You don't tell the driver where to take you; you type the destination into the app when you request the ride. Type "Disney's Fort" into the app, and select "Disney's Fort Wilderness Resort and Campground." There is only one place at Fort Wilderness that is accessible by private vehicle, and that is the same place where the riding stables are located. If the driver has any confusion, tell him to follow signs to Fort Wilderness and go thru the guard gate. Check-in for trail rides is at the building on the edge of the parking lot, to the right of the bus stop.
As PP said, do NOT go to Wilderness Lodge; it's an entirely different place. But if you somehow find yourself at Wilderness Lodge, there is a bus from there to Fort Wilderness.