The town really comes alive in the evening. Hands down, the best thing to do there is people watch. Get to Mallory Square before sunset to enjoy the buskers. They are spread out in front of the hotel next to it as well. There is free entertainment everywhere. Many bars have great bands and since they are open air, the volume is better from the outside of most venues. Grab a seat on a bench and enjoy.
We walk the entire waterfront at night, grab a chocolate covered key lime pie slice from Kermits, listen in as the ghost tours pass by, and just soak up the atmosphere. There is a lot of activity on Duvall street as well.
Plan on staying out late to really experience the magic of Key West.