Who made up that rule? If you are uncomfortable when you get out of the pool for the 10 seconds it takes to wrap a towel around you, that doesn't mean that you will be uncomfortable swimming before that.
It's not a rule...it's basic science. Evaporation has a cooling effect that becomes way exaggerated when the air temperature is much lower than the water in which your body was just submerged.
I'm not trying to say that anyone is banned from swimming when the temperature is "X" outside vs. the water temperature. However, for me personally, wrapping a towel around me does
not suffice to warm me up, when the rest of me is still wet ALL the way back up to my room. That WDW air conditioning is fierce, so you can bet that on an iffy day (like 65-70 degrees), WDW will still be cranking the A/C in the hotel lobbies/hallways, and coupled with the cooler air outside,
*I* would probably not go swimming, which is not to say that no one can, but I just don't like being cold and wet like that. I have a lot of thick hair, and it stays wet for a LONG time, so I feel cold for awhile, and it needs to be HOT for me to want to swim.
I last went to WDW mid Feb 2007. It was SO cold that week (there was FROST two nights we were there) that the very idea of swimming was laughable). The temps during the day did not even go past 60, ever.
We are going back next March. Sure, I'd love to swim, and will plan for it (at the very least, the hot tub a couple times), but if it's not HOT, I will not be swimming. I have a pool here at my apartment complex at home that is heated year round, plus we live in a very warm climate, so swimming is not a "must do" for me on vacation.