I wish I had taken my daughter when she was younger - we only started going just before she turned 6yo. The thing is, I would remember it even if she didn't, and I would LOVE to have pictures of her in front of the castle when she was smaller...so we could see her "growing up" in WDW.
Anyway...