We took our older boys when they were 12 & 14. We had a 1 & 2 year old at that time as well but they stayed home with Grandma, Auntie, and a few other trusted babysitters. That trip needed to be about the older boys for many reasons (that I won't go in to). We figured we'd take the younger kids when they were about 10. However, by about day 2 of that trip we realized we needed/wanted to bring them when they were younger...but we also knew we didn't want to deal with a stroller and we wanted them to be able to easily go on the majority of the rides. So our benchmark was that they needed to be 40" tall. That milestone was reached by the time the youngest one was 3.5 and we went to Disney when she was 4 (and her brother was 5). That age (4 or so) was perfect for us -- the kids didn't need naps, they didn't need a stroller, they could walk for long distances without complaining, and we didn't have to worry about diapers, potty training, or special food. I found out I was pregnant just before that trip and we also took that child when she was 4. We toured very leisurely when the kids were that age but have done several trips since then and our touring plan has gotten more intense.
That being said, we recently went to Disney with our son and his wife and 1 yo daughter (she was 14 months old at the time). She was a very well-behaved baby on the trip. She napped in the stroller and was also content to stay in the stroller and just take everything in most of the time. She had a few little meltdowns, but was calmed quickly and easily. However, having to deal with the stroller all the time and having to stop for diaper changes and nursing sessions (usually when she was tired) really slowed down our touring style...significantly. But that's ok...we just had to readjust our expectations for the trip (which we knew we would have to do before we even got there). Overall it was a very enjoyable trip but we still wouldn't change what age we took *our* kids to Disney for the first time. Being the grandparent is a different ballgame though since we weren't the ones responsible for the baby and all associated gear.
ETA: I would probably handle things differently if we went to Disney frequently (once or twice a year). If that was the case, I would probably take a child as young as about 6 mos. However, in our case, we thought we'd only be making 2 or 3 trips during our lifetime so I wanted the kids to be old enough to remember as much as possible. Of course that was 8 trips ago.....