We just returned from a Walt Disney World trip with our five young children. Meeting the characters is an absolute must for us, and the character meals have always been a great way for us to do that. So I'm not sure that we could ever feel right about taking a break from them.
That being said, I have to say that I agree that the buffets are rather chaotic with small children, and I dislike the fact that character meals seem to skew toward the buffets. However, we did get to meet characters without the buffet at about half of our table service destinations this time: Akershus, Be Our Guest, Cinderella's Royal Table, and Garden Grill. While all of our experiences, even with the buffet chaos, were awesome, I'd say the true table service restaurants that offer character greetings were just a tad more awesome.
Definitely when we plan our next WDW trip, we'll be leaning toward true table service as much as possible and only going with buffets when absolutely necessary to get a character experience we feel we can't miss. I'd say we consider Chef Mickey's to be iconic enough that we'd probably do it regardless, but I'd be fine with avoiding the other character buffets we did -- Crystal Palace, Hollywood and Vine, and 1900 Park Fare -- if those same character experiences could be had elsewhere. But all of those characters at those places -- the Winnie the Pooh gang, the Disney Junior gang, and the stepsisters -- are pretty much "must meet" characters for us.