My 4 year old loved Thunder Mountain, and pretty much all the rides he went on our first couple days at the parks, so our second day at MK he desperately wanted to try Space Mountain. He was just barely tall enough, but was able to go on. Well it really freaked him out, and he basically refused all other rides the rest of the day. He would do shows and things like Carousel of Progress, but he wouldn't go on any other coasters or faster rides. Haunted Mansion also scared him quite a bit.
I don't regret letting him try though! If he had said he didn't want to go on Space Mountain, that would have been fine but he wanted to try and he was proud of himself for going. After his Space Mountain freak out, when he said he didn't want to go on Splash Mountain and other rides, we didn't force him and it was fine. I think just let your 4 year old be in control and go with the flow a little bit.