Here's what we did...
DH and I (FH at the time
sat down and *really* discussed all the pros and cons of having so few guests as well as a Disney wedding. My family is very small, his is very large, so while it was okay with me, I wanted to make sure that he wouldn't have any regrets.
After we agreed that we would be okay with the number of people, we decided that although his family consisted of several more people, we wanted to split our guests evenly - 4 from my side, 4 from his. My family was fine with this, as was his (surprisingly, we were extremely worried that this would *not* go over well with his family).
Now, one event that helped us quite a bit... His sister got married about 6 months before us and had a huge affair - 100+ guests, full Catholic Mass, mansion reception. So we were seen as the good guys for not competing with her wedding as well as getting to see his entire family en masse then. We were approached at her wedding by several family members telling us they thought that we had the right idea with an intimate wedding, all positives, no negatives like "how dare you not invite us", etc. Personally, we watched everything thinking "we are *so* we're not doing this, that, and the other". We saved a lot of $, and really felt that we had time to spend with our guests.
In the end, everyone was happy and most importantly, we were happy with our decision and didn't regret having so few guests. Everyone was very supportive of us, and while we never ended up having any type of reception at home, we never missed that either.
However, every couple is different. While we were happy, I know many of my friends who wouldn't have been. It takes a lot of careful thought and honesty from both of you.
Good luck, I'm sure whatever you decide you'll be happy