The other downside is how much is costs to update a large home. More space means more money. Everything costs more. More windows to cover, more paint to buy, ect ect ect.
My Grandma pointed this out to me a few years ago too. She was right, LOL. We have a huge picture window in the living room - it took an extra-long rod + 6 curtain panels. When your rooms are bigger, even an 8x10 area rug might not be big enough. I had sticker shock at how much we spent to paint this whole house compared to previous - there's just more wall space. More carpet.
Our current house is 2000 + another 2000 in the unfinished basement. It's a little big for the two of us right now, but I could see where if I had 2-4 kids here I'd either want a slightly larger house or finished basement space so we're not all right on top of each other.