It's like if you build a house really cheap, then the area builds up around it at a much higher price per square foot. You have a $500 mortgage payment, but your neighbor's is $2,500. Are you going to sell your house based on what you paid? No, you're going to sell it based on what your neighbor paid.
Buffett made his. He's free and clear. It's easy for him to play class warfare after the fact; same for Hollywood, btw.