The reason what Sterling said was so horrible is the simple fact that it is part of a historical racist mindset that does go back to the days of slavery.
Sterling's words could easily be translated to:That is what troubled the majority of people. And that harkens back to the slave days when slave owners would become friendly with their slaves, some to the point of having children with them, but if ever one of them became too familiar or did it in front of someone else, they would be put in their place. Those scars still exist in the black community so while some might hear it as the ranting of old rich crazy man, others hear it as the haunting of time we are not quite over yet.Blacks are good for working, making you money, being friendly with and even sleeping with. But don't you dare elevate them to the same status as whites.