Another -- rarely discussed -- aspect is that the perpetrators of hate crimes are far more likely to come from among groups which are privileged (on the axis in question) because of the sense of invulnerability or entitlement that comes with such privilege. It's much harder to maintain a pretense (let alone sincere belief) in something like black supremacism while living in the United States, because the concrete indicators of financial success and political influence (i.e. income, holding positions in high office, heading up big business, etc.) all contradict such a doctrine. This doesn't absolutely prevent such a doctrine, but it does make it more difficult to hold on to. White supremacists, on the other hand, can point to real, concrete, pervasive advantages of "whites" over nonwhites in many aspects of daily life (which they then warp and distort to lend ideological support to all manner of nonsense). Put another way: you're less likely to believe both in the righteousness AND in the likelihood of getting away with something if and when your political and economic context either doesn't contradict that expectation, or at least does so only weakly.