A very common ground level definition of racism is "one who dislikes a person, or hates a group, solely because of their race." This usage is not defined in the dictionary and yet it is widely believed to categorize racists from non-racists.
The ability to cast a wider net and make more people racists is seen as a powerful tool in shaping society towards a preferred direction. Using racist as a shaming word can get people to modify their behavior. So control of the language is a political act. When you equate validity with a political act, you're making a false equivalence.
Your grandmother thinks, I assume, that racists are people who hate people of other races. She doesn't think of herself as a racist because she doesn't have any hatred to blacks. You're condemning her as a racist because you agree with the political definition of racism being the observation of statistically significant behavior patterns observed within a group.