It's worse to create a culture where all assertions of racism are dismissed.
When you analyze how people behave in discussions on race and racism, you can often understand them better when you figure out who they identify with in whatever scenario is being described as racist. More often than not, a person of color is going to identify with the victim of the racism. They're going to recall similar events in their life or the lives of their loved ones when they hear about the situation. As a result, they're going to find the allegation believable or, at the very least, worthy of investigation. For example, many black people have been followed around stores by those who suspect them of wanting to steal. So, if a black person tells a story where a shopkeeper followed her around a store, but ignored white shoppers, the black listeners are likely to agree that racism was involved.
Now, when you look at the white (usually conservative) people who frequently dismiss allegations of racism as "playing the race card", you'll often find that they identify with the person or organization being accused of racism. They identify with the accused because, like people of color, they recall events in their lives or the lives of their loved ones that remind them of situation. However, in their recollections, they are not the victim of the racism, they or their loved one is the person being accused of racism in a way that they perceive as unjust. So, for them, the more pervasive - and thus greater - evil is being called a racist. Therefore, when they hear an allegation of racism, they're more likely to minimize or deny it because, in their perceived experience, allegations of racism are usually just some oversensitive liberals whining about nothing and misinterpreting things. (This is also why the same people are often racist as well - because they have no impetus to examine their prejudices. They just blame everyone else.)
For the record, there are also people of color who are reluctant to take allegations of racism seriously as well and that usually comes from internalized racism which breeds a desire to be accepted by the dominant culture, particularly people in the dominant culture who don't like "other black people" or "other Hispanics", etc..
