I'm fairly certain that cracker specifically refers to racist white people, or at least a reference to the systematic oppression of blacks. There's really no equivalence between the use of a pejorative like this by a member of an oppressed minority group and the use of one by a member of the majority in power. It's never the word that matters, it's the systematic discrimination that it represents. And despite the whining of a few people who confuse loss of privilege with discrimination, there is no systematic discrimination of whites. Or males. Or Christians. Or heterosexuals. But there is systematic discrimination of basically everyone else.
In the case of the Tea Party, the shoe fits. They endorse a lot of racist positions. That is, they, a group almost exclusively composed of white, heterosexual Christians, endorse a lot of positions that hurt anyone who isn't white, heterosexual, or Christian. Also they endorse a lot of positions that hurt women, but I have no idea about the gender demographics of the Tea Party.
There is, again, simply no equivalence. Calling someone a cracker is not the same as pretty much any other slur, because cracker has never been indicative of systematic oppression. That's why the left isn't outraged. It's never the word that stirs outrage. It's the systematic oppression. Meanwhile the right gets up in arms over the words, and supports the systematic oppression.