He SHOULDN'T have power....that is different than the fact that he DOES.
The question you ask is akin to this situation.....a crime boss who NEVER, and I do mean never, actually pulls the trigger in murders he wants committed is no different that Carlson in this situation, at least according to your line of thinking. By all rights, he shouldn't have the power to influence someone to do something like that, but somehow he does. The same holds true for the words ANY of us speaks. In a perfect world, each person would have the capacity to think for themselves and not let the words of someone else drive us to stupid ****. That, however, would require us to be in a perfect world, which we aren't.
The simple point that the OP was trying to make was that when the words one speaks are indistinguishable from the words of racists, its not that out of line to call what you said racist, even if you aren't one as a person. One need not be a racist to express racist remarks.