The hallmark of rational debate is not that no one involved ever makes a mistake. It's that the debate centers around facts and logic. You can quibble with Ben Shapiro about how he interprets a set of facts, but you cannot fault him on not taking stock of facts. Moreover, if you dispute how he lays out an argument (i.e., if you spot a fallacy), anyone is welcome to explain the error in the question segment. And if students are ill-equipped to deal with this issue, it reflects very badly on their professors -- who, by the way, are welcome to see the lectures and are invited to voice objections during question segments.
Another note: your position requires someone to censor Shapiro and Coulter. You don't need to censor someone who is wrong, as long as you can respond to them.
First of all, putting these conservative figures in the same group as Klansmen and neo-Nazis is despicable. Klansmen and neo-Nazis have been involved in assaults, rape, and murders to name just a few things. If you don't fear to attend a neo-Nazi march considerably more than you fear to talk with Ben Shapiro, your sense of proportion is out of touch with reality. I'd go talk with Shapiro alone without a problem. But, a Nazi march? I don't know if I would feel safe, even if the army was there. The worst conservative pundits ever did was say things you don't like... That might be annoying, but it's fine.
Second of all, a college has no requirement to invite people over. However, if a group of students thinks that public figures like Shapiro, Knowles, Coulter, Prager or someone like that should be part of a discussion, they should be free to raise the funds and organize their events.
Third of all, conservatives students can make similar comments about faculty, no less than public speakers who are often invited. Imagine if things were reversed and they had the power to grant or deny access to college. Even if a group of conservatives thinks that Judith Butler is a presumptuous quack who uses indecipherable prose to hide the blatant idiocy of her identitarian claims, don't you think someone who wants to judge her work for themselves should be allowed to invite her?
Fourth of all, American campuses are all biased in the same political direction. If you had to let more people in of some kind, conservatives would probably be the best way to put a limit to the echo chamber effect, even if they're wrong in one way or another.
Here's the reasonable line: Shapiro never promoted violence of any sort, so he should be allowed to go anywhere he is invited in due form. He repeatedly condemned violence and repeatedly called people to not intimidate or insult each other over differences related to things like sexuality and religion.
Your criterion is problematic because campuses are staffed, populated and managed by mostly left-leaning people. You cannot trust that they will do this job properly, even if they tried. If you try to get rid of every "bigot" that could slip through, you'll end up giving some groups on campus too much discretion and they will use it to ban ideas they dislike.
Actually, Ben Shapiro has often been challenged by professors during his presentation. Presumably, you would expect they could come up with good arguments, but I've never seen one. I could stump him on economics (because I am a trained economist currently studying for a Ph.D.), but I suspect it would just be a polite disagreement over how to interpret some results. Or I would point to a few studies, key results and he would just concede the point and thank me for references -- which I have seen him do on occasions, by the way.
Contrast this with protestors. He's polite, calm, and invite people to talk. They are loud, violent and want to prevent people to talk... But, in your head, Shapiro is the problem?