Ideas are inanimate objects. I have no more problem with any idea being in a library than I would any table.
They are no more threatening than the voices in someone's head, and no more responsible for actions.
Do you have special knowledge about "voices in one's head"?
Just joking.
Seriously, ideas can be dangerous, because they have a direct relation to actions. Of course we can't and shouldn't "hold them responsible" as we do hold people responsible, that's trivial, a no-brainer: You can't put an idea on court or throw an idea in prison. That's obvious and not a big revelation.
But ideas matter. Why do you think companies invest millions, if not billions into advertizing? If ideas didn't lead to action, they wouldn't do that. And I doubt the Nazis would have had any chance, not even the slightest to get to power, if the same hate speech laws had existed in Weimar already as exist in today's Germany.
I agree that freedom of speech is a basic right and I am not comfortable with censorship or banning certain ideas. Both for ethical and pragmatic reasons (this will often backfire). So of course if there is censorship, extreme caution is warranted and it's better to err on the side of free speech, than on the side of censorship of dangerous ideas.
But I also believe that under certain conditions, certain speech and/or ideas are not just that. Depending on the context and situation, it may be incitement. When there is an angry mob and a gifted speaker rallies them up with calls for murder, the speech of that speaker is more than just harmless speech. It's fuel into a fire which will almost inevitably lead to action, much like good advertizing will inevitably cause enough people to buy a product.
Also, while freedom, such as freedom of speech, should only be limited by the freedom of others, certain speech may indeed violate the freedom of others. Think of libel, slander or call for murder against an individual or a group of people. These kinds of speech are illegal in most countries, as far as I know. A call for genocide, i.e., falls into the same category, IMO.