I think what he said was contemptible and should have 'some' consequences. People can react, condemn and show their disapproval in a legal manner.
But it was an expression of his opinion on a matter, not active discrimination. As distasteful as words of racism and bigotry are, if you make them out of bounds and you start censoring words and opinions, the list will grow and grow. There's a reason they don't limit free speech, because trying to use thought control never works out well. Naturally, you can't scream 'bomb' in an airport, disturb the peace or do other verbal escapades that lead to harmful action but that is enough.