The entire opposition to the Mosque is based on essentially a reincarnation of Jim Crow against Muslims. They are basically saying "you can worship just don't do it here". This argument is no different than the argument used against blacks during the Jim Crow era. The fact that these demonstrations had/have racist undertones is absolutely no surprise. They even harassed an African-American man, claiming they thought he was a Mosque supporter when in fact he never said his opinion on anything whatsoever. In reality it was because he looked like a Muslim.
I hope the Mosque gets built just to spit in the faces of all the bigots opposing it.