I don't think you have a "Right" to build your church wherever you like. You have a right, as does every other citizen, to build a building that fits within the legal zoning codes. People have a right to protest that if they wish.
Building a mosque or a church is not free exercising the right in regards to freedom of worship.
So for me the question would be this. If a pizza place was being built, would it be able to be protested? Yes. If it was a gun range being built, would it be able to be protested? Yes. If it was an apartment complex being built, would it be able to be protested? Yes. If it was a pool, a YMCA, or a homeless shelter could it be protested? Yes. Then I see no reason why a church or mosque can't be protested.
Unless these people are somehow keeping these individuals from attending another mosque, praying at their home, etc, to me its not infringing on anyones right to worship.