I don't want a Denny's or a pawn shop or a Wal-mart next door to me, but that's no reason to make those businesses illegal.
But if you had a legal brothel zoned for its own location, I may start up a business in the same plaza. Perhaps an after hours bar, or an after hours diner where customers or even prostitutes of the brothel could come to.
Or I may even place a competing brothel next door and the two of us could compete for business but gain the advantage of nearby location of our businesses together, such as increased police patrols to deal with dangerous situations. Much like what is done with areas with many night clubs or the diamond district in New York City.