1. No one is refusing to serve anyone. People are refusing to serve gay weddings. This is an important point that opponents would do better to recognize and deal with honestly, as trying to pass over it only allows proponents of religious freedom to make that point and demonstrate a false presupposition.
2. If a devout Muslim or Jew doesn't wish to serve a woman, they are welcome to open up a business where men only serve men and women only serve women. The rest of us have no more right to trample on their faith with our beliefs in this scenario than we would to force a Muslim grocer to carry bacon and beer, or a Christian bookstore owner to carry porn.
3. The distinction between those who discriminate for religious reasons and those who do not is an important legal one. The government has to justify it's actions when limiting our freedoms, after all, and when it comes to the State limiting our
religious freedoms, the standards the government has to meet are
quite strict indeed:
First, the burden must be necessary for the "furtherance of a compelling government interest." Under strict scrutiny, a government interest is compelling when it is more than routine and does more than simply improve government efficiency. A compelling interest relates directly with core constitutional issues.
The second condition is that the rule must be the least restrictive way in which to further the government interest.
So the government would have to argue that:
A) gay couples being able to access wedding cakes is a core constitutional issue for the state government, and
B) Only religious bakers who refuse to provide service are available, and there are no other accessible bakers for the gay couples to access, and therefore forcing a religious baker to make the cake is the least restrictive way of addressing that core constitutional issue
it's not a matter of "it's not fair" or "but I think it's mean".