so if I am black and the stores in my town don't want to serve me, then I have to drive however many miles it takes or move and you think that right should be protected?
It's a matter of competing rights. If we are to run over one person's rights, we have to have good justification for doing so.
There are three issues at play.
1. The right to Association, which carries with it an inverse right
not to Associate. Generally, we should no more force someone to sell to you than we should force you to buy from them (a law stating that all bakeries have to support same sex marriages is the equal inverse of a law stating that all gays have to support Chik Fil A, and both are abusive. That doesn't mean we can never pass them - sometimes the need can be truly terrible enough that we can force people to buy chik fil a, join the military, buy health insurance, etc.).
2. Freedom of Conscience and Religion, which is more strongly protected, because of its primacy. If you believe selling to someone of a different race is immoral, then no, we shouldn't force you to do so unless we can show a compelling state interest (a compelling interest is that which relates directly to core Constitutional issues) in using the threat of violence against you to violate your will and force you to do so, and when we do so, we are required to do so in the least invasive/restrictive manner.
3. The State has a compelling interest in ensuring that it's citizens are not locked out of entire industries. This has two expressions:
A: If a group is effectively denied access to a service (note, this is not the same as "denied the ability to use a particular provider of that service") because all providers refuse them, the state has an interest in forcing at least some of those providers to do so.
B: If a group is effectively denied access to an industry (for example, if we were to pass a law stating that all Grocers must first deny that the Lord their God is One, or handle and sell pork products, we would be effectively banning Jews and Muslims from being Grocers)
So, if (for example) every single provider of a service refused to provide to you because of your race within an area accessible by you,
then, yes, the State can make the argument (which does not mean it will win - if, for example, the providers in question is the one prostitute in a city, you still don't have the right to force them to have sex with you) that a compelling interest allows them to force a provider to provide.
If, however, there are half a dozen bakeries within a 10 mile radius who would love to make your Nazi or SSM or Race-Mixing or Whatever-Hot-Button-You-Like cake..... then no. There is no government interest in violating the rights of those particular bakers.