When you have a business the government regulates where it can be located, the type of building you can use, the type of sign you display, how you can fire and hire people, the minimum wage, how you dispose of waste, and fire and other safety requirements. You are also required to document your income and expenses to pay your taxes and get deductions. If you have a restaurant, you will be inspected to make sure that the food is protected properly from spoilage and contamination, that the employees wash their hands and wear proper clothing and hair nets etc, that you have pests under control, provide access to the disabled, have accessible restrooms and much more.
Out of all these regulations and requirements, none of which are loved by conservatives-libertarians, the one mentioned most often as onerous and unfair is the requirement that employees and customers are not subject to racial, religious etc. discrimination by business. Ironically, while advocating for allowing discrimination in public accommodations, they also argue that such discrimination would not happen very much, if at all, because people have changed since the Jim Crow days and it would harm the business.
My opinion is that the harm from business discrimination against potential employees, employees and customers outweighs the harm from being "forced" to tolerate undesirable races, religions etc in one's business. I don't believe that such discrimination will be as rare as claimed. These days it is as likely to be directed at Muslims and gays as blacks, but the tendency to ignorantly, irrationally and/or arbitrarily discriminate has not been sufficiently eliminated yet. For evidence, just look at all the racism and bigotry expressed on this forum and other public fora.
Also, I am willing to recognize the damage that it would cause to society and vulnerable people, something the conservative-libertarians ignore or downplay. In small towns, isolated and rural areas, just one or two discriminatory businesses could keep a significant portion of the population from getting a job, shopping, getting a place to live etc. without leaving town. Businesses have the ability to oppress people in their daily lives as much or more than government, especially in these days of a handful of mega-corporations, malls and big box retail dominating retail and services. Addressing government discrimination while allowing business discrimination requires tolerating discrimination and the hardship and oppression it will impose on unpopular minorities and society as a whole.