That might be the case- and I can imagine that most such owners just state that they turned down the individual for another reason.
At the same time, I believe there was a Court decision involving an Amish man who owned a carpentry business and who refused to women and who would also not hire non Amish men. If I remember correctly, the decision noted the size of the business and that the owner worked on site.
Likewise, I think there was another earlier case involving an Ultra Orthodox Jewish small business owner with near the same policies (would hire women in theory, but only for what he determined was a gender appropriate position- and the woman then had to comply with all Orthodox rules regarding inter gender contact while at work. This then negated her ability to do the job, so she was fired). The earlier court also cited the size of the business and that it was "owner occupied".
What I do know is that small, owner occupied boarding homes are exempt the anti discriminatory provisions of the Fair Housing Act:
https://fairhousing.foxrothschild.c...mrs-murphy-exemption-to-the-fair-housing-act/ .
Likewise, so are some small scale rental property owners are also exempt from the anti discrimination provisions:
https://fairhousing.foxrothschild.c...-fair-housing-act-not-many-but-here-are-some/
So, it could be possible that some small business owners are exempt the anti discrimination laws. I just don't know the specific cases.