The only thing that is particularly bothersome is your unwillingness to answer his question without you poisoning the well. If to even ask your opinion is itself racist, then conversation ends - which, one supposes, is at least your subliminal intent.
For those of us who have not spent the last 15 years in the wilderness finding a new coded rhetoric to discourage unwelcome questions the answer would be very simple:
A false accusation of racism is not necessarily racist, but it is at least as, or perhaps MORE, morally offensive than when an expression of racism is found. Not only is far more reputationally damaging to an individual, the consequences of a false charge of racism also enables the authentic ideological racist to exploit the increasingly broad and reckless usage against innocents. Just as McCarthy and his supporters reckless charges of communist agentry against ordinary liberal critics had the perverse effects of lending cover to those actual agents, who claimed they too were merely persecuted and idealistic reformers.
So yes, a false accusation is at least as bad as the alleged sin, if not far worse.
One's differentiation should be honest. There is the usual and commonly understood dictionary definition since the massive moral lessons of WWII, and there is the ludicrous "power structure" definitions developed by the tenured radical leftists of the last 20-30 years. The latter are not 'scholars' in any traditional sense, they are the product of a humanities departments "literary" fads and "studies" departments that have developed a specialized jargon and pet narratives of post modern clap trap.
The traditional understandings of race and prejudice were too simple, and many big issues remedied, from the 60s. So they since then they have spent decades getting grants and fat salaries searching for racism, sexism, patriarchy, ableism, genderism etc. with broader defintions, uncovering its "subtle" and "hidden" forms everywhere. And from this spun lots of new fads since the 90s: white privileging, intersectionality, whiteness critical studies, structural racism (which is now broader than the prior institutional racism).
And with all these, new faculty positions and departments for the socalled "scholars" of identity politics.