They have to be able to prove he's avoiding the women because they are women.
What's so totally awesome, is my industry is like 92% male, so I don't really need to purposefully avoid women, statistically I don't have to deal with them.
I've been in a position where I've trained drivers for employers I've worked for, and can you see why I might be uncomfortable with the idea of having to go in a vehicle, equipped with a sleeping bunk no less, alone with a woman for several hours, that is way outside my comfort level. I would outright refuse unless the company agreed at their cost to install a driver facing always recording camera in the truck. I don't believe that is objectively unreasonable. the last thing I want is he said/she said allegations.
it shouldn't be grounds for termination, you can't put up these zero tolerance sexual harassment policies and then give men with no interest in sexually harassing women no tools to avoid even the appearance of an impropriety. And nobody seriously believes that any less then 99.99% of women in the workplace just want to work, but it only takes one allegation to destroy careers, marriages, etc. And rumors can be very costly to the company as well as involved employees. so is OP evil for not wanting a piece of that problem? I don't think so.
and not only that, but the fact he has to secretly avoid these projects and can't just bring this issue to his supervisor is a good reason something is wrong, because a company should be willing to work with him in making sure there is adequate protection for both employees to prevent problems, like making a project involve three people instead of two, or require the project be done in the conference room instead of an office or what have you.
Think about it, a police officer who arrests a female suspect has to radio in his mileage and starting point and check back out when she's been dropped off at jail, is to create a record so that the timing can show nothing happened.