It's such a non-issue. Blatant racism, as a whole, these days are limited to little groups like this. The Klan is not a "violent" group anymore. All they are anymore is a bunch of inbred white hicks sitting around some rural church or VFW post, reminiscing about the "good ol' days".
If this pastor wants a whites-only meeting, go for it. It's not hurting anyone, and it's insignificant on any possible social level you can think of, except maybe local - and that's because it's a podunk area.