There are very good reasons to support or oppose AA which have nothing to do with racism. Probably you can bring up certain good arguments against which are not racist.
I can try to give a good argument in favor: Due to existing racism among the white population, blacks statistically are facing a disadvantage which lowers their prospects and opportunities. For example, white employers will less likely hire black employees, white teachers will give blacks worse grades, statistically. Affirmative Action is just a means to pose a counter-weight against these disadvantages for blacks.
You may believe this means is flawed, you may also believe this brings more bad than good. But unless we make empirical studies on that field, we won't know how well it works or how unefficient it is. We cannot know on the spot who is right. So you are free to disagree, but please pay people who believe AA is a good thing to do the deserved respect, instead of calling them "racists".