1.)You've not stated why people should not be held responsible for their associations.
2.)If you were a member of the KKK but said you didn't participate in lynchings, you'd still have a lot of splainin to do. If you were a member of the Nazi party but said some of your best friends were Jewish, you still might be judged accordingly. The same is true if you were a Fascist or a Communist.
3.)Muslims belong to a religion that hates Jews, Christians, 'infidels', murders Gays, control the way women dress or even whether they can appear in public. They have murdered little girls on the way to school, sent them into burning buildings, bombed innocent people eating a pizza or doing their shopping, or waiting at a bus stop, and the list goes on and on. There is no intellectual class.
To belong to any 'religion' like this is inexcusable, and should be discredited at every opportunity, just like any other hate group.