I think its because its the "root" of the slang term in question.
My maternal grandmother, rest her soul, was tbe sweetest woman you ever met, and horridly racist. I think it was because she was a housekeeper in the south during the depression, and at that time that was a black persons job, so she overcompensated.
The vitriol with which she said the word "nigra" was simply chilling.
Personally, the word is hurtful to some, so I refrain from using it for the most part. If I'm singing some song with the word ("ah" version not "er") I don't edit it, but I don't say it with extra glee either.
And I've used the "familiar" "ah" version in mixed company when it was too perfect to pass up and the company knew me well enough to not be offended. But only as a turn of phrase, "n**** please", for instance.
As far as the discussion in general, if the black community finds the word offensive then don't use it. The whole idea that a word can be reserved to one race is extremely racist, IMHO.