Robertson's biggest mistake with that quote was saying "they," as if he really had his finger on the pulse of the black community in early-60s Louisiana, and acting as if that was representative of the entire south. If he had just said "the people I knew," obviously not a first instinct, he's likely fine.
I don't think his racial comments were all that offensive. Ignorant, to be sure, but not offensive ... I'm willing to give him the BOTD that maybe he didn't know anyone adversely affected by Jim Crow. In certain areas of the South (mostly small towns where the white people weren't total assholes), it wasn't unheard of for the white folk to just ignore those laws. I think his gay comments were far worse than his racial comments.
I think his boiling homosexuality down to a preference for ass instead of the V is far more ignorant than saying that he, personally, didn't know any black folks who were upset about racist laws. He was born in 1946; the CRA didn't come down the pike until 1964; the last of Jim Crow ended in 1965. We don't know at what age he was pickin' cotton with the Negro folk. And the JC laws were far worse in the cities than they were in certain small towns (and, of course, they were far worse in OTHER small towns that I wouldn't go to NOW if I was a black guy).
He never said "blacks were happier before civil rights;" he also was an idiot for implying that because the few he knew weren't upset, that it was a representative sample of anything other than his anecdotal ignorant bull**** and acting like everything was hunky dory.