Not all of them.
Just like there were northerners who fought for the Confederates, that weren't slave owners and even opposed slavery. John Pemberton, commander of the Vicksburg garrison was a Pennsylvanian and opposed slavery. Pat Cleburne, an Irish immigrant; he opposed slavery. General Cleburne and 13 of his officers petitioned the Confederate government to abolish slavery in 1862. James Longstreet had pushed the agenda of freeing the slaves, then declaring war on The United States.
The opposite side of that coin, Mary Todd Lincoln's family were slave owners. The Emancipation Proclamation exempted slaves that were held in Union states.
There were more freedmen living in the south than in the north. Before the end of Reconstruction, freedmen had the right to vote in the south, but not in the north. There were freedmen in the south that owned slaves. Jim Crow was a result of Reconstruction. If Reconstruction hadn't been so brutal, Jim Crow probably wouldn't have happened.
Earlier, you said that former COnfederates were lucky they weren't exterminated; well, everyone knew that would be a bad idea, because it would have launched a guerilla war that would have lasted for decades. It's the reason that Lee was allowed so much lee-way at Appomattox: Grant was afraid that one mistep in the surrender process would send the Army of Northern Virginia into the mountains to carry on the war. Grant knew that his only single advantage over the COnfederates was is nearly unlimited source of warm bodies to use as cannon fodder, but also knew that that source wasn't infinite. There were more Federals dieing in battle than Confederates and hew knew that public opinion wouldn't stand for that, for long.
You say that the war was fought over slavery, but the historical facts tell a much different story.