Propery rights have been used by those on the far right to oppose lots of things that progressives favored including
*** a defense of slave ownership
*** a defense of secession by Southern slave holding states
*** opposition to Civil Rights legislation in the Sixties
*** labor rights for the last hundred and more years
It is a fact of history.
If libertarians want to apologize for their stand and change it, that would be welcomed.