It's actually a really weak-knee argument to try and claim that "Vice users are oppressed". It's a societal vice. A vice activity is naturally destructive to society. You as an individual forfeit your freedoms when you choose to engage in vices that attack the greater society.
Taxes and laws are 'An eye for an eye' in this respect.
When you as an individual make a choice to do clear viceful damage to the societal collective then that societal collective has a right to open your bank account and 'charge you' for your societal trespass.
So the Libertarian argument is actually illogic attempting to pass itself off as logic when it is anything but. "I should be able to puff smoke in your breakfast at the cafe and insodoing rape society yet society should not be able to penalize me". Absolute illogic.