Yes, every single person in history wanting to restrict another's freedom has started with the claim that the other person doesn't have freedom. Black people didn't have the right to be free. Women didn't have the right to own property. Japanese-Americans didn't have the right of due process. Because if you were to say there is a right to marriage, your fighting tooth and nail against someone else's most personal, intimate choices would look awfully bad, wouldn't it?And yes, no one can suppress something that does not exist.
You'll forgive me if "restricting that action is justified because I say they don't have the right to it" isn't a compelling argument in of itself.