Yeah, I don't see it.
There is no moral requirement to have sex exclusively for procreative purposes, or for everyone to have children, or to have as many children as physically possible.
"Natural ends" are irrelevant. Reproduction solely for the sake of reproduction is not an ethical principle. It is a choice that an individual can make, and a biological impulse, but there is no requirement to reproduce so that your children can reproduce, and their children can reproduce, and so on.
If I have sex purely to engage in pleasure, this in and of itself causes no harm. It does not stop me from having sex at some other time in order to procreate. It does not convince me not to have children at some later date. It does not change my neighbor's mental calculus about whether or not s/he should have children.
One of the problems that California's Prop 8 had in court, by the way, was precisely this issue -- its defenders were incapable of explaining how any straight marriages were actually harmed, let alone altered, by allowing same-sex marriage.
It's not a zero-sum game. The decision to pursue sex for pleasure does not prevent or exclude sex for procreation. No harm, no foul, no justification for regulation.