One man one woman or interracial persons were not the issue, it was eqaul protection for all. The very same constitutional principles that upheld Loving are the same that will uphold SSM. Equal protection under the law.
I though Loving would be sufficient because you argued that marriage was not in the constitution.
Here , here's Widsor...
Windsor v US
What has been explained to this point should more than suffice to establish that the principal purpose and the necessary effect of this law are to demean those persons who are in a lawful same-sex marriage. This requires the Court to hold, as it now does, that DOMA is unconstitutional as a deprivation of the liberty of the person protected by the Fifth Amendment of the Constitution.
The liberty protected by the Fifth Amendment’s Due Process Clause contains within it the prohibition against denying to any person the equal protection of the laws.See Bolling, 347 U. S., at 499–500; Adarand Constructors, Inc. v. Peña, 515 U. S. 200, 217–218 (1995). While the Fifth Amendment itself withdraws from Government the power to degrade or demean in the way this law does, the equal protection guarantee of the Fourteenth Amendment makes that Fifth Amendment right all the more specific and all the better understood and preserved