The US Supreme Court does not agree with you.
Game.
Set.
Match.
Madison actually says, that it is “a plain principle, founded in common sense” that The States are the final authority on whether the federal government has violated our Constitution! Under his discussion of the 3rd Resolution, Madison says:
“It appears to your committee to be a plain principle, founded in common sense, illustrated by common practice, and essential to the nature of compacts; that where resort can be had to no tribunal superior to the authority of the parties, the parties themselves must be the rightful judges in the last resort, whether the bargain made, has been pursued or violated. The Constitution of the United States was formed by the sanction of the States, given by each in its sovereign capacity. It adds to the stability and dignity, as well as to the authority of the Constitution, that it rests on this legitimate and solid foundation. The States then being the parties to the constitutional compact, and in their sovereign capacity, it follows of necessity, that there can be no tribunal above their authority, to decide in the last resort, whether the compact made by them be violated; and consequently that as the parties to it, they must themselves decide in the last resort, such questions as may be of sufficient magnitude to require their interposition.”
A bit further down, Madison explains that if, when the federal government usurps power, the States cannot act so as to stop the usurpation, and thereby preserve the Constitution as well as the safety of The States; there would be no relief from usurped power. This would subvert the Rights of the People as well as betray the fundamental principle of our Founding:
“…If the deliberate exercise, of dangerous power, palpably withheld by the Constitution, could not justify the parties to it, in interposing even so far as to arrest the progress of the evil, and thereby to preserve the Constitution itself as well as to provide for the safety of the parties to it; there would be an end to all relief from usurped power, and a direct subversion of the rights specified or recognized under all the State constitutions, as well as a plain denial of the fundamental principle on which our independence itself was declared.” [emphasis mine]
A bit further down, Madison answers the objection “that the judicial authority is to be regarded as the sole expositor of the Constitution, in the last resort”.
Madison explains that when the federal government acts outside the Constitution by usurping powers, and when the Constitution affords no remedy to that usurpation; then the Sovereign States who are the Parties to the Constitution must likewise step outside the Constitution and appeal to that original natural right of self-defense.
Madison also says that the Judicial Branch is as likely to usurp as are the other two Branches. Thus, The Sovereign States, as The Parties to the Constitution, have as much right to judge the usurpations of the Judicial Branch as they do the Legislative and Executive Branches:
“…the judicial department, also, may exercise or sanction dangerous powers beyond the grant of the Constitution; and, consequently, that the ultimate right of the parties to the Constitution, to judge whether the compact has been dangerously violated, must extend to violations by one delegated authority as well as by another — by the judiciary as well as by the executive, or the legislature.”
Madison goes on to say that all three Branches of the federal government obtain their delegated powers from the Constitution; and they may not annul the authority of their Creator. And if the Judicial Branch connives with other Branches in usurping powers, our Constitution will be destroyed. So the Judicial Branch does not have final say as
“…to the rights of the parties to the constitutional compact, from which the judicial as well as the other department hold their delegated trusts. On any other hypothesis, the delegation of judicial power, would annul the authority delegating it; 10 and the concurrence of this department with the others in usurped powers, might subvert forever, and beyond the possible reach of any rightful remedy, the very Constitution, which all were instituted to preserve.”
