Really? Please explain why you think the idea that Congress may trim the Supreme Court in a fight is absurd. In Ex Parte McCardle, the Court made clear that Congress may limit the appellate jurisdiction of the Supreme Court--and almost all the Court's jurisdiction is appellate. And as the Court noted, jurisdiction means the power to hear cases. In McCardle, Congress completely removed the Court's power to hear the case before it, even after the justices had already heard oral arguments.
Of course Congress was not usurping anything in McCardle, but rather exercising a power the Constitution gives it. The fact the Constitution devotes Article I to the Congress, but only gets to the Supreme Court in Article III, is no accident. It reflects the relative importance the Framers meant the two branches to have. Congress has several means of clipping the wings of federal courts, including the Supreme Court. So does the President, to a lesser degree. He can make a Supreme Court decision a dead letter simply by declining to enforce it, as President Lincoln did with Dred Scott v. Sandford.
If the Senate were ever to fail to appoint so many nominees to the Supreme Court for so long that natural attrition started to reduce the number of justices on the Court by very much, many Americans would no doubt want to put an end to it. And they could do that either by voting the Senators responsible out of office or by removing them through the impeachment process, and replacing them with ones more to their liking.