That's exactly what I'm saying, as it's exactly what happens in real life. When a lawyer or businessman is laid off from a $500k/year job, do they immediately take whatever job is available to them, regardless of pay? Of course not. They will spend weeks, months, or even years looking for a job with commensurate pay and responsibility. I know literally tens of people who have done this exact thing over the past few months and will probably be doing it for the near future. Every one of them has the skills necessary to get jobs making $10/hour somewhere, but it's a terrible career move and not something they can really consider.
This effect is even more pronounced among lower income groups. While that businessman might eventually relent and take a job paying $75k as a middle manager somewhere, someone laid off from a job making $60k a year is not likely to rush to accept a job offer at $5/hr.
In your hypothetical world where there is no minimum wage, the discrepancy would be even more pronounced. Many people would simply decide that a job offering them $3/hr is not worth it at any point whatsoever, and that their time would be better spent looking for a job closer to their previous $60k salary.
In theory, after some undetermined amount of time, some may do this. That's a long way from what was being discussed.