I am not a layman and I've worked in this field for over a decade. So I will try to explain and keep things comprehensible.
That seems awfully high. Either your source was waaaay off, or you are mis-remembering.
There is not and cannot be a set number..it's obviously going to change as the population changes. Also, it can be complicated because "jobs" and "employment" are different concepts and the jobs numbers are not used in the UE rate calculations.
There are 2 surveys in the Employment Situation Report. The "Jobs" numbers come from a survey of approx 145,000 businesses (557,000 individual establishments). The Jobs numbers exclude agriculture, the self employed, people who work in other people's houses, people working off the books, and unpaid family workers. It's also a count of jobs, so a person who works two jobs will be counted twice.
The Current Population Survey is a survey of approx 60,000 households. It includes everyone 16 and older and Employed are those who worked at least 1 hour for pay or 15+ hours unpaid in a family business/farm. It's a count of people and people are counted only once no matter how many jobs they have. It's also the survey that counts Unemployed and those Not in the Labor Force (neither working nor looking for work).
So to "keep up with the population" we're looking at the Employment to Population ratio...but that's Employment from the Household survey. You have to do some adjustments to get the number of nonfarm payroll jobs (making assumptions about self employed and agriculture etc). A decent tool is at
Jobs Calculator - Federal Reserve Bank of Atlanta
Leaving the labor force can happen in many ways. To leave the population altogether (which of course is a minus to the labor force as well), you can die, go to jail, join the military, get institutionalized, or leave the country. To leave the labor force and stay in the population you can retire or otherwise lose/leave your job and not (yet) start looking for work. Until you start looking, you're not unemployed. If Unemployed, you stop looking for work. Most of that is for personal reasons: school, family obligations, illness/injury etc.
The Labor Force bounces up and down all the time. It reached its peak back in 2000 and is unlikely to get that high again for a while. Keep in mind that many people are loosely attached to the labor force: students/stay home spouses/retirees/Paris Hilton..who don't necessarily need a job but often want one for some extra cash. So these groups dropping out, increased retirees, more people deciding they're better off staying home with the kids, can all drop the labor force (or the participation rate) without meaning anything dire.
It's not a shell game or a lie, it's just that technical definitions and methodology just doesn't, and can't, match casual/everyday language usage. You can say a 19 year old pothead living in his mom's basement is "unemployed" but is that really a good way to measure the job market? (yes, that was an extreme example to prove a point)