You mentioned manufacturing, among other things, as being more productive. In an age where improvements in technology is replacing the need for manufacturing jobs, we can't really expect manufacturing to be a huge job creator. Even if our manufacturing sector started growing like crazy, job creation in manufacturing is most likely going to lag behind other sectors which are more dependent upon low cost human labor.
The same with finance. Since there is virtually no need for shipping or material handling in finance, and since it is largely a matter of just shuffling numbers, many of it's tasks are very likely to being replaced by automation and computerization.
Any particular sector can grow in size and production, without having huge gains in jobs.