In many cases, they don't think they can succeed. They may have encountered failure, and just given up. Holding them back just reinforces the belief that they can't do it, so they just quit trying.
One problem is starting formal instruction before the child is ready. You try to learn to read, and don't succeed, so you decide you can't. Same with math. Someone tries to teach you fractions, or algebra, and you don't get it, and decide that you just don't get math. So, what do we do to reform the situation? Why, we start teaching academics at an earlier and earlier age, of course. It's a great solution politically, but a disaster to the learning process.
If walking had to be formally taught, we'd have a generation of crawlers. Kids learn to walk when they're ready. For some, that means nine months or so, for others perhaps 15. Do the earlier walkers turn out to be better at it? Not necessarily. For reading, it's the same thing. A kid who learns at 5 is not necessarily better at it than one who learns at 7, or 9. One of the best teachers of remedial reading I met told me he learned to read himself at the age of 15.