I think you're overlooking the effect of colonialism and, possibly, the enclosure of land (or other forms of private property) in creating conditions where children or adults might starve.
Laboring is better than starving, sure. However, why would people starve. If you look at primitive societies, people didn't starve for lack of a job. They lacked wage labor, or even slave labor, entirely.