Uh...well this seems somewhat rhetorical, but when your main aim is feeding your family or even more extreme, finding a place to sleep at night, its much more difficult to focus on education, and furthering oneself. Its just not a priority for you. Not to imply that you can't advance without a higher education, but its much less likely than it used to be. If I may reference Maslow's pyramid of needs
Higher income people have more of their lower needs taken care of, and thus can devote mental energy to things like doing well at work or school, or distinguishing themselves in some other way. Poverty does not allow one to do this. Like all things, it is not a guarantee of failure just like wealth is not a guarantee of success, but it is a defining factor.