These are various narrative pieces I've picked up. I'll try to put sources where possible.
Slavery, failure of Reconstruction, Redlining, racist policies in the South (and North) keep black population impoverished and disempowered, and tying masses of the black population to the land (thus missing the industrial boom) well into the 1940s
1. Franklin Delano Roosevelt + Labor leads to New Deal; socialist-organized (although not led) CIO strengthens labor (Kaye, McAlevey), although the unionization effort in the south lagged behind (in part due to the powerful and conservative politicians there, Republican AND Democrat)
2. After WWII, returning GIs would be harder to union-bust (it would look bad), and that meant it might be easier to unionize. As a result, the Taft-Hartley Act gets passed (with Republicans over-riding Truman's veto), which is heavily anti-union is does heavy damage to labor movement (Kaye)
McCarthyism guts the socialists from the labor movement, and as the most effective labor-organizers, this results in the modern beurucratic union, which begins the slow decline of labor, weakened institutional power, and heralds the "nationalist" turn (McAlevey)
Large wave of black (40% of Southern-born blacks) moved to the north in the 40s and 50s, as well intra-south rural-to-urban migration (and due to Jim Crow disenfrachisement, had little buying power for housing, had relatively little education, etc) - segregated nature of work also meant that migrating blacks would have to compete with already-present black population; Baby boomers led to a wave of young males, and more competition in the labor market (source)
Masked by gross overall prosperity, joblessness in inner/central cities grew as industry moving away (in part to escape union strongholds, and to take advantage of new infrastructure) from inner cities to suburbs and Sunbelt, and later on automation and foreign competition; already-poorer blacks were less able to leave (poor, low-credit, zoning, lack of public transport) than white counterparts (who themselves were often driven by racist aversion to blacks), and incentivized by federal investment in highway and home construction (which naturally invest away from cities, and into the suburbs) (source)
The post-FDR liberals (ie Kennedy, Johnson) unable to (but both clearly wanting to help, accurately identifed the root causes of the problem, and putting a lot of resources to the problems - just not enough) properly develop federal programs to help inner cities (with dual problems of much investment instead being allocated to suburbs, and the tax-base that funded city social programs having fled) (my expansion: the inability of inner cities to fully integrate in the working black population was in part due to the undermining of labor unions, starting in 1947, thus their work on the ground killed, as well as the political pressure they applied to FDR being much weaker on Kennedy and Johnson) (source)
As a result of the failure to deal with rising social problems, the resulting (symptom) rise of crime lead to local and state governments needing to use the cheapest option to appease the voting base - punitive law enforcement, and the beginning of the carceral state we know today; the prosperity outside of inner cities stigmatized those that were suffering, and the strained social control institutions reduced cost of crime by reducing risk of getting caught; All of this lead to a death spiral, collapsing communities, and the rise of turning to illicit means of income generation (source)
In reaction to this rising crime, defying the structural liberal criticisms (which are more expensive than punitive measures), Conservative law and order talking points (which exploited racist tropes) resonated; this same 'get tough' politics rose in popularity in black communities as well (source)
'The point is that waging an all-out war on the root causes of crime is equivalent to the task of building a large, redistributive welfare state that takes from the rich to give to the poor.' ... 'Thus, liberals did not fail to imagine what ought to have been done. Nor did they fail to attempt to do what ought to have been done. So why, exactly, did they fail? At root the issue is not one of attitudes or motivation, but capacity. The ultimate causes of liberal failure lie outside the state, in the incapacity of the American poor to compel redistribution from the rich.' (due to weakened labor power I'd say, but also the authors point out the costs of the Vietnam War) (source)
Racism was still a nucleus in this, but it's fundamental genesis wasn't in the 60s, it was during the Great Migration, when the North failed to integrate the working black population (source)(my expansion: again, in large part due to the early death of socialist-organized labor unions) (my expansion: rooted in policies such as red-lining)
'In short, we are arguing that American exceptionalism in violence and punishment is a symptom of America’s exceptional history. America has so many prisoners because its development path yielded some unique social problems, while its political economy prohibits redistribution from rich to poor on the European model. In a sentence, the story of American mass incarceration is the story of the underdevelopment of American social democracy.'; 'The failure at the federal level thus matters not because the federal government was the proximate agent of mass incarceration. It was not: neither under Johnson nor under later administrations. Rather, it matters because the persistent failure of the federal government to attack the root causes of crime left the task of managing the rise in crime to state and local governments. In this climate of high anxiety about crime, state and local legislators, mayors, city officials, prosecutors, and sheriffs made careers out of responding to a panicking public.'; 'One of the reasons for this is simply institutional. In the division of labor that characterizes American federalism, police, prisons, and the courts are mostly the responsibility of the states and municipalities, while most of the major social programs in American history have been invented and funded at the federal level. When local and state officials were bombarded by panicking electorates, it is no surprise that it was mainly to these tools that they would turn.'; 'However, this is not the whole story. After all, some states and municipalities do attempt to craft their own social policies. They can raise taxes and spend in redistributive ways. Thus, another answer is that they were subject to the same constraint that bound the federal government: the absence of a constituency that could force the rich to give to the poor.; (source)
Question: There is truth in Alexander's account of the carceral state though. The Drug War has lead to a large chunk (around a quarter) of the carceral population. What about Nixon's southern turn? Is racism deployed in more of a political manner, which enables impunity at the local level?
In the 70s, a corporate propaganda blitz (ie Koch brothers, Coors) further damaged the remaining liberal and labor position - following an insanely bad Carter administration, Reagan took office, and carried out an enormous effort to weaken labor, enact neoliberalism (ie bipartisan privatization with a parallel 'culture war'), amongst many other small tweaks to strengthen the position of Republicans to this day, and cemented the 'neoliberal' ideology of today. (Andersen)