The black death didn't discriminate and wiped out people of all social statuses equally. It upset power balances rapidly.
Barring absolute chaos, people can generally plan for years and decades from now. Declining population is something we all know is coming. Those with wealth will, for the most part, hold onto it. Those born into wealth will, for the most part, hold onto it. Japan is always mentioned as the prime example of declining population in these threads, and while the population is declining and demand for labor is increasing, wages are not rising but poverty is. Maybe it'll change 10 or 20 years from now, but the trend is downwards for the average person.
That's true but I think which bit of the population remains matters.
The demographic impact of a plague is pretty much the exact opposite of economic flight. In a plague the fittest and healthiest survive and are unencumbered by the elderly and unfit, but in a demographic flight the fittest and healthiest leave. Similarly in a birth rate collapse the demographics shift towards the elderly and away from the young and fit.