Dark Eldar, due to the Eye growing, feel Slaanesh clawing at their soul with increased ferocity, so they're forced to increase their raids to keep up. This creates instability in Commorragh, because there is just not enough souls to keep protecting all dark eldar, creating strife within.
At the same time, they recognise they need to help defend against orks/chaos and Nids/necron. Because orks, necron, nids and chaos are horrible subjects for soul torture, they really need humans, tau and Eldar to survive as a species.
For the Imperium... you know how now it's always 'they're at the brink of Armageddon'? Well, I say go for it. Let them take a major blow. Due to increased power of all the xeno and chaos forces, they just cannot hold everything anymore. And so, they're forced to make a terrible decision: they abandon a lot of their worlds so they can afford to concentrate on their core worlds (forge worlds, agriworlds and such). A lot of them look towards Tau or Eldar for protection, some form small independent empires. But a whole lot more just burn, falling to the maws of ravenous aliens or insane heretical cults. This rakes a deep scar in the psyche of humanity, feeling like they betrayed their own, feeling like they utterly failed their brothers and sisters, and realize just how far they've fallen from the Emperors ideal.
However, as they keep losing ground and are faced with total defeat, this forces them to finally evolve as a society, getting over a lot of their xeno- and technofobia, out of sheer necessity. Adapt or perish, y'know. And then something happens that gives them new hope, like a primarch returning. But they can't just reconquer the universe because of three reasons:
-Nids/necron war. Shit is really going down there, and so they're forced to 'help' the necron to try and contain the Great Devourer. But the necron themselves don't ally with them because they just see imperium/tau/eldar as potential biomass for the nids.
-in the Eye, things are getting worse every day. Just wave after wave of increasingly powerful orks into the grinder, with Chaos getting a big foothold in realspace (maybe by destroying the Cadia pillar things).
-a lot of former imperium worlds that survived accuse them of having abandoned them in their time of need. What will the new imperium do? Fall to their old ways and conquer them, or leave them be? This could be a very interesting avenue to explore in stories (stories like 'the last church' prove you can do this in the 40K universe), seeing how they deal with this as a culture, how they deal with the still fresh scars.
Tau keep advancing in tech, start getting a real empire going, and absorb a big deal of the worlds and people that the Imperium was forced to abandon. However, they also start getting problems (which prevents them from conquering everything):
-imperial overstretch, and protecting/keeping order in such a big empire. Lack of warp travel really plays into this.
-Increased Chaos influence leading to bigger cult activity. They'll be forced to either loose worlds to daemons everywhere, or start adapting some methods of the old imperium. Oh, and Tau realize they have no soul, have identity crisis
-Maybe tied into previous, they have an internal schism. Not all out war, but cracks start showing in the 'perfect' tau society. Maybe a bunch of Ethereals are having a dispute or something. Either way, this leads to a sort of 'shadow war' within the tau, giving room for lots of character development (what will the regular tau do if different ethereals start saying they need to do different things 'for the greater good'?
Basically, they'll be forced to 'grow up' as a species, and realize governing an enormous empire can't be done on nice words or ideology, but that you need real, sometimes awful policy for that.
Eldar lose a few maiden worlds/craftworlds to the nid/necron and ork/chaos wars, and are also moving closer to extinction. They basically struggle to hang onto their culture. But then, they find something in the Webway that could save them. For example, a way to close the Eye. Or they are forced to increase ties with 'lesser' races like tau and human, again also to preserve their species, shattering their image of superiority.