Yeah, by the end of the war, the Empire was literally down to less than half of what it was - only Cyrodil, High Rock, Orsinium, and Skyrim were left, and I'm not 100% sure about Orsinium. Summerset Island was the first place to break away, being where the Aldmeri Dominion started, and they invaded Valenwood and Elsweyr(and are busy going Auschwitz on both). The Empire had to release Hammerfell, Black Marsh, and Morrowind. Hammerfell actually pushed the Dominion back, much to the embarrassment to the Empire, while Black Marsh rolled into Morrowind and was doing to the Dunmer what the Thalmor were doing to the Bosmer... and that was before Red Mountain exploded.
As far as I know, the Aldmeri Dominion probably does have an army comparable to the Empire and Skyrim, and part of the Thalmor's plan is to draw the civil war out long enough to exhaust both sides so they can roll over them both. Also remember that Thalmor soldiers tend to have decent-quality Elven armor(which is a bit better than either Imperial or Stormcloak armor if I recall) and often toss magic around pretty hard, whereas the Imperials and Stormcloaks lack the infrastructure(and racial proclivity) to train an effective force of battlemages due to the former's collapse of the Mages' Guild and the latter's general distrust of magic.