Doomstacks are excessive but it's more realistic than one unit per tile. That's why I'd rather see some kind of combination between unit per tile limits and the old doomstacks - like actual army units which are comprised of collections of individual units that fight together in the same combat. You can only have one active army per tile (so any given tile can only attack or defend once per turn), and if the defending army on the tile dies the rest take some damage, maybe scatter as they retreat (the idea being that the front line was overrun and all the un-deployed units at the back get caught unprepared). The armies can be deployed outside of your territory, but might take a couple turns to get organized, with some units or unit types being quicker at that than others, maybe also affected by tech (e.g. improvements in war tactics and coordination). They'd be limited by either the number available in-combat tiles or supply or coordination (could go either way, at tank column tank takes up more space and has more complex supply requirements than swordsman infantry, but by then you'd also have radio and combustion engines so they're easier to supply), with Great General units either improving how many combat tiles you have available to field, or providing additional supply. Lone individual units would always be for combat, but will attempt to retreat if attacked by an army (and would get slaughtered if they tried to attack an organized army).
I'd also probably get rid of "ranged" attacks by "ranged" units (except at the late game with air units and cruise missiles and maybe artillery that can fire over dozens of miles - not catapults/trebuchets), but instead they're done by cavalry units in sorties (cities maintain their ranged attacks under the same logic - defenders make a dash out to harass the enemy then return). Any army with cavalry can make that attack, the strength depending on just how many cavalry units are in that army, so you can still penalize an opponent as he tries to move units individually without forming armies beforehand.
I don't know many 4X games that have used this concept. Call to Power 2 had the army grid (which is my chief inspiration for this), while most space 4X have fleet mechanics (GalCiv, Endless Space, etc) but there's rarely further tactics involved beyond throwing as much dakka/armor as you can manage into the fleet's supply cap.