I tend to do manual installs and keep a note of which loose files belong to which mod. Then again I tend not to go for the massive overhaul type mods or those with large numbers of loose files so it's not too hard to sort out a good load order.
I used to do that as well, but it started driving me insane. I kept it up for a year until I got fed up with random bugs seeping into my saved games of unexplained origin, despite following best practices and checking for conflicts myself.
I found that I could never "remember" what loose files belonged to which mod, so it forced me to keep a folder with every archive I'd installed: I'd then use intuition to guess what archive to search in for a particular file. It got unwieldy quickly, however, and it wasn't uncommon to end up with partial installs - remnants of texture packs and mods I'd installed a long time ago but lost track of - and end up with scripts of unknown origins conflicting with each other and doing weird things. This was especially noticeable when upgrading mods: who has time to manually compare the contents of each archive to see what has been deleted? I'd upgrade a mod, forgetting to check manually whether it had deleted any files, only to have my saved games go pear shaped a few hours later because the mod author had rearranged (or deleted) scripts, and the old ones were still there (happily conflicting with the new ones). If I was lucky I'd kept a copy of the old archive (or could re-download it), and I'd remember to check. If I was unlucky I'd just have to either ascertain the origins of all of my scripts from scratch, delete the scripts folder and reinstall everything (hoping that none of the "texture only" mods I'd skipped over to save time didn't have any scripts I was missing), or go through all of my upgrades sequentially to try to identify where something went wrong. Not an easy task if you've updated 10 or more mods at once.
Hopefully I'd remembered to delete the entire contents of a mod I'd installed "just to test it out", and hopefully I was able to recover the scripts and textures it had overwritten. If it overwrote something that I couldn't find in my archives I'd be left wondering where the file came from and how I could get it back.
None of the above addresses the "texture leftovers" I had, where the author decided to remove some textures from their mod because they were of poor quality or for some other reason, but I'd failed to notice during an upgrade, deleted the original archive, and now had textures installed with no way of knowing where they came from or whether they were necessary.
Realistically, there's just too many things that can go wrong. Somewhat ironically, I found I always lost track of what scripts and textures had been installed -- precisely the kind of control I assumed installing manually would help with. I finally gave up on the nightmare and installed Mod Organizer. I never looked back.
Nowadays I keep Skyrim uninstalled, though. My free time thanks me.