How feasible is it to just not wear armor and use your shield and [weapon of choice] instead?
I'd say in light of what TheBlueSteel stated that it's a bit more then that - it would matter more based on what your up against and if you power level Armor User skill.
Simply put, Armor reduces your characters speed if you don't power level it up to Grand Master or Legendary. Sometimes your speed is more important a defense then the Armor you might be wearing. Example - you get into melee with a Bronze Colossus. It don't matter if your wearing armor or not, once that Bronze Colossus gets a hit or two in, your character is done. The above example as well shouldn't be seen as an isolated case either. Against Archers you'll be better off with a higher speed then better Armor since as of now the Bolts and Arrows pretty well bypass Armor entirely most of the time; against enemies that use Lashes or Scourges you'll be better off with higher speed for pretty much the same reason.
Then to turn what I said above around, perhaps your in a situation where your armor is of great importance. If you enter a crypt or a dungeon for example, sometimes just entering a room with a lack of armor can be a death sentence. Even if you could reduce the chance of instantly being waylaid entering a room (like by not entering first) you could find yourself getting surrounding on different sides. Your chance of dodging/blocking the attack from the side or behind is usually quite reduced. Of course being in the outside world you could also end up in this situation when you get ambushed.