Things that can slow you down:
-Low agility. In adventure mode, agility may well be the most important quality you can have.
-Carrying too much stuff. Bolts, for example, can be fairly heavy. If you're carrying a lot of items, try dropping some of the heavier ones to see if your speed improves.
-Lying down. If you aren't standing up, your speed is quite a bit lower. Probably not your problem, but it's worth keeping in mind.
-Using a crutch. If you hold a crutch in one hand, you'll use it even if you don't need it, and keep using it even if you put it in a backpack. This slows you down considerably until you build up high crutch user skill. If this happened, you might be able to stop using the crutch by [r]emoving it into your hand, then dropping it.
-Sneaking also slows you down, but you wouldn't be sneaking and attacking at the same time unless you have legendary+ ambusher skill.
-Pain, weariness, nausea, and other impairments will slow you down, but those shouldn't be constant in every battle.
-Firing a bow or crossbow takes a long time, especially at low skill. There's likely to be pages of announcements every time you use a ranged weapon, but this wouldn't influence movement speed.
There may be other factors that could be influencing your speed, but those are the ones that come to mind.
Better speed should help against the goblins. So should Gearskull's combat tips. Having some sort of defensive skill is fairly crucial. If you don't mind breaking realism for an unfair advantage until you get a better feel for how to survive in adventure mode, you can collect multiple shields, then remove them from your backpack into your hands to increase your chance of blocking. Aiming for an enemy's foot or leg can remove their ability to stand, slowing them down and giving you a big advantage.
Right now, there's no way to give your followers supplies without using outside utilities. I'm sure this will be added eventually, but unfortunately, an archer who runs out of ammo is currently fairly useless. I don't know of any way to dismiss them, beyond waiting for them to get killed and then taking their armor.
Were the monsters prisoners when you found them? I think prisoners will join anyone who offers to take them away.