Three things.
There's a difference between "Digging through Stone" and "Taking down Walls". Right now, you can't dig through walls, only stone. You have to take down a wall. Now presumably if you bashed down a wall, it would be a heck of a lot faster.. assuming you had the raw power to do it. It would be harder to start digging through a smooth wall because there isn't a good place to start taking out chunks of rock. But once started, it would prove faster to plow through it than it would be to take each block down. Therefore, it should be harder to dig through a smooth wall than a rough-hewn wall, yet easier to dig through a smooth/constructed wall than to manually take apart said wall.
Tolkien's Goblins were actually intelligent creatures. If you read enough into early Middle-Earth lore (The Silmarillion, for example) you will understand why I say this. They were just lazy, overworked, and oppressed. They were timid creatures in the presence of power. A power which was always overshadowing them at any particular moment, and never allowed to grow on their own. On their own, Tolkien goblins would have probably prospered.. but they were always pushed back down and used as tools. Also, it was Saruman who helped developed other advanced things (gunpowder), not the goblins themselves. Technically, Goblins and Orcs may be considered the same thing (though this is sometimes of debate.. perhaps just different subraces of the same race). Do note, Saruman's big old army of Uruk-Hai are not the same thing.
Digging through soil, I suppose, is something I implied. It would be easier to dig through soil. Pathfinding the "shortest/easiest presumed tunnel-to-be-made to get inside a fort" shouldn't take into account stone layers unless absolutely necessary. Siege Craft should be considered easier than digging through stone, imo. My counter-digger defense idea of Moats "as deep as the soil goes" was intended to make digging in much harder (and dangerous) for your enemies. Soil should always be first option, and stone should be a more "desperate measure to get inside [unless we have engineers or dwarves digging for us]."