Thinking more about stone and mining, I came up with the following thoughts.
1) Ore. Having novice miners destroy ore makes no sense. Gold, for example, is so valuable that people pan for gold DUST in rivers. I don't care how many chunks the novice is putting it into, you'd be gathering it all up. It gets smelted later, so who cares if it's in a large chunk?
2) Stone on the other hand, need to be in nice large pieces for most uses. Crafts can probably get away with being made from rubble, but things like Statues, Doors, etc all need to come from nice big blocks of stone.
3) Gems require careful digging to avoid breaking them too much. Although even a small ruby is worth a bit.
So, I had an idea of how to do this.
Miners still put things into large or small chunks, based on skill. Small chunks, though, aren't lost. They are simply reduced to Rubble. Rubble can't be picked up and hauled the way large chunks are. You have to it into a bag, bucket, or cart (when those are added in). This makes it so that there is still an efficiency advantage for having your Legendary Miner doing the digging, but you don't lose ore when your novice decides to go gold mining. It just means you need a way to transport it back before you can use it.
For gems, I think you'd have a much smaller chance to get a large gem out of Rubble then out of large chunks. And perhaps some chance that you'd get nothing usable out of it at all, but even here I don't think you should get NOTHING out of the gem cluster. You can be pretty sloppy and still get something, although it might be worth less. Perhaps if gemstones start getting an inherent quality, we could have it that gems gotten from rubble have a lower average quality. I've always thought that gems SHOULD be a raw material with quality, since that's how it works in real life. After all, a flawless gem is worth way more then a flawed one.
This system works well regardless of whether we're using the partial amounts I proposed earlier, or simply as a change to the existing system.