Realisticly I believe rubble was used in medieval walls. I'm not exactly certain why they did that but if I had to give an answer I'd say that it increased the walls flexibility and made it harder for it to break when having boulders thrown at it, kinda like the difference between iron and steel.
I think that sounds correct. Also, the Romans used rubble/gravel in their early from of concrete and in their roads. As a side note, I think the dwarves would certainly be inventive enough to come up with that same Roman-style concrete and would love to see it implemented somehow. (Yes, I do realize that this is the second time I have mentioned the Romans. I think they're cool. Get over it.)
Aother thought just ocurred to me: wouldn't rubble be a nice way to get stone crafts to make more sense? You can carve a statue out of a stone as it is now, but you would require rubble to produce a bunch of little, stone crafts.
Rubble would make ore and coal make more sense too. You can't smelt iron out of a huge block of hematite; you need to break it into smaller pieces first. Similarly, you can't burn a huge block of coal very quickly or make it produce much heat. Rubble could solve both of these small, realism-based problems.