Depends.
I build fairly big above ground forts, which are really more like towns than motte-and-bailey castles (which wouldn't have a market, e.g. depot, at all), so there is enough room for everything. Late game houses are at least two stories tall, so there's plenty of space for stockpiling on the ground level.
There is a "market", where the depot is packed tightly together with various workshops and the houses where the craftsmen live.
Medic has their own house, where the hospital is located, which can be expanded if the need for hospital beds grows. The castle's barracks can be repurposed for a hospital temporarily, if the things get really bad.
I use individual dinning rooms (for commoners it just overlaps with their bedroom), so taverns are more like little gathering places where people meet (undistracted by furniture etc., as suggested in
this thread), scattered around the town. The ones outside the castle are open for all, while the ones inside are citizen-only.
The library is merged with the main temple, which is a dedicated building.
I don't do open quarries for fps reasons. The mines (together with the smelters and such) are usually outside the walls, but that is never a big problem because they are only used occassionally, and thus, don't require constant defense. There is a fence and a door, and I can move a bunch of guards nearby when the bars/blocks get hauled out, but that's about it. I keep the metal bars in various stockpiles within the town/castle, somewhere near the forges where they get processed as needed (fairly quick, as armor and weapons in my mod are expensive to make). Motivates me to be more cautious with resources during the sieges, which would hardly be the case with a nigh-infinite supply of metal within the walls.