Cities naturally engulf smaller neighboring hamlets. That's "urban sprawl" (or suburban sprawl) in a nutshell. Castles will naturally evade being completely unrecognizably being engulfed, but having a castle wind up completely surrounded by a town isn't that terrible, unless there is a good reason to put, say, a fortress on a border or behind some sort of Great Wall of China-style fortification, although in a case like a border, you just naturally stop the town at the border (unless it merges into a town on the other side of a border, as towns often do along peaceful borders). It's only the monastery that is likely to be abandoned and moved further away as they stop being peaceful places. (Also, things like werecreature dens are something towns would expand into, and those would necessarily be the sort of thing that would require confrontation to clear out.)
I remember having conversations about this years ago when cities were first being built, but generally, cities exist because there is one large primary source of employment, and everyone else is just trying to be physically close to that employer, either because they are laborers at that primary employer or because they are secondary employers that support that primary employer, with teritary employers crowding around those secondary employers.
In the case of a port town at the mouth of a river, for example, the docks where goods are transfered from river barges to oceangoing vessels are a major employer. Secondary employers are those that make ship-related goods, from shipyards to weavers of rope to taverns and boarding houses, and those will then need hemp farms for the rope fibers, woodcutters for the lumber, and food farms for the food. Especially in a world without cars, there's a maximum amount of distance people will realistically want to live from their employers, and a maximum distance they can go to buy for all the commodities they need to live.
Building "up" is cost-inefficient, and in a realistic sense, it's very hard to build more than five stories with medieval technology, and three-story houses are probably all the likely less skilled laborers could build. (There weren't building codes like we know them today in medieval times, and truly skilled craftsmen didn't make peasant apartments.) Hence, it tends to only happen when the people are constrained. Basically, when a city wants to expand out, but either is going past the limits of how far people will walk to their jobs, so they have to stack extra floors to compensate, or else when there is a physical barrier preventing expansion, such as a town being in a narrow valley. (One that comes to mind is
Robin Hood's Bay, which is built between steep slopes so that the town wound up building houses into the alleys between houses, creating a highly effective smuggling route, as there were passages through the insides of houses that allowed goods delivered at the docks to reach the city walls without ever touching the streets.)
And of course, there can be more than one primary employer as a city becomes large enough. New York City became hugely important as the port at the mouth of the Hudson River, but as it became larger, it attracted other industries, like the textiles/clothing industry, which jumped from being a secondary employer to a primary employer as it grew large enough to begin producing excess and exporting its clothing around the world. As more primary employers grow up, it effectively creates multiple city centers that create multiple overlapping cities, where people will face more pressure to move in close to these primary employers, and this tends to create the "metropolis" we think of.