You know the majority of shows labeled "harem" don't actually involve constant efforts of multiple girls to get into the heros pants? A few do, but most do not. The common feature of all harem shows is "attention", not sex. And all the girls in Dungeon want his attention, so it's a regular harem. Also, all harem shows have stronger first seasons because those contain the "meeting the girls" plot arcs, and thus are more interesting than the part that comes later, when it becomes more about slice-of-life and day to day interactions. You might be mistaking these "introduction" plot arcs for the "relationship building", I think.
The city isn't really that interesting either. Everything revolves around the same basic thing about gods/goddesses who have entourages of adventurers who level up in a single D&D dungeon. It's very weak as a setting. Because the gods can be "anything", the plot is at the same time "nothing". There's no substance to it and more gods and familias are just added in ad-hoc as it goes.
Things just tick along in the city, there's no dynamic aspect to the storyline whatsoever.
And his "quest for self-determination". Seriously? All his life goals are is to level up as an adventurer. Sure he has some backstory about family members killed off, but that's the most generic possible backstory and motivation of all time.
Don't pretend there's anything more than 1-dimensional in this series.