The issue is that people are talking past each other by not trying understand what the other person is saying when they say "harem". In anime, a harem is generally defined as any situation where there is one main male character, and almost all his interactions are with women. The characters do not need to be romantically connected. Any other main male characters will have "issues" which prevent them getting female attention, usually fat/ugly/bald/old/perverted/dickwads or similar.
This is such a common set-up for shows that it doesn't make sense to only limit the concept to the minority of shows where there are literally half a dozen girls chasing after a guy. Even most outright harem shows don't have that. See Love Hina for example. That's about the most classic harem you can get, and there is one designated love interest for the entire series.
Think about Steins;Gate, Clannad, Haruhi Suzimiya, Bakemonogatari. All of these revolve around one guy and a ton of girls who's onscreen lives revolve around him, and each series has a single secondary male character who none of the girls pay much attention to. Sure, you could not watch any of those shows because I told you there's a "harem", but you might be missing out on a show you'd like a lot.
btw, someone on myanimelist took offense at me labeling No Game No Life a harem, because NGNL doesn't feature a romance-related plot. Well, it has every harem element except for the romance, and I pointed out that there were no other notable male characters except the hero. They rebutted "but there was that one girls grandfather". Ok, so there's another male character, and it's a 70-year-old dude with a long beard. Seriously, I rest my case.