I read the manga before the anime release and it was 10 times more stupid. That two-tier system literally lasted for like 3 chapters before it all went to bust and forgotten.
I got the feeling that the author started with the two-tier system but decided to drop the concept pretty quickly. And it's very jarring because in the first part, Tatsuya is talking from the oppressed side, but when the writer retconned out the prejudice, Tatsuya is suddenly lecturing other low-borns about not trying to rise above their station, because winners and losers is the natural order of life. And the siser extols at length how the test is wrong because it didn't recognize onii-sama's greatness, rather than saying it's wrong to divide the school into winners and losers, and give all the resources to the winners. Basically the author is outright defending elitist discrimination, as long we don't make a mistake as to who are the deserving superior people vs the undeserving inferior people.
It's ultra-jarring because the author laid on the prejudice really thickly at the start yet by episode 4, the hero is claiming that the oppression is all imaginary, and is because of ingrained loserism by the losers. But most of them are first year students at an elite academy, which is stated to only take the top 1% of national scorers. It doesn't make sense for any of them to have such ingrained loserism, since they did not have a history of us/them division before entering the school.