Those are not problems with the design, those are decisions made about what they're trying to measure, which is the cross-section of three factors: submission to authority, aggression in the name of authority, and wanting to make others conform too. Obviously, hardly anyone would answer "yes" to a "are you a nazi" test, but plently of people score highly on the RWA test. So it's good at detecting nazi-follower like mentalities in general population. Hardly anyone is a literal nazi, but plenty of people seem to be potential nazis if you go by the test results. It's worth researching for that reason.
Yes, those traits are probably quite prevalent in any human population, and yes, they probably do correlate with having Nazi-sympathies, as any 13-year-old could tell you. But the crucial question is, does this test measure those traits or something else altogether? It's supposed to be a "personality test" disguised as an "attitude survey," but as the questions are loaded with clusters of dogwhistles and fighting words, you have to admit that it's so artlessly and ham-fistedly "disguised" that any distinction between psychology and politics becomes moot (and let's ignore the fact that it's bloody difficult to differentiate personalities from attitudes in any case.) The personality traits have been already grouped into two opposing camps because the test in fact takes the conventional left-right dichotomy as its unstated premise and starting point – instead of explaining the left-right split in psychological terms, it simply presupposes the status quo and runs with it for Great Science. Even the author himself seems to acknowledge the horribleness of his methodology, but in the end he rather hilariously brushes it off and comes out on top (in his own mind):
Third, you knew what the items were trying to measure, didn’t you, you rascal!
The RWA scale is a personality test disguised as an attitude survey, but I’ll bet you
saw right through it.8 In fact, you could probably take each statement apart and see
how I was trying to slyly tap the various components of the RWA personality trait.
Take that first-scored item, No. 3: “Our country desperately needs a mighty leader
(authoritarian submission) who will do what has to be done to destroy (authoritarian
aggression) the radical new ways and sinfulness that are ruining us”
(conventionalism). Well if you’re smart enough to do that, you’re smart enough to
realize how easily you might have slanted your answers to look good.9
8 The RWA scale is well-disguised. Personality tests are usually phrased in the first
person (e.g., “I have strange thoughts while in the bathtub”) whereas attitude surveys41
typically are not (e.g., “Bath tubs should keep to ‘their place’ in a house”). So it is
easy to pass off the RWA scale, a personality test, as yet another opinion survey. Most
respondents think that it seeks “opinions about society” or has “something to do with
morals.” Back to chapter
9 For the same good reasons, it’s out of bounds to give the RWA scale to your loved
ones, and unloved ones, to show them how “authoritarian they are.”
He's completely ignoring the fact that pretty much everyone taking the test realizes what they are being tested for, and that such traits as "
submission to authority, aggression in the name of authority, wanting to make others conform, etc." are generally considered
bad in our society, and most people are not willing to admit having them even to themselves. In other words, his data does not reflect "what people are like," but merely how they want to see themselves and how they want to be seen by other members of their social group.
Case in point: I got a "0% RWA" result, and as far as I know anything about myself, that does not sound very convincing. I mean, not even one percent? Oh come on, even Gandhi was at least 20% RWA, and Mother Theresa was probably somewhere above 95%. And I strongly suspect that if everyone in my department took this test, we'd see a statistically improbable row of zeroes accompanied by a beaming row of self-congratulatory faces. On the other hand, if you made members of a modern right-wing populist party take the test, we'd undoubtedly see a very narrow distribution of scores focused smack-dab in the middle of the continuum – the funny thing about contemporary European wingnuts is that very, very few of them self-identify as wingnuts, even in private.
Regarding the "predictive power" of this test in the mock UN scenario, we should ask: "What kind of people will publicly admit being Hitler-freaks?" The answer: Complete Morons. It's no wonder, then, that the people who got ~100% scores from this test fared badly in an intricate diplomatic game – they were the only ones too thick to figure out what all that dog-whistling was about.
And there's definite evidence that this is the case when you look at the large number of behaviours which are correlated with the test scores. For example high RWA scorers are also much quicker to ramp up the voltage in Milligram-style experiments, so RWA-style personality tests can give a new dimension of analysis into well established existing experiments. A test that tells you before-hand who's most likely to electrocute other people in the Milligram experiment has obvious value as a research tool, rather than just saying some people electrocute others more or less but we're not going to try and find out why.
This is pretty interesting because the Milgram experiment is another poignant example of a tendentious study with horrible methodology and unstated premises. Wikipedia lists James Waller's four objections to Milgram's study, and they are quite fatal to its central conclusions :
1. The subjects of Milgram experiments, wrote James Waller (Becoming Evil), were assured in advance that no permanent physical damage would result from their actions. However, the Holocaust perpetrators were fully aware of their hands-on killing and maiming of the victims.
2. The laboratory subjects themselves did not know their victims and were not motivated by racism. On the other hand, the Holocaust perpetrators displayed an intense devaluation of the victims through a lifetime of personal development.
3. Those serving punishment at the lab were not sadists, nor hate-mongers, and often exhibited great anguish and conflict in the experiment, unlike the designers and executioners of the Final Solution (see Holocaust trials), who had a clear "goal" on their hands, set beforehand.
4. The experiment lasted for an hour, with no time for the subjects to contemplate the implications of their behavior. Meanwhile, the Holocaust lasted for years with ample time for a moral assessment of all individuals and organizations involved.[18]
tl;dr: If your study tries to figure out why ordinary people like WWII-era Germans commit atrocities, you are probably doing real science. If your study takes "Nazis are Evil" as the premise and presents "Nazis are Evil" as the conclusion, you are not actually saying anything.
REALEDIT: Continued where I left off.
EDITEDIT: Much edit, very writing.