The "trans panic defense" isn't a formal affirmative defense in and of itself. Rather, it is a excuse to invoke a more conventional affirmative defense, usually insanity. Like most affirmative defenses, it is rarely used (remember - using an affirmative defense is inherently a confession of guilt ("I did it, but I don't think I should be punished normally for it because..."), and if the judge/jury doesn't buy it you're fucked), and it is even more rarely successful. Meanwhile, the majority of hate crimes are intimidation or assault - deliberate murder is a small portion of hate crimes, and hate-murders are a small portion of murders. Meanwhile, the biggest targets of hate crime in the US remain women and black people - others do get targeted, but they're not the ones with the biggest target on their backs other than as part of one of those two groups.
Making this more complicated is that rabid transphobia is something of an astroturfed bigotry, deliberately manufactured by the right because the older forms of bigotry they relied on are waning. The more "wild" variety is rooted in mockery and ignorance - still extremely cruel and difficult to deal with, but it isn't the "I sure would like to hunt your kind for sport" type of thing you see at some of the soapboxes.