That's fair enough, and I butt heads with fundamentalists too. That's why I tend to snap back when people seem to be firing on religion as a whole, rather than the problematic parts.
Even casual religion still generally professes to have some source of absolute truth in its dogma, though, and to blame fundamentalism is to ignore the dangers that can pose. To use the example most relevant to American politics, each version of the Christian Bible is a flawed book written by flawed men, filled with contradictions, factual impossibilities and moral precepts that were written for contemporary rather than modern audiences. It is certainly possible to believe it is morally instructive without being literally true, and to some extent that's required when doing science given methodological naturalism, but that's
technically heretical, since the book itself says it's to be read as literally true in a way nothing falsifiable can be. That simplicity is as attractive in some circumstances as it is toxic to rational inquiry. Thus all the contortions and outright fabrications apologeticists go through to distort science until it fits with what, under their worldview, has to be true because it says it is, even and especially when it's demonstrably counterfactual.
Religion is not inherently evil, and fundie assholes would still be assholes as atheists. It is, however, inherently incompatible with how we try to minimize wrongness, and that's dangerous in a scientific and democratic context. Focusing on the fundies can mask the more numerous people who cannot or do not want to restrict their faith and consequent assumptions of absolute truth to those things that are already unfalsifiable and manage their consideration of data accordingly.