I can think of three major potential reasons that would apply:
1) Experience: They may not be the cleanest, but at least they do their job for the most part, or they know how to go about performing specific areas very well.
2) Connections: They know people who would make it an utter pain to continue on without the bad apple.
3) Hard to Replace: Even if they don't fall into the other two categories, they might not be wanted to get rid of due to the fact that there might not be a person to replace them with. Attitude might be enforced by the general unwillingness to cooperate with the police that many people exhibit, and the general disdain given to them.
Maybe they're just assholes who are bad at their jobs.
There's a reason for everything but it doesn't need to be a good reason. I really doubt that all or even most bad apples are held on because they're a net benefit to the force.
I'm just stating a few that might cover potentially a sizable majority of cases combined. Bullshit is to be expected from a government organization.
Yeah, the problem with guns, if there is one, seems to be in the psychological aspect - encouraging the warrior cop mentality.
EDIT: And, also, I think there are better ways to combat that, although if you want to get into guns specifically I think we need to find a way to disentangle them from the machismo and liberty associations they seem to have. A good start would be repealing the 2nd amendment but ahahahahahahahahaha
FURTHER EDIT: That's not to say I want some kind of gun ban, but that there's a lot of cultural weirdness that comes from privileging weapons that way, and it's that aspect of culture that I think contributes to warrior cops, by contributing to a general "guns = warrior = blood and honor" mindset. I'd have it replaced with something more general, if I had my way, but I never will and it's pretty off topic to get into that.
From what I have seen, there are two views in terms of how a culture views weaponry. The "Sword" view is the one that is more frequently seen in the majority of the world, viewing a weapon as an extension of the person. To develop this mindset, at the founding of a nation, soldiers taken years of training to hone their skills, essentially making use feel a part of the person. America did not have this sort of beginning. It instead has the "Gun" view, where the weapon empowers the person. This mindset was established by the soldiers who founded the nation typically having a few weeks of training before entering combat, or simply entering it with skills they had learned from hunting practices of the time.
In addition, the tactics of the "Sword" mentality lead to a greater focus on the whole of the group and the long-term. Tactics inherit in the "Gun" mentality lead to a greater focus on the self and the short-term. There's problems in both. For Swords wanting to hang back too much, planing too far ahead without looking at what is going on in the present, or becoming entirely focused on one thing, without paying attention to others. Examples of the results of these problems would be both World Wars, focused treaties causing the first, and being too slow causing the second. For Guns, it is going in too fast, planning only for the short term, not considering the long term, and trying to take everything on at once. Examples of consequences would include the Collapse of Rome, and the American Vietnam war. Neither is good, neither is bad, they are just one of the major core bases of cultures.
I had a point in typing this up, but now that I have finished, I cannot for the life of me remember what it was now...