That's why the math is very rarely set up to be "we'd like to save you, but we're over budget as is" and tends to be more about probabilities of dying, shared by all (or some). No one is marked out to be screwed by not investing in some super expensive program to prevent terrorism against post offices (unless you're omniscient), it's just that people at the post office will have a slightly higher chance to die then they would have if you had. Statistically, eventually, with enough time and people someone will die who wouldn't have if you paid for the anti-post-office-terrorism program; it's just that that person could have been anyone. So the question becomes how much were you willing to pay for how long to save a life instead of, as you said, investing in housing, or research, or subsidies for the poor, or simply not spending it at all.
So money spent for life/QoL/likelihood-of-dying is actually a pretty good measure of government accountability (for lack of a better word). If we can save more lives with the same amount of money by investing in infrastructure or something, well, we should! Granted that won't work with everything, but even though its pretty gross (and gives rise to that sort of mindset that worries about "Death Panels"), it's really useful.