Liquids don't fall through floors right?
Is the damp/hot stone underneath a pool/reservoir/magma sea actually safe to dig?
Correct, digging underneath a pool/reservoir/magma sea is perfectly safe as long as you don't breech anything sideways.
Edit: Also, how lethal are infections from stagnant water? That is, how long does the dwarf usually live? I currently just use a stagnant water well, because I don't care for pumps and other waterworks.
Infection isn't necessarily caused by stagnant water, it's just not as good as normal water.
Infection chance (greatest on left):
no cleaning>cleaning with bloody water>cleaning with stagnant water>cleaning with normal water
Soap will also greatly reduce the chance of infection regardless of the type of water it is used with.
Once a dwarf actually has an "Infected" wound they are most likely doomed as infections only heal very rarely. However if the infection is a small enough one and the dwarf is tough enough dwarves have been known to survive for decades before finally giving into the infection and dying. Basically think of an infection as a constantly bleeding wound. The bigger the wound and the less tough a dwarf is the faster they will give in, but a very tough dwarf with a small wound can last extremely long periods of time before giving in, well long enough for them to die of some more fast and violent method.