Semi Molten Rock (commonly called SMR) is found at the bottom of the magma sea (including the parts where there's rock on top of it rather than magma), so you're digging in rock surrounded by the sea. The "hot stone" warning means there's magma adjacent to the hot tile (besides or below: I don't think there are any cases where above occurs naturally, although you can create it by moving magma).
Digging out rock where the magma is below only is safe, while breaching the sea to the side will have consequences...
You CAN get below the magma sea, but that's associated with some serious challenges and is best avoided until you think you're ready for it. It's possible to get through the SMR using some exploity techniques, but you won't be able to do it by accident.
My advice would be to breach the magma sea from above (so you have hot stone only below, not to the side) to check out the layout of the sea and then plug the breach to block critters living there from getting to you. Once that's been done I'd dig out the rock formations that are revealed, obviously leaving a shell intact.
Once you have seen where the magma see is, you can safely dig out the level above it, but have to deal with constant cancellations. When I do that (e.g. to build magma powered workshops powered by magma in a channel siphoned off from the sea and then blocked by magma safe drawbridges, to secure it from critters), I first dig a tunnel surrounding a rectangular area and once dug, designate one level or rock on the inside and one of the outside of the tunnel, because digging hot rock does not result in cancellation if the designation is on revealed rock (you're supposed to know that the rock is hot, so it ought to be an informed decision).