My policy on FBs has usually been to leave them alone if at all possible. Even a pathetically weak FB can destroy a fort, long after the FB is dead and gone. Although sending a kitten after it has it's merits. After the kitty scratches it to death, leave the feline hero in the cavern for a while to see if it gets sick. If so, you can put restrictions on any contaminated areas and wall them off and then things should be safe.
The downside of leaving a FB alone in a cavern is if you ever need to go into the cavern, like maybe to get some cave spider silk to satisfy a mood. I had a very bad experience in my last fort by leaving FBs alone in a cavern. I had designated some exploratory mining and was later surprised to see "Urist McHauler cancels store item in bin: Forgotten Beast". Uh-oh, that's not a good thing to see. I checked, and the mining had found an unexplored bit of the cavern, and Urist McHauler had been picking up a gem that had been dug up right where the tunnel met the cavern. I dispatched my closest squad and they got there while the FB was still in the tunnel. They killed after a fight involving large clouds of deadly dust, and as they were walking away they suddenly started spewing out miasma. I watched as they headed to my hospital, and while they were getting diagnosed, I was another "interrupted by forgotten beast" message. Oh shit, there were 3 of them in that cavern! This one was on my main stairs and left lots of blood and extract spread on stairs and a heavily trafficked hallway before it was killed. Every dog in my fort started having their feet rot off, as well as all the children and many of the adults who had worn out their shoes & socks. It was a classic syndrome mess.
So maybe leaving FBs alone in a cavern isn't such a great idea. But don't just send a few troops/kittens after a FB and expect to just kill it and forget about it. Even a wimpy FB needs careful prep to make sure you can contain any syndrome it my share.