Yea In order for fortress mode to work with a custom creature you need to have a full entry. Many of the things in the civ entries directly control fortress mode behavior and not having them would make fortress mode unplayable at best.
Its also a bit tricky to make sure the species to work as well, they have to be intelligent, have to be able to open doors, have a grasping part like a hand, can't fly. Many things that limit your options without knowing how to custom make a creature just for the mode.
Know that any other entities that are playable in fortress mode won't be able to trade with them, as civs with that setting rely on special case code to allow trading. The special case code turns off when it detects that yer not the same race, which prevents trade with them.
You'll also have to select your civ manually when choosing an embark site. As all fortress mode civs show up on that list with no obvious distinction. The best method if they both make detailed caves is to select a civ, then tab over to the predicted trade list. The species of the civ you currently have selected will be on the top of the trading list.
So I guess what I'm getting at is that using a non standard race in fortress mode is possible, but is one of the more difficult things to mod. You really should learn to mod though, while hard at first, once you start to figure the system out the rest just falls into place. And it's far easier than most of those other games out there.