I think design philosophies change a bit between setup types. Open, Closed, and Semi-closed all have different things that are important. Open (and even a bit of Semi-closed) setups require you to have ways around massclaiming or breaking the game, for example. Closed setups, on the other hand, require much more in the way of balance, since unexpected Abilities and win conditions can cause players to have a very bad time.
For this reason, I'll focus on a more generalized approach that will apply more to Closed and Semi-closed setups, and probably even more to BYORs, since I'm the most familiar with those.
Step 1. Who are the bad guys?
I think this is a really easy step to overthink in the game, but thinking about what alignments constitute your game is important. My philosophy is that I like to think of a couple of different "archetypes" and then choose one of them randomly. My archetypes are things like "Single scumteam", "multiple scumteams", "5 SKs", "Cult" (lol, jk, I would never run a Cult game... unless...?). How the setup is implemented in the number of players is probably the part I think the most about, but typically, you're going to aim for a 3:1 Town to scum ratio.
Step 2. Design space is infinite and scary
So, once you've got the easy part out of the way, it's time to start designing roles. My first runthrough is to gather a few ideas about what a Role would entail. Investigative, protective, manipulative, supportive, etc. How these can manifest changes, too. Maybe the Role has a power limitation, like ammo or evolution. Maybe it has a way to interact with other Roles. If I know who is using the Role, that also affects what I add to it. For instance, if I were going to design a Role for myself, I would focus my Role around the concept of "borrowed power." For FallacyofUrist, I would make a Role that has game-breaking capabilities. Everyone has strengths and weaknesses that changes how a Role is used, so considering that can be a good idea.
Step 3. A Role in a vacuum
With all these ideas, you can start making the Role and its Abilities. You can create all manner of Abilities from these ideas. This is probably the place where you want to try EVERYTHING! Want a Cop that can Inspect as many Players as they want but the Inspect is generalized? Do it. Want an Unkillable Vigilante? Do it. Make exactly what you want, and don't worry about if it balances or works later. You're bound to mess up a lot here, but that's what it's there for.
This is glossed over, though. So, let's go deeper. There are three distinct things you should worry about: What a Role does, what a Role wants to do, and what a Role should feel like.
If you're making an Investigative Role that doesn't investigate, you failed. This one isn't very common, but can come around when you're trying to design multiple distinct Roles. I sometimes give multiple Roles the same Ability, and that feels bad to me on a macro-level, so I try to remove these similarities, which cause these Roles to feel bad on an individual level. Sometimes, you might end up with two Roles that are similar (or the exact same, just to really screw with people), and that's okay! Make two Cops!
If you make an Investigative Role that isn't incentivized to investigate, you failed. This usually comes about when trying to balance out the Role. In an effort to keep the power level down, you might add in a power system, like ammo. But, if the cost to use the Ability is too high, no one will use it! If you make an Ability that has no reason to be used over your other choices, then you either made a bad Ability, or you made the other Abilities too good. Luckily, this is fixable, either by giving incentives to diversify (Say, you gain additional benefits for using an Ability on a specific Night) or by giving punishments for stagnating (Power limitations for using an Ability over and over, or X-Shot Abilities). Always think about a Roleblocker that blocks one Player the entire game: That's not okay!
If you make an Investigative Role that doesn't feel like it's investigating, you failed. This one is the least likely to happen, but it's one to think about. Like, let's say you made a Cop that Roleblocks whoever they investigate. Well, technically, this is an Investigative Role! But, you're going to find yourself using it for the Roleblocking before long, and this makes it feel like it's not really an Investigative Role, not least because it can be punishing to Town, but also because Roleblocks are simply more powerful. It's not a bad Role, to be sure, but it doesn't exactly feel like an Investigative Role. To me, this is really more about Player expectations, but I think it's important to consider.
Step 4. Interactions
After you have all the Roles, you can start seeing how they interact. Your Cop that Investigates as many Players as they want? What happens if they target the entire scumteam? Oops, that's a game over immediately! What stops this game over? You can add false positives and false negatives to cover for this, but there's also a meta angle to this, where you could tell the scumteam that there are no Millers (and possibly even given them a Miller ability!)
What about an Unkillable Vigilante? Well, if they're truly Unkillable, you can have the Mafia have a 1-Shot that allows them to bypass Unkillable. You can also have the Mafia have their own Unkillable Ability, or, even worse, a way to redirect the Kill, should the Vigilante claim themselves.
This is a hard step, because it relies on knowing how your Role is supposed to interact in the game, but also how the Player will pilot it. If you don't want to think too hard about the Player themselves, take a Role, compare it to the other Roles, and see if it breaks the game. You could possibly do this for three or four Roles, but I would really only test more than that for teams. How the Mafia interacts with itself and other Roles is super important. You never want a busted Mafia, nor do you want a Mafia that relies on one specific Player not dying.
Step 5 (but really Step 0). The vote is sacred
This is the last and final rule, but ultimately the most important: The Town has one true method to deal with their problems, and that is the vote. If you mess with the vote, you mess with the Town's ability to win. This can be things like stopping the execution, skipping the Day, execution avoidance, revives on execute, votestealing, vote manipulation, etc. I try to avoid these, especially for scumteams, because they can create incentives for abuse, as well as get to points in the game where the scum haven't attained equality, but are unable to be beaten even with perfect play. I think the Mafia can always win without a Mafiakill, but the Town can (usually) never win without voting. Therefore, if you mess with the vote, it better be for a good reason and it better be extremely limited. In general, it is always safer to leave it alone.
TL;DR: Make what you want, get creative, then think about if creativity is bad and you somehow broke the game, and DON'T WRECK THE VOTE THAT'S THE MAIN MECHANIC IN THE GAME!