Ten things I've learned about / realized about modding BYORs:
1: People are a lot more willing to accept random bullshit in a BYOR game. Obviously if there's too much, it won't be well received anyways, but by signing up for a BYOR you're signing up to a more flexible game than most. You could even say the random bullshit is part of the appeal of the BYOR genre.
2: Checks and balances and role weights are important. There are no vanilla roles - everyone has their own powers, ideally with a strong degree of uniqueness. But you have to prepare for each single player's 'special thing' by including multiple counterbalances for that 'special thing'.
3: Every player should have something special or unique - or occasionally shared in a manner that makes the sharing feel special. (Like for example a mason group.) The specialness shouldn't be able to dominate the whole game - at least, not by itself - but it should give the player a sense of 'this is awesome' attached.
4: Aside from the checks and balances thing, positive interactions should exist as well, or at least be possible. It's entirely okay if you don't predict everything that might come from an ability, okay for you to laugh along with the players when an infinite action loop accidentally shows up... but that sort of thing being possible also helps makes BYORs feel special. The intersection of the specialness of two players is allowed to be game-breaking if it's unlikely to happen. When or if it does happen, it'll be amusing for everyone.
5: It's perfectly fine to get creative in role name interpretation, especially if it feels like the player is trying to force a specific result (like with a Shakeragian role).
6: Third parties are very disadvantaged unless they either create a team for themselves (cult for example) or attach to an already existing team (mafia-ally for example). To compensate for this, third parties should be given just a bit more power or security.
7: The more work you give to yourself, the more bloated the game gets, the more likely things are to just... go downhill. Be careful with randomness, meta-abilities (such an ability that improves other abilities arbitrarily), day powers (you can mitigate this by having a specific time of day you evaluate day abilities), and so on. If you do include any of these bloating elements, don't use them pointlessly - use them as part of a player's special element, or as 'spice' in moderation.
8: In addition to individual checks and balances, you should also consider whole groups - and sometimes even the composition of those groups. Based on what I read in the aftermath of Mostly Vanilla Mafia 1, a third mafia member instead of a mafia-ally would have been reasonable, for example, and helped stop the immense town-sided swing that occurred in that game.
9: The game should ideally be just as fun for you to run as a mod as the fun the players will have. Your own well-being as the runner of the game matters too.
10: Double or even triple-check your action resolution docs. If you own up to your mistakes you'll probably be forgiven, but better to avoid making them in the first place.
With all this laid out? Now I feel ready to run FBYOR5, once it's my turn.