"I don't know how to solve that."
I was thinking of doing a suggestion for a Sanity system (another one) that involved all sort of natural and supernatural alterations of your personality giving you Psychosis, Neurosis, and minor forms of Sociopathy until it goes into full blown insanity that makes your character near unplayable until they recover.
It even was supposed to divide the kind of stress between: Reality Weakening, Psychological need, and Overloads. (For example magic forcing your Jollyness above homeostasis would cause the mind to create mental barriers in the form of psychosis and thus you could suffer blackouts)
It could work and the way I was designing it, it was supposed to be possible even in Adventure mode as the character's self-perception (which I can't explain without spoilers) is supposed to give indicators of sevear mental stress.
So combined with a magic system your character could, through magic stress, gain an OCD causing him to scratch a lot causing his hands to constantly be "Bruised" with light bleeding.
Wow I just realised that one be one sure way to get rid of Casual magic... Sure you could survive but your slowly driving the magician insane. You need breaks