Competence bonuses stack as far as I know, so he shouldn't get untyped, especially if you say 'untyped'. It should be competence, which is what most bonuses from that sort of thing are.
Don't put 'at least 5 levels'; they never do that, and there's no point. Just put 'the ability to cast(or you can say prepare) 3rd level arcane spells'. That's even a bit much to me, only really need to know 2nd level, just add something else so you still for sure have to be level 5, or even 7. It seems like a monk/wizard PrC after all.
Possibly have a specific spell needed to be able to cast/know. A lesser version of Tenser's Transformation or the Pathfinder equivaleny, perhaps.
Additionally, you could consider having a minimum amount of unarmed or natural weapon damage(which must be able to be lethal). Needing Improved Grapple is kinda dumb. A BAB minimum, for example, could work. Needing to have flurry of blows could also work for a prereq; just sayin'.
The proficiency thing is dumb. They really shouldn't gain any proficiencies. Everyone is considered proficient with unarmed strike. However. I would suggest adding some skills; Athletics, Jump, maybe a Knowledge or two. If you're able to have any Perform skill as a class skill, you should have all of them as class skills, just silly otherwise.
Instead of Muscled Fist, have it so any Cestus, Punching Dagger, Spiked Knuckle, Gauntlet, etc. that the Muscle Wizard uses(there needs to be a better name, even Arcane Wrestler could do in a pinch, we just need to think of it) can use either the base weapons damage, critical rate, alignment/magic/material for overcoming damage reduction, etc., or the muscle wizard's unarmed version's of those, whichever is better.
Rather than flurry of blows stacking, I would have the unarmed damage stack with monk levels. Of course, I don't know much about monks in pathfinder, so maybe I'm speaking nonsense.
Muscled Maneuver should a. Be competence, not untyped(you don't type untyped!), b. be greater than that but more specific., and c. Have a better name. For example, maybe an every other level progression. Or the current progression, but twice that for grapple.
Muscle wizard oath is silly. Just do it like the monk does; loses all muscle wizard abilities if wearing armor, using a shield, or carrying a medium or heavy load. Spell failure is already a thing. Instead of the ridiculous 'YOU MAY NEVER USE WEAPONS AGAIN AHAHAHAH' just specify for the offensive abilities that they only work when unarmed or using a muscle wizard weapon. Some defensive abilities, too, perhaps.
A good ability might be one that makes it so all of the monk abilities that rely on Wisdom now rely on whatever the main spellcasting ability for the class you used to qualify is. If you had multiple classes that would qualify like that, choose one.
Rather than Empower (Ability) I would just give him a general ability around fourth or fifth level that lets him cast a certain number of spells per day spontaneously empowered(if he prepares spells, the spells themselves must still be prepared ahead of time). Possibly similar abilities relating to Extend, Maximize, and even Quicken. However, they must be arcane spells, and if they are direct damage dealing spells, must require melee touch attacks.
Should get a bonus at some time or another to concentration checks whilst grappling or the like. Probably fairly soon in the leveling, and a pretty decent bonus. Might even get to take 10 eventually.
Master of Muscle is OP, just being honest. There's whole prestige classes dedicated to doing that sort of thing alone.
I CAST FIST could use improvement for sure. It's just kinda silly(and over-powered) right now.
Muscle Wizard Abilities and DCs and such should either be based on the muscle wizard's Constitution, not Strength, or their primary spellcasting ability.
Spellcasting progression is a bit fast. Every other level would probably be best, without a level the first level of the class(so it would might go 0/1/0/1 etc.).
More abilities relating to using spell slots to do cool things with your fists, more abilities relating to using your fists to do cool things with spells, and more abilities relating to crushing your enemies, seeing them driven before you, and hearing the lamentation of their women(no not really, just more abilities relating to using spells whilst choking the life out of them).
This kind of prestige class isn't really the kind to have a capstone ability, I think. Most prestige classes with capstone abilities aren't typically combo-class PrCs. Some are, but they typically aren't incredible capstones, or anything, just fairly nice things to have.
Sidenote; Grappling rules in Pathfinder are, frankly, dumb. They are one of the few things I don't think improved from 3.5(didn't get worse, though...). You can take the most decadent knight in the world, clad in adamantine masterwork(or magical) full plate, with a matching adamantine tower shield, wielding a masterwork shortsword, and muscle wizard here can punch him to death, because he effectively attacks(and, to boot, takes up their actions/disarms/trips them all at once) as a touch attack. Now, if he was actually using a touch attack like shocking grasp, that'd make sense. But he's not. He's punching them. Where? I don't even know, because it's full plate. I guess he's wrenching their arm violently enough to rip it off. Sure, at high levels, he'll be less effective because people's BAB will be higher. At high levels, though, he is a wizard, and it doesn't matter. Right now, he's more dangerous than DreXav, at least against a single target. DreXav can only power attack you and drop you a good 20 hit points. Muscle Wizard here can take you down for 6 hit points, and your next turn is taken up by trying to escape, you don't have an action to attack back, and then it repeats. His only weakness is if there are large numbers of enemies, so while he's grappling they can try to stab him in the face(that's true for most of us, actually, except Daethwin, whose only weakness is fighting in general).