IronTomato, if this is as bad as you make it out to be, there's absolutely no way you can get your brother to stop pestering you. Pestering you is his favourite thing, and he won't stop doing that if he can do anything about it. If you beat him up, he'll make sure your parents will discourage you from attempting to lay hands on him again. If you ignore him, he'll find ways to force you to stop ignoring him, for example by wrecking your stuff.
A thing you can try to do about this: How about, instead of actually being annoyed, you just act annoyed. Of course it won't be easy to stop being actually annoyed, but you can do it, just keep in mind that all his tricks are getting old. Don't stop acting annoyed, otherwise he'll use new tricks. Do that until you're immune to his pesterings. Now it's your turn to do the ol' psychological manipulation, and that is limited only by your creativity. How about you occasionally just "miss" the part where he annoyed you? Not too often, mind you. On the other hand, you could get really annoyed about things he actually didn't even intend to be annoying. You could also act badly on purpose, being all hammy and stuff. Or how about you annoy him back? Persistence is the key there.
There's a requirement for that though, you actually have to understand him. He can read you like an open book, that's why he's so effective at annoying you. You need to start reading him too. Before you say that he's impossible to understand, being mentally unstable doesn't entail being unpredictable. If his actions aren't what you'd expect from a sane person, then you need to stop trying to interpret them as if he were one.
Finally, if that doesn't work, you can still just endure him until you can move out. Then you can live in peace, make some nerd friends, and it will be so much more satisfying to watch him crash and burn.
<standard internet disclaimer here>