Yes, PE should be compulsory for all ages. I've had very good and very bad experiences with PE teachers over the years, and the fact is that they're no different from other teachers when it comes to quality variation. I took PE in Elementary School because it was compulsory and loved it most of the time, it wasn't offered at all by my Middle School, and I've taken it every year in High School so far, even though only two of those were compulsory. I'm not taking it for my Senior year because I ran out of PE classes that sounded worthwhile that I haven't already taken.
Yes, being able to perform well in PE is very subjective, and as such grading should be based upon participation, like in my school. There was a "thug life"-type guy in my PE class last year. He was probably the most fit out of all of us, but he almost certainly got the lowest grade because of how little he participated. I was probably in the lowest bracket of fitness in the class, but I showed up every day, dressed out, and participated, so I ended up with over a 95% average. That's a good system in my mind.
Also, they need to make you learn to swim in elementary PE if you don't have a disability preventing it. Seriously, the number of people who can't swim in my High School class is pathetic, pathetic, pathetic. At least 60% of them would drown almost instantly in seven or eight feet of water, and that's just sad. It's like they want to die via drowning.