I don't like how they removed useful language features like operating overloading simply because they had the potential to be abused. It's programming, everything has the potential to be abused. Don't handicap the programmer as a result of this.
I don't think the standard libraries are well-designed or well-implemented on a fundamental level (favouring inheritance in situations where composition makes much more sense, for example). Swing is also a painful messes.
I don't like it's pseudo-namespace import system and think it mainly an annoyance.
I don't like how implementation details defined several syntax features (Current terrible ways around the lack of generics use type erasure? Well clearly the features of generics must be limited as a result of this -_- Yes that was sarcasm).
I don't like how they've refused to evolve the language meaningfully or add any new and major additions syntax in a long time. Recent announcements with Java 8 have started to move towards some evolution, but it's very little and very late.
I don't think it's a good language to learn how to program with, and consider it a little better than Visual Basic in terms of teaching some bad habits. Because Java has a lot of little things I haven't even listed that add up to make it a terrible language in my eyes.
I do think C# got right a lot of what Java has gotten wrong. Hence my personal recommendation of the former.
C# has disposables for when RAII is useful for clean code.
C# has delegates and lamdas, because newing up an anonymous class to define how a simple addition or comparison occurs in a flexible library is madness.
C# has optional overflow detection for when overflow really can matter.
C# has unsigned integral types, because those are also useful.
C# supports the floating point IEEE 754 standard, because, you know, standards matter.
C# has a decent generic implementation.
I have complaints with C# (as I do with every language), but not as many as I do of Java.
Now, unsigned integrals and "closures" are going into Java 8 but like I said, it's very little and very late.
Still better than PHP though. If you want a good write-up on PHP, I recommend this article: PHP - A Fractal of Bad Design
http://me.veekun.com/blog/2012/04/09/php-a-fractal-of-bad-design/