I've never gotten too wrapped up in physical attraction myself. I can recognize individuals who "look good" or have aspects of them that are beautiful, but it isn't something based on commonly-held ideals, and only sometimes applies to people generally thought of as "beautiful". It's also not something that generally attracts me to people... I generally have to know something about who they are as a person before I find myself attracted to them.
For example: I found Viggo Mortensen (the actor who played Aragorn in the Lord of the Rings trilogy) to be a sexy man. What I mean by this is that I have a great deal of respect for how he did all his own stunts as Aragorn, and even managed to parry what could have been a lethal knife-throw during a mishap while filming. He also has a well-shaped jaw, a nice voice, and a good attitude about what he does with his wealth. I don't find myself particularly attracted to Viggo, but I do very much like him as a person.
Another example: I had a creative writing course with a girl who I had something of a crush on. She was tall, round-faced, dark-skinned, and wore a lot of aesthetically beautiful patterned clothes. What attracted me to her most were not these things (they were just other aspects of who she was), but instead were her kindness, her creativity, her soft-spoken tenacity, her openheartedness, and her ability to write poetry and prose. She had a great attitude about life and lived it beautifully, and that is what attracted me to her.
I'd not worry about not getting instantly enamored of people, at all. It just means you've got a better idea of what you're looking for, or at least a more cerebral one. Just be who you are, and don't worry about it. Take the road less-traveled, for it can make all the difference.
P.S. Instead of being "Partly Asexual", you also may just have a low libido. :3