Chainmail bikinis on the girls whilst all the guys are in sensible armor is one of the giveaways. e.g. it's in how a character is used rather than them just being "attractive".
Whilst it's almost impossible to define a "line", there are clear examples where everyone can agree that it's about teh sex.
What I do is simply apply the "service" point of Fanservice. Something in addition to what is there beyond the norm.
Fanservice near the invisable line becomes failed fanservice by its very nature (afterall a scene meant to be sex you up where people cannot tell it is sexing you up is not doing its job).
Where fanservice however becomes huge negatives are when the Fanservice is at the expense of the character or the scene. Also when the desire to make a character "hot" puts them through a cookie cutter that severely limits the design choices.
If you see a gratuitous shirtless scene, then it's fanservice for the straight women/gay men/bi people
This isn't nessisarily true but it is a very common mistake.
You see Straight Men want to see sexy male heros because even straight people are "attracted" to the same sex. Now this may not be a sexual attractiveness but it could be because of projection or ordinary magnitism.
To put it another way, to quote Dave Chippelle "Even Women like Women with large breasts" (Not vibratum).
Oddly enough I've seen a tv show (By Disney no less) actually break this down quite maturely.
Mind you I guess it depends on how we are defining Fanservice.