Usually stuff like this boils down to communication and compromise.
Do you feel one way or another? Talk about it. Let her know where you're at. Don't let things fester. (I'm talking about both girls, here.)
For the friend, tell her you have a loose rule about dating friends and that you hadn't thought of her that way, so it may take some time for you to adjust. If you think she's relationship material let her know that you're open to the idea, but you want to it happen without forcing it too much. If you're exclusive with the other girl, say that you consider your integrity to be paramount but wouldn't cross the idea off at some future date should things not work out.
As for things not working out with the other girl (And this applies to all relationships), if life has taught me anything it's that certain things don't matter enough to destroy a relationship. If there's one thing that just bugs you a little bit, like she likes Taco Bell more than Taco Time, it's not the end of the world. Naturally this applies to more than just food choices, but you need to sit down and ask yourself if a relationship is worth ending over something that you could easily discuss and compromise over. It's a two way street so you both have to be willing to give a little to try to make the other person happy. It's a huge warning sign if she's completely unwilling to compromise over something little. On the flip side don't be the punk that expects everyone else to conform to your opinions. You'll never find someone who jives perfectly with every aspect of your being and it's healthy to have someone you trust challenge you from time to time.
Now, if it's something major, like she murders and drinks the blood of newborn lambs every second Thursday of the month and you cannot abide, then it's time to discuss that with her and say that it's kind of a huge deal to you that someone you date doesn't slaughter baby animals. Be open, honest, and willing to let go if she finds her beliefs as important as you find them repulsive - But for the love of God don't use that word. You want to be honest, not insulting.
Things get murkier when we get to personality quirks and physical characteristics, but the same principles apply. Just be absolutely 100% sure that it's a deal breaker before telling her that you can't stand that she nags you about small things or complain that eating a bowl of anchovies and hot sauce for breakfast every day makes her breath stink real bad and she has terrible taste in hot sauce brands. Just keep an eye out for crazy and don't go near it. If you're getting a creepy vibe that she'll stalk you for ten years before burning your house because you won't get back with her, then it's best to get out ASAP before she starts to see you as The One.
Just remember: Communication and compromise, except in case crazy. In case of crazy, just run.