I read a series of books by Orson Scott Card when I was a kid. What I didn't know was, they were a sci-fi/fantasy retelling of the book of Mormon. I also didn't know all that much about mormonism, I thought it was just another weird american thing like the Amish or something. Some time into the first or second book, the plot began to take weird turns, that really rubbed me in the wrong way, and transformed a relatively progressive society into a stone age thing where the women were suddenly reduced to be breeding machines and the gay character "came to his senses" and got married so he could have kids. This was of course all "the right thing to do" for these characters. I was really baffled at these turns in what had looked like a promising sci-fi/fantasy tale and I wasn't really motivated to read that to the end. That's something that really puts me off about books, when I feel like I'm being lectured about something I totally disagree with.
As for Lovecraft - I read that stuff as a kid and didn't really notice the racist implications until later. The racism isn't the point of these stories and I find it possible to ignore it. The area I grew up in used to be pretty ethnically homogenous and it really blew my mind when I first saw as a kid how in big cities like Berlin, Paris or Brussels some quarters look like they are on another continent. I can on some level understand how someone who never left his hometown and knew all about the world from books and letter correspondence must have felt in New York. It also helps to read a Lovecraft biography, with his life, he could have turned out much weirder. His masssive correspondence and his scientific mind eventually helped him recognize how wrong some of views were towards the end of his life and his latest stories don't actually use any racist stereotypes anymore.