Okay, so Homura's basic motivation is that she will do whatever she thinks she needs to in order to protect Madoka. That never changes. Rebellion is basically the story of Homura taking that defining characteristic to its tragic extreme. You can pinpoint the exact instant during her conversation with Madoka in the flower field when her
hamartia shows up - she changes her mind about how she ought to have behaved at the end of the series. She does this because she mistakenly explains what happened to Madoka - and so Madoka replies with an answer to the wrong hypothetical. In truth, Madoka
won't never see anybody again, so the foundation of Homura's belief that Madoka never really wanted her wish is flawed.
Later, she realizes what Kyuubey is doing. Apparently, these guys are threats to Madoka. Now, the immediate danger is over, and if Homura dies then the only link the Incubators have to try and control Madoka is gone.
But what if there's another way? Homura's in mid-witch freakout mode at this point - she's in no position to think rationally, and moreover has very good reasons to expect the Incubators to keep trying. It'd be odd if she
didn't get paranoid that the Incubators might figure something out - and then she finds that she has some vaguely-defined opportunity.
She can apparently interfere with Madoka.
Now, this is where that flawed reasoning from earlier takes over. From Homura's perspective, she's taking on a burden so that Madoka won't have to anymore. She's protecting Madoka from the Incubators by taking enough power to do it. She's willing to sacrifice anything for this, even their continued friendship, because she's convinced that she knows what's best for Madoka. Even better than Madoka, herself, does. Sure, she winds up hating herself for it - but she's willing to offer up even her conscience on the altar of Gotta Save Madoka. Look at the way she describes herself as the Devil, and her Clara dolls' behavior -
Homura is the edgelord here. I mean, I'm not disagreeing with your assessment of Gen's tendency toward Grimdarkness, but there actually did seem to be some merit to this development.
And all of it is rooted in the one immutable aspect of Homura's character that we already had - her need to protect somebody, and refusal to trust anybody else to get it done. There are tons of problems with the movie, but I gotta say, this one wasn't - it's probably the movie's chief virtue.