"Programming" doesn't really apply here, as such, when you consider evolutionary circuit design and stuff. God, I need to dig up those old bookmarks but I think that drive is dead...
Guy took a very fancy reprogrammable circuit board and used some very elaborate training regimens to make it output a pulse at a certain frequency--or something like that. Started out fully randomized, but you provide it with various parts that it can utilize if the evolutionary algorithm stumbles across a use for them. Eventually it produces the desired effects, but the circuit just looks like nonsense, wired up in weird ways with spurious parts--and if you try to tweak any of them, it could break unexpectedly. So he goes to his PhD mentor and asks him for help interpreting the circuit that this algorithm created, and his mentor just says "You're wasting my time, there is no interpretation. It found an analog solution in a big wide solution space and it's not in a region that we should bother trying to comprehend, because we wouldn't succeed." The circuit board didn't do anything that humans couldn't design faster and better--but it found its own way, using its own parts, and given enough time you could tackle some very weird or tricky problems with that evolutionary approach.
And that's how a singularity-level AI is going to operate. It wouldn't be "programmed" as we know it. It would grow in this weird organic fashion, where we couldn't remotely hope to make sense of it...because there is no sense, there is only function, only operation with no guarantee of rhyme or reason. Surely as the AI becomes more advanced, it will come up with actual designs to improve itself, but those actual designs could be so mind-bendingly outside of our comprehension that again, why bother?