There's no doubt that work on science is accelerating, though. And honestly there's a lot of improvement to be had just in the way we use computing power, never mind making things smaller and more powerful. The way programmers are nowadays, they just throw around pointers and integers and floating points where bitfields will do, rabble rabble, kids etc.
I'd get into the singularity and crap, but that'd be derailing the thread.
Just because we haven't seen time travellers doesn't mean time travel is impossible, nor does it mean the world will be destroyed. There are a couple possibilities that preclude either, such as time travel accessing/creating parallel dimensions, time travel destroying sections of time ahead of the destination entirely (which would mean time travel would only exist from the perspective of the time traveller and whoever he convinces he's from the future, so unless it's already been invented or the traveller is in the loony bin/government custody, there are no time travellers), time travel only being possible with a receiving device, and time travel just never being bothered with to our era, which although seems unlikely, we have no way of predicting what our motives will be in the future.
Although I do think there's a good possibility that we will destroy ourselves with a biological or nanomolecular weapon in the nearish future, considering how commonplace the tools to create both are becoming.