A very interesting new book by Nick Bostrom, Superintelligence, addresses the questions that will be raised by the appearance of Artificial Intelligence (AI) Systems that are vastly smarter than humans.
So far, researchers have concentrated their efforts on the development of artificial intelligence systems that are as intelligent as humans, the so called strong AI. This is a tall order and it may yet take many years until we reach that point.
However, as Nick Bostrom points out in this book, there is no reason to believe that, once developed, strong AI systems would remain approximately as intelligent as humans. Once an AI system with human-level intelligence comes into existence, it will certainly be able to improve itself rapidly past that level.
As Irving John Good, a statistician that worked with Alan Turing at Bletchley Park, pointed out, a smarter than human machine is probably the last invention man will ever need to create. After that, such a machine can invent all sorts of new technologies, including the ones related with AI.
Nick Bostrom describes in a very clear and convincing way how a superintelligence may develop out of research in AI and neurosciences, using a number of different paths that may include whole brain emulation, strong artificial intelligence or highly connected communities of human brains.