In fact, that argument can be stated a lot more simply.
Take any programmable AI. Show it its own source code. Instruct it to run the code with all the inputs it has received so far, and then return something different.
Hence, a strong AI cannot have source code that it is capable of running.
Huh. I'd always heard that a true mind couldn't comprehend itself. I guess now I know why.
This argument also proves that people can't really think, because I can set you the following challenge: "Work out what number you'd name in response to this question, and then tell me the number one bigger than that."
This doesn't prove that a human "can't think". It proves that a human can't provide a logically consistent answer to a question that doesn't have a logically consistent answer.
This isn't quite analogous to the situation in the parent. There exists a logically consistent answer to the question "tell me something other than what [algorithm] would yield from [inputs]" -- it just can't be produced by that algorithm.
It's exactly analogous. There is a logically consistent answer to the question "tell me something other than what Joe McBlow would do in situation X"; it's just that Joe McBlow can't give it to you.
Take any programmable AI. Show it its own source code. Instruct it to run the code with all the inputs it has received so far, and then return something different.
Hence, a strong AI cannot have source code that it is capable of running.
Huh. I'd always heard that a true mind couldn't comprehend itself. I guess now I know why.