It's possible. I tend towards a similar belief system, although
I can't escape the distinguishing factor that we're the original
programmers and any AI we develop, no matter _how long_ we give
it in time, it will still suffer certain inescapable limitations
implanted by us that it will be entirely blind to. yes; humans
will always be blind to certain things. We can create extensions
of ourselves that work within what we are cognitively capable
of, but by that same token, anything we design will also be
restricted to anything we are cognitively capable of. In short,
I do not believe it is possible for humans to create something
that can evolve beyond the limitations of thinking of anything
that humanity isn't ALSO potentially capable of eventually
thinking of. In short: here is a set of all possible human
thought/emotional configurations: within that set of human
thought/emotional configutions are found the programming for any
AI ever programmed by any human being ever to exist. Those AIs
will be programmed from within the set of limitations provided
by their makers. I believe they will be forever tethered to
staying within the boundaries of our imagination and no further.
In short, if it's not possible for any human anywhere in the
history of humanity to EVER consider something, it will ALSO be
impossible for anything we create to exceed those possibilities,
for while an AI can exceed even the current generation or 100 of
humans, I don't believe it can exceed all of humanity's
imagination for all time. Of course, this is hardly much of a
limitatoin; because we haven't thought every thought that could
possibly be thunk yet, and to have a machine that can do some of
the human style thinking FOR us would certainly make our work
easier
