The copyright industry would never accept that. Where’s the money for them?

Ahh. Paul McCartney. Looks like Lemmy has finally found a billionaire it likes.

I’m sure it is The Beatles’ activism for social change that won people over. Who could forget their great protest song “The Taxman”, bravely taking a stand against the 95% tax rate. Truly, the 60ies were a time of liberation.

Thank you for the long reply. I took some time to digest it. I believe I know what you mean.

I can also say that the consciousness resides in a form of virtual reality in the brain, allowing us to manipulate reality in our minds to predict outcomes of our actions.

We imagine what happens. Physicists use their imagination to understand physical systems. Einstein was famous for his thought experiments, such as imagining riding on a beam of light.

We also use our physical intuition for unrelated things. In math or engineering, everything is a point in some space; a data point. An RGB color is a point in 3D color space. An image can be a single point in some high dimensional space.

All our ancestor’s back to the beginning of life had to navigate an environment. Much of the evolution of our nervous system was occupied with navigating spaces and predicting physics. (This is why I believe language to be much easier than self-driving cars. See Moravec’s paradox.)

One problem is, when I think abstract thoughts and concentrate, I tend to be much less aware of myself. I can’t spare the “CPU cycles”, so to say. I don’t think self-awareness is a necessary component of this “virtual environment”.

There are people who are bad at visualizing; a condition known as aphantasia. There must be, at least, quite some diversity in the nature of this virtual environment.

Some ideas about brain architecture seem to be implied. It should be possible to test some of these ideas by reference to neurological experiments or case studies, such as the work on split-brain patients. Perhaps the phenomenon of blindsight is directly relevant.

I am reminded of the concept of latent representations in AI. Lately, as reasoning models have become the rage, there are attempts to let the reasoning happen in latent space.

You do it wrong, you provided the “answer” to the logic proposition, and got a parroted the proof for it.

Well, that’s the same situation I was in and just what I did. For that matter, Peano was also in that situation.

This is fixed now, and had to do with tokenizing info incorrectly.

Not quite. It’s a fundamental part of tokenization. The LLM does not “see” the individual letters. By, for example, adding spaces between the letters one could force a different tokenization and a correct count (I tried back then). It’s interesting that the LLM counted 2 "r"s, as that is phonetically correct. One wonders how it picks up on these things. It’s not really clear why it should be able to count at all.

It’s possible to make an LLM work on individual letters, but that is computationally inefficient. A few months ago, researchers at Meta proposed a possible solution called the Byte Latent Transformer (BLT). We’ll see if anything comes of it.

In any case, I do not see the relation to consciousness. Certainly there are enough people who are not able to spell or count and one would not say that they lack consciousness, I assume.

Yes, but if you instruct a parrot or LLM to say yes when asked if it is separate from it’s surroundings, it doesn’t mean it is just because it says so.

That’s true. We need to observe the LLM in its natural habit. What an LLM typically does, is continue a text. (It could also be used to work backwards or fill in the middle, but never mind.) A base model is no good as a chatbot. It has to be instruct-tuned. In operation, the tuned model is given a chat log containing a system prompt, text from the user, and text that it has previously generated. It will then add a reply and terminate the output. This text, the chat log, could be said to be the sum of its “sensory perceptions” as well as its “short-term memory”. Within this, it is able to distinguish its own replies, that of the user, and possibly other texts.

My example shows this level of understanding clearly isn’t there.

Can you lay out what abilities are connected to consciousness? What tasks are diagnostic of consciousness? Could we use an IQ test and diagnose people as having or not consciousness?

I was a bit confused by that question, because consciousness is not a construct, the brain is, of which consciousness is an emerging property.

The brain is a physical object. Consciousness is both an emergent property and a construct; like, say, temperature or IQ.

You are saying that there are different levels of consciousness. So, it must be something that is measurable and quantifiable. I assume a consciousness test would be similar to IQ test in that it would contain selected “puzzles”.

We have to figure out how consciousness is different from IQ. What puzzles are diagnostic of consciousness and not of academic ability?

Because I don’t think we have a sure methodology.

I don’t think there’s an agreed definition.

Strong AI or AGI, or whatever you will, is usually talked about in terms of intellectual ability. It’s not quite clear why this would require consciousness. Some tasks are aided by or maybe even necessitate self-awareness; for example, chatbots. But it seems to me that you could leave out such tasks and still have something quite impressive.

Then, of course, there is no agreed definition of consciousness. Many will argue that the self-awareness of chatbots is not consciousness.

I would say most people take strong AI and similar to mean an artificial person, for which they take consciousness as a necessary ingredient. Of course, it is impossible to engineer an artificial person. It is like creating a technology to turn a peasant into a king. It is a category error. A less kind take could be that stochastic parrots string words together based on superficial patterns without any understanding.

But we may be able to prove that it is NOT conscious, which I think is clearly the case with current level AI. Although you don’t accept the example I provided, I believe it is clear evidence of lack of a consciousness behind the high level of intelligence it clearly has.

Indeed, I do not see the relation between consciousness and reasoning in this example.

Self-awareness means the ability to distinguish self from other, which implies computing from sensory data what is oneself and what is not. That could be said to be a form of reasoning. But I do not see such a relation for the example.

By that standard, are all humans conscious?

FWIW, I asked GPT-4o mini via DDG.

I don’t know if that means it understands. It’s how I would have done it (yesterday, after looking up Peano Axioms in Wikipedia), and I don’t know if I understand it.

Just because you can’t make a mathematical proof doesn’t mean you don’t understand the very simple truth of the statement.

If I can’t prove it, I don’t know how I can claim to understand it.

It’s axiomatic that equality is symmetric. It’s also axiomatic that 1+1=2. There is not a whole lot to understand. I have memorized that. Actually, having now thought about this for a bit, I think I can prove it.

What makes the difference between a human learning these things and an AI being trained for them?

I think if I could describe that, I might actually have solved the problem of strong AI.

Then how will you know the difference between strong AI and not-strong AI?

I don’t see why the example requiring training for humans to understand is unfortunate.

Humans aren’t innately good at math. I wouldn’t have been able to prove the statement without looking things up. I certainly would not be able to come up with the Peano Axioms, or anything comparable, on my own. Most people, even educated people, probably wouldn’t understand what there is to prove. Actually, I’m not sure if I do.

It’s not clear why such deficiencies among humans do not argue against human consciousness.

A leading AI has way more training than would ever be possible for any human, still they don’t grasp basic concepts, while their knowledge is way bigger than for any human.

That’s dubious. LLMs are trained on more text than a human ever sees, but humans are trained on data from several senses. I guess it’s not entirely clear how much data that is, but it’s a lot and very high quality. Humans are trained on that sense data and not on text. Humans read text and may learn from it.

Being conscious is not just to know what the words mean, but to understand what they mean.

What might an operational definition look like?

Obviously the Turing test doesn’t cut it, which I suspected already back then.

The Turing test is misunderstood a lot. Here’s Wikipedia on the Turing test:

[Turing] opens with the words: “I propose to consider the question, ‘Can machines think?’” Because “thinking” is difficult to define, Turing chooses to “replace the question by another, which is closely related to it and is expressed in relatively unambiguous words”. Turing describes the new form of the problem in terms of a three-person party game called the “imitation game”, in which an interrogator asks questions of a man and a woman in another room in order to determine the correct sex of the two players. Turing’s new question is: "Are there imaginable digital computers which would do well in the imitation game?

One should bear in mind that scientific methodology was not very formalized at the time. Today, it is self-evident to any educated person that the “judges” would have to be blinded, which is the whole point of the text chat setup.

What has been called “Turing test” over the years is simultaneously easier and harder. Easier, because these tests usually involved only a chat without any predetermined task that requires thinking. It was possible to pass without having to think. But also harder, because thinking alone is not sufficient. One has to convince an interviewer that one is part of the in-group. It is the ultimate social game; indeed, often a party game (haha, I made a pun). Turing himself, of course, eventually lost such a game.

All I can say is that with the level of intelligence current leading AI have, they make silly mistakes that seems obvious if it was really conscious.

For instance as strong as they seem analyzing logic problems, they fail to realize that 1+1=2 <=> 2=1+1.

This connects consciousness to reasoning ability in some unclear way. The example seems unfortunate, since humans need training to understand it. Most people in developed countries would agree that the equivalence is formally correct, but very few would be able to prove it. Most wouldn’t even know how to spell Peano Axiom; nor would they even try (Oh, luckier bridge and rail!)

Self aware consciousness on a human level.

How do you operationally define consciousness?

I find it funny that in the year 2000 while attending philosophy at University of Copenhagen I predicted strong AI around 2035.

That seems to be aging well. But what is the definition of “strong AI”?