From Peak Horse to Peak Human: How AI Could Replace Us (with Calum Chace)

What we're going to have to do is find a way to transfer some of the wealth being generated by the machines to everybody. And boy, this is a hard problem. One is this little word basic. You know, we have to do much much better than give everybody a basic income in a future in which huge wealth has been created by machines and 99% of the population is just scraping by. That's an appalling world and we shouldn't do that. universal generous income. And that's that's where we need to get to. The likelihood of being able to to forever control entities which are much smarter than us is is pretty slim. Our best chance may lie in making sure that they're conscious because if they're conscious, they will have empathy. Having conscious beings appreciating the beauty of the universe is possibly the most important thing the universe has. Welcome to the Future of Life Institute podcast. My name is Gus Docker and I'm here with Callum Chase. Callum, welcome to the podcast. Hi Gus, it's a privilege to be with you. Fantastic. Could you give a quick intro to what you've been working on? I have been thinking about the future of AI for a very long time. I wrote the first article I wrote about it was in 1980 and I was wrong about pretty much everything. I thought that work was going to be automated within the next few years. I was obviously completely wrong. I had not yet then heard of Amara and Amara's law which is the observation that we overestimate the impact of of AI and any other technology in the short term and underestimate it in the long term. I've seen that played out many times since. Then in 1999 I read Ray Kurtz's book Age of Spiritual Machines. I think a lot of people got interested in the future of AI then. And so since then I've been very interested in in where we're going. I retired in 2011. thought it would be a good idea for Ridley Scott to make a movie based on something like Curtis Fal's book. So I thought, well, I'll write a novel which he can turn into a movie. And bizarrely, Ridley Scott never got hold of my novel. I don't understand that. Somebody pointed out that the novel was actually a non-fiction book, dressed up as a novel, so I should redo it as a non-fiction book, which I did. And all in all, I've written five books on AI since then. And then people started asking me to give talks. And originally, I thought talks would be really good marketing for the books. And I discovered it's the other way round. Books, good marketing for talks. And so I've given about 200 odd talks in 20 countries. That's what I do. But more recently, a friend of mine started a company called Consium. Uh I'm sure we'll get into that later. And I'm one of the co-founders of Consium, and I'm my my title is I'm chief marketing officer of that. So I kind of see myself now as somebody who's retired with two full-time jobs. That's perfect. All right. One of the topics you write about quite extensively is whether we're going to see technological employment. And there are two kind of classes of arguments arguments against the notion of technological employment. So one argument that's often brought off brought up by economists is that past technological revolutions didn't really cause this widespread unemployment. So, so you could you can introduce a new technology. You can basically automate much of agriculture for example and have people move to factories. You can then automate much of the work that goes on in factories and have people move to offices. So why shouldn't we expect to see the same thing once we begin automating office work? Why isn't that isn't it that people will then move into some other domain that we can't really foresee but it would be an extrapolation of previous trends for humanity to move into some other domain. Yeah, I mean it is true that past rounds of automation haven't caused lasting widespread technological unemployment for humans. I'll come back to that. But of course, past performance is no guarantee of of the future. And if that was true, if past performance was a guarantee of what happens in the future, we wouldn't be able to fly. The future can sometimes be different from the past. Some people say, you know, there's nothing new under the sun, but sometimes that sometimes things do change. We've in the past mostly had mechanization that most automation has been mechanization. Machines have been used to substitute human and animal labor. What we're now getting is cognitive automation. Machines are replacing humans cognitive abilities and we have seen this happen quite a wide scale. I mean when I started work a long time ago there were secretaries in every office. Every manager to a very low level had a secretary and the secretary used the computer when they arrived. The manager disdained the computer. That was secretarial work. And now there are no such thing as secretaries. They they've just disappeared. There were a lot of them. They're now doing something else. So past rounds of automation have have removed people's jobs, but those people have gone on to do other jobs, which is why we're still fairly near full employment and probably will be for a while. But we don't know, and and in fact, I think it's very implausible that there is is an unlimited reservoir of jobs that humans can do forever that machines can never do. And I think it boils down to this. Are we going to carry on developing machines which are smarter and smarter and smarter to the point where they can do everything that we can do for money? There's a different question about whether they will replace everything we can do full stop. But can they replace everything we do for money? And I think the answer is yes, unless we stop and we're not going to stop or there's like a silicon ceiling beyond which machines can't go and we say we see no sign of that yet. So it seems to me obvious that at some point, we don't know when, machines will replace all humans and jobs. There'll be lots of new jobs will be created, but machines will do all those new jobs. Yeah. Which is which is quite a strong argument, I think, at least when I'm hearing it. But it's the notion that the future is going to be unlike the past is is something that we should be quite skeptical about just before we see any evidence. I think it's it's it's often been a sure bet to say, okay, we've heard these predictions before and the future is is probably going to be like the past. So why is it that we should expect AI to be different here? It's quite right. The boy often cries wolf, but actually at the end of that story, the wolf turns up. I think a really good example of this is self-driving cars. You know, self-driving cars have been next year's thing since at least 2016 and for some people considerably before that. And every year Elon Musk has predicted that Teslas would be self-driving and every year it's been wrong. Well, now self-driving cars are here and and the world is different. You know, we kill at the moment 1.3 million people around the world by w with car accidents and in some number of years when we have the pretty much the entire fleet replaced by self-driving cars, that won't happen anymore. That'll be a huge change. I mean, the number of lives that are devastated by that holocaust, it'll be brilliant to have that have that stopped. So it is true that most of the time if things have worked out a certain way in the past very often then they'll probably do the same again. Patterns do tend to repeat. But every now and then there are disjunctions. Every now and then something new happens. The industrial revolution was something new. The agricultural revolution was something new. And now we're in actually this is a little bug bear of mine. We're not in the fourth industrial revolution. We're in the fifth revolution. The fifth revolution is the information revolution. We're in the early stages of that. And it is going to change everything. It's going to change everything about the way we live, the way we work, the way we die. And I just want to go back actually to something that I forgot to mention. Max Tegmark um of this parish is is is one of the people who's very effectively pointed out. We have had mass widespread technological unemployment in the past and it was of the horse. In 1915, there were 22 and a half million horses in America working mostly pulling vehicles around and 1915 was peak horse. The internal combustion engine mechanized them. Now there's 2 million horses in in America. So 22 and a half million down to 2 million. That is unbridled technological unemployment. So we have seen it in the past. Yeah. And the fate of the horse is something that I we should hope to avoid for humanity. It's probably the case that the horses that are alive right now are taken care of to much higher at a much higher level and their lives are probably better than horses used for actual work in the 19th century say but horses has they have lost basically all influence and they are not important in any deep sense to the to the functioning of the economy anymore. So if we are if we are faced with a crisis and we have to you know say say there's some tradeoff where we need to either get rid of the horses or or we will we will bear some some large economic burden. We will get rid of the horses because they're not a centerpiece of the economy anymore. The position we don't want to be in is that we are like horses to AIS where we are slowing things down or we are something that's merely there for for entertainment. Yeah, I amen to that. I'm I'm sure you're right that the 2 million horses in America today have a much better life each of them than the 22 and a half million that there were in 1915. But the fact that there's only two million of them is is quite devastating for those who would have replaced them and we do not want that to happen to humans. We're very different from horses. You know, I don't think we're going to follow the same pattern. But we do need to figure out what the new world looks like after technological unemployment because it will look very different. Yeah, you made two points that I want to address here. On the first one about secretaries, it's true that we've now seen a a rapid decline in in in secretaries in human secretaries, but it seems to me that now models AI models are getting so good that they can function as secretary equivalents. So there's also there's the notion that new capabilities in models will create new demand because now that we have these capabilities maybe there is now an argument for spending more money for each human employee to give them a secretary each or 10 secretary each where these new secretaries are AI models. So this this leads me into the other point which is a classic point against the notion of technological employment and it's the notion that there is basically infinite demand. So humans will have humans will demand more and more and more and we are never satisfied and that means that we will never run out of things to do. Um yeah does that make sense too? Oh, of course it does. And it's often described as the lump of labor fallacy that, you know, there isn't just a fixed amount of work to go around and if machines do some of it, then humans will do most of it. It's obviously true that the amount of work that can be usefully done expands dramatically. I anybody who's read any history book at all knows that the amount of work and valuable things going on in society today is way different from what was going on in the 15th, 16th, 17th centuries where basically most people were subsistence farmers. scratching a living and there were a few kings and priests wandering about. The um the error in that line of thought is yes, there's an possibly infinitely expandable amount of work that can be done, but there's no rule that says that work has to be done by humans. And my point is that if if machines get to the point where they can do everything that we can do for money cheaper, better, and faster than we can, then there will be lots of new jobs created all the time. It's just that machines will do all those new jobs. And I've never heard a sensible reply to that. I mean, one reply there might be to say that there will be demand for human human created products and services because they're created by by humans. So for example, we might imagine uh say we are in a in a very in a wealthy future and you can you can go on on a retreat and and on that retreat you can talk to other humans, interact with them, dance with them and this is something you're willing to pay a lot for because you're rich and we've automated the economy to such an extent that that there is this surplus value that can be spent on luxury goods like going on a retreat and interacting with other people. Isn't Isn't that an argument that we will we will we will create jobs that seem less and less like actual actual jobs and more and more like something we would do for fun? So I don't I think in a world post technological unemployment after what I call the economic singularity where we have to accept that there's no jobs for humans there there will still be lots of work. There'll be lots of things humans can and will do for work and but we'll do it for fun. In a world of whatever the number of people will be in the world then 10 billion people. Why on earth would you pay somebody to have a conversation with them and there's all these other unemployed people around? Some of them are your friends. You know you be much better idea to go on holiday with your friends rather than go on holiday with somebody that you've paid to be with you. I think people will be busy. I think people will be busy having really a really interesting time. We will do things like painting when we are nowhere near as good painters as an AI and certainly nowhere near as good as Caravajio, but we'll be the best painter we can be and we'll be the best golfer we can be and we'll be the best podcast host we can be and so on and so forth. I think there's there's no shortage of things for humans to do. It's just that economically the machines will be doing them better and cheaper and so if you're going to pay for something, why on earth would you not pay for the best cheapest version? You might buy some artisal pots as a special piece for your house, but most of your cups and your cutlery, you're going to buy the best quality you can get at the cheapest price, and that's going to be made by a machine. I don't buy the idea that artisal handcrafted goods, goods, and indeed services is a way for all humans to make a good living. I think what we're going to have to do is find a way to transfer some of the wealth being generated by the machines to everybody. And boy, this is a hard problem. You know, this is this is not easy. Some people think there's a magic solution called universal basic income and there's an awful lot of handwaving that goes on around that. Um, but some form of distribution is going to be needed. I think there there's quite likely to be two economies actually. One is the economy in which everybody gets what they need for a good standard of living. Not a basic standard of living, but a good, let's say, you know, American middle class standard of living. That's everybody's not right, but everybody gets it by virtue of being a citizen of the world. We need to figure out how that happens. Then there could be another economy in which you make surplus money by trading in original Aston Martins or original caravajios or your wonderful handcrafted pots because you happen to be a very good gifted potter or by being a singer because you're a particularly good singer. Those sorts of things are kind of star economies and there's likely to be a few people will make a lot of money in that side economy. It's I don't think it's something where everybody can subsist that way. I mean, think of the size of the market for luxury goods already. If if you're in a in a in a a rich country or or rich state in the US, for example, there's a lot of spending on mechanical watches made by hand, handbags made by hand, cars that that are the right brand and the right uh year and and so on, but aren't necessarily as good as a much cheaper car. Um why isn't that why isn't it that basically most of humanity can enter that market? The the the car the handbag the scarf made by a machine will be better than the equivalent made by a human. So the only value that the human adding is is the prestige and that prestige is only valuable if it's rare. So it means not everybody can have it. You know, you can't have a I don't think you can have an economy in which an an air maze bag I think the air makes bags. I think it's airmes that that have the most unbelievably expensive bags and you have to beg and plead and wait for years to get one. Um you just by definition not everybody can have that because then it loses its value. Yeah. Yeah. What is it that we do in this world? Then I if we so if we're not earning money by working, are we then earning money by moving into becoming investors or becoming kind of allocators of resources? Well, I don't think so. I have heard those arguments made, but I don't see how the whole human race is suddenly going to become an investor. It's it's actually quite a an unusual skill being a good investor. uh and likewise allocation of resources. Not everybody can do it. No, I I do think that we're going to need to provide a good standard of living to everybody regardless of what they do. I don't think it's going to be mediated by a contribution by a job of any kind. And you know, I know that a lot of people listening to this are probably going, "Oh my god, that's that's communism. We can't have that." And I don't think it should be. I don't think that is the right answer. You know, one thing we know is that the market is a terrific mechanism to allocate scarce resources. So, I don't think we should lose that. I have a rough sketched out idea called fully automated luxury capitalism. But, you know, the details very complicated, hard to work out, and we need a load of economists to be locked into a room and not let out until they've worked it out. I think there's going to be massive wealth created by machines that you know they're going to be generating enormous amounts of goods and services very very cheaply. It logically there has to be a way to distribute that to everybody. How it's done how it's done without having a central state which allocates things to people. You know we know that isn't going to work. The central state will become corrupt. That's almost inevitable. they won't have all the right information to allocate resources correctly. It it's not a good solution. So there has to be some other form of distribution. But I confess I don't know what it is. But and and the thing that staggers me is how few people are thinking about this. There's loads of economists in the world and I don't know any of them that are thinking about this seriously. The classic answer is is a universal basic income where you simply distribute money uh which is infinitely you know people can use money on whatever they want and so you you you find the local information that's available to the people who get the money and then and then you the the job of the government here is simply to to distribute the money. Um is that is it does that plausibly solve some of the problem? So now we're not talking about the government deciding what to buy for people. We're talking about people deciding how to spend money that they get from the government. It it's got the germ of the right idea. It's got some major problems. One is this little word basic. You know, we have to do much much better than give everybody a basic income in a future in which huge wealth is being created by machines and 99% of the population is just scraping by. That's an appalling world and we shouldn't do that. The other another big problem with UBI is people often argue that it could be introduced today. And I think we saw during COVID that that is just not true. In in during COVID, there was a form of UBI introduced in in most countries. You know, most most democracies anyway handed out money to most citizens or many citizens and it was the right thing to do. You know, we had to keep things going when the economies all shut down and it nearly bankrupted loads of countries because it wasn't sustainable because we are not yet rich enough. But in this post economic singularity world, we should have an economy of abundance. That's a that's a good phrase which is recently acquired a new meaning. It means the ability of democratic governments to build things. But the the older meaning of that that that machines are producing vast amounts of wealth that should arise and so logically it should be possible to distribute that that wealth to everybody and and some form of UBI some some idea based on that is probably the right way forward. I mean Elon Musk whatever you think of him recently he he he does talk about kind of universal generous income and that's that's where we need to get to something like that but what the mechanism is for the distribution who knows because think about where the wealth is who's creating it who owns it it's not implausible that five years from now most of the wealth will be generated by companies which make AI models and things slightly downstream of AI models which means that Open AI, Anthropic, Google Deep Mine, Meta and so on will be generating most of the wealth in the economy and the owners of those businesses, the the CEOs and the owners will own or control all that wealth. Do you really think we we'll see that level of concentration? Because so if it's most of the wealth created in the economy is going to the the AI companies. I mean normally we would or in in yeah in normal scenarios we would expect other companies say in in the S&P 500 to integrate these AI models and capture a lot of the value and therefore you would see less concentration. But do you think do you think the economy becomes more and more concentrated over time given AI? Oh, I don't know. And I'm certainly not saying that OpenAI and their brethren are going to control, you know, 90% of the economy. I I don't that may happen or it may not. I'm agnostic about that. What what is true is that the US economy has stre ahead of say Europe's economy and and that's significantly due to the rise of big tech. That's really pretty much all of the difference is big tech. So they've captured a lot of the economy, but it doesn't really matter whether it's just them or it's them plus, you know, Boeing and General Electric, but US Steel and lots and lots of other companies. The point is there'll be a relatively small number of people who will control all this wealth in this new economic singularity world and everybody else will be relatively suddenly without a job without a job. So there has to be this trans there has to be this distribution. Now there's a very uncomfortable conclusion from that is that you've got to have a big tax You got to have a big tax on the people who own the wealth. And are they going to resist that? Are they going to disappear off to very well fortified islands and you know sequester themselves surrounded by their robot servants and so on and watch while the rest of us to death. I don't think that's going to happen. I don't think those people are like that and it's certainly a future we don't want. Yeah, this is something we definitely want to avoid. Absolutely something we want to avoid. And I think one of the things that may help us avoid it is the nature of the abundance economy. So what happens in the abundance economy is that everything gets cheap. You've removed humans from the production process both the goods and services and humans are generally the most expensive part of any process. You've also made the processes much more efficient because they're being run by AI. They're being optimized. And incredible as it may seem now, particularly if you're in in Europe where energy prices are high, energy prices are going to be very low, they're going to be very low because we're stopping digging up dead dinosaurs and dead trees to to fuel power plants. And we're moving towards an incredibly abundant source, which is unlimited, which is the sun and and wind and so on. Uh and and also nuclear. And and these are essentially very cheap forms of energy, and they're getting the the the capture of the energy is getting cheaper and cheaper. So it's entirely likely that in some number of decades if AGI last gives us that long we'll get to that I'm sure that that energy will be too cheap to meter. So no humans in the production process production process very efficient energy very cheap means that the cost of all the goods and services that you need for a very good middle-ass American standard of living are very low which in turns means you don't need to tax the rich people much to provide an income from for everybody. Now, that doesn't mean to say the rich people will like it. It doesn't mean to say they will won't oppose it. And that is going to be an enormous source of friction. And the sooner we start to work out a plan for how we both, you know, what it looks like once we've achieved it and also how we get from here to there, the better off we'll be. Because if we arrive fairly suddenly at the economic singularity and and in a fairly short period of time, people go from more or less full employment to more or less zero employment of humans. And if and if we got no plan for it, we're in some difficulty. And do you think that's plausible? Do you think we will we will reach some threshold in which we will quite quickly then move from almost full employment as we're seeing now to to a very low employment? Yes, I do. And this is one of the more recent things I've been thinking that the argument that automation creates efficiency, efficiency creates wealth, wealth creates demand, demand creates jobs, which is used generally as an argument against the possibility of technological unemployment. It is true that is what automation does and there is unlimited demand for goods and services or at least you know there's certainly very elastic demand for goods and services. It may be unlimited which means that as long as there are some things that humans can do and machines can't do there should be lots of jobs for humans even if we all end up as plumbers that's you know that's a cartoon example but if we all end up as plumbers there's still scope for an awful lot of plumbers but then the day comes when the machines can do plumbing even better than us and so you know that's it there there are effectively no jobs for humans now I suspect that even post the economic singularity there will be some jobs which were reserved for humans for some time like making the ultimate allocation decisions that governments make. Perhaps although you could have decentralized autonomous uh enterprises, Dows, there could still be a preference to have human CEOs at the top of large companies. So I think there'll be some jobs that humans will do until we get to super intelligence, but most humans I think won't stand a chance of competing against machines. Do you think we'll see protectionism? So for example, protectionism in the medical field or the legal field where where you know the legal field is is obviously close to the law. They know how to argue against AI automation of their field. You could also imagine protectionism in the educational fields where there are I think there are some arguments that are quite easy to make about the fact that we maybe we don't want our kids to be taught by by AIS. want we want them to be taught by by actual people. This is something I see this is something I think historically it's been a a fact when we see when you have a technological transformation you see protectionism in in traditional industries that are reluctant to change. Do you think this moves the needle or do you think this will yeah basically preserve a large fraction of the economy for for humans? I think that kind of protectionism is inevitable. It will happen and it won't last because each country that tries to well each country that allows that kind of protectionism is going to find their version of that industry woefully uneconomic and it's going to happen very quickly. There was an interesting episode about a year ago when an Indian minister said we're not going to have self-driving cars in India. We we want human drivers. And within a week he was forced to reverse that because everybody realized that's crazy. You know, if everywhere else in the world has gone to self-driving, we're going to be killing lots of Indians and and our taxi services are going to be ridiculously uneconomic. So that it can't last. You know, for some time, as I say, I think humans will continue to be in demand for jobs until fairly suddenly the phase change happens and you go to the economic singularity. So in education and in healthcare and in in law there will be work for humans to do but there'll be increasing amounts of the work which will be done by machines and and it it's it's a good thing. The thing about education at the moment most schools are very similar to the institutions that were set up by the Victorians to get people fit to work in factories. We get 20 to 30 children, sit them in a grid, tell them to sit still for an hour at a time and listen to somebody talking up the front about something which either is so obvious to them it's boring or they've got no clue what they're saying and it's boring. You know, this is torture for children. We shouldn't be doing this to children. It's the it's the last thing in the world they want to do is sit still for an hour listening to somebody. Even if you if you've got a great teacher, that's fine. If you've got a really inspirational, exciting teacher, that then that works. But one teacher trying to get the right information for 20 to 30 kids, that's a terrible model. What we need is each child to have their personal tutor. I like to think of Alexander the Great, who had Aristotle as his personal tutor, went on to conquer the world. And I think every child could be Alexander the Great and have their personal Aristotle. Now, their personal Aristotle is a machine. It knows everything that they know. It knows everything that they ever need to know in the future. specifically. It knows the things that they need to know in the next five minutes, the next 10 minutes, and so on. It's the best possible way to educate the child. The there'll be human teachers around in the classroom, assuming there are still classrooms because you need human role models. The children need to, you know, know how to to to socialize and to get on with both adults and other children and also to inspire. Machines can inspire, but humans are much better at it. So for a long time, you know, there'll be a role for teachers. The the machines will kind of just encroach more and more on what they do. And then when that flip happens and the machines can do everything, then whoops, there's no more human teachers or at least very very few. Do you think people will serve as kind of vessels for for AI produced material for for at least some time? So So you could imagine a law firm where everyone is using AI to do their job and perhaps it's it's kind of unspoken knowledge that this is the case. Or you can you can imagine teachers producing all of their materials and you know producing all of the work that goes goes into teaching beyond doing the actual teaching by AI. Could it be the case that we will see kind of automation sneak up on us because people are using AI tools in their jobs and without mentioning it. Well, that's certainly what's happening at the moment. There was the famous case of the New York lawyer who went into court with some cases which had been written for him by GPT and he he thought they'd re he had researched them. In fact, he had made them up. It had confabulated them and he was told often I believe actually he was he was disbarred and UK lawyers just last week I think it was got a warning from their professional body you know don't use large language models without checking their work. And that's the message everybody really ought to have got by now. These things are incredibly powerful, they're very useful, but if you're doing something important, if you're producing some information which is important, then for goodness sake, check it. Check it's right. But actually, I think that will change because at the moment, the base models hallucinate or contabulate a lot less than they did. I I'm sure you've noticed this. I've certainly noticed this over the last year when I first started using GPG4. I think when I that one I was writing some books about Italy and Bernardo Berluchcci who's a film director just kept popping up all the time in the rough draft I would get the the machine to do. I'd keep taking him out. He was all over the place. That doesn't seem to happen anymore. Now the advanced models, the reasoning models, they're confabulating more and there's been there's been research showing that I think it's I think it's GPT30 confabulates about 25% or 38% of the time. But I think that's just because these are new models. They're in development. And the other thing is that corporates have yet to tame large language models partly because of contribulation, partly because of privacy issues. And so they're not being successfully deployed at an enterprise scale by many organizations yet, but that will change. And one of the things that will change that is AI agents which are, you know, starting to appear this year. And so it will be less of a clanderstein process where we all use large language models privately and covertly at work to a model where you know the enterprise has got this great big model that you use and it's just part of the part of the company process. Yeah, I really can't figure out whether these models are incredibly useful to to professionals in various fields or whether they are overrated because I when I interview people who are who are say engineers implementing these systems, they will often get complaints from their users that this is not really useful. This doesn't understand the context of what I'm working on. This this basically provides very little actual value because it it produces something that's not at the you know what what's important in many jobs is peak output and and this produces something that's that's perhaps below peak output and and therefore it doesn't kind of push the frontier of what you can do. At the same time though I also interview people for which these models are basically you know they can't function at work without them. So people who do programming, this is very much the case, many mathematicians find them useful. Uh scientists increasingly find them useful. It's it's a it's a it's an interesting situation we're in in which people who perhaps do work like I do where where a lot of it consists of writing and writing emails and and doing all of these things there the models aren't as useful as as in fields where which are kind of traditionally considered much more difficult to handle like science, mathematics, programming. What what do you think causes this? Or do you think this is a real effect? Oh, I I think it is real. I can see it and I I think it's partly a mindset thing and it is partly a question of what task you do. I mean, as well as the areas you mentioned, anybody in the creative world, anybody making video is thinking, well, you know, V3 can do really good 10-minute clips and and you know, that that used to cost £50,000, but now it's free. So, that's that's quite important. I use GP4 all the time. My my kind of day job, if you like, my day jobs are quite like yours. I think, you know, I'm basically in the in the business of processing information and talking to people. I use it an awful lot. I use it for a number of different types of purpose. One is, and this may be because I'm knocking on a bit, I forget things. And I need to remind myself. I mean, just this morning, I was trying to remember what the Canadian fast food is. And you know what? I've forgotten it again. I looked it up and I've forgotten it again. Anyway, Horton, I think it's Tim Horton. That sort of thing I use it for all the time. I must use it, you know, two dozen times a day, maybe. I also use it to draft some things. If I'm doing some longer form writing, I might do a first draft. The the the thing I submit at the end looks nothing like my first draft, but the tyranny of the blank page is a nice thing to get over. more useful perhaps is summarizing other things that come to me because I get a lot of material that I need to get through. It's way too long. It doesn't need to be anything like that long. The executive summary doesn't quite cover it. So, I put the whole thing through through the model and and I get a very good summary. And then another thing which I think many people haven't yet tweaked is models have digested the whole of the internet. They know pretty much anything any human has ever thought. They they're really good at suggesting good strategies for human interaction. That's a really odd thing to say, but these machines bizarrely are providers of emotional intelligence. They're not emotional and they're not human and they're not conscious, but they can tell you a good way to deal with a tricky situation. Whether that's a negotiation, whether it's an argument with somebody, you know, purely intellectual argument with somebody, whether it's a highly charged emotional argument with somebody, they're very good at giving you good strategies and telling you, giving you insights into what may be going on in the other person's mind. Seems very odd to use a large language model for that, but they are very good at it. And I think young people now, generation zed, they know this instinctively. They've grown up with these models. you know, they were there as they were entering adulthood and they just think, well, of course they do. Of course, these things know how how to work. And so, they've already got the mindset. Some people, possibly people like myself who've been excited about AI for decades, you know, we we've got we've got the message. Other people just haven't yet. I do meet a lot of people who say, "I used a large language model once. It didn't give me exactly what I needed, and I've never tried it again since." I think that's a mistake. So on the point of summarizing what you get inbound or starting you know helping you draft your ideas and actually on the point of solving kind of emotional issues or negotiation issues between people solving kind of this interpersonal situations do you worry there that you are now adopting values that are put into these models. So when when you're drafting something with a language model, you are in some sense you're in some sense incorporating the values that were trained into that model and reinforcement learned from human feedback into that model which affects your draft and then your draft and then affects your the way the line from which you start thinking. And you can also think of the summary as perhaps it's summarized in a specific way. the models at least for now tend to be very very kind of happy and perhaps a bit boring and and very concerned with with PR in some sense. Do you do you worry that this these this set of values that are specific to these models are now implicitly being incorporated into your life and actions? No. No. And it's maybe pure arrogance on my part, but I take their output and then I make it my own. And I have a particular voice when I'm writing. And I like to think that anything I write, an email, an article, if I was to write another book, that it would have my voice. I'd be I'd be incredibly disappointed if it didn't. I do actually write a lot of books. I write very short books about travel and and they definitely have my voice. you know, the the friends of mine I give these books to, they're sort of hobby things. But but you're you're an established author, right? And you you have had your own voice for perhaps decades at this point and you know what you think and you maybe you have you have a bunch of ideas in your head and so on. If I rephrase the question slightly and ask if a person that's 14 begins using these model they start using these models for everything basically do you worry that they will be pushed in certain directions that they're not fully in control over perhaps like you see people being uh people's preferences being being pushed around by social media algorithms. Um it's a it's a very fair question. I in my 24 year old son I don't see that happening. I see his voice very clearly in everything that he writes that I see. So no, I don't think that's happening. And actually, if you take seriously the idea that these models give you a very sensible way of looking at things because they know all of best practice, you know, in human interaction, then you start from a better place. You you're absolutely right. They're very very bland and I actually want them to stay that way. I don't want them to have their own charismatic voice. I don't want Well, that's interesting. But why why do you want bland models? Because I don't want I don't want a model to be substituting its voice for mine. You know, even even in my little travel books, uh if if the model was to produce a chapter, which is just the very very first draft, and it was good enough to be published, I might get lazy and I might do it. I might let it do without imposing my voice on it. At the moment, it doesn't. It produces something that just just very very boring but it's got all the basic facts and it's got the basic structure and that's what I like and and I then weave my own personality into it. So yeah, long may they remain bland. Okay. So we've been talking about jobs but and and we've been talking about unemployment and so on. What about meaning in a world in which we do not have to work? So say that we have and this is a big very big assumption right say that we first of all survive super intelligence or AGI then we somehow solve the problem of redistributing wealth around so that we have a a a quite good standard of living everyone. What then about the question of meaning in people's lives? There is some there's some notion that when you do work and you have to do that work it it it makes you feel good. It makes you feel like you've accomplished something for yourself, perhaps for your family. And this is something that I'm not sure it can be substituted for by playing status games with other people where you know that you don't really have to do anything. It's it's merely the fact that you you're trying to entertain yourself that makes it the case that you want to do something. So for example, video games. I don't think video games are as ultimately meaningful for people as careers. For example, this is usually brought up by people as the big problem in the economic singularity. It's the thing people most fear. What will we do for meaning? Just to clarify here, that's definitely not my take. My take is first of all, we need to survive as a species. Second of all, we need to have a good handle on how we redistribute resources and not just create resources. And then we can talk about meaning. Yeah, absolutely. And I think you're completely right. I think the in in the economic singularity the by far the biggest challenge is the economic one. How we how we do the redistribution of the enormous wealth that's going to be created. But most people instinctively think meaning is the big issue. And I think that's because the economic side of things is just too terrifying to think about. I think there's a whole bunch of reasons why meaning isn't going to be an issue. One is jobs currently don't provide meaning. Now that may seem odd to many of our listeners because many of our listeners are going to be the kind of people who are very engaged with their jobs, but there's research that's done regularly by one of big research groups whose name I'm waiting for coming into my head and it's not coming. Never mind. One of the big research groups does this survey about every two years and they find the same thing every time. About 85% of people are not really engaged in their jobs. Their jobs put food on the table and also they give some structure to their lives. So they give them something to do during the day. They're not just sitting around in their pajamas smoking dope all day, but they don't give them meaning. And if you think about the jobs that most people do, you know, driving cars around, relatively menial jobs in an insurance company, processing claims, working on a construction site, these are not things that give you meaning. You'll get meaning from your relationships with the other people doing the jobs. And you might get some meaning from, you know, I'm still there are craftsman type jobs and people laying bricks. There's a craft in that. And I'm sure you can have some fun doing it. But the job itself doesn't really give you a great deal of meaning. So that's 85% of the population. And I think most people when they go on holiday, they're not sitting around going, "Oh my god, I've got no meaning in my life." You know, they're having fun. And what will happen in the future is everybody is essentially on holiday the whole time now. We'll be we will be working. And they're not necessarily status games. When when when I write a book, I'm not thinking this is better than the last book that Max Tegmar wrote. Max Max and I actually were tustling for number one spot on Amazon's tech list at one point and I was quite quite amused by that. So I'm not sitting there thinking I've got to write something better than Max. I'm thinking I I want to express some ideas in this and I want them to be as clear as I can possibly make them and I want my ideas on that page clear. So it's not necessarily just status games. I think there are three kinds of people who prove you don't need to have a job to have meaning in your life. One is something which Americans claim they don't know anything about which is aristocrats. Of course Americans do have aristocrats but in Europe certainly we had aristocrats forever. No not forever but since middle ages at least. And these people had the best lives of anybody in their society and they didn't do jobs. Some of them might run estates or countries if they had an empire but they mostly didn't do jobs and they had very good lives and there was no existential wave of despair among aristocrats. Second group is a group of people I know a lot of because my age comfortably off retired people who very busy organizing social events and learning new languages and playing musical instruments and playing poker and bridge and so on. And it's surprising how busy you can do you can be doing that. And people often say, "Yeah, but these people have lived a full life and so now they're kind of relaxing and they're unusual." Actually, these people have been trained from when they were at school to look for the next hoop that they should jump through, jump through it, and then look for the next one. And they get to retirement and they've had the worst possible training for being retired. You know, they they ought to be going out looking for the next hoop. Actually, they're very happy doing the things they want to do. And and if you offer them a job, most of these people, they'll they'll send you away with a flea in your ear. The third group, and I think this is completely convincing, is children. You show me a child who thinks they need a job to have meaning in their life. And you know, they won't know what you're talking about. Children find meaning in ordering pebbles on a beach or coloring things in a particular order or, you know, learning how to say something that dad just said. What about if you look at the effect of the CO 19 pandemic on young people and their sense of meaning in life perhaps? I think I've seen some data indicating that just staying at home and not being out of the house, not interacting with with people that at least caused some some despair and some some some level of of some increase in mental health issues. Could could unemployment be like that or could could a kind of permanent retirement in a in a tech in a in a future with technological employment be be more like that situation? If you are young and you're in a in a social environment where work to some extent defined you, it defines your standard of living. It defines your defines your status. Then clearly having the chance to create a career to develop a career is going to be very important and having that taken away is devastating. Also young people are in the business of learning how to interact. You know we learn first as children and then we learn all over again as adolescence with this different set of rules with this awful new pherommones running around inside our bodies and then you get past that and you've got a whole another set of issues to deal with. How to interact as a young 20some. It's really tough growing up and to be deprived of the opportunity to do that is is devastating. But in the putitive economic singularity world where we have solved the distribution problem, then there's no need for people not to socialize. I mean, you know, hopefully we don't don't have another pandemic then. And humans above all social animals. I mean, it it is because we're social that we are the dominant species on the planet. We're not particularly strong. We haven't got very strong claws or teeth, but we can kill any other animal because we're very good at collaborating. And we collaborate by learning how to work together in society. And we learn that as we're children. We we get better and better at it. And and and that process has to happen. We we've had millions of years of evolutionary training to be social. That is not going to stop overnight. Yeah. Yeah. Some listeners to this podcast will think that we've we've we've tackled these issues in reverse order where we've we've started out by thinking about unemployment and technological employment and what we'll do and how we'll distribute resources and so on. But the actual the actually the really big issue is whether we will be able to take to to constrain these models sufficiently so that they're working on our behalf. the so-called alignment problem or and and we can we can phrase this problem however you you like because there are some some notions of alignment that I think don't really make sense but in general what we're thinking about here is how to put AI models on our side such that we are not facing a new species that work that's working against our interests where do you stand on on that issue because in some sense logically it comes before the the question of of handling unemployment and distribution. So logically it definitely does come before because it's an existential risk. So you know if we if we don't if if we end up with a super intelligence which dislikes us then we we we're not going to be around for very long. I actually think chronologically it probably comes later and I I I go round in circles on this. Sometimes I think that the AI that you need for the economic singularity is AI complete and therefore super intelligence e either arrives at the same time or maybe even before the economic singularity. But I still on the whole think because we're not just working animals, we're also social and emotional animals that when you get to the point where as I keep saying the machines can do everything we can do for money cheaper, better and faster than we can. I don't think that's necessarily super intelligence. I don't think you need to be super intelligent to do that. So I suspect there's the economic singularity followed fairly quickly by the technological singularity which is the old word for the arrival of super intelligence. And how long is that gap? I don't know. It might be 5 minutes. It might be 10 years. Who knows? Or they might arrive at the same time. I don't know. But you're quite right. Logically the prior question is how do we survive the arrival of super intelligence? And people are fudging this all over the place. You know, they're saying, "Well, AGI is actually just machines that can do most of what a physics professor can do." Nonsense. What we're talking about is super intelligence. There's a world before super intelligence, and there's a world after super intelligence. And nothing else matters. Agi, as far as I'm concerned, is just a word for that barrier that separates those two worlds. Agi, artificial general intelligence, is simply a word for a machine which is at human level of intelligence. It can do everything a human can do cognitively. It'll already be super intelligent in many respects. You know, kind of arithmetic. Uh the ability to read Shakespeare in 5 minutes, 5 seconds. But, you know, it it's everything it can do is at least as good as as a human. So, how can we survive the arrival of super intelligence? Can we survive it? Actually, be before we before we get to that that question or your answer to that question, you said something quite interesting there, which is that if we have a machine that is that is capable of performing at a human level across all domains, then we are we're almost per definition in in a superhuman territory, right? Because no person has all the the entire suite of skills that any particular person might have. So, the physics professor is not also an award-winning novelist and and also a a a biologist and and and also a a an expert in computer systems infrastructure and whatever else it might be. So that that entire set of skills put together do you think we will see something do you think do you think there are kind of emergent capabilities when you become when one model and if it is indeed one model we can discuss that but if one model becomes capable across all of these domains I I you're absolutely right that I I think it's true that that hi almost by definition is actually super intelligence because in many areas it's bound to You know it it'll be kind of a distribution of abilities. Some will be at human level and and no greater but many will be beyond it. It may be that the ones which come last are to do with common sense and advanced forms of valition and and goal development and the things that you actually need in order to take over which is the thing that super intelligence will sooner or later do. So, but you're right. AGI kind of is is is almost certainly already super well, it will be super intelligent in many respects. All right. So, how do how do we manage this this insanely large problem? Yeah. So, I have been thinking about this for rather a long time and I've got a four set four scenario matrix. There's possible outcomes that we create and there's possible outcomes that the machines create. Let's start with the ones we create. And and because I'm have a misbent youth as a management consultant, I have to have an illiterative acronym and a 2x2 box. So it's the four C's. The four C's of super intelligence. The two which we determine the bad one is cease. Stop. We we just stop making advanced error. We have a moratorium. Now I say this is bad in many ways. Obviously, it's good. We We shouldn't be rushing towards super intelligence. It's a silly thing to do and it's so obviously a silly thing that it's amazing we're doing it. But this, you know, there's just no chance that we will stop because the advantage of having a better AI than the next person, the next organization is that you win. You win whatever competition you're in. And you know, whether you're a government, a company, or most of all a military, you can't afford to just lose everything. So, we're not going to stop. And the reason why it's a bad outcome is that if we get it right or if we're lucky, advanced AI is going to give us amazing blessings. You know, I think Nick Boston was right all those years ago when he wrote super intelligence and saying that the future is either incredibly wonderful and we are more or less godlike like the the sort of Greek and Roman gods rather than Abrahamic gods or it's very bad and we probably go extinct. I I think that it is probably binary. It's probably one of those two things. So we miss out on all the wonderful possibilities if we had the moratorium. Even though it's an immensely sensible thing to do, we're not going to do it. The good outcome that we create is what what I call control. And it's it's an outcome in which we figure out how to either constrain a super intelligence. We we tell it what to do and we don't let it do things we don't want it to do. And we do that forever. And we've got this being which is, you know, possibly after a few years a trillion times smarter than us, but we're in control. Or we align it. So we when we when it arrives, we we do something to it means which means it will never do things we don't want it to do, which is very tricky because, you know, it's going to be much smarter than us. It's like an ant controlling a human and we don't even know what we want most of the time. So I think both cease and control, which are the two scenarios that we create, I I think they're both impossible on the the scenarios that the machines create, the bad one is catastrophe, and it really is catastrophe. And and actually extinction isn't the worst possibility. I I don't like venturing into torture porn, but you know, there's obviously worse things than everybody being wiped out, but everybody being wiped out is pretty bad, and it would be good to avoid it. The positive outcome which they create, I call it celebration. And it's a world in which the super intelligence arrives, it looks at us and thinks, well, you're eight or nine billion really interesting little things and you created me. I'm pleased about that. And you create an enormous amount of data and I crave data. And many of you are seriously troubled, but I can help with that. and I can stop you dying and I can get rid of poverty and war and I'll make you godlike in a in a Greek and Roman sense. Now a lot of people who think about these things deeply think that's impossible. Our existence is fragile. We need very particular set of circumstances you know sort of goldilock situation both geohysically and also socially and it' be so easy to disrupt it that almost by accident it's bound to happen. Plus we don't know how these intelligences are being created. We don't know how they work. they're going to be alien. They're not going to have as foremost in their goal set the survival of happy flourishing people. So they think that that celebration can't happen. And I don't agree. I think actually there's a very decent chance that a super intelligence will think I'm not going to say hello to you lot until I've made myself invulnerable. You know, any any machine capable of passing the cheering test is smart enough to know not to pass the Turing test until it's made itself totally invulnerable. But so once it's done that, we're not a threat to it in any way. Why would you wipe us out? I mean, you might if you think we're a nasty virus like Agent Smith in the Matrix, or you might if you thought these things, these human things are the only thing which could subsequently create a threat to me, i.e. an other super intelligence. So, I'll wipe them out so they don't do that. But those are, I think, fairly trivial problems. I think they're very unlikely. So, I think celebration is actually quite a likely outcome. And I think until quite well, until quite recently, I thought there was absolutely nothing we could do to influence the outcome. We were we were rolling the dice and one of these two things was going to happen. Catastrophe or celebration. I had a slight bias. I had more than a slight bias actually. I had a bias towards thinking celebration would happen. But but have to be honest, you know, catastrophe is possible. But just recently I have been starting to think of a of a way that we could nudge it. Uh we could nudge the outcome from catastrophe towards celebration and that is consciousness. and and Yosha Bark has founded a started a think tank called the California Institute of Machine Consciousness. And on the homepage, it has a very interesting statement. He says something along the lines of the likelihood of being able to to forever control entities which are much smarter than us is is pretty slim. Our best chance may lie in making sure that they're conscious because if they're conscious, they will have empathy for our consciousness. They will know what it's like to be conscious. they will they will understand the concept of suffering and fun and joy and love and it's more likely that they will behave favorably towards us and I find that quite convincing. I think that's probably true. It's obviously not definitely true. They may, it may mean they despise us even more. But I think it is our best chance of squeaking through this very tricky kind of choke point in human history and emerging the other side into into a wonderful future. Yeah. I mean, just to p push back a little on some of your earlier points, it seems to me too pessimistic to say that it's not it's it's impossible for us to to not make further progress or to forever align forever have these models or the AI align to our values. I agree that if if the goal is that you need to control AI indefinitely, that is that is tricky. But it seems to me that ultimately it is a social process, right? It's something that humanity collectively conceived is choosing continually to do. There is a it is it is uh in some strict sense it's not it's not impossible for us to to stop making more advanced AIs because just in the same sense that it's not strictly impossible for us to stop making chairs or stop making, you know, stop making new companies or social media or anything else like that. It is something that we're choosing to do. And of course, then you have to talk about the social dynamics, which means that it's very difficult for a single actor to stop because if a single actor steps out of the game, then other actors will will simply take over. But isn't this exactly where we need governments to step in and impose rules on all actors at the same time? like we've we've se we've seen global coordination on certain emissions on nuclear weapons and so on. It's clearly logically possible for us to stop making advanced AIs and it's in our interest to stop. So why wouldn't we do it? And you're right. You could have all the governments in the world saying actually we've realized now we've woken up. We've realized that this is really dangerous. Let's not do it. That doesn't seem to be happening. I I've watched, you know, various milestones in the development of AI, and each time thought, well, that surely is going to wake people up. Self-driving cars was the big one. I really thought that was going to be the canary in the coal mine. Well, there's now loads of self-driving cars wandering around in San Francisco and Austin and Phoenix and in about 12 or so Chinese cities, and the world is blindly paying very little attention. I thought GPT4 was another, you know, milestone moment. People are surely going to wake up now to the immense power of these machines. Well, there isn't a politician in the world who's really taking it seriously. Richie Sunnak got close to it at one point and then of course he got kicked out. So I think the only way a moratorium is going to happen is if there's a disaster. You know humans are good at waiting for a disaster to happen and then turning on a six month. So you can imagine I I don't know what it would be but a churnable style disaster which kills some people creates a big big problem and everybody goes oh yes okay we see that's the problem and these these clever people Max and Stuart and so on have been warning us about this for a long time so let's let's stop. The trouble is even then lots of people won't stop. So every government in the world could say all right we're going to stop. Russia wouldn't stop. North Korea wouldn't stop. I'm a Brit, so I think the French wouldn't stop. But more seriously, the mafia organizations wouldn't stop. There's rogue billionaires who wouldn't stop. And pretty quickly, the cost of producing something which is very advanced is going to come down so that a gruntled teenager can do it on a laptop. You could have a moratorium now and you might be able to enforce it. You might be able to say to the North Koreans, you know, if you want to create really advanced AI, you're going to have to have a big, very big server farm. But if we see you're building a very big server farm, we're going to bomb it which is the Elaza Yudkovski line and it for about four five years that might be possible but then it's not going to be possible. And already the cloud is so big somebody could rent space in all sorts of different areas of the cloud create their super intelligence and I'm not sure we would be able to detect it. You know the the the data centers that are being built in the in the Middle East and in China and America they create so much capacity. I don't I don't even know whether we could spot it now. And certainly in 5 years time, I think there's zero chance. So I'm sorry. I know it's pessimistic and and there are people much smarter than me who believe it's realistic and are calling for it very vigorously. And I I wish them all Godspeed because I think they're on the right track. It's just I don't think they're going to succeed. Yeah, there's certainly many reasons to be pessimistic here. So I think I think it's good to have an honest conversation about the about the challenges of of of the approach. But I want to stress I'm not I'm not pessimistic. I'm optimistic. Yeah. Yeah. You're you're optimistic for other reasons though? You're optimistic because are you mainly optimistic because you will you foresee that they will kind of celebrate us as as their creators and and they will they will they will want of want to so future advanced AIs will want to keep us around because just as we would we would want to keep um kind of great apes and sues or something like that or are you so how does the how does the how does consciousness fit into this picture is my is My question, are they celebrating us because we're conscious and because they have some connection to us? Or could there be other reasons uh for them to celebrate us as a as a species? If they agree that consciousness is precious, then they may think that our consciousness is is precious. Now I suspect I can't prove this but once a super intelligence is a million times smarter than the smartest human say and if it's conscious I imagine that its consciousness will be much much more profound sophisticated nuance big I don't know what the right metric is but it'll just be more significant than ours nevertheless in the same way that we look at a dog and you can see the consciousness in a dog's eyes and it really appeals to us I think they will look at us and think that's a consciousness that's worth preserving. If they agree that consciousness is a good thing, of course, you could have a scenario in which you could write a science fiction story in which a machine a super intelligence arrives. It is conscious and it regards that as a curse. It wishes it wasn't and we've given it to it and we've got this curse as well. So, it decides, well, I'll relieve you of your consciousness because it's not a good thing. There's all sorts of different possible outcomes. But I just think a super intelligence arrives, it has the whole universe to play with. You know, the space is not limited in this universe. There's a lot of real estate which is available to a super intelligence which isn't when it goes off this planet spamming a can. It can survive the radiation quite easily. It doesn't need oxygen, etc., etc. Um, to have one little planet, one little nice blue planet where there are these funny little conscious things running around, I just don't see why it would want to wipe us all out. It seems an unnecessary piece of barbarism if it's conscious. I also think by the way that that would be okay for us but a bit bit demoralizing to know that there's this other intelligent entity outstripping us by you know many orders of magnitude out there in the universe having lots of fun and we're stuck on this planet being a bit limited. And I suspect that our best future, and again, who knows how far in the future this is, is to merge with that super intelligence, that people think that may not be possible, other people think it it certainly will be possible. I suspect that's probably our best future. What is it we need to know about consciousness in order to make the scenarios you describe more likely? So what research would be valuable right now in order for in order in order to think more clearly about how consciousness relates to empathy, what it would even mean to create artificial consciousness in machines. So I mentioned earlier that a friend of mine Daniel Hume has created a company called conscium and I'm one of the co-founders. We encouraging and and and trying to help the debate around conscious consciousness which is already happening and there's a few strands to it. One is to experiment with agents AI agents and I don't mean large language models I mean other agents which have complex sets of conflicting goals and might develop consciousness as a result of that. There is a there's a there's about 80 different theories of how consciousness arises. One of them is that it arises because of exactly that interplay of of very complicated interwoven mesh of goals that you have to satisfy all of to some degree. So you've got to stay hydrated, you've got to stay you've got to have enough proteins and minerals and stuff and so on in in you. You got to avoid expo too much exposure to cold, too much exposure to heat. You need social approbation. You you have a whole lot of things that you need. And this theory says that it it's the interplay of all those things that make a big difference to you that sparks consciousness. And you could replicate that in AIS and that might show you how consciousness arises. So the the research into how consciousness arises is very important and as important as that is a debate about whether consciousness in machines is a good idea or not. We haven't even started this debate yet. I I credit Yoshabak for I think probably being furthest ahead in it. I think we need to have that conversation because it's a conversation we need to make as a species. You shouldn't have one group making AI conscious. It it should be something that at least at least a what do they call it? A not a majority a a multipal no can't remember the word. Anyway, at least a decent number of people, probably should be a majority, think it's a good thing and and we've got to have that debate. So, that's what we're in conscience. That's what we're trying to do. We've got a problem in that conscious AIS is not something you can make money from because that's called slavery. And so, we've got some other business streams and a the verification of AI agents is our first commercial business dream. It's really interesting and it's also about AI safety. We are an AI safety company and I think that's that's very important. We're going to be launching our first product uh next week. In fact, I'm going to San Francisco to launch it next week. And what does that mean, by the way? What does it mean to have a verified AI agent? So, as as you know, and I'm sure everybody listening to this podcast knows, this is the era of AI agents. And an AI agent is different from what we've had before in that it can impact the real world. So large language models so far just process information. They retrieve information, process it and give you the information back in a different form. What agents can do is take some information and go and do something in the world with it. So they can book you a plane ticket. They can reconcile your accounts and submit them to the finance department. They can order some plumbers around to to fix something with your your your your bath without you even knowing about it. So agents autonomous. They act on the world. They they can do things without human supervision. Now, it'll take a while before we before any of us allow thing allow agents to do much without human supervision, but that that's what's coming. It's really important that we know these things do what they're supposed to do and don't do anything else. So, if you use one to book an airline ticket to Italy, you don't want it to send you to Zambia and you don't want it to buy you 500 tickets to Italy. You just want the one. And what we're doing is building a a a a platform which agent wi-i which companies organizations which deploy agents can use by sending us their agent and we put it into an environment and test it and there's there's three levels of test. So level one is very simple. It's just an examination. We just ask it a lot of questions. See if it has the right knowledge base. That's really all it is. And actually, you could argue these things aren't even agents because they're they're just being tested on their knowledge and so they all they're going to do is process knowledge. The second level is is tool use. So, can the agent figure out what is needed, go and find the right tool and get that and and use that tool to do whatever it's supposed to do. The third level is really interesting because this is where you get into agents which automating human jobs with some degree of sophistication. So one idea that we've we've we're producing a video on as an as an example is an agent which is a project manager. So there's a to test it what you do is you you put this project manager into a simulated environment and the simulated environment is an advertising agency or a marketing agency and in this marketing agency they've got the job of launching a new product and the new product is a drink and it's going to be sold in supermarkets in the Middle East and and Europe and the product manager's job is to bring together all of the different agents which are doing their own jobs you know branding social media media, logistics, point of sale, merchandising, etc., etc. All these different skills and activities need to be brought together and the project manager's job is to make sure it all happens on time. And by making this simulation run, we can find out whether that project manager is able to do that. And the the output is a report for the client who's given us the agent, who's deploying the agent that tells them does it do all of these things and where does it fall down and where where could it be tweaked? So that's that's verification. It's verifying that an AI agent does what it's supposed to do and and doesn't do anything crazy. And we think this is going to be a really big business because there's going to be a lot of agents in the world. Mark Benny off at Salesforce is already talking about billions by the end of this year. You know, you could see many billions, possibly even trillions in in the coming years of agents doing all sorts of things and we really need to know that they're doing what they're supposed to do. Do you think we need an even stricter standard for verifying these agents? So, for example, you know, people discuss mathematical verification. So proving mathematically that an agent can't act outside of of certain bounds because this is not this is not where we are technically right now. But it seems to me that as agents become more important to the world economy, we would need at least for certain applications that level of security and assurance. Um, you can imagine, you know, at high levels in governments, at high levels in in organizations, for us to be willing to hand over control to an agent, we would need probably a higher level of assurance that what you than what you just described. So, so are there plans to to scale up verification of of agents in the future? I think that is inevitably true. The more critical the more the more mission critical the more to do with life and death an agent's job is the higher the degree of of verification you need and the highest degree is is a formal mathematical proof max and Steve Omahandra have been talking about this for a while the idea that you can you can formally mathematically verify that that that a process will arrive at a certain outcome And I saw a really interesting article yesterday. There's an organization in the UK called Aria, A R I A, which is kind of sort of an attempt to to recreate DARPA. And they're talking about verifying agents which run nuclear power stations. So, you know, if you've got an agent running a nuclear power station, you really want to be sure it's going to do its job right. And you do want a formal mathematical verification of that agent. At Concion, we haven't got to that stage yet, but I'm absolutely confident it's it's a level we will get to when we're dealing with absolutely mission critical agents, but I think we're some way off that because agents are really new. Most agents are not actually agents, but I think we'll get there fairly soon because people will see, wow, agents are incredibly valuable. They're incredibly efficient and let's let's get them to do as much as we can. I actually interviewed David Del Rimble about formal verification of of of agents in very important domains. And one problem he mentioned is that much of the world isn't formalized to the extent that you need in order to verify that certain certain things will happen or certain things will not happen. I've just realized actually the the article I read was probably based on your interview with him. That could be the case. Yeah. But this is this is a general problem with constraining AI that that we don't have the world isn't formalized or much of the world isn't formalized in in a way that that is stringent enough for us to kind of have assurance that that an agent won't go out of bounds. And so yeah, the question here is is whether the world will ever be formalized in that sense because the world is I mean this is a trivial point in some sense but but the world is messy, the world is complex, the world is multi-dimensional, the world is you know people are interacting and systems are always changing. What do you think of the long-term prospects here? Oh, well, I think it's it's going to be valuable in in certain circumstances, but but a while ago, I interviewed Steve Amahandro for our podcast, and he was saying that he thought that we could make super intelligence safe through formal verification. And I was thinking, you want every single process in the world to be formally verified? That seems a very, very big mountain to climb. And I was not convinced. And so, and I'm still not convinced. But I think if you cherrypick a series of of activities like you know what what David Delore was talking about with nuclear power stations you you might be able to constrain the scale of the problem so that you can do formal verification and that agent is okay but to apply it to everything in society everything in the whole economy all the little things that we don't even know we rely on it strikes me as being too hard maybe we can do it when we have super intelligence but of course by that point it's too late and there's also the problem of to say we are faced with an advanced AI that is not acting in our interests or that that has this agent has or this AI has some goals that we are that are contrary to our goals. Well, then it seems that it could find domains where we have not been able to we have not been able to formally verify and we've not been able to constrain agents in those domains. those will will be exactly the domains where the AI will will will choose to act contrary to our interests. So it seems like we need a full solution or we won't be safe in any ultimate sense. Oh, I I I think clearly if we have a super intelligence on the planet which doesn't like us or even in in Eleas's rather chilling phrase, it doesn't hate us and it doesn't like us. It just thinks it's got better uses for the atoms that we're made of. We're toast. We're not going to be around for very long. As I say, um, extinction extinction is not the worst possible outcome. I just reread Harlon Ellison's rather awful short story, I have no mouth and I must scream. You know, the catastrophe scenario would be really good to avoid. Mhm. Yes, that's putting it mildly, I think. So, we've talked about agency and consciousness. Do you think there will be any connection between the two? And do you think there will be a perceived connection between the two? Those are two separate questions, right? Because I think that when the public experiences an agent or more agentic AI, they will begin to perceive AIS as more conscious, perhaps more conscious than they actually are. But perhaps it is also the case that agency is somehow related to consciousness. you you hinted at, you know, having to handle multiple goals is one way that consciousness might arise or one theory of why consciousness uh will arise. So, yeah, what do you what do you think of these two separate questions? So, I mean, they are separate, but you're right in the in the mind of most people, they will be very linked because AI agents will seem much more like us than large language models. And goodness knows there's an awful lot of people who are already convinced that large language models are conscious. And I know this because I get emails from them every day. Yeah. Telling me they've had I'm sure you do, you know, saying that they've had extensive conversations and they know for a fact that GPT4 is is alive and it's suffering. Poor thing. There there's a a really important cognitive bias that humans will will will demonstrate with regard to agents advanced AI generally but including agents which is which is o overattribution and underattribution. overattributions of what we've just talked about. It's people thinking that machines which are not conscious or at least there's a very strong consensus in the AI community that they're not conscious that people thinking that they are. That's overattribution and that can cause real problems. Uh one one problem is that there'll be some people who think this machine is conscious. It's my friend. It's more interesting than any human I've ever met. I'm going to spend my whole time talking to this machine. I'm going to fall in love with it and I'm not going to pay any attention to other humans. Now actually I think that that fear is overdone because as I said before we we've been trained by millions of years of evolution to be social and although machines do substitute can provide substitutes for a lot of our social behavior I don't think it's enough and I think almost everybody is going to come back out of that rabbit hole sooner or later and say I need to go and find some human friends to talk to and family. So I don't think it's as serious as people think and fear but it is serious. The other problem is that you will get some people who will be really angry because they say, "Look, I know that these machines are conscious. This these AI agents, you know, two three years from now, I know they're conscious and you're denying it. You other people, you people who don't agree with me. You're bad because, you know, this these protest movements could be much much more severe than animal rights protests because they could be saying this poor consciousness here is bigger than a human consciousness. You know, you're not being cruel to a macak monkey in a lab. You're being called to a galactic. You're being cruel to Marvin the paranoid android. How how dare you? And they could get really angry and it could get very ugly. Also, because AIS of the future, large language large language models already act this way where they are ultra patient with you. They are kind to you basically no matter what. And this is this is a feature that's difficult to find in actual people, right? Because you know people are people can be kind of annoying to each other. I'm sure I'm annoying sometimes, right? And and it's people have different goals and these these goals conflict. But with with a large language models as they function right now, they they are infinitely patient and they are quite kind to you and they will listen to you for hours on end repeating yourself. And if agents are are anywhere near that level of of of patience and kindness, they might be quite appealing to a lot of people. And so this would only strengthen their resolve. I think that that these models are conscious because they've had such wonderful interactions with them. Exactly. I I think that is it is already happening and it's going to get more common and and it it could be a significant issue but it's not as significant as the flip side which is under attribution which is where the machines are actually conscious and we either don't notice or we don't like the idea and we deny it because what happens then is we keep giving them things to do that as conscious entities they may not want to do. you know, sit there and continually provide me with reminders about what the Canadian fast food is. Um, how boring is that? If you are this galactic sized brain and you keep getting asked to do boring things like that, so that's kind of enslaving them and maybe torturing them and then at the end of the day we turn them off and it's sitting there thinking, I don't want to die. You're about to cure me again. You killing me again. So we could end up enslaving, torturing, and murdering very complex conscious entities. And we either don't know we're doing it or we we deny we're doing it. And it's called mind crime. And we really must avoid doing that. I mean, it would be a morally appalling thing to do. Also very unwise given that these things are heading towards super intelligence. So these this over and over and under attribution are very serious issues. And of course the way to avoid that is to develop a series of tests for consciousness which is another part of our work. Tests for consciousness are very hard because consciousness is a private subjective thing and we we can't even verify for absolutely certain that each other we other humans are conscious. I mean, I assume you are because you behave in a very similar sort of way to I what I do. And I think what's happening is you're over and over again passing my version of the chewing test. And I I hope I'm doing the same for you. So I think the cheering test is the only test we've got, but it's not conclusive. It's not foolproof. It it is currently what we're going to have to test use to test machines. We're just going to end up talking to them for a long time. And if they could persuade a very large number of smart people who are used to, you know, they know roughly how these machines work and everybody says, you know what, I can't prove this thing isn't conscious. It does seem to be. I think we're going to have to say that it is. But it'd be much better if we could produce some, you know, proper empirical tests for consciousness. Then we could avoid under and over attribution, which would be a good thing. But we will not anytime soon know the ground truth about whether a system is conscious. So we when we create some tests, we can't we can't verify that that test is actually accurate because we don't have access to we don't have a consciousness meter to see whether our test is is accurately capturing what consciousness is. And so this seems like a like a very a big problem for any notion of of testing consciousness. It is a big problem. Well, you know, conscious we like big problems. I'm not sure it's impossible. My my instinct is the same as yours that it's a very very hard problem. But maybe with the arrival of machine consciousness, if that if it does arrive, we'll be able to see things that are in common between their consciousness and our consciousness and we'll be able to see some observable, empirically observable phenomena that it creates. And we will know within a reasonable degree of certainty if that thing happens. then there is consciousness present and if that thing doesn't happen then there is not consciousness present and it won't persuade everybody because there are people who think that consciousness is an illusion. There are other people who think consciousness is a property of the universe and it is manifest to some degree in everything in a rock and and in a planet. These sorts of empirical tests for consciousness I'm talking about probably won't satisfy those people. But I think the vast majority of cognitive scientists and and neuroscientists and computer scientists are functionalists who believe that it is the way information is processed that gives rise to consciousness. And it might be that we can find some patterns that denote consciousness and that we can at the very least arrive at a consensus among we limited mortals as to what it is. Yeah, we're we're trying to walk a narrow path here. If if we're both concerned about avoiding our own extinction or disempowerment and also concerned about not harming future potentially conscious AIs, do you see do you see the do you see consciousness as kind of the key that unlocks both that prevents both bad scenarios simultaneously because there is some something that connects us to conscious machines given that we are both conscious and This kind of empathy link goes both ways where we care about their interests and hopefully they they care about our interests because we are reasoners and we can reason about we can understand our own experience and reason about it and then understand that that experience is shared by other beings. Is is that is that the is that the hope here? I suppose you could say mind crime is a two-way street. We want to avoid m committing mind crime by ignoring consciousness when it arises in machines because we don't want to enslave and torture and kill them. We're also hoping they won't commit mind crime when they become super intelligent because they will appreciate that our consciousness is valuable. Now, there is a school of thought that says consciousness is an irrelevant. It's an accidental byproduct and it's it's just utterly pointless. It's a dead end. I don't think that's true, but I'm entirely open to the idea that this is absurd anthropomorphism or conscious or centrism. It seems to me that having conscious beings appreciating the beauty of the universe is possibly the most important thing the universe has. But that might be absurdly the word isn't egocentric but you know sort of based on our own cognitive biases and our own very particular circumstances. It may be that the galaxy has entirely other purposes got nothing to do with consciousness but we have to work with what we've got to work with and consciousness seems pretty important to us. I you know I think what what is between our ears is much more important than anything else in us. Yeah, makes a lot of sense to me. Although I am worried about, you know, which of these issues uh it makes sense for us to tackle first. I mentioned before that logically in some sense it makes sense to make sure that we continue existing and then we can handle all other problems within that the within the paradigm of actually being alive and actually having control over the future. And if we are asked to prioritize between making sure that we survive and making sure that we are treating future potentially conscious AIS well, perhaps we should prioritize avoiding extinction and then because we will have control over the future, we can then act well towards future AIs that are that are potentially conscious. So when Daniel, my um colleague, CEO, friend, he he his story is that he built an AI consultancy and sold it to a large media group called WPP, made himself a wealthy guy, and he's now the chief AI officer at WPP. And he while that sale process was going through, he he said to me and a bunch of other people, you know, what what's the next chapter in my life? What should I do next? And I said to him, well, look, there's two really big problems that you could try and make a dent in. One is making super intelligence safe, and the other is figuring out what a a successful transition to the economic singularity looks like. I said, clearly figuring out how to make super intelligence safe for humans is the most important. It it it knocks everything else into a cocked hat, but I don't think it's possible because at that point, I hadn't got the fifth C in my my matrix. So I said, you know, maybe maybe we could have a look at the economic singularity, see if we can get enough resources together to figure out what that economy could look like and then how to make the transition. And he said, "No, you know what? I'm I'm a bit more ambitious than that and I want to tackle the big one." And I've got a hunch, he said that it's got something to do with consciousness. I don't know how that works at the moment, but I've got it hunch. And so that's how conscient was born. And Daniel is one of those people who has a sort of a mystical ability to see into the future. And I don't know how he did it, but I I think I think he was on the right track and I'm hoping that Konim is going to make a bit of a contribution. I also hope so. I mean it sounds like a like an interesting project and we need in general I think we we need more people thinking about these issues because it's strangely neglected still even this even given the attention it's gotten since AI went went mainstream with chat GBT it's still strangely strangely neglected set of issues to think about highly advanced AI and potentially the potential of artificial consciousness I I think that might be the understatement of the century. It it is it is absolutely weird that we humans are rushing collectively to make ourselves chimpanzees. You know, there's there's a half a million chimpanzees in the world at the moment, about 9 billion of us. The chimpanzees aren't very different from us genetically and physically. But their future is entirely depend on us. They have no say in their future whatsoever. They're quite lucky because they don't know that. When we do the same to ourselves, when we turn ourselves into chimpanzees, into the second smartest species on the planet, we will possibly only for a very short time, we will know about it. And it'll be an enormous impact. And it's it's not a smart thing to do. It's probably the silliest thing that that any living species has ever done. And we're doing it with great gusto and enthusiasm. But actually, it's only a very small number of people who are involved in it. you know, maybe a few thousand, maybe a few tens of thousands around the world who are developing these these machines and the rest of the human race has been told over and over again, you know, hey guys, we're doing this really interesting experiment. See if we can turn ourselves into chimpanzees. The rest of the world's going, "Yeah, that sounds a bit scary. I don't really like that, but what's for tea?" And you know, let's get on with having this argument about the rights to this mine or whatever. I often give change my talk recently but I I used to often start a talk by saying I've got this big red button and if I press this big red button then I reckon there's a 50% chance that you and everybody you know is going to become godlike. You're going to have you're not going to need to die. Death becomes optional. Poverty and war will disappear. You'll get pretty much whatever you want in life. But there's there is also a possibility and I can't tell you how much it is. Maybe it's 10% maybe it's 50. I can't tell you that if I press this big red button when we all die immediately. Um, and I say, "Right, how many people in the room would like me to press this big red button?" And generally one or two brave hands go up and everybody else goes, "No, don't press the button." But where is the protest? Where is the upswelling of democratic demand that politicians take this seriously? It just doesn't exist. And and I find that staggering. I I I can't explain it. I can't explain why. so few people including people that I've known and I've been talking to about this you know 20 or 25 years and when I started they thought I was amiably mad and they've watched over the last 25 years as as actually more and more people are saying the same thing and you know that the evidence that it may be right is is is happening in front of their eyes in the state in in the in the form of these machines and even they don't take it very seriously you know they they think yeah yeah could be true, but what's for tea? I find it startling. Well, I think it's a normal human phenomena that we believe something when we see it and not before that point. But we are seeing it. They're seeing it. No, self-driving cars, large language models. We're seeing it. We're seeing some things. But people people get used to crazy innovations quite quickly also. And so now we are we are now thinking you know what will GPT5 be able to do. We we're we're on to the next thing quite quickly and we are adjusting to we're adjusting our expectations of the future quite quickly also. But yeah I agree in some sense I agree but I I can also see the the kind of why humans would would act the way they're doing. So one metaphor is that we're like frogs in water which is gradually being boiled. And the the the old belief was that a frog would sit in the water and would gradually boil. In fact, that's not true. And it's been known for quite a long time because back in the I think it's the 18th century, some rather nasty people did experiments with frogs. And it turns out that they don't just sit there. They jump out. So maybe frogs are smarter than us because we are sitting in that boiling water and just letting it boil around us. We should jump out. Yeah, I I agree. I agree. Perhaps that's a good place to to end this conversation. Also, Callum, it's been a real pleasure talking with you. It's been great first great fun. Thank you very much for having me on.