Your next best friend may be 100% AI w/ Purnendu Mukherjee

[Music] it's the 26th century and you're moving through a world as a cybernetically enhanced super soldier on the planet of reach you have a big mission in front of you as Master Chief you are the Last Hope For Humanity against a hostile Alliance of alien forces only you can stop the Covenant well only you with the help of your trusty AI sidekick a hologram with a cropped haircut and shimmering ocean blue skin Cortana actually it was 2001 and I just hooked up my Xbox to play the first series of Halo Combat Evolved I was 11 at the time and my brother our friends and I would pile into our Punjabi living room to play a campaign in split screen mode I was totally sucked into the game invested in the story and Cortana was key to that with our humor and emotional depth even though she's just a blue hologram ironically she was the key to the game's Humanity she was my trusty co-pilot guiding me through these alien worlds and throughout the game I felt a real Bond starting to emerge it seemed totally novel at the time but I was forming a friendship with a non-player [Music] character I'm belaval sidu and this is the Ted AI show where we figure out how to live and thrive in a world where AI is changing everything [Music] sometimes things in the world of technology are complicated and need careful explaining sometimes they just need a little hard truth I don't think anyone is going to buy a banana with crypto at any point in the foreseeable future I'm Liz oir the host of slates what next TBD your cleare eyed guide to Technology power and the future Friday and Sunday wherever you get your podcasts now non-player characters or NPCs for short have always been a big part of video games in singleplayer text based computer games players could interact with NPCs in a very limited way just putting in a very specific command and having the NPC respond from a predetermined script think the original king's quest where players type to directly into textbox to interact with NPCs who were there only to move the plot along this evolved into single player games like Halo where NPCs like Cortana had more Dynamic personalities and quirks written for them though were still limited by the design of the game and therefore static now if you think about multiplayer games they allow for human-to-human interactions think second life where you're interacting with other players through their avatars what's so immersive about that is you're engaging with the real people who are responding dynamically to you and you're responding dynamically to them human to human NPCs exist in this world and while they can be interesting you would never confuse these NPCs as fellow Gamers NPCs just haven't been real enough to be mistaken for actual people up until now these NPCs have been relatively passive but now with generative AI those NPCs no longer have to choose between a limited number of responses they can script in real time just like a human player would reacting to what's happening inside of the game World suddenly video game worlds can be infinitely more immersive and interactive as these virtual characters become more integrated into our daily lives and maybe even become our friends are we going to start spending all our times in these Virtual Worlds will our interactions with NPC start to become unrecognizable from our interactions with humans and what do we gain or lose if this happens this is the domain of convey spelled con vaai a platform that enables developers to create NPCs with humanlike conversational abilities convey says their goal is to help developers create virtual characters that can converse in the present moments and build long-term human relationships with human players there are over a thousand projects being actively built on their platform by creators ranging from AAA Studios to Indie developers and what makes convey so interesting is that they're making technology that enables game developers to create NPCs that are so like liik it won't matter who or what is on the other end berenda Mukerji is the CEO of convey and our guest on the show today he has a lot to say about our evolving relationship with NPCs or as he likes to call it AI characters and like me he was a total gaming nerd pendu welcome I'd love to start talking about the origins of convey I get that you were a gamer before you worked at Nvidia but obviously there's a lot of areas in game development you could have branched into why were you drawn to the notion of AI NPC specifically before I started at Nvidia when I was doing my thesis work I literally saw this language model wave coming I wrote this that while language models are going to get bigger and better and will potentially even have um abilities to have human-like conversation it is still not going to have the same level of understanding as we humans do because we humans don't think from text in text out or text to text we think from a 3D world around us right since we're born we first understand Locomotion like moving around the 3D World and then we attach words to these objects we assign meaning right so so basically we are multimodal creatures where do we find such a m multimodal environment where we could potentially have these AIS live train and and and iterate themselves Virtual Worlds and what kind of virtual worlds would we have uh people that can provide feedback to this AI uh heavily populated worlds are games right so all those connected together like if we have to create this humanlike mind within a virtual world NPCs non-player characters or let's call it AI character embodied in a way are one of the best vehicles to do that it's almost like you know we've got these rich environments where you can embody this sort of AI agent and have it experience very similar stimuli to what we might experience in the real world which is a perfect segue into the evolution of AI NPCs right this is a non-playable character this is sort of cessing the side thing in this like you know kind of like the side dish to go with the main course which is the game itself and so I'm kind of curious um what's your what's your historical perspective on how AI NPCs have evolved from their earliest stages to the complex entities that we see today in games and virtual experiences to talk about the history of games I mean um there are these pioneering genre defining games all the way from halflife that you know like single-handedly Define the firstperson shooter genre and of course counter- strike like the multi multiplayer aspect of it that you could have many people in the same world right in in in various ways of gamep play basically revolutionized gaming you know they could play with each other so you don't need new gameplay as long as people can involve with each other so so like that has evolved NPCs has of course gotten better Mo mostly uh on the visuals or you know animation front uh but not on the intelligence front as much so what we are seeing it's almost like a Cambrian explosion of uh characters and and AI agents that can not only be very humanlike in terms of interaction with the players just like players played with players now ai can also play with players both as friends or enemies you know cooperatively or competitively I I think it's this magical moment where now we've of course got these Behemoth large language models but you also have the you know kind of multimodal models hitting hitting the scene where it's not just that they can understand text these models can understand audio can understand imagery can understand even video right and so I'm curious to dig into how that affects human AI relationship so how do you see AI NPCs sort of changing the nature of player engagement and emotional investment in both games and experiences firstly the way these NPCs are becoming very humanlike like there is going to be a large set of people in the world that will Big Time benefit from it mainly because there is a big big chunk of players who don't like engaging with real people or or nervous or afraid to do that uh they feel much comfortable if they know it's not a real person and that will help them open up they'll help them socialize in terms of people that let's say are playing single player games or multiplayer games now they can engage with this uh with with the set of AI characters and have a more engaging time ideally and lastly let's say if it's in a multiplayer environment people will still enjoy engaging with people but now they have another reason to have fun together with other people and then from a relationship standpoint basically I think it is important for for companies like ourselves to to look ahead in terms of the positives as well as the dangers it is going to quick fill in the gaps where a human doesn't exist today right whether it's uh uh you know like just being friends or from a romantic angle or or maybe someone that is a mentor or a guide and not just like chat GPT like text in text out but very much gamified immersive environment that can um that can reach them and they can they can effectively have this mentor of an of an AI so I think overall I definitely am an optimist and I see the positive sides there could be um potential darker you know sides dystopian sides that needs to be uh addressed and understood and informed I mean I I think it's it's very fascinating the like it's one thing to talk to your chat GPT app right and you see a voice emanating from your phone it's another thing entirely to let's say be talking to your Mentor you know and it's embodied as like a humanoid character that like has the same sort of expressiveness that you do it suddenly becomes this sort of more lifelike experience right and so as NPCs become more lifelike what are those ethical considerations that come into play es especially regarding player relationships and you know AI behaviors the number one thing that I think AI needs and this is a bit controversial but like if you if you think deeply enough right the biggest uh fear for AI is centralization and a few entities responsible for these relationships when a kid grows up uh with an AI teacher and mentor and that's their all you know uh that's a level of relationship that no company in the earth should own it's theirs absolutely wherever they want to take it and um what can enable this I think decentralized blockchain technology can provide true ownership that is going to be very very essential uh along with confidential Computing uh that can uh help ensure that that their data remains theirs their memories and relationships remains theirs yeah I mean you brought up a bunch of very interesting points right it's it's like um if you do have this future where let's say we have an oligopoly of companies that sort of own the models that mediate your relationship with these digital characters right and especially if this is like kids who are growing up talking to these you know NPCs or AI agents whatever you want to call them yeah you are building like a very rich history of sort of their hopes wishes anxieties worries desires and how those evolve over time right and you know these agents are getting to a place where it's not just like oh yes you say this and I will say this it's like they remember the context and glean insights from your previous conversations and you know maybe the right way to solve that is with decentralized AI and you alluded to confidential Computing as well is like can we keep this sort of very rich data um you know as close to the user as possible or you know and even if you are going to learn and improve models off of it you do it in this like privacy preserving way which you know itself sounds like a tall order I'm really Keen to dig into sort of the experiential and sort of emotional impacts of these type of AI agents right and so like one example that I keep going back to is hey even if you're playing a solo single player game you could have this like AI Jarvis or Cortana that sort of understands you your context and is with you helping you navigate this sort of Game World um how close are we to having that sort of companion in games the primary aspect that is missing would be the I mean two some extent right so is the multimodal aspect of it for that to happen at scale like very much similar to Jarvis of Iron Man uh is uh aware of the entire context that means every nuon cranny of the room that you are in you know like they're able to see that process that along with your uh digital presence so we are not far away like H it may not be at 100% of Iron Man's Jarvis capability but like if you play the game it'll feel like that how does all of this change the way these experiences are authored right the the analogy that keeps coming to my mind is sort of the narrative division of Westworld is that the right way to think about how you're authoring these type of experiences with these rich characters the emerging nature of large language models are very interesting The Narrative designers and writers are still the better storytellers right we have to have a right mix of both controlled evolution of characters controlled evolutions of stories as well as the open-ended and emerging behaviors uh of these generative AI models which is where the balance and challenge is and we are providing all the necessary tools for developers and designers to do that to come back to the West World analogy that you know not only did Westworld had these crafted characters that um that were one of of our favorites they evolved right they evolved into something else entirely than what was originally written right and and evolved in a very meaningful manner remember what was in West world that started the whole thing about these characters evolving is memories once you have that their personalities is evolved they remember things and not only are it's enough to just give them long-term memory you have to keep them up all the time go about their day make decisions interact with other real people or other Ai and those interactions will change their decisions and their Pathways and some will be very high intensity experiences and that will shape their personalities and this will be ideally best put in servers that can have up to like 250 500 people these these a inpcs will always be there living their life you know like uh their experiences will will shape them and they will start making decisions that will be quite be wildering we already see that you know like we did this Nvidia demo where we had now two characters and both of them were chatting and uh we just know like we just let them talk to each other right like just to see how how what they chat about and while chatting they Nova made an order that hey can you bring me a drink and Jin was like sure let me get that for you and he went ahead and brought a drink and gave it to her and uh it was so wild that they are not only just talking with each other they're like carrying out actions and they're giving commands to each other the demo that you refer to which went stupid viral on the internet I think it got everyone excited to just imagine where games are going next right and so I'd love for you to talk a little bit about the types of experiences and characters people are building with convey that are exciting to you both for gaming and non-gaming purposes the non-gaming examples are primarily in learning and training Ed education related areas and then there is brand ambassadors which could be in the likeness or digital human of a celebrity or a non-descript model um who is AI powered and knows all about the brand can guide people how to use the product and what not these AI characters can be even location aware like you scan a particular QR code anywhere while walking in the street and they spun up right there and tell tell you what the directions are it's not a Far Future Okay where we start seeing these embodied AI characters literally everywhere and very much in public spaces all the way from take take your favorite mall to your favorite airport you will see large screens with these AI characters standing there welcoming you but now you can talk to them and ask them which way to go um you know like uh this is my Airport ticket which way do I go where where is the security check any kind of information dispense or transactional engagement these these characters would be uh would be perfect for real world use cases that we are already seeing now finally coming to gaming well games are going to be effectively The Matrix right so uh what do I mean by that it's going to be so real that you would prefer living there all right so which is which is a dystopian future that we have to beware it will be pretty darn engaging where instead of the machines taking over we would be willingly submitting ourselves to these game worlds which will be an extremely exciting with all of these Technologies converging all the way from your VR devices uh VR augmented reality and all of that to um extremely highspeed internet the cloud computing aspect of it where these extremely high definition world can be literally rendered on the cloud and stream to yourself along with these AI NPCs in these worlds in these metaverse like worlds it will become a much easier way for people to put themselves in let's call it the Matrix or the metaverse all right so uh where where you can be there live there almost uh to engage with your friends and you know learn certain things and and engage with these AI NPCs and that is uh a future not too far away that people will start doing that people already do that by the way in a major way in a lot of the social Virtual Worlds okay if you think especially the younger generation right like it's growing up almost in these worlds that is true but also there is uh a very small minority who have been doing it for a while there's a large audience a concurrent daily active user base for something like second life where we recently launched right and uh these people come back daily live their life talk to their friends for many years now and there are new newer platforms like VR chat uh you know like I know people that regularly go there they party on the weekends once the challenges of on boarding and the challenges of uh the friction is reduced to get in those worlds a lot of people will start going there all right let's let's take that VR chat example I think that one's fascinating it's like yeah people are buying expensive full body tracking setups with you know expensive headsets and computers to have this High Fidelity embodied experience in Virtual Worlds but right now on the other end are other humans that they meet right how do you feel about you know there being a time let's say in a couple years where you're talking to somebody in VR chat and you literally cannot tell if they're human or not does it matter at that point because like I don't know maybe these AI agents would be far more thoughtful and nice to you than perhaps real flawed humans so I'm curious how you think about that especially in the context of this Matrix analogy that you're making you literally reminded me of this movie called Transcendence and Johnny Depp was the lead character there literally that's the case there basically Johnny Depp dies and uh before dying he actually uploads himself like full full neuron scan of his brain and uploads himself into the internet and um and when he comes back after his actual biological body has passed back people would often ask him are you real are you aware and his answer would be if you cannot tell does it matter so so that is going to happen no doubt it's already there from a text standpoint text in text out it will be hard and um today totally right and the other other aspects would be the visuals of it the animations of it and things like that but also what might be a giveaway is if they are always of a particular certain certain personality type they need to have a wide array of personalities and even uh Ecentric AI characters you know that are kind of awkward and and some of them are mean and some of them are super nice and uh you know like uh you get the full High School experience yes exactly those are going to be necessary for for for people to engage it's it cannot just be like very assistant like so the more the variability the better you know like people would like and and engage with them and people will find their own type some people are drawn and attracted to toxic people right so so basically that they they will have all kinds of AI basically in in these worlds and you you'll choose your Pi what you want it reminds me of the conversation with the architect in The Matrix where the architect outlines that hey you know like we made a utopian simulation but nobody bought it it just felt too artificial and almost introducing the flaws of our Humanity kind of is what made it a sticky experience um which which I think is very fascinating right cuz these environments and experiences need to Mirror the full range of emotions um that we experience right right so that that's what we noticing you know like we did this demo room and people enjoy talking with the meanest character wow okay like they would want to walk away and that that mean character would say something provocative that would draw them back uh to talking to that character there's something that we have to be uh conscious about as well like that is literally the one of the reasons that Facebook and Tik Tok became so popular because the newsfeed was programmed for uh maximum emotional turbulence like the content that that was the most provocative draw people there we don't want to do with our stuff so so like the right balance between engagement and what's good for the people is something that we plan to do now there are many negatives to outweigh the positives here but Peru has spoken with me about being a lonely kid growing up and I had to ask him about the way these experiences can benefit people you alluded to you know growing up as a kid you felt isolated and different than other kids and I just imagine you know I I relate to that experience I was so like deeply into computer graphics and visual effects and all this other stuff that like nobody else cared about at the time and I obviously I found my Escape through the internet and you know OG IRC forums and PHP forums I'm kind of curious what does this do for people you know uh who may feel lonely today like what role can these AI characters play in sort of enriching their lives big time you know big time there's of course online communities where you could meet those like-minded people but like they may not be available uh at the same time but you have this AI character who could effectively have all the right interest like think of your best friend that you connected with the most right that understands you before you even say it right so these AI characters can potentially be that for them you know like uh it is risky but that is where we are evolving too right like undoubtedly and and do you find in that situation like let's take go back to a younger you or me do you imagine the experience being that these systems can sort of infer what you need or would would I be like curating the combination of my like three best friends plus a little sprinkle of like I don't know John G and you know some other VFX supervisors that I really like and let me throw in a Sprinkle of Alan Watts and a Sprinkle of like you know Einstein in there how would people Define sort of the their best friend if you will in this in this space yeah that's a very hard one to answer because we don't choose uh I mean we kind of choose our best friends uh eventually but uh we don't choose based on Vibe right based on Vibe right exactly so uh and and common interest and and things like that is how how it evolves but we don't exactly choose their things what they are interested other things that they are interested in and whatnot right so maybe it will be a multifarious world with lots of different AIS if these characters have to evolve if you keep changing them you will not see their character Evolution right so you basically go and socialize and you start with your one and maybe they will adapt to your interests and things like that and eventually they will have their own unique experiences too that will shape them effectively like um how it shapes you imagine friends that grew up together uh maybe this AI can also go out and have have its experience when when you are not there right so they have stories to tell what happened today it's kind of mind-blowing what you're saying almost makes me feel like we've been at this phase of technology and the internet where we can organize the world's information and make it universally accessible and useful to use the the Google mission statement but now we're heading into a world where we can make the world's people and personalities universally accessible and useful yes and a lot of the technology we are creating you know all the way from facial expressions and hand gestures and and emotional voices basically Bing the mind of these non-player characters are going to be the same um technology that may be used for a lot of these social robots there's obviously both utilitarian and delightful experiences uh your customers are building what are you looking forward to to we have set the vision we have created the tools uh and people are developing in terms of the immediate uh not not just immediate like medium-term 3 to 5 years plan is basically ensure that we have redefined gaming in a very positive way we have enabled these learning and training experiences at scale you know that that are changing lives of people in a very positive manner uh brand experiences product information uh real world embodied characters I love the Creator Centric approach I think it's so important that we don't forget that creators are going to author these experiences thank you so much for your time thank you so much yeah it was it was great chatting around the time pendu and I had this conversation he invited me to the convey headquarters where I got to see their NPC Innovations first hand and let me tell you it was pretty wild when I walked in they had this massive monitor with an AI anime character on it and the people it convey told me to just have a conversation with it they told me I could talk to it in Hindi which I was psyched about then I tried to press a little deeper I think there's this really human urge to try and push the boundaries of an AI system to prove that it's intelligent and human enough to exist beyond the constraints of corporate language most of my efforts were in vain though because as soon as the AI came back with these canr responses it kind of ruined some of the effect even though it was giving me unscripted responses it still had parameters the AI allowed me to go off script and even though the model I was talking to recognize what I was saying it was still responding based on its parameters the guardrails set by the company while an AI like this might be great at helping you fight a lethal force of aliens it's hard to know if it will ever reach the Messier more human parts of how we relate to one another [Music] the tedi show is a part of the Ted audio Collective and is produced by Ted with Cosmic standard our producers are Ella feder and Sarah McCrae our editors are B bansang and Alejandra Salazar our showrunner is Ivana Tucker and our associate producer is Ben Monto our engineer is Asia polar Simpson our technical director is Jacob Winn and our EX producer is Eliza Smith our fact Checker is Julia Dickerson and I'm your host baval sadu see y'all in the next one [Music]