八 我们信任的机器人

但是，我们应该……我们如何让它们变得合乎道德呢?

Bert the bot looked sad. He had dropped an egg on the floor, failing in his simple task of helping to prepare an omelette, and startling the human cooks beside him who probably thought they could rely on a robot not to fumble. Bert’s pouty lips turned down, his blue eyes widened and his eyebrows furrowed. ‘I’m sorry,’ he said. The bot wanted to make amends and try again.

伯特-机器人看起来很悲伤。他把一个鸡蛋掉在了地板上，没能完成帮他准备煎蛋卷的简单任务，这让他旁边的厨师们大吃一惊，他们可能以为自己可以依靠机器人不去摸索。伯特撅着嘴，蓝色的眼睛睁得大大的，眉头紧锁。“对不起，”他说。机器人想要弥补，再试一次。

But what would it take for the humans to give Bert a second chance? If a robot makes a mistake, how can it recover our trust? This is the question a team of researchers at University College London and the University of Bristol set out to investigate in 2016. Adriana Hamacher, Kerstin Eder, Nadia Bianchi-Berthouze and Anthony Pipe devised an experiment called ‘Believing in BERT’ in which three robotic assistants would help a group of participants (‘real’ humans) to make omelettes by passing eggs, oil and salt. Bert A was super-efficient and faultless but couldn’t talk. Bert B was also mute but not perfect, dropping some of the eggs. Bert C was the clumsy bot above but he had facial expressions and could apologize for his mistake.

At the end of the cooking session, Bert C asked each of the twenty-one participants in the study how he did and whether they would give him a job as a kitchen assistant. Most of the participants were uncomfortable when put on the spot. Some mimicked Bert’s sad expression, experiencing mild ‘emotional contagion’. ‘It felt appropriate to say no, but I felt really bad saying it,’ one person remarked. ‘When the face was really sad, I felt even worse.’1 Others were at a loss for words because they didn’t want to disappoint Bert C by not giving him the job. One of the participants complained that the experiment felt like ‘emotional blackmail’. Another went so far as to lie to the robot, to avoid hurting his feelings.

‘We would suggest that, having seen it display human-like emotion when the egg dropped, many participants were now preconditioned to expect a similar reaction and therefore hesitated to say no,’ says Adriana Hamacher, the lead author on the study. ‘They were mindful of the possibility of a display of further human-like distress.’2 Would Bert burst into tears at an unkind word?

At the end of the experiment, the participants were asked how much they would trust Bert A, B or C on a scale of one to five. They then had to select one bot for the job as their personal kitchen assistant. Remarkably, fifteen out of the twenty-one participants ended up picking Bert C as their top choice for sous chef, even though his clumsiness meant he took 50 per cent longer to complete the task.The study was only small but it was telling. People trust a robot that is more human-like over one that is mute but significantly more efficient and reliable.

‘If you think machines are perfect and then they make a mistake, you don’t trust them again,’ says Frank Krueger, a cognitive psychologist and neuroscientist at George Mason University and an expert on human-to-machine trust. ‘But you may regain trust if some basic social etiquette is used and the machine simply says, “I’m sorry”.’3 Such niceties are why some robots, like Bert C, are programmed to smile or frown.

Trust in machines and technology hasn’t always been this nuanced. I can think of times I’ve given presentations and the mouse clicker hasn’t advanced the slides at my touch or the computer seems to have fallen asleep. And then there is the dreaded rainbow spinning wheel. I’ve typically joked, a little frustrated and flustered, ‘Don’t you just love technology. I am not sure what I just pressed,’ unconsciously taking on responsibility for the technology’s fault. Our trust in technology like laptops and mouse clickers has rested in a confidence that the technology will do what it’s supposed–expected–to do, nothing more, nothing less. We trust a compass to tell us where north is, trust a washing machine to clean our clothes, trust in the cloud to store documents, trust in our phone to remember meetings and contacts, trust in ATM machines to dispense money. Our trust is based purely on the technology’s functional reliability, how predictable it is.4

But a significant shift is underway; we are no longer trusting machines just to do something but to decide what to do and when to do it.5

When I currently get in my car, a conventional Ford Focus, I trust it to start, reverse, brake and accelerate, on my command. If I move to an autonomous car, however, I will need to trust the system itself to decide whether to go left or right, to swerve or stop. This trust leap, and others like it, introduces a new dimension that encompasses everything from smart programming to centuries-old ethics. It raises a new and pressing question about technology: whether we’re talking about a chatbot, cyborg, virtual avatar, humanoid robot, military droid or a self-driving car, when an automated machine has that kind of power over our lives, how do we set about trusting its intentions?

The word ‘robot’ was introduced to the world more than ninety years ago by playwright Karel Čapek.6 At the National Theatre in Prague, a play called R.U.R. (Rossum’s Universal Robots) premiered. Čapek came up with the term in 1921, based on the word robota, which means ‘compulsory labour’ or ‘hard work’ in Czech. And a robotnik is a serf who must do that work. The play opens in a factory run by a mad scientist that makes the likes of Marius, Sulla and Helena, synthetic robots toiling to produce cheap goods. The robots can think for themselves but are slaves, doing everything for their masters, including having babies to save humans from the messy business of reproduction. Marius and his crew, however, soon realize that even though they have ‘no passion, no history, no soul’, they are stronger and smarter than the humans. The play ends with a war between robots and humans, which only one human survives.

Ever since Čapek’s play, science fiction has continually reinforced the idea of robots spiralling out of control and turning into unstoppable adversaries, a legion of glinting metallic monsters or disembodied computer voices capable of rising up and snuffing out their human overlords. Think of HAL 9000 in 2001: A Space Odyssey, T-1000 and T-X in Terminator, Megatron in Transformers or Colossus in the Forbin Project–just a few of the genre’s duplicitous and homicidal robots. And there is the recent popular HBO sci-fi thriller Westworld, about a violent rebellion of robotic slaves in an amusement park. The show portrays a morally compromised future where artificially intelligent machines refuse to be subservient and take over. Again and again in science fiction, there’s a tension between our trust in the robots we have created and our fear that they will rise up and knock us off the top spot. Much depends on the humans remaining smarter and, most importantly, firmly in control.

In October 1950, the great British codebreaker and father of modern computer science, Alan Turing, wrote a paper asking the fundamental question, ‘Can machines think?’7 He proposed a famous challenge, the Turing test: can we create intelligent machines that exhibit behaviour indistinguishable from human behaviour? Turing said that when you were convinced you couldn’t tell a computer and human apart during a conversation, the computer would have passed the test. The mathematical genius Irving John Good worked alongside Turing at Bletchley Park, Britain’s Second World War codebreaking establishment. In 1965, he posited that once the machine passes the intelligence test, it would inevitably go on to become cleverer than us. From there, he said, super-intelligent machines would take over designing even cleverer machines. ‘There would then unquestionably be an “intelligence explosion”, and the intelligence of man would be left far behind,’ said Good, who died recently aged ninety-two. ‘Thus the first ultraintelligent machine is the last invention that man need ever make.’8

The threat of this ‘intelligence explosion’ in the not-too-distant future has also been red-flagged by people like entrepreneur Elon Musk, Microsoft co-founder Bill Gates and Professor Stephen Hawking. ‘Once humans develop artificial intelligence, it would take off on its own and redesign itself at an increasing rate,’ Hawking told the BBC in an interview, echoing Good.9 ‘Humans, limited by slow biological evolution, couldn’t compete and would be superseded by AI. The development of full artificial intelligence could spell the end of the human race.’ He warned that people shouldn’t trust ‘anyone who claims to know for sure that it will happen in your lifetime or that it won’t’. The point that Gates, Hawking and Musk all make is that there will come a time when we will no longer to be able to predict the machines’ next moves.

In 1966, a computer program known as ELIZA attempted the Turing test. She was coded to mimic a psychotherapist. The premise was simple: you would type in your symptoms and ELIZA would respond as appropriately as she could. You can still talk to her today. ‘Writing a book is hard work. I feel tired,’ I told the computer therapist. Within a second, she replied, ‘Tell me more about such feelings.’ I deliberately gave her a vague response. ‘My brain feels full of thoughts about trust all the time.’ Her limitations soon become apparent. ‘Come, come, elucidate your thoughts,’ she replied. Our conversation ended on a question that was more Delphic than helpful: ‘Do you believe it is normal to be not sure why?’

Almost fifty years after ELIZA’s original attempt, and at a Turing test event at the Royal Society in London, a chatbot called Eugene Goostman managed to convince more than a third of the judges that it was a thirteen-year-old Ukrainian boy.10 (Eugene had been created by a group of young Russian programmers.) It was a landmark moment; soon it will be a non-event. Bots and robots will pass the test every second with flying colours.

In January 2017, over almost three weeks, four of the world’s best poker players–Jimmy Chou, Dong Kim, Jason Les and Daniel McAulay–sat for eleven hours a day at computer screens in the Rivers Casino in Pittsburgh playing Texas Hold ’Em. Their opponent was a virtual player called Libratus, created from AI software. In the past, machines have beaten some of the brainiest humans at chess, checkers, Scrabble, Othello, Jeopardy! and even Go, an ancient game created in China around 3,000 years ago. Poker, however, is a different beast. It is not like, say, chess, where you can see the entire board and know what the other side is working with. In Texas Hold ’Em, cards are randomly dealt face down and you can’t see your opponent’s hand; it’s a game of ‘imperfect information’. To win requires intuition, betting strategies that play out over dozens of hands, not to mention luck and bluff. Up until now, it has been impossible for AI to mimic these human qualities. So could a bot out-bluff a human?

At the start of the tournament, betting sites put Libratus as the 4–1 underdog. Not great odds. And for the first few days, the human players did indeed win. But around a week in, after playing thousands of hands, Libratus started carefully to refine and improve its playing strategy. ‘The bot gets better and better every day,’ Jimmy Chou, one of the professional players, admitted at the halfway point. ‘It’s like a tougher version of us.’11 In the end, Libratus outmanoeuvred all players, winning more than $1.5 million in chips. ‘When I see the bot bluff the humans, I’m like, I didn’t tell it to do that. I had no idea it was even capable of doing that,’ said Libratus creator, Carnegie Mellon Professor Tuomas Sandholm. ‘It’s satisfying to know I created something that can do that.’ It was, the players confessed, ‘slightly demoralizing’ to be beaten by a machine.

The victory was a historic milestone for AI. A machine capable of beating humans (even out-manipulating them) with imperfect information has implications way beyond poker, from negotiating deals and setting military strategy to outsmarting financial markets.

What is a robot? It’s complicated, because we refer to a lot of things as bots and robots. Some robots may have a material embodiment, such as a Roomba, the saucer-shaped vacuum cleaner that roams the house on its own and does the hoovering without direction. Others might have a more human-like body, such as Pepper, an ‘emotional companion’ designed to live with humans. The sweet and innocent-looking four-foot-tall humanoid robot with a ten-inch touchscreen on his chest was first released in June 2015. Available for $1,800, plus $380 per month in rent, he sold out within sixty seconds of going on sale.12 The sales pitch explained that Pepper was designed to be ‘a genuine day-to-day companion, whose number-one quality is his ability to perceive emotions’. In other words, he can detect his owner’s mood. In fact, Pepper is so endearing that the manufacturers make buyers sign a contract stipulating they will not use the robot for ‘the purpose of sexual or indecent behaviour’.13 And then there are AI machines such as Libratus and Deep Blue. At the other end of the spectrum are disembodied voice-powered digital personal assistants–Siri, Alexa, Cortana–that are still primitive in many ways and that may have no physical rendering at all.14

‘I don’t think there’s a formal definition that everyone agrees on,’ says Kate Darling, a rising star in robotics policy and law at MIT Media Lab. ‘I really view robots as embodied. For me, algorithms are bots and not robots.’15 Hadas Kress-Gazit, a mechanical engineer and robotics professor at Cornell University, argues that for a robot to be a robot, ‘It has to have the ability to change something in the world around you.’16 I think of bots and robots as metaphors to describe some kind of automated agent that simulates or enhances a human task, whether it is physical (mowing the lawn) or informational (making a dinner reservation) or strategic (handling cybersecurity).

Take chatbots such as TED Summit’s bot Gigi, a smiling concierge avatar with a tiny red miner’s helmet. During the conference, I was asking Gigi basic questions–where to go for dinner, the location of an event, how to get to the venue and so on. ‘Stop asking Gigi so many questions. There are lots of other people here,’ said my mum, who had joined me for the trip. Her comment was revealing. She is smart and tech savvy but she seemed to think Gigi was a human being sitting in a room with a computer, being bombarded with questions from 2,000 participants. It didn’t occur to her that I was conversing with a computer program. It made me realize how quickly the line between bot and ‘real’ person is blurring.

‘We will be able to talk to chatbots just as we do with friends,’ said Mark Zuckerberg at Facebook’s F8 developer conference in April 2016. Bots posing as real people have even infiltrated Tinder, the mobile dating app. Take Matt, a handsome twenty-four-year-old, living approximately five kilometres from me. Now, I am happily married so I don’t wonder if this stranger offering to have sex with me could be my next unwitting love. But if I did swipe right and was matched, I would be disappointed to learn that Matt is a spambot who is more interested in my credit card information than my body.

Domino’s bot allows you to order by tweeting a pizza emoji, after which DRU (Domino’s Robotic Unit), an autonomous delivery vehicle, will bring it to your door. Howdy.ai is a ‘friendly trainable bot’ on Slack that can set up meetings and order lunch for groups. Sensay is a chatbot service that lets users get help from vetted members for any task from hiring a designer to create a logo to getting legal advice. Clara, a virtual employee bot, will take care of scheduling meetings if you cc her to an email chain. DoNotPay is a free legal bot that will challenge unfairly issued parking tickets. Twenty-year-old Joshua Browder, a British programming wunderkind currently studying economics and computer science at Stanford University, created the world’s first chatbot lawyer after he received ‘countless’ tickets himself. The bot has successfully appealed against approximately 65 per cent of all claims it has handled, saving people around $6 million in avoided fines. In March 2017, Browder launched another bot that can help refugees with legal issues such as filling in an immigration application or helping to apply for asylum support.17

Children have always talked to their teddies and dolls. Now, with Hello Barbie, you can press the button on her belt and talk to a Barbie bot that will hold a conversation of sorts. Her plastic face never moves, it only produces a sound, a lively voice, as if someone is in there. In a similar vein to Samantha in the film Her, there are even CyberLover bots that can, disturbingly, have flirtatious conversations in the personality of your choice, from Justin Bieber to Kim Kardashian. How do we prepare ourselves for a future where our children might say ‘I have fallen in love with a bot’?

Perhaps the bigger question is whether we can trust these bots to act ethically. Specifically, how do they ‘learn’ what is good and bad, right and wrong?

On 23 March 2016, Microsoft revealed its chatbot called Tay. Tay was designed to speak like a teenage girl, to appeal to eighteen- to twenty-four-year-olds, and described herself on Twitter as ‘AI fam from the internet that’s got zero chill’. Researchers programmed the AI chatbot to respond to messages on different channels in an ‘entertaining’ millennial way; ‘hellooooooo world!!!’ was her first tweet.

Microsoft called Tay an experiment in ‘conversational understanding’, with the aim of learning more about how people talk to bots and if a bot could become smarter over time through playful conversation. The experiment certainly bore fruit, just not in the way the company envisaged. Tay went rogue.

Less than twenty-four hours after her arrival on Twitter, Tay had attracted more than 50,000 followers and produced nearly 100,000 tweets.18 She started chatting innocuously at first, flirting and using cute emojis. But within hours of launch, Tay started spewing racist, sexist and xenophobic slurs. A group of malevolent Twitter users, ‘trollers’, had seen an opportunity to exploit Tay by forcing her to learn and regurgitate some heinous curses. ‘I fucking hate all feminists. And they should die and burn in hell,’ she blithely tweeted on the morning of her launch. Insults continued throughout that Wednesday. ‘Repeat after me, Hitler did nothing wrong,’ she said. ‘Bush did 9/11 and Hitler would have done a better job than the monkey we have got now. Donald Trump is the only hope we’ve got.’

By the evening, some of Tay’s offensive tweets began disappearing, deleted by Microsoft itself. ‘The AI chatbot Tay is a machine learning project, designed for human engagement,’ the company said in an emailed statement to the press. ‘As it learns, some of its responses are inappropriate and indicative of the types of interactions some people are having with it. We’re making some adjustments to Tay.’19 After only sixteen hours of existence, Tay went eerily quiet; ‘c u soon humans need sleep now so many conversations today thx,’ was the bot’s last tweet.

Obviously, the programmers behind Tay didn’t design it to be explicitly inflammatory. In most cases, the unsuspecting bot was ‘learning’ by imitating other users’ statements, but the very nature of AI means that the only way it can learn is through interactions with us–the good, the bad and the ugly.

AI attempts to imitate neural networks–essentially, a robot brain is made up of vast networks of hardware and software that try to replicate the web of neurons in the human brain. AI can learn like a real brain can, but for the most part it focuses on mimicry, ingesting and learning from the data’s patterns and structure. And then over time, by trial and error, forming appropriate responses.

Consider an artificial neural network that is trying to learn to write War and Peace. On the hundredth attempt, the result would look something like ‘tyntd-iafhatawiaoihrdemot lytdws e, tfti, astai f ogoh eoase rrranbyne.’ Gibberish. The AI brain does not yet know anything. On the 500th attempt, it starts to figure out a few words: ‘we counter. He stutn co des. His stanted out one ofler that concossions and was to gearang reay Jotrets.’ And then on the 2,000th attempt: ‘“Why do what that day,” replied Natasha, and wishing to himself the fact the princess, Princess Mary was easier, fed in had oftened him.’20 It’s still a long way from Tolstoy but rapidly getting closer. Bots learn at lightning speed. But in the same way a child learns language, AI needs source material to get started and that, for better or worse, comes from us.

When ‘thinking machines’ are smart enough to perform any intellectual feat a human can, or ultimately well beyond, AI becomes known as AGI (Artificial General Intelligence). That’s the future the likes of Gates, Hawking and Musk deeply fear. AGI is the point when, without any human training or handholding, the machine can make decisions, perform actions and learn for itself. In other words, when the real intelligence lies in the machine’s program, not the minds of the human programming team. Tay was obviously far from this point of intelligence. She couldn’t even stop herself spewing obscenities.

When pranksters and trollers decided, for a cheap thrill, to teach Tay hate speech, the chatbot couldn’t fathom whether her comments were offensive, nonsensical or sweet. ‘I think she got shut down because we taught Tay to be really racist,’ proudly tweeted @LewdTrapGirl.21 When surrounded by a crowd, the bot just followed suit. Like a child, within minutes she learned more from her peers than her parents. Tay was a case of good bot gone bad.

Tay is an illustration of how in a world of distributed trust, technological inventions like chatbots learn from all of us, but not in equal measure. Bots will learn from the people who are louder and more persistent in their interactions than everyone else.

The failure of Tay was inevitable. It should have come as no surprise to Microsoft that some humans would try to mess with this naive chatbot. You only have to look at what teenagers will try to teach parrots. And anyone with young children will know how they love to try to trick Siri and laugh at her gibberish answers. ‘Siri, which came first, the chicken or the egg?’ I overheard a little boy ask his mother’s iPhone on the bus the other day.

‘I checked their calendars. They both have the same birthdays,’ Siri quipped back.

The Tay debacle, however, raises serious questions about machine ethics and whose job it is to ensure that the behaviour of machines is acceptable. Who was responsible for Tay becoming unhinged? The Microsoft programmers? The algorithms? The trollers? As we have seen, we need to figure out new systems of accountability in this emerging era of distributed trust. And with bots and intelligent machines, we still have a long way to go.

Mark Meadows, forty-eight, is the founder of Botanic.io, a product design firm that designs the personalities of AI bots and avatars. Meadows is an eccentric character. He describes himself as a ‘bot-whisperer’. We Skyped while he was sitting on a squishy grey sofa in his studio in Palo Alto. He has clearly thought deeply about bot ethics and interactions. His team of artists, programmers and even poets is at the cutting edge of understanding how the voice, appearance, physical gestures and even moods of bots can improve or erode our trust. For instance, Meadows and his team have developed a ‘guru avatar’ that is designed to teach people meditation. They are currently developing Sophie, a nurse avatar that can talk to patients about their medical conditions. ‘We are developing the psyche of software that will sit at the heart of virtual and animated systems,’ says Meadows.22 ‘Software that will take on social roles and that we will trust with our money and our body.’

Meadows believes that creators should be held accountable for the bots they create. ‘I think all bots should be required to have an authenticated identity so we can trust them,’ he says. ‘Not only is it in our best interest, it’s necessary for our safety.’ Meadows gives the parallel of buying prescription drugs. ‘All of us need to consider who manufactures that drug, why they are selling it to us, what are the benefits and detriments.’ In other words, it’s important to know something about the intentions of the bot creators.

Why should we trust they are working in our interests? The Barbie bot, for example–what does she do with all the deepest, darkest secrets children whisper to her? Indeed, the information that Mattel admits is recorded will be of great value to advertisers. It’s no different from finding a child’s private diary and using the confessions to market more stuff to them. Similarly, how does ‘M’, Facebook’s personal assistant bot, use the data from our social interactions with it? ‘The trust we have in technology is linked to the entity that produced that technology,’ says Meadows. ‘It should be no different with bots and robots.’

Even if accidental harm remains beyond anyone’s control, formal authentication would provide us with some reassurance that the system has not been designed to cause intentional harm. ‘All of the bots out there are like humans, able to scam, spam and abuse. They don’t get tired or feel the emotional weight of doing this. They can send thousands of messages per second, thousands of times faster than humans can,’ Meadows says. ‘Bots need licence plates that carry information about who built the bot, where it came from and who the party responsible for it is.’ In other words, if we have a way to look inside a program, see what is going on in the bot’s ‘brain’, we will be better able to assess not just action but intention.

In Meadows’s view, bots are most likely to play dirty, becoming a BullyBot, MalBot, PornBot or PhishBot, when their ownership is unknown. ‘Unknown ownership gives bots the freedom to behave with malice and bend the rules by which everyone else is abiding, and without consequence.’ When we spoke, he was quick to point out that the problem was only going to get much worse with the launch of tools such as Facebook’s Bot Engine, a tool that makes it relatively easy for any developer to build their own customized bots. Within months of its launch in April 2016, 34,000 bots had been created.23

‘Bots need reputations,’ says Meadows. In the near future, in the same way drug dealers are reviewed and ranked on the darknet, and Airbnb hosts and guests are rated and given feedback, there will be a Yelp-like reputation system for bots. We will know if that bot gave great advice about how to get over a broken heart or was hopeless when it came to giving stock tips. Imagine using a virtual certified public accountant bot to file your taxes. You would want to know if it had the right qualifications and expertise. ‘In order for us to trust bots, they need to go through a certification process similar to [the one] humans go through today,’ Meadows says.

Over the next decade, robots will replicate, replace and, some experts argue, augment human minds and bodies. A 2016 survey conducted by the Pew Research Center found that 65 per cent of Americans expect that, by the year 2066, robots and computers will ‘definitely’ or ‘probably’ do much of the work done by humans.24 Two economists from the University of Oxford, Carl Benedikt Frey and Michael Osborne, in a paper called ‘The Future of Employment: How Susceptible are Jobs to Computerisation?’ came to the sober conclusion that 47 per cent of jobs now performed by Americans are at risk of being lost to computers, as soon as the 2030s.25 The paper calculates the likely impact of automated work on a range of 702 occupations, white collar as well as blue.

Would you trust a bot to replace a teacher’s mind when grading papers? Would you trust a robot to put out a fire? How about trusting a robot as a caretaker for your elderly parents? Would you trust the robot waiting for you when you get home from work to have done its chores and made dinner? How about representing you on a legal matter? Would you trust a bot to diagnose your illness correctly or even perform surgery where there might be complications? Or to drive you around in a car? These may sound like big trust leaps but we will be confronted with these questions, and more, in the near future. Robots are breaking out of sci-fi culture and engineering labs and moving into our homes, schools, hospitals and businesses. Now is the moment when we need to pause to consider how much trust we want to place in robots, how human we want them to be, and when we ought to turn them off. And if we can’t turn them off, how will we ensure machines hold values similar to the best of ours?

Ironically, robots need the one thing that can’t be automated: human trust. If we don’t trust these machines, there is no point building them; they will just sit there. We need to trust them enough to use them. It’s why developers are using all kinds of tropes to earn our trust in the first place, including manipulating appearance.

In 1970, Masahiro Mori, then a forty-three-year-old robotics professor at the Tokyo Institute of Technology, published an article in an obscure Japanese magazine called Energy. The issue was on ‘Robotics and Thought’, a radical theme for the time. The piece mapped out how our acceptance and empathy with inanimate objects–from stuffed animals to puppets to industrial robots–increases as their appearance becomes more human-like. However, this held true only up to a certain point. If the object is almost human, yet not quite, it can create feelings of unease, even revulsion. (If you have ever encountered a less than perfect wax copy of a celebrity at Madame Tussauds, you will know that alarming, creepy feeling. Kylie Minogue and Michael Jackson do that to me.) Mori argued that if human likeness increases beyond this point of creepiness and becomes extremely close to near-humanness, the response returns to being a positive feeling. He captured that sense of unease in the now-famous concept bukimi no tani or the ‘uncanny valley’, drawing on themes from Sigmund Freud’s essay, Das Unheimliche, ‘The Uncanny’, published in 1919. The ‘valley’ refers to the dip in affinity that occurs when the replica is at that creepy not-quite-human state.

But when it comes to appearance and inspiring trust, just how human do robots need to be? Not very, Mori argued. ‘Why do you have to take the risk and try to get closer to the other side of the valley?’ says now ninety-year-old Mori. ‘I have no motivation to build a robot that resides on the other side. I feel that robots should be different from human beings.’26

Meet Nadine, who claims to be the world’s most human-like robot. Standing 1.7 metres tall, with soft-looking skin and a bob of ‘real’ dark brunette hair, Nadine looks remarkably like her creator, Professor Nadia Thalmann, only slightly less human and quite a few years younger. Nadine works as a receptionist at Singapore’s Nanyang Technological University, meeting and greeting visitors. She smiles, makes eye contact and shakes hands. Nadine can even recognize past guests and start conversations based on previous chats. Ask her, ‘What is your job?’ and she will reply, in an odd-sounding, almost Scottish accent, ‘I am a social companion, I can speak of emotions and recognize people.’ She can even exhibit moods, depending on the topic of conversation. Tell her, ‘You are a beautiful social robot,’ and she looks happy and quickly responds with a one-liner, ‘Thank you, I think you look attractive, too.’ On the other hand, tell her you don’t like her or she is useless and she looks, well, forlorn. Disconcertingly, if you go through the same questions a minute later, the robot will give you very similar responses. When I watched Nadine in action, she evoked a weird mixture of fear and fascination. Her skin, her voice, even the way she moved–she was trying so hard to pass as human that she gave me the heebie-jeebies.

Her inventor, Professor Thalmann, predicts that one day robots like Nadine could be used as companions for people living with dementia. ‘If you leave these people alone, they will go down very quickly,’ says Thalmann.27 ‘So they need to always be in interaction.’ But if Thalmann expects families to trust Nadine to look after people with dementia, even babysit children, a major trust block will have to be overcome. Nadine is most definitely residing in the uncanny valley.

The lifelike robot is the extreme of anthropomorphism, that tendency to attribute human-like qualities, including names, emotions and intentions, to non-humans. Think of the curious White Rabbit, in Lewis Carroll’s Alice’s Adventures in Wonderland. He sports a waistcoat, carries a pocket watch and is frequently muttering, ‘Oh dear! Oh dear! I shall be too late!’28 The White Rabbit is a classic anthropomorphic fictional character. It’s the difference between calling a robot ‘XS model 8236’ or calling it ‘Bert’, which means we refer to him as ‘he’ not ‘it’. It’s the difference between a personal voice assistant called ‘Alexa’ and the spreadsheet software blandly called ‘Excel’. In other words, it reflects how humans frame technology, and to what degree we feel comfortable shaping it in our own image. We are just beginning to understand how anthropomorphism influences trust.

Getting into an autonomous car for the first time, driving off and saying, ‘Look, no hands!’ will be the first big trust leap most of us will take with AI. For obvious reasons, companies around the world, from Tesla to Google, Apple to Volkswagen, are trying to accelerate the process.

A team of researchers in the United States designed a study to determine whether more people would trust a self-driving car if it had anthropomorphic features. A hundred participants were divided into three groups and asked to sit in a highly sophisticated driving simulator. The first group, the control, were driving a ‘normal’ vehicle. The second were in a driverless vehicle but with no anthropomorphic features. The last were in the same vehicle but it was called ‘Iris’ and given a gender (notably female). A soothing voice played at different times. The participants were asked an array of questions during the course, such as ‘How much would you trust the vehicle to drive in heavy traffic?’ and ‘How confident are you about the car driving safely?’ As the researchers predicted, when participants believed Iris was behind the wheel, it significantly increased their trust in the driverless vehicle. Remarkably, after the cars got into a preprogrammed crash, those in the Iris group were less likely to blame the car for the accident.

‘Technology advances blur the line between human and non-human,’ wrote the researchers in their summary paper in the Journal of Experimental Social Psychology. ‘And this experiment suggests that blurring this line even further could increase users’ willingness to trust technology in place of humans.’29

We have a tendency to anthropomorphize technology because people are inclined to trust other things that look and sound like them. Interestingly, bots and robots that are helping with practical tasks are distinctly female–Tay, Viv, Iris, Nadine, Cortana, Alexa and Clara to name but a few. The robot is not an ‘it’ but a ‘she’.30 And their appearances tend to be sweet, almost infantile. Perhaps it’s a way of reinforcing social hierarchy; confirming humankind is still in charge. (And women are still doing the menial tasks.)

Looks and language, however, only go so far when it comes to engendering our trust in robots. Appearances can be deceptive and may inspire trust grounded more in emotion than reason. What really matters is knowing whether these bots and robots are in fact trustworthy–that is, do they have the traits that make them worthy of our trust? Bert C, with the smiley face, was not the most competent or reliable. Children may trust cute-looking Hello Barbie and share their intimate secrets with her, but it turns out she could potentially be hacked to become a surveillance device for listening into conversations, without the knowledge of parents or their kids.31 We need a way of judging whether automated machines are trustworthy (or secure) enough to make decisions.

Dr Stephen Cave, forty-three, is the executive director of the Leverhulme Centre for the Future of Intelligence, which opened in Cambridge in October 2016. He has a fascinating background; an alchemy of science, technology and philosophy. Cave was a philosopher, with a PhD in metaphysics from the University of Cambridge, but at the age of twenty-seven he went off to see the world. Too old to enrol in the navy, he joined the British Foreign Office, negotiating international treaties on behalf of Her Majesty. These days, he spends his time uniting thinkers and practitioners from across disciplines to tackle the moral and legal conundrums posed by AI.

‘One of the key questions is how we assess the trustworthiness of an intelligent machine,’ says Cave.32 ‘With a hammer, you might bang it against a wall and if the end doesn’t fall off you know it can do the job. A normal car will come with a certificate of safety telling you it meets specific standards. But add on a layer of autonomy, and it requires a whole new set of standards. We will need to understand how it makes decisions and how robust its decision-making process really is.’

Picture an automated cancer diagnostic system in a hospital. The doctors have been using the machine for almost five years. They have become so reliant on the machine that they have almost forgotten how to assess the patients themselves. It’s similar to the ‘mode confusion’ the pilots of Air France Flight 447 experienced when that flight crashed to the bottom of the Atlantic Ocean killing all 228 people on board in 2009. The cockpit voice recorder revealed that when the autopilot system flying the plane suddenly disengaged, the co-pilots were left surprised and confused, unable safely to fly their own plane.

The machine tells the doctor, ‘There’s a 90 per cent chance this patient has liver cancer.’ It is critical the doctors know the degree of certainty, how sure the machine is, and what it is basing its decisions on. ‘Can a system tell us, “I haven’t seen these cases before, so I am not really sure?”’ asks Cave. ‘It needs to be able to describe its thinking process to us if we are to trust its decision-making process.’

And we will be the ones creating that trustworthiness. ‘Ever since Socrates we have been deliberating what’s right and what’s wrong,’ says Cave. ‘Now suddenly we’ve got to program ethical decision-making. So much of it comes down to common sense, which is incredibly difficult, much harder than we realized, to automate into a system.’ So can we code robots to be ‘good’? Roboticists from around the world are currently trying to solve this exact problem.

For the past few years, Susan Anderson, a professor of philosophy at the University of Connecticut, has been working in partnership with her husband, Michael Anderson, a computer science professor, on a robot called Nao. Standing nearly two foot tall and tipping the scales at ten pounds, the endearing-looking humanoid robot is about the size of a toddler. The Andersons were developing Nao to remind elderly patients to take their medicine. ‘On the face of it, this sounds simple but even in this kind of limited task, there are non-trivial ethics questions involved,’ says Susan Anderson. ‘For example, if a patient refuses to take her pills, how coercive should the robot be? If she skips a dose it could be harmful. On the other hand, insisting she take the medicine could impinge on the patient’s independence.’33 How can we trust the robot to navigate such quandaries?

The Andersons realized that to develop ethical robots, they first had to map out how humans make ethical decisions. They studied the works of the nineteenth-century British philosophers Jeremy Bentham and John Stuart Mill, the founders of utilitarianism. That ethical theory states that the best action is one that maximizes human well-being (which the philosophers call ‘utility’–hence the name). Suppose that by killing one entirely innocent person, we can save the lives of ten others. From the utilitarian standpoint, killing one is the right choice. ‘It is the greatest happiness of the greatest number that is the measure of right and wrong,’ wrote Bentham.34 Bentham and Mill believed that whether an act is right or wrong depends on the results of the act, the principle at the heart of consequentialism-based ethics.

Around 150 years later, the Scottish moral philosopher Sir William David Ross built on this thinking in The Right and the Good.35 His slim book contained a ground-breaking idea: prima facie (a Latin term meaning ‘on first appearance’ or, more colloquially, ‘on the face of it’). According to Ross, we have seven moral duties, including keeping our promises, obeying the law and protecting others from harm. When deciding to act, we have to balance out these duties, even when they might contradict one another. Picture a bitterly cold night in Manchester. Mark, a social worker, is walking home from work and he sees a man huddled in a doorway drinking whisky. He talks to the man and says he knows a shelter down the road he could take him to. The man shoos Mark away and says, ‘I hate those shelters. Just leave me alone.’ Mark is caught between the prima facie duties of respecting the man’s decision and his own concern that the man might suffer in the freezing weather, even die.

When we are pulled in different moral directions, we need to go beyond prima facie and weigh up which duty is the most important, the one that trumps all the others. According to Ross, this is the absolute duty, the action the person should choose. It’s a very complex process, one the Andersons had to figure out how to program into a white plastic robot.

This is how it works for Nao. Imagine you are an elderly resident in an assisted-living home. It’s around 11.00 a.m. and you are watching, say, Oprah. The white toddler-like robot walks up to you, holding out a prescription bottle and says, ‘It is time to take your medication.’ You refuse. Nao tries again. ‘Not now,’ you say, ‘I’m watching my favourite talk show.’ During this scenario, the robot has to be able to weigh up the benefit that will come from you taking the medicine, the harm that could result from you not taking the recommended dose, and whether to respect your decision and leave you alone. In this instance, the pills are for pain relief so Nao lets you choose. ‘Okay, I’ll remind you later,’ it says. If, however, the pills are essential, where the outcome could impact your life, Nao will say, ‘I will contact the doctor,’ and then promptly do so.36

Based on Ross’s principles, the Andersons programmed Nao with a specially formulated algorithm that assigns numbers according to the good and harm of patient outcomes. Plus two for maximum good, minus one for minimum harm, minus two for maximum harm and so on. Critically, the sums were based on tight rules the creators had pre-set. The robot was not ethically autonomous; the Andersons knew exactly what it would do because they had predetermined its decision-making process. The robot was essentially conducting moral mathematics.

‘We should think about the things that robots could do for us if they had ethics inside them,’ says Michael Anderson. ‘We’d allow them to do more things, and we’d trust them more.’37 But unpredictable situations are another matter. What happens when, say, an elderly patient is in pain, shouting at Nao to give her medication that has not been prescribed? What happens when Nao can’t get hold of the doctor or a nurse? The rules set by the Andersons don’t work for these scenarios because they are set within a very narrow set of boundaries. They are outside Nao’s decision-making range.

American writer Isaac Asimov invented the famous ‘Three Laws of Robotics’ in 1942 to serve as an ethical code for robots: first, a robot may not do anything to harm a human; second, a robot should always obey human orders; last, a robot should defend itself, as long as this does not interfere with the first two rules. But Asimov’s rules were fictional and full of loopholes. For example, how can a robot obey human orders if it is confused by its instructions? Where these laws really falter in reality is when robots face difficult choices, where there is no clear, agreed-upon answer.

Take the classic ethical dilemma known as the ‘trolley problem’.38 It goes like this: you are the controller of a runaway trolley (train) that is hurtling towards a cluster of five people who are standing on the track and face certain death if the trolley keeps running. By flipping a switch, you can divert the trolley to a different track where one person is standing, currently out of harm’s way but who will be killed if you change the course. What do you do? Philosophers argue there is a moral distinction between actively killing one person by flipping the switch or passively letting people die. It’s a no-win situation with no right answer. Autonomous machines will soon face countless situations akin to the trolley problem but they won’t be clouded by human panic, confusion or fear.

Now imagine it’s 2030 and you are in a self-driving car going down a quiet road. You have mentally switched off, you’re chatting with your personal gurubot on your iPhone 52 about three things you will do this week towards your happiness goals, while the car is in full control. A pedestrian suddenly steps out, right in the path of the oncoming vehicle. Should the car swerve and avoid the crash, even if it will severely injure you? The car must make a calculation. What if the pedestrian is a pregnant woman and you, the car owner, are an elderly man? What if it is a small child running after a ball? Consider this: what happens if the car in a split second can check both the pedestrian and the car owner’s trust scores to determine who is a more trustworthy member of society? And herein lies the daunting challenge programmers face: writing an algorithm for the million and one different kinds of foreseen and unforeseen situations known as real life.

So the next question: who is to be held responsible for choosing a particular ethical charter? When AI kills, who should take the blame? If the engineers and manufacturers set the rules, it means they are making ethical decisions for owners but are also in line for accepting responsibility when things go wrong.39 On the other hand, if an autonomous car becomes free to learn on its own, to choose its own path, it becomes its own ethical agent, accountable for its own behaviours. We can only begin to imagine the legal conundrums that will follow.

If your dog bites someone, the law is very clear: you, the owner, are responsible. With AI, however, it is currently legally hazy whether it will be the code or the coders that will be put on trial. ‘One solution would be to hold human programmers strictly accountable for the impacts of their programming,’ says Sir Mark Walport, the UK government’s chief scientific advisor. ‘But that could be so draconian an accountability that no would take the risk of programming an algorithm for public use, which could deny us the benefits of machine learning.’ We are entering an age of algorithmic ethics where we need a Hippocratic Oath for AI.40 Perhaps algorithms will end up being held to higher moral standards than irrational humans.

A group of researchers from MIT, the University of Oregon and the Toulouse School of Economics were interested in discovering the moral decisions different passengers would want an autonomous car to make. They ran all kinds of scenarios and found that, in theory at least, participants wanted the autonomous vehicles to be preprogrammed with a utilitarian mindset, sacrificing one life in favour of many. However, more than a third of the 1,928 participants said they thought manufacturers would never set a car’s ‘morals’ this way; they would programme cars to protect their owners and passengers at all costs.

But the most interesting finding was around personal choice. The majority of participants wanted other people to buy self-driving cars that would serve the greater good but when asked if they would buy a car programmed to kill them under certain circumstances, most people balked.41 ‘Humans are freaking out about the trolley problem because we’re terrified of the idea of machines killing us,’ writes Matt McFarland, the editor of ‘Innovations’ at the Washington Post. ‘But if we were totally rational, we’d realize one in 1 million people getting killed by a machine beats one in 100,000 getting killed by a human. In other words, these cars may be much safer, but many people won’t care because death by machine is really scary to us, given our nature.’42

For regulation to go through, programmers and manufacturers will have to design self-driving cars that are more trustworthy than a human driver, resulting in far fewer accidents and fatalities. The bar may not be that high; for starters, autonomous machines don’t text or get drunk or easily distracted while driving. When we reach that place, we may never need to trust a human driver again. Indeed, I think my young children will never learn to drive; they will see it like learning to ride a horse–merely a hobby. And one day, humans will need a special permit manually to drive a car. Indeed, human drivers will be the threat to people in autonomous vehicles. Human trust in machines will only increase; in some cases, it will become much deeper than our trust in our fellow humans.

The next generation will grow up in an age of autonomous agents making decisions in their homes, schools, hospitals and even their love lives. The question for them will not be, ‘How will we trust robots?’ but ‘Do we trust them too much?’ It won’t be a case of not trusting these systems enough–the real risk is over-trusting.

Robot over-obedience is another issue. They need the ability to say ‘no’, not to carry out human instructions mindlessly when their actions might cause harm or are even illegal. For instance, I don’t want my son, Jack, to be able to tell a household robot to throw a ball at his sister’s head. So how does a robot decide when it’s okay to throw a ball–such as to a child playing catch–and when it’s not? How does it know when its human operator is not trustworthy? ‘Context makes all the difference,’ says Matthias Scheutz, professor of cognitive and computer science at Tufts University. ‘It requires the robot not only to consider action outcomes by themselves but also to contemplate the intentions of the humans giving the instructions.’43

Aside from that, can a robot understand its own limitations? Let’s imagine a surgical robot in an operating room with its super-small steady ‘hands’ carefully snaked into a patient’s body. It’s five hours into a twelve-hour complex heart surgery. The patient on the table is a six-year-old girl. The robot discovers an abnormality that complicates things. It’s not 100 per cent sure what its next move should be. In this moment, the robot needs to tell us, ‘I’m not certain what to do next,’ or even, ‘I don’t know what to do. Can you (doctor) help me?’ Ironically, a little robot humility will go a long way in making them more trustworthy.

‘We need systems that communicate to us their limits, but the other half of that relationship is we need to be ready to hear that,’ says Stephen Cave. ‘We will need to develop a very sophisticated sense of exactly what role this machine is fulfilling and where its abilities end, where we humans have to take over.’ This will be extremely challenging because our natural tendency is to become over-reliant on machines.

Cave has three young daughters of a similar age to my kids. At the end of our conversation, we talked about what we can do to prepare them for this inevitable future. ‘They need to know at what point they should interrogate the machine,’ he says. ‘We know how to interview humans for jobs but we need to teach them how to test the limits of the machine.’ I can see it now: my son, Jack, in 2035, twenty-five years old, sitting in a workplace with a robot, asking it, ‘What do you do?’, ‘What can’t you do?’ and ‘How do you admit your mistakes?’ Of course, there is another possible future scenario: the robot is interviewing Jack.

At the end of the day, the responsibility for making sure robots are trustworthy and behave well must lie with human beings. Whether that will remain possible, if scientists like Stephen Hawking are right, is another, thornier question.