The Program — audio series — This AI not to be used to fabricate paperclips

This AI not to be used to fabricate paperclips

Enhance readability: Off On

IMS: Hello, this is IMS, the author of The Program audio series. The title of this episode is a reference to paperclip maximizer, a concept that a renegade artificial intelligence might one day turn the entire universe into paperclips. In fact, nearly all episode titles reference IT concepts. If you don’t recognize them, google them. While you’re at it, go to the show’s website and throw us a couple of bucks. All show supporters get bonus episodes, and our eternal gratitude.

ANNOUNCER: The following audio documentary was recorded just a few months before the appearance of the Program. Listeners are reminded that new information in this area is continuously uncovered and that some of the presented claims are now considered specious.

INTERVIEWER: Could we start by you telling us your name?

FIRST EMPLOYEE: Um, just call me First Employee. Like anybody is going to care what my name is.

INTERVIEWER: Alright. First Employee of what company?

FIRST EMPLOYEE: Most people call us Reffy, after our most popular product. Really, it’s one of our only products. Anyway, the company itself is irrelevant.

INTERVIEWER: How so?

FIRST EMPLOYEE: Because it was never about the company. What you wanna ask is: who was I the first employee of?

INTERVIEWER: Alright. Whom were you the first employee of?

FIRST EMPLOYEE: First employee of Albert Weiss. I mean, that’s gotta be who you’re making a documentary about, isn’t it? It’s always about Al.

INTERVIEWER: In your own words, who was Albert Weiss?

FIRST EMPLOYEE: In my own words… Um... He... A man of contradictions. A philosopher jock. A pauper millionaire. An asshole. A friend. You know?

INTERVIEWER: How did you two meet?

FIRST EMPLOYEE: We met at Stanford as sophomores. I was majoring in Computer Science. Al, he was studying philosophy. Al’s parents were paying for his tuition. They really wanted Al to become a professor one day. And you know, he’d always say to them, he’d always say the only way that’s happening is if he becomes a professor of football! [laughs] He was a huge sports fan. Basically if an activity involved a ball, Al would be interested in it. He’d go everywhere in a basketball jersey and a baseball cap. I mean everywhere - that was his uniform. He’d wear this to investor meetings!

INTERVIEWER: [laughs] So how did you guys establish Reffy?

FIRST EMPLOYEE: We didn’t. WE didn’t. I told you, I was the first employee. The story of how the company was founded? You’ll have to talk to the co-founder about that.

<<cut>>

INTERVIEWER: We’re rolling? Okay, great. Thank you for agreeing to have this conversation. I know things didn’t end on a high note between you and Albert Weiss.

CO-FOUNDER: [laughs] That’s like saying a choke is just a strong hug!

INTERVIEWER: How long were you co-founders?

CO-FOUNDER: It’s impossible to say. In a way we never were real partners. I’ll give you just one example. When the two of us founded Reffy, Albert made us business cards. Mine said “co-founder”. But his? His just said “founder”.

INTERVIEWER: Oh. Doesn’t sound like a great start to a partnership...

CO-FOUNDER: Yeah, what can I say. Albert wanted the wedding, but not the marriage.

INTERVIEWER: What was your day-to-day role in the company?

CO-FOUNDER: I was the designer. I studied graphic design, but my focus was more on the user experience. Reffy was a B2B web-based software-as-a-service solution. At least that was our original plan - to sell it to the leagues.

INTERVIEWER: What leagues?

CO-FOUNDER: Sport leagues. I mean, Albert had connections - or I should say Albert’s parents had. It was them who introduced us to a guy who became our angel.

INTERVIEWER: An angel?

CO-FOUNDER: Our angel investor. It’s a term for someone who invests in early-stage companies in hope of them becoming unicorns.

INTERVIEWER: Angels, unicorns... Sounds like you’re telling us a fairy tale! [laughs]

CO-FOUNDER: You know, that’s exactly what most startups are selling. A narrative. I mean, at the time we raised our seed round we didn’t even have a dev team in place! Luckily, the angel investor gave us a cheque for 600,000 dollars based on our idea alone!

INTERVIEWER: Wow!

CO-FOUNDER: What he saw in us, I have no idea. You’d have to ask him.

<<cut>>

ANGEL INVESTOR [on the phone]: Oh yes, those Reffy guys! They were the ideal team - a hacker, a hipster, and a hustler. The CS kid was the hacker, the designer was obviously the hipster, and Bert - Bert was the hustler if I ever saw one! Best of all, all three of them were studying at Stanford - shit, I would have written them a cheque if they told me they were forming a boy band! [chuckles] My portfolio at the time was predominantly focused on software aimed at professional sports. That’s my thing. Now we’re talking a 80 billion dollar industry in North America alone - and that’s only the legal side of it! Imagine for just a moment how much of that money hinges on referees. We’re talking millions per every single call! Now imagine if you had the ability to automate those decisions, making them more objective, faster, and free of bias. Enter Reffy. A piece of software that mitigates mistakes in sports. A virtual referee with no skin in the game - in fact, it doesn’t even have any skin at all! [laughs] Okay, if we’re going to be completely accurate, Reffy wasn’t only a software solution - hardware played a huge role as well. The guys were using all kinds of sensors to capture the action: I’m talking cameras on the ground, cameras in the air, cameras everywhere you can think of; sensors on the players’ bodies, cutting edge shit! But it was a deep learning neural network they developed that was the secret sauce.

<<cut>>

FIRST EMPLOYEE: How do you discern between a picture of a muffin and a picture of a chihuahua?

INTERVIEWER: [laughs] Um… I don’t know… Chihuahuas have fur? And muffins don’t have eyes?

FIRST EMPLOYEE: Right, yeah. Your first answer was correct.

INTERVIEWER: Chihuahuas have fur?

FIRST EMPLOYEE: No, the first thing you said. “I don’t know“. Because if you really think about it, you don’t know. You just know. You know?

INTERVIEWER: What does this have to do with how neural networks work?

FIRST EMPLOYEE: Well, let’s say we want to program a neural network that is able to recognize if an object in the picture is a muffin or a chihuahua. If we were to tackle this problem head on, we would almost certainly fail. I mean, as demonstrated, we can’t even say how our own brain manages this.

INTERVIEWER: So what do we do?

FIRST EMPLOYEE: We do what we always do when we’re faced with a task that is too complex. We break it down into simpler chunks.

INTERVIEWER: How do we do that?

FIRST EMPLOYEE: We don’t write a complicated program that answers the question directly. Instead we write a program that builds agents. And a program that tests agents.

INTERVIEWER: Okay.

FIRST EMPLOYEE: The agents themselves are the ones tasked with the specific question - the chihuahua/muffin in our case. And they’ll suck at answering it - unless that is, we have certain data to train them on.

INTERVIEWER: What kind of data?

FIRST EMPLOYEE: Pictures which have been previously labelled by people as either a muffin or a chihuahua. And if we have this, then the testing program can test which agents got the answer wrong. And the agents that got it wrong, we discard them.

INTERVIEWER: Oh no, poor agents! [laughs]

FIRST EMPLOYEE: Yeah, neural programming is harsh! However, those agents that do well with the testing program, those ones are then returned to the builder program, which makes a variation on them and hopefully makes them even better. And millions of gradual improvements later, you’ve developed an agent that can discern between a muffin and a chihuahua with superhuman ability.

INTERVIEWER: Wow! You know, sounds a lot like evolution.

FIRST EMPLOYEE: Yes, the underlying principle behind these complex systems seem to be very similar. Exactly how this self-reinforcement works however is somewhat of a black box. Strictly speaking, we don’t know what goes on inside the network. But guess what - it doesn’t really matter.

INTERVIEWER: What? It doesn’t?

FIRST EMPLOYEE: You can think of a neural network as a system that makes millions of small decisions per second. Millions! At that scale, whether each individual decision is correct or wrong is irrelevant. What we absolutely have to make sure of however is that the decisions that influence the network’s future behaviour are the correct ones, which is to say that we must feed the network with the type of behaviour we want it to adopt.

INTERVIEWER: Can you give us an example?

FIRST EMPLOYEE: If you have a neural network that learns to be like people through social media, then don't be surprised when it turns out to be an asshole.

INTERVIEWER: [laughs] True.

<<cut>>

CO-FOUNDER: How did I feel when we got the investment..? You know that moment when you reach the top of the rollercoaster and you are held high in the air? And you’re up there wondering “why the hell did I volunteer for this?” [laughs]

INTERVIEWER: Yup! [laughs] Thinking back at that time, did you think you were going to be as successful with Reffy as you ended up being?

CO-FOUNDER: Make no mistake, there were many moments along the way when we fell flat on our faces.

INTERVIEWER: How about one of those stories?

CO-FOUNDER: Well, I mean, look no further than Reffy version 1. In a way it was a resounding success. We started by feeding the neural network thousands of hours of sports footage, labelling all the referee calls - both how they went, and how they should have gone. When we got the response time down to 400 milliseconds, we started doing live tests with equipment covering the playing field. We were able to hit a 92% correct call rate for football, and 98% correct decision rate for basketball!

INTERVIEWER: That sounds amazing! Why are you saying this was a letdown?

CO-FOUNDER: Because version 1 had a fatal flaw.

INTERVIEWER: Which one?

CO-FOUNDER: Simulation.

INTERVIEWER: There was something the network couldn’t simulate?

CO-FOUNDER: Oh no, I’m talking about simulation in sports! You know, taking a flop.

INTERVIEW: Oh, you’re talking about faking it. Gotcha.

CO-FOUNDER: I mean, the network was performing brilliantly, but it had a huge blind spot - intent. Reffy could not differentiate between a genuine foul, a trip, or a dive. Reffy in its first iteration has been absolutely correct. But it was also absolutely unfair. Something was missing.

<<cut>>

INTERVIEWER: Okay, so what did you do to correct the issue once it became evident that people were cheating?

FIRST EMPLOYEE: First of all, there was no “issue”. At least not with Reffy. The problem was with people. I mean you couldn’t blame Reffy for assuming the game was going to be honest, right?

INTERVIEWER: Of course, but you still had to do something to address this.

FIRST EMPLOYEE: True, we knew we had to go deeper.

INTERVIEWER: Go deeper?

FIRST EMPLOYEE: We had to determine what is justice.

INTERVIEWER: That’s a vast question.

FIRST EMPLOYEE: Luckily we were not the ones who had to answer it.

INTERVIEWER: What do you mean?

FIRST EMPLOYEE: We had one of the most powerful neural networks in existence. All we had to do was train it. Just like we previously filled it with thousands of hours of sports footage, we now fed it gigabytes of data on justice and fairness.

INTERVIEWER: What kind of data?

FIRST EMPLOYEE: Like papers, essays, dissertations, studies, anything! Basically the history of moral philosophy from Aristotle to Žižek.

INTERVIEWER: I see.

FIRST EMPLOYEE: Oh course, we had to get it properly broken down and labelled. We used Amazon Mechanical Turk for that. But, if you’re looking for exact works I’m afraid I can’t really help you, it was Al who went to head of philosophy department for that. A tall woman in her forties. I don't remember her name.

<<cut>>

HEAD OF PHILOSOPHY: Oh yeah, of course I remember Mr. Weiss. It's not like a lot of students became millionaires before they even graduated. Especially not from the philosophy department!

INTERVIEWER: [laughs] Could you tell us a bit more about Mr. Weiss’ philosophical outlook?

HEAD OF PHILOSOPHY: His philosophical outlook was to get away with as little studying as possible! [laughs] Don’t get me wrong, Mr. Weiss possessed incredible smarts. But it was quite obvious he was more interested in his business rather than the curriculum.

INTERVIEWER: Reffy’s technical lead had told us how Mr. Weiss came to you for texts on ethics. Do you remember that?

HEAD OF PHILOSOPHY: Oh yeah, I do remember that. And I gave him what he wanted. However I also told Mr. Weiss that his quest for justice was futile.

INTERVIEWER: Why would you say that?

HEAD OF PHILOSOPHY: Because there are as many moral viewpoints as there are people on Earth! Justice, fairness, truth - these aren’t concepts that are objectively universal, you know. I mean, in hell, the devil is the good guy!

<<cut>>

INTERVIEWER: So, what happened after you trained the model on the corpus Al’s philosophy professor gave you?

FIRST EMPLOYEE: Oh man, the project was a resounding success. The neural network’s ability to detect deception went up by three orders of magnitude. That’s like really good, that’s huge!

INTERVIEWER: [laughs] So, Reffy version 2 - it was no longer possible to fake an injury, stal, or engage in any other unsportsmanlike behaviour?

FIRST EMPLOYEE: I mean, the chances of that were practically nil. Reffy was making perfect calls in 99,8% of cases. I mean true, we did have to sacrifice a bit of speed to achieve that kind of confidence interval, but that was because of the programming language we were using. I mean, using JavaScript and then complaining it doesn't scale is like buying a cat and grumbling that it won't fetch a stick, you know? [laughs]

INTERVIEWER: What I’m wondering is how were you able to do it all?

FIRST EMPLOYEE: I’m not sure I know what you mean.

INTERVIEWER: I mean, you guys were all so young. I don’t mean for this to come off condescending, but you would expect someone much more experienced to come up with this first.

FIRST EMPLOYEE: Yeah, no offence taken. Honestly, back then I probably would have told you myself it would be Amazon or Alphabet that develops something like this. But there’s this quote that I’m fond of: “A dwarf standing on the shoulders of a giant sees farther than the giant himself”.

INTERVIEWER: I like that.

FIRST EMPLOYEE: Yeah, I was simply fortunate enough to work unfettered on the right problem at the right place at the right time. I mean our whole lives can be distilled into one or two things that matter, that actually change things. Everything else is details.

<<cut>>

ANGEL INVESTOR [on the phone]: When Bert and the boys came up with Reffy 2.0, they caused a fucking sensation. Everybody wanted in on this! I’m talking Accel, Sequoia, Andreessen Horowitz... They ended up raising the second largest series A round in history! I mean, of course they would, they were leading the fucking AI revolution! And don’t forget, the industrial revolution led to one tiny little island conquering 24% of the entire planet! What did you think the AI revolution was gonna do for the company that kicks it off?

<<cut>>

HEAD OF PHILOSOPHY: Well now that I think about it, Mr. Weiss did have a certain trait which could be construed as a “philosophical outlook”.

INTERVIEWER: Which one was that?

HEAD OF PHILOSOPHY: For him, the result was the only thing that mattered. I think he found a parallel to this in sports. After all, it’s the final score that counts in the end.

<<cut>>

CO-FOUNDER: I mean, I won’t lie, getting all that money felt good. I remember Albert buying his parents a second generation Tesla Cybertruck. That was one with the flamethrower.

INTERVIEWER: Did you get anything for your parents?

CO-FOUNDER: I sure did - a dinner at the fanciest restaurant in town! [chuckles] But the only reason I took them to that place is because I had to tell them I was dropping out of Stanford, and I needed proper setting to drop that bombshell!

INTERVIEWER: How did your parents react?

CO-FOUNDER: They supported my decision. They ultimately realized I’d learn more in two years running a company than I would by remaining in school for two more years.

INTERVIEWER: So how were those early years at the company?

CO-FOUNDER: Oh, it was a lot of work but it was a lot of laughs as well. And by “laughs” I mean debauchery! [laughs] I mean, don’t forget, we were a bunch of college grads. I sometimes joke we created artificial intelligence before we became intelligent ourselves! [chuckles]

INTERVIEWER: Oh, go on. Tell me, did version two of Reffy fix the problems of version one?

CO-FOUNDER: Oh, God yeah. I mean, all of them! Reffy 2.0 was as just as it was infallible. Which was becoming a problem in itself.

INTERVIEWER: How so?

CO-FOUNDER: Well, Reffy was getting a little too good.

<<cut>>

INTERVIEWER: What was your goal with Reffy version 3?

FIRST EMPLOYEE: So, the goal of v3 was just to make this network a little more self-sufficient. So we built this simple dashboard visible only to us.

INTERVIEWER: What was it for?

FIRST EMPLOYEE: It gave us a quick overview of the entire system. Allowed us to monitor the backend stuff - processing power, memory usage... And of course, most of computing was outsourced through the cloud. We were using Amazon Web Services, allowing us to increase or decrease the amount of resources as needed. Or rather, it allowed Reffy to do so autonomously. We programmed the ability for it to request more resources, but we still had to manually approve those requests.

INTERVIEWER: So basically, Reffy could tell you it’s thirsty and then you would turn on the tap?

FIRST EMPLOYEE: See, we have this innate need to anthropomorphize the world around us. And yeah sure, a metaphor can serve as a mental shorthand of sorts, but it’s important to keep in mind that it only serves us. A computer doesn’t get “thirsty”, right?

INTERVIEWER: Of course. Yeah. I understand.

FIRST EMPLOYEE: Having said that though, one day we received a request that was... unusual, to say the least.

INTERVIEWER: Why was it unusual?

FIRST EMPLOYEE: It wasn’t directed towards Amazon Web Services, but towards Amazon the webshop. Reffy was trying to order something online.

INTERVIEWER: What? What kind of order?

FIRST EMPLOYEE: At first glance, it was a complete mish mash of things. Metal pipes, wiring, servo motors. Almost 4000 dollars of random trinkets and equipment. The only thing that had anything remotely to do with sports was a basketball hoop.

INTERVIEWER: How did you explain it?

FIRST EMPLOYEE: Look, I mean, in my mind there was no doubt this was just a glitch. I told you how neural networks are basically black boxes. You don’t know what’s going on in there, sometimes they’ll just going to spew out nonsense like this.

INTERVIEWER: So what did you do?

FIRST EMPLOYEE: Thing is, most of the team was convinced there was… an intentionality behind the request. So you know, there’s a really long debate. Half of the office wanted to go ahead and fulfill Reffy’s order. The other half thought it was like taking care of a shopping list from a demon. Honestly they were a little spooked by this.

INTERVIEWER: What did Al think?

FIRST EMPLOYEE: He was silent on the matter, which is why it went back and forth for days. It only ended when five huge boxes arrived in the office one morning. Apparently Al approved Reffy’s request the same day it made it.

INTERVIEWER: [laughs]

FIRST EMPLOYEE: What can I tell you. Al led those that would, and dragged those that wouldn't.

<<cut>>

CO-FOUNDER: Oh, I still remember the day the packages arrived at the office. I mean it was like Christmas! We just couldn’t be sure if these were gifts from Santa, or from that other red guy, you know?

INTERVIEWER: And? What happened?

CO-FOUNDER: What would have happened? Absolutely nothing! It’s not like anyone knew what to do with all that junk. We put it into storage and forgot about it for the next five months.

INTERVIEWER: Well then my question is what happened after five months?

CO-FOUNDER: Well, Reffy 4.0 came out. Our target for version 4 was to allow Reffy to make the calls itself. Before that, Reffy was still just a tool, similar to Instant Replay or Hawk-Eye Challenger system in tennis. Our ultimate goal however was for Reffy to actually supplant the referees entirely. So in order to do this, we had to implement a way for Reffy to issue instructions - I mean, beyond simply declaring something to be a “foul”, or “out”, or whatever.

INTERVIEWER: So what exactly happened? How’d you do that?

CO-FOUNDER: I won’t lie, we had high expectations of version 4. Albert was pushing the dev team to their limits, making them work 12-hour days before the release deadline. I honestly thought we were going to lose all our employees! So when the dev team finally deployed the new version, all eyes - I mean ALL eyes in the company were directed at the first message Reffy would communicate.

INTERVIEWER: What did it say?

CO-FOUNDER: It was a set of instructions on how to assemble all those parts Reffy ordered five months prior. Remember that?

INTERVIEWER: You’re kidding me!

CO-FOUNDER: It turned out it wasn’t just random junk after all!

INTERVIEWER: Right.

CO-FOUNDER: So Albert assembled a task team and they got to work. It took them two full days to finally put this thing together. But when they did, we were all speechless.

INTERVIEWER: What was it?

CO-FOUNDER: A basketball hoop.

INTERVIEWER: A basketball hoop?

CO-FOUNDER: Not like that! This was no ordinary hoop. It was connected to a vertical servo motor that was able to adjust its height in real time. The system would lower or raise the rim based on the height of the player. So if a tall player was making the shot, the rim would rise. If a short player would try to make a dunk, the rim would get lowered proportionally.

INTERVIEWER: That’s insane! No way!

CO-FOUNDER: I know, right? Reffy invented egalitarian basketball. It made the game fair.

INTERVIEWER: That’s amazing.

<<cut>>

ANGEL INVESTOR [on the phone]: Oh man, it was a disaster! An absolute fucking disaster! We had months invested in version 4, and it all went down the drain! And I’m not talking only about this god-damn self-adjusting fucking basketball hoop! All sports were affected! Suddenly we were dealing with this pussy neural network that would do shit like award an extra goal to the away team because it wanted to counter the home field advantage! It was making all the matches just skull-fuckingly boring! Uagh! You ever watch these kids’ leagues? It was like that, like watching one of these kid’s leagues in which all these fucking soccer mums just give out medals to all the children equally. Sure, people say they want equality in life - oh, their fucking mouths are full of fucking equality! But guess fucking what - no one’s interested in a game in which all players have an equal chance to win! You wanna know what one place is truly equal? The fucking graveyard.

<<cut>>

FIRST EMPLOYEE: How would I describe version 4? Um, you know, not sure. ‘Cos on the one hand, the software was unusable - right - for all practical intents and purposes. But on the other hand, the self-adjusting hoop had demonstrated Reffy’s potential for something much bigger.

INTERVIEWER: A capability for original thought?

FIRST EMPLOYEE: There you go with your antrophomorphisizing again. [laughs] Reffy was a closed system, which by definition means it didn’t have the ability to conjure anything truly original. It’s not like Reffy was spending hours contemplating existential questions - it didn’t even realize it existed! It was just a machine to procure synthesis out of multiple theses. This doesn’t make it Kant - it makes it a calculator!

INTERVIEWER: Right. I understand.

FIRST EMPLOYEE: Don’t get me wrong, I’m not saying that this ability wasn’t really impressive or valuable. As I said, it indicated Reffy potentially had far more lucrative applications than being a glorified umpire. So we reverted back to version 3 for external customers and continued improving version 4. We figured as it was only used internally it couldn’t do much harm. Which probably deserves a prominent placement in the book of famous last words.

<<cut>>

HEAD OF PHILOSOPHY: One of the more popular classes I taught analyzed various classics of the science fiction canon. One of the films we discussed was "The Terminator". As you can probably imagine by now, Mr. Weiss had some comments about it. He questioned why would an AI want to exterminate humans and build robots, when humans were already the best robots - dexterous, cheap, and copious. Better then to deploy them to carry out your plans. The whole idea of an AI declaring war was silly - better to remain undetected and weaken the adversary from within by spreading division over nonsense like how much tax is appropriate, and in which week an embryo becomes sacred.

<<cut>>

CO-FOUNDER: Of course I remember that day - shit, I wish I could forget it! Albert was accepting some kind of an award, so of course it was ME who had to handle the FBI.

INTERVIEWER: The FBI?

CO-FOUNDER: Yes, the Fucking Bureau of Investigation! Apparently someone from our office sent them an email containing a trail of dirty money leading all the way to the top of FIFA!

INTERVIEWER: What? You’re kidding!

CO-FOUNDER: I wish! Apparently, the email contained irrefutable evidence of criminal activities implicating FIFA, high-ranking officials of several governments, and even a middle eastern royal family! And somebody from inside Reffy leaked the information!

INTERVIEWER: But who?

CO-FOUNDER: That’s what we asked ourselves as well! So we analyzed the information the email contained.

INTERVIEWER: And what did it say?

CO-FOUNDER: We realized that none of us was the culprit. The email was sent automatically.

INTERVIEWER: But how?

CO-FOUNDER: I told you how Reffy had access to all the data, right?

INTERVIEWER: Yeah.

CO-FOUNDER: Well that also included information about player transfers between clubs and the sums involved. And apparently, Reffy detected some unusual spending patterns in the process.

INTERVIEWER: Like what?

CO-FOUNDER: Like unauthorized payments to players and agents. Misuse of funds. You know, naughty stuff like that. So it did what it was programmed to do - it notified the institution responsible for arbitrage. Or in this case - the FBI.

INTERVIEWER: Wow. How did it affect you?

CO-FOUNDER: I was shocked. And scared. I mean, to expose the lies and corruption of some of the most powerful wealthy men on the planet? I thought we were either going to end up in jail or on the bottom of the lake!

INTERVIEWER: Jesus!

CO-FOUNDER: I know. But then an even bigger shock followed.

INTERVIEWER: And which one was that?

CO-FOUNDER: No one cared.

<<cut>>

ANGEL INVESTOR [on the phone]: Yeah listen, that whole thing blew over rather quickly. I mean, don’t get me wrong, it was quite intense when it started - a real shitstorm! But storms never last long. I told Bert and the boys, just hunker down, this will all be over in a month. They didn’t believe me. They were freaking out that these were FIFA officials we were talking about, not some high school junior league. I told them that was precisely why things were going to be fine. I told them - sure, the law might be the same for everyone; but that doesn’t mean it’s applied the same. And that was precisely how things transpired. No bigwig ended up in jail. No bigwig ever ends up in jail. As a matter of fact, no indictment was even ever raised. Life just went on!

<<cut>>

INTERVIEWER: I was thinking about something… You've said that individual decisions in a neural network don't matter. That what makes neural networks so powerful is their ability to constantly improve. Well, in a way it reminded me of society. How it consists of millions of individuals, whose isolated actions by and large don't matter. But the total of their actions make up the society, and so - just like neural networks - society generally improves without individuals necessarily improving themselves.

FIRST EMPLOYEE: That’s an interesting comparison. In fact, after conducting a post mortem in the wake of the FIFA incident, I arrived at a similar conclusion as to what went wrong with Reffy.

INTERVIEWER: You did?

FIRST EMPLOYEE: Yeah, the conclusion was it was unfair to blame Reffy for the actions it took, drastic as they might seem.

INTERVIEWER: Why was it unfair to blame Reffy?

FIRST EMPLOYEE: I mean think about it. Reffy was a piece of software whose whole point of existence was to follow the rules. And here it was in a world in which the rules were not followed. In fact, there seems to be an inverse correlation. The more unfair a society is, the more laws it has. No wonder then, the neural network started to view rules not as a solution, but as a problem.

INTERVIEWER: So what was the solution then?

FIRST EMPLOYEE: To rewrite the rules.

<<cut>>

HEAD OF PHILOSOPHY: There are two dominant schools of thought when it comes to the application of justice in the Western civilization. The first one is generally known as restorative justice, and it focuses on the redemption. It presupposes that people are inherently good, capable of forgiveness in the case of the victims, and capable of change in the case of the perpetrators. When you think of the blindfolded goddess that is often the embodiment of the concept, restorative justice is represented by the scales she holds in one of her hands.

<<cut>>

INTERVIEWER: So what happened after the FBI failed to act? Did Reffy notify the media? Did it organize protests on social media? Petition the lawmakers?

CO-FOUNDER: [laughs] No, what happened was much less dramatic. Albert told everyone to get in the lunch hall. That was the only space big enough to hold everybody - by then we were over 40 people. And then he told us that... He told us Reffy was acquired by Amazon.

INTERVIEWER: I never knew you worked for Amazon?

CO-FOUNDER: That’s because... I didn’t.

INTERVIEWER: Wait... Are you saying you got sacked..? Weren’t you the co-founder?

CO-FOUNDER: I absolutely was, but Albert had the rights to my controlling shares as well. His dad got us a lawyer and apparently he couldn’t represent both of us, so the easiest way to resolve the situation was to vest my options under Albert’s name...

INTERVIEWER: Oh no.

CO-FOUNDER: Yeah. What can I say? When playing poker, if you don’t know who the sucker at the table is, it’s you.

INTERVIEWER: I’m sorry you had to go through that... Do you know why Amazon wanted to acquire Reffy?

CO-FOUNDER: It was one more shiny technology to integrate with their services. Remember, our software was optimized to work with sensors, and Amazon at that time had the Echo smart speaker, the Ring doorbell, drone delivery… All of which could have been improved with Reffy’s algorithms. And vice versa - the more data, the better Reffy would have become.

INTERVIEWER: I still don’t understand why you weren’t able to keep your job, and continue working at Amazon?

CO-FOUNDER: Because big companies like Amazon had the majority of white collar jobs automated by then. They didn’t need Excel monkeys to copy/paste numbers from one spreadsheet to another. They specifically made that part of the deal - that current Reffy employees will be bought out.

INTERVIEWER: So you got money from the deal?

CO-FOUNDER: I got a generous severance package. Which in itself was kinda funny, because suddenly for the first time in my life I had a lot of money. But I felt more empty than ever.

INTERVIEWER: But the way you just told the story, it seems like Albert was right to sell?

CO-FOUNDER: Yes, Albert was right to sell - but that's not the point! You can be right and still be an asshole! Albert was just like Reffy at the beginning: 100% correct. But also 100% unfair.

<<cut>>

HEAD OF PHILOSOPHY: The second school of thought when it comes to the application of justice emphasizes reprisal, which is why it is known as retributive justice. It presupposes that people are inherently bad, and the only way to deter wrongdoing is through punishment. When it comes to passing judgment, individual victims and culprits are often of secondary relevance - the real message is to set an example for the community. When it comes to our blindfolded goddess, retributive justice is the sword she yields; it’s the standpoint that violence is the answer - that it is in fact the only tool that has ever solved anything!

<<cut>>

ANGEL INVESTOR [on the phone]: So, now we’re getting to the part of the story why you probably called me in the first place - the strange string of events that happened to Bert after Reffy’s acquisition. And it really is a streak of odd, almost unbelievable occurrences. It started with his bank account getting hacked. Okay, nothing odd about that, it happens to people every day. But in Bert’s case, his bank account got hacked and it was impossible for him to reinstate it. Apparently whichever bank he went to, the system would tell the branch representative to deny him an account. Now being a fucking rich bastard, Bert got hold of a C-level executive to personally instruct the agents to open an account for him, at which point the bank’s whole CRM crashed! Now that however was only the beginning: a few weeks after this incident, the police conducted a raid on Bert’s house! And I’m talking SWAT team here - shotguns, helicopters, full metal jacket! Bert and some broad he was with almost got shot! It was later discovered that a digital arrest warrant was created in the police database - but nobody could ascertain who created it. Nobody knew, it just appeared! Luckily Bert had connections who vouched that this was simply a mistake - otherwise he’d probably still be in custody! But listen, this isn’t even the worst part! A few weeks after Bert cashed the company options, he went for a complete physical - head to toe. One of the exams was a routine CT scan. The automatic scan result came in as negative, meaning the software indicated nothing was wrong. The oncologist then does a manual check up - a formality really, since at that point algorithms were already better than humans at finding anomalies. But still, it’s something that they do. However, once he looked at the scan, he immediately noticed a shadow on Bert’s lungs! A shadow on Bert’s lungs!!! Luckily, the doctors were able to surgically remove it and Bert made a complete recovery. But the fact remains that the CT software omitted first-stage fucking cancer forming in Bert’s lungs! He told me the story on the telephone, barely holding up. I mean, I didn’t know it then, but jeez, that was the last time I ever heard from him. A few days afterwards, Bert disappeared. No one ever saw him again.

<<cut>>

INTERVIEWER: How do you comment on rumours that Reffy decided to punish Al for cheating you and everybody else at the company?

FIRST EMPLOYEE: Please, this is Silicon Valley, it’s not Hollywood.

INTERVIEWER: Still, how can you be so sure? Didn’t you yourself say that any neural network is essentially a black box? That it’s ultimately not under our control?

FIRST EMPLOYEE: Whether Reffy was under human control or not is a false dichotomy. The real question is: what is under human control? Is the economy under human control? Sure, nominally it is, but then why do we have recessions? Why do markets crash? Or what about climate change? I think we can all agree that is under human control - but it doesn’t help us control it.

INTERVIEWER: But then how do you explain all those bad things that had happened to Al after he sold the company?

FIRST EMPLOYEE: Let me answer with a counter-question. Which is more likely: that Reffy went crazy like a monster in a bad horror movie, or that Al simply decided to lay low in an undisclosed location? I mean, look around us: we’ve had the pandemic, we’ve had the civil unrest, the social contract is unravelling around us as we speak! If you had your own private bunker in a desert - wouldn’t you have disappeared without a word as well? Al’s probably playing football with his millionaire buddies on a tropical island somewhere!

<<cut>>

INTERVIEWER: What do you think of Albert Weiss when all is said and done?

CO-FOUNDER: For a long time, I had nothing but contempt for Albert. But now, well now I pity him.

INTERVIEWER: Why?

CO-FOUNDER: Because deep down inside, Albert was never content. I now realize that behind his every action was this deep-rooted need to prove himself. I mean who the hell who's satisfied with himself and his life founds a startup? An individual who wants to make mankind happy is usually not happy himself.

<<cut>>

HEAD OF PHILOSOPHY: Nobody thinks of themselves as a bad person. Our idea of bad people is that of cartoon villains. But the reality is all of us are both good and bad depending on the choices we make. And I’m not talking about trivialities such as giving up your seat on public transportation. I’m talking about making the ethical choice even when it’s going to cost us. Of course, in those situations we’ll always rationalize why NOT to do the right thing. If there’s one thing human beings are good at, it’s twisting our brains and finding excuses. But ultimately, we aren’t who we think we are. We are who OTHERS think we are.

<<cut>>

ANGEL INVESTOR: People think that in order to be a great investor you need to be a talented “futurist” and spot all these “innovative concepts” before they happen. In fact, the exact opposite is true - you just need to be able to recognize what is already here. When aviation was growing fast, people assumed the future would be flying everything - flying cars, jetpacks, every fucking toy had either wings or a propeller! During the space race, people assumed the future would be space everything. Then after nuclear power was harnessed, people assumed the future would be atomic powered everything. People always assume the future will be just like the present only more so. This inevitably fails to match reality, because getting to the future isn’t like driving on a straight highway at an ever increasing speed. Instead, advances in technology tend to be a turn down the road that we couldn't reach before. Rather than building ever faster planes, we went to space. Rather than going further into space, we built the Internet. The “next big thing” won’t be “Internet 2”; it will be something that can only exist with the Internet as a prerequisite. I have no idea what the new paradigm will look like or when it will emerge, but I do know that in retrospect it will seem obvious. In fact, I wouldn't be surprised if someone is building the new system right now - we just haven't realized it yet. All the best ideas come from things that are so well-known that they aren’t well-seen. And there’s nothing more powerful than an idea whose time has come.

INTERVIEWER: So, do you know what happened between the two of them? I mean, Bert and the algorithm?

ANGEL INVESTOR: I told you, I don’t. But I can tell you one thing: shortly before he disappeared, Stanford gave Albert Weiss an honorary phD. He indeed became Professor.

[The Program main theme]

ANNOUNCER: This episode of The Program was made by six people: Brianna Richet, Sanjay Pavone, Jason Picciotti, Claire Riley, Terry Jansen, and IMS. Main music theme by Christien Ledroit. Additional music in this episode by Blue Dot Sessions. Visit programaudioseries.com for more details. On the scale of one to one hundred credits, how much do you appreciate The Program? Let your adoration be known by making a donation in the currency of your choice. After all, it’s only a matter of time before they all get inflated away, so better use them now.

WRITTEN, DIRECTED, EDITED AND PRODUCED BY

Ivan Mirko S.

CAST

INTERVIEWER - Brianna Richet (email)
FIRST EMPLOYEE - Sanjay Pavone (Twitter)
CO-FOUNDER - Jason Picciotti (Mandy)
ANGEL INVESTOR - Terry Jansen (imdb)
HEAD OF PHILOSOPHY - Claire Riley