Get in touch

Artificial intelligence is all about automating human behavior — and that includes bad behavior. How can technologists write software that helps rather than harms? Paul and Rich discuss the rise of apps that encode shady approaches like loan sharking into automated tools, and whether social pressures may eventually right those wrongs — or if that change will come too late.

Transcript

Paul Ford The next podcast should be us just looking at our password managers [oh, no, no] just figuring out what the hell we’ve done to ourselves. 

Rich Ziade Oh yeah. It’s a wasteland. [music fades in, plays for 15 seconds, ramps down]

RZ Hey Paul.

PF How you doing, Rich?

RZ Doing okay, actually.

PF Yeah, it’s the actually is the operative term right now, right? 

RZ Yeah, I mean, I’m doing very well. To our good friends who are listening to us in the year 2025. I couldn’t be better. [music fades out]

PF Yeah. Yeah. We’re getting by, we’re figuring it out. My AC is dead. So as New York, I got about two weeks to resolve this issue [yeah] and it’s on the roof. I may just pass out. 

RZ No, just fluids and just do the right thing. Take care of yourself.

PF There’s a funny thing that happens. You finally have to just accept that things are going to be what they are and Postlight’s doing okay. And the kids are healthy and I have in firm parents, but they’re making it work. And so like, I better go for a bike ride.

RZ You should go for a bike ride. 

PF You’ve been doing your morning workouts pretty steady. 

RZ Yeah. Very steady. In fact, I think I’ve lost weight at home. 

PF Yeah! I can tell.

RZ Also I’m I’m not eating out as much. It makes you eat very differently.

PF Boy are we not. God. What do you miss? What do you miss?

RZ Sushi. I do miss sushi. But you know what? I’m going to, I’m ready to take the leap forward here. Right? There’s I ordered out Thai food on Saturday night and it was lovely. 

PF Oh, you haven’t been ordering, we’ve been ordering like once a week. 

RZ Yeah. We’re going to start ordering out. Think we’re going, it’s just, it also brings us a little relief. [mhm] Like just not even think about cooking every minute. 

PF I’ll tell you what’s, what’s fascinating is, you know, we’ve been eating a lot of like regular stuff, noodles, pasta, things that are simple. We also have now a hundred, a hundred pounds of flour in our closet. And I bake bread on the weekends, which is a whole other world. 

01:56 

RZ Why would you do that? 

PF Because it’s one of the outings that, one of the things that forces the shopping is the lack of like bread, like the lack of simple staples. [Oh yeah] And so once you get the, once you figure that strategy out, you’re going outside like half as much. 

RZ Interesting. 

PF Been interesting that way. So look, actually, we’re speaking about global things. There is a website that I’m increasingly fond of that came out. A brand new website with knowledge on it. It’s called restofworld.org. 

RZ Ooookay. 

PF And what it’s about the tech industry, [okay] except I’m going to tell you, there’s a lot of tech coverage about like Facebook and algorithms and what tech is doing to our culture and how it’s ruining democracy and all that stuff. And that’s good. It’s important. [yeah] You gotta have it. There’s a lot of the websites out there that are doing the good work. This one has a different angle. It’s basically what the hell is happening outside of America and Silicon Valley in terms of the tech industry.

RZ Fascinating stuff. 

PF So this, this platform is set up to cover that world to be like, how are things going? You know, here and there and elsewhere. And there is a pattern that they described that I want to talk about, which is [okay]. Okay so the article is Morris Kiruga wrote this article and it is about a popular financial technology app, a FinTech app in Kenya and Nigeria called OKash. O K A S H. I won’t read the title cause I want to like, let me just play it out. Ask me for some money and I’ll be the bank. 

RZ Okay. Can I have $100?

PF Oof, that’s a lot of money, but let’s, let’s start you at 15. No problem. Here you go. Install the app. There’s your 15 bucks. 

RZ Okay. Thanks. 

PF Can you pay me back? 

RZ Uh, what schedule? 

PF Well, just, just go ahead and pay me back. Like fine schedule. Everything’s good. Pay me back. Go ahead. Do it. 

RZ Okay. 

PF Here’s, give me my money. Okay. Give me my money now. Okay. I got $30 for you if you need it. Cause you were really on top of the 15. 

04:00 

RZ Okay. Yeah. I mean, I can always use money. I use the money. I mean, I use it it’s capital to me. I invest it. So I’ll take $30. 

PF Hey, you got that $30 for me? Good. How about 60? 

RZ Okay. Ah, yeah, I’ll take it. 

PF Hey, wait a minute. [yeah] Wait a minute. Hey, you’re a little late paying back the 60 bucks. 

RZ Okay. But you know, I, it’s a loan. I will get it back to you. Just give me, I just need a little time. 

PF Here’s what’s going to happen, buddy. We, when you installed the app, we got every single person in your social network. We know all of them, all their email addresses and phone numbers. And we will be letting them know if not cleared by 4:00 PM. 

RZ Mmmmm… No.

PF Yup. That’s what we’re going to do.

RZCan we talk about this? Can we just talk this through, right?

PF Well, you’re not yet. You didn’t pay me by four. So I guess not. So let’s, uh, let’s see what happens to your friends and family and coworkers when they know, click. The guy paid back the $15. Right. [yeah] And he, you know, then [yeah] he paid back the 20, then they get you. They literally are like, we’re going to just keep turning this up because we make our money when, when you’re paying us a little interest on these, on these loans. So, I mean, it’s a fascinating pattern cause actually, you know, you brought this…

RZ Is it unethical to ask you that question?

PF Yeah. Yeah. Just flat out.

RZ Yeah. What if the tool said, look, if you don’t make payments, we are going to contact all your contacts? Do you, are you sure? Yes or no.

PF I mean, you know, for all, you know, that’s the terms of service on the Apple thing [Rich laughs]

RZ Oh probably. It probably is. 

PF Very likely, very likely. I have given someone the ability, like the right to completely get into everybody’s business in my world. Like messed up patterns in the kind of tools and frameworks that we use get called out in awkward ways. Right? So if you subscribe to some service, let’s say this has happened. This has hit everybody a couple of different ways. And our like very nerdy world, which is some VC is touting some new thing and you jump in, you’re like, all right, I’ll sign up for it. You log in through Twitter and you don’t, and they’ve actually locked this down a lot better than, than they used to. But you give it permission to tweet on your behalf. [mhm] And it’s like, don’t worry. We’re never going to do it. You know? And then like one day you wake up and it’s like, I’ve got over 44 followers on Friend Master. And you’re like, what the hell is, what? 

RZ Yeah.

06:17 

PF And your friends are like, that’s weird. And so, everybody makes fun of that…

RZ My mom signs up to these things all the time. Like I was invited to all sorts of weird shit.

PF See, so you and I are part of a cohort that socially will punish organizations that do that. And, and then they have to write an apology note and show that they understand it. We apply collectively social pressure [mhm] because we don’t like when people do stuff like that on our behalf, those are norms [yeah] that we gave them the actual permission to do it. But then when they did it, we were like, nuh uh, that’s not cool. 99.9% of the world has absolutely no social capital that they can expand to tell people how to do terms of service.

PF No. And not only that, the value prop is so strong for them in many cases like, wait a minute, you’re gonna give me this tool? That’s going to give me, I can use it as a shopping list maker for nothing? Okay. And then you’re signed in, but you know, in the TOS it says, we’re going to actually look at what’s in your shopping list. Pass that data along to Google. Next thing you know, every time you think about toilet paper, you’re going to see ads on the LA Times website. And it’s like, ohhhh, okay. [Rich laughs]

PF Yup. 

RZ I think most of the time people are like, I guess, I mean, it’s such a good tool. It’s so handy. [Yeah] It’s fine. You just sort of go along with it. 

PF I think it’s, we accept a certain amount of creepiness as the, just the part of existing.

RZ I bet if they had it in big bold letters, exactly what they’ll do to you. If you don’t pay your debts, people are still signing up left and right. 

08:00 

PF  You need $15 right now. And look, I mean, it’s like Venmo and people pay each other. And like, there’s a social signal. It’s not, it’s not bananas that this is going to happen. Right. Except that apparently this, this app in particular got a rep. Like people were like, Whoa, what just happened [Well I mean..] to me? Because when they call your mother-in-law it’s tattling.

RZ Yeah. I mean, look, what is software? Assist software is automating what would otherwise be human behavior. And let me tell you, there is that loan shark in the village who will let everyone know if you don’t pay them back. So some software developers like, you know what? I can code that up. I can code up Leon and his buddies. [Rich laughs]

PF You know what’s tricky and people should read the article. Right. But as it, as it unfolds, like of course they’re, you know, owners for all these financial platforms and like all this stuff, what you see is that, so right. You’re automating that behavior. [yeah] You’re, you’re adding this structure. You’re putting that pattern in. Where things go wrong, where product thinking tends to fall down is when you take a portfolio based approach to human behavior. And you’re like, [Rich laughs] you know what, if we did that, you know, and if we get the money at all costs, right. And here’s, what’s tricky…

RZ They probably viewed it as a deterrent. And look, I think there’s such power in these tools, especially tools that give you money, that what people are willing to give up their, you don’t even have to be underhanded. You could lay it all out and they’ll still do it.

PF Here’s the future. As far as like, so I look at this and like, okay, this’ll probably get regulated. But the lesson to take away and the lesson to take away, I don’t think is like, ”don’t follow dark patterns in your products’.” Like we all know that. The lesson to take away is things that you would never cultural categories that you would never assume would be subject to automation [mhm], like automating legal action or automating like broadcast style collections are utterly possible because when somebody is accessing you to provide a service to you, they get a huge amount of ambient information back. And so for the most part, it’s what you were talking about. I’m going to get an ad. I’m going to show him this toaster over and over again, and really cross my fingers. Because if I can get 0.1% of them to buy toasters, every everybody’s going to have a good Christmas, but what’s happened. So we’re like, okay, I see that. I kind of see the database. They want to sell me a toaster. I’ve been targeted to buy stuff. My whole freaking life fine. Okay. 

RZ Yeah.

10:19 

PF That’s how it is. Robots are trying to make me get my wallet. [sure] But like when you start getting automated lawsuits or medical stuff, or just, you know, I see that like your insurance starts to figure it out. [yup] That’s when we start to freak out and look, I think what is real and where we have to adapt is that no matter what we say the rules are and what we like, or don’t, it is this unstoppable force. And it doesn’t matter whether you think it is good or bad, unless you are like the President of the United States, you are going to be living in a world in which things that it was inconceivable, that computers would do it automatically like sue you, or, you know, [yeah] tell you, tell you who to date or grade your papers. All of these things are coming. And so like, how are you going to negotiate as an individual? I don’t think we have really good guidelines for how you negotiate in a world like that, where robots decide your fate. 

RZ Yeah. I agree. Uh, I don’t think we know. I think we’re going to step in shit a bunch of times and then adjust. I think, I think where you, where you can draw lines is, is these tools should not be using tactics that would be viewed as unfair or intimidation and whatnot from humans. I mean, the truth is they are used. I mean, you ever get those letters in the mail that you couldn’t tell it was an ad, really? 

PF Yeah.

RZ It looks like, it looks almost like a bill that may be, Oh, when you register a company, when you register a company…

PF Oh my God, yeah. Yeah, yeah, yeah.

RZ They send you these letters that are just shady because it looks like the state of Delaware sent you the letter, but really it’s just the guy named Joe. Who’s trying to hit you up for a fee to protect your name or whatever there is they’re selling.

12:03 

PF Oh, I can’t remember which publishing company. I think it’s kind of one of the big glossy ones that publishes the fancy magazines. Somebody billed them like $2 million and they just paid it. [Rich laughs] 

RZ There you go. There you go.

PF It wasn’t good. Like, it was definitely a bad day for, bad day at work where they’re like, well, why are $2 million in payments going to Joko? 

RZ That’s right. That’s right. And you get utility bills that like pay us through the, pay it through us and it’ll be your bill will be cheaper. And, but the bill that like the, the proposal, like the pitch, it comes in a bill, it looks like a bill itself. Right?

PF Oh yeah. Well, and it’s the same look at the email marketing. Like it’s this is what blows my mind is how dumb they are. Right. Cause I get five pitches a week and, or probably more like five a day, frankly selling me software services overseas because they want to supplement Postlight or just anything I get, Oh, we also get like, Hey, why don’t we, we’re going to do a video profile of your company for this very legitimate sounding cable network. And you’re like, and then you go look at it and you’re like, no, okay. No, you’re not. You’re going to charge me $14,000 to get in like a 19 year old to read a press release. [Rich chuckles] Um, right. And so when, when you look at that, and then you think for a minute, Rich, come with me into the evil side for one minute, the amount of information we have and the quality of narrative that you can produce. Like if you search for stock results on a middle tier company, you will find like 50 articles written by robots. That’s a, that’s a normal thing. [yup] Okay. So there’s that right? What could we do to really get people to pay more attention? Like you think about the automated profile that you could put together of me and that, that database, and then the kind of emails that you could write that would be really personal and really helpful. Like I got one the other day that was fascinating. Somebody pulled the report for me and sent me like some statistics and the thing was a garbage. But what they did was they said, I know that this person with this email address is interested in technology companies. 

14:09 

RZ Mhm.

PF And I have this database of market sectors. So I’m going to make them this weird little report. [yeah] And then they’re going to want to pay attention to me. And it’s like, I saw it through it really quickly. Actually, if you had any sense of quality, you could get my eyes. You could really I’m gettable. [sure] But nobody, nobody will put the work in because they’re just going to optimize for like that 1% of chowder heads who will hit the button.

RZ And they, and you know, that’s success. 1% is a huge success. [yeah that’s right] I mean, that’s the game, right? It’s not that different. I mean, look, it’s a milder form of scam, right? Like my mom gets calls all the time. She’s like, look, I found cocaine and you’re in the trunk of a car you abandoned in Texas two months ago. [Paul laughs] If you don’t wire me money. And I got to tell you, you gotta understand, my mom is so incredibly entertained by the whole thing that she engages. She’s like really? 

PF Ohhh, cocaine?

RZYeah. Yeah. She’s like really so lots and lots of money’s worth. Huh? And the guy is yelling at her at her saying, listen to me, I’m serious right now. Right. 

PF Oh I love those guys. 

RZ And my mom is hysterical laughing. But you know, what, is there an old, you know, is there an older man or an older woman who doesn’t know what’s going on? It’s like one out of every hundred 50 calls. 

PF Oh, totally.

RZThat’s good. That’s really good.

PF Lenny. Just look for Lenny telemarketing, which is this chatbot that a guy made. And it’s like an audio chat bot for telemarketers. That is from 2008 and it is not advanced technologically. It’s just an old guy who likes to talk about his kids. And he just, if you listen to it, it’s just him going like, yeah, well, no, hold on a minute, hold on a minute. Can you give that back? 

RZ [Rich laughs]Oh that’s so good. 

PF And one of them goes for like 45 minutes and like literally they just burst into flames because the guy, you know, and he’s like, no, no, no, no, wait a minute. My daughter’s at university. Very proud, very proud. 

16:05 

RZ [Rich laughs] That’s so good.

PF And I mean, it’s, it’s some of the most stellar stuff. It’s almost just like, it must want to put it onto the background. It’s so good. 

RZ That’s great.

PF Humans, when they behave like robots are susceptible to all the things, all the honeypots that catch robots, right?

RZ Sure!

PF Like when we put humans on a script, they lose their flexibility. And so then like you end up in this position, this is what I think happens in the future, which is that your personal platform becomes all these different agents that are essentially filters, keeping automated systems from getting to you. Right. 

RZ You think we’ll pay a fee for that. 

PF You already do because you pay for your, you pay for Google. And now they’re filtering spam calls because the government couldn’t come up with policy to block bazillions of spam calls that started to come through all of a sudden, a couple of years ago on everybody’s mobile phone. [yeah] And now I, my, when it pops up, it says suspected spam caller. And I go, yeah, I don’t want to, I don’t want to talk to that. That’s not for me. So like I’m now paying two or three giant mega corps [yeah] who can’t, who could automatically protect me, but choose not to, I’m paying them passively through various kinds of fees and services to take care of me that way. And I think that will apply to, you know, there will be a setting for older people who are vulnerable, where you can like turn it up to max, like no incoming calls without approval. [yeah, yeah] And we just don’t have the immune system at the local level because everyone got so excited to just like generate all that revenue.

RZ You know, I think going back to Okash, I think the thing about Okash is, and the reason they get more latitude is that there’s a very clear relationship. There’s a transactional relationship between the user and the company. And I think what people, what’s happened over the last 20, 30, 40 years is that most people don’t realize that they’re entering into all kinds of relationships, like actual, formal legal relationships, where they are allowing certain things to happen. Right.

18:07 

PF Oh, and you know, you know what the great tell is here? You know how you know how bad it is is you go into your password manager now. 

RZ Oh, it’s unbelievable. Isn’t it? 

PF[Paul laughs]And you’re like, why do I have 1500 accounts? [yeah] Like why, what happened?

RZ You know, there is a, there’s a counter to all this and that’s a deeply ethical, smart, engaging, or, you know, corporate entities and organizations out there that just do good solid work and don’t want to be, uh, don’t call us if you’re doing the, the shady stuff.

PF Well, you know what? They get in touch all the time. Cause I see all the sales emails. You know what the most amazing filter is for all of this is price. [no] Nobody on the shady side ever wants to pay retail. 

RZ [Rich laughs] It’s very true. It’s very true. It’s kind of amazing, isn’t it?

PF It’s really wild. Cause I’ll see stuff and I’ll be like, I shouldn’t even bother to reply to this, but I’ve got a bad gut feel. But I always like to kind of play things out from a sales perspective. Like we can always say, no. I like to play things out to just kind of see where they go. And it’s how I learned about the market. Boy. They, they melt down when they see what it costs to build software. They cause they they’re just wired to rip you off. Right? Like they’re just like, no, no, we’ve an amazing deal here. You should come in and build this for free [yeah] because then I’m going to be able to get you some opportunity. And I’m like, no, that’s not really our business works. [yeah] But you know, if you want to, if you’d like to see our capabilities deck and they’re like, bahhh[Paul chuckles]

RZ You know, because also these are people that are used to it, you know, marketplaces where it’s all a million little tiny transactions rather than anything meaningful. Right. 

PF No that’s right and it’s somebody’s cousin [Rich laughs] and now suddenly… It’s also like, I’ll tell you very, very few blockchain-related ideas ever, I mean there’s so serious when they show up and I’m just like, great. Would love to talk to you about the amazing thing you’re doing. I love this technology stack. Let’s talk about it. And they can’t seal the deal. [no, no] They just, because I think what happens is like the price is so volatile that they’re like, Oh no, we can’t do that anymore. And then they call back again, like six weeks later and they’re like, Oh yeah, yeah, actually, no, everything’s fine. Bitcoin’s up above 9,000 again. 

20:14 

RZ  Yeah well they’re trying to enter the world of actual commerce and see what happens. Right. 

PF So I think actually what you just said is exactly right, right. Which is that we as a strategically and not even consciously decided that we would only function in the real economy and  that really throws people off about the companies they’re getting, they can get really confused. So look, what we are is a real place that works with real companies and organizations to build real products for real users. Usually I’ll tell you what, we get an MVP out, minimum viable, viable product we like to ship. And we start shipping code really quickly after people get in touch. But then Rich, tell the people how long we stay. 

RZ We don’t leave. Like the party’s over. Everybody’s like throwing the solo cups away and we’re still there. [mhm] We’re just going to sit there. 

PF Well, you know what we are, we’re the friend who sticks around and does the dishes.

RZ Help clean up! Yes. Absolutely [Rich laughs] I like this analogy.

PF We clean up and then, yeah, we do the dishes and we talk, now, we don’t overstay our welcome. If it’s time to go home, it’s time to go home. 

RZ Absolutely. Absolutely.

PF But then we might even, we will show up the next day to help get the trash out. [music fades in]

RZ Yes. We are driven by relationships more than we are about projects. And I think that’s, that’s been kind of a difference for us. 

PF And that is, that is really who we are. 

RZ Who are we?! We haven’t said the name of the company, Paul.

PF Oh my god, we are Postlight. We are your digital product partner. We are here to help you.

RZ Yes.

PF With your digital transformation and you know, that is a lot of hand waving. But what we actually do is say, how are we going to build it? How are we going to get in front of users? What needs to happen? Whether it’s management or humans or training or new software, new approaches, what needs to happen to make it happen in your organization? And then we do the work. Unlike most other consulting firms. I won’t name any names. We build the freaking thing. 

22:05 

RZ Reach out to us though. We love to talk. Hello@postlight.com. 

PF Hello@postlight.com. 

RZ We are based in New York City, ambiently, but we’re around the world and we love to chat, hit us up.

PF Oh and check out restofworld.org. That was a great article and good, good conversation fodder. 

RZ Yeah. Have a great week. 

PF Bye! [music ramps up, ends]