This week Paul and Rich talk to two former chief technology officers (CTOs) who are also good friends and frequent collaborators: Camille Fournier, who was previously at Rent the Runway, and Kellan Elliott-McCrea, who was previously at Etsy. They discuss the role of the CTO within a company, share experiences from the trenches, compare their experiences managing engineers versus managing CEOS, and swap stories about the most colossal technical outages that happened on their respective watches (Kellan took down Yahoo Messenger [literally]; Camille ruined everyone’s Thanksgiving [well a network switch was reconfigured, it was a bad day])
Paul Ford: Hey, Rich!
Rich Ziade: Yo!
Paul: We’re back again for Track Changes, the little podcast that can.
Rich: Hell yeah.
Paul: Hey Rich, have you ever been a chief technology officer?
Rich: I’ve never formally had the title.
Paul: Have you done the work?
Rich: I would think so.
Paul: Is it fun?
Paul: It’s not fun?
Rich: Um…it has its moments.
Paul: I thought it would be really interesting to talk to some chief technology officers, because these are the people who decide how the big apps and the big web sites actually get made.
Rich: OK. So, you found some?
Paul: Yeah, two examples, one is Etsy, it’s a little web site you might have heard of.
Rich: Mmmm hmmm.
Paul: The other one is Rent the Runway.
Rich: Also heard of it.
Paul: OK. We got the CTOs, former CTOs —
Rich: Of both.
Paul: — to come on in and talk to us.
Paul: These two people together could manage probably something like three or four hundred engineers at any given moment.
Rich: And easily half —
Camille Fournier: I think maybe even more than that.
Paul: Maybe even more.
Rich: Probably More.
Camille: Yeah. I’m going to go ahead and guess more.
Paul: The person who just spoke is Camille Fournier. Camille, thank you for coming in.
Camille: Thank you for having me.
Paul: You were, until not too long ago, the head of engineering and then the CTO of Rent the Runway.
Paul: That is here in Manhattan?
Camille: Yes, it is.
Paul: How long were you there?
Camille: About four years.
Paul: Immediately to your left is Kellan Elliott-McCrea.
Kellan Elliott-McCrea: Hello.
Paul: Who I’ve known for a long time and was the CTO of Etsy.
Paul: How did you guys meet?
Camille: My friend Harry —
Kellan: Ah, yes, Harry.
Camille: — who is one of my best best best friends for a long time, and I think I remember the time we met was at Tom and Jerry’s.
Kellan: Sounds likely.
Camille: This was right after I had started at Rent the Runway.
Paul: Bar right above Houston Street.
Camille: Yes, yes.
Paul: OK, yeah.
Camille: Some event for the Foursquare people.
Kellan: Good Dark and Stormies.
Camille: Yeah, there was a lot of Foursquare people there because Harry was running engineering at Foursquare and he introduced me to Kellan. Like all my good friends, the first time I met him, I thought he was kind of obnoxious.
Paul: Kellan might be a little bit obnoxious.
Rich: Aloof is what I’m getting.
Paul: No. I mean he’s just … Sorry, we’ll come back to that.
Kellan: [indignant noises]
Rich: This is useful, because I met Kellan ten minutes ago, and I’m trying to get him to smile at me.
Paul: Like he’s a toddler?
Rich: Well, no…
Paul: Are you holding up brightly colored pieces of paper?
Rich: Can I make an impression on Kellan in these first ten or fifteen minutes? I struggle with it.
Paul: When a man’s been the CTO of Etsy, do you think you can easily make an impression? Think about what he’s seen.
Camille: Oh, man.
Camille: So many crafts.
Paul: I’m kind of fascinated by the two of you, because you’re CTO pals! That, to me, is a cool thing to be.
Kellan: We have a whole CTO club.
Paul: Do you really? Is there a CTO club in New York City?
Camille: There are multiple CTO clubs in New York City.
Kellan: There are many CTO clubs.
Paul: Tell me about this. I don’t understand anything about this. We should actually explain for our audience that this is chief technology officer.
Paul: It’s a pretty serious role in an organization, and what it means is that — Rich, I would say that engineers are expensive and often complicated individuals.
Paul: And so a CTO is somebody who can wrangle not just a couple, but usually dozens, hundreds of engineers, and get them to build really big abstract technologies and ship them and make money from them.
Rich: Well, the make money happens on the other side of the wall.
Kellan: I would say that’s an aspirational goal. [laughter]
Paul: To be a CTO is actually to be able to wrangle an enormous amount of abstractions and it’s also not necessarily a role that people talk about in New York City as much as like — this is not Silicon Valley. We’re in a different place.
Camille: Thank God.
Paul: Right, OK. Why do you say, “Thank God?”
Camille: Well, so for me, I live in New York at least partly because I have spent time in San Francisco and the Valley, and I did not want to live in a place where my career was, like, “the thing,” where everyone around me — you can’t throw a can and not hit an engineer or another CTO. It’s just everywhere. I like to hear people talking about things that are not tech when I go out to dinner.
Paul: Fair enough. What about you?
Kellan: Same. Well first, there was a girl, but that aside, the other reason for being in New York — the girl being my wife of going on two decades, but whatever. The other reason being very much what Camille said. In San Francisco you meet people. “Yeah, I’m in tech, but I have this side project at night.” I work at a start up during the day, but at night, I’m doing a start-up.”
Paul: I have a night start-up.
Kellan: I have a night start-up. I have day and night start-ups. In New York, there are more and more tech — certainly when I moved here, it was kind of weird to be in tech, it’s now pretty common. I live in Williamsburg. It might as well be the Mission, but people at night at least do other things. At night, I’m in a band, at night I’m in the theater, at night I’m like, there is something else that you do.
Paul: Sure. I drink wine until blackout.
Kellan: Yeah, exactly.
Camille: Only if you’re a CTO.
Rich: There’s a recurring theme taking hold. If you listen to our first-ever podcast, I think I made this exact point, about how New York is about all the other stuff, and that’s what makes it New York.
Paul: We had John Lax come in and talk about being at Facebook for a year.
Paul: He’s gone over, in a good way, he’s got a big job and he’s doing a lot of work. We had to get him to break out a lot of acronyms for us, because we were just like, “Wait, what is that?” I can’t even remember what they were, it was like “high value individuals,” or individual contributors, “ICs” —
Rich: ICs. The weird stuff.
Kellan: Yeah. We probably talk about ICs daily.
Kellan: You know, it’s funny, some people ask me, “I thought Etsy was from San Francisco?” I’m like, “Who would have sold something on the site if we’d been in San Francisco?” Of course it’s from Brooklyn. There actually is an amazing sewing community in San Francisco, but you can’t imagine bootstrapping something like Etsy in San Francisco. Where would you find anyone to use the tool?
Paul: Fair enough. Everyone is too busy crafting a start up at night.
Kellan: Yes. Exactly.
Paul: Nobody is knitting Minions.
Kellan: And digital goods came much later in the life cycle of the product.
Rich: Question for both of you: where are you originally from?
Camille: I was born in DC, but raised in Tallahassee, Florida, actually.
Kellan: I’m from the Bay Area, Santa Cruz, California.
Rich: Oh wow! So you’re hating on home, huh?
Kellan: You know, San Francisco is going through a dark and dystopian time. This is not the San Francisco that I grew up with. They sort of built a floating walkway summoned by your iPhone over the top of the city right now, and it kind of casts a shadow on everybody. [laughter]
Paul: So now you two have — I want you to explain what Fiasco is.
Camille: Oh boy.
Paul: It’s on your Linkedin page, somewhat —
Camille and Kellan in unison: Yeah.
Kellan: First, you have to understand that we’re both rather unemployed at the moment. We both have projects and we have some projects together and some separate projects, some of them pay the bills, some of them fail to pay the bills. One of the things that you find leaving the CTO job is there’s a fair amount of decompression to do.
Paul: Talk about the compression for one second. What was that like? What’s hard about being a chief technology officer?
Camille: Ugh, ugh, ugh. How do you answer this question tactfully? What isn’t hard about being it, I guess?
Paul: I think that managing lots of people — people are hard, right?
Camille: People are hard, but I would say the hardest part at the end of the day is the executive part of the job.
Camille: I think it’s dealing with the CEO, dealing with the board, dealing with the other executives. Trying to both protect the engineering team and keep it … Engineers are fragile creatures, and they want to be coddled and they have their needs and you want them to be happy because they’re so hard to hire. Also, the business has things that need to get done. You’re constantly the person that feels like they’re in between those two worlds. That’s not all of it, but that’s a huge part of just being stuck in the middle of all those negotiations.
Kellan: I think the best CTOs, there are a lot of different possible means for the CTO, including the first person dumb enough to take the job with this company. There are lots of possible ways you get to be CTO, but the best CTOs are that bridge between the executive function and the engineering function. If you’re in a tech startup at least, you probably have the largest org, you certainly have the most expensive and sort of needy organization, and you’re both the site of, like I said, you’re a major expense for the company, but you’re also the site of what differentiates the company. And you’re trying to build something special, and there are aspects — there’s a reason tech start ups are a thing, there are aspects of engineering culture that allow you to envision the future in different ways that run counter to standard business culture. Your job is in that middle line.
Paul: Like what, though? What’s an example?
Kellan: Classic ones, around decentralized decision making, access to information, good ideas come from anywhere, failure is part of learning. These are not your standard executive belief systems.
Paul: I noticed, I remember once reading the Etsy blog, and you guys are very big on post mortems.
Kellan: We are very big on post mortems.
Paul: Tell the audience what a post mortem is. Maybe both of you, too. Were you also a post mortem fan?
Camille: Yeah, yeah. I think Etsy certainly did a great job of making that practice really wide-spread in the tech industry. People have been doing retrospectives and looking at events for a long time. Of course, a post mortem is when something goes wrong, particularly when you have an engineering outage, and you make an effort, once you’ve gotten things stabilized and things are back working OK, to understand and learn what happened.
Paul: So how do I do a good one?
Kellan: Sure. I’m pretty dorky about this topic. We are the church of post mortems. Or Etsy was the church of post mortems. I would back up a couple of steps and say, first, it’s not about something went wrong. I know better than you, but we’re going to operate on the subject that Rich is a competent and well-meaning individual. [laughter]
Rich: Big leap.
Kellan: Big leap. We’ve just met. I like to give everyone the benefit of the doubt. So you have a theory, there are reasons you’ve developed the theory that the thing you’re about to do is going to work. When it doesn’t work, that’s surprise. So it’s not necessarily, I mean, sure, something went bad, the interesting part is, you have a theory, your theory was wrong, the delta between your theory and what happened, that’s surprise.
When there is surprise, you do a post mortem, because something about the environment led you to believe that what you were going to do as a competent, well-intentioned individual was going to work. The environment misled you. There were missing tools, there were missing practices, there were things that were broken about it. The tests were passing and it actually turned out the tests had just been commented out and they always passed, or whatever it was. There were things that misled you.
And so now we’re going to delve into why you were surprised. You start off a good post mortem by getting a good timeline. You try and get an objective timeline of what happened. This is going to be very useful for a bunch of reasons later, but the primary reason you do the objective timeline is everyone comes in freaked out.
Paul: So are you, and I’m talking to both of you —
Paul: Are you making the timeline or are you saying, “Go make a timeline.”
Kellan: You’ve gotten people in the room. Hopefully people are bringing some information about the timeline, but the first part of this collective, like, we’re all going to sit in the room and talk about why we got surprised, is objective.
Paul: How many people should be in this room?
Kellan: There’s not a hard and fast rule. Obviously, it gets harder the more people that are in the room.
Camille: Yeah, I would say, the interesting thing to me about post mortems, just to interject for a minute, is that I think Etsy is even, John Allspaw, who’s one of the big post mortem folks from Etsy has run classes on doing post mortems. You can take training, it’s not a long training, but you can take training on how to run a post mortem. At the end of the day, it’s kind of like any other meeting. It depends a little bit on who’s there and your discipline in following a certain process.
Some of my favorite post mortems have actually been business post mortems. I think you can’t describe that necessarily as, “Someone had a theory and then they were surprised.” Often the surprise was that nobody had a theory at all, there was no plan, and then many things —
Kellan: Someone always had a theory about why it was a good idea. Even if it was unexamined.
Camille: Maybe that.
Kellan: There’s always a theory about why, I mean, there are often unexamined theories, like, “I was hungry and we needed to get this done.”
Camille: I’m just saying.
Kellan: There are theories that play in.
Camille: It’s always a bit different. You set up the timeline, but the number of people in the room, the discipline with which you do it. The interesting thing about the post mortem is the goal is not to be blaming people for fat-fingering something, mistyping something, having a bad theory, but just to actually make explicit the unstated assumptions and make explicit the things that, “Whoops! We really didn’t think that could go wrong. What can we learn from this?”
Kellan: The fascinating thing is the person who’s most likely to blame someone is the person who made the mistake. “I just wasn’t paying attention. Human error. I should have been doing a better job.” That’s kind of useless. The fact that you weren’t perfect is not actually news to any of us. This is not actionable information. And so a huge piece of trying to get past the blame is trying to get past the person who was the primary actor blaming themselves. Because once we get past that, we can start to understand what it is that led them to stir their false confidence to act.
Paul: OK. So now you’ve started an organization called Fiasco.
Camille: No, no, no. We are interns.
Kellan: We are interns.
Camille: We are interns at Fiasco.
Paul: So you’ve both left CTO-ing, and you both sort of decided, “the hell with it” at the same moment in your lives, roughly.
Paul: You just keep running into each other on the street.
Rich: They both have the same tattoo.
Camille: Which is no tattoo.
Paul: [laughter] It’s of a cloud service, and you can’t see it, but it’s there.
Rich: Would you CTO again? Yes/no?
Camille: Yes, but not next.
Paul: Oh, what’s next?
Camille: Hopefully not CTO. [laughs]
Kellan: It would depend on the people. It’s a terrible job.
Paul: What makes it terrible? What makes it good, what makes it terrible?
Kellan: A huge piece of what makes it terrible, like any other senior management job, and certainly any senior management job that sits atop a creative profession: you got here because you were good at doing something else that you really enjoyed. As your team gets larger and larger, if you are being responsible, all of the fun parts of your day, you’ve delegated. The only thing that manages to make it to your desk are the people problems, and actually, you’ve got people for the people problems. The only thing that manages to make it to your desk are the people problems that they didn’t want to deal with. By definition, you are dealing with the things that were not easy and do not have easy answers, and look nothing like the reason that you started doing this work.
Rich: I think that definition is actually a universal one.
Rich: That’s a C-blank-O explanation that he just gave.
Paul: It has nothing to do with tech, actually, at that point.
Paul: Tech, I think, adds its own set of wrinkles to it, because of —
Camille: The drama.
Rich: — the subculture bubble that you tend to be nurturing inside.
Kellan: Starting with a real distaste and sort of scorn on management as a profession, which is part of the tech subculture.
Camille: Mmm hmmm.
Kellan: And so there’s a lot of self-hatred that goes into being a tech executive.
Paul: Because you should be a genius programmer.
Kellan: Yeah, exactly. That’s the real work.
Camille: The best organizations don’t need managers, man. They are just, like, free! They just self-organize —
Paul: Like GitHub, or Valve?
Paul: So there’s a lot of —
Camille: If we were smart enough, we could just lead ourselves.
Paul: What listeners don’t know is that this is actually a real theme in tech culture is that maybe management shouldn’t be there at all. It’s just getting in the way of true productivity.
Rich: You could argue that the best CTOs appear to be not touching anything.
Kellan: Most of the time, right? And again, the sort of fifth order leadership or whatever, people think that they’re doing it themselves or whatever the various Jim Collins — I think the best CTOs seem like they’re not doing anything most of the time.
Rich: Seem, I think is the key.
Kellan: Also, there does have to be — I’m a strong believer in this not everyone agrees — like, there does have to be teeth. There does have to be a sense that these are the boundaries, these are the lines. As long as you stay inside these lines, everything is fine.
Kellan: When you step outside these lines, then we have a conversation.
Kellan: If that is absent, as it so often is in organizations, then you have a problem.
Paul: That’s one of the things with partnering with Rich, is Rich is the teeth. You are very clear about boundaries in the organization that we’ve built. I’m learning that, as we partner.
Rich: Yeah. I think conveying where those boundaries are is good. I think constantly going around and biting people doesn’t get you there. I think signaling out that those boundaries are there, and again, I think you’re speaking pretty universally here. I’m sort of the operational lead of a shop, which is not CTO, but very, a lot, just about I’d say 90% of what you’re laying out here applies.
Camille: It’s a little bit like raising toddlers, which all of us are familiar with, right?
Kellan: I thought you hated the analogy of parenting and managing.
Camille: I do, because I’m a woman, and therefore, I turn into mom. But. Nobody likes mom! You don’t want to be mom.
Rich: I disagree with your term, “toddlers.” I’ve got to be watchful here. We’ve got a team of people — if I sit here and agree, if you jump in and agree with this, Paul.
Camille: No, I don’t mean —
Rich: “Yeah! They are like toddlers!”
Camille: I only mean like toddlers, or like children and one of the things, if you read any parenting literature, that you are supposed to set boundaries for your children.
Camille: So yes, your team is not children. They are adults, and in fact this is one of the things that I love about the tech culture, is that you do want them coming back to you and having a conversation, but at the end of the day, for organizations to run successfully, there have to be standards. You may re-evaluate those standards, you may question them occasionally and change them. It’s not that they have to be set in stone, but they have to exist.
Paul: So I’m not supposed to help engineering in the toilet? That’s just been a —
Rich: I’ve caught you twice, and seriously, one more time, we’ve got a whole other set of issues. I think you’re laying it out exactly. I think the other wrinkle here is some of the best engineers, they have this view that what they do, that the craft that they’re doing is just pure and it’s too bad that it’s been tainted by business and tainted by all these other interests, because there is a purity to what they’re doing.
Paul: You see that in, like, language battles. That’s when people start to —
Paul: Yeah, monads, right? Like the whole Haskell scene. Now, you were all PHP for about seven hundred years.
Kellan: Etsy was a diverse set of technologies when I arrived five years ago. We standardized on PHP among other things. PHP has this amazing quality, you never have to argue about the elegant way to do anything.
Paul: That’s true, you just, it’s just done.
Rich: He’s pulled out that argument before, I have a feeling.
Kellan: It’s just never part of the conversation.
Paul: Yeah. It’s true.
Kellan: There’s a way to do it.
Paul: Tires will burn for so long, you just light them on fire and they burn and burn. [laughter]
Kellan: I meant to listen to the podcast so I had some idea of what we were going to talk about.
Paul: No, it’s just mostly us throwing people under the bus.
Kellan: Absolutely. I started to listen to the last one and I just couldn’t take it anymore. I think it was Rich said something like, “Well, you should let people use exciting platforms,” or something like that.
Camille: No. Boo.
Kellan: I was just like, “No!”
Paul: Oh, this is what you were angry about.
Kellan: No no no no no. People are excited about the technology, you’ve already lost the first battle.
Rich: Remind me of this?
Paul: I don’t remember —
Rich: If you’re selecting a technology —
Kellan: You’re like, “Use something that people are excited about!” I was just like, “Oh my God, if they’re excited about the platform, your world is in for pain.”
Paul: You said that out loud, but you’ve been badly burned by exciting technology before.
Rich: Yeah. Yeah.
Paul: Yeah. There have been ugly situations in the past where people were committed to a certain language, I don’t even want to say it.
Rich: The truth is, when Etsy started, there was probably whatever, four, six engineers —
Rich: Two. That enthusiastically got behind —
Kellan: One of his jobs was to make animated flames, the other one wrote a lot of code.
Rich: OK, so one. That person picked up what they felt really good at, and what they were excited about, and that planted the seed.
Paul: I’m pretty sure Buzzfeed is still mostly Perl, actually, because it came out of the old, a lot of the old Movable Type world, and so on and they really —
Kellan: But Etsy evolved from them. Etsy is ten-plus years old at this point, and there are the lost years. There was the lost year re-writing into Java, and the lost year re-writing into Python, and the lost year re-writing into Scala. There was that one page on the site —
Paul: Scala got you. It got you.
Rich: You know, I like this history. It fits Etsy, this history of Etsy.
Paul: It’s true.
Rich: It’s like that imperfect scarf that you embrace its imperfections.
Kellan: Very much so.
Paul: So it’s just knitted together, the whole stack.
Rich: And it’s fine! There’s only one of them.
Kellan: Well. There were purges.
Paul: You, I’ve noticed, are both of you computer science majors?
Kellan: God, no.
Camille: I am, yes.
Paul: You are? OK.
Camille: I’m the academic.
Rich: I think you just offended Kellan there.
Paul: What were you, Kellan?
Kellan: I was going to a small, and I mean small, 700-student liberal arts college in Western Massachusetts that didn’t have majors.
Paul: Or razors, or anything probably.
Kellan: Really. [laughter] It’s probably best known for the heavy drug culture associated with Halloween. That’s probably the thing it’s most of well-known for.
Rich: Name of the school?
Kellan: Hampshire College.
Paul: Oh, global systems, all that stuff. They like a good abstract major.
Kellan: Yeah, absolutely. Please, Division III, not a major. So it was all self-guided. There were a couple of computer science classes in there, but they were taught in the cognitive science department, in LISP, by the one guy that was really interested in AI and genetic programming. I only made it through two and a half years of that before dropping out and doing my first start up in Perl.
Paul: Right. And there wasn’t a lot of internet at that point.
Camille: It was the Stone Age, so… [laughter]
Paul: I knew you a little bit way back then.
Rich: What years are these?
Rich: ’95, OK.
Paul: Well, yeah —
Rich: It’s just coming out.
Kellan: Well I mean, by the time our startup was up and running, start of ’97, we were convinced we’d already missed the internet.
Paul: Yeah. That happened.
Kellan: Then we got acquired by Palm, who decided the web was a fad and shut us down.
Paul: And then did miss the internet.
Kellan: Did actually miss the internet.
Camille: It can happen.
Kellan: It can actually happen.
Paul: Camille, you’re Carnegie Mellon, a little school nobody’s heard of, and then, UWis actually has a great program that not everybody knows about.
Paul: Master’s in comp-sci from University of Wisconsin, which is one of the better comp-sci programs out there.
Camille: Yeah, yeah.
Kellan: She’s actually qualified for this.
Paul: Well that’s the thing, I noticed, again, Kellan’s public persona is like, management, thinking thoughts, post mortems. Yours tends to be, “Let’s consider the Paxos algorithm in this context.”
Camille: You know, it’s a little bit of both. I did a lot of distributed systems work before I accidentally tripped into management, kind of. I had one of those, “There’s no one to lead this team so I guess it should be me, let’s make this happen” experiences, which is, I don’t know if that’s why people go work for startups? It was actually kind of why I went to work for a start up and I’m glad I did it. Before that, I was a distributed-systems developer for a really long time.
Kellan: I would like to say I am technical.
Paul: I know! [laughter]
Camille: I think it’s sort of funny that the impression of me is way more technical than the impression of Kellan, because I think of Kellan as as technical as I am and as talking about tech stuff as much as I do and I think of myself as talking about management stuff probably too much.
Paul: I don’t know. As I’ve been watching your profiles, you’re more pure comp-sci. You’re not likely to come out and be like, “As Leslie Lamport wrote in blah blah — ”
Rich: A little bit of hacker, is my read.
Camille: That’s true.
Paul: Kellan, you actually won awards for destroying —
Kellan: Uh, yeah.
Paul: Kellan, did you bring down, what part of Yahoo? You brought something down.
Kellan: OK. We’re not actually telling the story chronologically. I normally tell it in reverse-chronological order.
Paul: Like Pulp Fiction?
Kellan: Absolutely. So at Etsy, we had an award called “The Three-Armed Sweater Award” that we granted to people who most spectacularly broke the site in a given year. We can talk about some of the really spectacular fails, because they are spectacular. Complex systems, it turns out, is a never-ending joy. But it was inspired by an earlier award at Flickr, which was called the “Grant-Pattishall Award” named after two high profile contributors.
Paul: That had like a bowl or something, right? It was like a trophy?
Kellan: It was a trophy, it was coined actually, as I think I was sitting there with my head in my hands after a particularly spectacular outage and Aaron Cope, who is a mutual friend of Paul and I, brought it over, and I was like, “Here, you’ve won the…” There was also a chrome wall mounted, it was actually my first exposure to Etsy, was another award we had at Flickr, which was a chrome wall-mounted tyrannosaurus rex, mid-ejaculation, which you can decide if you want to edit that out of the podcast or not.
Paul: No, that stays in.
Kellan: First thing I ever saw that was acquired on Etsy.
Paul: These are awards for screwing up.
Kellan: These are awards that we gave to people for spectacularly screwing up. In my case, we were about to launch people-tagging, which was a very scary and very complicated project…
Paul: On Flickr.
Kellan: On Flickr.
Kellan: We took the day off, and we’re just working on minor bugs, I was pretty tired. I pushed a very small change to the Flickr home page, and for a very small percentage of the fifty million users or something, it caused an infinite loop. I’ve always maintained it was really Allspaw’s, who was running operations, fault. I detected the error within twenty minutes, and he wasn’t able to get the fix out to the servers.
Paul: So an infinite loop means the page just hangs and it’s bad.
Kellan: Yeah. Just spinning. The computers are just spinning, using more and more, they’re just trying the same thing over and over and over again, to the point where they can’t do anything else. Even a very small percentage having this infinite loop, it quickly flooded all of the computers, and we couldn’t push out the fix. I was just kidding about Allspaw.
Paul: So you did a denial of service attack against yourself.
Kellan: On ourselves. The denial of service attack is coming from within the building. We tried to fix it and we tried to fix it and the computers are so busy, and we eventually just had to call MUD, which was the primary Yahoo data center at that point and say, “Re-start the computers.”
Paul: Like, hit the button.
Kellan: Yeah. Hit the button. And you sort of hang up the phone and you’re like, “They know to do rolling re-starts, right?” We’re talking about a lot of computers. The next thing you hear is, “Is Yahoo Messenger down for anyone else, or just for me?”
Paul: Oh no.
Paul: People don’t think about Yahoo Messenger a lot, but that was how the internet chatted.
Kellan: Yeah, so we had blown a circuit in Yahoo’s primary data center, and that was —
Paul: What did that take to get back up?
Kellan: Patience. It took patience.
Paul: Camille, what about you? What’s the worst thing you ever had happen on your watch, or that you personally did?
Camille: Let me think about that.
Paul: I can tell a story —
Camille: Oh! I do have a story. Thanksgiving. Thanksgiving of like four years ago. I really wish I could remember exactly what happened, but long story short, Rent the Runway went down. The problem is, nobody’s watching, because it’s Thanksgiving, and so I am hanging out with our mutual friend Harry and his wife and my husband and we’re watching one of those Nic Cage movies, American History? American Treasure?
Paul: National Treasure!
Camille: National Treasure! There you go, we’re watching National Treasure.
Paul: Proud I knew that.
Camille: And I get a call on my phone and it’s the CEO, and she’s like, “The site is down.” And I’m like, “Oh.” And so, of course, I have to figure out what the hell went wrong. I believe it was not even my team’s fault, it was the fault of the —
Kellan: It’s always someone else’s fault.
Camille: Well, no, in this case it was —
Kellan: That’s what you learn in post mortems.
Camille: Well, whatever. The cause of the outage, let’s put it that way, the cause of the outage was like a network switch getting re-configured by our hosting provider?
Paul: Oh, on Thanksgiving, that’s a great day to do it.
Camille: On Thanksgiving. Because, you know, that’s the day to do it. Of course, I’m like, this is beyond my depth, so I’m having to call our head of ops, who finally picks up his phone, and he’s in New Orleans and a little tipsy and we finally get him on and he figures out who to call. Unfortunately this is not nearly as spectacular as taking down a whole data center.
Paul: No, but there’s an expanding graph of Thanksgivings being ruined, which is kind of amazing.
Camille: Yes, and you know, in retrospect, it’s like, no one was really using —
Paul: Renting the Runway, at that moment.
Camille: This was before the days when everybody would have their Black Friday deals easing in to the day of Thanksgiving. I agree, Kellan’s giving me this look like, “The site should never be down, it should be up. How dare you?”
Kellan: No. Not at all. I was just thinking it’s not before… We had a policy for years at Etsy that no one took Thanksgiving. Thanksgiving was just not a holiday we did at Etsy, because you’re a commerce site.
Camille: That’s hard.
Paul: Wow. That sucks.
Kellan: Yeah. That was the phrase. Thanksgiving’s going to suck, but we promise Christmas is going to be great.
Camille: That doesn’t work if you are doing —
Kellan: Actually, we never actually followed up on making Christmas great.
Rich: What an awkward phrase.
Camille: That doesn’t work if you are doing rentals, and your biggest rental day is New Year’s Eve.
Kellan: Yeah, fair enough. Different commerce profiles.
Camille: Yeah. Different commerce profiles.
Paul: I had a client once say the words, “Do you want to see your kids or do you want to feed your kids?”
Kellan: That’s a special one.
Kellan: One of the other things that came up, this is probably the moment — Camille and I are actually running a training for engineering managers and directors right now, but the training that I really want to run some day, and I want to find a safe space to run it, is managing your CEO. When the CEO calls there’s this whole adrenaline experience that you have, because the CEO never calls for a good reason.
Kellan: It’s just never a good thing.
Rich: “I just want to say, I was thinking about you.”
Kellan: Yeah. The email is like, “Hey, do you mind jumping on the phone? I’ve just got a quick phone call.” You’re like, this is never going to end well.
Paul: It’s true, it’s never, “Hey, it’s Thanksgiving and one of the things I’m thankful for…”
Rich: The email subject line, “Tried to reach you?” [collective groans of recognition]
Kellan: Tried to reach you.
Paul: Tried to reach you. Checking in.
Kellan: I think there’s a whole — it’s just a question of finding a safe space. I think there’s a whole management-CEO…
Paul: I can provide you that safe space, actually. We’ve had people come and give semi-off-the-record talks. If you ever want to come and talk about CEO management and get a bunch of nerds out in New York City, we can make that happen.
Rich: Boy, will they come.
Paul: They really will. It’s amazing, people really, this is a hot subject. All right. You guys are going to need to go.
Paul: You have to… sounded a little worse than I meant.
Kellan: It’s like, “This is not working.” [laughter]
Rich: We tried.
Paul: This is actually incredibly useful. I feel you’ve been really modest about the stuff that you’re up to. What are you up to? People are going to hear this, what do you want people to know about where you’re at right now and what do you need?
Camille: If you are actually interested in management-y things, I’m doing a few things in that area, personally. I am writing a column for O’Rielly called “Ask the CTO.”
Camille: In fact, if you have any questions that you would be interested in getting a very long and detailed point by point answer to, because that’s kind of my style, I would love to to hear them. I’m doing that. I am actually also working on a book around engineering management, and it’s literally, the angle for this is very much on the stages that you tend to go through as a manager in an engineering management career. Mentoring, being a tech lead, having a team, having multiple teams, being in more senior management, dealing with all the things from both a management, but an engineer’s perspective. How do you think about these special creatures that are engineers and help them do well and enjoy the career yourself?
Paul: How do I find that O’Rielly column?
Camille: That’s a good question. I think if you search “O’Reilly Ask the CTO” you’ll find it. Orielly.com if you look under the writers, for my name as well.
Kellan: It’s also linked from your blog.
Camille: Yes. It is linked from my blog.
Paul: Which is…
Paul: E-l-i-d-e-d branches.com?
Camille: Mmm-hmm. (affirmative)
Paul: What’s the story behind that URL?
Camille: I have whilefalse.blogspot.com, and it was called elidedbranches is the title of the blog, and then when I could finally get the URL, I cold not get whilefalse, unfortunately, so I got elidedbranches instead.
Rich: It’s a granola brand.
Paul: Wildfalls is? OK.
Kellan: While false.
Camille: While false.
Paul: While false?
Camille: While false, yes.
Paul: Oh, while false!
Camille: While false is in the compiler will elide it out.
Paul: I was hearing “wild falls”.
Rich: That’s why I said granola.
Rich:: No. Wild falls? No.
Paul: I see.
Camille: That really fell flat.
Paul: We all got in big trouble there.
Rich: It’s a fruit and nut mix. [laughter]
Paul: So you are out in the world doing things and welcoming input.
Paul: OK. What about you, Kellan? What’s next for you?
Kellan: I’m making a note to answer the question first rather than second, after Camille, next time I’m asked that question.
Paul: Bought a dog…
Kellan: Yeah. I have a small child who I spend a lot of time with. But I’m also doing consulting. I do a little bit of engineering-leadership consulting. I try to work almost one day a week if I can pull it off. Try to remember how to write code. It turns out that it is not like riding a bicycle. Actually, the muscle is the focus muscle, is the one that goes after you’ve been ticking on fifteen-minute management time.
Paul: They change everything, once you go over.
Camille: He’s still writing in PHP, you know he is.
Kellan: Not true.
Paul: Honestly, I’ll take that over any —
Kellan: Then, like I mentioned, the other thing that Camille and I are working on together is we’re actually launching our development program that is a network for managers and directors. We both have benefited from our peer networks, from coaching, and one of the things I tell people, I’m like, “How do I solve this problem?” I’m like, “You should go find some people to talk to.” They’re like, “How do I do that?” I’m like, “I don’t have time to solve that problem for you.” So now I’m solving that problem for folks. So that’s the other thing we’re working on together.
Paul: So that’s you two together.
Kellan: That’s us together.
Paul: What’s a problem I might have? So I have a problem, which is I have a big project that I need to get done by the end of the year. Do I come and talk to you, what do I do?
Kellan: So what we’re trying to do is we’re actually trying to build cohorts of people that talk to each other and actually have trust networks coming out of that you know twenty-odd people who have the same types of problems that you do who you might be able to talk to.
Paul: So do I as a company owner come and talk to you? What do I do?
Kellan: Yes. You as a company owner who may have engineering managers who you actually may want to be better at their jobs, come and talk to us.
Paul: Sure, and they really do. They actively want to be more engaged.
Kellan: The broad areas of thinking about it is the sort of things that all engineering managers are thrashing through are, “I have to build a team. How do I build this team?” You never have enough people. I asked a friend the other day, “How can I help?” It’s like, “Do you know thirty people that can start next week?” I’m like, “I’ll send you the list.” I don’t have that list. Then there’s running it day to day. “Is it supposed to feel like this?” It’s like, “Yes, that is normal.” Then occasionally, “Is it supposed to feel like this?” “No, that is deeply fucked. Let’s de-bug this problem.” Those are kind of the rough areas.
Paul: So I can’t hire you guys, but I can come talk to you.
Kellan: Yes. Actually you can hire me.
Camille: Well, you can pay us money to come talk to us.
Paul: Fair enough. I can’t hire you as CTOs right now, today.
Camille: Yes, no.
Kellan: You can hire me one day a week.
Paul: OK. Yeah.
Camille: Yeah, I might be interested in a similar arrangement for a little while anyway. I think the reason we’re doing this is we really do see that New York City, because it’s an awesome place and it’s not just the center of the tech world, has a little bit of a lack of community for people in more senior roles in tech, and there are not as many seasoned CTOs, there’s not as many seasoned engineering directors and engineering managers. There are a lot of groups for CTOs and VPs, that level and CEOs, but there are not that many groups for people that are below that level.
Camille: Part of what we’re really interested in doing with this particular project that we’re working on together is helping people form those networks and just helping that cohort here in New York become more qualified.
Kellan: Level up.
Camille: So that when they become VPs and CTOs themselves, they’ll be better at their jobs, hopefully. Frankly, I have too many friends working for CTOs and VPs that are just not —
Camille: They’re not doing it, right? It’s a problem I think in the tech industry that we don’t respect management and it causes a lot of unhappiness in the engineering world I think. I’d like to see that problem improve a little bit.
Paul: That’s powerful. That’s great. So is that Fiasco?
Kellan: No, Fiasco is something totally different. I think it’s at the heart — first rule of Fiasco is that you don’t talk about Fiasco — I think it’s at the heart of the challenge that New York has as a tech industry. We are relatively young to this work. People are always asking, “How do you hire engineers in New York?” It’s like, “You hire engineers in New York the same way you hire them anywhere else. It’s hard work.”
Paul: Or you can go to postlight.com.
Kellan: Or you can go to postlight.com and they will supply all of your needs.
Rich: That’s [email protected].
Paul: It’s a great email address! But There are other ways to solve problems besides hiring my company …
Kellan: Fair enough, that’s legit. Even if we have great engineers and obviously there’s great design talent in New York, there is this dearth of management talent. It’s a real, it’s a desert. That’s what we’re trying to solve.
Paul: OK. Does that have a name? This new cohort?
Camille: It should, but it doesn’t.
Paul: So it’s that new. So right now if I want to talk to you about that or come to an event or whatever, none of that fully exists, but it will exist soon and I get in touch with Kellan or I get in touch with Camille.
Paul: OK. Do you guys want to host an event at our space?
Paul: We’d be happy to enable that.
Kellan: Sounds good.
Paul: Awesome! Now we’ve got a plan.
Kellan: All right. Cool.
Rich: Thank you guys for coming.
Paul: This is super great.
Camille: Yeah. This was fun.
Rich: This was a lot of fun.
Paul: Camille, thank you.
Camille: Thank you for having me.
Paul: Kellan, thank you.
Kellan: My pleasure.
Paul: Well, you know, I think typically when people leave, you kind of want to trash talk them, but they were great.
Rich: They were really good.
Paul: They are very very smart, very together people.
Rich: Yeah. They actually care about the perspective they’re in and they’re not just sort of, “Ah, the hell with that. That was ridiculous, I want to do something else.”
Paul: No, they want to build a culture where people can do meaningful work.
Paul: Which I admire. Would you work with those people?
Rich: I really would.
Paul: I would too, right?
Rich: As I’m listening to both of them talk, I’m thinking to myself, “Why did I have to work with that other person?”
Paul: I know, seriously. These are really strong…these would be great CTO’s in your organization.
Rich: I know I just caused eleven people who I’ve worked with to wonder if —
Paul: A lot of Linkedin connections are going to die tonight. “Why am I suddenly down to 492 connections?” So, listen, if you like this program, and some of you do, the way to demonstrate that is not necessarily to give us any money or to do anything that would actually require any severe effort or for you to suffer in any way. I don’t want you to suffer. What I want you to do is move one finger over the trackpad or the mouse and go to iTunes and give us a rating. Four stars would be fine, five stars would actually be —
Paul: Good. Just really good. Write a little review. People are writing reviews, and what that lets the world know is that maybe they should listen to this. Maybe they should jump into the community that is Track Changes, and that gives us the will that we need to continue.
Rich: Yeah, to move forward.
Paul: If you need us for anything, not just to give us money and hire us for things, but to ask us questions, we love it. We love hearing from anyone. You can tell us what you like, you can tell us what you don’t like. Just send an email to [email protected]. That goes straight to Rich and me, and we love hearing from people.
Paul: We’ll see you next week. Rich, let’s head back to the office.
Rich: Let’s go. Back to work!