SDS PODCAST

EPISODE 349:

HUMAN-IN-THE-

LOOP ALGORITHMS

IN RETAIL

Kirill Eremenko: This is episode number 349 with Chief Algorithms

Officer at Stitch Fix, Brad Klingenberg.

Kirill Eremenko: Welcome to the SuperDataScience podcast. My name

is Kirill Eremenko, Data Science Coach and Lifestyle

Entrepreneur. And each week we bring you inspiring

people and ideas to help you build your successful

career in data science. Thanks for being here today

and now let's make the complex simple.

Kirill Eremenko: This episode is brought to you by SuperDataScience,

our online membership platform for learning data

science at any level. We've got over 2,500 video

tutorials, over 200 hours of content, and 30 plus

courses with new courses being added on average once

per month. So all of that and more, you get as part of

your membership at SuperDataScience. So don't hold

off. Sign up today at www.superdatascience.com.

Secure your membership and take your data science

skills to the next level.

Kirill Eremenko: Welcome back to the SuperDataScience podcast,

everybody, super excited to have you back here on the

show. And I've got a question for you. What do you

think Netflix, Amazon, Spotify and Uber all have in

common? Oh, that's right. They're all algorithms

companies. And right now, according to Forbes, we're

living in the golden age of algorithms. This is a time

when companies that are based on algorithms that use

algorithms, not just data science and data analytics,

but specifically algorithms, those companies are the

ones that are going to thrive. And today's guest, Brad

Klingenberg, is the Chief Algorithms Officer at Stitch

Fix.

Kirill Eremenko: So let me tell you a bit about Stitch Fix. Stitch Fix is

an online personal styling service. They were

established in 2011, so just a mere nine years ago. But

in that time, they've already accomplished so much. In

2017, they went public. As of February 2018, they

were valued at $2 billion. $2 billion. Right now, they

have over 3 million active users worldwide. And on

their algorithms team, which is headed by Brad, wait

for it, they have 125 people on that team. How cool is

that? And so in this episode, Brad walked us through

exactly what is happening at Stitch Fix, and how

they're using algorithms to their advantage. This

episode can roughly be broken down into three core

components. We'll be talking about algorithms quite a

lot, and you will hear about the sheer diversity of

applications of algorithms at Stitch Fix. It's mind-

blowing.

Kirill Eremenko: They use algorithms for everything from

personalization, to NLP to process texts, to computer

vision to process images, other models to solve the

traveling salesman problem. It's hard to imagine a

single activity in the business that is not underpinned

by an algorithm. So that's a very exciting component of

what we talked about. Another really cool thing that

you'll hear about is how Stitch Fix combines humans

and algorithms together. This is a company that is

completely destroying the myth that algorithms will

replace humans. In this company, algorithms work

side by side with humans, and you'll find out exactly

what that means, how they accomplished it, and how

people are satisfied with that process.

Kirill Eremenko: And finally, the third part is what will be interesting to

you if you're an executive or manager, you want to

lead a company at some point, it's about how to build

an algorithms-driven company. What does it take to

make sure that your business has algorithms in its

DNA? What are the trade offs, what are the

advantages, and what are the pitfalls to look out for?

In a nutshell, this is going to be an extremely

insightful episode about algorithms and algorithm-

driven companies. So without further ado, let's

welcome Brad Klingenberg, Chief Algorithms Officer at

Stitch Fix.

Kirill Eremenko: Welcome back to the Super Data Science podcast

ladies and gentlemen. Super excited to have you on

the show, and today's guest is Brad Klingenberg from

Stitch Fix, calling in from San Francisco. Brad, how

are you going today?

Brad Klingenber...: I'm doing quite well Kirill, thank you.

Kirill Eremenko: Very excited to have you on the show. It's been a while.

We originally planned this for several months ago and

then my schedule didn't work. Your schedule didn't

work. But finally we're here. We've going to be talking

about algorithms and Stitch Fix. Are you excited about

this?

Brad Klingenber...: Absolutely. It's a pleasure, and I'm glad we could align

the schedules and make it happen.

Kirill Eremenko: Yeah. Same here. And what I really appreciate, already

talking to you a little bit just before the podcast now,

you're so excited about the company you work for. You

talk about Stitch Fix, and we can feel it in the tone of

your voice. What is the one thing that you would say,

that makes you most excited about waking up in the

morning and going to work?

Brad Klingenber...: Oh, it's so hard to choose just one. So I might cheat

and say two. So Stitch Fix, just the problems that we

work on are just fascinating. And so I feel, after having

been here for more than six years now, I still learn and

get to think about new things every day, which is

exciting. And secondly, the people are great. So I think

that combination of working on interesting things,

with colleagues that you enjoy spending time with, is

pretty hard to beat, and keeps you engaged in the long

haul.

Kirill Eremenko: That's fantastic, to continue learning after six years.

Would you say your learning has shifted? Maybe at the

start was more about algorithms, and now it's about

leadership? Or is it still a combination of both?

Brad Klingenber...: I think it's a healthy combination of both. So my

journey at Stitch Fix, today I'm the Chief Algorithms

Officer, or lead our algorithms team which, which is

largely a data science and platform organization that is

focused on using data to improve our business. And so

I joined Stitch Fix about six and a half years ago. So

an individual contributor. And so certainly, it's taking

on people management and larger and larger teams.

The focus of my work has shifted. So there's definitely

a lot to learn there. But part of what's still so exciting

to be here, is we have these wonderful, difficult open-

ended problems around making our clients happier

and buying inventory. And there's really a richness of

this problem space where many of the fundamental

things we work on are just not solved. And that keeps

me engaged on a technical level as well as managerial

level.

Kirill Eremenko: That's, that's fantastic. And congratulations. This is a

first time I've heard the title Chief Algorithms Officer.

Did Stitch Fix invent this title, and what does the title

entail?

Brad Klingenber...: It's a great question. I can't make, with certainty, the

claim that it's unique. But it's certainly unusual. I

think what it reflects, as much as anything, is really

the unique role of the algorithms team in the

company. So data science is a really interesting field,

and you have different companies organized quite

differently. And very often you'll find data science or

machine learning and algorithm development is a sub-

function of engineering or other verticals at a

company. And Stitch Fix is somewhat unusual in

actually having an algorithms organization that is a

top-level department, alongside other functions of a

business. And I think this has been important

historically, and just generally represents the role of

data science at the company.

Kirill Eremenko: Yeah. It's a very cool thing that's going on now. I've

recently, well maybe a year ago, read a blog

somewhere, that before data-driven companies were

crushing it, but now it's the algorithm companies that

are crushing it. Like you look at Amazon, they have

their recommender system uplifts their revenue by

30%. You've got Google with their algorithm for

displaying ads. Uber with their matching algorithms.

And basically, all of the top companies are using

algorithms. And it's really cool to see a company in the

retail space, in the fashion space also following along

those lines. And would you say that this is something

common that your competitors are also using or is it

something a massive innovation that makes Stitch Fix

stand out?

Brad Klingenber...: I think certainly across technologies, the examples you

cited at a Google or Amazon, there are many places

where algorithms are used in core parts of the

business. That's fairly unusual, certainly in fashion

and retail. I think a lot of that has to do with, as

much, with the aim of the company, as what the data

that the companies have. And so one of the advantages

of the Stitch Fix model, which is, Stitch Fix's personal

styling service, a retailer, it helps people find what they

love. But not in the traditional sense of shopping. So,

we're not a store where you browse or filter your way

to inventory. It's really a service where a personal

stylist actually can make decisions, and we bet on

those recommendations by sending them to clients.

Brad Klingenber...: And as part of that, we close a feedback loop that's

unavailable to many retail models. So I'm sure if you

think about the last time you were trying something on

in a store, and it didn't quite fit, or you didn't quite the

style and you left behind in the dressing room,

chances are extremely good that nobody learned

anything about you or anything about the inventory.

In Stitch Fix, part of the magic of the model is our

clients are generally very excited to share feedback

with us. And so when we send things to clients, we get

to learn about what they and what they don't like.

That helps us better serve them, but also better

understand how to use inventory to make other clients

happy.

Brad Klingenber...: And so the algorithms we use at Stitch Fix, since the

top, this feedback loop that really opens a lot of

opportunities to use data, not just in a sense of

building a dashboard that is providing the insight into

something, but in a very active sense, where we can

actually use data to drive to recommendation

algorithms at the heart of the business, to make

decisions about managing inventory or marketing. And

so, I do think it's a bit of a part of a vanguard data

being used in a much more active way, in many

different businesses that I'd expect to see continue to

play out in the coming decades in other industries as

well.

Kirill Eremenko: Yeah, definitely great that you're leading the way in

that sense. You mentioned a bit about Stitch Fix,

about how you've closed the feedback loop and that

you're in the fashion and design industry. Could you

tell us a bit more about the company? So for somebody

who's never heard of Stitch Fix or never used the

service, how does it all work? How does it all play out?

Like for instance, if I want to go and buy something,

some item of clothing, could you walk us through the

customer experience? So you paint a bit of a better

picture of how the service works before we dive into

more, the algorithms and that side of things?

Brad Klingenber...: Yeah, absolutely. So Stitch Fix is a personal styling

service. The way it works from a client perspective is

you come to us and tell us a bit about your

preferences. So how much you typically to spend, how

you like your clothes to fit, and a variety of things to

help us get to better understand your sense of style.

And then with that, we'll actually pick some things to

send to you. So, it's unlike a traditional retailer or even

most e-commerce models, we're actually making

choices on the behalf of clients. So, rather than just

for yourself, Stitch Fix is actually making the choice of

what to send to you, and being a personal stylist, in

the truest sense.

Brad Klingenber...: And so of course, it's quite important to get that right.

And that's where the focus on using data to better

understand the preferences of our clients, and the

inventory that they're going to love becomes so

important. And so as a client, you'll get a shipment

that we call a fix, and have a few days to try things on

at home. Try it on, with your mirror, with other things

that you own. You keep what you love, and send back

the rest, and you are able to continuously engage over

time in that way.

Brad Klingenber...: More recently, we've also opened up some new ways of

engaging that are quite exciting. And so for example,

being able to get recommendations for an outfit that

goes with things that you've purchased with us in the

past. And so whereas we're really exploring more ways

to make it an even richer experience, but I think the

key distinction from traditional retail is really the

company acting as a personal stylist, and us making a

really literal bet on our recommendations.

Kirill Eremenko: So what kind of payment model is... Like for instance,

if I like an item, do I need to pay for it after you've

shipped it to me? Or is it a membership fee that I pay

every month? How is this monetized? What is the

revenue model for the company?

Brad Klingenber...: So the way it works is that you, as a client, would pay

$20 for effects as a styling fee. And then, if you keep

anything from your fix, that's applied toward what you

keep. And then if you keep five things, you actually get

a discount beyond that. And so you can look at the

styling fee essentially, is only something that you end

up paying if you don't keep anything. So if you keep

anything that we send you, it's applied toward that.

And then that model includes both the shipping to you

and any returns that you'd to make to the company.

Kirill Eremenko: Okay. Got you. And so that's what you mean when you

say, you close the feedback loop, that by seeing what

the client returned, you're able to adjust your

algorithm. So even if we take for instance, the best

retailer who takes care of all of their data possible, the

only data they actually see is... They don't see what

the client is trying on, but they see what the client is

purchasing. And then from there they can say, "All

right, so the clients purchased these five pairs of

jeans, or this specific pair of jeans. So we can learn

from that." But you can learn from that, and in

addition, you get to learn from what they didn't like.

Brad Klingenber...: Indeed. So that's exactly right. Learning why you

didn't like something is often just as useful as learning

why you did like something. So for example, if

something doesn't quite fit you because it's too big or

too small, that's a lot of information about how you to

wear things. But beyond the implicit feedback of what

people keep and what they choose to return, our

clients can also share much richer feedback. So we

ask structured questions about, "How do you like the

size?" Or the style of things that are sent. And even

have clients share freeform texts feedback with us.

Brad Klingenber...: So part of the model is really this combination of

algorithmic decision making, but also then, expert

human curation. So every fix that we send is actually

curated by a stylist who's a Stitch Fix employee, and

they engage with clients on a variety of unstructured

ways. So when you request a fix, you can include a

free form requests note. And when you're providing

feedback on things that you've received, likewise, you

can include freeform text feedback on what you've

liked, what you haven't, to help the stylists and also

our algorithms get to know you even better.

Kirill Eremenko: So just to clarify, that means that as a client, my

stylist is not just an algorithm, it's an algorithm and a

human being.

Brad Klingenber...: Yes, absolutely. And so when we're styling effects,

which is our term for choosing things to send to a

client, there's really this combination of

recommendation algorithms, and the tools that our

stylists use. But ultimately in every fix, every single

item that we select is picked by a stylist. And it's really

this human-in-the-loop process, and human-in-the-

loop in a very strong sense. So not just having expert

humans label things so that algorithms can learn, but

actually having an active human decision making in

our production loop.

Kirill Eremenko: I like that term, active human-in-the-loop. Much better

than just labeling stuff. That's very cool. And so I think

we'll get to that in a bit, about how humans and

machines work together at Stitch Fix. I think that's a

very important topic. But for now, I want to ask you

just to put this into perspective for people who are

listening who might even have used the service before,

but don't know the intimate details of the company.

Two questions. How many people do you have on your

team? And also, how many clients does Stitch Fix

serve?

Brad Klingenber...: So the Algorithms team at Stitch Fix, which is the

team that I lead as Chief Algorithms Officer includes a

data scientists and platform engineers. And we have,

today, over 125 people on the team. And so, a very

large organization, or company size. [inaudible

00:18:01] a couple things. I think, both the incredibly

important role that the data science plays at the

company, and the breadth of applications. And so,

happy to chat about that more. And then, to the

second part of your question, we have over three

million active clients. So [crosstalk 00:18:22] a large

population as well.

Kirill Eremenko: That's a lot of clients. You have a lot of data. I love this

about algorithm-driven companies, that the better the

service, the more clients you'll have. The more clients

you'll have, the more data, meaning the better the

service will become. It's a self-fulfilling prophecy.

Brad Klingenber...: Oh, absolutely. And I can certainly remember six years

ago when I just joined the company, the volume of

shipments we were making was much smaller. And

that has a real impact on the amount of time it takes

to run experiments and the amount of signal that you

have to work with. And today, we're able to run

experiments, much more quickly than I'd ever dared

hope, years ago.

Kirill Eremenko: That's crazy. When you joined the company, how big

was the team then?

Brad Klingenber...: So, the data science team was led by Eric Colson, my

predecessor in the Chief Algorithms Officer role. And I

joined as one of the first data scientists. And so I think

Eric and maybe three other people, depending on how

you count, something like that.

Kirill Eremenko: Wow, that's crazy. Okay. Very interesting. So that's the

scale of the company. Massive company. 3 million

active users. Over 125 people in the Algorithms team.

So let's dive into a little bit. Let's dissect this, the types

of algorithms you use. I think this could be a very cool

case study for somebody not necessarily even just in

the retail space, but who wants to see how AI and

algorithm can drive a company, and it can be at the

foundation of a business. How it's already possible,

even at the start of the 2020s. Oh, well, it's been

possible even in the previous decade you've been doing

for a while.

Kirill Eremenko: So the first question I would have is, there's two types

of filtering that are quite well known. Collaborative

filtering versus content-based filtering. In your

algorithms tour, which by the way, I recommend for

everybody listening to this, I will link to it in the show

notes. It's algorithms-tour.stitchfix.com amazing.

Completely blew my mind. I loved reading through it.

Such a volume of information and so diverse. It really

explains how this whole thing runs.

Kirill Eremenko: So I wanted to talk a little bit about some points in

this algorithms tour. So collaborative versus content-

based filtering. In the tour, it says that you use

collaborative filtering plus mixed effects modeling.

Could you go into a bit of detail on that? Why and how

would you use collaborative filtering plus mixed effects

modeling, and what are those things?

Brad Klingenber...: Yeah, absolutely. I think it's helpful, just as general

context, to think about the types of data that we have,

when we're thinking about what to send to people. And

broadly speaking, I like to think of there being three

categories. So there are things that clients tell us

about themselves. And that could be through the

initial onboarding survey where clients could tell us

the sizes they to wear and their budget, preferences. It

can also be things like, our clients are able to rate

images in our iOS app, and on the web, that help us

understand their style preferences. But generally

things we know about our clients.

Brad Klingenber...: On the other ends, Stitch Fix as a retailer, we buy and

hold inventory. And so as part of creating or buying

that inventory, we have a lot of information about it.

So that could include structured measurements, or

even subjective ideas about when you might want to

wear this, or different occasions it might be suited for.

And then finally, the third category is really feedback.

So as we send things to clients, learning about what

they like, in a way that helps us better serve them, and

helps us get to know them over time. But then also

helps us do a better job managing our inventory in a

way that creates better client outcomes.

Brad Klingenber...: And so, in general, in thinking about this

recommendation problem, you have those three

categories to draw from. And so, as you mentioned, a

content-based recommendations is something that

might be very focused on the things we know explicitly

about our clients, and things are know explicitly about

our inventory. And we know that there's quite a lot to

do there. And you mentioned the mixed effects model,

highlighted in the algorithms tour, is one example of a

type of algorithm, not unlike other applications where

you use things matrix factorizations where you're

essentially trying to learn either through a transaction

history or through feedback that people have provided,

representations of clients and inventory that are useful

for making future predictions. So not just something

that's an explicit attribute given by the client, or that

we know about our inventory.

Kirill Eremenko: So for instance, let's compare it to a very well known

company that has an algorithm in its base, Netflix,

right? Netflix is the, probably... It is the most valued

company in terms of movie production right now. And

I think it's actually been labeled the most valued

membership platform in the world. Anyway, the point

is, they use filtering, in terms of a content filtering.

They look at content-based filtering. They look at, "All

right, what movies have you watched, or what shows

have you watched? What are the shows similar to

those shows to recommend to you?" And then

collaborative filtering or... I don't know what they use.

It's proprietary information, but they could use

content-based filtering, plus collaborative filtering.

They could look at, "Okay, you've watched these

shows, and then this other person is similar to you in

these certain ways. So they've watched those similar

shows. What else have they watched? Now we can

recommend that too."

Kirill Eremenko: Whereas, what you can do is, you can do both those

things, but in addition to that, you can take it to the

next level. You actually have feedback. Your users tell

you implicitly, feedback, saying that by returning

items, you understand they don't those things. Plus, in

addition, they might fill in some surveys, or provide

your textual information. Maybe get on a call with your

stylist. Basically... Or explain attributes about

themselves. Basically, give you additional contextual

information that other companies such as Netflix

wouldn't have. And you're able to add that in. And

that's what is called the mixed effects modeling. Am I

on the right track here?

Brad Klingenber...: So the mixed effects modeling refers to a very

particular type of statistical model that it is useful, for

example, in learning empirically, that a... For example,

a particular blouse does better with clients of a certain

style preference than another. And so it's an example

of an algorithm that can make those kinds of empirical

inferences. One family of algorithm that we've used to

some success here is something called the

factorization machine, which is something closely

related to matrix factorization, but it makes it easy to

incorporate the types of explicit attributes that you've

mentioned.

Brad Klingenber...: And maybe to add, I think one, actually interesting

feature of the Stitch Fix recommendation problem that

makes it different from recommending movies in

Netflix, or music at Pandora or Spotify is, again, that

we have this human stylist in the loop. And so in

addition to learning through the choices our clients

make, and what to keep and what to return, and how

they rate things, we get to learn through the way that

stylists interact with our recommendations as well. So

being able to learn from what a stylist chose to send,

but what they didn't choose to send, that was

recommended by an algorithm. And so there's actually

this interesting dual feedback character to the

problem.

Kirill Eremenko: So basically, you're using pins or images that the

clients liked, in order to get additional insights, which

they cannot explicitly tell you, and add that into the

model. Tell us a bit about that. How does that work?

Brad Klingenber...: Yeah, absolutely. So one of the best examples of this

might be our style shuffle feature in our mobile app

and on the web, which lots of people rate different

images. And through those ratings, what we're eliciting

from clients is just their reaction to something, and

whether they like the style. And through looking at all

the many things that they share with us, then we're

able to derive some representations of their

preferences that are useful on many different parts of

the company.

Kirill Eremenko: Got you. And let's go into a little bit of technical details

here. I found a very cool example of how you deal with

this visual information. Because there's lots of ways

you could transfer visual information into machine

information. How does this whole system work, where

you create a vector for a specific style, or a specific

item even? Like you have vectors representing different

items. And then based on the cosine similarity,

basically the angle between the vectors, you can tell if

two items are similar or not. Is this a common

approach to encoding images for further comparison

and filtering? Or are there some innovations that you

can share with us, that you are able to create along

the way?

Brad Klingenber...: Yeah, it's a great question. I think... So, certainly,

recent years have brought a lot of incredible

improvements in computer vision generally. One of the

applications that we were just chatting about though,

it combines not just trying to process an image, but

really understand how a client reacted to it. And their

response to it. And so, one interesting aspect of our

business is because we have stylists who largely

interact with our inventory through software tools that

allow them to make accurate recommendations, is we

have great image assets of essentially all the inventory

that we carry. And by showing that to clients and

getting the reactions to it on a tool Style Shuffle, we're

able to learn an embedding or a vector representation

of the client preferences for that item, that are really

useful. And even simple applications as you noted by

looking at cosine similarities. And it's really the

application of those embeddings throughout the

business, that are really a differentiated application of

that data.

Brad Klingenber...: So for example, by playing Style Shuffle, we get to

learn about your visual style preferences, in a way that

leads to the really material improvements in the

recommendations that we make, and therefore the

outcomes that the clients have, how happy they are

with their fixes. And so it's really a very useful tool

that shows up in a lot of places, including helping to

pick a stylist for you in the first place. So helping to

choose from amongst the stylist, the good style you're

fixed, to find the ones that will be most successful.

Kirill Eremenko: So that's also another algorithm to not just pick your

clothes but pick a stylist in the first place. Right?

Brad Klingenber...: Yes, absolutely.

Kirill Eremenko: Wow. And what are the criteria there?

Brad Klingenber...: Well, there's a lot of things. I think part of the value

proposition for many clients, it's actually creating a

longterm relationship with a stylist who gets to know

you over time. And that free form interaction that I

mentioned earlier, being able to request things to your

stylist, and provide feedback, create [inaudible

00:30:06], in many cases, a real relationship. And so,

for many clients, they're happy to stick with a stylist

who's gotten to know them and really knows what they

love, and brings that human touch. Of course, for

clients who are getting their first fix, or looking for a

new stylist, there's a question of, "Well how do we find

that the stylist who you're going to like the most?"

Brad Klingenber...: And one of the important signals that goes into this is

really understanding the strengths and weaknesses

different stylists have, with respect to different styles.

So, we all have our own preferences, and you might be

better or worse at styling different types of clients. And

so we try, again using this idea of a vector

representation, or embedding representation of styles,

something we call [lasing 00:30:56] Style, truly use

that to understand the types of clients that stylists are

likely to be successful with.

Kirill Eremenko: I thought for a second, you said a vector

representations of stylists. That would be funny.

Brad Klingenber...: Yeah. At least the preferences that they share when

picking things for clients.

Kirill Eremenko: Yeah. Got you. Okay. All right. So you have algorithms

to find the right clothes. You have algorithms to match

customers to stylists. You probably have NLP to

process text, right?

Brad Klingenber...: Yes. NLP is an interesting thing for us too, because

again, the human in the loop nature of the problem,

it's quite difficult, for example, for even with cutting

edge NLP, to beat the stylist capacity to see a note

from a client that might be something like, "Hey, I'm

going on a vacation. Send me something for the

beach." To really understand that intent, it's hard to

beat the human capacity. At the same time, we can do

a lot of things with that text, algorithmically. And so

we work on, I think, learning from texts, both,

algorithmically, but then also through the role that our

stylists play in the process.

Kirill Eremenko: Yeah. And you can pick out some keywords, and help

stylists to sift through the text faster, I guess. I read

that you also have algorithms to even optimize how

your pickers go through your warehouse to pick up the

items that need to be sent. Is that true?

Brad Klingenber...: It is true. So we have a network of warehouses across

the country, that we use to fulfill our shipments to

clients. And within the walls of the warehouse, there's

a lot of opportunities for optimization, and again,

using the data we have, to help make things more

efficient. So we have a team focused on what we call

operations algorithms that among other things, thinks

about how to optimize the operation of our

warehouses.

Kirill Eremenko: Wow. That is very impressive. So basically you use

algorithms pretty much for everything. Is there

anything in the business you don't use algorithms for?

Brad Klingenber...: There might be a couple of small pockets here and

there, but I think actually one of the interesting stories

of the algorithm that we make at Stitch Fix is just the

breadth of engagement it has with the business. So

there's a lot of things we haven't even mentioned yet.

And actually, the fact that we have 125 people on the

team, more than anything, reflects the number of

different problems that we're working on. So if you drill

down into any particular problem, like our core

recommendation algorithms, you'll find there's less

than 10 people working on those problems. And so the

size of the team, more than anything, reflects that it's

really an enormous breadth of applications across the

business that the team is working on.

Kirill Eremenko: Yeah, it's incredible. How do you even keep your mind

on top of all these things as Chief Algorithms Officer?

To me, it sounds impossible to deal with all these

different algorithms in different parts of the business.

And then just keep up to date with what's going on.

Brad Klingenber...: I think there's couple of things that help with that. So

one is, I get to spend a lot of time with the team and to

understand the problems aren't working on. And it

certainly is a lot, and I certainly can't keep it all in my

short term memory all the time. I think beyond that

though, another special feature of the team is really

our strong culture of bottom-up innovation. And so the

development of better algorithms, or better ways to use

data to run the business doesn't rely on me having a

great idea, and then propagating that down the

organization to be implemented. But really, rather the

opposite. So some people who are closest to working

on the business problem really seeing opportunities,

exploring new models, and bringing that to life. And

so, my role of leader of the team is less somebody

directing things and a fine grain from above, and more

a like a gardener just trying to make sure conditions

are good so that all sorts of different things can grow.

Kirill Eremenko: I love that analogy. Analogy of a leader as a gardener.

That's perfect. Can you give us an example of a recent

bottom-up innovation that you think pop to mind?

Brad Klingenber...: Yeah, absolutely. I think many of the most significant

improvements we've made over the years have arisen

in this way. And I think even the Style Shuffle rating

experience that I mentioned, is generating such useful

data, was actually born out of just as a side project of

one of the data scientists on the team who decided to

try something, and prototype something. And it really

got traction, has become a real product here. But

that's something that is a great example of an

innovation that was not asked for in any executive

way.

Kirill Eremenko: Fantastic. I think this is a good time to make a small

plug for your hiring efforts with such a grand team,

and growing so fast. To me, it sounds like an amazing

place to work, especially when you can drive

innovation bottom-up, do experiments, choose what

area of business you want to work on, choose the type

of work, whether you want to do NLP, you want to do

traveling salesman problems, you want to do computer

vision. Pretty much all AI is there. You mentioned at

the start, before we started the podcast, you're hiring.

Can you tell us a bit about more about that? If

somebody is interested, what kind of roles you're

hiring for, and where can they apply?

Brad Klingenber...: Yeah, absolutely. So I would say in my time here, we

have been in a mere permanent state of hiring, and

that certainly continues to this day. The company and

team continue to grow. We seek people from a variety

of different backgrounds. And actually one of the really

fun things of working at Stitch Fix is just the variety of

backgrounds and skills that people bring. And this is

true within Algorithms, but in the broader company as

well. And so we have a team that has enormous variety

of different backgrounds, from the computer science

and statistics, to the sciences like physics, social

sciences, psychology, epidemiology. We really run in

the gamut. And so, we draw from a lot of different

backgrounds and find a lot of value in having that

diversity of training.

Brad Klingenber...: And in general, have a fairly strong preference for

people who are able to be generalists. And really, not

be too hyper specialized into any particular subset of

data science. And a lot of this has to do with the

business problems that we solve, and asking a very,

very open-ended question like, how should we manage

our inventory? How could we do a better job buying

things that our clients are going to like? Which are

just very open-ended problems that are amenable to

techniques from a variety of different fields.

Kirill Eremenko: Got you. And I'm looking at your careers page. You've

got algorithms platform engineer, data platform

engineer, data scientist, data acquisition algorithm,

data science and client recommendation, data science

and merchandise. Very, very cool. Most of these roles

are in San Francisco. Is your whole team based in San

Francisco?

Brad Klingenber...: It is, yes. So the algorithms team is in our

headquarters here in downtown San Francisco.

Kirill Eremenko: Okay, fantastic. All right. So if anyone's looking for a

cool job, this is your opportunity. Check out

stitchfix.com/careers/jobs. Okay. So the other thing I

wanted to ask you is, it's very evident that, and I think

I read this somewhere, that algorithms are a part of

the business model at Stitch Fix. Just by the way you

do business, this is a just like a bonus, not just like a

cherry on top of the cake. This is the cake of Stitch

Fix. What can other companies learn that? We've got

about 10% of our audience, executives and leaders

and business owners. What can other companies learn

from that? What takeaways would you recommend, or

you could share from having the algorithms as a part

of the business model?

Brad Klingenber...: So I think for Stitch Fix, for a very long time, and

much credit to Katrina Lake, our founder and CEO, for

foreseeing the opportunity. Data and algorithms have

been a key part of our differentiation strategy. And

really, to be able to really predict what people are going

to like, and to be able to give it to them in a fun and

convenient way. And going back to the framing of

making a bet on the recommendations, that's just

really important that we're good at winning that bet.

As a retailer, our strategy is not to carry the largest

selection of things, or to carry things at the very lowest

price, or to deliver them faster than other retailers, but

to find things that you love. And so it was really that

personalization, and ability to understand and learn

from you, is at the core of the business. And so, I think

to your question, we really have data and data science

at the heart of the business, and it's part of the core of

the strategy. It's not a second order optimization. But

really, a first order existential strategy for the

business.

Kirill Eremenko: Got you. And so, does that make it easier for the

business to grow and to understand its users? What's

the main advantage of having that data science as the

core?

Brad Klingenber...: I think in businesses where you first have the data, so

either the feedback loops exist, or there are ways to

learn from your customers or other stakeholders, you

are able to really take full advantage of everything that

you know. And even beyond using the algorithms to

make those decisions, having a culture that is very

data-oriented and empirical helps to make good

decisions about how to think about uncertainty, how

to think about measurement. And it really, in general,

I would say, just brings a more scientific approach to

driving a business forward. And so where the business

model supports it, I think, making investments in data

problems that are really of first order importance to

the businesses is the right way to go.

Kirill Eremenko: Got you. So increases certainty and predictability of

your business. What is the main challenge of having

algorithms as part of the business model, or having

data science at the core? We've seen the advantages.

They're very clear. It's obvious how now, you've grown

to this level, how much more streamlined things are,

and how different things are to a competitor that

wouldn't have that in a business structure. But what

would you say is the one or two biggest challenges

you've had along the way?

Brad Klingenber...: So we mentioned that we're hiring. I think data science

is just in a wonderful period of rapid growth. And

you're finding applications across many different

industries, which means that as a data scientist,

everybody has many different choices of what to work

on, and where to work. And so, one thing we're always

working on is bringing very talented people into the

company. And that's certainly a big part of what I

think about. I think another challenge of using data

and algorithms in this more active sense that we

discussed earlier, is trying to figure out what is the

best way to combine human judgment in algorithm

decision making? And when do you bet on an

algorithm, and when do you bet on human judgment?

And there's no fixed optimal answer to that.

Brad Klingenber...: And I think we're on something of a journey, of really

understanding what is that optimal boundary between

your algorithm making or human decision making?

And what is really, the right way for humans and

machines to work together in a way that enables folks

at such Stitch Fix to do things that they wouldn't

otherwise be able to do?

Kirill Eremenko: Let's talk a little more about that. This is a really cool

topic, which I identified before I started the podcast.

One of the interesting themes at Stitch Fix, the

humans plus machines. And if we think about it,

companies like... Again, taking the case of Netflix, I

imagine that all of that, the whole recommendation

process can be done entirely by a machine. And if we

take Amazon, especially there, with the volume of data

that's coming in, and number of transactions all the

time, again, that's most likely being done fully by

machines. So what is your view on the situation of

Stitch Fix? Do you think that the service that you

provide could ever be done entirely by machines, or

are humans an integral part of this whole setup in this

whole service that you have created?

Brad Klingenber...: So this is a question that I'm inevitably asked, anytime

I speak at public events, or give interviews. I think my

view is that the service would not be nearly as good

without the role of the human stylist in helping to pick

things for our clients. Nor is the work that we do on

building algorithms. It doesn't have the aim of actually

automating away the human component. I think the

way that we think about things is really how to

combine humans and machines in a way that that's

better than either alone. And so, the goal is really not

just automate things away, but to equip stylists with

capabilities they wouldn't otherwise have.

Brad Klingenber...: And I think, we talked about earlier, the possibilities

for interacting with clients where you can exchange

free form text, you can exchange image data, those are

all things you can work on algorithmically. And there's

really exciting advances, especially in recent years. But

it is quite difficult, still, to match the human part for

many of these things. And I think really, the human in

the loop is a really integral part of our strategy for

solving these problems. And I would expect it to stay

that way for a very long time.

Kirill Eremenko: Fantastic. And yeah, active human loop, as we

discussed at the start, very cool. How do your

stylists... What do your stylists say about this

experience for them? How do they find it, working with

artificial intelligence hand-in-hand?

Brad Klingenber...: It's certainly a core part of their experience. And I

think you can really, in some sense, think of the

process of picking items to send to clients as being

something of a dance between algorithms and stylists.

And so, we learn a lot from stylists. And so as I

mentioned, the feedback loops we use to improve our

algorithms are not just how clients respond to things,

and how they rate them, which of course is very

important, but also, the decisions our stylists are

making. And if we make a change to an algorithm,

does that make stylists have to search more for the

things that they want to send clients? Or does it make

them make decisions more easily? And really, taking a

view of styling as a combined system that involves

both algorithmic decision making and human decision

making.

Brad Klingenber...: And so we pay a lot of attention to stylists, or

interacting with algorithms. In both, quantitative ways,

so the stylists can tell us about the recommendations

that they're seeing, also in qualitative ways. So just

spending time with stylists and the leaders of the

styling organization to really understand their

experience with the tools, but also opportunities we

have to always be making things better.

Kirill Eremenko: Got you. And you mentioned that like three million

people, massive market, a huge market presence. We

spoke briefly before the podcast, that you're

expanding, or you've expanded to the UK. Tell us a bit

about that. Is that your first international expansion?

And what does it look like now, in terms of growing

your global presence?

Brad Klingenber...: It is. So, yeah. We were quite excited to expand into

the UK, and set up a business center. So that's our

first international business. The past few years have

been a story of expansion in the US market. So

starting as a styling service just for women and then

expanding to men's and kids and even different size

profiles, so were petite and plus size within women's,

for example. And so, international expansion is

similar, and a natural opportunity that we have. And

yeah, we're excited about growing our UK business.

Kirill Eremenko: Fantastic. And what does that mean for the algorithms

team? Are you going to stay in San Francisco? Or are

you going to start hiring in Europe as well?

Brad Klingenber...: So for now, everybody's still in San Francisco though

we do spend quite a bit of time with our colleagues in

London, both virtually and also by visits to London. I

always love to go when I get a chance. And yeah, it's

an exciting problem to work on. It's a different it's a

different market of clients that we have. Largely

different inventory. And when we launched of course,

there was a bit of a cold start problem. You have all

new inventory, all new clients, and starting a new

business. And of course, we have the benefit of

everything we know about our US business and all the

algorithmic approaches that work there. But it's really,

I think, as time goes on, the feedback starts to

accumulate and you can... There's a flywheel that that

kicks in, that helps drive things algorithmically.

Kirill Eremenko: That's very cool. And if you do, at some point, expand

your algorithms team to Europe as well, then you'll

have access to a whole new talent pool European

professionals and data center.

Brad Klingenber...: Absolutely. Yeah. I think that is a very compelling

consideration. And then thinking about hiring folks, as

they've very large pools of talent that exists. Of course,

not just in US but Europe as well.

Kirill Eremenko: Fantastic. And what does the future look for

personalization, using artificial intelligence? Because

that's predominantly the core part of the algorithm, or

output of the work with algorithms is personalization

and making it very user-specific, all these

recommendations. And that doesn't necessarily have

to be just applied in the space of fashion, design,

tailored clothing. It can be applied in lots of different

industries. From your experience, extrapolating just,

in general, onto personalization, what do you think the

future looks in terms of AI developments and data

science playing a huge part in this space?

Brad Klingenber...: So I think from a consumer perspective, in retail, but

in many other consumer services or products, we're in

an era of just overwhelming choice. And you see this

with retailers competing to carry the most broad

inventories. And Stitch Fix, I think, is a compelling

counter position to that. The service is really a

personalization and making it easy to find things that

you love. Not to show you 1,000 pairs of jeans that you

could choose from, but actually just to send you the

pair that fits just right, and that you love. And I think,

as you know, the value proposition there, while quite

well suited to fashion and apparel, need to be limited

there. And so I think, really, any industry where

there's this overwhelming burden of choice and

selection and having to do a lot of work to find things

that you love, is an opportunity for personalization,

which of course, there's a very successful one driven

by data to actually help people find things that they

love and relieve them of the burden of doing the

curation themselves.

Kirill Eremenko: I love it. I love that getting rid of that whole choice

paralysis problem. I think I've experienced that. You go

in. There's so many things to choose from. Before, and

I'll tell you, in my life, like even 20 or 15 years ago, you

go into a shop and there's only one thing that suits

your measurements, or what you wanted. You buy it,

then you're happy. Okay, it almost fit what I wanted.

But now you go into shop, there's at least 10 things

that would suit you, and they're all different colors,

shapes, sizes and so on. They're all perfect for you.

And then you buy one of them, and you walk out

feeling miserable because you are thinking of the

missed opportunity of buying all these other things,

and how much better they would have been. So it's

like you're bring us back to that feeling of satisfaction

from your purchases, because you're taking the choice

paralysis out of the equation.

Brad Klingenber...: Absolutely.

Kirill Eremenko: Yeah. It sounds very exciting. But then I've got an

interesting, more philosophical question then. Do you

think that... Is your work on algorithms, or these

kinds of personalization algorithms, will, or have the

potential of driving the world to a dystopian future

where we no longer can make choices for ourselves?

All we do is, "I'll wait for an algorithm to make a

choice. What are we going to eat? What are we going to

drink? What movie are we going to watch? Where are

we going to go for holiday? What are we going to

wear?" Basically, in that sense it's like taking the

human, not just out of the workforce, but rather

taking the human out of their life. And it sounds a

little bit scary, if you think about it that way. Maybe a

bit farfetched, but what are your thoughts on that?

Brad Klingenber...: Yeah, I wouldn't worry too much about that quite yet. I

think the general trends in so many settings is, again,

for this ever increasing selection, and just broader and

broader choices for people to make. And I think there's

certainly both risk and opportunity in narrowing what

you're exposed to, too dramatically. But I think the

prospect of that becoming so prevalent that it has

dystopian implications is probably pretty remote, in

my opinion.

Kirill Eremenko: Probably in the meantime, it's really great, what you're

doing. And another consideration is that it helps

reduce waste, right? When there's so much on offer,

things that don't get purchased and they go out of

fashion, they get thrown away. But when you can

personalize to the extent where you know your

customers really well, you can reduce the inventory

size. And that means the less waste, less carbon

emissions, less impact on the environment, things like

that. So even from that standpoint, this is the way to

go, I think.

Brad Klingenber...: Yeah. It's a wonderful aspect of this model, where

we're able to run a very efficient inventory, largely

because of how much we know about our clients. In

addition to this, the general structure of the model.

And something we haven't talked about too much

today is there's quite a lot of effort in the algorithms

team, in thinking about how to just make better

decisions about what to buy, and how to manage

inventory. And you can think about it in some sense

as this giant portfolio optimization problem. How do

we build an assortment of inventory that's going to

delight our millions of clients, and do so efficiently?

And the power of personalization here is really to

enable us to do a really good job, buying things we

know our clients are going to like, which, to your

point, in addition to being more efficient from a

business perspective, it's also more efficient and less

wasteful than other models.

Kirill Eremenko: Totally, totally. Well, it's been really cool dissecting all

of this and diving into some of the technical aspects

and also the business implications, ethical

implications. Really, really cool. But we're slowly

coming to the end of the podcast, and Brad I wanted to

ask you, do you have any final parting thoughts for

data scientists that are listening to this, who want to

progress their career, or want to maybe get into the

space of data science, who are by now, I hope, are very

excited about personalization and what AI can do in

this space? Any thoughts or wishes for them?

Brad Klingenber...: I think it's a tremendously exciting time to be in a

career in data science. And one I... Stitch Fix, and our

models are extremely interesting. But I think, even in a

larger sense, I think we're part of a transformation that

will play out over a lot of different industries as we

really bring data and algorithms and just scientific

approaches in general, to running and building

businesses. And so I think the opportunities will only

increase, and keep an open eye out for hard and

interesting problems to work on.

Kirill Eremenko: Fantastic. Fantastic. Thank you so much Brad for

coming to the show. Very exciting conversation. I really

enjoyed chatting with you. And before I let you go, can

you give us some places, online, where people can

follow you, connect with you, maybe follow Stitch Fix?

What are the best places to get more information and

stay in this loop?

Brad Klingenber...: Absolutely. I think more than anything, I'd recommend

checking out our blog, multi-threaded, which is a

technical blog of the company that includes the

algorithms tour that you mentioned, but then also a

pretty regular cadence of really interesting blog posts

about the way that we use algorithms at Stitch Fix.

Kirill Eremenko: Got you. So that's multithreaded.stitchfix.com. Very

exciting. And one more question I had for you before

we finish up, what is a book you can recommend to

our listeners for them to further their careers or just in

general their lives?

Brad Klingenber...: Yeah, great question. I think one really interesting area

of machine learning research these days is causal

inference. And this is something that there's really

active research there. And I think, for folks who are

interested in learning a bit about that, but might not

have encountered it in their training or experience,

there's a pretty general audience book called Mostly

Harmless Econometrics, that I think is a fun

introduction to the subject. And a little more broadly

for life, I've really enjoyed recently reading a book

called Origins by Lewis Dartnell, and which was about

the role of geology in shaping human history. And I

think it's a fun story of how we got to where we are

today.

Kirill Eremenko: Very cool. Have you read Sapiens by Yuval Noah

Harari?

Brad Klingenber...: I have. And it's in the same genre of large tellings of

the human story over thousands of years.

Kirill Eremenko: Yeah, it's really cool. And there's another one, Silk

Roads, Peter Frankopan. That's another really good

one.

Brad Klingenber...: Yup. One of my favorite genres.

Kirill Eremenko: Yeah. Very cool. Okay, thank you. So the

recommendations are Mostly Harmless Econometrics,

and Origins. On that note, Brad, once again, thank

you so much for coming on the show, spending some

of your time with us. I'm sure this is going to be very

inspiring to lots of data scientists out there.

Brad Klingenber...: Thank you. It's a pleasure.

Kirill Eremenko: So there you have it, everybody. I appreciate you being

on this show with us today, and hearing from Brad

Klingenberg from Stitch Fix. I hope you enjoyed this

episode as much as I did, and got value out of those

amazing insights that Brad was sharing with us. My

personal favorite part about our conversation today

was the sheer volume and diversity of applications of

algorithms at Stitch Fix. This is really cool to see how

a company is using algorithms in absolutely

everything they do, and at the same time, they are not

edging out humans. They are finding ways for

algorithms to coexist with humans and they're getting

the advantage of having both working hand-in-hand.

Kirill Eremenko: I'm really impressed at how Stitch Fix is, in my view,

leading the path of becoming an algorithms-driven

company. They're not just using it for one application

to set themselves aside from the competition. They're

using it across the board. They actually have an

algorithms team and the Chief Algorithms Officer. How

cool is that? Totally. Totally love this conversation, and

I'm sure you have your personal favorite takeaway

from today as well.

Kirill Eremenko: And as usual, you can find the show notes for this

episode at superdatascience.com/349. That's

superdatascience.com/349. There, you'll find the

transcript for this episode, as well as any materials we

mentioned, including the careers page for Stitch Fix. I

highly recommend checking out this company,

especially if you're in San Francisco. This could be a

fantastic place and sounds like a fantastic place to

work at. They're hiring. So check it out. It's data

science of all kinds in stores, whatever you like. I'm

sure you'll be able to find it at Stitch Fix.

Kirill Eremenko: Also in the show notes, you will find a link to the

algorithms tour that I mentioned. Also very cool read,

something to browse through. It will probably take you

more than a day to get through all of it. It's very well-

written, highly animated, very insightful. Highly

recommend checking it out. And finally, you'll also find

the blog of Stitch Fix that Brad mentioned. That could

be a very useful read for you as well. So lots of

materials there. The link is

superdatascience.com/349.

Kirill Eremenko: And one ask I have for you today, if you enjoyed this

episode, share it with somebody. Share it with

somebody who is interested in algorithms, who is

maybe in the space of personalization, or online

services, so they can get an idea of what's happening

in this space. If you know any data scientists, machine

learning engineers who are passionate about

algorithms, this could help them build their career

structure. And also if you know any business owners

who might be looking to build an algorithms-driven, or

at least data-driven company, Stitch Fix is a great

testament, a great example to strive for. And I'm sure

they would find it useful. Very easy to share. Just send

them the link, superdatascience.com/349. And on that

note, we're going to wrap up for today, and I look

forward to seeing you back here next time. Until then,

happy analyzing.