The Motley Fool sent a team of analysts down to Austin, Texas for South by Southwest Interactive this week.And in this episode ofIndustry Focus: Consumer Goods, Vincent Shen is joined by Dylan Lewis to share insights from industry favorites:Netflix(NASDAQ: NFLX)and Walt Disney(NYSE: DIS).
For the leading streaming service and global entertainment giant, product testing is always happening behind the scenes. Find out how both companies leverage various tools at their disposal to put forward the perfect customer experience.
Continue Reading Below
A full transcript follows the video.
10 stocks we like better than Walt DisneyWhen investing geniuses David and Tom Gardner have a stock tip, it can pay to listen. After all, the newsletter they have run for over a decade, Motley Fool Stock Advisor, has tripled the market.*
David and Tom just revealed what they believe are the 10 best stocks for investors to buy right now... and Walt Disney wasn't one of them! That's right -- they think these 10 stocks are even better buys.
Click here to learn about these picks!
*Stock Advisor returns as of February 6, 2017
This episode was recorded on March 14, 2017.
Vincent Shen: Welcome to Industry Focus,the podcast that divesinto a different sector of the stock market every day.I am your host Vincent Shen and it's Tuesday, March 14th. We'recontinuing with our special coverage ofSouth by Southwest interactive this week.The Motley Fool sent a team of analysts down to Austin, Texas to get a firsthand look at everything coming out of the conference. Calling in today from Austin is Dylan Lewis. How are you, Dylan?
Dylan Lewis: Doing OK. Taking a little break fromchecking things out at the convention centerto give you a call on the hotel phone.
Shen: Sure. Ouroverarching topic for today's episode reallycomes down to product testing. We will be looking at thisin the context of two very well knownconsumer companies, those beingNetflixandWalt Disney.Starting with the leading streaming service, Dylan, what did Netflix share at their talk atSouth by Southwest this year?
Lewis: The session hosted by a Netflix rep wasprobably one of the ones I was most excited about when I was looking at the schedule. I went to a session called Design Like A Scientist: A/B Testing UX at Netflix. There's a lot of letters in there, some acronyms, but the idea here was, there was a designer from Netflixbasically giving a chat about: How does Netflix make changes to the platform, keep theirexisting users happy, and then bring in new ones, as well? That'skind of the big-ticket here, and that's what they walked through.
Shen: Sure.I have heard previously aboutNetflix and their focus on data.I'm really interested to hearwhat they talked aboutat the conference. But the way I think about it sometimes, to put it in context for listeners, it'skind of an old Hollywood versus new Hollywood. If you think about the old model of how movies and TV shows are made, and how Netflix can apply a more modern, data-focused approach,it comes out very superior to the old way, where as recently as even ten years ago,you have industry insiders, people writing scripts, directors, agents, they'remaking these incredibly important decisions, often with millions of dollars in production budgets in the balance, and it's based onsomewhat unquantifiable things like star power or intuition. So, a leading actor or director may have a strong track record at the box office, but even the biggest stars have their flops.
Theclosest things I've been able to think of,in terms of a more standardized approach to creating content, it might besequels and franchises where you can count on the previous success and an established audience of an earlier film. You also have things like remakes that bank on nostalgia. And then there's trends, you often see trends at the box office, like, dystopian young adult books often get turned into movies. That was a trend recently. Thenzombie films. They will become extremely popular, but eventually they peter out after oversaturating the market. But Netflix, as we'll get to, takes a very different approach, and they're really seeing what will work in terms of the content they'll produce, and also how they present that to their customers, right?
Lewis: Yeah. They're in kind of a unique position. They control the delivery of their content. They know exactly how much time people are spending watching certain series,which ones are incredibly popular, when people start to tune out, all that stuff. So,they have the analytics on that side with their platform. Whatthis conversation really focused on was, how do they update the way that theydeliver that content in a way that's super user friendly andkeeps people satisfied?And really, this comes down to A/B testing. This is something that a lot of online businesses do. It's actually something we do a little bit at the Fool here and there. What it means is, every day, Navin, who is one of the guys who led this conversation, NavinIyengar, his team ofdesigners are testing out these little changes to Netflix'splatform. And they're doing it on a very small subsection of customers. Andbasically, they're looking to see how those users, that small section of users,interacts with the changescompared to a control group that's just getting the standard experience. And very often, what they'redoing here is trying to drive thatimproving customer satisfaction, like I talked about. They'll usually tie that to some core metric. Maybe it's log-ins per month, maybe streaming hours or customer retention, something like that. Ultimately, if what they applied to that very small subsection, the test, beats the control, the standard experience, they'll roll out that new design element, or that new functionality to the larger whole Netflix userbase.
Shen: OK. AndI'm going to ask you,did they give you any examples of how they use this data to change things that people who are onNetflix might be familiar with, that they have seen or noticed? One thing that really fascinated mewhen I was doing my research on this, the level of detail they have for everysubscriber, they know exactly what you were watching to a shocking level of detail. Time and day of the week, check. Where you're located, check. Device you're watching on, whether it's a tablet, a TV, PC, check. What yousearch for and how you rate content, check. Howyou browse the library, even, or your scroll-through, rewind, pause,all of those things are included. It'sinteresting, the idea that this iskind of real-time feedback theyreceive, because they can recommend content to a user, and after that subscriber watches that content, they'll often rate it, and voila, you know right then and there if theiralgorithms are accurate for thatspecific subscriber, because in the end,obviously, only the subscribers who reallydiscover more interesting content that they like on Netflix will want to stay with the service. This will play into loyalty and retention, if they ever want to raise their rates again, for example.
An article I found in Wired delves into this as well,how the company's really delivering it even in the imageryit uses for TV shows and moviesin its Netflix library, and how they'll do avery detailed analysis of the colors and the cover art that they use,and how that can tell the company aboutuser preferences andbehavior. But,did the speakers at the conference share details of how that really played through in terms of A/B testing?
Lewis: Yeah, they used a couple high-profile examples that I think a lot of regular users of Netflix might see and actually really like. I think one of the main ones, one of the most recognizable versions of this testing format being eventually rolled out to the larger audience, was Netflix's post-play function. This is basically the mechanism that gets you the next episode rolling as soon as you finish a show. That is what leads so many viewers into that binge-watch cycle. "Oh, the next episode is going to start in 15 seconds." That is something that they tested into because it helps keep people on the platform longer, and it boosted retention, it boosted satisfaction, people were getting more out of the service, they were watching more. I think that's a staple of the online offering right now.
Shen: OK. Yeah, Inoticed that. I don't remember exactly when that rolled in,but now, for me, especially as I've been discovering some new series,I've also been trying to push my wife to watch stuff like The OA,and I have to say it is very conducive -- as you mentioned, you play that first episode, you like it,and once it automatically goes through,you don't have to touch the remote on a Rokuspecifically at all,it can definitely push you into binging those three or four more episodes, and before you know it, Netflix is abigger and bigger part of your entertainment experience and options.
Lewis: Yeah. If you're looking for how the internal team thinks about testing, and what it looks like on the design side, and theconsiderations they have,the speaker did walk through their framework a little bit, and talked about some of the keys they focused on. One ofthe big things was this idea of,test before you invest. Do you remember a couple years ago, they totally redesigned the Netflix site? It went from being an older-tech look to a very fresh,current modern look that we see today. But,when you're talking about overhauling the design of an entire site, that'spretty daunting and it involves a ton ofdesign work. Before they went wholesale with it, what they decided to do was test a redesign mainpage. That's it. All the other pages were in the old skin. Once they saw that the newly designed page was better, and wasperforming better for the metric they were looking at,they then applied it to all of the subpages, all the show-specific things, all the subcategories, all that type of stuff. So, you see,even within the testing environment,they can get very granular before they decide to apply it broadly to the whole platform.
Shen: OK. What else do they look for, in terms ofguiding principles for them in this testing process?
Lewis: We talked about the amount of data they can collect on their users, and I think that the notion of the metric being their compass is really what underlies all ofthe work that they do. They might be looking at a problem like, "How do we get people to watch more?" Andall the design elements are going to be aimed at a specific metric, it's going to be episode starts, or hours streamed, or something like that. But the idea of having a very specific number that they're focused on, that keeps the design work organized, it keeps it fairly unified, and generally aligned toward that goal. If you want to talk about another example of some of this work beingsuccessfuland later being rolled out broadly on the platform, any time you're looking at a title on Netflix, and you put your cursor over it, there's a play button when you hover over it. That's something they tested into. Ultimately, they wantpeople to be watching a ton on the platform, and the less information they show --they show interesting graphics, and maybe the rating, and a title, and avery brief description, butpeople want to get to playing the episodes. And they realize that very early on in a control group, and then rolled it out to the rest of the platform. So, that's another example.
Shen: OK. Something that you had mentioned to me before the show that I thought was really interesting wassome of the things that Netflix has spoken of of how people say they prefer certain things,and how the company discovers that,when they observe their actions --because they have this very granular data, andobserving how people use the service --those things don't always match up, and how thecompany needs to focus on what theircustomers actually do,rather than what they say. How did thatplay out?
Lewis: Yeah,I think that's probably one of the funniest ones. We think, as consumers, we know what we want. Thatisn't always the case, though. So,they asked this survey question and said, "What'sone thing that you would like to know more aboutbefore signing up for Netflix?" And 46% of respondents said they wanted to know all the movies andTV shows that are available. You'll notice, if you go to the sign-on page, they don't make a fullcatalogue available. You can't find that in some of the on-boarding they have for new users. And why? That'ssomething the company tested into to see. It turns out, when you're getting to these sign up pages for Netflix, what really drives people to take the plunge and give it a shot is the free trial. When they offered up people a look at the catalogue,even a small part of it, they got so bogged down inhunting for specific titles, what they found worked best in terms of bringingpeople on to the system and keeping them was getting them in, and having them sign up and actually use the service. And so, that's a case where, theremight have been a little bit of a disconnect between what we thought we wanted and how to find that out, and how we actually acted and were interested in as consumers.
Shen: Wow. Ican't say that I'm too surprised. I have personally experiencedsitting in a group with my friends,trying to figure out what we're going to watch together, we're on Netflix. AndI can see, with that specific example,how you can get bogged down going through the different choices,and I can see how the company wants to get people into thatcontent viewing as quickly as possible. But for investors, turning the roles arounda little bit for the investors who are listening,this level of granularity andinsight into the customer base, it reallyallows the company to do things like pay $300 million a year forexclusive rights to Disney content,for example, or spend $4.5 million per episode for House of Cards, or $9 million per episode for their original series Sense8 as well. And they have a lot more confidence here to makethese kinds of investments, andultimately have more confidence that they will pay off. They can takeall of this information they have at their disposal, and you combine that with the estimates that Netflix is spending,content budget for 2017 is, I think, about $6 billion. Andyou realize why the company is able to grow so quickly. Its subscriber count was up 25% in 2016, hit 94 million total as of their most recent report. Any other takeaways here, Dylan, for you on this company or their process, or what you heardat the conference, before we move on?
Lewis: One thingI do think was pretty interesting,and it underscored the importance of this testing approach, was the speaker said, "Whatwe found over the years of testing is that our intuition is generally wrong." And so, these are guys that areprofessional designers, and people who workin the business of online media, andeven they don't have the greatest finger on the pulse of what people want,and the best way to deliver it to them. That is why they have thisvery rigorous testing environment setup. You may have this hunch about how people behave, but until you have the data to back it up -- or, you might see that you made one assumption that was totally unfounded, and it turns out that's not apriority for users, but,it's nice for them to have these numbers,because otherwise, they would be flying blind.
Shen: Sure. Now, we turn our attention to Walt Disney. From what you described to me before the show, Dylan, it seems like a lot of the testing the company discussed at the conference here was around experiences and their theme parks, and how they can push the envelope with new technology, potentially. Can you give us some more detail?
Lewis: Yeah,the session that I attended that was relevant toDisney was called Using AI and Machine Learning to Extend the Disney Magic. We hada couple different folks from Disney Corporate there. Andas the title implies, this was really a talk all aboutwhat Disney is doingin the space of artificial intelligence and machine learning. This isobviously a nascent field,and one that I think we're probably not going to know that we're seeing for quite some time. Butone thing that I was really struck by was theemphasis on consistency witheverything Disney does. They think of theircharacters, their stories, all these different ways thatpeople interact as touch points they have with theemotions and the characters that bring out thoseemotions, and how important it is forall of them to be consistent, in line with the character and the Disney brand. So, they'reinvesting heavily on the tech side to see what they can do, and how they might be able to continue to bring thesecharacters to life, but I think they're doing it very carefully.I think a lot of this is going to be behind closed doors for a very long time.
Shen: Sure.I have to ask you,you mentioned artificial intelligence and machine learning,I'm really curious, for me,in the context of a theme park, I'm thinking,maybe the typical person in a costume representingfamous Disney characters like Mickey Mouseget replaced, potentially, in the future,by a robot in the costume instead. Did they give you any examples, or showcase any of these things they're working on?
Lewis: They talked a little bit about what different ways AI might beIncorporated by Disney in the future. Theytalked about chat bots, or some kind of kids chat apps, maybe some quiz apps,things like that. They talked about the idea of havingDisney characters at the park powered by AIor machine learning as well. Butone thing that I think you have to keep in mind here is that emphasis on consistency, and theexperience always being so positive and so seamless. There are a lot of major challenges with AI and machine learning for as far as we've come. In the R&D lab, when they are working on this, and they have some very small groups of people interacting with whatever they're creating, they have total control. In the theme park,they have slightly less control,but I think they're still dictating adecent amount of the interactions and the experiences that their technology will run into. If you'rethinking about the consumer products side, and maybe, eventually, being able to buy something that is AI-powered off of supermarket shelves or something like that, this is a Disney-branded product, I think we're way away from that, because it's a matter of control,and understanding the environment that the technology is going to be used in.
Shen: Sure.I'm not surprised that they'retaking a more measured pace and approach to this, because a lot of different things could happen in a park with a kid,whereas if you have somebody right now who's in the custom,they can react dynamically as necessary. But,if you think now about certain toy companies,they're kind of entering this with the AI. You see Barbie dolls now that kids can talk to and have a conversation with, and the Barbie will have some pre-programmed responses, but it's learning, and connected to a cloud-likeinfrastructure that allows them to bea little bit more dynamic when interactingwith the kids that are playing with the dolls. But, what else were they able to share, in terms of some developments in the space?
Lewis: Well, they showed a couple different R&D projects that they're working on. One was an AI machine learning-fueledversion of Pascal from the movie Tangled, and another one was this project they called Jake. Unfortunately, the videos for these aren'tavailable anywhere. But I think really, what we saw withwhat they were willing to show us wasDisney studying how people interact with theserenderings of the characters, and what they do with them,and if they're able to still build that emotional connection,because I think that's what's so important for them as a company -- maintaining that, and not having any of that lost, if theyever decide to go over to the machine side.
Shen: OK. I'm really surprised, I was looking into other ways they've adopted this kind oftechnology in their theme parks, across the company. In terms of the history of the company,it was really interesting to find thatas early as the 1960s, they adoptedanimatronics at theirEnchanted Tiki Room attraction, which hassinging and dancing birds. From what I was reading, in terms of news at the time, this bird that would sit there and beckon park guestsinto the attraction was so popular, it was so groundbreaking at the time, there would becrowds that would gather outside to see this happen. Think about it, in the sixties, there was nothing like that before. Itreminds me of other things that thecompany has done,pushing the envelope a little bit, or at least, be willing to invest to push theenvelope. Think about Pixar leadingHollywood at the time with computer animated films,Disney was willing to pay over $7 billion to acquire theexpertise and computer animation,which, obviously, their films are focused on now, andintellectual property there. And also, they have things,I haven't been to a park in some time,but my brother told me recently in his trip that they have their MagicBands now. So,available to guests, this is RFIDtechnology that really makes the guest'sexperience very seamless. You can enter parks, enter hotel rooms, with this band. Just some examples of how the company isdefinitely not inexperienced in terms of pushing technology and finding ways to innovate and make experiences at the park, and with their products, that much more rich and fun to play with.
Lewis: Yeah,one of the panelists said, "We'renot a tech company,but everything we doseems to be under piles and piles of techas we continue to move along as a company." So, I think users and consumers that go to the park and things like that, they'regoing to continue to see the experience that theycurrently have at the park forquite some time. And they mighteventually be interacting with AI and machine learning at the park, and not even realize it. I thinkif Disney had its druthers,that's exactly what would happen. They areso much about capturing the magic and having you not see the strings that are pullingeverything in the background, andcultivating this really incredible experience. I think once the technology gets there, you'redefinitely going to be interacting with it quite a bit more at the theme parksin particular.
Shen: OK. Very cool. That is a wrapon our discussion for today. Just as a preview,is there anything else coming up atSouth by Southwest this week that you'll be covering on the other shows,or that you're really excited to see?
Lewis: Simon Erickson isgoing to give me a little break, and he's going to talk on the Healthcare show onWednesday, and I believe he's also going to talk withSean O'Reilly on the Energy and Industrials show on Thursday. Some very fun stuff lined up there. Simon is going to be talking health and med-tech with Kristine tomorrow, and I'm hoping to land acouple interviews, we'll see what happens. Friday's episode is still in the air. I have an interview that I booked and locked down with a self-driving car expert. So, we might wind up airing that and double-dipping onself-driving car technology. But if anything elseinteresting comes up, theremight be a change of plans. Either way, we'llpublish the episode as a bonus episode, andmake sure that all of our listeners are gettingall of the content that we put out.
Shen: Sounds good. Thank you, Dylan!
Lewis: It's a pleasure, Vince.
Shen: That wraps up our discussion. You canreach out to usand the rest of the Industry Focus crew via Twitter @MFIndustryFocus, or shoot an email to firstname.lastname@example.org. Don'tforget to check out podcasts.fool.com for more Foolishcontent.
People on the program may own companies discussed on the show, and The Motley Fool may have formal recommendations for or against stocks mentioned, so don't buy or sell anything based solely on what you hear during the program. Thanks for listening and Fool on!
Dylan Lewis owns shares of Walt Disney. Vincent Shen has no position in any stocks mentioned. The Motley Fool owns shares of and recommends Netflix and Walt Disney. The Motley Fool has a disclosure policy.