At the Insights Association’s Corporate Researchers Conference in Orlando on October 10, Siddhi Sundar of Netflix and Mark Doherty of Chadwick Martin Bailey discussed developing a segmentation for product design.

Siddhi began by encouraging attendees to take the time to read the Netflix Culture Deck. “Netflix has been disruptive and transformative, and we take our culture super-seriously,” she said. (All quotes are paraphrases.) “It’s been the secret sauce to evolving from a DVD-by-mail firm to a streaming company to an entertainment studio. Netflix is distinct, disruptive, engaged, growing – and accomplishes this through emphasizing freedom and responsibility. As a member of the insights team, you’re given a ton of freedom to drive the insights forward, with no processes for approval, no red tape, just produce great consumer insights. Answer the research questions, with whatever research partners we choose, without anyone really telling us if this is okay or not. Insights at Netflix is not a support role: we are equal partners, with autonomy. The consumer insights team is really unique; we create research programs that best meet the needs of the business. It’s really empowering: you define the research that you think is most important for the business and execute. As a result, we are answering questions in a much broader scale. We’re really pushing our answering of business related and focused questions. We’re not just providing a narrow sense about usability, for instance.”

“The Netflix-CMB partnership covers design, fielding, analytics, and reporting,” Siddhi said. “CMB is the rare partner that over 3-4 years went from smaller projects to every major strategic project.”

Mark chimed in: “I’ve never run across a company that has that much freedom and that much accountability. The autonomy is remarkable. What you hear about culture is true. And as a result, we really have to bring our A game every single time. We ended up in so many long productive conversations about our models, with their wicked smart (as we say in Boston) PhDs. We answer their questions about the models, their usage, and how it connects with end business use. We really to get to know each other.”

“Netflix has smart researchers who manage projects from beginning to end: general contractors who subcontract out sampling, programming, modeling. It’s different from how we typically work. But we handle a range of work, including discrete choice, MaxDiff, segmentations, and our graphic design staff provides visualization, infographics, posters - even for projects we weren’t otherwise involved with. Netscape is a prototype for self-sufficient clients who need just a bit of help.”

Siddhi said, “Netflix’s challenge is managing and innovating the user experience in nearly 200 countries, with one consumer product that is both global and personalized. There are a lot of variables: watching at home or on the go, watching together or alone, watching on different devices (TV in the living room, cell phone on a train, laptop in the bedroom). We want to successfully design for every person globally and consistently without limiting the user interface. Our question was: Who we are designing for?”

To answer that, and to answer most questions, Netflix relies on “uncovering needs via triangulation.” Netflix does extensive A/B tests, collects behavioral data, conducts survey research, and does qualitative research. “If Reed Hastings and a new employee both come up with a new idea, the winner is the winner of the A/B test. It’s democratizing.” For qualitative work, “The research team is constantly in the field. We will go to customers’ homes, shadow them for a day, understand the why in ways impossible through data alone.”

Netflix wanted a segmentation methodology for product design segmentation, not for marketing. To build the segmentation, the team triangulated the following:

  • “Survey research and behavioral data, combining the results from a comprehensive, multimarket attitudinal and behavioral survey with analysis of Netflix behavioral data.”
  • “Analysis workshop: collaborative sessions with the CMB team, Netflix Insights, and the UX designers to review potential segmentation schemes.”
  • “Qualitative research: conducted in-home and shadowing ethnographies of members for mobile, TV, and web in each segment to understand pain points and innovation opportunities.”

Siddhi said, “Behavioral data is not enough to run meaningful segmentations. It wouldn’t tell us what we needed to do to be better years from now.” Mark elaborated: “For business use around innovation and the future, just getting a sense of what they are doing now is not enough: they might be doing it, but are unhappy about what they are doing.”

CMB created a hierarchical cluster analysis that generated more than a dozen hierarchical segments, displayed in a dendogram. “In the two days of analysis workshops, working with Siddhi and her team and user design, we balanced the art with the math. How to get to a reasonable number of segments? Sometimes we might choose business segments that trumped the math. It was critically important to have actual users of the eventual segmentation embrace it and be part of its creation. We had many long fascinating conversations.”

Siddhi said, “What was super special about that was these were not data people. We had the UX designers, who live in a data-versed culture, but don’t understand hierarchical cluster analysis. But they were there as we went from the numbers to the segments. We developed these segments across 3 different device UIs (web, mobile, and TV), identifying the pain points for each segment. We then traveled to 6 different markets to bring these segments to life.

As strategic as this project was, by moderating in field and in homes, the team members were there to ask follow-up questions, and this super-amplified the impact and established trust in the process.”

For deliverables, CMB and Netflix produced segment videos, 360 VR stations, data posters, and took over a floor of a Netflix headquarters with a museum-style exhibit. “We created compelling, narrative videos of each segment, highlighting unique stories, streaming contexts, pain points, and opportunities. The VR stations compiled 360 video montages of each segment’s streaming environments for a VR experience from GoPro cameras of where people watch Netflix: with Google Cardboard they could feel like they were watching Netflix in a train in Mumbai, or an apartment in Poland, or a multigenerational family home in Mexico. The data posters were per segment and included survey data and behavioral data that added color to the insights in the videos. Being Netflix, we had a documentary crew following us, and their documentary helped socialize the results.”

Mark added, “Connecting the qual, quant, and behavioral data wasn’t oversimplified. The video was not just about the qual, the posters were just not about the quant. It was always about how the three played together.”

Siddhi said, “We even turned a conference room into a living room in India! For other places, we changed Wi-Fi settings and speed so they could see what it was like to stream in different environments. It wasn’t just the research team socializing the results: different disciplines were sharing learnings and manning stations.”

To apply the data, Netflix is using a “How Might We” framework to bridging insights to action in multidisciplinary teams. For example, “How might we make Netflix accessible to all members regardless of data connectivity?” This was a global need: Siddhi said, “We productized mobile downloads in 2016 to solve for this need.”

Key takeaways from the study:

  • “Mandate high engagement with the end users throughout to cement the usefulness of the research.”
  • “Strike the appropriate balance between art and science.”
  • “Leverage all available resources to uncover the best segmentation.”
  • “Assemble the right team for each state—play to your team’s strengths!”