The Experimentation Layer

Designing for Reusability

Code reuse has been a major concern of mine throughout my career. I graduated with a computer science degree and was taught the value and beauty of loosely coupled, modular code with a clear separation of concerns. In fact some of my most rewarding experiences have been writing user interface frameworks (a few dozen over the last 30 years).

The same goes for designing experiences. Early on I realized that interaction designs could also be reused. Sometime after reading the gang of four book (Design Patterns) I started thinking that patterns could be used to define experiences as well. It was after that I stumbled upon Jennifer Tidwell's excellent work which would later become the book Designing Interfaces. A few years later, I went on to help launch the Yahoo! Design Pattern library and write a book on the subject.

I mention all this to set the context that I am a huge fan of reuse.

Designing for Throwaway-ability

In my prior role at Netflix, I quickly discovered a wrinkle to the reuse mantra. If you know a little about Netflix you will know that experimentation is the bread and butter of refining the experiences there. The big "aha!" moment came (2008) when I counted the number of experiments we fielded vs the number of experiments that became a lasting part of the site. I discovered that

Only about 10% of the UI code written to craft the experience lasted more than a year. 90% of the UI code needed to be thrown away.

Could that really be right? In some cases we were testing 10+ test cells over a couple of months time. Only 1 cell (or sometimes a combination of cells) was selected as a winner. The winning cells often led to a new hypothesis that led to another set of test cells being fielded (perhaps another 10+ cells). And of those a new cell (or control) would end up as the winner. It doesn't take too many 9 out of 10 losers to start seeing that a lot of code needed to be thrown away over the course of a year. (I am over-simplifying this a bit. There is often reuse between test cells as well as the number of cells fielded will vary.)

Now to a typical user the experience doesn't seem to change much in the short term. User A might be in a control cell (the standard experience). And in the next set of tests User A might still get allocated to the standard experience again. So to User A the site seems pretty stable. However if you plot the arc of the experience year over year you will see some substantial changes occur (think: Amazon, Netflix, Twitter, Facebook) as newly minted experiences are released to the full user population.

The epiphany for me was that for the most volatile part of the experience...

We needed to design for throwaway-ability and not as much for reusability.

What does that even mean? Design for throwaway-ability? It means not over-investing in the scaffolding for reuse. It means designing a way to select code for experiments in a simple grokkable manner (like using the file system or packaging mechanisms for bundling experiments together). It means externalizing the logic for selecting experiments as much as possible. It might even mean copy and paste reuse to make it easy to modularize the experiments. It means don't start with writing components, instead start with creating the experience.

Heresy!

When I had this epiphany it certainly felt heretical. Over time I have come to see the wisdom of knowing when to reuse and when not to reuse. However, lest I be next for a good stake burning for giving others a license to writing crappy code (since one could argue that it will be thrown away anyway ;-) let me provide some nuance.

Reuse is Still a Good Thing

Not all of the UI layer changes with experimentation. The nuance is the UI layer contains different levels of volatility. The code to make a button or render a template doesn't change with an experiment. However, the placement of the button or the contents of the template will often vary from test to test.

For the more stable areas of the experience, the great news is we have a rich set of open source libraries and frameworks that can provide a lot of this reuse right out of the box. Examples include component libraries like YUI or jQuery; templating solutions like mustache or dust; MVC patterns like those expressed in backbone.js; common javascript extensions like those found in underscore.js and so on. In addition each organization will find certain aspects of their user experience that benefit heavily from thoughtful reuse.

The truth is volatility increases as you get closer to the user experience and reuse opportunities increase as you move down the software stack.

This funnel illustrates the relationship of volatility and reuse 

as you move up and down the software stack.

UI Layer = Experimentation Layer

What I want to impress upon you is a simple thought.

The UI layer should be thought of as the experimentation layer

Think of the topmost layer of the user interface code (especially the HTML, CSS and Javascript for experiments) as the experimentation layer. As a corollary

You should always create an experience before thinking about creating a component 

In other words seek to use before reuse. I have found in my conversations at PayPal, talking about the UI layer as an experimentation layer is a great way to shift the conversation to the experience rather than to spending months building components and interface scaffolding. The goal is to get code in front of the user as fast as possible. Instead of starting with reuse, we start with the experience.

What about Consistency?

Now let's be clear, experimentation is not the silver bullet that will create the perfect experience. A lot of understanding about the user, scenarios, and just good solid design principles are needed to make a good start on the experience. Experimentation can often lead to local optima instead of finding the overall experience. But given a good experience, experimentation can allow you to refine the experience in the right direction.

One of the valid complaints that designers will often raise about experimentation is that it can lead to different experiences across the product families as the experimentation often is chasing specific business metrics in a specific area of the business. This can lead to some inconsistency when we allow the experience to be experimented on (at Netflix we call this the Frankenstein phase). There is a natural tension between innovation and standardization and the truth is healthy products will swing back and forth between loosening the constraints and innovating and then pulling back in the reigns and standardizing.

Design standardization (read: patterns, guidelines) can be used to bring consistency. But I have often seen design standards teams position themselves at odds to innovation. And on the converse I have seen design innovation teams totally ignore basic patterns that could be employed. Both are wrong. Consistency without informed learnings during phases of innovation is the dead letter. And innovation without the learnings of the past & best practices is just wild speculation.

Engineering teams will struggle with when to invest in the reuse scaffolding and when to simply create an experience that is quick to build and quick to tear down.

The Bottom Line

The key thing to remember is everything must be in service to the user's experience. It's way to easy to forget this and get in an endless cycles of finessing the UI - which I must remind you might not even be the right experience.