2 posts on Collaboration

Every decision we make involves weighing tradeoffs, whether that is done consciously or not. From evaluating whether an extra bedroom is worth $800 extra in rent, whether being able to sleep lying down during a long flight is worth the $500 upgrade cost, to whether you should take a pay cut for that dream job.

For complex high-stakes decisions involving many competing tradeoffs, trying to decide with your gut can be paralyzing. The complex tradeoffs that come up when designing products ^[1] fall in that category so frequently that analytical decision-making skills are considered one of the most important skills a product manager can have. I would argue it’s a bit broader: analytical decision-making is one of the most useful skills a human can have.

Structured decision-making is a passion of mine (perhaps as a coping mechanism for my proneness to analysis paralysis). In fact, one of the very first demos of Mavo (the novice programming language I designed at MIT) was a decision-making tool for weighed pros & cons. It was even one of the two apps our usability study participants were asked to build during the first Mavo study. I do not only use the techniques described here for work-related decisions, but for any decision that involves complex tradeoffs (often to the amusement of my friends and spouse).

Before going any further, it is important to note a big caveat. Decision-making itself also involves tradeoffs: adding structure makes decisions easier, but consumes more time. To balance this, I tend to favor an iterative approach, adding more precision and structure only if the previous step failed to provide clarity. Each step progressively builds on the previous one, minimizing superfluous effort.

For very simple, uncontroversial decisions, just discussing or thinking about pros and cons can be sufficient, and the cost-benefit of more structure is not worth it. Explicitly listing pros and cons is probably the most common method, and works well when consensus is within reach and the decision is of moderate complexity. However, since not all pros and cons are equivalent, this delegates the weighing to your gut. For more complex or controversial decisions, there is value in spending the time to also make the weighing more structured.

The tradeoff scorecard

What is a decision matrix?

A decision matrix, also known as a scorecard, is a table with options as rows and criteria as columns, with a column in the end that calculates a score for each option based on the criteria. These are useful both for selection, as well as prioritization, where the score is used for ranking options. In selection use cases, the columns can be specific to the problem at hand, or predefined based on certain principles or factors, or a mix of both. Prioritization tends to use predefined columns to ensure consistency. There is a number of frameworks out there about what these columns should be and how to calculate the score, with RICE (Reach × Impact × Confidence / Effort) likely being the most popular.

The Tradeoff Scorecard is not a prioritization framework, but a decision-making framework for choosing among several options.

Qualitative vs. quantitative tradeoffs

Typically, tradeoffs fall in two categories:

• Qualitative: Each option either includes the tradeoff or it doesn’t. Thing of them as tags that you can add or remove to each option.
• Quantitative: The tradeoff is associated with a value (e.g. price, effort, number of clicks, etc.)

Not all tradeoffs are all equal. Even for qualitative tradeoffs, some are more important than others, and the differences can be quite vast. Some strengths may be huge advantages, while others minor nice-to-haves. Similarly, some weaknesses can be dealbreakers, while others minor issues.

We can model this by assigning a weight to each tradeoff (typically a 1-5 or 1-10 integer). But if quantitative tradeoffs have a weight, doesn’t that make them quantitative? The difference is that the weight applies to the tradeoff itself, and is applied the same to each each option, whereas the value of quantitative tradeoffs quantifies the relationship between tradeoff and option and thus, is different for each. Note that quantitative tradeoffs also have a weight, since they also don’t all matter the same.

In diagrammatic form, it looks a bit like this:

These categories are not set in stone. It is quite common for qualitative tradeoffs to become quantitative down the line as we realize we need more granularity. For example, you may start with “Poor discoverability” as a qualitative tradeoff, then realize that there is enough variance across options that you instead need a quantitative “Discoverability” factor with a 1-5 rating. The opposite is more rare, but it’s not unheard of to realize that a certain factor does not have enough variance to be worth a quantitative tradeoff and instead should be modeled as 1-2 qualitative tradeoffs.

The overall score of each option is the sum of the scores of each individual tradeoff for that option. The score of each tradeoff is often simply its weight multiplied by its value, using 1/-1 as the value of qualitative tradeoffs (pro = 1, con = -1).

While qualitative tradeoffs are either pros or cons, quantitative tradeoffs may not be universally positive or negative. For example, consider price: a low price is a strength, but a high price is a weakness. Similarly, effort is a strength when low, but a weakness when high. Calculating a score for these types of tradeoffs can be a bit more involved:

• For ratings, we can subtract the midpoint and use that as the value. E.g. by subtracting 3 from a 1-5 rating we get value from -2 to 2. Adjust accordingly if you don’t want the switch to happen in the middle.
• For less constrained values, such as prices, we can use the value’s percentile instead of the raw number.

Explicit vs implicit tradeoffs

When listing pros and cons across many choices, have you noticed that there is a lot of repetition? First, several options share the same pros and cons, which is expected, since they are alternative solutions to the same problem. But also because pros and cons come in pairs. Each strength has a complementary weakness (which is the absence of that strength), and vice versa.

For example, if one UI option involves a jarring UI shift (a bad thing), the presence of this is a weakness, but its absence is a strength! In other words, each qualitative tradeoff is present on all options, either as a strength or as a weakness. The decision of whether to pick the strength or the weakness as the primary framing for each tradeoff is often based on storytelling and/or minimizing effort (which one is more common?). A good rule of thumb is to try to avoid negatives (e.g. instead of listing “no jarring UI shift” as a pro, list “jarring UI shift” as as con).

It may seem strange to view it this way, but imagine you were trying to compare and contrast five different ideas, three of which involved a jarring UI shift. You would probably list “no jarring UI shifts” as a pro for the other two, right?

This realization helps cut the amount of work needed in half: we simply assume that for any tradeoff not explicitly listed, its opposite is implicitly listed.

Putting it all together

Your choice of tool can make a big difference to how easy this process is. In theory, we could model all tradeoffs as a classic decision matrix, with a column for each tradeoff. Quantitative tradeoffs would correspond to numeric columns, while qualitative tradeoffs would correspond to boolean columns (e.g. checkboxes).

Indeed, if all we have is a grid-based tool (e.g. spreadsheets), we may be stuck doing exactly that. It does have the advantage that it makes it trivial to convert a qualitative tradeoff to a quantitative one, but it can be very unwieldy to work with.

If our tool of choice supports lists within cells, we can do better. These boolean columns can be combined into one column as a list of all relevant tradeoffs. Then, a separate table can be used to define weights for each tradeoff (and any other metadata, e.g. optional notes).

I currently use Coda for these tradeoff scorecards. While not perfect, it does support lists in cells, and has a few other features that make working with tradeoff scorecards easier:

• Thanks to its relation concept, the list of tradeoffs can be actually linked to their definitions. This means that hovering over each tradeoff displys a popup with its metadata, and that I can add tradeoffs by selecting them from a popup menu.
• Conditional formatting allows me to color-code tradeoffs based on their type (strength/weakness or pro/con) and weight (lighter color for smaller impact).
• Its formula language allows me to show and list the implicit tradeoffs for each option (though there is no way to have them be color-coded too).

There are also limitations however:

• While I can apply conditional formatting to color-code the opposite of each tradeoff, I cannot display implicit tradeoffs as nice color-coded chips, in the same way as explicit tradeoffs, since relations can only display the primary column.
• Weights for quantitative tradeoffs have to be baked in the formula (there are some ways to make them editable, but )

---

1. I use product in the general sense of a functional entity designed to fulfill a specific set of needs or solve particular problems for its users. This does not only include commercial software, but also things like web technologies, open source libraries, and even physical products. ↩︎

You may be familiar with this wonderful illustration and accompanying blog post by Henrik Kniberg:

It’s a very visual way to illustrate the age-old concept that that a good MVP is not the one developed in isolation over months or years, grounded on assumptions about user needs and goals, but one that delivers value to users as early as possible, so that future iterations can take advantage of the lessons learned from real users.

The MVP as a range

While not quite what Henrik intended, I love this metaphor so much, I have been using it to describe shipping goals when writing product specs. I find they are understandable to anyone who has seen Henrik’s illustration, and fit nicely into a fixed time, variable scope development process, such as the Shape Up methodology that we use at Font Awesome.

1. 🛹 The Skateboard (aka the Pessimist’s MVP): What is the absolute minimum we can ship, if need be? This is the most bare-bones set of features, without which we cannot ship at all. It skews more utilitarian: it has the basic functionality we need, but its UX is very rough, even embarrassing. Anything that can be flintstoned is flintstoned. This is meant to be less-than a traditional MVP.
2. 🛴 The Scooter (aka the Realist’s MVP): It is the minimum set of features we want to ship that will still provide value and fulfill enough user needs across enough user segments to be worth it. Its UX is more well thought out than the skateboard but anything nontrivial to implement is punted unless essential. This is closer to a traditional MVP.
3. 🚲 The Bicycle (aka the Optimist’s MVP): The wishlist or stretch goals. If everything goes really well, what else can we ship? This may include UX improvements, “sprinkles of delight”, and features that are nonessential but have high I/E ratios. This is where we aspire to be, but we are not going to be heartbroken if we don’t get there.
4. 🏍️ The motorcycle: These are improvements that are beyond even the optimistic MVP, but we want to get to sometime in the near future.
5. 🚗 The car: Improvements that we can ship in the medium to longer term future.
6. 🏎️ The race car (aka the North Star UI): This is the ideal product we would ship if we were not bound by ephemeral constraints like time, resources, performance considerations, or backwards compatibility.

The meat is the first three stages, since they directly affect what is being worked on. The more we go down the list, the less fleshed out specs are, since we know they will change once we have input to customers.

The most controversial of these is the last one: the race car, i.e. the North Star UI. It is the very antithesis of the MVP. The MVP describes what we can ship ASAP, whereas the North Star describes the most idealized goal, one we may never be able to ship.

It is easy to dismiss that as a waste of time, a purely academic exercise. “We’re all about shipping. Why would we spend time on something that’s not even feasible?” I hear you say. However, in the rest of this essay I will argue that it is one of the most important milestones, and fleshing it out pays dividends in the long run.

More deliberate product design

TODO: Race car with arrows pointing to car, motorcycle, bike, scooter, skateboard.

Whether you realize it or not, the North Star is the only actual input into this process. Every other stage, the skateboard, the scooter, the bike, the motorcycle, the car, are all outputs. They are all derived from the North Star, like peeling layers off an onion. In fact in some contexts the process of breaking down a bigger shipping goal into milestones that can ship independently is literally called layering.

The process is so ingrained, so automatic, that most product designers don’t realize that they are doing it. They go from race car to car, or even motorcycle so quickly they barely realize there was anything else there to begin with. Thinking about the North Star feels like a guilty pleasure — who has time for this daydreaming? We gotta ship, yesterday!

But the race car is fundamental. Without it, there is no skateboard — you can’t reduce the unknown. Without a solid North Star, your MVP is a confused jumble of design decisions and compromises, so tangled it becomes impossible to tell them apart.

To stick with the transportation metaphor, a skateboard might be a good MVP if your ultimate vision is a race car, but it would be a terrible minimum viable ferry boat — you might want to try a wooden raft for that.

TODO: first, the most basic raft possible. Then a simple sailboat, then a speedboat, then a yacht, and finally a ship.

This North Star may (and likely, will) change down the line, informed by customer feedback. That’s okay and par for the course. We don’t need to wander aimlessly with no destination, to be able to course correct.

Perhaps counterintuitively, spending time fleshing out a North Star UI can actually help you ship. Allow me to explain.

Simplify problem solving

A common problem-solving strategy in every domain, is to break down a complex problem into smaller, more manageable components and solving them separately. Product design is no different. The concept of a North Star UI breaks down tough product design problems into three more manageable components:

1. North Star UI (NSUI): What is the ideal solution?
2. Ephemeral constraints: What prevents us from getting there?
3. Compromises: How close can we reasonably get given these constraints?

In many simpler problems, their difficulty is concentrated in only one of these components, in which case this framework does not help much. Where it really shines is the really tough product problems, where the first and third question are both hard. Far easier to answer them separately than trying to answer both at once.

Facilitate team alignment

TODO: Two people arguing. One has a speech bubble with a skateboard, the other a speech bubble with a wooden raft. The first also has a thought bubble with a car, the second a thought bubble with a ship.

When the North Star UI is not clearly articulated, it doesn’t mean it doesn’t exist. It just means that everyone is following a different North Star.

Since MVPs are products of the North Star, this will manifest as difficulty reaching consensus at every step of the way.

Debating at a different level of abstraction than what produced the original disconnect is generally a recipe for nonterminating divergence. It pays off to zoom out and resolve the root cause separately, rather than waste time and energy debating its byproducts one after another, like fighting off a Lernaean Hydra one head at a time.

Having the space to flesh out the North Star UI separately not only eliminates future disagreements before they happen, it also strips away a lot of noise.

Often, what is fundamentally a North Star disagreement will masquerade as a disagreement about practical constraints. It feels easier to cite practical constraints than to debate the merits of an idea directly. Fleshing out the North Star UI separately eliminates this deflection at the root. Here is a story that may sound familiar: Alice has designed an eigensolution, — an elegant solution that addresses several user pain points at once. She is aware it would be a little tricky to implement, but she thinks the tremendous improvement in user experience is worth it and she can layer it in such a way that it can ship incrementally. When she presents her idea to the product team, Bob dismisses it as “this is way too much work, it’s not worth doing”. However, what he is actually thinking is “this is a bad idea and any amount of work towards it is a waste”. Instead of spending time figuring out whether Alice’s concept is a good idea, they spend their time discussing how much work it is and whether it could be reduced. As a result, they fail to reach consensus because the amount of work was not the core issue.

It is important to answer the questions above in order, and reach consensus on what the North Star UI is before moving on to the compromises. This way, we are aware of what is an actual design decision and what is a compromise driven by practical constraints. Articulating these separately, allows us to debate them separately. It is very hard to evaluate tradeoffs collaboratively if you are not on the same page about what we are trading off and how much it’s worth. You need to know both the cost and the benefit to do a cost-benefit analysis.

North Star UIs can improve MVPs via user testing

Conventional wisdom is that we strip down the North Star to an MVP, ship that, and then iterate based on input from real users. But did you know you can actually get input from real users without writing a single line of code?

While common knowledge among usability folks, this seems unheard of in product management circles. You don’t need to wait until an implemented MVP to do user testing. You can do user testing as early as a low-fi paper prototype, with the user telling you where they would click or tap and the facilitator mocking the response. This allows you to user test your North Star UI directly and adjust your MVP accordingly without having to wait for a whole deployment cycle.

Obviously, this works better for some types of products than others (it is notably hard to mock rich interactions or UIs with too many possible responses), but it is a powerful tool to have in your arsenal. It can be particularly useful useful when there are vastly different perspectives within a team about what the North Star UI is, or when the problem is so novel that every potential solution is on shaky ground. Even the best product intuition can be wrong, and there is no point in evaluating compromises if it turns out that even the “perfect” solution is not actually a good one.