‘Lean Analytics’ by Croll & Yoskovitz

This is one of the most important product management books I’ve read to date. Absolutely fantastic. Like most of the Lean Series, lessons can be applied to organizations of any size.

Feel free to check out my highlights in Evernote after reading Lean Analytics: Use Data to Build a Better Startup Faster.

Some excerpts:

“The core idea behind Lean Analytics is this: by knowing the kind of business you are, and the stage you’re at, you can track and optimize the One Metric That Matters to your startup right now. By repeating this process, you’ll overcome many of the risks inherent in early-stage companies or projects, avoid premature growth, and build a top a solid foundation of true needs, well-defined solutions, and satisfied customers.”

• “A good metric Changes the way you behave – This is by far the most important criterion for a metric: What will you do differently based on changes in the metric? Accounting metrics like daily sales revenue, when entered into your spreadsheet, need to make your predictions more accurate. These metrics form the basis of Lean Startup’s innovation accounting, showing you how close you are to an ideal model and whether your actual results are converging on your business plan.”

• “Much of lean analytics is about finding a meaningful metric, then running experiments to improve it until that metric is good enough for you to move to the next problem or the next stage of your business. Eventually, you’ll find a business model that is sustainable, repeatable, and growing, and learn how to scale it.”

• “We sometimes remind early stage founders that in many ways, they aren’t building a product. They’re building a tool to learn what product to build… Lean Startup is focused on learning above everything else, and encourages broad thinking, exploration, and experimentation. It’s not about mindlessly going through the motions of build>measure>learn – it’s about really understanding what’s going on and being open to new possibilities.”

• “Rally doesn’t release software, but instead ‘turns features on for users and customers.’ Most features have a toggle that allows Rally to turn them on or off for specific customers… Underneath Rally’s feature development process, the company is focused on measurement. ‘We have an internal data warehouse in which we record everything from server/database kernel-level performance measurements to high-level user gestures derived from HTTP interactions between the browser and our servers,’ says Zach (product manager at Rally). The goal is to make sure the team can measure feature usage and performance. ‘When we develop a new feature our product team can form theories about how much usage warrants further development of that feature… As we are toggling on the feature we can compare to actual data. Because the data includes both usage and performance information, we can quickly understand, in real time, the impact a feature is having on the performance and stability of our production environment.’… Rally has taken measurement to the next level. In a way, Rally is 2 companies- one making lifecycle management software, and one running a gigantic, continuous experiment on its users to better understand how they interact with the product itself. This requires a lot of discipline and focus, as well as considerable engineering effort to make every feature testable and measurable, but it’s paid off in less waste, a better product, and a consistent alignment with what customers want.”

• “When Etsy adopted a continuous deployment approach to engineering, its initial business dashboards included registrations per second, logins per second, checkouts per second, new and renewed listings, and screwed users (distinct users seeing an error message). Importantly, these are all rate-based metrics designed to quickly highlight that we might have broken something… More recently, Etsy has been trying to make it clear how various features contribute to a sale. For instance, we can attribute the percentage of sales that come directly from search, but we’ve found that visitors who first browse, and then search, have a higher conversion rate.”

• “A concierge approach in which you run things behind the scenes for the first few customers lets you check whether the need is real; it also helps you understand which things people really use and refine your process before writing a line of code or hiring a single employee.” Similar to early tests ran with Bill Me Later, as well as the musical balance beam toy from IDEO.

• “Circle of Friends was a simple idea: A Facebook application that allowed you to organize your friends into circles for targeted content sharing… By mid-2008, Circle of Friends had 10 million users… But there was a problem. Too few people were actually using the product. According to Mike, less than 20% of circles had any activity whatsoever after their initial creation… So Mike went digging. He started looking through the database of users and what they were doing…And he found a segment of users-moms, to be precise-that bucked the poor engagement trend of most users. Here’s what he found… Their messages to one another were on average 50% longer. They were 115% more likely to attach a picture to a post they wrote. They were 110% more likely to engage in a threaded conversation. They had friends who, once invited, were 50% more likely to become engaged themselves. They were 75% more likely to click on Facebook notifications… The numbers were so compelling that in June 2008, Mike and his team switched focus completely… Ultimately, the company moved off Facebook, grew independently, and sold to Sugar Inc. in early 2012… Mikes success with Circle of Moms was his ability to dig into the data and look for meaningful patterns and opportunities. Mike discovered ‘unknown unknowns’ that led to a big, scary, gutsy bet that was a gamble-but one that was based on data.”

• “Everything is an experiment you can learn from.”

• “Tim Ferriss, in an interview with Kevin Rose, said that if you focus on making 10,000 people really happy, you could reach millions later. For the first launch of your MVP, you can think even smaller, but Ferriss’s point is absolutely correct: total focus is necessary in order to make genuine progress.”

• “The more engaged that people are with your product, the more likely they’ll stay.” 7 questions to ask yourself before building a feature:

1. Why will it make things better?
2. Can you measure the effect of the feature?
3. How long will the feature take to build?
4. Will the feature overcomplicate things? Complexity kills products. It’s most obvious in the user experience of many web apps: they become so convoluted and confusing that users leave for simpler alternatives.
5. How much risk is there in this new feature? Building new features always comes with some amount of risk. There’s technical risk related to how feature may impact the code base. There’s user risk in terms of how people might respond to the feature. There’s also risk in terms of how a new feature drives future development, potentially setting you on a path you don’t want to pursue.
6. How innovative is the new feature?
7. What do users say they want?

• Leading Indicator Examples:

1. Facebook – Determined a user would become ‘engaged’ later if he reached 7 friends within 10 days of creating an account.

1. Zynga – Looks at day 1 retention. If someone came back the day after she signed up for a game, she was likely to become engaged.
2. Dropbox – Changes that someone becomes engaged increases significantly when he puts at least 1 file in one folder on one of his devices.
3. LinkedIn – Tracks how many connections a user establishes in a certain number of days to estimate longer-term engagement

“Just because you have a lot of data doesn’t mean you’re data-driven. Sometimes, starting from scratch with a small set of data collected to solve a specific issue is more likely to get executive sponsorship because the problem is bounded and constrained, whereas nobody knows what controversies are lurking in the larger amounts of data exhaust the data organization has collected over the years.”

“Data doesn’t just lead to better decisions. It also improves organizational efficiency. You can create a flatter, more autonomous organization once everyone buys in to data informed approach, because rather than needing to propagate an opinion across the organization, you can let the facts speak for themselves… Whether in a leadership position or not, you can make your organization more data-centric.”

• “Whenever you look at a metric, ask yourself, “What will I do differently based on this information?” If you can’t answer that question, you probably shouldn’t worry about the metric too much. Consider for example, ‘total signups.’ This is a vanity metric. The number can only increase over time (classic up and to the right graph). It tells us nothing about what those users are doing… ‘Total Active Users’ is a bit better assuming you’ve done a decent job defining ‘active’ but it’s still a vanity metric that will gradually increase over time, too unless you do something horribly wrong. There real metric of interest-the actionable one-is ‘Percent of Users Who Are Active.’ This is a critical metric because it tells us about the level of engagement your users have with your product. When you change something about the product, this metric should change, and if you change it in a good way, it should go up. That means you can experiment, learn and iterate with it.The point here is that you’re doing something based on the data you collect”

• “This insight is forever changing what it means to be a business leader. Once, a leader convinced others to act in the absence of information. Today, there’s simply too much information available. We don’t need to guess-we need to know where to focus. We need a disciplined approach to growth that identifies, quantifies, and overcomes risk every step of the way. Today’s leader doesn’t have all the answers. Instead, todays leader knows what question to ask.”

• “Discipline is key to success in a larger, later-stage startup, particularly in the furious heat of execution.”