Only one out of ten startups manages to succeed. However, this does not mean that nine out of ten ideas are bad. In most cases, failure is due to an ill-defined goal and insufficient focus on metrics that measure how the idea is working towards its goal.
The main focus of this article is to elicit the value of startup metrics for creating a successful product. Railsware is a product studio. We build products for our clients as well as our own. The latter include developer tools such as Mailtrap, Smart Jira Checklist, OrgMap, and others. We’ve made our way through the thorny path of product building time and time again, and are ready to share our experience. In this blog post, we’ll reveal the best insights Railsware product managers and engineers have gained into business startup metrics over the years. Let’s get started.
What is required for a product to succeed over a long period of time?
At the heart of any successful product lies a thorough analysis and some luck. Let’s look at the factors that define whether a product succeeds or fails:
- Market fit
Unfortunately, a lot of promising ideas fail because they don’t meet the market need. The product/market fit concept is fundamental for the startup world. It defines the interplay between a product and a customer. When your customers begin to sell your product for you by sharing their positive experience with others, it is the evidence of a true market fit. You need to build a product that people not only love, but need and use daily. That defines the product’s value on the market.
Your startup may be called scalable if it can grow fast without significant loses. Growth is one of the fundamental goals, which we’ll describe later, and is highly desired by investors and product managers. Rapid growth rates mean your idea is a rising star. If a startup does not scale, it will lead to the opposite outcome – shrinking. In this case, the idea turns into a shooting star, which, unfortunately, are not actually capable of fulfilling a wish.
- Money flow
This success factor is closely linked to the previous one. If your startup scale rate is too slow, there is a risk of running out of cash and finding yourself in the graveyard of deceased ideas. Even if you already have a minimum viable/lovable product or prototype but experience a rapid drain of cash, this may indicate a poor growth rate or an inappropriate development strategy. In this case, you’re doomed to fail unless substantial changes are implemented.
- Focus on the right metrics
Weird though it may sound, proper use of startup metrics is also one of the fundamental success factors. Products with a good focus on various types of analytical and statistical data have a better chance of staying afloat in the intensely competitive ocean that is the startup space. Here is a real case study provided by one Railswarian from his former job experience:
A startup was at its early stage with an MVP of a video debate platform and was looking for an investor. They had big plans to go public and support election campaign in the US. Goals were established, but they neglected to work on analytics associated with their product. No attention was paid to measuring product stickiness and churn rate, not to mention downloads. This neglect prevented the startup’s evolution, and it failed in attracting any investor. The product was shut down.
Some might say that the idea was poor or they were out of luck. And what if the goal vector was wrong and needed to be changed in the course of time? That’s another mistake inherent in fading startups – they make mistakes in setting priorities within the product pipeline. You can’t be stuck on a single goal for the entire development cycle. Product goals change just like people’s life goals do.
Startup goals at different stages of product development
Product goals do not come out of thin air. They are a result of the collaboration between product manager, engineers, marketing specialists, and other parties involved. To define product goals, you need to leverage business drivers that are vital for the continued success and growth of your startup. You need to go top-to-bottom and consider the company vision and objectives, what stage the company is on now, what type of high-level goals is active, etc.
There are three major types of high-level goals: customer satisfaction, growth, and revenue. Each of them is applicable to a particular pipeline stage. For example, in the Calendly project, we had several iterations aimed at customer satisfaction at the beginning. In other words, we focused on the product quality. You never know how many other startups are working on the same idea at a given moment, so you need to work hard to make your product lovable. When the product was ready and released, our focus shifted to customer acquisition and profit-making, which refers to growth and revenue goals respectively. Practically all startups undergo this goal-setting pattern.
Goals at the prototype stage
In the beginning, you have the global goal of creating a product the people will love. This is your alpha and omega. However, at the stage of the prototyping, the major goal is to acquire knowledge in regard to the idea’s feasibility. You’ll get some feedback, do market research, analyze appropriate metrics or do other activities that are meant to prevent you from costly ‘wrong choices’ in a startup. Surveys and feedback collection are especially important when developing a prototype. In the case of Mailtrap, our initial goal was to solve a problem our dev team was having. Later on, the idea began to evolve, and at the stage of prototyping, we wanted to understand whether the larger market had any interest in a fake SMTP server for testing emails without spamming real users. It very much did, and so we moved to the next stage with a new goal in mind.
Goals at the MVP stage
At the stage of a minimum or nano viable product, you still want to make a product that people will love, but with the atomic goal of watching customer feedback and tracking metrics. You do have your product now, and you need to make it great. With Mailtrap’s MVP, our major goal was to bring the maximum value, gather feedback, and achieve customer’s love.
It may happen that at this stage, the goal changes its direction. For example, another Railsware’s product called OrgMap was meant to be used internally as a company management tool. We also had plans to promote it externally as an efficient managerial solution for mid-sized companies for workflow organization. At some point, we understood that we were still looking for a selling factor and the key components to succeed in the market. As a result, we decided to focus solely on the needs of our organization and kept the product for our internal use only.
Goals at the production stage
Your product is already on track. Now, it’s important to stick to your vision and avoid offshoots that may decrease the product value and business profit. Profit is your driving goal at this stage. All startups need to make money. However, this goal should not overshadow the objective of creating a great product.
At this point, the focus may be on customer acquisition (increasing user base, elaborating loyalty programs, retention activities, etc.) as part of your growth strategy. That kind of goal was set at Calendly, Mailtrap, and many other products you can check out in our Railsware case studies.
If goals are measurable values, how and what should you measure? Product metrics give more clarification on that.
Key startup metrics and how to measure them
When you collect proper metrics, you get an optimized picture of your product, and the decision on what to do further becomes evident. Traditionally, data is divided into several types depending on the focus area like finances, customer engagement, marketing, and so on. If your aim is to understand how to scale profitably, you should take advantage of financial metrics for startups including the recurring and/or total revenue, lifetime value, customer acquisition cost, etc. Customer engagement metrics include monthly active users, churn rate, downloads and others.
There are different frameworks to measure and analyze metrics. One of the most famous ones is Google’s HEART, which refers to Happiness, Engagement, Adoption, Retention, and Task Success. Each element is measured with the help of attitudinal metrics. For example, to measure user happiness at Jira Checklist, we used the net promoter score (NPS) metric. NPS is more applicable for later stages, while at early stages, you may limit yourself to surveys because of the small number of users. At Mailtrap, we conducted surveys via Typeform, Twitter, and UserVoice to get user feedback.
Another framework that deserves to be mentioned is AARRR – startup metrics for pirates. It has nothing to do with fraudulent or blackhat activities and is not meant exclusively for outlaws. This five-step model for analyzing product growth consists of Acquisition, Activation, Retention, Referral, and Revenue. The idea of the framework is to shape a behavior model of an average customer by answering the following questions:
- Where do customers come from? => acquisition
- How many of them are satisfied with the product at once? => activation
- Are there any revisits over time? => retention
- Will they share the product-related experience with their friends? => referral
- Will this behavior bring in money? => revenue
Apart from the mentioned options, there are many other ways for you to effectively diagnose and understand problem drivers and think through them. All you need is determine key dimensions and set criteria to shape your own framework for decision-making. In the following table, you’ll find our vision of which metrics are better to use at certain product stages.
|North Star metric||✔|
The table shows a common way of things, but exceptions are possible of course. Now, let’s take a look at key metrics for startups, some of which Railsware leveraged for different projects.
North star metric
This type of startup metrics is often referred to as one metric that matters (OMTM). Some say it is a leading indicator of success. North star metric belongs to the category of output metrics, which are helpful for long-term goal setting.
In practice, it looks as follows: to become an active user, he or she should perform X number of activities like sending messages, watching videos, sharing content, etc. As a result, this number becomes a foundation-stone for the product team. It defines the desired goal – what should we do to make people perform X more activities? There are numerous examples across the web.
Slack’s north star is 2K messages per user. Facebook wants you to make friends with seven people within ten days. For LinkedIn, you need to have your profile filled out and get a minimum requested number of connections.
At the same time, we have to warn you that buying into the one key metric philosophy at the production stage can lead to failure because it captures one dimension of your product and is not actionable (it is more like a scoreboard). With that in mind, a better way to focus on growing your product is to pick the most relevant metrics from the following ones.
At the stage of production when you’ve already built a product that people love, the major goal is to generate revenue. Hence, let’s start our discovery with key financial metrics for startups, amid which the monthly recurring revenue (MRR) is a core one.
It is one of the major startup valuation metrics for SaaS business. MRR is simply how much money your product brings in per month. Sometimes, companies use ARR, which refers to the annual recurring revenue. The metric is used to measure the revenue components which are recurring naturally and do not include some one-shot fees and special payments. We also rely on MRR at both Mailtrap and Smart Jira Checklist.
How to calculate:
Mailtrap offers two paid subscription plans: Fly-trap and Bee-trap for $9.99 and $24.99 correspondingly. Let’s suppose we have 30 Fly-trap users and 10 Bee-trap users. The MRR formula will be the following:
MRR = 30 * 9.99 + 10 * 24.99 = $549.6
When your business scales up, and the number of subscription plans and options grows, it’s highly recommended to use the average monthly recurring revenue for calculations. You need to multiply the number of customers you have by the average of their monthly fees.
The example above is shallow and is unlikely to help in real life calculations associated with upgrades, downgrades, reactivations, churns, discounts, etc. Today, you can find many services that can do hard calculations for you.
Customer Acquisition Cost shows how much money you spend to get one customer. It includes all marketing costs such as advertising, SEO, sales reps commissions, sales support call centers, etc. This key startup metric is important to both product owner and investor. CAC is useful for analyzing the startup scalability. The difference between the money spent on generating leads and the money returned shows the level of profitability. If your costs for getting a client exceed the money your product can earn, you’re heading in the wrong direction.
Another thing that you can measure with this metric is the efficiency of your marketing strategy or team. It’s possible that some expenses yield nothing and you’d better cut them off. As a result, you’re optimizing the return on your advertising investments.
It is a paradox that CAC grows in the course of expanding the clientage. For example, the first 1,000 users may cost $1 each, while the next 5,000 will be pricier – $1.5. Eventually, the cost of each forthcoming client may rise up to $5 and more. Therefore, you should not neglect the metric that shows the scope of leads attracted through each communication channel.
How to calculate:
You need to divide all marketing expenses by the number of acquired customers. As a rule, this figure is calculated at the rate of a year.
CAC = 50,000/42,000 = $1.2
Thus, with $50,000 spent on acquiring 42,000 customers, your CAC equals $1.2 per each customer.
MRR is at the heart of the SaaS business, while CAC and LTV are vital for startup unit economics. LTV is the lifetime value of a client within the entire relationship with the product. It is a tricky metric, which shows how much money you’ll earn on an average user during the product life cycle. Hence, you can estimate not only the long-term value of a customer, but also the future net profit per one customer with CAC included.
LTV concept is mostly used to compare CAC with the cash flows to be expected from that customer in the future. If these cash flows are significantly higher than the CAC, it is a sign to increase marketing costs. Your startup may be deemed healthy if your LTV/CAC ratio is three and above.
How to calculate:
In this article, Bill Gurley introduced an unfolded formula for LTV calculation and some warnings regarding using this metrics. We suggest you leverage a basic simplified and more understandable version.
LTV = ARPU*Gross Margin/Churn rate
- ARPU – average revenue per user;
- Gross Margin per month – the difference between revenue and variable costs per user;
- Churn Rate – the rate of cancelled subscriptions.
In addition to the above core financial SaaS startup metrics, there are plenty of others you may want to use. These include Customer Conversion Rate (to measure your costs and activities aimed at turning leads into clients), Average Revenue per Account/User (to measure how much money you get from an average paying client per month), Annual Contract Value or ACV (to measure the value of your clients per year, as well as ROI of your sales and marketing investments), and so on.
One way or another, MRR remains the core financial metric each SaaS startup should strive to increase. If users love your product, they should pay more. But it is not always so. Many products face a challenge in achieving a balance between value and price. Therefore, they change their prices quite often.
For example, at some point, the Mailtrap team understood that a big number of active users does not bring an adequate MRR. The reason was that the free plan users could get an extended set of functionalities (much of the product value) for free. The balance between the value and revenue was broken. Therefore, we had to fix that by limiting functionalities for free subscription users and stimulating them to shift to one of the paid options.
User engagement metrics
Now we are ready to switch to marketing metrics aimed at evaluating how many people joined, how long they spend on the page, how much they sign-up, how many pages they visit on average, the bounce rate, and other indicators of user engagement. These are essential growth metrics for startups.
This metric defines the number of active users per day (DAU) or month (MAU). You can also encounter the ADAU acronym. It’s determined by dividing the average number of users active on any given day by the number of days in a month.
The concept of ‘active user’ will look differently depending on the company it is applied in. The DAU/MAU rate is unique for each product, and you can pick a particular metric to understand retention. In some cases, activity metrics are expressed through the rate of new users or the number of users at a particular period like weekends or weekdays. For example, a social media-oriented app will benefit most from analyzing the daily usage, while enterprise software products are more interested in the weekly analysis. At Mailtrap, we focus on active usage metric expressed through the number of emails a new user sends during each month. It goes without saying that the bigger DAU/MAU rate you have, the better.
Stickiness or the DAU/MAU Ratio
When you divide the number of daily active users by the number of active users per month, you’ll find out how “sticky” your product is. As a rule, this metric shows the long-term prospects of a startup. The higher the stickiness, the higher income-generation rate you can achieve and build a strong user base.
It’s important to note that stickiness is not about product downloads or sessions. This metric relates to regular users, hence it lays the emphasis on retaining customers. As a rule, to become sticky, a product should possess the qualities including usability, reliability, usefulness, high entertainment level, curiosity, emotional and social effect. Having a product associated with all of them means you’ve hit the jackpot.
Session-related metrics: number, length, interval
Regardless of the product to be released in the market, a set of session-related metrics including the number of sessions per user, average duration and interval is vital to analyze the behavior of your users. Besides, these metrics are easy to track.
The number of sessions is measured per user or cohort of users (for example, those who use a particular subscription plan) within a certain period (week, month, quarter, year). Everything is clear here.
Analysis of the length or duration of sessions helps you understand the users’ needs and interests. For example, short-term usage may be frequent for business users who visit your product between meetings or during meal time. At the same time, longer sessions may be associated with a user’s longer leisure time. Time in-app is a similar metric to session length, but it focuses on the number of hours a user spends per day/week/month. This metric is popular for startups whose revenue directly depends on the time spent in-app.
The interval between sessions is another important metric to learn from users. When you spot long intervals, you can assume the users’ activities during that time and adapt the features of your product correspondingly. Therefore, you’re motivating the users to increase their time in-app and decrease intervals.
That’s it for business metrics for startups aimed at user engagement. The next section deals with making your user or customer satisfied.
User happiness metrics
If you do not measure how happy your users are, you might never know why they unsubscribe from your product. This category of metrics is self-reported, meaning it is difficult to measure happiness without asking users directly about their experience. Amid numerous options to evaluate user happiness, we emphasize the following:
We’ve already mentioned the Net Promoter Score as the growth metric for startups. We used this metric at Smart Checklist for Jira. It reflects the level of user satisfaction with the product.
Traditionally, NPS is defined through 11-point scale (0 to 10). To get these points you need to ask users a specific product-related question. For example, How likely are you to recommend Smart Checklist to a friend or a colleague? 0 means “Not at all likely” and 10 means “Extremely likely”.
As a result, you get three categories of users: Promoters, Passives, and Detractors. Passives are of no interest since they have no strong attitude to your product (they may recommend it to others and switch to its alternative equally well). The other two categories are important because they allow you to calculate NPS by subtracting the Detractors from the Promoters in percents.
Here is the NPS we had at Jira Smart Checklist in 2017:
Detractors (0-6 score): 13.33%
Passives (7-8 score): 48.89%
Promoters (9-10 score): 37.78%
NPS = 37.78% – 13.33% = 24.44%
In general, if the result you get is above zero it already means success. However, to become a company with an excellent score, you need to cross the 30% threshold. The next benchmark is a world class NPS exceeding 70. Webscale Networks and Vibes are the members of this top-level club.
This metric defines users or customers who stopped using your product or canceled their subscription. Churn rate is a good indicator of product health and fundamental for startups which are based on a monthly subscription.
Many startups pay more attention to the opposite metrics – retention rate. However, it isn’t as useful and actionable as the rate of churned customers. By the way, churn can be classified as a financial metric because the customers you lose can be converted into the revenue your product could have generated. It is interesting that the metric is fluid, meaning it is impossible to maintain zero churn rate even if your product is superawesome. Some users will leave anyway. However, great startups and businesses strive to keep the rate as low as possible all the time. The optimum churn rate for a SaaS startup is 2-5%.
The calculation of this metric is rather simple. You need to divide the churned customers by the total number of customers as of the previous month. It is also recommended to put more emphasis on the gross and net revenue churn. To get the gross revenue churn, you need to divide the churned MRR by the MRR as of the beginning of the month. The net revenue churn includes another variable – after-sales MRR – and looks as follows:
Net revenue churn = (churned MRR – after-sales MRR) / MRR as of the beginning of the month.
The difference between the gross and net lies in the measurement objects. With the gross churn, you get a realistic picture of your revenue churn and a clearer understanding of the product’s health. With the net churn, the picture looks more positive, because it includes the after-sales revenue.
Feedback as a metric
Bet you expected to see another combination of capital letters as the next user happiness metric? However, we decided to show you the importance of user feedback, which is also a metric. Our Mailtrap team can prove it since the product has been evolving thanks to user suggestions provided via different communication channels. Here are some examples of them:
“Would it be possible to have an email address for each inbox in Mailtrap?” “Would you consider adding a way to configure hard- and soft-bounces?” “I want to have more advanced email forwarding rules”
However, if we had implemented any desire of any user, we would have gotten bogged down in constant product tweaking. The solution is to allow users to vote for a particular feature on public resources.
User feedback is extremely important at the MVP stage, where the atomic goal is to watch what customers like and dislike. At production, feedback decreases in priority, but does not disappear entirely.
The above-described are key business performance metrics startups should rely on. You can tune them according to your project requirements. At the same time, we recommend you think wisely and don’t dissipate your efforts on too many metrics. If a metric does not give you weekly value, it’s most likely useless.
What would you say if you knew that your product was downloaded one million times or your startup gained five million sign ups? These figures are impressive and should definitely be shown to your friends or relatives, but not investors. That’s all the number of downloads can do in terms of efficiency – draw attention. All the rest is vanity.
Vanity metrics are the reverse side of data that really matters for growth. If you focus on the real metrics, you are on the way to make your product more sticky and attractive which in turn converts into user happiness. The focus on vanity metrics gets you nothing but shiny objects to impress less savvy investors. As an example, check out the following use case:
At the X project (NDA does not allow us to put its name here), we were tracking every action in the app but weren’t benefiting from every metric we tracked. Meanwhile, the use of Mixpanel for analytics cost quite a lot – more than $10k per month. As soon as we reviewed our activities and metrics we use, we decided to move analytics to our own platform. Eventually, the focus on actionable metrics allowed us to reduce our expenses.
You can check out Eric Ries’ article on this: “Why vanity metrics are dangerous”. The point is that if the metric you leverage has no influence on the bottom line, it’s not worth your time. The road to success is paved with actionable metrics that the lean startup should opt for.
We hope that the main course consisting of different-purpose data for analysis was tasty. And if you spare some room for dessert, we would be glad to serve it.
Innovative metrics are specific to the product. For YouTube, these are the number of likes or watches; for Mailtrap – how many emails are “trapped”; for hosting providers – data transferred per hour per device, and so on. If a metric is actionable for one product, it does not mean it will work for another one.
Innovative metrics are not an active ingredient meaning you should use a mix of traditional (MRR, CAC, etc.) and product-specific (likes, CPC, etc.) data for analysis.
First few tools to start working with lean startup metrics
You can measure some of the above-mentioned data using a calculator. It’s not difficult to calculate CAC or NPS when you have all the data. The question is where to get these data from? There are many ways to do that, as well as a plethora of tools. We are glad to share our experience and introduce some options from the Railsware’s toolbox.
Mixpanel is a user analytics platform to measure user behavior and interactions across a mobile or web app. Mixpanel allows you to do an analysis of your product usage. However, some information like the app discoverability, path of users, what steps they undertake, how many times they perform a single action, how much time they spend on a step and others cannot be taken directly from the tool. To get this data, you can use a lightweight and flexible command-line JSON processor called jq. Mixpanel can export data in JSON, and then you can quickly manipulate with it and transform it into the form you need using jq.
It’s important not to mix up jq and JQL provided by Mixpanel (a powerful query language to analyze and learn from data). We import the data and query Mixpanel raw data source using JQL through Blockspring (Google Sheets add-on) and get results in the appropriate format, so we don’t have to manually run JQ. Besides, it can be automated at a certain trigger like every hour or so.
Another cool way to look at metrics with Mixpanel is its dashboard. Before it came out, we had to build segmentation by ourselves. There were too many search events, and that was a problem. So, we had to fix the way certain events are tracked. The dashboard released us from these unnecessary tasks.
Google offers a number of great tools for data analytics, data visualization, dashboarding, etc. We usually take advantage of Data Studio and Google Analytics to work with different types of metrics. These are very usable and functional for analytics needs of our products such as Mailtrap and Jira Smart Checklist. We also use GSheets a lot. It helps us analyze ideas (filter bad ones) and versatile data related to user happiness and behavior (where the visitors come from, drop-offs, user scroll depth, etc.).
Another analytics tool we leverage mostly for dealing with financial metrics is ChartMogul. It allows you to visualize key money-related indicators like MRR, churn rate, average revenue per account, annual run rate, etc. The tool holds itself as a sort of replacement of spreadsheets. It automates subscription revenue and customer reporting, thus saving hours of work. In addition, ChartMogul provides numerous turnkey integrations (PayPal, Stripe, Shopify, etc.), as well as a CSV upload tool and Import API.
Such tools as Crashlytics and Fabric aren’t in the Railsware’s toolbox but are highly recommended by one of our experts. They are useful for dealing with metrics from a tech perspective. Crashlytics is a crash reporting solution that gives you actionable insights into app issues. With it, you can track, prioritize, and fix stability issues to improve the quality of your product. Fabric is the Crashlytics functionality expansion into mobile app analytics. It is also known under the name Answers. Both tools are aimed at helping users make smart decisions and save their time.
Where to go from there?
The principal thing about metrics is not to choose proper indicators but to make the right decisions based on the figures you get. Not only is the metric itself important, but also the insights behind it. The ability to interpret metrics is the most important skill for startup founders. If something is difficult to understand (because of the lack of data or for other reasons), the only way forward is to dig and delve. You may find a lot of stories about startups that failed not because of the lack of metrics, but because they did a poor analysis. The following story shows how important a thorough analysis is for the improvement of product performance:
At the Y project (NDA does not allow us to put its name here), we had an unpopular feature – tracker, which we decided to put into cold storage. At some point, the data analysis showed a rising demand in the feature and the growth of its DAU. A thorough interpretation of metrics allowed us to figure out what users want from using the feature. Based on this information, we focused our efforts on improving it and got it into the top five features of the product. The power of metrics and its proper interpretation resulted in the increase of user retention and making the product more lovable.
In general, you should consider metrics as a call to a specific action. If your MRR goes down, you need to get it reversed by adding a new feature, improving marketing, revising subscription plans, and other relevant activities. If your NPS goes down, it means your product is not as sticky or lovable as before. Your customers may be getting taken by a competitor offering better features or more flexible pricing.
Why metrics are vital for both startup founders and product owners
Readers with data-driven DNA do not need any explanation of how beneficial data can be. But some of you might have only started your path in product development. Therefore, let’s highlight some key takeaways based on the above.
- Metrics are a measurement system of a product’s evolution towards its goal. They let you understand the product success consisting of how users use your product, how much product brings in revenue, and what makes users particularly happy about your product.
- Metrics shape evidence-based decision-making
- Metrics help resolve disputes in the team, being an unbiased indicator of what’s going on with the product.
- Metrics are a good motivator if you see that the goal you chose is getting closer.
- You can and should measure not only the financial achievements of your product but also user happiness and engagement thereto.
- User feedback is an important metric as well.
- Vanity metrics are good to impress your friends but inefficient for pursuing your goal.
- The choice of a metric suitable to your product is much more important than a framework or formula to measure it.
- The ability to interpret metrics is the most important skill for startup founders.
- A good product manager should be able to gain insights by interpreting metrics.
- Your startup needs to focus on metrics in pursuing the goal of building a lovable product.