My notes: The Basics of Growth, Engagement, and Retention from a16z podcast

[Below are my notes from listening to the a16z Podcast: The Basics of Growth, Engagement, and Retention by Sonal Chokshi, Jeff Jordan, Andrew Chen. These are my notes from their great work]

Growth and acquisition, what startups mostly care about in the early days, because you don’t really have any active users but the other part of this is that you see all the users show up and how active they are starts to change over time…

Once you have users, how do you keep them engaged, retain them, and even “resurrect” or re-engage them?

Strategies for measuring engagement

Early on it’s all about getting users. If you’re widely successful at doing that you run out of users (or you start running low on users) and you have to go to engagement.

So Pinterest has a very high-quality problem right now. Most women in America, have downloaded the Pinterest app. But much more so they need to engage and re-engage the existing audience.

We love engagement from an investor standpoint because it’s just, you know, that it shows stickiness.

You can often hack your way into new users. It’s really hard to hack your way into true engagement.

What we mean by engagement is actually interacting with them and seeing their activity.

often analyze this is looking at cohort analysis.

Where we’ll look at kind of each batch of users that’s joining in each week and really start to dissect like well, how active are they really and to compare all these cohorts over time. You’re basically putting the users that come in from a particular timeframe, let’s say it’s a week, and you’re putting them into a bucket, right? And what you’re doing is you want to compare all of these different buckets against each other.

Build the engagement curve for each one, how active are they

Do this to try to spot any positive trends

Compare vs your new layers on the cake

The evolution of most of these companies as they’re getting bigger tends to start with acquisition, then focus much more on churn and retention, and then ultimately also to layer in resurrection as well.

the best models are built off of the cohort curves

Using cohorts really give you a sense of their network effects, and network effect is the business gets more valuable to more users that use it; if it gets more valuable, your newer cohorts should behave better than your early cohorts.

In a network effects businesses we always ask, “Show me the data, cohort curves, or ”

segmenting it.

segmenting it based on market geography, why that’s so valuable is because then you can compare markets against each other. You can say, “Well, you know, this market which is like, has much more density in terms of the numbers of scooters behaves like this.” And you can start to draw conclusions, sort of a natural A/B test in order to do that.

And I think the similar kind of analysis you can do for B2B companies is for products that have different sized teams using it. If you have a really large team that they are all using a product, well, are they all using the product more as a result? And let’s compare that to something that maybe only has a couple. … And so this way you can start to kind of disassemble the structure of these networks and do they actually lead to higher engagement.

So after you get the new users, the way that you have to think about it is around quality, right? You have to make sure that the new users turn into engaged users. One of the things people often talk about is just sort of this idea of like an “a-ha” moment or a magic moment where the user really understands the true value of the product.

And once you get that, then you have the opportunity to keep them in this engaged state over time. A lot of the entrepreneurs I work with are trying to figure out what is my magic moment that then creates the awareness of the value prop.

So almost all the early activity in a company is, “Okay, how do I get the users?” As you get users, you get more and more leverage from efforts at activation and retention and engagement.

Once you have an active base of users and customers, what starts to get really interesting is to really analyze what are the things that actually set that group up to be successful really long-term sticky users versus what are the behaviors and profiles where users aren’t successful, right? You actually throw your data science team on it to figure out all the different weights for both behavioral as well as the demographic and sort of profile-based stuff on there. And so one of the first things that you figure out is that each one of these products actually has this ladder of engagement where oftentimes new users show up to do something that’s, valuable but potentially infrequent. And you need to actually level them up to something that happens all the time.

For example, when you first install Dropbox, the easiest thing that you can do is you can use it to just sync your home and your work computers, right? And that’s great but really the way to get those users to become really valuable is for them to share folders at work with their colleagues. Because once they have that and people are dragging files in, and they’re really starting to collaborate on things, that’s like the next level of value that you can actually have on a daily basis versus this thing that kind of is in the background that’s just syncing your storage.

“ladder of engagement”

Step one is really segmenting your users into this kind of engagement map, oftentimes you’ll see this visualized as a kind of state machine where you have folks that are new, you have folks that are casual, and you want to track how much they’re moving up or down in each one of these steps.

And then once you have that, then the question is, okay, well, great, how do you actually get them to move from one place to the other? First there’s content and education; they need to know in context that they can actually do something.

The second is of course if your product has some kind of monetary component, then you can use incentives like $10 bucks off your next subscription if you do this behavior that we know for sure gets you to the next step

The third thing is really just like refining the product for that particular use case, maybe there are certain kinds of products that are transacted all the time and so you maybe want to waive fees or you give some credits or you do something in order to get people to get addicted to that as a thing.

In order to develop your strategy, you really need to understand how your users are behaving at a really refined level.

engagement metrics

Frequency: how often are you using the product regardless of the intensity and the length of the sessions and all that other stuff. Literally just the frequency of sessions.

daily active user divided by monthly active user ratio, and that gives you a sense for how many days is a user active.

And when it works and when it doesn’t. There’s a lot of nuances around when to apply it and when not to.

Because it really depends on the product you have, the type of nature of use it has, etc.

L28: A frequency diagram that basically says, okay, show a bar showing how many users have visited once in that month, then twice in the month, and then three times in the month, and then four times in the month. And you kind of build that all the way out to 28 days.

I actually have heard this referred to as a smile because the one use is always pretty big. Done right, it starts to increase at the end. So you basically get a smile.

If you take a step back, it’s a precondition for investing in a venture business essentially that there’s growth. If it’s end market you want to see growth, but growth by itself is not sufficient. Investors love engagement

DAU/MAU and L28 are really hard to game. Whereas growth can be very easy to game

You get to see these incredibly seductive growth curves but our job is essentially to be cynical and just say, okay, we need to go below that because there are a lot of talented growth hackers who can drive growth. I looked at a number of businesses that had tens of millions of users and no one ever came back.

Network effects

network effects by definition are that a network becomes more valuable the more users that use it. What happens on the engagement side with network effects? What are the things we should be thinking about in that context?

Typically network effects, if they’re real, manifest in data.Things like the cohort curves improve over time

When you look at the data, what you really figure out is that network effect is actually like a curve, and it’s not a binary yes/no

So for example, if every city was a data point, and you graphed on one side the number of restaurants in the city versus the conversion rate for that city, you would quickly find that when cities have more restaurants, the conversion rate is higher.

The number of restaurants you have as a percent of that market’s restaurant universe; because having 100 restaurants in Des Moines is different than having 100 restaurants in Manhattan.

what you then quickly figure out is that there’s some kind of a diminishing effect to these things often in many cases. So for example, in rideshare, if you are gonna get a car called 15 minutes versus 10 minutes, that’s very meaningful. But if it’s five minutes versus two minutes, your conversion rate doesn’t actually go up.

If you can express your network effect in this kind of a manner, then what you can start to show is, okay, yeah, we have a couple new investment markets that maybe don’t have as many restaurants or don’t have as many cars but if we put money into them and invest in them and build the right products, etc. then you can grow.

Engagement vs. retention

The real difference is that engagement obviously varies depending on the product, the type of thing it is, whether it’s weather or ebook, and retention is are you still using it after X amount of time.

The chart that I’d love to really see is one that was like a bunch of different categories, retention versus frequency versus monetization. I think you got to be, like, really good at least on one of those axes.

What kind of time were you competing for in the first couple years of the smartphone? You were competing against literally I’m gonna stare at the back of this person’s head, or I can use some cool app that I downloaded. Versus these days you actually have to take minutes away from other products.

You have that competition. It is a big overhang right now in consumer investing because you have to displace someone’s minutes.

Measuring retention

You adjust the time period that you’re relevant on. If the average diner dines twice a year…Then that’s your time frame.

One of the things that companies can often do is to measure upstream signal. So for example, Zillow, you’re probably not gonna buy a house very often. Maybe a couple times in your life. However, what’s really interesting is they can say, “Well, you know, maybe folks aren’t buying houses but at least are we top of mind? Are they checking the houses that are going on sale in their neighborhood? Are they opening up the emails? Are they doing searches?”

“dance” between architecting and discovering. Like, you might know some things upfront because you’re trying to be intentional and build these things, and then there are things that you discover along the way as your product and your views and your data evolves. How do you advise people to sort of navigate that dance?

You iterate. You develop hypotheses. You put it out there and you test the hypothesis. I think my product’s gonna behave this way. And then, did it?

Probably the most important thing, marketing can be art, marketing could be science; in the consumer internet, it’s more science. Some companies can effectively do TV campaigns, large media budgets, things like that. For me, the better companies typically just rip apart their metrics, understand the dynamics of their business, and then figure out ways to improve the business through that knowledge. And that knowledge can feed back into new product executions or new marketing strategies or new something. It’s constant iteration but it’s informed by the data at a level that on the best companies is really, really deep.

Ultimately, you have a set of strategies that you’re trying to pursue and you pick the metrics to validate that you’re on the right track.

There's a lot of “nature versus nurture” kind of parts to this. Your product could just be low cadence but high monetization, and so you shouldn’t look at, DAU/MAU. And so you have to find really the right set of metrics that show that you’re providing value to your customers first and foremost and then really build your team and your product roadmap and everything in order to reinforce that.

Find the loops and the networks that exist within your product because those are the things that are gonna keeps your engagement improving over time even in the face of competition.