June 13, 2024

Rachel Salvador

Behind the metric: how we developed “Time Spent Learning Well”

We develop proprietary metrics to help us improve our product!

At Duolingo, we’ve developed a lot of proprietary metrics that help us improve our product in very specific and unique ways. One of the most important metrics that we created is Time Spent Learning Well (TSLW).

This isn’t just any metric—it’s an essential quality metric that helps us optimize for learning alongside growth. TSLW has evolved over several years to become the version we use today—let’s look at how we got here, what it measures, and how we use it!

How we arrived at TSLW

Since Duolingo is a learning app, we had to find creative ways to approximate teaching efficacy. Our Efficacy Lab conducts rigorous research to measure the efficacy of Duolingo, but these studies aren’t helpful in conducting A/B tests on a daily basis. It’s extremely important that we have a proxy metric for learning, because that’s what we’re here to do! So how did we land on Time Spent Learning Well?

Total Sessions

We started with Total Sessions, which measured the total number of sessions people were completing (from a path lesson to a Match Madness round to a quick Story review). The idea was more sessions = better for learning. If you do more, you learn more, right? Not quite!

We found that Total Sessions was an imperfect metric because session length (time spent doing the activity) was very variable. We wanted learners to advance through their course and encounter harder content, and naturally, harder lessons might take longer. So if a learner has 15 minutes a day to spend doing Duolingo, that might be 5 easier sessions or 3 harder sessions. In this case, fewer sessions was not necessarily a bad thing for the learner! The metric wasn’t accurately measuring engagement or learning—we were biased towards people who were grinding on shorter, easier sessions.

Time Spent Learning

Next, we evolved to Time Spent Learning (TSL). This focused on time learners spent in learning activities, which includes: regular lessons, timed challenges, review sessions, lessons focused on conversation or listening practice, stories, etc. The challenge: we had to figure out how to make it relatively resistant to outliers.

In the beginning, Total TSL ended up being skewed to a small set of studious learners. And if we changed features that impacted competitiveness—like Leaderboards—Total TSL grew mostly due to these very active learners. That didn’t help us reach our goals because a) we were motivating a small group of learners.

We decided to develop a “standard” for how much time each learner should spend on Duolingo—after conducting several formal studies, discussions with our internal learning and curriculum experts, and assessing current learner behavior, we decided to optimize for a higher percentage of learners spending at least 15 minutes/day on Duolingo. Learning a new language takes time, and the more time you spend practicing, the better you’ll get. If you only spend 5 minutes a day on Duolingo, it will take you a really long time to reach your goals.

Once we looked at how each learner was spending their 15 minutes/day, we realized not all learning time is equal—after all, lessons on the path are the ones that teach you new concepts, skills, and vocabulary!

A path lesson is any lesson node that moves you down the path, including: personalized practice, Stories, and Unit Review.

Time Spent Learning Well

Once we realized that, we landed on Time Spent Learning Well. This metric favors certain sessions that have better impact on learning—particularly those that move you down the Duolingo path. An independent study from researchers at Northern Arizona University and East Carolina University found that “the number of completed lessons was the strongest predictor of learning gains.” While all of the exercises and activities on Duolingo have value, we found that most of the time, for most learners, going down the path is the most helpful. This is how learners encounter new and harder content, as well as the review that we’ve sprinkled throughout lessons! So when determining the formula for TSLW, we knew we had to pay special attention to the value of those lessons.

The rough formula for measuring TSLW is:

How we influence TSLW

Since we wanted learners to move down the path, we had to find ways to incentivize them to do so! Here are a few features we found positively impact TSLW (and keep the app entertaining):

Daily Quests: These are optimized very carefully for TSLW. The order of the Quests matters! The first is supposed to “get you going” and is usually the “easiest” to complete. They get harder and harder, and typically value activities that move you forward, i.e.: finish a unit or read the next Story on your path.

Monthly Challenges: To encourage good learning habits, we switched the Monthly Challenge to be Quest-based instead of XP-based. We wanted to encourage people to return daily and progress through different activities.

Leaderboards: This is one of our most popular features in the app, but we wanted to make sure it incentivized the best learning behavior. We’ve increased XP along the path so that XP rewards are proportionate to effort and learning outcomes. With this change, lessons on the path help our learners climb the leaderboard!

Testing everything

At Duolingo “test everything” is one of our operating principles—and we’re constantly iterating on different features to see how they impact our most important metrics! Here are a few things we learned about TSLW while testing:

Shorter isn’t always better… We tried out making lessons shorter. Overall, this wasn’t the case, and ultimately the shorter sessions hurt TSLW.

…But longer seems to be! We did find that adding new content always had a positive impact on TSLW. We hypothesize that learners enjoy making progress and engaging with fresh content.

Avoiding learner burnout We know that not everything on Duolingo is going to positively affect Time Spent Learning Well… and that’s OK! Our philosophy is: “they can’t learn if they churn,” which means we all need a break every now and then.

Keep learning!

The most important thing about TSLW is the “L”—above all, we want to make sure learners advance towards their goals! We’re still refining TSLW, but we’re confident that metrics like this help Duolingo move towards its mission of developing the best education in the world and making it universally available 🎉