Using Experiments to Launch New Products
Martyn Goddard/Getty Images

More than ever before, managers are using large-scale randomized controlled trials (i.e., experiments) to guide decisions. This has led to impressive gains for organizations ranging from Amazon to the UK government. We are excited about the rise of experiments in organizations and have spent much of the past few years thinking about how to design and interpret them. At the same time, we’ve seen that experimentation remains uneven across and within organizations, and many companies struggle with ways to start or expand experimenting.

One simple and often overlooked way for larger companies to experiment is to randomize the introduction of new products across a set of markets. To see how this can be valuable, consider how Uber rolled out its Express Pool service in 2018.

At the time, the company was already running UberPool, a service that allows passengers heading in the same direction to share rides and costs. With UberPool, passengers are picked up and dropped off wherever they like, as with other Uber services. But with the Express Pool service, which costs even less than UberPool, passengers are generally asked to walk short distances to meet their rides and to reach their destinations.

In 2018, in the run-up to the launch of Express Pool, Uber tasked one of us (Duncan, who manages a group of economists and data scientists within the company) with assessing how likely it was to succeed. How many riders would opt in, and how would the service affect the broader—and more complex—Uber ecosystem?

To answer those questions, Duncan and his team conducted an experiment, launching Express Pool in six large markets and then comparing metrics in the launch cities with those in others.  Leveraging recent advances in experimental methods—especially a statistical method that allowed Uber to use a weighted combination of other cities to form a more-suitable “synthetic” control group—the team was able to tease out the ways in which the rollout was influencing Uber usage. Unsurprisingly, Express Pool created new kinds of trip matches. But the experiment also accounted for the effect that Express Pool had on existing Uber products and made clear that launching it would make good business sense. As a result, Uber was able confidently to introduce Express Pool to many of its major markets.  This confidence, and the finding that inspired it, would not have been possible without the experiment.

Online marketplaces now abound, ranging from Uber and Airbnb to Rover and Tinder. And Uber is not alone among these companies in turning to market-level experiments to test new products and innovations.

Airbnb (where Jeff used to work as a data scientist) recently ran an experiment to test the impact of a new landing-page design on search-engine ranking and traffic. To run the experiment, Airbnb exploited the fact that it had landing pages with different URLs for different markets (San Francisco, Boston, New York, etc.). This meant that they could randomize the different URLs to include the new design or not, thereby isolating the design’s effect on search-engine traffic. And by doing that they were able to show that the new design was a success: the new landing page, it turned out, was driving a ~3.5% increase in search traffic, an improvement corresponding to tens of millions of incremental visitors per day for the platform. Based on these findings, Airbnb launched the new design for all markets.

It’s not just tech companies who can use large-scale experiments to test new products and innovations. Consider what a restaurant chain might do when deciding whether to offer a new turkey-avocado sandwich. One traditional approach to a decision like this might be to roll out the sandwich in a couple of strategically chosen stores, run some focus groups, and study historical sales of other products. If people seem to like the sandwich in those stores, the business could roll it out in all of its stores and hope that it will succeed nationally. This type of approach would provide insight into the issue, but it has significant limitations. For example, it would be hard to know if the new sandwich crowded out other purchases. And it would be challenging to see whether this increased the overall number of customers. If the chain were to complement this approach with a large-scale randomized trial, by rolling the sandwich out to a set of randomly selected markets, they could learn much more about the effects that adding the sandwich might have broadly on sales (both for new and existing products), customer retention, and customer satisfaction.

We’ve seen companies miss important opportunities for experimentation, and we’ve seen experiments that suffer from implementation and interpretation challenges.  For companies looking to test new products experimentally, here are some guidelines for getting started:

1) Decide what metrics matter to you most, and then come up with hypotheses about how they might behave. Invest in data collection and decide up-front what experimental outcomes will constitute success or failure. Create a map from data to decisions. Remember, it’s great if more people buy a product, but not so great if this leads to more customer support calls.

2) Choose a random subset of markets (e.g., regions, cities, or franchises) in which to launch the product. The results of market-level experimentation are often noisy, so once you have a set of markets in mind, think carefully about whether you’ll be able to detect the effects that you’re hoping for. (Power calculations require a lot of assumptions but can help figure this out.)

3) Make sure to track not only whether your new product is working but also how its launch affects existing products. Express Pool on its own might look like a success, but if it doesn’t sufficiently grow the overall market for rides, it’s probably less valuable than it seems. Similarly, when Airbnb launches new products, the company needs to think about how bookings through existing products are affected. And when Starbucks rolled out its sous vide egg bites (if you haven’t tried them, we recommend you do!), it needed to consider not only sales of the egg bites but also whether they crowded out other menu items.

4) Make sure you understand why your product is succeeding or failing. Top-line metrics like revenue and sales don’t tell the whole story. Is the new product improving outcomes for some types of customers while harming them for others? Did the new product help one part of your acquisition funnel but hurt another? Do these moves align with your pre-experiment hypothesis? Understanding why a metric has moved can help you not only make a rollout decision but also understand how to innovate within a product space.
How to Create a Customer Referral Program That Works

Gaining the trust of your customers is one of the most important things you can do for your business. Why? First, you're creating positive customer experiences, which leads to customers coming back to your website.

There is another important reason why you should care. When your customers have faith in your products and services, they are more likely to refer you to a friend.

Nielsen reports that customer referrals are the No. 1 most trusted form of advertising. What does this mean for you and your business? It means that if you want to make the most of your company, you need a referral program that works.

Below I explain how you can develop a solid referral program and get more conversions for your business, as well as improve the public’s opinion of your company.

What is a referral program?

In a nutshell, a referral program is a system set up through your business that offers rewards to customers for sharing the good word about your business. The reward can be anything from a discount to exclusive content, and anything in between.

There are a number of ways you can incentivize customers to spread the word about your company. The most important thing to remember is you have to start with a killer customer experience every single time.

Make it easy.

The first and most important rule is you have to make it easy. If customers have to jump through a million hoops just to get a coupon, they are going to give up and leave. Considering that referrals can boost conversions by up to 70 percent, you can't afford to lose potential conversions because of a complicated process.

So, you know you have to make it easy to complete. Now you have to pinpoint your referral program rewards and figure out how to get your plan out there.

Figure out your reward system.

There are multiple ways you can reward your customers for referring their friends to your business. You have to figure out how much you make on each order and determine how much you can give the customers and still make a profit.

For example, if giving a 30 percent discount causes you to break even, you may want to drop down to a 15 or 20 percent discount. Regardless of the numbers, your customers will be appreciative that they are getting a discount on their order, especially if you follow the first tip and make it easy for them to refer their friends.

Most businesses do a standard percentage off discount. You can also offer points if your business has a points system that translates into money off an order. If the customer is only 100 points away from a free purchase, and you offer 500, you better believe they are going to refer their friend.

Another common way to reward your customer is with a free gift. If they refer a customer, they get a free gift with your next order. A free gift for the customer and their friend could convert them and cause them to make the referral.

How to implement your program.

Once you've decided to start a referral program, you have to determine how you're going to send out your referral proposition. The most common way to is to send emails to returning customers. You can do this by using an email marketing program to build a lead list of former customers.



When designing your email, you have to build a custom referral proposition that catches the attention of your customers. First, start with a catchy heading.

Once you’ve decided on a subject line, it's time to move to the body text. You should make it short and sweet. For example:

Hi, there! It's [Name] here from [Name of company]. We noticed that you recently made a purchase from us. Did you love your experience? Now is your chance to save 30 percent on your next order! Simply refer one of your friends today and you both get a coupon code for 30 percent off your next order.

It only takes a few minutes, and we want to make sure you don’t miss out on this limited time offer. Simply follow the directions below for your referral code.



It's that simple.


Now that you know what customers expect from a referral plan, you're ready to start your own program.

You'll want to carefully monitor your analytics and make sure that your email marketing is working. You may have to make some minor tweaks to your referral program over time. You can build a perfect system based on your business model with practice and patience.

Just remember to make your system easy for customers, clearly define your rewards and target your lead list. You'll soon have an explosion of customer referrals for your business.

5 Tips to Market Research like a Pro

Are you a business owner looking to increase your conversions? If so, one thing you need to consider is how you are gathering and implementing market research.  

In a nutshell, market research is the process of gathering information about your audience learn more about them. It involves taking time to pinpoint the problems, needs, and interests of the people who buy your products, and the people you want to purchase your product.

Creating a thriving business involves constant market research so you can develop new, helpful creations and update existing products. Below are five tips to help you start doing market research like a pro.

1. Develop Buyer Personas

One of the first things you have to do is discover more about your customers. If you don’t have a firm grasp on your audience, it’s hard to figure out what kind of people you should target in future ad campaigns.

There are plenty of tools available to help you develop customer profiles. As you fill out information about your customers, you’ll start to develop personas. These personas include important key information such as their interests, most common problems, and lifestyle trends.

You can use this information to your advantage. Part of market research is discovering a common problem and coming up with a solution to that issue. The more customer personas you develop, the more information you’ll have at your fingertips

2. Conduct A Survey

A problem that many new businesses face is they simply don’t have a lot of customer information to use for their own market research. Luckily, there is a solution to this problem. If you want to gather more information about the people who buy -- or may buy -- your product, consider adding a survey to your website.

Surveys are great options for your business website for multiple reasons. First, it allows you the chance to write a custom questionnaire that customers can take part in. Try to include a few questions with text boxes so they can type in their own response. You want as much information as possible so you can build successful, accurate profiles for your market research.

You can also use surveys as a means to get more sales. Some business owners find that adding the option to do a survey in exchange for a discount, such as 10 percent off the order, is more than enough incentive for customers to answer your questions. This is a win-win situation. You’ll gather more information about your customers AND get more sales.

3. Upload Your Lead List to Facebook Ads

Social media is an excellent tool that every business owner should use to promote their business. As it turns out, social media, specifically the Facebook Ad program, is a great way to gather market research information about your audience.

When you use Facebook Ads, there’s a feature called “Custom Audiences” that lets you import your marketing email lists. How does this help? Well, when you upload your email segments, Facebook automatically finds the profiles of the people on the list.

After Facebook has gathered all the profiles, it builds charts that display trends among the people you added to the list. You can see what kind of habits and interests these folks have and use it when conducting your market research. For example, you may learn that many of your customers use Youtube. If that’s the case, you can come to the conclusion that many potential customers use YouTube, making it an ideal place to advertise.

4. Research Your Competitors

Odds are, if you’re in business, you have competition. There are different levels of competition, and you can leverage bigger businesses that offer similar products by diving into their customer statistics and website

There are multiple websites that can help you by showing you relevant marketing research data on your competitors. Some programs let you look at statistics on how well content does on your competitor's websites. This kind of research can give you insight into the topics that have a chance of interesting your audience. As an example, if you have a website about pets and your competitor’s articles on pet care skyrocket above all else, you can bet that this is the kind of content your readers are going to want to see. Delivering excellent, relevant content to your audience is another part of solid market research.

5. Set Up Multiple Small Ad Campaigns

The final way you can enhance your market research is by doing small sample ad campaigns. Once you’ve gathered customer data, you can start to set up email marketing campaigns to gauge their response.

Set up your ad campaigns to target certain groups so you can figure out what portion of your audience is likely to click-through and make a purchase. You can break down your campaigns by age, annual income, internet habits, and more.

These small tests are pivotal in finding out what kind of marketing works with your audience. As you develop a rich marketing research file, you’ll be able to pinpoint the people most likely to buy your product or service.


Everyone needs market research. It is a time consuming, yet an important part of any business. You have to know and understand your audience if you want to grow your company, get sales, and build strong relationships with the people who buy your product.

As you continue to develop your research, you’ll discover that people are complex and diverse. They have specific wants and needs that are begging for solutions. If you use your data properly, you can build great products that truly help people.

Take these five tips and start to implement them in your market research. Gather data, keep notes, learn what people like and you could soon have a business more successful than you ever imagined.

No comments