“Houston, we have a problem.”
These famous words were spoken by Jim Lovell, the Commander of the Apollo 13 space mission.
The Apollo 13 mission was slated to be the third manned trip to the Moon, but the team faced a massive crisis when an oxygen tank exploded in space, making the spacecraft unable to complete its mission. This extremely high-stakes and high-pressure scenario had no traditional, out-of-the-box solutions. The astronauts and engineers had to think creatively and innovate to get the spacecraft safely back to Earth.
Luckily, in data science, we do not have to innovate in an unforgiving environment of life-and-death situations. However, we are often asked to creatively adapt to real-world constraints where there is no perfect solution. This blog series explores two real-world experimentation scenarios your team might face, and how to navigate them when traditional methods fall short.
In Part 1, we’ll look at what to do when you can’t create a traditional control group, and how to still make reliable causal inferences despite that constraint.
Including a control group in your experiment is important to establish a baseline for comparison and to account for external factors that enable causal inference. Ideally, the control group should be similar to your treatment group(s) in all aspects, except for the experimental intervention. This setup allows you to make stronger cause-and-effect claims because the only potential cause of any observed difference is the treatment.
However, there are some real-world scenarios where recruiting a suitable control group is simply not possible.
The Scenario: Your company is rolling out a loyalty rewards program, offering discounts to frequent customers.
The Problem: Here, a control group would be impractical because of two main reasons: self-selection bias and an unavoidable interaction.
Self-selection bias occurs because customers who opt into loyalty programs are not randomly selected, making it difficult to create a true control group of customers who haven’t joined the program but are otherwise similar. Pharmaceutical drug trials also deal with this issue. While people cannot self-select into having the disorder being treated, they do self-select to join the experiment pool.
Secondly, the existence of the loyalty program may indirectly influence non-participants (e.g., via word of mouth), a phenomenon known as a spillover effect.
PSM provides a way to calculate the average treatment effect of an intervention by making treated and untreated groups comparable on observed characteristics (e.g., demographics, previous purchase history, online behavior, etc.).
What this means in practical terms: Creating a fair comparison.
With PSM, you can unlock new analytics capabilities. This method helps identify similar customers across different segments (e.g., loyalty tiers) while accounting for a multitude of characteristics. It creates a fair comparison in your data between groups that may come from diverse customer segments, channels, or campaigns.
These are the basic steps for PSM:
RDD can estimate causal effects when assignment is determined by a clear cutoff value (e.g., a customer must hit a spending threshold to be eligible for the loyalty program). Researchers compare units just above or just below the cutoff to determine the treatment effect. The key assumption is that units near the cutoff are similar; the only difference between them can be attributed to the intervention.
What this means in practical terms: Finding critical thresholds.
RDD can answer questions about the effectiveness of your segmentation and targeting strategies. For example, it can reveal whether loyalty program tiers drive meaningful behavioral changes. It also helps optimize budget allocation by identifying points of diminishing returns and opens new frontiers by identifying customers just below key thresholds who could be nudged into higher-value behavior with minimal investment.
These are the basic steps for RDD:
When a suitable control group doesn't exist, SCM allows you to create a comparable “synthetic” control group using a weighted combination of other units.
What this means in practical terms? Quantifying the unquantifiable.
SCM helps quantify the unquantifiable, enabling you to measure the impact of broad campaigns or market changes where a true control group is impossible. It also supports compelling storytelling by visualizing what would have occurred without your intervention.
These are the basic steps for SCM:
Alternatives to traditional A/B testing offer varying levels of causal inference depending on the nature of the data and the specific business context. Each method discussed above, and additional ones not mentioned here, come with their own set of assumptions, nuances, and limitations. However, when applied correctly, they can be incredibly powerful methods for drawing valid conclusions from data. Selecting the right approach can allow you to gain meaningful, actionable insights that drive informed decision-making.
Our statistical and experimentation experts bring years of hands-on experience in advanced data analysis methods with tailored solutions for your specific business needs. When traditional methods fall short, our team of advanced analytics SMEs step in to provide reliable, causal insights that drive better decision-making.
Not sure on your next step? We'd love to hear about your business challenges. No pitch. No strings attached.
