How do you handle multivariate testing or sequential testing scenarios?

One practical example: Let's assume your company offers great quality for a great price. "Quality" and "Low Price" might be two Independent factors which are both important to your audience. In a multivariate test (MVT) you could create different versions of elements that show a different intensity of quality whily other elements show different price signals. Running an MVT is best practice for identifying tradeoffs between different factors but it rquires high sample sized to eliminate statistical errors.

The best way to understand multivariate testing is by the limitations it overcomes. Let's say you're planning a really big AB test, where you've developed a brand new page and you want to send half of your traffic to it. How do you know what aspect of that new page is responsible for the improvement? And worse - what if one aspect of the page is single-handedly responsible for dragging down the performance of your experiment? These are not things that surface from a normal AB test that you run, but it's where MVTs shine. By breaking down your experiment into it's component parts, you get to point to specific changes as responsible for a proportion of the improvement, and specific changes as dragging the experience down.

Multivariate testing, like great leadership, explores the power of combinations and teamwork. It's not just about testing different elements, but understanding how they interact and complement each other. Like a team with diverse skills, each variation - be it a headline or image - contributes uniquely. The goal is synergy, where the collective impact maximizes outcomes like conversions or engagement. It's a journey of discovery, finding the optimal blend that resonates with your audience, much like finding the right team dynamic in a successful organisation.

Multivariate testing goes beyond A/B testing by simultaneously evaluating multiple elements and their interactions. It's like a science experiment for your website, allowing you to discover the synergistic effects of different design or content variations. Advantages of Multivariate Testing: Unveils hidden interactions: It exposes how elements influence each other, revealing unexpected winners or losers. Optimizes for the whole experience: It goes beyond single elements, finding the best overall combination for your goals. More efficient than individual A/B tests: It tests multiple variations simultaneously, saving time and resources.

Testing approach to take depends on the project requirement and priorities. In general, MVT answers the “what ifs” of creative testing. It lets you test numerous creative elements simultaneously, discovering optimal combinations faster. With MVT, you can explore which headlines work best with specific images and calls to action. Automation makes MVT practical, as it handles the complexity of testing multiple variables at once. Sequential testing is methodical and controlled, but it may miss synergistic effects between elements. This type of testing provides detailed insights but at a lower pace.

Art of A/B testing! In the enchanting realm of data-driven decisions, multivariate and sequential testing weave a spell of optimization. Multivariate testing, my dear, is like a grand symphony, orchestrating multiple variables to uncover the most harmonious user experience. Sequential testing, a more agile approach, gracefully adjusts the sample size as data unfolds, minimizing precious time and resources. Both these methods, when wielded with finesse, empower businesses to navigate the labyrinth of choices, identify the winning combinations, and craft truly magical user journeys.

When you're doing multivariate testing, try juggling different elements at once but keep it manageable—focus on each part so you can see what's really making a difference. If you're doing sequential testing, it's like taking things one step at a time, good for rolling out changes gradually or when you've got limited resources. Whatever you're testing, make sure you've got clear goals and pick the right things to measure. Give each variation enough time and visitors to get reliable results, and learn from each step to make the next one even better. Oh, and don't forget to jot down everything you learn so you can use it to improve things later on!

Define specific hypotheses for each variation to ensure clarity in what you're testing and why. Use a robust statistical analysis tool to accurately measure the interaction effects between variables in multivariate tests. For sequential testing, carefully plan the order of tests to avoid carryover effects and ensure tests do not influence each other's outcomes.

Handling multivariate testing or sequential testing scenarios requires careful planning and execution. For multivariate testing, I identify multiple variables to test simultaneously, ensuring each combination is statistically significant. I use specialized tools to manage and analyze complex experiments efficiently, such as factorial designs or Taguchi methods. For sequential testing, I implement methodologies like Bayesian testing or sequential probability ratio testing (SPRT) to make interim decisions while minimizing sample size and time. Continuous monitoring and adjustment based on interim results help optimize testing efficiency and accuracy.

Actually, sequential testing isn't about testing one variant at a time. Rather, sequential testing is an approach that moves the decision boundaries every day or so instead of having a fixed one. The decision boundaries are fairly hard to cross early on during a test so that only test with very obvious winners are called. As the test runs, the boundaries get tighter and higher - generally resulting in a shorter runtime than standard frequentist testing.

I think this is a misperception of sequential testing. Sequential testing isn't about testing one variation at a time. It's a way to still use Frequentist statistics (aka p-values that control for false positives) for experimentation without requiring a fixed horizon. That is, you are allowed to "peek" and stop a test early if you have a win or a loss. Sequential testing spreads your error control across each analysis instead of saving it all up for the final analysis at the end. It is designed to guarantee the same error control (ie - if 95% is your goal - you still have 5% guaranteed false positive rate if you "peek" 4 times by checking with 4 analyses each week of a 4-week run time).

Sequential testing, like effective decision-making, focuses on adaptability and responsiveness. It's about evaluating one variation at a time, mirroring how we tackle challenges step by step. The goal is efficiency - to identify what works or doesn't quickly, akin to agile leadership that pivots with changing circumstances. This method, like good leadership, is dynamic, making decisions based on real-time evidence rather than rigid plans. It's a reminder that sometimes the best path forward is to assess, adapt, and act as you go.

Benefits of Sequential Testing: Faster results: No need to wait for a set duration, potentially reducing testing time significantly. Data-driven decisions: Stops based on evidence, not arbitrary timeframes, leading to more informed choices. Resource efficiency: Tests fewer variations in total, saving resources for other optimizations. Challenges to Consider: Increased risk of errors: Early stopping can lead to false positives or negatives if not done carefully. Requires sophisticated analysis: Continuous analysis demands robust statistical methods and expertise. Less interpretable results: Understanding long-term impact might be harder compared to traditional A/B testing.

Traditional hypothesis testing was built with the idea that you will collect your entire dataset and then run the statistical analysis only once. However, this is very inefficient in modern day A/B testing. Modern A/B tests prefer calculating statistics on the fly as data is being collected and give a decision as soon as the defined threshold is crossed. This method of continuous monitoring of statistics and early stopping are collectively referred to as Sequential Testing. However, sequential testing leads to an increase in false winners. Hence, there is a need for proper statistical corrections to adjust for the increase in error rates. If you are doing sequential testing, make sure to use sequential statistics with it.

Sequential testing is a type of experimentation that involves testing multiple variables in a series of steps, with each step building on the previous one. It's used to refine and optimize experimental designs by iteratively testing and refining hypotheses based on the results of previous tests.

Factorial testing makes sense if you have done a radical redesign and want to quickly and efficiently find the most important elements (main effects) on the page for further optimization and where they might interact with each other. It can also be a powerful way to identify interactions with concurrently running experiments if multiple experiments are setup as a factorial design. To be clear, however, factorial design (sometimes called MVTs) are not the opposite of sequential design.

Multivariate testing is like exploring a complex puzzle, where each piece - an element or factor - plays a crucial role. It's about understanding interactions and their collective impact on user behavior. This approach helps in crafting the most resonant design or content, tailored for your audience. However, it demands more - more traffic, more time, akin to a leader needing more resources for a larger vision. Use it when conditions are right: high traffic, a clear hypothesis, and a defined goal. It's a strategic choice, aligning resources with objectives.

Think of your website or app like a sandwich. A/B testing is like testing two different sandwiches—maybe a ham sandwich vs. a turkey sandwich—to see which one people like more. But what if you want to know the effect of changing the bread, meat, and/or sauce, so that you can discover the winning combination of factors (and the best sandwich)? That’s where multivariate testing comes in. Use multivariate testing when you want to test multiple elements at once and understand how they interact to affect your overall goal—like conversions or engagement. Keep in mind though, multivariate testing is often more complex than regular A/B testing.

It is important to ensure the right sample size because each variation adds the possibility for a statistical error. Example: If you test 2 factors with 4 variations, you end up with 16 combinations + control version. All the traffic will be split between these 17 variations so you need a big sample size to identify the statistical differences between the variations. Make sure you use a proper sample size and / or runtime calculator to make sure you don't run into statistical pitfalls.

Use multivariate testing when you want to understand the complex interactions between multiple variables and their impact on a particular outcome. This is particularly useful when you're trying to identify the most effective combinations of variables or when you're dealing with a large number of variables.

You would use sequential testing if you pre-calculated your fixed-horizon runtime and found a runtime that was longer than your available window. Or perhaps you felt strongly that you could beat the estimated lift/MDE. Or - maybe, you just wanted to be able to peek or had an organization that needed to be able to pull the plug quickly if it wasn't working - or wanted to implement quickly if it was a win. All of these are good use cases for sequential design. But to be clear - these are not opposites of factorial / MVT tests. They are as opposed to a fixed-horizon test.

Use sequential testing when you're trying to refine and optimize an experimental design over time. This is particularly useful when you're dealing with a complex problem or when you're trying to test multiple hypotheses.

It's really important in a multivariate test that it's setup correctly in order to be able to interpret the results. Often, I see tests which are intended to be multivariate setup as a univariate test with multiple changes present in a single variant. Instead, you need to make sure you have a variant for each combination of variables. In a test with two different sets of copy and two different images you would need 4 variants. Something to watch out for is that the number of combinations increases very quickly as you add variables and thus your available sample gets diluted across all those combinations. If your available sample is very large, this may not be a problem. However it might cause issues if you tend to have limited samples.

Ultimately, the human is making the decisions and the human will be held responsible for them. So the human has to use the method they can defend best, regardless of the outcomes, for it to actually work.

To handle multivariate testing, use a robust design that accounts for potential interactions between variables, and use statistical methods like regression analysis or machine learning algorithms to identify relationships between variables.

To handle sequential testing, use a phased approach, with each phase building on the previous one, and use techniques like Bayesian updating to refine your hypothesis based on new data.

Begin by establishing a baseline and then systematically introduce variations, monitoring their impact on key metrics. Use statistical methods like Bayesian analysis or sequential hypothesis testing to make informed decisions about when to stop or continue testing each variation.

In the "frequentist framework", when you check a test result multiple times and stop based on what you see, you MUST require stronger evidence before declaring a win. Otherwise, you’re giving yourself more "bites at the apple" and inflating your false positive risk. The beauty of frequentist methods is they require almost no assumptions and p-values are easy to interpret, but the p-value is designed for "one test only". A good rule of thumb: multiply your p-value threshold by the number of "peeks" you plan. For example, if you look once each week for 5 weeks, and stop anytime you achieve a certain significance level, then you should require p < 0.01 to maintain a true 5% false positive rate.

Something to consider when it comes to multi-variate testing is whether or not it's truly necessary or if a univariate test would deliver the necessary data. For example, if you're redesigning a landing page with a lot of different elements, you could set this up as a multi-variate test with each element as a variable and lots and lots of combinations of variables. The alternative would be to test the new page against the old one as a single variant. You avoid a lot of complexity in your test and still learn if your new page works better. What you would miss out on is understanding why the new page works better. The question is; does it matter?