Designing the future of A/B testing + shaping AI strategy at Optimizely
Product design
B2B SaaS
Master's project sponsored by Optimizely
CLIENT
UW Master’s
project sponsored by Optimizely
TIMELINE
Feb - August 2025
TEAM
1 UX researcher
3 product designers
MY ROLE
Product design
UI + visual design
Design strategy
OVERVIEW
Context
Optimizely is a B2B SaaS startup that helps marketing teams improve performance through a suite of products designed to support their workflow. One such product is Web Experimentation (WebEx), an A/B testing platform where marketers can test digital experiences, measure uplift, and optimize websites for higher conversions.
With nearly 1,300 customers, WebEx is a core offering, yet it has a 65% retention rate, the lowest of any Optimizely product. Many customers who purchase it struggle to get started - 75% don’t meet the minimum “success” bar of running two statistically significant tests per quarter, and 82% of customer churn comes from this underperforming group.
Our team was approached by two product managers at Optimizely to evaluate and improve the first-touch onboarding experience for WebEx customers. Shortly after project kickoff, Optimizely launched Opal, an LLM built into the WebEx platform. Following the launch of Opal, priorities at Optimizely shifted, as did our project scope. Our stakeholders asked us to envision a “next generation” version of WebEx where AI plays a central role in helping customers ramp up and achieve early success with A/B testing on the WebEx platform.
OVERVIEW
Solution
We created WebEx 2.0, an AI-powered A/B testing platform that makes it easy for any team, regardless of experience level, to run impactful tests, scale their experimentation program, and drive meaningful business growth. WebEx 2.0 leverages Opal AI to provide users with personalized, contextual help during A/B test setup, ensuring they feel confident throughout the process and that their experiments follow best practices for capturing relevant, quality data and reaching statistical significance.
Meet Opal: the proactive AI assistant
In the original WebEx platform, Opal functions as a passive, chat-based LLM. The few contextual AI features within original WebEx focus on pre-setup test idea generation and post-setup results interpretation, not the test setup process itself. WebEx 2.0 incorporates Opal across all stages of test setup, speeding up the process while proactively guiding users through experimentation best practices.
Show, don’t tell: unveiling the connections during experiment setup
In the original WebEx platform, each A/B test element - from traffic allocation to audiences to metrics - is hidden in its own tab, making it difficult for users to understand connections between them and double check their full test setup at a glance. WebEx 2.0 offers an improved information experience for users by consolidating test elements in an ordered, interactive node layout paired with an editable website preview showing their experiment variations.
Launch confidently: checkpoints for every step
In the original WebEx platform, there isn’t any guidance or feedback provided around whether tests are ready to go live. WebEx 2.0 features an intuitive experiment preview experience, a launch readiness checklist, a test duration estimator, and a shareable experiment summary to ensure users can launch with confidence.
Going beyond: turn one win into many
In the original WebEx platform, tests end without guidance for next steps. WebEx 2.0 leverages Opal to suggest follow-up test ideas based on the results of concluded experiments, allowing users to easily iterate for continuous impact and by extension, scale their experimentation program.
OVERVIEW
Impact
We presented WebEx 2.0 to 20+ stakeholders at Optimizely across product, design, engineering and research and handed off a high-fidelity prototype using Optimizely’s design system. Our work has since influenced AI strategy for Web Experimentation, inspiring shipped Opal-based features like the “Experiment QA Agent” which allows users to double check the quality of their test setup before it goes live and get suggestions for improvement.
“I’ve been so impressed with this team. We’ve taken a ton of this work and already started implementing it which I think is the biggest sign of success. [Customers] can use our tool appropriately to build a really bad test, which is a pitfall that we run into a lot. So a lot of the work that this team did and recommendations that they gave us are already being folded into Opal. Today we are launching the first beta of the Optimizely Experiment QA Agent, which is the feature they talked about that gives warnings and updates and helps [the user] build a really good test. It is being built into Opal and going into our beta today.”
-Britt Hall, Vice President of Product, Digital Optimization
OVERVIEW
My contribution
As a product designer, I drove the design direction of WebEx 2.0, collaborating cross-functionally with research and product management to ensure designs were rooted in real user and business needs. I owned the design of the AI powered experiment setup flow as well as the post-results “get follow-up test ideas” feature.
Design strategy
I helped translate initial research insights and stakeholder priorities into a clear product vision that was used as a roadmap to guide decisions throughout the design phase of the project.
Rapid prototyping
I rapidly designed and iterated on 10+ interactive prototypes at varying stages of fidelity, incorporating feedback from stakeholders, subject mater experts, and user testing to ensure resonance with our target audience, experimenters new to WebEx.
Visual + UI design
As the sole team member with past experience in visual design, I mentored teammates on design best practices and oversaw the implementation of Optimizely’s design system, Axiom, across the entire high fidelity prototype, driving correct and consistent usage.
User research
I participated in 3 rounds of user testing and 12 SME interviews. I moderated concept + usability tests, recruited participants, synthesized data, and helped craft research study plans.
PROCESS
Coming soon! 🛠️
This case study is a WIP - check back soon for more about my design process!