Skip to main content

Skill Guide

User Research & A/B Testing

User Research & A/B Testing is the systematic practice of gathering qualitative and quantitative user insights to inform product decisions, and then using controlled experiments (A/B tests) to validate those decisions with statistical rigor.

It directly ties product development to user needs and measurable business outcomes, minimizing costly guesswork. Mastering it enables organizations to de-risk innovation, optimize conversion funnels, and build user-centric products that drive sustainable growth.
1 Careers
1 Categories
9.2 Avg Demand
30% Avg AI Risk

How to Learn User Research & A/B Testing

Focus on: 1) Core research methods (surveys, user interviews, usability testing) and their appropriate use cases. 2) Foundational statistics (sample size, significance, p-value) to understand test validity. 3) The standard A/B testing workflow: hypothesis → variant design → traffic allocation → result analysis.
Move from theory to practice by: 1) Designing and executing end-to-end research plans for a specific feature or user journey. 2) Running A/B tests on low-risk elements (e.g., button color, copy) to understand platform mechanics and common pitfalls like sample ratio mismatch or novelty effects. 3) Integrating quantitative A/B test data with qualitative user feedback to form a complete user narrative.
Master the skill by: 1) Architecting a research and experimentation strategy that aligns with multi-quarter business OKRs. 2) Managing a portfolio of concurrent experiments across the product lifecycle, balancing speed and rigor. 3) Developing advanced statistical methods (sequential testing, Bayesian approaches) for complex scenarios. 4) Mentoring junior researchers and evangelizing a data-informed culture within the organization.

Practice Projects

Beginner
Case Study/Exercise

Optimizing a SaaS Pricing Page

Scenario

Your company's new SaaS product has a 3% trial-to-paid conversion rate. The pricing page is identified as a key drop-off point.

How to Execute
1. Conduct 5-7 moderated usability interviews with trial users to identify confusion points (e.g., unclear value proposition, complex plan comparison). 2. Based on findings, form a hypothesis: 'Simplifying the pricing grid to highlight the 'Recommended' plan will increase click-through to checkout by 15%.' 3. Design a simple A/B test: Control (current) vs. Variant (simplified grid). Use a platform like Google Optimize to run the test for 2 full business cycles. 4. Analyze results for statistical significance and document learnings.
Intermediate
Project

Onboarding Flow Experimentation

Scenario

User activation (completing key setup steps) within the first week is 40%. You need to improve this metric.

How to Execute
1. Analyze quantitative data (funnel drop-offs) to pinpoint the largest friction point. 2. Conduct qualitative 'think-aloud' usability tests on that specific step. 3. Develop multiple hypotheses (e.g., progress bars, tooltips, simplified form). 4. Design and run a multi-armed bandit or A/B test to compare 2-3 competing variants against the control. 5. Measure impact on activation rate AND downstream retention (7-day, 30-day) to avoid short-term wins that hurt long-term engagement.
Advanced
Project

Experimentation Program for a Fintech App

Scenario

As the Head of Product, you are tasked with increasing user trust and loan application completion rates in a regulated, high-stakes environment.

How to Execute
1. Establish an experimentation framework with clear ethics and compliance guidelines for testing in finance. 2. Prioritize a backlog of experiments using an ICE (Impact, Confidence, Ease) score model, focusing on trust signals (security badges, social proof) and simplifying complex forms. 3. Implement a sophisticated testing platform with proper segmentation (new vs. returning users) and guardrail metrics (e.g., must not decrease support tickets). 4. Run sequential tests and use Bayesian analysis to make faster decisions with smaller samples. 5. Create a 'learning repository' and present quarterly insights to leadership, showing the cumulative revenue impact of the experimentation program.

Tools & Frameworks

User Research Tools

UserTesting (moderated/unmoderated testing)Qualtrics/SurveyMonkey (surveys)Maze (usability testing & analytics)Dovetail (research repository)

Used for gathering qualitative insights at scale. UserTesting for recruiting and testing, Maze for rapid prototype testing, Dovetail for synthesizing and storing research data.

A/B Testing & Experimentation Platforms

OptimizelyVWOGoogle OptimizeLaunchDarkly (feature flagging)

Platforms for implementing, managing, and analyzing controlled experiments. Optimizely and VWO are enterprise-grade; Google Optimize is accessible for web; LaunchDarkly is used for feature-level testing and gradual rollouts.

Analytics & Statistical Tools

Amplitude / Mixpanel (product analytics)Google Analytics 4Statsig (experimentation analytics)R/Python (for custom analysis)

Amplitude/Mixpanel for deep funnel and cohort analysis. GA4 for web traffic and basic testing. Statsig for advanced experiment analysis. R/Python for statistical modeling when built-in tools are insufficient.

Mental Models & Methodologies

Jobs-to-be-Done (JTBD) FrameworkDouble Diamond Design ProcessStatistical Hypothesis TestingExperimentation Backlog (ICE Score Prioritization)

JTBD to uncover core user needs. Double Diamond to structure the research-to-ideation process. Hypothesis Testing to ensure scientific rigor. ICE Scoring to objectively prioritize which experiments to run next.

Interview Questions

Answer Strategy

Test for structured, rigorous thinking. The candidate must outline the full lifecycle. Sample Answer: 'First, I'd define a clear hypothesis tied to a metric, e.g., 'Changing the button from grey to green will increase checkout click-through by 5%.' I'd ensure proper randomization and control for segments (mobile vs. desktop). I'd calculate the required sample size for 95% confidence and 80% power. After running the test for at least one full business cycle to avoid day-of-week effects, I'd analyze the primary metric and check guardrail metrics (e.g., revenue per user didn't drop). If the result is significant, I'd roll it out; if not, I'd analyze secondary data or user feedback for why.'

Answer Strategy

Test for synthesis skills and intellectual humility. The candidate should show they can reconcile quantitative and qualitative data. Sample Answer: 'In a test, we saw a 10% lift in clicks from a new, more aggressive upsell banner, which aligned with our hypothesis. However, user interviews revealed it was perceived as 'spammy' and eroded trust. I stopped the rollout. We used the quantitative data to identify the high-performing element (the offer itself) and the qualitative data to redesign its presentation (subtler placement). The revised version passed both click-through and sentiment checks.'

Careers That Require User Research & A/B Testing

1 career found