ab-test-analysis

phuryn/pm-skills · pm-data-analytics/skills/ab-test-analysis/SKILL.md

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

Software EngineeringNo license★ 25853⑂ 2783commit 18468a9

Install skillView source on GitHub

What this skill does

A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

Context

You are analyzing A/B test results for $ARGUMENTS.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

Instructions

Understand the experiment:
- What was the hypothesis?
- What was changed (the variant)?
- What is the primary metric? Any guardrail metrics?
- How long did the test run?
- What is the traffic split?
Validate the test setup:
- Sample size: Is the sample large enough for the expected effect size?
  - Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
  - Flag if the test is underpowered (<80% power)
- Duration: Did the test run for at least 1-2 full business cycles?
- Randomization: Any evidence of sample ratio mismatch (SRM)?
- Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
Calculate statistical significance:
- Conversion rate for control and variant
- Relative lift: (variant - control) / control × 100
- p-value: Using a two-tailed z-test or chi-squared test
- Confidence interval: 95% CI for the difference
- Statistical significance: Is p < 0.05?
- Practical significance: Is the lift meaningful for the business?
If the user provides raw data, generate and run a Python script to calculate these.
Check guardrail metrics:
- Did any guardrail metrics (revenue, engagement, page load time) degrade?
- A winning primary metric with degraded guardrails may not be a true win
Interpret results:

| Outcome | Recommendation | |---|---| | Significant positive lift, no guardrail issues | Ship it — roll out to 100% | | Significant positive lift, guardrail concerns | Investigate — understand trade-offs before shipping | | Not significant, positive trend | Extend the test — need more data or larger effect | | Not significant, flat | Stop the test — no meaningful difference detected | | Significant negative lift | Don't ship — revert to control, analyze why |

Provide the analysis summary:

## A/B Test Results: [Test Name]

**Hypothesis**: [What we expected]
**Duration**: [X days] | **Sample**: [N control / M variant]

| Metric | Control | Variant | Lift | p-value | Significant? |
|---|---|---|---|---|---|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
| [Guardrail] | ... | ... | ... | ... | ... |

**Recommendation**: [Ship / Extend / Stop / Investigate]
**Reasoning**: [Why]
**Next steps**: [What to do]

Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.

Installation

Copy a source-pinned command for your client. You run it yourself.

Destination: .claude/skills/ab-test-analysis · pinned to the source commit

# Run from your project root
git clone https://github.com/phuryn/pm-skills.git .skillboard-tmp
git -C .skillboard-tmp checkout 18468a95b427e70e258b51389796367c6f684e7d
mkdir -p ".claude/skills"
cp -r ".skillboard-tmp/pm-data-analytics/skills/ab-test-analysis" ".claude/skills/"
rm -rf .skillboard-tmp

Review the source before running. This copies files into your project; it is not a one-click install and does not verify runtime safety.

New here? Step-by-step setup – from installing the agent to using the skill

Operating systemAgentI already use Claude Code

1
Install Git
Install Git with your package manager (Debian/Ubuntu shown; use dnf/pacman on others). Official download.
```
sudo apt update && sudo apt install -y git
```
2
Install Claude Code
Requires Node.js 18+. Install the CLI globally: Official docs. (command-line tool)
```
npm install -g @anthropic-ai/claude-code
```

Get the skill, pinned to its exact version

Open a terminal and run these steps (they clone the source, check out the exact commit, and copy the skill into place):

# Run from your project root
git clone https://github.com/phuryn/pm-skills.git .skillboard-tmp
git -C .skillboard-tmp checkout 18468a95b427e70e258b51389796367c6f684e7d
mkdir -p ".claude/skills"
cp -r ".skillboard-tmp/pm-data-analytics/skills/ab-test-analysis" ".claude/skills/"
rm -rf .skillboard-tmp

Destination: .claude/skills/ab-test-analysis

4
Use it in Claude Code
Open Claude Code in your project. It reads skills from the folder above, so ab-test-analysis is now available. Always review the source before using it – static checks cannot prove runtime safety.

Security & Audits

Scanner static-checks@0.1.0 · commit 18468a95b427. Static checks cannot prove runtime safety – review the source and the exact diff before installing. How checks work.

No static rules matched. This is not a safety guarantee.

ab-test-analysis

phuryn/pm-skills · pm-data-analytics/skills/ab-test-analysis/SKILL.md

Software EngineeringNo license★ 25853⑂ 2783commit 18468a9

Install skillView source on GitHub

What this skill does

A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

Context

You are analyzing A/B test results for $ARGUMENTS.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

Instructions

Understand the experiment:
- What was the hypothesis?
- What was changed (the variant)?
- What is the primary metric? Any guardrail metrics?
- How long did the test run?
- What is the traffic split?
Validate the test setup:
- Sample size: Is the sample large enough for the expected effect size?
  - Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
  - Flag if the test is underpowered (<80% power)
- Duration: Did the test run for at least 1-2 full business cycles?
- Randomization: Any evidence of sample ratio mismatch (SRM)?
- Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
Calculate statistical significance:
- Conversion rate for control and variant
- Relative lift: (variant - control) / control × 100
- p-value: Using a two-tailed z-test or chi-squared test
- Confidence interval: 95% CI for the difference
- Statistical significance: Is p < 0.05?
- Practical significance: Is the lift meaningful for the business?
If the user provides raw data, generate and run a Python script to calculate these.
Check guardrail metrics:
- Did any guardrail metrics (revenue, engagement, page load time) degrade?
- A winning primary metric with degraded guardrails may not be a true win
Interpret results:

| Outcome | Recommendation | |---|---| | Significant positive lift, no guardrail issues | Ship it — roll out to 100% | | Significant positive lift, guardrail concerns | Investigate — understand trade-offs before shipping | | Not significant, positive trend | Extend the test — need more data or larger effect | | Not significant, flat | Stop the test — no meaningful difference detected | | Significant negative lift | Don't ship — revert to control, analyze why |

Provide the analysis summary:

## A/B Test Results: [Test Name]

**Hypothesis**: [What we expected]
**Duration**: [X days] | **Sample**: [N control / M variant]

| Metric | Control | Variant | Lift | p-value | Significant? |
|---|---|---|---|---|---|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
| [Guardrail] | ... | ... | ... | ... | ... |

**Recommendation**: [Ship / Extend / Stop / Investigate]
**Reasoning**: [Why]
**Next steps**: [What to do]

Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.

Installation

Copy a source-pinned command for your client. You run it yourself.

Destination: .claude/skills/ab-test-analysis · pinned to the source commit

# Run from your project root
git clone https://github.com/phuryn/pm-skills.git .skillboard-tmp
git -C .skillboard-tmp checkout 18468a95b427e70e258b51389796367c6f684e7d
mkdir -p ".claude/skills"
cp -r ".skillboard-tmp/pm-data-analytics/skills/ab-test-analysis" ".claude/skills/"
rm -rf .skillboard-tmp

Review the source before running. This copies files into your project; it is not a one-click install and does not verify runtime safety.

New here? Step-by-step setup – from installing the agent to using the skill

Operating systemAgentI already use Claude Code

1
Install Git
Install Git with your package manager (Debian/Ubuntu shown; use dnf/pacman on others). Official download.
```
sudo apt update && sudo apt install -y git
```
2
Install Claude Code
Requires Node.js 18+. Install the CLI globally: Official docs. (command-line tool)
```
npm install -g @anthropic-ai/claude-code
```

Get the skill, pinned to its exact version

Open a terminal and run these steps (they clone the source, check out the exact commit, and copy the skill into place):

# Run from your project root
git clone https://github.com/phuryn/pm-skills.git .skillboard-tmp
git -C .skillboard-tmp checkout 18468a95b427e70e258b51389796367c6f684e7d
mkdir -p ".claude/skills"
cp -r ".skillboard-tmp/pm-data-analytics/skills/ab-test-analysis" ".claude/skills/"
rm -rf .skillboard-tmp

Destination: .claude/skills/ab-test-analysis

4
Use it in Claude Code
Open Claude Code in your project. It reads skills from the folder above, so ab-test-analysis is now available. Always review the source before using it – static checks cannot prove runtime safety.

Security & Audits

Scanner static-checks@0.1.0 · commit 18468a95b427. Static checks cannot prove runtime safety – review the source and the exact diff before installing. How checks work.

No static rules matched. This is not a safety guarantee.

ab-test-analysis

What this skill does

A/B Test Analysis

Context

Instructions

Further Reading

Installation

Install Git

Install Claude Code

Get the skill, pinned to its exact version

Use it in Claude Code

Security & Audits

ab-test-analysis

What this skill does

A/B Test Analysis

Context

Instructions

Further Reading

Installation

Install Git

Install Claude Code

Get the skill, pinned to its exact version

Use it in Claude Code

Security & Audits