01
How many clips does a conclusion need?
More than most people use. Because a single clip can reach a hundred times another for reasons unrelated to quality, a handful of posts is dominated by whichever one got lucky. Judging a change needs enough posts that no individual result moves the median much, which in practice is dozens rather than several.
02
Sample size against what it can show
The spread between your best and worst clip is usually larger than the effect of anything you deliberately changed, which is what makes small samples actively misleading.
| Clips posted | What you can conclude | What you cannot |
|---|---|---|
| One | Something from its retention graph | Anything from its reach |
| Three | Very little | Whether a change worked |
| Ten | A rough sense of your range | Small differences |
| Thirty | Whether a change moved the median | Fine distinctions |
| A hundred | Reliable patterns | Still not why a single clip flew |
03
Using the median instead of the best
When comparing two periods, look at the median clip rather than the best one. The median is the performance of your typical clip, and it barely moves when one post gets lucky. If the median rises after a change, the change is probably helping.
Averages can mislead because a single viral clip pulls them far upward. The median ignores outliers and shows what usually happens.
Plot the median week by week on a simple chart. A steady upward slope over two months is far more convincing than any single good week, and it is much harder to fool yourself with.
04
Changing one thing at a time
If you change the style, the posting time and the clip length all at once, you cannot tell which change mattered. Pick one variable, keep the rest steady for a few weeks, and then compare. It is slower but far more informative.
Write down the start date of each change, so you can match it with your analytics later without guessing.
05
Metrics that stabilise faster
Views swing wildly between clips, but some measures settle down more quickly. The share of viewers who stay past the first few seconds and the share who watch to the end tend to be more consistent, and they respond more directly to what you changed.
Track those alongside views. They often show whether a change helped weeks before total views can.
06
Knowing when to trust a result
A useful rule of thumb is to wait until you have at least a few dozen clips under a new approach before drawing strong conclusions. If the result then points the same way as the early signs, it is likely real.
If results flip back and forth, the change probably has little effect, and your attention is better spent elsewhere.
07
Give it a month and a median
- Use the median, not the average. One outlier destroys an average and barely moves a median.
- Run a change for a month minimum. Shorter windows are noise.
- Write down what you changed and when. Otherwise you cannot tell later what produced what.
Produce enough to have a median. Nothing to pay for the first.
Questions people ask
- Can I learn anything from one clip?
- From the retention graph, yes, because that is about the clip itself. From its reach, very little.
- Why is the variance so large?
- Because distribution compounds. Small differences in early response produce enormous differences in eventual reach.
- What if a change obviously works?
- Obvious effects show up quickly and are rare. Most real improvements are small and need patience to see.
- Should I stop testing then?
- No, change one thing and give it long enough. The mistake is concluding after three posts, not testing at all.
- What is worth tracking?
- Median reach and median watch time per month, against what you were doing that month.