A pilot study research paper section adds disproportionate credibility for very little effort — the methodological equivalent of a pre-flight check.
Pilot studies are the cheapest insurance you can buy against unworkable measures, badly worded items, and unrealistic data-collection timelines. Yet most papers either skip the pilot entirely or report it as a single defensive line. Reviewers read both signals the same way: as a missed opportunity to demonstrate methodological seriousness.
This guide covers when a pilot is necessary, what it should actually test, how to scope it without over-engineering, and how to report it so it strengthens — rather than burdens — your methodology. For broader context, see our pillar on how to write a methodology section that reviewers respect.
You're using new or adapted measures, an under-tested population, an online or remote protocol, or a complex experimental procedure.
You're using validated, off-the-shelf instruments in their original language, with a well-understood population, and a familiar procedure.
The default for empirical work outside the second category is to run something. Even a 10–20 participant pilot adds a credible sentence to your methodology, and almost always surfaces a fixable problem before it becomes an unfixable one in the main study.
What a pilot study actually tests
A pilot is not a mini version of your main study. It's a calibration exercise — focused on the things that can break and the things you can still change. Four targets matter most.
Instrument clarity
Do participants understand the items as you intended? Reverse-coded items, ambiguous wording, and technical jargon often surface here.
Procedure feasibility
Does the data-collection flow work end-to-end? Timing, drop-off points, technical glitches, and instruction clarity all surface in a pilot.
Preliminary reliability
Do scales hold together as expected? Cronbach's α and inter-item correlations on the pilot can flag problem items early.
Effect-size estimation
For inferential work, the pilot can yield a rough effect size to inform the main study's power analysis — though never as the sole input.
How to scope a pilot — the 10% rule
A useful rule of thumb is 10% of your projected main sample, with a floor of 10 and a ceiling of 50. Below 10, you can't meaningfully test reliability; above 50, you're spending budget that the main study needs. For interview-based work, three to five pilot interviews are usually sufficient to test protocol flow without saturating the main pool.
The pilot sample should resemble the target population but should not be drawn from it. If you pilot with five participants from your eventual sample of 400, those five must be excluded from the main analysis — and that's a small loss compared with the contamination risk of including them. Where possible, recruit pilot participants from a parallel, comparable population.
Pilot effect sizes are notoriously unstable — a 15-person pilot can produce effect estimates that swing wildly with one or two participants. Use pilot data alongside published effect sizes from comparable studies, or treat the pilot estimate as an upper bound to be checked against the literature. See our supporting guide on how to justify sample size in research for the full justification framework.
How to report a pilot study — the seven elements
A pilot report sits inside the procedure or design subsection of your methodology, typically 80–150 words. Seven elements cover the reviewer-expected content.
- When the pilot was run — month and year, relative to main data collection.
- Pilot sample size and characteristics — comparable to but separate from the main sample.
- Recruitment method — and confirmation that pilot participants were excluded from the main study.
- What was tested — instruments, procedure, timing, technology, or all of these.
- What was learned — specific issues identified, not generalities.
- What was changed — every modification made to the main protocol because of the pilot.
- What was retained — confirmation that no further changes were needed for elements that worked.
"The pilot is the only place in your methodology where finding a problem is good news. Surface it now, fix it now, report it now."
The three pilot-reporting mistakes reviewers catch
1. The decorative pilot. "A pilot study was conducted to ensure validity" tells reviewers nothing. State what was tested, what surfaced, and what changed. If nothing changed, say that too — and explain how you reached that judgement.
2. Pilot data folded into main analysis. Including pilot participants in the main dataset, especially after protocol changes, biases your analysis and invites methodological challenge. Pilot data should stay separate unless explicitly justified as part of an internal pilot design (which then requires a different framing entirely).
3. The "we changed everything" pilot. If the pilot prompted wholesale changes to instruments, procedure, and sample frame, the reported version no longer reflects the original design. State this explicitly. Reviewers respect transparent revision histories far more than glossed-over ones.
Internal versus external pilots — know the difference
An external pilot is a standalone calibration run, separate from the main study, with participants excluded from main analysis. This is the default and the safest design for most methodological purposes.
An internal pilot is integrated into the main study — the first wave of participants serves as both pilot and main study, with their data included if no protocol changes are made. Internal pilots can save resources but require strict pre-registration of the criteria under which the protocol would be modified. Without pre-registration, internal pilots blur into p-hacking territory and should be avoided.
For most early-career researchers, the safest default is an external pilot with explicit exclusion from the main sample. The slight efficiency loss is worth the methodological clarity it produces in peer review, especially for the first few papers where reviewer scrutiny is highest and any ambiguity about data provenance can derail the submission.
Closing — small effort, large credibility return
The strongest pilot studies aren't elaborate; they're targeted. A focused 15-person pilot that tested three specific elements, made two specific changes, and is reported in five clean sentences will outperform a sprawling 60-person mini-study reported in a single defensive paragraph.
Run the pilot. Document what you found. Write the paragraph that turns calibration into credibility. It's one of the highest-leverage moves available in a methodology section.
Need a second pair of eyes on your methodology?
Our PhD editors will review your pilot reporting, sample-size justification, and overall methodology for reviewer-readiness — before your paper reaches an editor's desk. First consultation is free.
Book Free Consultation → Or read: How to Justify Sample Size in a Research Paper