When a farmer or a program team finishes an on-farm trial, there's usually a moment of anticipation. The data is in. The yield monitor files have been processed. The comparison is right there in the spreadsheet: treated versus untreated, new practice versus standard practice.

And the number looks good.

Here's the question worth asking before anyone acts on it: is this result defensible?

Not "is this result interesting?" Not "does this confirm what we thought?" But: if someone who did not run this trial looked at the data and the methodology, would they trust the conclusion?

That question matters a lot. Because on-farm trial results don't just live in spreadsheets. They get presented to partners. They influence purchasing decisions and program design. They go into reports that funders use to justify continued investment. And increasingly, they are the evidence base for claims that have legal and commercial consequences.

A result that can't withstand scrutiny isn't just a wasted season. It's a liability.

What "Defensible" Actually Means

Defensibility is not about perfection. On-farm trials happen in real fields under real conditions, and anyone who has reviewed trial data seriously knows that the conditions are never perfect.

Defensibility means that the result can be explained and examined by someone outside the trial team, and that explanation holds up. It means that when someone asks "how do you know this wasn't just field variability?" you have an answer. When someone asks "how many times did you replicate this?" you have a number. When someone asks "what statistical test did you use, and why?" you can answer that too.

It also means that the result is accompanied by enough context to interpret it correctly. A yield bump of four bushels per acre means something very different on a field with high spatial variability than on a field where conditions were uniform. A result that shows no statistical significance is still valuable information, but only if the person reading it understands that the trial had enough replication to have the power to detect the effect size that was being looked for.

Defensibility is not about convincing skeptics. It's about communicating evidence accurately, so that people making management decisions based on that evidence are making them with the right information.

The Replication Question

The single most common defensibility gap in on-farm trials is insufficient replication.

When a result comes from a single comparison, one strip of treatment, one strip of control, there's no way to separate the treatment effect from field variability. The result is an observation, not a finding. It's interesting. It might point in a useful direction. But it cannot support a management decision with any confidence, because you don't know if the difference you're seeing is the treatment or the particular patch of ground where the treated strip happened to land.

Replication is what allows you to average across locations, and averaging across locations is what allows you to say with confidence that you are seeing a treatment effect rather than a field effect.

Four replications per treatment is a reasonable minimum for most on-farm trials, though the right number depends on the variability of the field and the size of the effect you're trying to detect. A power analysis before planting tells you exactly how many replications you need to have a reasonable chance of finding a real effect, at your chosen level of statistical significance.

When a trial result is challenged, replication is usually the first thing a reviewer looks at. If the answer is "we had two strips," the conversation tends to end there.

Statistical Significance and What It Doesn't Tell You

Statistical significance gets misunderstood more often than almost any other concept in research trials.

A result that is statistically significant means that the observed difference between treatments is unlikely to have occurred by chance, given the variability in the data. It does not mean the result will replicate in every field. It does not mean the treatment is a good idea. It does not mean the difference is large enough to matter economically.

A result that is not statistically significant means the trial could not detect a reliable difference between treatments. This can happen because there is no difference. It can also happen because the trial lacked the statistical power to detect the difference.

Both outcomes are informative. Neither is informative without context.

When you report a trial result, report the p-value and the confidence interval. Report the effect size in agronomic terms: not just "statistically significant" but "a yield difference of X bushels per acre, which at current commodity prices represents a return of Y dollars per acre against a treatment cost of Z." That translation from statistical language to economic language is what makes a research trial result into a management decision tool.

The Documentation Standard

A defensible result is a documented result.

Documentation means the trial protocol exists in writing, and it was written before the trial was planted, not reconstructed afterward. It means the field map shows treatment assignment and replication layout. It means the data collection methods are specified: what instrument was used, what quality checks were applied to the yield monitor data, what covariates were recorded and why.

It also means that deviations from the protocol are noted. If a replicate had to be moved because of a drainage problem. If the application rate on one strip was different than intended. These notes don't invalidate a result, but they're part of the honest interpretation of it.

The question to ask at the beginning of every trial, before the seed goes in the ground: if someone outside this team had to reconstruct what we did from our records alone, could they?

If the answer is no, the documentation needs more work.

Who Is Asking for Defensible Results

The audience for defensible trial results has expanded significantly in the last decade.

It used to be primarily university researchers and extension professionals who cared about rigor. Now that group includes commercial partners who want evidence to support product claims, conservation program funders who need to demonstrate practice efficacy to their own funders, technology companies using trial data to train models and validate recommendations, and regulatory bodies that are increasingly scrutinizing the evidentiary basis for agricultural claims.

Each of these audiences is asking a slightly different version of the same question: why should I trust this result?

The answer is not "because we ran a trial." It's because the trial was designed to answer a specific question, with enough replication to detect the effect size that matters, with randomized treatment assignment, with documented methods, and with analysis that is appropriate for the data structure.

Programs that are generating this kind of evidence are building a genuinely valuable asset. Results that meet this standard can be aggregated across locations and seasons, used to support product registrations or program claims, published in practitioner journals, and shared with partners who bring them additional credibility.

Results that don't meet this standard are interesting, but they stay in the spreadsheet.

Download The On-Farm Trial Operations Playbook to make sure you are audit ready from the first grower enrollment until you hand over the commercial-ready report.