A club will not trust a new scoring engine on my word.So I proved it against the one they already run.
144,539 comparisons against Sailwave, the software most clubs already use, drawn from 1,345 results files those clubs published themselves across ten countries. The interesting part is not the match rate. It is that I did not choose the test data, and that you can run the whole thing yourself.
Complete, one-off · July 2026 · code, data and report public on GitHub
No club adopts a scoring engine on a promise.
A sailing club's results are the whole point of the season. Get the scoring wrong and you have handed the trophy to the wrong boat in public. Flying Start needed its own scoring engine, and no club is going to move off the software it has run for years because the person who wrote the replacement says it is fine.
"We tested it" is not an answer. Neither is a unit test suite, because a unit test only ever asserts what its author already thought of. The rules of sailing are full of cases nobody thinks of: a boat that came to the starting area but did not start, a discard that cannot be discarded, six different handicap systems, fifteen penalty codes, and a tie-break procedure that runs to its own appendix.
The only convincing evidence is a corpus you did not choose.
Twenty years of results, scraped off the public web.
Sailwave is the software most clubs have used for two decades, and thousands of clubs publish their finished results with it. Those results are the output of a mature engine, checked by the people whose season depended on them, and nobody involved has any interest in flattering Flying Start.
A scraper walked the Sailwave public results directory and pulled down every results file it could parse. That corpus, not a hand-written fixture set, became the test. Every race in it was rescored by the new engine and compared, boat by boat, against the score the club actually published.
Figures from the report in the repository, v2, July 2026. The corpus nearly doubled between v1 and v2, from 75,693 comparisons to 144,539, when five further penalty codes were added to the engine.
Every mismatch explained, not discarded.
The rule that makes this worth anything: a disagreement is never dropped because it looks small. It is classified, and the classification is written down and published. A 1% mismatch rate with 1% unexplained is not evidence of anything.
Major mismatches, all traced to the test data rather than the engine: guest boats inflating a fleet size in the scraped XML, and a boat whose rating changed mid-series so the two runs were scoring different fleets.
Minor mismatches, all rounding: one to three seconds of corrected time from intermediate rounding, and tenths of a point where the bonus point table is held at slightly different precision. None of them changes a position.
Files below 90%, each read individually. Mostly boats with an IRC rating that the scraper labelled as PY, which is a limitation of the scraper and is recorded as one.
Genuine ambiguity found in the rules themselves. Sailwave uses a code, TLE, that the Racing Rules of Sailing do not define, and different clubs score it differently. There is no correct answer to match against, so the choice is documented instead.
What that leaves is a claim with a shape: no mismatch in 144,539 comparisons was caused by the scoring engine's logic. Not "it works". Not "we are confident". A number, a method that produced it, and a list of everything that did not fit.
The run also found nothing wrong with Sailwave, which is worth saying plainly. Twenty years of use is its own kind of verification, and the new engine had to earn its way to agreement with it rather than the other way round.
You do not have to believe any of this.
The engine, the scraper, the comparison code, the source log and the full results are in a public repository. So is the report, with its mismatch tables intact. If the number is wrong, the way to find out is to run it again.
An unreproducible verification is a marketing claim with a decimal point in it. Publishing the method is the part that makes it evidence, and it is also the part that means I have to be right.
Two caveats, because they would surface in the first ten minutes anyway. The soak test script imports from paths inside the Flying Start monorepo, so running it standalone needs those imports adjusted. And the scraper reads live pages, so a fresh scrape will not return byte-identical data to the one in the repository.
Sailwave is a trademark of Jon Mayall. Flying Start is not affiliated with Sailwave, and all the results data used here was already published publicly by the clubs that produced it.
This is not really about sailing.
Any replacement of a calculation engine has this shape: a legacy system nobody wants to touch, an organisation that cannot sign off the new one, and a body of historical output sitting right there. Billing runs, payroll, pricing, tax, commissions, actuarial models, grading. The output is the fixture set, and usually nobody has thought to use it.
The work is the harness and the honesty about the mismatches, not the cleverness. It took days rather than months, and it turned an argument about trust into an argument about a number, which is a much easier argument to finish.