Skip to content

feat(runner-shared): accept percentiles in walltime benchmark stats - #630

Open
fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles
Open

fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles

Conversation

@fargito

@fargito fargito commented Nov 17, 2026

Copy link
Copy Markdown
Member

Add an optional percentiles_ns field to the walltime BenchmarkStats: p1 to p99 of the time per iteration, in nanoseconds like the other _ns stats.

Integrations already hold every round in memory but only report min, max, mean, stdev and quartiles. Reporting the full percentile set lets the backend keep the shape of each benchmark's distribution.

The field is optional and skipped when absent, so results files from current integrations keep parsing, and exec-harness keeps writing the same JSON.

Refs COD-4241

@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 53 untouched benchmarks
⏩ 6 skipped benchmarks1


Comparing feat/walltime-percentiles (76030c0) with main (ffdafa6)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Integrations can report p1 to p99 of the time per iteration in
`stats.percentiles_ns`. The field is optional: results files without it
keep parsing, and it is left out of the JSON when absent.

Refs COD-4241
@fargito
fargito force-pushed the feat/walltime-percentiles branch from 322d73b to 76030c0 Compare November 18, 2026 14:39
@fargito
fargito marked this pull request as ready for review November 19, 2026 09:19
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

Adds optional percentile values to wall-time benchmark results while preserving existing result files and execution-harness output.

  • Walltime benchmark results can carry optional p1–p99 values.

No actionable issues found.

Reviews (1) · Last reviewed commit: "feat(runner-shared): accept percentiles ..." · Reviewed by Greptile

pub q3_ns: f64,
/// p1..p99 of the time per iteration: `[24]`, `[49]` and `[74]` equal `q1_ns`, `median_ns` and `q3_ns`
#[serde(default, skip_serializing_if = "Option::is_none")]
pub percentiles_ns: Option<Vec<f64>>,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question: Would it make sense to have a struct here for those values to prevent issues where they are inserted in the wrong order? Or if one value too much/little is pushed? Or just for easier understanding?

Or maybe use a [f64; 100]? Wdyt?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess [f64; 100] is a good call. For the struct, I'm not sure, since we'd have to sort anyway when building. If the max is too big for f64, then we'll definitely not be able to store it in the DB, and also it's quite unlikely, since it would mean an iteration of over 100 days 🤣

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants