Repository navigation
Conversation
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
Integrations can report p1 to p99 of the time per iteration in `stats.percentiles_ns`. The field is optional: results files without it keep parsing, and it is left out of the JSON when absent. Refs COD-4241
322d73b to
76030c0
Compare
[Medium impact] The PR appears safe to merge. SummaryAdds optional percentile values to wall-time benchmark results while preserving existing result files and execution-harness output.
No actionable issues found. Reviews (1) · Last reviewed commit: "feat(runner-shared): accept percentiles ..." · Reviewed by Greptile |
| pub q3_ns: f64, | ||
| /// p1..p99 of the time per iteration: `[24]`, `[49]` and `[74]` equal `q1_ns`, `median_ns` and `q3_ns` | ||
| #[serde(default, skip_serializing_if = "Option::is_none")] | ||
| pub percentiles_ns: Option<Vec<f64>>, |
There was a problem hiding this comment.
Open question: Would it make sense to have a struct here for those values to prevent issues where they are inserted in the wrong order? Or if one value too much/little is pushed? Or just for easier understanding?
Or maybe use a [f64; 100]? Wdyt?
There was a problem hiding this comment.
I guess [f64; 100] is a good call. For the struct, I'm not sure, since we'd have to sort anyway when building. If the max is too big for f64, then we'll definitely not be able to store it in the DB, and also it's quite unlikely, since it would mean an iteration of over 100 days 🤣
Add an optional
percentiles_nsfield to the walltimeBenchmarkStats: p1 to p99 of the time per iteration, in nanoseconds like the other_nsstats.Integrations already hold every round in memory but only report min, max, mean, stdev and quartiles. Reporting the full percentile set lets the backend keep the shape of each benchmark's distribution.
The field is optional and skipped when absent, so results files from current integrations keep parsing, and exec-harness keeps writing the same JSON.
Refs COD-4241