Support vitest 5#86
Conversation
GuillaumeLagrange
changed the title
| } | ||
| } | ||
|
|
||
| function patchRunBenchmarks(): void { |
There was a problem hiding this comment.
By the way, you can do that just in a setup file without overriding the runner. This should also make it support the browser(?)
If there is any feedback, I would love to address them on vitest side; we would also like to use codspeed after Vitest 5 is out in our own repo
There was a problem hiding this comment.
@sheremet-va thanks for the heads up, currently trying out this approach.
The main pain point I am running into is that we are struggling to highjack the benchmark runner when we are in simulation/analysis mode.
When in this mode, what we used to do is highjack the whole run function, to have a very simple sequence of
- warmup
- manually call optimizer
- start valgrind instrumentation
- execute code once
- stop
But I'm struggling a bit to find a good way to replicate the behavior without overriding the whole runner.
I still do not have a proper api request, nor am I sure it's the way to go for the analysis mode, becuase maybe running browser mode under valgrind is not a good idea in the first place, and the main motivation for not highjacking the whole runner is actually to be as compatible as possible with the browser mode.
There was a problem hiding this comment.
Could it be possible to have a way for the plugin to hook/customize the createTinybench function? Maybe by defining some sort of abstract "benchmark backend" interface that we could fullfill, that would default to a tinybench implementation. It would also make it easier to switch away from tinybench on your end at some point.
The way tinybench is instanciated/ran does not allow for much flexibility on our end, and I cannot find a good implementation without overriding the whole runner as we did before for our analysis mode.
I'd love to exchange over this over discord/slack/a call if you want, or we can keep things here on github.
There was a problem hiding this comment.
Feel free to join the discord - https://chat.vitest.dev/
I will give you the ecosystem role, we can discuss it there. I just need your nickname
There was a problem hiding this comment.
I've joined and pinged on via MP, my nickname on the server is the same is github
GuillaumeLagrange
force-pushed
the
cod-2931-prepare-compatibility-with-vitest-5
branch
5 times, most recently
from
25ef927 to
e04f0da
Compare
Merging this PR will degrade performance by 59.17%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Memory | test_recursive_fibo_20 |
19 B | 65,624 B | -99.97% |
| ❌ | Memory | long body |
960 B | 26,344 B | -96.36% |
| ❌ | Simulation | wait 500ms |
10.2 ms | 13.3 ms | -23.09% |
| ❌ | Simulation | wait 1sec |
20.8 ms | 24.5 ms | -14.92% |
| ❌ | WallTime | switch 1 |
144 ns | 168 ns | -14.29% |
| ❌ | WallTime | switch 2 |
168 ns | 192 ns | -12.5% |
| ❌ | WallTime | test_recursive_cached_fibo_10 |
1 µs | 1.2 µs | -10.31% |
| ⚡ | Simulation | switch 1 |
313.8 µs | 105.3 µs | ×3 |
| ⚡ | Memory | wait 500ms |
64.1 KB | 23 KB | ×2.8 |
| ⚡ | WallTime | test sync baz 10 |
228 ns | 192 ns | +18.75% |
| ⚡ | WallTime | test_iterative_fibo_10 |
276 ns | 240 ns | +15% |
| 🆕 | WallTime | iterative fibo 15 |
N/A | 840 ns | N/A |
| 🆕 | WallTime | iterative fibo 20 |
N/A | 816 ns | N/A |
| 🆕 | WallTime | recursive fibo 15 |
N/A | 22.3 µs | N/A |
| 🆕 | WallTime | recursive fibo 20 |
N/A | 239.8 µs | N/A |
| 🆕 | Memory | iterative fibo 15 |
N/A | 0 B | N/A |
| 🆕 | Memory | iterative fibo 20 |
N/A | 0 B | N/A |
| 🆕 | Memory | recursive fibo 15 |
N/A | 19 B | N/A |
| 🆕 | Memory | recursive fibo 20 |
N/A | 19 B | N/A |
| 🆕 | Simulation | iterative fibo 15 |
N/A | 17 µs | N/A |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing cod-2931-prepare-compatibility-with-vitest-5 (6791750) with main (1999ec9)
Footnotes
-
53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Vitest 5 reworked the benchmark backend: the dedicated `NodeBenchmarkRunner` and the `vitest/runners` / `vitest/suite` entrypoints are gone, benchmarks now run inside `test()` through the unified `TestRunner`, and tinybench moved to v6 (stats moved from `result.benchmark` to `task.result.latency`, `includeSamples` became `retainSamples`). Detect the installed Vitest generation and select the integration seam behind a `VitestBackend` abstraction so the rest of the plugin never inspects the version: - v3/4 keep the custom benchmark runner per instrument mode (analysis/walltime). - v5 installs instrumentation from a setup file that patches the shared `TestRunner.runBenchmarks` static. A setup file (rather than a custom `test.runner`) leaves the runner untouched for non-benchmark tests and also applies to the browser pool.
Now that the plugin's own dev dependency tracks Vitest 5, add a dedicated Vitest 4 example so the legacy (v3/4) benchmark seam keeps explicit coverage alongside the existing with-vitest-v3 example. Mirrors that example, pinning vitest ^4.1.9. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>
GuillaumeLagrange
force-pushed
the
cod-2931-prepare-compatibility-with-vitest-5
branch
from
6791750 to
f511dee
Compare
GuillaumeLagrange
force-pushed
the
cod-2931-prepare-compatibility-with-vitest-5
branch
from
f511dee to
c3a356b
Compare
2 participants
No description provided.