-------------------------------------------------------- Benchmark: benchmark/test_runner/mock-timers.js -------------------------------------------------------- What we are trying to find: The optimal number of iterations (n) that produces consistent benchmark results without wasting time. How it works: 1. Run the benchmark multiple times with a specific n value 2. Group results by configuration 3. If overall CV is above 5% or any configuration has CV above 10%, increase n and try again Configuration: - Starting n: 10 iterations - Runs per n value: 30 - Target CV threshold: 1% (lower CV = more stable results) - Max increases: 4 - Increase factor: 10x Testing with n=10: .............................. Configuration: {"mode":"enable-empty-apis","n":10} CV: 9.06% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setTimeout","n":10} CV: 5.90% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setInterval","n":10} CV: 3.90% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setImmediate","n":10} CV: 4.60% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-Date","n":10} CV: 5.07% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-scheduler.wait","n":10} CV: 4.32% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-AbortSignal.timeout","n":10} CV: 3.60% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-all","n":10} CV: 4.66% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-default","n":10} CV: 4.03% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTimeout","n":10} CV: 4.94% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setInterval","n":10} CV: 6.09% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setImmediate","n":10} CV: 6.99% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"scheduler.wait","n":10} CV: 5.37% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"AbortSignal.timeout","n":10} CV: 4.32% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"Date","n":10} CV: 5.58% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTime","n":10} CV: 5.60% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"runAll","n":10} CV: 4.57% (lower values mean more stable results) Stability: ✗ Unstable Overall average CV: 5.21% ✗ Overall CV (5.21%) is above 5% → New best n: 10 with average CV: 5.21% Testing with n=100: .............................. Configuration: {"mode":"enable-empty-apis","n":100} CV: 3.58% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setTimeout","n":100} CV: 4.56% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setInterval","n":100} CV: 3.76% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setImmediate","n":100} CV: 4.01% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-Date","n":100} CV: 5.29% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-scheduler.wait","n":100} CV: 4.24% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-AbortSignal.timeout","n":100} CV: 5.25% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-all","n":100} CV: 4.47% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-default","n":100} CV: 3.53% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTimeout","n":100} CV: 4.23% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setInterval","n":100} CV: 12.55% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setImmediate","n":100} CV: 4.91% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"scheduler.wait","n":100} CV: 6.46% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"AbortSignal.timeout","n":100} CV: 4.52% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"Date","n":100} CV: 5.34% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTime","n":100} CV: 4.65% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"runAll","n":100} CV: 4.21% (lower values mean more stable results) Stability: ✗ Unstable Overall average CV: 5.03% ✗ Overall CV (5.03%) is above 5% ✗ 1 configuration(s) have CV above 10% → New best n: 100 with average CV: 5.03% Testing with n=1000: .............................. Configuration: {"mode":"enable-empty-apis","n":1000} CV: 3.53% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setTimeout","n":1000} CV: 6.01% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setInterval","n":1000} CV: 4.37% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setImmediate","n":1000} CV: 5.87% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-Date","n":1000} CV: 5.57% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-scheduler.wait","n":1000} CV: 4.87% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-AbortSignal.timeout","n":1000} CV: 6.04% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-all","n":1000} CV: 4.50% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-default","n":1000} CV: 5.49% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTimeout","n":1000} CV: 4.65% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setInterval","n":1000} CV: 4.76% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setImmediate","n":1000} CV: 2.66% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"scheduler.wait","n":1000} CV: 3.59% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"AbortSignal.timeout","n":1000} CV: 7.78% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"Date","n":1000} CV: 4.29% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTime","n":1000} CV: 5.97% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"runAll","n":1000} CV: 4.60% (lower values mean more stable results) Stability: ✗ Unstable Overall average CV: 4.97% ✗ Overall CV (4.97%) is above 5% → New best n: 1000 with average CV: 4.97% Testing with n=10000: .............................. Configuration: {"mode":"enable-empty-apis","n":10000} CV: 8.15% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setTimeout","n":10000} CV: 5.97% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setInterval","n":10000} CV: 5.19% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-setImmediate","n":10000} CV: 4.42% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-Date","n":10000} CV: 6.34% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-scheduler.wait","n":10000} CV: 5.38% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-AbortSignal.timeout","n":10000} CV: 8.10% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-all","n":10000} CV: 4.23% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"enable-default","n":10000} CV: 2.95% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTimeout","n":10000} CV: 5.40% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setInterval","n":10000} CV: 4.32% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setImmediate","n":10000} CV: 3.56% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"scheduler.wait","n":10000} CV: 4.19% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"AbortSignal.timeout","n":10000} CV: 7.05% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"Date","n":10000} CV: 14.86% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"setTime","n":10000} CV: 3.88% (lower values mean more stable results) Stability: ✗ Unstable Configuration: {"mode":"runAll","n":10000} CV: 8.84% (lower values mean more stable results) Stability: ✗ Unstable Overall average CV: 6.05% ✗ Overall CV (6.05%) is above 5% ✗ 1 configuration(s) have CV above 10% → Current best n remains: 1000 with average CV: 4.97% Maximum number of increases (4) reached without achieving target stability Best n found: 1000 with average CV=4.97% CV by configuration at best n: {"mode":"enable-empty-apis","n":1000}: 3.53% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-setTimeout","n":1000}: 6.01% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-setInterval","n":1000}: 4.37% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-setImmediate","n":1000}: 5.87% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-Date","n":1000}: 5.57% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-scheduler.wait","n":1000}: 4.87% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-AbortSignal.timeout","n":1000}: 6.04% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-all","n":1000}: 4.50% ⚠️ This configuration is above the target threshold of 1% {"mode":"enable-default","n":1000}: 5.49% ⚠️ This configuration is above the target threshold of 1% {"mode":"setTimeout","n":1000}: 4.65% ⚠️ This configuration is above the target threshold of 1% {"mode":"setInterval","n":1000}: 4.76% ⚠️ This configuration is above the target threshold of 1% {"mode":"setImmediate","n":1000}: 2.66% ⚠️ This configuration is above the target threshold of 1% {"mode":"scheduler.wait","n":1000}: 3.59% ⚠️ This configuration is above the target threshold of 1% {"mode":"AbortSignal.timeout","n":1000}: 7.78% ⚠️ This configuration is above the target threshold of 1% {"mode":"Date","n":1000}: 4.29% ⚠️ This configuration is above the target threshold of 1% {"mode":"setTime","n":1000}: 5.97% ⚠️ This configuration is above the target threshold of 1% {"mode":"runAll","n":1000}: 4.60% ⚠️ This configuration is above the target threshold of 1% Recommendation: You might want to try increasing --max-increases to continue testing with larger n values, or adjust --cv-threshold to accept the current best result, or investigate if specific configurations are contributing to instability.