Benchmark

[mre]

We should test hyperjson with the benchmark data provided in gojay.
Thanks to @arnecls for the link.

[mre]

More benchmark data:

• Benchmark data
• Comments on benchmark data
• Another benchmark
• Some ultrajson benchmarks

[emmatyping]

There are also some good benchmarks in the python performance suite. https://github.com/python/performance/tree/master/performance/benchmarks (see bm_json_loads.py and bm_json_dumps.py).

[mre]

Nice find @ethanhs. One would have to check, which cases we already cover and add the remaining ones to our benchmark. If there's anything in particular we aren't testing yet, we would be happy to merge PRs which add that.

[emmatyping]

I'll take a look at that some time this weekend. FWIW, it seems that there is a huge performance hit on Windows. I ran pip install . and ran the benchmark tests. It would seem that hyperjson is an order of magnitude slower than even the json module (!), though perhaps I am misreading the output?

---------------------------------------------------------------------------------- benchmark '100 dicts array: deserialize': 5 tests -----------------------------------------------------------------------------------
Name (time in us)                               Min                   Max                Mean              StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-100 dicts array]           26.2000 (1.0)        209.1000 (1.0)       35.2515 (1.08)      13.9826 (1.12)      37.5000 (1.29)     11.6000 (58.00)     617;555       28.3676 (0.92)       8439           1
test_loads[rapidjson-100 dicts array]       28.6000 (1.09)     3,164.2000 (15.13)     32.5589 (1.0)       26.8670 (2.15)      29.0000 (1.0)       0.2000 (1.0)      493;3503       30.7136 (1.0)       20367           1
test_loads[json-100 dicts array]            33.3000 (1.27)       419.3000 (2.01)      38.9969 (1.20)      15.9375 (1.27)      33.8000 (1.17)      3.0000 (15.00)    871;2135       25.6431 (0.83)      13870           1
test_loads[simplejson-100 dicts array]      54.5000 (2.08)       289.7000 (1.39)      58.6965 (1.80)      12.5252 (1.0)       55.5000 (1.91)      0.8000 (4.00)     777;2589       17.0368 (0.55)      11779           1
test_loads[hyperjson-100 dicts array]      401.9000 (15.34)    1,674.6000 (8.01)     501.3356 (15.40)    213.6069 (17.05)    415.4000 (14.32)    72.8000 (364.00)      65;77        1.9947 (0.06)        679           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark '100 dicts array: serialize': 5 tests ------------------------------------------------------------------------------------
Name (time in us)                               Min                   Max                Mean              StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[rapidjson-100 dicts array]       19.2000 (1.0)        134.4000 (1.0)       22.7310 (1.0)        6.3979 (1.0)       19.8000 (1.0)       6.6000 (6.00)      555;312       43.9929 (1.0)       20921           1
test_dumps[ujson-100 dicts array]           21.4000 (1.11)       231.1000 (1.72)      27.5418 (1.21)      12.5107 (1.96)      21.7000 (1.10)      4.8000 (4.36)    2079;2374       36.3084 (0.83)      12331           1
test_dumps[json-100 dicts array]            43.9000 (2.29)       314.7000 (2.34)      62.6061 (2.75)      16.9113 (2.64)      58.9000 (2.97)      1.1000 (1.0)     1196;2359       15.9729 (0.36)       9479           1
test_dumps[simplejson-100 dicts array]     103.4000 (5.39)     1,953.8000 (14.54)    138.1775 (6.08)      56.1377 (8.77)     114.8000 (5.80)     46.2000 (42.00)     430;406        7.2371 (0.16)       6415           1
test_dumps[hyperjson-100 dicts array]      307.0000 (15.99)    1,274.9000 (9.49)     423.3172 (18.62)    104.2502 (16.29)    412.2000 (20.82)     7.1000 (6.45)      214;469        2.3623 (0.05)       2141           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark '256 Trues array: deserialize': 5 tests ---------------------------------------------------------------------------------
Name (time in us)                               Min                 Max                Mean             StdDev              Median               IQR             Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-256 Trues array]            6.0000 (1.0)      174.7000 (1.60)       7.0628 (1.0)       3.3077 (1.0)        6.3000 (1.0)      0.2000 (1.0)      2680;8138      141.5870 (1.0)       49505           1
test_loads[simplejson-256 Trues array]       7.9000 (1.32)     109.3000 (1.0)       10.4251 (1.48)      4.8933 (1.48)       8.2000 (1.30)     2.6000 (13.00)    2564;2614       95.9221 (0.68)      29240           1
test_loads[rapidjson-256 Trues array]        9.5000 (1.58)     348.1000 (3.18)      12.1959 (1.73)      4.9872 (1.51)      10.2000 (1.62)     3.0000 (15.00)    4254;4213       81.9946 (0.58)      52357           1
test_loads[json-256 Trues array]             9.8000 (1.63)     260.5000 (2.38)      12.0491 (1.71)      5.2017 (1.57)      10.7000 (1.70)     0.2000 (1.00)     3076;7648       82.9935 (0.59)      39683           1
test_loads[hyperjson-256 Trues array]      172.9000 (28.82)    612.9000 (5.61)     188.5422 (26.70)    50.3771 (15.23)    175.0000 (27.78)    2.9000 (14.50)      109;240        5.3039 (0.04)       1883           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark '256 Trues array: serialize': 5 tests -----------------------------------------------------------------------------------
Name (time in us)                               Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[rapidjson-256 Trues array]        5.0000 (1.0)         97.1000 (1.38)       8.6817 (1.0)       5.4984 (1.48)       5.2000 (1.0)       9.1000 (22.75)    8545;304      115.1842 (1.0)       33223           1
test_dumps[ujson-256 Trues array]            7.7000 (1.54)        86.9000 (1.23)       9.4707 (1.09)      3.7154 (1.0)        7.8000 (1.50)      1.8000 (4.50)    2026;2528      105.5884 (0.92)      28819           1
test_dumps[simplejson-256 Trues array]      11.8000 (2.36)       276.9000 (3.93)      15.5102 (1.79)      9.6132 (2.59)      12.4000 (2.38)      3.8000 (9.50)    1222;1292       64.4736 (0.56)      21979           1
test_dumps[json-256 Trues array]            13.6000 (2.72)        70.5000 (1.0)       15.5118 (1.79)      4.3276 (1.16)      14.2000 (2.73)      0.4000 (1.0)      579;2210       64.4669 (0.56)      12854           1
test_dumps[hyperjson-256 Trues array]      153.4000 (30.68)    1,130.7000 (16.04)    204.9592 (23.61)    66.3087 (17.85)    207.2000 (39.85)    50.1000 (125.25)    111;109        4.8790 (0.04)       3938           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------- benchmark '256 ascii array: deserialize': 5 tests ----------------------------------------------------------------------------------
Name (time in us)                               Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[simplejson-256 ascii array]      24.6000 (1.0)        250.5000 (1.0)       36.7018 (1.06)     13.3701 (1.0)       33.4000 (1.18)      1.5000 (1.0)     2942;9459       27.2466 (0.94)      19961           1
test_loads[rapidjson-256 ascii array]       27.8000 (1.13)     7,115.9000 (28.41)     34.5528 (1.0)      53.6676 (4.01)      28.3000 (1.0)       4.6000 (3.07)     409;2585       28.9412 (1.0)       19048           1
test_loads[json-256 ascii array]            36.3000 (1.48)       406.5000 (1.62)      43.3847 (1.26)     17.5771 (1.31)      37.5000 (1.33)      3.6000 (2.40)    1672;2794       23.0496 (0.80)      18282           1
test_loads[ujson-256 ascii array]           42.3000 (1.72)     4,082.7000 (16.30)     54.7054 (1.58)     41.2656 (3.09)      43.1000 (1.52)     17.7000 (11.80)    719;1009       18.2797 (0.63)      14389           1
test_loads[hyperjson-256 ascii array]      173.6000 (7.06)     1,143.9000 (4.57)     246.3564 (7.13)     93.8418 (7.02)     233.0000 (8.23)     25.4000 (16.93)    133;1112        4.0592 (0.14)       3732           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark '256 ascii array: serialize': 5 tests ------------------------------------------------------------------------------------
Name (time in us)                               Min                   Max                Mean              StdDev              Median                 IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-256 ascii array]           30.6000 (1.0)        363.1000 (1.44)      38.8535 (1.0)       13.0850 (1.0)       31.0000 (1.0)       18.3000 (18.30)    1210;465       25.7377 (1.0)       25840           1
test_dumps[rapidjson-256 ascii array]       34.5000 (1.13)       434.6000 (1.72)      40.3999 (1.04)      18.4996 (1.41)      34.8000 (1.12)       1.2000 (1.20)     967;3817       24.7525 (0.96)      18553           1
test_dumps[json-256 ascii array]            40.3000 (1.32)       673.1000 (2.66)      58.3193 (1.50)      20.9119 (1.60)      54.4000 (1.75)       1.0000 (1.0)     1108;4592       17.1470 (0.67)      11416           1
test_dumps[simplejson-256 ascii array]      44.2000 (1.44)       252.7000 (1.0)       65.5099 (1.69)      30.6722 (2.34)      58.8000 (1.90)      23.6000 (23.60)     544;301       15.2649 (0.59)       3962           1
test_dumps[hyperjson-256 ascii array]      563.1000 (18.40)    2,485.2000 (9.83)     702.8139 (18.09)    197.4451 (15.09)    758.3000 (24.46)    191.0000 (191.00)      31;22        1.4229 (0.06)       1422           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------- benchmark '256 doubles array: deserialize': 5 tests -----------------------------------------------------------------------------------
Name (time in us)                                 Min                   Max                Mean              StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-256 doubles array]           23.1000 (1.0)        205.6000 (1.0)       29.4764 (1.0)       13.6464 (1.0)       24.3000 (1.0)       6.7000 (2.39)    1640;1669       33.9255 (1.0)       25446           1
test_loads[simplejson-256 doubles array]     111.9000 (4.84)       843.0000 (4.10)     149.8376 (5.08)      78.2478 (5.73)     113.2000 (4.66)     35.1000 (12.54)     243;250        6.6739 (0.20)       2090           1
test_loads[json-256 doubles array]           117.5000 (5.09)       658.9000 (3.20)     127.3151 (4.32)      29.5957 (2.17)     118.6000 (4.88)      2.8000 (1.0)      441;1279        7.8545 (0.23)       7413           1
test_loads[rapidjson-256 doubles array]      122.5000 (5.30)       867.9000 (4.22)     147.5637 (5.01)      61.2402 (4.49)     124.1000 (5.11)     11.7000 (4.18)      187;438        6.7767 (0.20)       2216           1
test_loads[hyperjson-256 doubles array]      480.1000 (20.78)    2,319.4000 (11.28)    528.8467 (17.94)    154.7483 (11.34)    485.6000 (19.98)    14.4000 (5.14)      132;290        1.8909 (0.06)       1969           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark '256 doubles array: serialize': 5 tests ------------------------------------------------------------------------------------
Name (time in us)                                 Min                   Max                Mean              StdDev              Median                 IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-256 doubles array]          103.0000 (1.0)      1,370.6000 (1.59)     136.2883 (1.0)       55.6587 (1.0)      137.8000 (1.0)       34.7000 (14.46)     192;192        7.3374 (1.0)        5311           1
test_dumps[json-256 doubles array]           217.3000 (2.11)     1,996.3000 (2.32)     306.5094 (2.25)     121.5316 (2.18)     291.2000 (2.11)       8.5000 (3.54)      97;1104        3.2625 (0.44)       3800           1
test_dumps[simplejson-256 doubles array]     220.9000 (2.14)       859.8000 (1.0)      237.0034 (1.74)      55.6794 (1.00)     221.9000 (1.61)       2.4000 (1.0)        78;132        4.2193 (0.58)       1145           1
test_dumps[rapidjson-256 doubles array]      316.9000 (3.08)     2,395.7000 (2.79)     345.5097 (2.54)     103.2005 (1.85)     319.4000 (2.32)       4.4000 (1.83)       65;483        2.8943 (0.39)       2151           1
test_dumps[hyperjson-256 doubles array]      356.1000 (3.46)     1,409.0000 (1.64)     450.4111 (3.30)     152.6869 (2.74)     476.6500 (3.46)     120.5000 (50.21)       21;21        2.2202 (0.30)        754           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark '256 unicode array: deserialize': 5 tests --------------------------------------------------------------------------------------
Name (time in us)                                   Min                    Max                  Mean              StdDev                Median                 IQR            Outliers         OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[rapidjson-256 unicode array]        435.7000 (1.0)       4,260.2000 (1.01)       615.0744 (1.0)      199.5003 (1.0)        588.8000 (1.0)       17.8000 (1.0)        81;346  1,625.8195 (1.0)        1622           1
test_loads[ujson-256 unicode array]            543.3000 (1.25)      4,200.9000 (1.0)        874.6399 (1.42)     358.4677 (1.80)       728.5000 (1.24)     177.3000 (9.96)        43;43  1,143.3277 (0.70)        456           1
test_loads[hyperjson-256 unicode array]        676.4000 (1.55)      4,682.4000 (1.11)       915.6950 (1.49)     271.0047 (1.36)       914.5000 (1.55)      24.1000 (1.35)       38;322  1,092.0667 (0.67)        918           1
test_loads[simplejson-256 unicode array]     2,233.5000 (5.13)     10,043.4000 (2.39)     3,188.1642 (5.18)     823.4900 (4.13)     2,996.3500 (5.09)     843.9000 (47.41)       50;10    313.6601 (0.19)        330           1
test_loads[json-256 unicode array]           2,932.8000 (6.73)      8,371.8000 (1.99)     3,104.1767 (5.05)     623.3183 (3.12)     2,957.4000 (5.02)      87.5250 (4.92)         8;34    322.1466 (0.20)        253           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------- benchmark '256 unicode array: serialize': 5 tests --------------------------------------------------------------------------------------
Name (time in us)                                   Min                   Max                  Mean              StdDev                Median                 IQR            Outliers         OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-256 unicode array]            246.6000 (1.0)      1,890.5000 (1.42)       371.1694 (1.0)      126.5183 (1.27)       348.6000 (1.03)      35.3500 (5.44)       88;494  2,694.1875 (1.0)        2151           1
test_dumps[simplejson-256 unicode array]       331.8000 (1.35)     1,328.6000 (1.0)        386.0724 (1.04)      99.8662 (1.0)        338.1000 (1.0)      111.4000 (17.14)       87;25  2,590.1879 (0.96)       2226           1
test_dumps[rapidjson-256 unicode array]        444.4000 (1.80)     2,402.8000 (1.81)       562.8477 (1.52)     189.5023 (1.90)       593.9000 (1.76)     148.9750 (22.92)       24;24  1,776.6795 (0.66)       1123           1
test_dumps[json-256 unicode array]             487.2000 (1.98)     1,433.8000 (1.08)       519.2264 (1.40)     127.0358 (1.27)       494.1000 (1.46)       6.5000 (1.0)        17;135  1,925.9420 (0.71)        692           1
test_dumps[hyperjson-256 unicode array]      2,915.5000 (11.82)    5,360.5000 (4.03)     3,790.2426 (10.21)    499.9566 (5.01)     3,918.6000 (11.59)     48.1500 (7.41)        34;37    263.8354 (0.10)        141           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------- benchmark 'complex object: deserialize': 5 tests -------------------------------------------------------------------------------------
Name (time in us)                                Min                   Max                  Mean              StdDev                Median                 IQR            Outliers          OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-complex object]             81.9000 (1.0)        349.8000 (1.0)         84.6603 (1.0)       11.1232 (1.0)         83.0000 (1.0)        0.9000 (1.0)        66;359  11,811.9123 (1.0)        3239           1
test_loads[json-complex object]             102.1000 (1.25)     1,156.0000 (3.30)       146.1027 (1.73)      50.0471 (4.50)       137.4000 (1.66)      10.8000 (12.00)    376;3073   6,844.5022 (0.58)       7850           1
test_loads[rapidjson-complex object]        110.6000 (1.35)     1,280.3000 (3.66)       173.8301 (2.05)      68.2279 (6.13)       147.6000 (1.78)      24.9500 (27.72)     370;551   5,752.7449 (0.49)       5236           1
test_loads[simplejson-complex object]       157.1000 (1.92)       696.7000 (1.99)       171.8350 (2.03)      51.2130 (4.60)       158.8500 (1.91)       1.8000 (2.00)      110;477   5,819.5373 (0.49)       2182           1
test_loads[hyperjson-complex object]      1,541.8000 (18.83)    5,841.3000 (16.70)    1,752.0044 (20.69)    537.2374 (48.30)    1,601.7500 (19.30)    159.0500 (176.72)       8;26     570.7748 (0.05)        228           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------- benchmark 'complex object: serialize': 5 tests --------------------------------------------------------------------------------------
Name (time in us)                                Min                   Max                  Mean              StdDev                Median                IQR            Outliers          OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-complex object]             70.2000 (1.0)        654.3000 (1.86)        80.5986 (1.0)       34.1854 (1.22)        71.1000 (1.0)       2.6000 (1.0)      710;1691  12,407.1649 (1.0)        9931           1
test_dumps[rapidjson-complex object]        101.5000 (1.45)     1,214.7000 (3.46)       151.5755 (1.88)      62.2900 (2.23)       137.2000 (1.93)      2.7000 (1.04)     503;2217   6,597.3742 (0.53)       8197           1
test_dumps[json-complex object]             151.3000 (2.16)       351.0000 (1.0)        227.7167 (2.83)      27.9296 (1.0)        240.3000 (3.38)     51.4000 (19.77)       563;1   4,391.4210 (0.35)       1417           1
test_dumps[simplejson-complex object]       231.7000 (3.30)     2,291.5000 (6.53)       308.8501 (3.83)     145.2978 (5.20)       243.1000 (3.42)     97.8000 (37.62)     285;283   3,237.8172 (0.26)       2919           1
test_dumps[hyperjson-complex object]      1,760.5000 (25.08)    5,066.1000 (14.43)    1,828.2322 (22.68)    268.4500 (9.61)     1,770.2000 (24.90)    26.9000 (10.35)       15;18     546.9765 (0.04)        199           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'composite object: deserialize': 5 tests ------------------------------------------------------------------------------------
Name (time in us)                                Min                   Max                  Mean              StdDev                Median                IQR            Outliers          OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-composite object]           25.0000 (1.0)        155.7000 (1.0)         34.2759 (1.0)       16.9594 (1.0)         27.8000 (1.0)      11.8000 (1.82)      456;444  29,175.0561 (1.0)        7103           1
test_loads[json-composite object]            67.5000 (2.70)       765.4000 (4.92)        80.2378 (2.34)      34.0275 (2.01)        70.3500 (2.53)      6.5000 (1.0)      543;1658  12,462.9488 (0.43)       9690           1
test_loads[rapidjson-composite object]       69.2000 (2.77)     2,169.7000 (13.94)       86.4409 (2.52)      42.3356 (2.50)        73.2000 (2.63)     12.6000 (1.94)     591;1009  11,568.5972 (0.40)       7943           1
test_loads[simplejson-composite object]      88.2000 (3.53)       369.8000 (2.38)        99.0856 (2.89)      17.9334 (1.06)        89.9000 (3.23)     22.1000 (3.40)      693;267  10,092.2851 (0.35)       7771           1
test_loads[hyperjson-composite object]      959.8000 (38.39)    4,844.3000 (31.11)    1,103.1980 (32.19)    379.3468 (22.37)    1,008.8000 (36.29)    73.8000 (11.35)      50;112     906.4556 (0.03)        918           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'composite object: serialize': 5 tests ------------------------------------------------------------------------------------
Name (time in us)                                Min                   Max                Mean              StdDev              Median                 IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-composite object]           25.2000 (1.0)        144.2000 (1.0)       26.7945 (1.0)        4.8106 (1.0)       25.5000 (1.0)        2.5000 (8.33)      118;118       37.3210 (1.0)        4270           1
test_dumps[rapidjson-composite object]       32.5000 (1.29)       183.4000 (1.27)      36.0976 (1.35)      12.7038 (2.64)      33.0000 (1.29)       0.3000 (1.0)       302;555       27.7027 (0.74)       5423           1
test_dumps[json-composite object]            46.6000 (1.85)       489.3000 (3.39)      68.6469 (2.56)      29.1511 (6.06)      62.1000 (2.44)      22.2000 (74.00)     780;377       14.5673 (0.39)       7497           1
test_dumps[simplejson-composite object]      61.4000 (2.44)       398.9000 (2.77)      66.3618 (2.48)      11.9575 (2.49)      63.5000 (2.49)       1.0000 (3.33)     424;1250       15.0689 (0.40)       7906           1
test_dumps[hyperjson-composite object]      602.8000 (23.92)    1,565.8000 (10.86)    683.5941 (25.51)    103.5540 (21.53)    613.3000 (24.05)    201.7000 (672.33)      357;2        1.4629 (0.04)       1026           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

Results (53.92s):
      70 passed

[mre]

@ethanhs thanks for your tests. We made quite a few improvements recently, so it might be worth trying again. You can simply install with pip install hyperjson now.
One thing that came to mind is, that you might be testing with a debug build. This was the default in the Cargo.toml for a while (required for profiling), but was since removed.

[emmatyping]

Sorry for the delay, I ended up having to build from source since I am running Python 3.7, and I had to figure out that yajl doesn't support 3.7 (Pipenv was particularly unhelpful here)

Anyway, here are the results of my latest run (I switched processors from an i7-5820k to a Ryzen Threadripper 1950X):

------------------------------------------------------------------------------ benchmark '100 dicts array: deserialize': 4 tests -------------------------------------------------------------------------------
Name (time in us)                              Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-100 dicts array]          26.5000 (1.0)      115.7000 (1.91)     27.2107 (1.0)      1.5892 (1.23)     26.9000 (1.0)      0.1000 (1.0)     1061;1936       36.7503 (1.0)       22832           1
test_loads[hyperjson-100 dicts array]      48.2000 (1.82)      60.6000 (1.0)      49.2244 (1.81)     1.2905 (1.0)      49.0000 (1.82)     0.2000 (2.00)        17;61       20.3151 (0.55)        291           1
test_loads[simplejson-100 dicts array]     49.0000 (1.85)     161.8000 (2.67)     50.7473 (1.86)     3.4427 (2.67)     50.2000 (1.87)     0.5000 (5.00)     641;1277       19.7055 (0.54)      14145           1
test_loads[rapidjson-100 dicts array]      64.0000 (2.42)     151.6000 (2.50)     64.7998 (2.38)     2.3452 (1.82)     64.3000 (2.39)     0.2000 (2.00)     893;1359       15.4321 (0.42)      11548           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark '100 dicts array: serialize': 4 tests ---------------------------------------------------------------------------------
Name (time in us)                              Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-100 dicts array]          22.3000 (1.0)      125.4000 (2.14)      23.3430 (1.0)       3.4317 (3.74)      22.8000 (1.0)      0.3000 (1.50)     578;2700       42.8394 (1.0)       26596           1
test_dumps[rapidjson-100 dicts array]      24.0000 (1.08)      93.3000 (1.59)      24.9715 (1.07)      2.6849 (2.93)      24.5000 (1.07)     0.7000 (3.50)      408;873       40.0457 (0.93)      22989           1
test_dumps[hyperjson-100 dicts array]      24.7000 (1.11)      58.7000 (1.0)       25.2348 (1.08)      0.9172 (1.0)       25.1000 (1.10)     0.2000 (1.0)      659;1229       39.6278 (0.93)      23810           1
test_dumps[simplejson-100 dicts array]     99.1000 (4.44)     753.1000 (12.83)    106.5488 (4.56)     30.3878 (33.13)    104.0000 (4.56)     1.6000 (8.00)       27;286        9.3854 (0.22)       3045           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark '256 Trues array: deserialize': 4 tests -------------------------------------------------------------------------------
Name (time in us)                              Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-256 Trues array]           5.8000 (1.0)       39.3000 (1.0)       6.0530 (1.0)      0.5898 (1.09)      6.0000 (1.0)      0.0000 (1.0)    1269;43225      165.2083 (1.0)       96154           1
test_loads[hyperjson-256 Trues array]       8.2000 (1.41)      45.0000 (1.15)      8.3714 (1.38)     0.5422 (1.0)       8.3000 (1.38)     0.0000 (inf)    1463;17634      119.4539 (0.72)      75758           1
test_loads[simplejson-256 Trues array]      9.3000 (1.60)     162.4000 (4.13)     10.9177 (1.80)     1.8791 (3.47)     10.7000 (1.78)     0.2000 (inf)      892;3398       91.5945 (0.55)      43479           1
test_loads[rapidjson-256 Trues array]      11.3000 (1.95)      56.3000 (1.43)     11.4680 (1.89)     0.7680 (1.42)     11.4000 (1.90)     0.0000 (inf)     798;18179       87.1995 (0.53)      52911           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark '256 Trues array: serialize': 4 tests --------------------------------------------------------------------------------
Name (time in us)                              Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-256 Trues array]           7.0000 (1.0)      176.9000 (5.30)      7.2587 (1.0)      1.6623 (1.91)      7.1000 (1.0)      0.0000 (1.0)    1608;21859      137.7649 (1.0)       88496           1
test_dumps[rapidjson-256 Trues array]       7.8000 (1.11)      33.4000 (1.0)       8.0584 (1.11)     0.8683 (1.0)       8.0000 (1.13)     0.1000 (>1000.0)  903;3843      124.0941 (0.90)      62501           1
test_dumps[hyperjson-256 Trues array]      13.2000 (1.89)      79.1000 (2.37)     14.3822 (1.98)     5.8395 (6.72)     13.5000 (1.90)     0.3000 (>1000.0)  876;2258       69.5304 (0.50)      50252           1
test_dumps[simplejson-256 Trues array]     13.9000 (1.99)     193.6000 (5.80)     16.2828 (2.24)     3.9865 (4.59)     15.5000 (2.18)     0.2000 (>1000.0) 1894;5700       61.4145 (0.45)      35715           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark '256 ascii array: deserialize': 4 tests -------------------------------------------------------------------------------
Name (time in us)                              Min                 Max               Mean            StdDev             Median               IQR             Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[simplejson-256 ascii array]     30.5000 (1.0)      236.6000 (3.14)     33.5635 (1.03)     5.6790 (1.43)     31.3000 (1.0)      2.7000 (15.43)    2160;2293       29.7942 (0.98)      22372           1
test_loads[hyperjson-256 ascii array]      30.5000 (1.00)      75.3000 (1.0)      32.7298 (1.0)      3.9625 (1.0)      31.3000 (1.0)      0.4000 (2.29)     3117;4003       30.5532 (1.0)       26456           1
test_loads[ujson-256 ascii array]          37.7000 (1.24)     126.1000 (1.67)     39.8000 (1.22)     5.0793 (1.28)     38.1000 (1.22)     1.4000 (8.00)     2131;2399       25.1256 (0.82)      19231           1
test_loads[rapidjson-256 ascii array]      38.6000 (1.27)      82.2000 (1.09)     40.8158 (1.25)     4.2425 (1.07)     39.1000 (1.25)     0.1750 (1.0)      2114;3427       24.5003 (0.80)      15975           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark '256 ascii array: serialize': 4 tests --------------------------------------------------------------------------------
Name (time in us)                              Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-256 ascii array]          36.0000 (1.0)      185.9000 (2.26)     36.7395 (1.0)      3.7906 (2.75)     36.2000 (1.0)      0.2000 (1.0)      564;2243       27.2187 (1.0)       24155           1
test_dumps[rapidjson-256 ascii array]      41.5000 (1.15)     203.0000 (2.46)     43.9578 (1.20)     7.7568 (5.62)     42.4000 (1.17)     2.0000 (10.00)     269;658       22.7491 (0.84)      13423           1
test_dumps[hyperjson-256 ascii array]      49.3000 (1.37)      82.4000 (1.0)      50.4728 (1.37)     1.3806 (1.0)      50.2000 (1.39)     0.7000 (3.50)      836;839       19.8126 (0.73)      15129           1
test_dumps[simplejson-256 ascii array]     55.1000 (1.53)     117.0000 (1.42)     57.8957 (1.58)     4.8371 (3.50)     56.3000 (1.56)     0.6000 (3.00)    1082;2267       17.2724 (0.63)      13870           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark '256 doubles array: deserialize': 4 tests --------------------------------------------------------------------------------
Name (time in us)                                 Min                 Max                Mean            StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[hyperjson-256 doubles array]       19.0000 (1.0)       60.9000 (1.0)       20.0989 (1.0)      1.0490 (1.0)       20.0000 (1.0)      0.2000 (2.00)     924;3094       49.7539 (1.0)       32052           1
test_loads[ujson-256 doubles array]           20.9000 (1.10)      90.3000 (1.48)      21.2940 (1.06)     1.6211 (1.55)      21.1000 (1.05)     0.1000 (1.0)      877;2837       46.9617 (0.94)      32259           1
test_loads[simplejson-256 doubles array]     109.9000 (5.78)     226.9000 (3.73)     114.0146 (5.67)     3.9091 (3.73)     113.5000 (5.68)     1.3000 (13.00)    272;1235        8.7708 (0.18)       8355           1
test_loads[rapidjson-256 doubles array]      163.0000 (8.58)     283.4000 (4.65)     166.5846 (8.29)     3.2674 (3.11)     166.1000 (8.31)     1.6000 (16.00)     370;511        6.0030 (0.12)       5784           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------- benchmark '256 doubles array: serialize': 4 tests -----------------------------------------------------------------------------------
Name (time in us)                                 Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[hyperjson-256 doubles array]       28.1000 (1.0)         59.8000 (1.0)       28.5005 (1.0)       0.9845 (1.0)       28.3000 (1.0)      0.1000 (1.0)      839;3615       35.0871 (1.0)       26316           1
test_dumps[ujson-256 doubles array]          141.6000 (5.04)       733.8000 (12.27)    144.7966 (5.08)     14.6225 (14.85)    143.6000 (5.07)     0.8000 (8.00)       45;929        6.9062 (0.20)       5964           1
test_dumps[simplejson-256 doubles array]     298.1000 (10.61)    1,344.2000 (22.48)    307.1069 (10.78)    43.7562 (44.45)    303.2000 (10.71)    3.3000 (33.00)      21;133        3.2562 (0.09)       3100           1
test_dumps[rapidjson-256 doubles array]      309.0000 (11.00)      672.4000 (11.24)    316.2584 (11.10)    20.6584 (20.98)    314.1000 (11.10)    3.2000 (32.00)      56;117        3.1620 (0.09)       3004           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark '256 unicode array: deserialize': 4 tests -------------------------------------------------------------------------------------
Name (time in us)                                   Min                   Max                  Mean              StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[hyperjson-256 unicode array]        292.1000 (1.0)        533.7000 (1.0)        295.9779 (1.0)        9.9837 (1.10)       294.5000 (1.0)       3.1000 (1.0)         34;70  3,378.6306 (1.0)        2317           1
test_loads[rapidjson-256 unicode array]        526.6000 (1.80)       764.7000 (1.43)       545.1310 (1.84)       9.0738 (1.0)        544.6000 (1.85)      4.1000 (1.32)      113;133  1,834.4214 (0.54)       1716           1
test_loads[ujson-256 unicode array]            686.2000 (2.35)       966.0000 (1.81)       699.1723 (2.36)      15.3390 (1.69)       697.6000 (2.37)      7.0000 (2.26)        47;53  1,430.2625 (0.42)       1294           1
test_loads[simplejson-256 unicode array]     2,283.0000 (7.82)     5,014.3000 (9.40)     2,348.5469 (7.93)     229.7757 (25.32)    2,315.4500 (7.86)     25.9000 (8.35)        11;25    425.7952 (0.13)        424           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark '256 unicode array: serialize': 4 tests -----------------------------------------------------------------------------------
Name (time in us)                                 Min                   Max                Mean              StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[hyperjson-256 unicode array]      218.0000 (1.0)        259.2000 (1.0)      226.6817 (1.0)        3.8400 (1.00)     225.9000 (1.0)      2.6000 (1.0)       434;266        4.4115 (1.0)        3467           1
test_dumps[simplejson-256 unicode array]     316.8000 (1.45)       354.9000 (1.37)     322.2012 (1.42)       3.8219 (1.0)      321.3000 (1.42)     3.3000 (1.27)       179;71        3.1037 (0.70)       2074           1
test_dumps[ujson-256 unicode array]          402.1000 (1.84)     1,353.7000 (5.22)     410.9501 (1.81)      42.1465 (11.03)    407.4000 (1.80)     3.6000 (1.38)        15;91        2.4334 (0.55)       1755           1
test_dumps[rapidjson-256 unicode array]      610.3000 (2.80)     2,253.4000 (8.69)     678.6976 (2.99)     118.2246 (30.93)    659.5000 (2.92)     6.2000 (2.38)       24;177        1.4734 (0.33)       1279           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'complex object: deserialize': 4 tests --------------------------------------------------------------------------------
Name (time in us)                              Min                 Max                Mean            StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-complex object]           91.6000 (1.0)      212.4000 (1.0)       92.8738 (1.0)      2.6292 (1.0)       92.4000 (1.0)      0.4000 (1.0)      332;1087       10.7673 (1.0)        8704           1
test_loads[simplejson-complex object]     160.2000 (1.75)     288.8000 (1.36)     167.7454 (1.81)     3.7794 (1.44)     166.9000 (1.81)     1.9000 (4.75)      274;328        5.9614 (0.55)       5110           1
test_loads[hyperjson-complex object]      171.1000 (1.87)     245.1000 (1.15)     174.1984 (1.88)     3.6186 (1.38)     173.1000 (1.87)     1.5000 (3.75)      308;625        5.7406 (0.53)       4769           1
test_loads[rapidjson-complex object]      219.4000 (2.40)     257.1000 (1.21)     221.9343 (2.39)     2.9511 (1.12)     221.2000 (2.39)     1.2250 (3.06)      154;562        4.5058 (0.42)       3989           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'complex object: serialize': 4 tests ----------------------------------------------------------------------------------
Name (time in us)                              Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-complex object]           81.3000 (1.0)        458.3000 (2.52)      85.7690 (1.0)      15.9285 (5.57)      83.2000 (1.0)      1.4000 (1.56)      315;913       11.6592 (1.0)       10870           1
test_dumps[rapidjson-complex object]      128.3000 (1.58)       200.4000 (1.10)     130.6006 (1.52)      3.0583 (1.07)     130.0000 (1.56)     0.9000 (1.0)       284;857        7.6569 (0.66)       6120           1
test_dumps[hyperjson-complex object]      150.7000 (1.85)       182.0000 (1.0)      153.7086 (1.79)      2.8607 (1.0)      152.9000 (1.84)     1.1000 (1.22)      490;988        6.5058 (0.56)       5831           1
test_dumps[simplejson-complex object]     221.3000 (2.72)     1,962.1000 (10.78)    235.6258 (2.75)     62.1258 (21.72)    226.2000 (2.72)     3.9000 (4.33)       84;237        4.2440 (0.36)       2925           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark 'composite object: deserialize': 4 tests ------------------------------------------------------------------------------
Name (time in us)                               Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_loads[ujson-composite object]          30.3000 (1.0)       85.0000 (1.0)      31.0842 (1.0)      2.3733 (1.26)     30.6000 (1.0)      0.2000 (1.0)      681;1282       32.1707 (1.0)       11002           1
test_loads[simplejson-composite object]     77.4000 (2.55)     116.3000 (1.37)     79.8718 (2.57)     1.8853 (1.0)      79.4000 (2.59)     0.4000 (2.00)     867;1045       12.5201 (0.39)       9277           1
test_loads[rapidjson-composite object]      83.0000 (2.74)     221.8000 (2.61)     84.8234 (2.73)     3.9772 (2.11)     83.7000 (2.74)     0.7000 (3.50)      407;879       11.7892 (0.37)       5721           1
test_loads[hyperjson-composite object]      86.3000 (2.85)     291.9000 (3.43)     89.6700 (2.88)     8.0607 (4.28)     88.1000 (2.88)     2.1000 (10.50)     474;705       11.1520 (0.35)       6888           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'composite object: serialize': 4 tests -------------------------------------------------------------------------------
Name (time in us)                               Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_dumps[ujson-composite object]          27.0000 (1.0)      139.8000 (1.59)     28.1336 (1.0)      3.3845 (2.24)     27.6000 (1.0)      0.3000 (1.0)       329;824       35.5447 (1.0)       13832           1
test_dumps[rapidjson-composite object]      31.7000 (1.17)     200.1000 (2.27)     33.7203 (1.20)     4.3713 (2.90)     33.2000 (1.20)     0.5000 (1.67)      153;665       29.6558 (0.83)       9001           1
test_dumps[hyperjson-composite object]      59.7000 (2.21)      88.1000 (1.0)      61.6890 (2.19)     1.5093 (1.0)      61.6000 (2.23)     0.8000 (2.67)      808;602       16.2104 (0.46)       9303           1
test_dumps[simplejson-composite object]     72.4000 (2.68)     391.7000 (4.45)     78.4170 (2.79)     8.1614 (5.41)     75.2000 (2.72)     3.4000 (11.33)    996;1233       12.7523 (0.36)       6477           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

All around things look much much better. I'm impressed with how well hyperjson performs at the moment, even beating ujson several times!

[mre]

While there are still some outliers this is beginning to look competitive.
I will profile test_loads[hyperjson-composite object] to see if there is any low-hanging fruit.
Have a beefy new Linux machine now that I can use for that purpose.

Does anybody know of some webservice for profiling code? Was already thinking about getting a Hetzner Cloud instance and connecting it to Github with some API...

[alexprengere]

Hi! I recently switched from json to orjson and observed a significant performance boost when serializing JSON (typically 5x faster on my workload).
I think this is currently the fastest Python JSON library, so it would be really interesting to have orjson in your benchmark numbers.

Edit: I just saw #64, so I can see it is already integrated in your scripts, just not visible in the figures from the README

[mre]

Sure thing. @alexprengere would you be interested in creating a PR for adding it?

[alexprengere]

As I wrote in the edit, I think this is already done 😉 with 02e1884.
The way I understand the code, I think a make plot is enough to generate new graphs. I only have a Linux virtual machine, but if you feel the benchmark results will be relevant, I can generate graphs and make a PR.

[mre]

Would for sure be interesting to run make plot on a VM to see if it all works and get some preliminary benchmarks yes.
If anyone else also wants to chime in and provide some benchmarks for different machines please do!

[alexprengere]

So here are the plots on my Fedora 33 Virtualbox. I can post the full reports if necessary. A couple of pain points I encountered when running those:

• I had to manually install matplotlib
• when running the benchmarks, by default I had no benchmark.json generated, so I had to manually add the option poetry run pytest benchmarks --compare --benchmark-save=benchmark.json, perhaps something changed accross versions