14.

7M Sign in

Search for articles, questions, tip

articles Q&A forums stuff lounge ?

Fastest Hash Function

for Table Lookups in
C?!
Sanmayce Rate me: 4.57/5 (16 votes)

30 Oct 2019 CPOL

Brutally optimized hash function for general table

lookups

Download Lookuperorama.c_r6 - 7.5 MB

Download source - 2.4 MB
Download ips - 7 MB

Introduction

This article is for those who need the FASTEST

hasher for their SMALL keys.

Update 2019-Oct-30

Fasten your belts, the fastest got boosted by

reducing 3 instructions...

Hide Shrink Copy Code

// Dedicated to Pippip, the main character

in the 'Das Totenschiff' roman, actually
the B.Traven himself, his real name was
Hermann Albert Otto Maksymilian Feige.
// CAUTION: Add 8 more bytes to the buffer
being hashed, usually malloc(...+8) - to
prevent out of boundary reads!
// Many thanks go to Yurii 'Hordi'
Hordiienko, he lessened with 3
instructions the original 'Pippip', thus:
//#include <stdlib.h>
//#include <stdint.h>
#define _PADr_KAZE(x, n) ( ((x) << (n))>>
(n) )
uint32_t FNV1A_Pippip_Yurii(const char
*str, size_t wrdlen) {
const uint32_t PRIME = 591798841;
uint32_t hash32; uint64_t hash64 =
14695981039346656037;
size_t Cycles, NDhead;
if (wrdlen > 8) {
Cycles = ((wrdlen - 1)>>4) + 1; NDhead
= wrdlen - (Cycles<<3);
#pragma nounroll
for(; Cycles--; str += 8) {
hash64 = ( hash64 ^ (*(uint64_t *)
(str)) ) * PRIME;
hash64 = ( hash64 ^ (*(uint64_t *)
(str+NDhead)) ) * PRIME;
}
} else
hash64 = ( hash64 ^ _PADr_KAZE(*
(uint64_t *)(str+0), (8-wrdlen)<<3) ) *
PRIME;
hash32 = (uint32_t)(hash64 ^
(hash64>>32)); return hash32 ^ (hash32 >>
16);
} // Last update: 2019-Oct-30, 14 C lines
strong, Kaze.

// https://godbolt.org/z/i40ipj x86-64 gcc

9.2 -O3
/*
FNV1A_Pippip_Yurii:
FNV1A_Pippip(char const*, unsigned int):
mov rax, QWORD PTR [rdi]
mov rax, QWORD PTR [rdi]
cmp rsi, 8
cmp esi, 8
jbe .L2
jbe .L2
lea rax, [rsi-1]
lea ecx, [rsi-1]
shr rax, 4
xor edx, edx
lea rdx, [8+rax*8]
shr ecx, 4
movabs rax, -3750763034362895579
add ecx, 1
sub rsi, rdx
lea eax, [0+rcx*8]
add rdx, rdi
sub esi, eax

movabs rax, -3750763034362895579

movsx rsi, esi

add rsi, rdi

.L3:
.L4:
xor rax, QWORD PTR [rdi]
xor rax, QWORD PTR [rdi+rdx*8]
add rdi, 8
imul rax, rax, 591798841
imul rax, rax, 591798841
xor rax, QWORD PTR [rsi+rdx*8]
xor rax, QWORD PTR [rdi-8+rsi]
add rdx, 1
imul rax, rax, 591798841
imul rax, rax, 591798841
cmp rdi, rdx
cmp ecx, edx
jne .L3
jg .L4
.L4:
.L3:
mov rdx, rax
mov rdx, rax
shr rdx, 32
shr rdx, 32
xor eax, edx
xor eax, edx
mov edx, eax
mov edx, eax
shr edx, 16
shr edx, 16
xor eax, edx
xor eax, edx
ret
ret
.L2:
.L2:
movabs rdx, -3750763034362895579
movabs rdx, -3750763034362895579
mov ecx, 8
mov ecx, 8
sub ecx, esi
sub ecx, esi
sal ecx, 3
sal ecx, 3
sal rax, cl
sal rax, cl
shr rax, cl
shr rax, cl
xor rax, rdx
xor rax, rdx
imul rax, rax, 591798841
imul rax, rax, 591798841
jmp .L4
jmp .L3
*/

Two of five benchmarks in Yann's suite (xxHash)

show these 3 instructions matter!
Note: GCC 7.3.0 executable was used, on i5-
7200U @3.1GHz turbo.

Let's see what reducing of above 3 instructions,

along with telling Intel v19.0 not to unroll, deliver
on my i5-7200U:

Hide Shrink Copy Code

KAZE_www.byronknoll.com_cmix-
v18.zip_english.dic:
44880 lines read
131072 elements in the table (17 bits)
Pippip_Yurii: 2325 [ 6822] !
Speed Brutality !
Pippip: 3046 [ 6822]
Totenschiff: 3308 [ 6818]
Yorikke: 2672 [ 6883]
wyhash: 5072 [ 6812]
FNV-1a: 4310 [ 6833]
CRC-32: 5251 [ 6891]
iSCSI CRC: 3631 [ 6785]

"KAZE_IPS_(3_million_IPs_dot_format).TXT":
2995394 lines read
8388608 elements in the table (23 bits)
Pippip_Yurii: 407608 [476410]
Pippip: 443241 [476410]
Totenschiff: 511103 [476467]
Yorikke: 530381 [506954]
wyhash: 551765 [476412]
FNV-1a: 716070 [477067]
CRC-32: 605808 [472854]
iSCSI CRC: 391876 [479542]
! Still 1st !

"KAZE_Word-list_12,561,874_wikipedia-en-
html.tar.wrd":
12561874 lines read
33554432 elements in the table (25 bits)
Pippip_Yurii: 2018116 [2084750]
! Speed Brutality !
Pippip: 2148478 [2084750]
Totenschiff: 2313835 [2084381]
Yorikke: 2383182 [2099673]
wyhash: 2787755 [2081865]
FNV-1a: 3123546 [2081195]
CRC-32: 2998909 [2075088]
iSCSI CRC: 2154190 [2077725]

KAZE_www.gutenberg.org_ebooks_100.txt:
138578 lines read
524288 elements in the table (19 bits)
Pippip_Yurii: 17178 [31196] !
Speed Brutality !
Pippip: 18174 [31196]
Totenschiff: 20336 [31134]
Yorikke: 23733 [31139]
wyhash: 28118 [31260]
FNV-1a: 49246 [31178]
CRC-32: 41789 [31210]
iSCSI CRC: 22606 [31248]

KAZE_www.maximumcompression.com_english.di
c:
354951 lines read
1048576 elements in the table (20 bits)
Pippip_Yurii: 41439 [53393] !
Speed Brutality !
Pippip: 44554 [53393]
Totenschiff: 48462 [53546]
Yorikke: 49988 [53782]
wyhash: 61612 [53996]
FNV-1a: 68586 [53896]
CRC-32: 65252 [54020]
iSCSI CRC: 46324 [53915]

KAZE_enwiki-20190920-pages-
articles.xml.SORTED.wrd:
42206534 lines read
134217728 elements in the table (27 bits)
Pippip_Yurii: 8253384 [5996107]
! Speed Brutality !
Pippip: 8734972 [5996107]
Totenschiff: 9215109 [5996598]
Yorikke: 9271283 [6011605]
wyhash: 11241704 [5991525]
FNV-1a: 12017273 [5980248]
CRC-32: 11570725 [5843653]
iSCSI CRC: 8433784 [5803092]

It turns out, no hasher can outperform FNV1A-

Pippip-Yurii on English wordlists, of any size :P

The three benchmark suites:

Peter Kankowski's hash.c compiled with

Intel v19.0;
Yann Collet's xxHash-0.7.2 compiled with
GCC v7.3.0;
Sanmayce's Lookuperorama.c compiled
with Intel v19.0 and GCC v7.3.0;

Lookuperorama.c_r7-.zip (252,329,722 bytes):

https://drive.google.com/file/d/1Uote1BFXJNwvM
ep13jlIfGyUAJlO1v20/view?usp=sharing

Update 2019-Oct-27

Wanted to clarify the "SMALL" term by running

Yann's hashBench suite taken from his github
repository, it currently features 5 different
scenarios/benchmarks, more in-depth testing at
his blog:
http://fastcompression.blogspot.com/2019/03/pre
senting-xxh3.html

It goes like this (on my laptop with i5-7200U

@3.1GHz turbo):

Hide Shrink Copy Code

E:\Lookuperorama.c_r6\xxHash-
0.7.2_Kaze\tests\bench>main --list
available hashes :
FNV1A_Pippip, xxh3, XXH32, XXH64, XXH128

E:\Lookuperorama.c_r6\xxHash-
0.7.2_Kaze\tests\bench>main --maxs=384 --
minl=7
=== benchmarking 5 hash functions ===

benchmarking large inputs : from 128 bytes

(log7) to 128 MB (log27)
FNV1A_Pippip, 12854.7, 11440.3,
8929.2, 7174.9, 6636.1, 6394.0,
6283.3, 6228.9, 6195.0, 6182.2,
6149.4, 6134.6, 6148.0, 6116.3,
6130.5, 5988.9, 6026.9, 6018.3,
6019.6, 6023.4, 6026.5
xxh3 , 12284.3, 10600.1, 14781.2,
17586.9, 19274.7, 20418.2,
20950.6, 21046.0, 21261.7, 21080.3,
21184.1, 21063.7, 20386.1,
20333.4, 19406.0, 14742.3, 13837.3,
13064.6, 13355.9, 13316.7, 13297.0
XXH32 , 5726.2, 5414.3, 5763.7,
5961.1, 6059.9, 6116.2,
6147.3, 6162.2, 6161.0, 6163.9,
6170.6, 6171.3, 6170.8,
6166.7, 6110.5, 5897.7, 5890.4,
5885.5, 5893.9, 5897.4, 5808.5
XXH64 , 6729.2, 9030.5, 10232.1,
10780.8, 11709.2, 12022.1,
12175.7, 12262.5, 12264.8, 12305.7,
12330.0, 12313.2, 12323.3, 12343.0,
12113.4, 11102.2, 10961.6, 10973.9,
10985.3, 10983.9, 10957.8
XXH128 , 9355.2, 7935.8, 11494.6,
14218.5, 16195.2, 17538.0,
18248.4, 18559.3, 18504.8, 18585.5,
18541.0, 18513.6, 18173.4,
18149.1, 17032.7, 13786.5, 13095.0,
12948.7, 12683.0, 12806.4, 12764.3
...

My hope was 'Pippip' to "hold the line" up to

280 bytes long keys, yes, the tweets maximal
length, by chance the XXH3 brutal domination
starts somewhere at this mark.

Update 2019-Oct-26

It's time for FASTESTNESS!

When tweaking my matchfinder, I saw how many
ideas were left unexploited in old hashers,
exploittime.

First the console run of Peter Kankowski's hash

suite (being one very precise micro-benchmark)
at https://www.strchr.com/hash_functions, it
shows the Pippip's supremacy while hashing the
dictionary of most powerful compressor on
Internet - CMIX:

The last column (in square brackets) shows

collisions, the number before collisions is the top
time.

If you happen to hash English words (~12 million

distinct) or Shakespeare's verses, 'Pippip'
screams:

Hide Shrink Copy Code

KAZE_www.byronknoll.com_cmix-
v18.zip_english.dic:
44880 lines read
131072 elements in the table (17 bits)

Pippip: 2463 [ 6822] !

Screaming speed !
Totenschiff: 2619 [ 6818]
Yorikke: 2674 [ 6883]
wyhash: 4371 [ 6812]
YoshimitsuTRIAD: 4035 [ 7006]
FNV-1a: 4283 [ 6833]
Larson: 4297 [ 6830]
CRC-32: 4342 [ 6891]
Murmur2: 4393 [ 6820]
Murmur3: 4320 [ 6874]
XXHfast32: 4573 [ 6812]
XXHstrong32: 4660 [ 6819]
iSCSI CRC: 3600 [ 6785]

"KAZE_Word-list_12,561,874_wikipedia-en-
html.tar.wrd":
12561874 lines read
33554432 elements in the table (25 bits)

Pippip: 2143388 [2084750]

! iSCSI CRC in the mirror,
are you
kidding me, what a beautiful brutality !
Totenschiff: 2300133 [2084381]
Yorikke: 2367380 [2099673]
wyhash: 2790935 [2081865]
YoshimitsuTRIAD: 2517971 [2084931]
FNV-1a: 3119297 [2081195]
Larson: 3017590 [2080111]
CRC-32: 2976146 [2075088]
Murmur2: 2858856 [2081476]
Murmur3: 2864098 [2082084]
XXHfast32: 3084063 [2084164]
XXHstrong32: 3191575 [2084514]
iSCSI CRC: 2155141 [2077725]

KAZE_www.gutenberg.org_ebooks_100.txt:
138578 lines read
524288 elements in the table (19 bits)

Pippip: 18195 [31196] !

Commentless I am !
Totenschiff: 20313 [31134]
Yorikke: 23758 [31139]
wyhash: 28252 [31260]
YoshimitsuTRIAD: 27014 [31316]
FNV-1a: 49282 [31178]
Larson: 48770 [31406]
CRC-32: 41691 [31210]
Murmur2: 31558 [31203]
Murmur3: 31336 [31308]
XXHfast32: 24637 [31146]
XXHstrong32: 27266 [31118]
iSCSI CRC: 22487 [31248]

KAZE_www.maximumcompression.com_english.di
c:
354951 lines read
1048576 elements in the table (20 bits)
Pippip: 44669 [53393] !
Fastest on all major English words,

a tear is falling !
Totenschiff: 48633 [53546]
Yorikke: 49951 [53782]
wyhash: 61668 [53996]
YoshimitsuTRIAD: 54825 [53658]
FNV-1a: 68106 [53896]
Larson: 67358 [54076]
CRC-32: 64756 [54020]
Murmur2: 62863 [53857]
Murmur3: 62782 [53983]
XXHfast32: 68146 [53411]
XXHstrong32: 70334 [53391]
iSCSI CRC: 46122 [53915]

Simply, all my benchmarking experience led to

writing the fastest hash function for tiny keys
(1..64 bytes long) - FNV1A-Pippip, let us see
what numbers say, in the showdown, other
participants are:

Wang's WYHASH - the current #1 in speed

department according to SMHASHER
Yann's XXH (taken from latest v0.7.2) family
- somewhere on top according to SMHASHER
SSE4.2 iSCSI CRC32
SHA3-224

The hardware CRC32 was the topmonster for a

long time, no more, now the supermonster
'Pippip' dethroned it, see for more versatile
benchmarks the IDZ Intel Developer Zone forum
(a link further below).

I have written Lookuperorama.c - the first both

SYNTHETIC and REAL-WORLD benchmark for
hash-table lookuping, current revision 6 is
attached as .ZIP at the top.
The .C source and how to compile it under GCC
and ICC:

_MakeEXE_Lookuperorama_GCC.bat

Hide Shrink Copy Code

gcc -O3 -msse4.1 -fomit-frame-pointer

Lookuperorama.c
-o Lookuperorama_GCC-7.3.0_WY_64bit.exe -
D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
DHashInBITS=24 -DHashChunkSizeInBITS=24
-DRAMpoolInKB=5120 -DBtreeHEURISTIC -
D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_WY

gcc -O3 -msse4.1 -fomit-frame-pointer

Lookuperorama.c
-o Lookuperorama_GCC-
7.3.0_Pippip_64bit.exe -D_N_XMM -
D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
DHashInBITS=24 -DHashChunkSizeInBITS=24
-DRAMpoolInKB=5120 -DBtreeHEURISTIC -
D_WIN32_ENVIRONMENT_
-DLongestLineInclusive=64 -D_N_Pippip

gcc -O3 -msse4.1 -fomit-frame-pointer

Lookuperorama.c xxhash.c
-o Lookuperorama_GCC-7.3.0_XXH64_64bit.exe
-D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
DHashInBITS=24 -DHashChunkSizeInBITS=24
-DRAMpoolInKB=5120 -DBtreeHEURISTIC -
D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_XXH64

gcc -O3 -msse4.1 -fomit-frame-pointer

Lookuperorama.c xxhash.c
-o Lookuperorama_GCC-7.3.0_XXH3_64bit.exe
-D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
DHashInBITS=24 -DHashChunkSizeInBITS=24
-DRAMpoolInKB=5120 -DBtreeHEURISTIC -
D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_XXH3

gcc -O3 -msse4.1 -fomit-frame-pointer

Lookuperorama.c
-o Lookuperorama_GCC-
7.3.0_CRC32C_64bit.exe -D_N_XMM -
D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
DHashInBITS=24 -DHashChunkSizeInBITS=24
-DRAMpoolInKB=5120 -DBtreeHEURISTIC -
D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_CRC32C

gcc -O3 -msse4.1 -fomit-frame-pointer

Lookuperorama.c
-o Lookuperorama_GCC-7.3.0_SHA3_64bit.exe
-D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
DHashInBITS=24 -DHashChunkSizeInBITS=24
-DRAMpoolInKB=5120 -DBtreeHEURISTIC -
D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_SHA3

_MakeEXE_Lookuperorama_ICL.bat

Hide Shrink Copy Code

icl /TP /O3 /arch:SSE4.1 Lookuperorama.c -

D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
D_icl_mumbo_jumbo_ /FAcs -DHashInBITS=24
-DHashChunkSizeInBITS=24 -
DRAMpoolInKB=5120 -DBtreeHEURISTIC
-D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_WY
copy Lookuperorama.exe Lookuperorama_ICC-
v19.0_WY_64bit.exe /y

icl /TP /O3 /arch:SSE4.1 Lookuperorama.c -

D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
D_icl_mumbo_jumbo_ /FAcs -DHashInBITS=24
-DHashChunkSizeInBITS=24 -
DRAMpoolInKB=5120 -DBtreeHEURISTIC
-D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_Pippip
copy Lookuperorama.exe Lookuperorama_ICC-
v19.0_Pippip_64bit.exe /y

icl /TP /O3 /arch:SSE4.1 Lookuperorama.c

xxhash.c -D_N_XMM
-D_N_prefetch_4096 -D_N_alone -
D_N_HIGH_PRIORITY -D_icl_mumbo_jumbo_
/FAcs
-DHashInBITS=24 -DHashChunkSizeInBITS=24 -
DRAMpoolInKB=5120
-DBtreeHEURISTIC -D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_XXH64
copy Lookuperorama.exe Lookuperorama_ICC-
v19.0_XXH64_64bit.exe /y

icl /TP /O3 /arch:SSE4.1 Lookuperorama.c

xxhash.c -D_N_XMM
-D_N_prefetch_4096 -D_N_alone -
D_N_HIGH_PRIORITY -D_icl_mumbo_jumbo_
/FAcs
-DHashInBITS=24 -DHashChunkSizeInBITS=24 -
DRAMpoolInKB=5120 -DBtreeHEURISTIC
-D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_XXH3
copy Lookuperorama.exe Lookuperorama_ICC-
v19.0_XXH3_64bit.exe /y

icl /TP /O3 /arch:SSE4.1 Lookuperorama.c -

D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
D_icl_mumbo_jumbo_ /FAcs -DHashInBITS=24
-DHashChunkSizeInBITS=24 -
DRAMpoolInKB=5120 -DBtreeHEURISTIC
-D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_CRC32C
copy Lookuperorama.exe Lookuperorama_ICC-
v19.0_CRC32C_64bit.exe /y

icl /TP /O3 /arch:SSE4.1 Lookuperorama.c -

D_N_XMM -D_N_prefetch_4096
-D_N_alone -D_N_HIGH_PRIORITY -
D_icl_mumbo_jumbo_ /FAcs -DHashInBITS=24
-DHashChunkSizeInBITS=24 -
DRAMpoolInKB=5120 -DBtreeHEURISTIC
-D_WIN32_ENVIRONMENT_ -
DLongestLineInclusive=64 -D_N_SHA3
copy Lookuperorama.exe Lookuperorama_ICC-
v19.0_SHA3_64bit.exe /y

Funny, I tried latest Intel v19.0 compiler and GCC

7.3.0, the choice of GCC team not to unroll
proved superior since tests are with short keys,
ICC decided to unroll, try and see the results
yourself.

For '1001 Nights' (15,583,440 bytes) and 10

hash-tables 24bit each, ICC v19.0, i7-3630QM,
Windows 10:

Hide Copy Code

Hasher | Collisions |
Synthetic/Taxed Hashing Speed for ALL keys
|
------------------------------------------
----------------------------------------
SSE4.2 iSCSI CRC32 | 23,737,621 |
208,334,772 KEYS-PER-SECOND / 5,565,507
Keys/s |
SHA3-224 | 23,737,389 |
104,465 KEYS-PER-SECOND / 102,052 Keys/s
|
------------------------------------------
----------------------------------------

For '1001 Nights' (15,583,440 bytes) and 10

hash-tables 24bit each, GCC 7.3.0, i7-3630QM,
Windows 10:

Hide Copy Code

Hasher | Collisions |
Synthetic/Taxed Hashing Speed for ALL keys
|
------------------------------------------
----------------------------------------
FNV1A-Pippip | 23,738,813 |
269,144,006 KEYS-PER-SECOND / 5,026,910
Keys/s |
WYHASH | 23,738,215 |
134,688,314 KEYS-PER-SECOND / 5,026,910
Keys/s |
XXH64 | 23,737,218 |
115,861,992 KEYS-PER-SECOND / 4,869,819
Keys/s |
XXH3 | 23,735,377 |
174,702,219 KEYS-PER-SECOND / 5,026,910
Keys/s |
------------------------------------------
----------------------------------------

Note 1: Collisions are cumulative for all the 10

hash-tables.
Note 2: 'Taxed' means it is polluted with many
random/uncached RAM accesses.
Like every other website we use cookies. By using our site you
This is how the console dump for 'Pippip' looks
acknowledge that you have read and understand our Cookie
like:
Policy, Privacy Policy, and our Terms of Service. Learn more

Ask me later Hide

AllowShrink
cookies Copy Code

F:\Lookuperorama.c_r6>Lookuperorama_GCC-