]> The Tcpdump Group git mirrors - tcpdump/commit
ISO: avoid undefined behavior and integer overflow in the fletcher checksum calculation
authorBill Fenner <[email protected]>
Tue, 11 Oct 2022 20:10:46 +0000 (13:10 -0700)
committerFrancois-Xavier Le Bail <[email protected]>
Wed, 8 Jan 2025 14:50:19 +0000 (15:50 +0100)
commit2634bd1c16c3914d7809b925ecaeb3fcf7af65cc
treeb07c621206aa86f8fa236aa746cc95053cdcbf5b
parentd6ec7b78c7d8fe7628f396ad1b4d96747dcf6af5
ISO: avoid undefined behavior and integer overflow in the fletcher checksum calculation

The fletcher checksum calculation would sometimes left-shift
a negative number, which is an undefined operation.  Rework the
code to avoid this.

checksum.c:186:20: runtime error: left shift of negative value -36
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior checksum.c:186:20

Unlike some checksum routines that use the defined semantics of
2's-complement unsigned overflow to their advantage, this one
gets the wrong value if it is allowed to overflow, due to the
use of mod-255.

Convert c1 to uint64_t to avoid overflow.

checksum.c:163:16: runtime error: unsigned integer overflow: NNN + NNN
  cannot be represented in type 'unsigned int'

Use integers during subtraction to avoid implicit conversion to unsigned
when calculating both x and y

checksum.c:172:18: runtime error: unsigned integer overflow: NNN - NNN
  cannot be represented in type 'unsigned int'
checksum.c:172:9: runtime error: implicit conversion from type
  'unsigned int' of value NNN (32-bit, unsigned) to type 'int' changed
  the value to -NNN (32-bit, signed)
checksum.c:173:12: runtime error: unsigned integer overflow: NNN - NNN
  cannot be represented in type 'unsigned int'
checksum.c:173:9: runtime error: implicit conversion from type
  'unsigned int' of value NNN (32-bit, unsigned) to type 'int' changed
  the value to -NNN (32-bit, signed)

(backported from commit c5b54bfbd68b03f7997feaa277db30d399975a4d)
checksum.c
tests/TESTLIST
tests/fletcher-checksum-negative-shift.out [new file with mode: 0644]
tests/fletcher-checksum-negative-shift.pcap [new file with mode: 0644]