Fix check for malformed frames.

Fix check for malformed frames.

Don't get confused by some malformed frames - always return -1 - and
assign the result of extract_header_length() to an int, so we don't get
warnings when we compare it against -1.  (Its return value always fits
in an int.)