Consider enabling BlockBuffering or exposing buffering option

[dylex]

Currently http connections are opened as unbuffered, so each individual call to connectionWrite results in a packet on the wire (as I noticed looking at tcpdumps). Especially for RequestBodyStreamChunked this results in a lot of overhead per chunk. It seems like it would be safe and result in much better performance to simply switch the connection to block buffered and do an explicit flush after the entire request is sent.

[snoyberg]

The API is intended that you will do the buffering yourself, such as by using a Builder. Higher level libraries like http-conduit and conduit-extra make this particularly easy. Forcing http-client to handle the chunking instead would be problematic, since it could introduce unwanted delays.

[dylex]

That makes sense and is reasonable, but my complaint is more that for each chunk, the delimiters go into separate packets. For example, I'm seeing on the wire (during tcp slow start), length "\r\n", ack, chunk, ack, "\r\n", ack, which is two extra round-trips. It would be better to include the delimiters with the chunk.

[snoyberg]

The right solution here would involve an API change, namely changing the request body from a stream of ByteStrings to a stream of Builders. Otherwise, we'll end up with a lot of buffer copying. That said, buffer copying is probably better than three packets per chunk, so I'm not opposed to making the change.

Can you confirm that for the non-chunked case, there is currently no overhead detected with tcpdump?

[dylex]

Sounds good. I'd be happy with a new RequestBodyStreamChunkedBuilder or something to fix this, too.

Yes, for RequestBodyStream I see packet boundaries only between each popped block (and between the headers and the first block, of course).

[snoyberg]

A fix for this is now on master, can you test that out?

[dylex]

Looks good. I now see one packet per chunk, and the request is successful. The large request I'm testing with is now 351 total packets instead of 980. Thanks!

[snoyberg]

No problem. Uploading new version to Hackage now.