PREV NEXT

⏮ Custom partition  Building Data Streaming Applications with Apache Kafka Java Kafka producer example ⏭

Additional producer configuration

There are other optional configuration properties available for Kafka producer that can play an important role in

performance, memory, reliability, and so on:

buffer.memory: This is the amount of memory that producer can use to buffer a message that is waiting to be sent
to the Kafka server. In simple terms, it is the total memory that is available to the Java producer to collect
unsent messages. When this limit is reached, the producer will block the messages for max.block.ms before
raising an exception. If your batch size is more, allocate more memory to the producer buffer.

Additionally, to avoid keeping records queued indefinitely, you can set a timeout using
request.timeout.ms. If this timeout expires before a message can be successfully sent, then it will be
removed from the queue and an exception will be thrown.

acks: This configuration helps in configuring when producer will receive acknowledgment from the leader
before considering that the message is committed successfully:

acks=0: Producer will not wait for any acknowledgment from the server. Producer will not know if the
message is lost aton
Find answers any
thepoint
fly, orinmaster
time and is not committed
something by the today.
new. Subscribe leader See
broker. Note
pricing that no retry will
options.
happen in this case and the message will be completely lost. This can be used when you want to achieve
very high throughput and when you don't care about potential message loss.

acks=1: Producer will receive an acknowledgment as soon as the leader has written the message to its local
log. If the leader fails to write the message to its log, producer will retry sending the data according to the
retry policy set and avoid potential loss of messages. However, we can still have message loss in a scenario
where the leader acknowledges to producer but does not replicate the message to the other broker before it
goes down.

acks=all: Producer will only receive acknowledgment when the leader has received acknowledgment for all
the replicas successfully. This is a safe setting where we cannot lose data if the replica number is sufficient
to avoid such failures. Remember, throughput will be lesser then the first two settings.

batch.size: This setting allows the producer to batch the messages based on the partition up to the configured
amount of size. When the batch reaches the limit, all messages in the batch will be sent. However, it's not
necessary that producer wait for the batch to be full. It sends the batch after a specific time interval without
worrying about the number of messages in the batch.

linger.ms: This represents an amount of time that a producer should wait for additional messages before
sending a current batch to the broker. Kafka producer waits for the batch to be full or the configured linger.ms
time; if any condition is met, it will send the batch to brokers. Producer will wait till the configured amount of
time in milliseconds for any additional messages to get added to the current batch.

compression.type: By default, producer sends uncompressed messages to brokers. When sending a single
message, it will not make that much sense, but when we use batches, it's good to use compression to avoid
network overhead and increase throughput. The available compressions are GZIP, Snappy, or LZ4. Remember
that more batching would lead to better compression.

retires: If message sending fails, this represents the number of times producer will retry sending messages
before it throws an exception. It is irrespective of reseeding a message after receiving an exception.

max.in.flight.requests.per.connection: This is the number of messages producer can send to brokers without
waiting for a response. If you do not care about the order of the messages, then setting its value to more than 1
will increase throughput. However, ordering may change if you set it to more than 1 with retry enabled.

partitioner.class: If you want to use a custom partitioner for your producer, then this configuration allows you
to set the partitioner class, which implements the partitioner interface.

timeout.ms: This is the amount of time a leader will wait for its followers to acknowledge the message before
sending an error to producer. This setting will only help when acks is set to all.

Support / Sign Out

© 2021 O'Reilly Media, Inc. Terms of Service / Privacy Policy