Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-18583

Improve/fix KRaftMetadataCache.getPartitionReplicaEndpoints

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.7.2, 3.8.1, 3.9.0
    • 3.8.2, 3.9.1, 4.0.0
    • core
    • None

    Description

      The KRaftMetadataCache.getPartitionReplicaEndpoints method, called when a replica selector is being used (e.g. for Fetch From Follower), currently takes a single TopicPartition for which to calculate node endpoints per replica ID, but then goes on to process all the partitions of the topic. As a result it seems to both calculate endpoints which are not relevant for the given partition and to recalculate the relevant endpoints multiple times as replicas of many different partitions.

      This could be very detrimental to performance if it touches topics with larger number of partitions.

      Unlike its KRaft counterpart, ZkMetadataCache.getPartitionReplicaEndpoints seems to do the right thing, processing only the partition info of the given partition. Also the problem doesn't seem to be a recent regression - it looks like it was present in the original form of KRaftMetadataCache introduced more than 3 years ago.

      Attachments

        Activity

          People

            dimitarndimitrov Dimitar Dimitrov
            dimitarndimitrov Dimitar Dimitrov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: