Python - Remove Duplicate Dictionaries characterized by Key

Given a list of dictionaries, remove all the dictionaries which are duplicate with respect to K key.

Input : test_list = [{"Gfg" : 6, "is" : 9, "best" : 10}, {"Gfg" : 8, "is" : 11, "best" : 10}, {"Gfg" : 2, "is" : 16, "best" : 10}], K = "best" Output : [{"Gfg" : 6, "is" : 9, "best" : 10}] Explanation : All keys have 10 value, only 1st record is retained. Input : test_list = [{"Gfg" : 6, "is" : 9, "best" : 10}, {"Gfg" : 8, "is" : 11, "best" : 12}, {"Gfg" : 2, "is" : 16, "best" : 15}], K = "best" Output : [{"Gfg" : 6, "is" : 9, "best" : 10}, {"Gfg" : 8, "is" : 11, "best" : 12}, {"Gfg" : 2, "is" : 16, "best" : 15}] Explanation : All values of "best" are unique, hence no removal of dictionaries.

Method : Using loop 

This is brute way in which this task can be performed. In this, we iterate for each dictionary and memoize the Key, if similar key's same value occur, then that dictionary is avoided in resultant list of dictionaries.

# Python3 code to demonstrate working of 
# Remove Duplicate Dictionaries characterized by Key
# Using loop

# initializing lists
test_list = [{"Gfg" : 6, "is" : 9, "best" : 10}, 
             {"Gfg" : 8, "is" : 11, "best" : 19},
             {"Gfg" : 2, "is" : 16, "best" : 10},
             {"Gfg" : 12, "is" : 1, "best" : 8},
             {"Gfg" : 22, "is" : 6, "best" : 8}]

# printing original list
print("The original list : " + str(test_list))

# initializing Key 
K = "best"

memo = set()
res = []
for sub in test_list:
    
    # testing for already present value
    if sub[K] not in memo:
        res.append(sub)
        
        # adding in memo if new value
        memo.add(sub[K])
    
# printing result 
print("The filtered list : " + str(res))

METHOD 2 :  use list comprehension.

 Here are the steps to implement it:

1. Initialize the input list:
2. Specify the key to be used for identifying duplicates
3. Use list comprehension to filter out duplicates:
4. This line of code creates a set of tuples where each tuple represents a dictionary in the input list. The dict() function is then used to convert each tuple back into a dictionary, and the resulting list contains only unique dictionaries.
5. Print the filtered list:

# Initializing the input list
test_list = [{"Gfg" : 6, "is" : 9, "best" : 10}, 
             {"Gfg" : 8, "is" : 11, "best" : 19},
             {"Gfg" : 2, "is" : 16, "best" : 10},
             {"Gfg" : 12, "is" : 1, "best" : 8},
             {"Gfg" : 22, "is" : 6, "best" : 8}]

# Specify the key to be used for identifying duplicates
K = "best"

# Use list comprehension to filter out duplicates
res = [dict(t) for t in {tuple(d.items()) for d in test_list}]

# Print the filtered list
print("The filtered list : " + str(res))

The filtered list : [{'Gfg': 2, 'is': 16, 'best': 10}, {'Gfg': 8, 'is': 11, 'best': 19}, {'Gfg': 6, 'is': 9, 'best': 10}, {'Gfg': 22, 'is': 6, 'best': 8}, {'Gfg': 12, 'is': 1, 'best': 8}]

The time complexity of this approach is O(nlogn) due to the use of sets and tuples.

 Auxiliary space complexity is O(n) because we are creating a new list of dictionaries.