Python - Remove Duplicates from a list And Keep The Order

While lists provide a convenient way to manage collections of data, duplicates within a list can sometimes pose challenges. In this article, we will explore different methods to remove duplicates from a Python list while preserving the original order.

Using dict.fromkeys()

dict.fromkeys() method creates a dictionary with keys from the list, automatically removing duplicates because dictionary keys are unique. The order is preserved where dictionaries maintain insertion order.

a = [1, 2, 2, 3, 4, 4, 5]

b = list(dict.fromkeys(a))
print(b) 

[1, 2, 3, 4, 5]

Explanation:

• dict.fromkeys(lst) creates a dictionary with list elements as keys.

Let's see some more methods to remove duplicates and keep the order in Python List.

Using a Set()

This method iterates through the list, adding elements to a set to track duplicates and using list comprehension to create a new list that retains only unique elements.

a = [1, 2, 2, 3, 4, 4, 5]

seen = set()
b = [x for x in a if not (x in seen or seen.add(x))]
print(b)  

[1, 2, 3, 4, 5]

Using a Loop

This is a simple method that involves iterating over the list and appending elements to a new list if they are not already present.

a = [1, 2, 2, 3, 4, 4, 5]

b = []

for x in a:
    if x not in b:
        b.append(x)
print(b)

[1, 2, 3, 4, 5]

Explanation:

• Loop through the original list.
• Check if the element is already in unique_list.
• Append it only if it’s not present.

Using OrderedDict

Before Python 3.7, dictionaries were not guaranteed to maintain order. To remove duplicates while preserving order in older versions, we could use collections.OrderedDict.

from collections import OrderedDict

a = [1, 2, 2, 3, 4, 4, 5]

b = list(OrderedDict.fromkeys(a))
print(b)  

[1, 2, 3, 4, 5]

Explanation:

• OrderedDict.fromkeys() ensures unique keys while maintaining order.