How to Calculate Jaccard Similarity in Python

In Data Science, Similarity measurements between the two sets are a crucial task. Jaccard Similarity is one of the widely used techniques for similarity measurements in machine learning, natural language processing and recommendation systems. This article explains what Jaccard similarity is, why it is important, and how to compute it with Python.

What is Jaccard Similarity?

Jaccard Similarity also known as Jaccard index, is a statistic to measure the similarity between two data sets. It is measured as the size of the intersection of two sets divided by the size of their union.

For example: Given two sets A and B, their Jaccard Similarity is provided by,

Jaccard Similarity

[Tex]\text{Jaccard Similarity }J(A, B) = \frac{|A \cap B|}{|A \cup B|} [/Tex]

Where:

• [Tex]|A \cap B| [/Tex] is the cardinality (size) of the intersection of sets A and B.
• [Tex]|A\cup B| [/Tex] is the cardinality (size) of the union of sets A and B.

Jaccard Similarity is also known as the Jaccard index or Jaccard coefficient, its values lie between 0 and 1. where 0 means no similarity and the values get closer to 1 means increasing similarity 1 means the same datasets.

Computing Jaccard Similarity

EXAMPLE: 1

Output:

AnB = {1, 2}
AUB = {1, 2, 3, 4, 5, 6, 8, 9}
J(A,B) = 0.25

EXAMPLE: 2

The Jaccard similarity can be used to compare the similarity of two sets of words, which are frequently represented as sets of unique terms.

Output:

Jaccard Similarity: 0.25

Significance of Jaccard Similarity

The Jaccard similarity is especially effective when the order of items is irrelevant and only the presence or absence of elements is examined. It is extensively used in:

1. Text Analysis: Jaccard similarity can be used in natural language processing to compare texts, text samples, or even individual words.
2. Recommendation Systems: Jaccard similarity can help in finding similar items or products based on user behavior.
3. Data Deduplication: Jaccard similarity can be used to find duplicate or near-duplicate records in a dataset.
4. Social Network Analysis: Jaccard similarity can be used in social networks to detect similarities between user profiles or groups.
5. Genomics: Jaccard similarity is employed to compare gene sets in biological studies.

Jaccard Distance

The Jaccard distance is a measure of how different two sets are i.e Unlike the Jaccard coefficient, which determines the similarity of two sets. The Jaccard distance is computed by subtracting the Jaccard coefficient from one, or by dividing the difference in the sizes of the union and the intersection of two sets by the size of the union.

Jaccard Distance

[Tex]\begin{aligned} \text{Jaccard Distance}J_D(A,B) &=1-\text{Jaccard Similarity}J(A, B) \\&=1- \frac{|A \cap B|}{|A \cup B|} \\&=\frac{|A \cup B|-|A \cap B|}{|A \cup B|} \\&=\frac{|A\triangle B|}{|A \cup B|} \end{aligned} [/Tex]

Where:

• [Tex]|A \cap B| [/Tex] is the cardinality (size) of the intersection of sets A and B.
• [Tex]|A\cup B| [/Tex] is the cardinality (size) of the union of sets A and B.
• [Tex]|A \triangle B| [/Tex] represents the cardinality (size) of symmetric difference of sets (A) and (B), containing elements that are in either set but not in their intersection.

The Jaccard distance is often used to calculate a nxn matrix For clustering and multidimensional scaling of n sample sets. This distance is a collection metric for all finite sets.

Example 1:

Output:

Jaccard distance: 0.75

EXAMPLE 2:

Suppose two persons, A and B, went shopping in a department store, and there are five items. Let A = {1, 1,1, 0,1} and B = {1, 1, 0, 0, 1} sets represent items they picked (1) or not (0). Then &#x2018Jaccard score&#x2019 will represent the similar items they bought, and Jaccard Distance measure of dissimilarity and is calculated as 1 minus the Jaccard similarity score:

Output:

Jaccard Index: 0.75
Jaccard Distance: 0.25

Conclusion

The Jaccard similarity coefficient is a useful tool to check the similarity of sets, with applications ranging from text analysis to recommendation systems to data deduplication. You may quickly compute Jaccard similarity to improve your data analysis and decision-making processes by learning the formula and employing Python’s capabilities.