drop duplicates pandas

2025

3 min read

·

11-03-2025

10

0

Share

Pandas is a powerful Python library for data manipulation and analysis. A common task in data cleaning is removing duplicate rows. This guide will walk you through various methods to effectively drop duplicates in your Pandas DataFrames, covering different scenarios and considerations. Learning how to efficiently drop duplicates is a crucial skill for any data scientist or analyst working with Pandas.

Identifying and Understanding Duplicate Rows

Before diving into the methods, let's clarify what constitutes a duplicate row. A duplicate row is a row that has identical values across all its columns compared to another row in the DataFrame. Pandas provides flexible ways to identify and handle these duplicates, allowing you to specify which columns to consider when checking for duplicates.

Example DataFrame

Let's start with a sample DataFrame:

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'Alice', 'Bob', 'David'],
        'Age': [25, 30, 22, 25, 30, 28],
        'City': ['New York', 'London', 'Paris', 'New York', 'London', 'Tokyo']}

df = pd.DataFrame(data)
print(df)

This will output:

      Name  Age      City
0    Alice   25  New York
1      Bob   30    London
2  Charlie   22     Paris
3    Alice   25  New York
4      Bob   30    London
5    David   28     Tokyo

Methods for Dropping Duplicates in Pandas

Pandas offers several methods to drop duplicate rows, each with its own nuances:

1. DataFrame.duplicated()

This method identifies duplicate rows. It returns a boolean Series indicating whether each row is a duplicate (True) or not (False).

duplicates = df.duplicated()
print(duplicates)

This will output:

0    False
1    False
2    False
3     True
4     True
5    False
dtype: bool

2. DataFrame.drop_duplicates()

This is the primary method for removing duplicate rows. It returns a new DataFrame with duplicates removed.

df_no_duplicates = df.drop_duplicates()
print(df_no_duplicates)

This will output:

      Name  Age      City
0    Alice   25  New York
1      Bob   30    London
2  Charlie   22     Paris
5    David   28     Tokyo

By default, drop_duplicates() removes all duplicates, keeping the first occurrence.

3. Specifying Subset for Duplicate Detection

Often, you might want to consider only specific columns when identifying duplicates. The subset parameter allows you to specify which columns to consider. For instance, to remove duplicates based only on 'Name' and 'Age':

df_subset_duplicates = df.drop_duplicates(subset=['Name', 'Age'])
print(df_subset_duplicates)

This will output:

      Name  Age      City
0    Alice   25  New York
1      Bob   30    London
2  Charlie   22     Paris
5    David   28     Tokyo

Notice that even though Bob and Alice appear twice, only the first entry is kept since the combination of Name and Age uniquely identifies each row.

4. Keeping the Last Occurrence

The keep parameter controls which duplicate to keep. The default is 'first'. To keep the last occurrence, set keep='last':

df_keep_last = df.drop_duplicates(keep='last')
print(df_keep_last)

This will output:

      Name  Age      City
2  Charlie   22     Paris
3    Alice   25  New York
4      Bob   30    London
5    David   28     Tokyo

5. inplace=True for Modification

To modify the original DataFrame directly instead of creating a new one, use inplace=True:

df.drop_duplicates(inplace=True)
print(df)

This will modify df directly, removing the duplicates. Use caution with inplace=True, as it alters the original data.

Handling More Complex Duplicate Scenarios

Sometimes, you might have near-duplicates where values are slightly different but semantically the same (e.g., slight variations in spelling). Advanced techniques such as fuzzy matching might be needed in those cases, which are beyond the scope of this basic guide.

Conclusion

Pandas provides efficient and flexible tools for handling duplicate rows. By understanding the drop_duplicates() method and its parameters (subset, keep, inplace), you can effectively clean your data and prepare it for further analysis. Remember to choose the method that best suits your specific needs and always back up your data before making changes using inplace=True.