E.g. Single column:

E.g. Multiple columns:

We can use the methods we saw before for analysis on a subset of data, e.g.:

The pd.unique() method in Pandas can be very useful for exploratory data analysis to understand the distinct values present in a column and get a quick sense of the variability/cardinality of data in a column using nunique(). Can be combined with value_counts() for frequencies of unique values.

Let’s explore “unique”, “nunique” and “value_counts”, the syntax for using them is:

DataFrame[‘Column’].unique()

DataFrame[‘Column’].nunique()

DataFrame[‘Column’].value_counts()

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.

Here the unique documentation

E.g.: Let’s analyse how many stores are in the dataset and their frequency:

We can see that there are 2 stores and that are 326 customers register in strore 1 and 273 in store 2.

By reordering the rows of a DataFrame based on one or more columns, sort_values() reveals patterns, relationships, trends, and anomalies that may be obscured in unsorted data. Sorting numeric data into ascending or descending order highlights outliers and distributions. Chronologically sorting date columns provides temporal context.

Next, we will sort the customers by their first name.

The syntax for sorting the data is:

\# *Sort the data from lowest to highest*
DataFrame.sort_values(by = ‘Column’)

 

In this example DataFrame is the name of the DataFrame, Column is the name of the Column we want to sort by.

The sort_values documentation

By selectively removing rows based on logical criteria, you can focus on specific observations and patterns. For example, filtering numeric columns by thresholds reveals outliers and anomalies. Filtering categorical data on classes studies their distributions separately. Extracting rows with missing values helps quantify and understand NaN. Row filtering enables segmenting data like customers by region or users by age group.

If we wish to look at subsets of the data, we will need to filter or group it. Let’s start by learning to filter it.

To do so, we need to be able to choose a specific row from our table.

The syntax for filtering a DataFrame is:

\# *Filter the DataFrame*
DataFrame[ DataFrame[‘Column’] == ‘value’ ]

 

In this example DataFrame is the name of the DataFrame, Column is the name of the column we want to filter on and value is the value we’re interested in.

Lets make a subset of store “2”:

Fundamentally, groupby() enables dividing up heterogeneous data into homogeneous chunks to ask focused, comparative questions. It facilitates studying inter-group variability and intra-group consistency. Any aggregates, transformations, and visualizations applied within groups can reveal insights not visible in the full data. If we want to look at a customer’s aggregate transactions, we can use the .groupby() method to aggregate the data.

The syntax for aggregating the DataFrame is:

\# *Aggregate the DataFrame by group*
DataFrame.groupby(‘group’)

 

We can choose what kind of aggregate output we want for the columns by appending the appropreate method. E.g. sum()

\# *Aggregate the DataFrame by group*
DataFrame.groupby(‘group’).sum()

 

If we want to apply different aggregation methods to each column, we can with a different syntax

\# *Aggregate the DataFrame by group*
DataFrame.groupby(‘group’).agg({
‘Column1’ : ‘sum’, \# *take the sum of Column1*
‘Column2’ : ‘mean’, \# *take the mean of Column2*
‘Column3’ : ‘max’ \# *take the max of Column3*
})

In this example, group is the name of the Column we want to group by and sum/mean/max are the aggregation methods.

Here is the groupby documentation

For the next example we would like to replicate the distribution of clients per store (we did it with unique ;):

We can see that the output returns all the columns of the dataset, to clear it we can select a single (or multiple coulumns if needed) with the following syntaxt:

Let’s use a different dataset to exemplify better the groupby method, for these we are going to load the payment dataset:

We can make an analysis using group by of different aggregations (like counts, sums, min and max) on the amount feature. In this case we make an analysis by staff member and the sales they made:

The mean can also be analysed:

Here the list of the aggregations funtions available in Pandas:

Reference: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.sum.html

Visual representation of the data enables detection of patterns, relationships, trends, and anomalies that may not be observable in the raw data alone. Fundamentally, graphical exploration leads to new hypotheses, questions, and insights that drive the analysis forward. Generating effective visualizations during EDA is both an art and a science. Master the key plot types, know their strengths and weaknesses, and enrich your analysis with impactful, intuitive graphics. Visual data exploration should be a core part of any data scientist’s workflow. Here are some examples of visualisations and their benefit:

• Plots such as histograms, scatter plots, and box plots provide summaries of distributions that reveal insights into the shape, skew, outliers, and clusters within the data.
• Visualizing data categorized by color or facet makes comparisons across groups explicit.
• Time series plots surface seasonalities and temporal patterns.
• Correlation plots highlight relationships between variables.

Pandas allows us to plot data from the DataFrame. Pandas will plot for you most of the standard plots you would want.

Lets analyse the distribution of the amount of payments in the dataset with a histogram.

The syntax to make a histogram is:

DataFrame.hist(column = ‘Column’)

In this example DataFrame is the name of the DataFrame, Column is the name of the numerical column you would like to make the histogram on.

We can identify three peaks on the distribution on payments: the first one in 1 with 3,000 occurrences, the second on the value of 3 with about 3,500, and the highest in 5 with more than 5,000 occurrences; not many payments are made above the monetary value of 6.

Exploratory data analysis is a crucial first step in any data science undertaking, and Python’s Pandas library provides the perfect tools to perform effective EDA. By dedicating time to activities like data importing, cleaning, visualization, segmentation, and hypothesis generation using Pandas’ versatile built-in capabilities, you gain intimate understanding and derive insights that enable building better models and analytics systems downstream.
Thorough EDA leveraging Python and Pandas leads to more informed conclusions and impactful data-driven decision-making by uncovering key findings, relationships and anomalies that would otherwise go unseen. The power of these flexible data analysis tools makes EDA an indispensable part of the data science process.

I hope you enjoyed this quick introduction to this interesting subject. Make EDA a priority step in your data science projects!