Exploratory Data Analysis with Python and Pandas | Rockborne

4. Exploring the data

Next, we will explore the data by using the .head() method to see the top rows and the columns attribute to see the names of all the columns.

• df.head() – View first 5 rows
• df.tail() – View last 5 rows
• df.shape – Number of rows and columns
• df.columns – Return the names of the columns
• df.dtypes – Data type of each column
• df.info() – Index, datatype and memory information
• df.describe() – Summary statistics for numeric columns

Lets explore each of this method starting with “head”, the syntax for using it is:

DataFrame.head()

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.
The head documentation

In [3]:

##Calling the dataset will return the first 5 and last 5 rows of the dataset.
customer

Out[3]:

599 rows × 10 columns

In [4]:

##Now lets try head:
customer.head()

Out[4]:

Lets explore this method “tail”, the syntax for using it is:

DataFrame.tail()

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.The head documentation

In [5]:

##Tail:
customer.tail()

Out[5]:

Lets method “shape”, the syntax for using it is:

DataFrame.shape()

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.The shape documentation

In [6]:

##Shape, number of rows and number of columns:
customer.shape

Out[6]:

(599, 10)

Lets the method “columns”, the syntax for using it is:

DataFrame.columns

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.
The columns documentation

In [7]:

customer.columns

Out[7]:

Index(['customer_id', 'store_id', 'first_name', 'last_name', 'email',
       'address_id', 'activebool', 'create_date', 'last_update', 'active'],
      dtype='object')

Lets the method “dtypes”, the syntax for using it is:

DataFrame.dtypes

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.
The dtypes documentation

In [8]:

customer.dtypes

Out[8]:

customer_id     int64
store_id        int64
first_name     object
last_name      object
email          object
address_id      int64
activebool       bool
create_date    object
last_update    object
active          int64
dtype: object

df.info() should become standard practice when loading in new data in Python. It provides an informative data quality “dashboard” to guide deeper investigation or cleaning as needed. Catching potential problems early mitigates headaches later in analysis. The ease and speed of df.info() allows quick iterations to check data manipulations. Leveraging this single line of code accelerates EDA and sets the stage for reliable data science workflows.

• Provides concise summary of DataFrame contents – shape, data types, memory usage
• Identifies any incorrect or unexpected data types that could cause errors later
• Easily spots missing values represented as NaN with a count per column
• Can be used to compare before and after DataFrame changes during data cleaning

Lets explore each of this method exploring “info”, the syntax for using it is:

DataFrame.info()

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.The info documentation

In [20]:

customer.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 599 entries, 0 to 598
Data columns (total 10 columns):
 #   Column       Non-Null Count  Dtype 
---  ------       --------------  ----- 
 0   customer_id  599 non-null    int64 
 1   store_id     599 non-null    int64 
 2   first_name   599 non-null    object
 3   last_name    599 non-null    object
 4   email        599 non-null    object
 5   address_id   599 non-null    int64 
 6   activebool   599 non-null    bool  
 7   create_date  599 non-null    object
 8   last_update  599 non-null    object
 9   active       599 non-null    int64 
dtypes: bool(1), int64(4), object(5)
memory usage: 42.8+ KB

df.describe() is an extremely useful method in exploratory data analysis to summarize numerical data. Rapidly provides an insightful overview of key statistics that support exploratory analysis; the output reveals details about quality, variance, and anomalies that warrant further examination. Here are some key reasons it provides value:

• Calculates standard summary statistics like count, mean, std dev, min, max, and quartiles. Gives overview of distribution.
• Statistics help identify outliers, skewness, and shape characteristics.
• Count of non-null values highlights missing data if lower than total rows.
• Summary stats can reveal insights like high standard deviations and long tails.
• Provides a “first look” at numeric attributes to guide more advanced analysis.

Lets explore this method “describe”, the syntax for using it is:

 DataFrame.describe()

In this example DataFrame is the name of the table, in this case “customer” is already a dataset.
The describe documentation

In [22]:

customer.describe()

Out[22]:

In summary, Pandas provides many useful functions to explore and summarize key details about an unfamiliar dataset. Methods like df.head(), df.tail(), df.shape, df.columns, df.dtypes, and df.info() quickly provide an overview of the structure, size, and data types.
df.describe() generates descriptive statistics to study numeric columns. Taken together, these methods help you efficiently inspect the contents, completeness, and integrity of the data.

customer['first_name']

0           MARY
1       PATRICIA
2          LINDA
3        BARBARA
4      ELIZABETH
         ...    
594     TERRENCE
595      ENRIQUE
596      FREDDIE
597         WADE
598       AUSTIN
Name: first_name, Length: 599, dtype: object

customer[['first_name', 'email']]

customer[['first_name', 'last_name']].describe()

customer['store_id'].unique()

array([1, 2], dtype=int64)

customer['store_id'].nunique()

2

customer['store_id'].value_counts()

1    326
2    273
Name: store_id, dtype: int64

customer.sort_values(by='first_name').head()

customer[customer['store_id'] == 2].head()

customer.groupby('store_id').count()

customer[['store_id', 'customer_id']].groupby('store_id').count()

payment = pd.read_csv('payment.csv')
payment.head()

payment[['staff_id', 'amount']].groupby('staff_id').sum()

payment[['staff_id', 'amount']].groupby('staff_id').mean()

payment.hist(column='amount');