Descriptive Statistics with Pandas

Pandas is the cornerstone library for data analysis in Python, and one of its greatest strengths is the rich set of tools it provides for descriptive statistics — methods that summarize, aggregate, and explore datasets without requiring you to write statistical formulas from scratch. Whether you are working with sensor readings, experimental measurements, survey responses, or financial records, these tools let you rapidly understand the shape, spread, and relationships hidden inside your data. This topic walks through each major category of descriptive analysis in depth, with practical examples drawn from realistic engineering and scientific contexts.

Summary Statistics with describe()

The fastest way to get a broad overview of a DataFrame is the describe() method. When called on an entire DataFrame it returns a new DataFrame whose rows are named statistical measures — count, mean, std, min, the 25th percentile (25%), the median (50%), the 75th percentile (75%), and max — and whose columns match every numeric column in the original data. This side-by-side layout makes it trivial to compare the scale and variability of different variables at a glance.

import pandas as pd

# Suppose we have temperature and pressure readings from a reactor
data = {
    'temperature_C': [210, 215, 209, 222, 218, 211, None, 216],
    'pressure_bar':  [1.2, 1.3, 1.1, 1.5, 1.4, 1.2,  1.3, 1.3],
    'flow_rate_Ls':  [5.1, 5.3, 5.0, 5.8, 5.6, 5.1,  5.2, 5.4]
}
df = pd.DataFrame(data)

print(df.describe())

The output will show, for example, that temperature_C has a count of 7 (because one value is missing), a mean near 214 °C, and a standard deviation that reflects the spread of readings. You can immediately see whether columns share a similar scale or whether one variable dominates the others numerically — essential information before feeding data into any model or visualization.

By default describe() operates only on numeric columns. To include object (string) or categorical columns you can pass include='all', which adds statistics like unique, top (the most frequent value), and freq (its frequency count) for non-numeric columns.

print(df.describe(include='all'))

Individual statistical methods are also available directly on a Series (a single column). These are useful when you need one specific number rather than the full summary table, or when you are computing statistics programmatically inside a function.

• df['temperature_C'].mean() — arithmetic mean, ignoring NaN values by default (skipna=True).
• df['temperature_C'].std() — sample standard deviation (divides by n − 1 by default, matching the Bessel-corrected formula used in most engineering standards).
• df['temperature_C'].min() and df['temperature_C'].max() — the extreme values, useful for setting axis limits or checking for physically impossible readings.
• df['temperature_C'].median() — the 50th percentile; robust to outliers in a way the mean is not.
• df['temperature_C'].quantile(0.9) — arbitrary quantile; here the 90th percentile.

print("Mean temperature:", df['temperature_C'].mean())
print("Std deviation:",    df['temperature_C'].std())
print("Max pressure:",     df['pressure_bar'].max())
print("Min flow rate:",    df['flow_rate_Ls'].min())

For categorical or discrete data, value_counts() is the go-to method. It counts how many times each unique value appears in the column and returns the results sorted from most to least frequent. This is invaluable for understanding the distribution of labels, product codes, failure modes, or any other non-numeric variable.

status_data = pd.Series(['pass', 'fail', 'pass', 'pass', 'fail',
                         'pass', 'inconclusive', 'pass', 'fail'])

print(status_data.value_counts())
# pass            5
# fail            3
# inconclusive    1
# dtype: int64

You can also normalize the counts to proportions by passing normalize=True, which immediately gives you the relative frequency — equivalent to the empirical probability of each category.

print(status_data.value_counts(normalize=True))
# pass            0.555556
# fail            0.333333
# inconclusive    0.111111

Aggregation with groupby()

Summary statistics computed over an entire column treat every row the same. In practice, data almost always contains one or more grouping variables — a plant site, a shift identifier, an experimental condition, a material grade — and the interesting question is how numeric outcomes differ between those groups. The groupby() method splits the DataFrame into sub-groups defined by the values of one or more columns, applies a statistical function to each group, and combines the results into a new DataFrame.

data2 = {
    'shift':        ['A','A','A','B','B','B','C','C','C'],
    'temperature_C':[210,215,209,222,218,221,211,213,210],
    'yield_pct':    [88.1,89.0,87.5,91.2,90.8,91.5,88.8,89.1,88.5]
}
df2 = pd.DataFrame(data2)

# Mean of every numeric column, grouped by shift
print(df2.groupby('shift').mean())

The result has one row per unique value of shift (A, B, C) and one column per numeric variable. You can instantly see that shift B operates at a higher temperature and achieves a higher yield, which might warrant further investigation.

When you need more than one statistic per group, use .agg() and pass a list of function names as strings. Pandas applies every function in the list to every numeric column and returns a hierarchically indexed result.

grouped = df2.groupby('shift')['yield_pct'].agg(['mean', 'std', 'count'])
print(grouped)
#           mean       std  count
# shift
# A     88.2000  0.762972      3
# B     91.1667  0.360555      3
# C     88.8000  0.300000      3

You can also apply different aggregation functions to different columns in a single call by passing a dictionary to .agg(), where keys are column names and values are the functions to apply to each.

result = df2.groupby('shift').agg({
    'temperature_C': ['mean', 'max'],
    'yield_pct':     ['mean', 'std']
})
print(result)

Grouping by multiple columns simultaneously works by passing a list of column names to groupby(). The resulting index is a MultiIndex with one level per grouping column. This is particularly useful in factorial experimental designs or when data is segmented by two independent factors, such as plant site and shift.

data3 = {
    'site':  ['North','North','South','South','North','South'],
    'shift': ['A','B','A','B','A','B'],
    'yield': [88.0, 91.0, 87.5, 90.5, 88.5, 91.5]
}
df3 = pd.DataFrame(data3)

print(df3.groupby(['site', 'shift']).mean())
#                   yield
# site  shift
# North A        88.250
#       B        91.000
# South A        87.500
#       B        91.000

After grouping by multiple columns you can call .reset_index() on the result to flatten the MultiIndex back into regular columns, which is often necessary before plotting or merging with another DataFrame.

Handling Missing Values

Real-world datasets almost always contain missing values — sensors drop packets, operators skip fields, instruments go offline. Ignoring missing data without understanding its extent and pattern can silently corrupt every downstream calculation. Pandas represents missing data as NaN (Not a Number) for numeric columns and as None or pd.NA in other contexts, and it provides a consistent set of methods for detecting, removing, and imputing these gaps.

The first step is always to count the missing values. df.isnull() returns a boolean DataFrame of the same shape as df, with True wherever a value is missing. Chaining .sum() collapses that boolean DataFrame column-by-column, yielding the count of missing entries in each column.

print(df.isnull().sum())
# temperature_C    1
# pressure_bar     0
# flow_rate_Ls     0
# dtype: int64

Dividing by len(df) converts counts to fractions, giving you the missing rate per column — a useful diagnostic when deciding whether a column is reliable enough to use.

print(df.isnull().sum() / len(df))

Once you understand where the gaps are, you have two main strategies: removal and imputation.

Removal with df.dropna() deletes every row that contains at least one NaN. This is the safest option when missing values are rare and random — the remaining data is complete and unbiased. However, if missingness is systematic (e.g., a sensor always fails at high temperatures), dropping those rows introduces bias.

# Drop every row with any missing value
df_clean = df.dropna()
print(df_clean.shape)   # (7, 3) — one row removed

You can restrict removal to specific columns with the subset parameter, and use thresh to keep rows that have at least a given number of non-NaN values — useful when a row missing only one of ten sensors is still valuable.

# Only drop rows where 'temperature_C' is missing
df_clean2 = df.dropna(subset=['temperature_C'])

Imputation with df.fillna(value) replaces NaN values with a specified constant or computed value instead of removing the row. Common strategies include:

• Constant fill: Replace with a domain-specific sentinel, e.g., 0 for a counter that never ran, or -999 as a flag for a downstream system.
• Mean or median fill: Replace with the column's average (suitable for approximately normal data) or median (more robust to outliers).
• Forward fill (method='ffill'): Propagate the last known value forward in time — natural for sensor time series where a brief dropout should carry the previous reading.
• Backward fill (method='bfill'): Fill from the next valid observation backwards.

# Fill with the column mean
df['temperature_C'] = df['temperature_C'].fillna(df['temperature_C'].mean())

# Or forward-fill for time series data
df['temperature_C'] = df['temperature_C'].fillna(method='ffill')

It is important to compute the fill value (e.g., the mean) on the training portion of your data when working in a machine-learning context, then apply the same value to validation and test sets. Computing the mean on the full dataset before splitting leads to data leakage.

Correlation and Relationships

Descriptive statistics for individual columns tell you about each variable in isolation. Correlation measures the linear relationship between pairs of variables — whether they tend to move together (positive correlation), in opposite directions (negative correlation), or independently (near-zero correlation). The Pearson correlation coefficient ranges from −1 (perfect inverse linear relationship) to +1 (perfect direct linear relationship), with 0 indicating no linear association.

df.corr() computes the pairwise Pearson correlation for every combination of numeric columns in the DataFrame and returns a symmetric matrix. The diagonal is always 1.0 (a variable is perfectly correlated with itself).

import numpy as np

np.random.seed(42)
n = 100
df4 = pd.DataFrame({
    'ambient_temp':  np.random.normal(25, 3, n),
    'power_draw_W':  np.random.normal(500, 50, n),
    'efficiency_pct': np.random.normal(85, 2, n)
})
# Introduce a real relationship: efficiency drops as ambient temp rises
df4['efficiency_pct'] -= 0.4 * df4['ambient_temp']

print(df4.corr())

Reading the matrix: the value in the row ambient_temp and column efficiency_pct reveals how strongly ambient temperature and efficiency are linearly related. A value near −0.9 would confirm a strong inverse relationship — as the environment heats up, efficiency falls.

When you only care about the correlation between two specific columns, you can call the method on one Series and pass the other as an argument:

r = df4['ambient_temp'].corr(df4['efficiency_pct'])
print(f"Pearson r = {r:.4f}")

You can also select a non-default correlation method by passing method='spearman' or method='kendall' to either df.corr() or Series.corr(). Spearman correlation measures monotonic (not necessarily linear) relationships and is more robust to outliers, making it a better choice when data is ordinal or skewed.

Correlation matrices are most informative when displayed as a heatmap, because the human eye can quickly spot clusters of high or low correlation that would be invisible in a table of numbers. The seaborn library provides seaborn.heatmap() for this purpose:

import matplotlib.pyplot as plt
import seaborn as sns

fig, ax = plt.subplots(figsize=(6, 5))
sns.heatmap(df4.corr(), annot=True, fmt='.2f', cmap='coolwarm',
            vmin=-1, vmax=1, ax=ax)
ax.set_title('Correlation Matrix — Reactor Data')
plt.tight_layout()
plt.show()

In the heatmap, colors near deep red (or blue, depending on the colormap) indicate strong positive (or negative) correlations, while pale or white cells indicate weak associations. The annot=True argument overlays the numeric values, combining the precision of a table with the readability of a color map.

A critical caveat: correlation does not imply causation. Two variables can be highly correlated because both are driven by a third (confounding) variable. Always interpret correlations alongside domain knowledge.

Plotting Directly from Pandas

Pandas integrates directly with matplotlib, allowing you to produce publication-quality plots with a single method call on a DataFrame or Series — without manually extracting arrays and passing them to plt.plot(). Under the hood, df.plot() calls matplotlib, so any plot it creates can be further customized with standard matplotlib commands.

The core interface is df.plot(kind=...), where kind specifies the chart type. The most common options are 'line' (default), 'scatter', 'bar', 'barh', 'hist', 'box', 'kde', and 'pie'.

A scatter plot between two columns is created by specifying both x and y and setting kind='scatter':

ax = df4.plot(x='ambient_temp', y='efficiency_pct', kind='scatter',
              alpha=0.6, color='steelblue',
              xlabel='Ambient Temperature (°C)',
              ylabel='Efficiency (%)',
              title='Efficiency vs Ambient Temperature')
plt.tight_layout()
plt.show()

The negative slope in the scatter plot visually confirms the negative correlation computed earlier. Adding alpha=0.6 makes overlapping points semi-transparent, revealing data density in crowded regions.

A histogram of a single column is produced by calling .plot(kind='hist') on a Series. The bins parameter controls how finely the range is divided — too few bins hide the shape of the distribution; too many bins show noise rather than structure. A common starting rule is to try the square root of the number of observations.

ax = df4['power_draw_W'].plot(kind='hist', bins=20, edgecolor='black',
                              color='coral',
                              title='Distribution of Power Draw')
ax.set_xlabel('Power Draw (W)')
ax.set_ylabel('Frequency')
plt.tight_layout()
plt.show()

One of the most practical features of pandas plotting is the ax parameter. By passing an existing matplotlib Axes object to df.plot(ax=ax), you embed the pandas-generated plot inside a larger subplot layout. This allows you to arrange multiple diagnostics — a time series, a histogram, a scatter plot — in a single figure, which is standard practice in engineering reports and dashboards.

fig, axes = plt.subplots(1, 3, figsize=(14, 4))

# Panel 1: histogram of ambient temperature
df4['ambient_temp'].plot(kind='hist', bins=15, ax=axes[0],
                         title='Ambient Temp Distribution',
                         color='skyblue', edgecolor='black')

# Panel 2: histogram of efficiency
df4['efficiency_pct'].plot(kind='hist', bins=15, ax=axes[1],
                           title='Efficiency Distribution',
                           color='salmon', edgecolor='black')

# Panel 3: scatter plot
df4.plot(x='ambient_temp', y='efficiency_pct', kind='scatter',
         ax=axes[2], alpha=0.5, color='purple',
         title='Efficiency vs Temperature')

for ax in axes:
    ax.set_xlabel(ax.get_xlabel() or '')  # tidy up labels

plt.tight_layout()
plt.show()

The key insight is that fig, axes = plt.subplots(1, 3) creates the figure and three Axes objects; each pandas .plot(ax=axes[i]) call fills one of those slots. Without the ax parameter, each .plot() call would open a new figure window, making it impossible to assemble a composite layout.

Together, these tools — summary statistics, grouped aggregation, missing-value management, correlation analysis, and direct plotting — form a complete workflow for the exploratory data analysis (EDA) phase of any data-driven project. Running them in sequence on a new dataset typically surfaces 80 % of the insights you need to ask the right follow-up questions and choose appropriate next steps, whether that means cleaning the data further, engineering new features, or selecting a statistical model.