Introduction to Pandas: Series and DataFrames

Pandas is one of the most widely used libraries in the Python data science ecosystem. Built on top of NumPy, it provides high-level, flexible data structures and a rich set of tools designed specifically for data manipulation, cleaning, analysis, and exploration. The name "Pandas" is derived from "Panel Data," a term from econometrics referring to multidimensional structured data sets. Whether you are working with financial time series, survey results, machine learning feature tables, or database exports, Pandas gives you a consistent and expressive interface for getting the job done.

To use Pandas, you import it with the conventional alias pd:

import pandas as pd
import numpy as np  # often used alongside pandas

Understanding Pandas begins with understanding its two core data structures: the Series and the DataFrame. Every other capability in the library — grouping, merging, pivoting, time-series resampling — is built on top of these two objects.

What is Pandas?

At its heart, Pandas is a library for working with labeled, heterogeneous, and often incomplete tabular data. Unlike a plain NumPy array, which requires all data to share the same type and uses only integer positions to reference values, Pandas attaches meaningful labels — called an index — to every row and every column. This labeling is what makes Pandas so powerful: you always know which observation or variable you are operating on, even after sorting, filtering, or merging.

One of the most practically important features of Pandas is its graceful handling of missing data. In the real world, data sets almost always have gaps — a sensor reading that failed, a survey question left blank, a database join that found no match. Pandas represents missing values using the special floating-point sentinel NaN (Not a Number, from NumPy) or the newer pd.NA for nullable integer and boolean types. Crucially, most Pandas operations simply skip over NaN values by default rather than propagating errors or silently producing wrong answers. You can then explicitly decide to fill those gaps with a meaningful substitute using .fillna(), or to remove incomplete rows or columns entirely using .dropna().

s = pd.Series([1.0, float('nan'), 3.0, float('nan'), 5.0])
print(s.sum())       # 9.0  — NaN is skipped
print(s.fillna(0))   # replaces NaN with 0
print(s.dropna())    # removes NaN entries

Pandas also integrates seamlessly with Matplotlib, Python's primary plotting library. Every Series and DataFrame exposes a .plot() method that internally calls Matplotlib, so you can produce line charts, bar charts, histograms, scatter plots, and more in a single line of code without manually extracting arrays and passing them to a plotting function. This tight integration makes exploratory data analysis (EDA) extremely fast.

import matplotlib.pyplot as plt

df = pd.DataFrame({
    'year': [2019, 2020, 2021, 2022, 2023],
    'revenue': [120, 135, 98, 160, 175]
})
df.plot(x='year', y='revenue', kind='line', marker='o')
plt.show()

Beyond analysis and plotting, Pandas is a first-class I/O library. It can read data from and write data to an enormous range of formats. The most commonly used functions include pd.read_csv() and df.to_csv() for comma-separated files, pd.read_excel() and df.to_excel() for spreadsheets, pd.read_json() for JSON documents, and pd.read_sql() or pd.read_sql_query() for pulling query results directly from a relational database via a SQLAlchemy connection. This means that whatever your data source, Pandas can likely consume it and hand you back a ready-to-use DataFrame in just one or two lines.

# Reading different file formats
df_csv   = pd.read_csv('sales_data.csv')
df_excel = pd.read_excel('report.xlsx', sheet_name='Q1')
df_json  = pd.read_json('records.json')

# Writing back out
df_csv.to_csv('cleaned_sales.csv', index=False)

The Pandas Series

A Series is a one-dimensional labeled array. You can think of it as a supercharged Python list or a single column from a spreadsheet — it holds a sequence of values, but each value is associated with a label from the index. A Series has two main components: its values (a NumPy array under the hood) and its index (the array of labels).

You create a Series with pd.Series(data, index=labels). The data argument can be a Python list, a NumPy array, a scalar value, or a dictionary. When you supply a dictionary, the dictionary keys automatically become the index labels.

# From a list with an explicit index
temperatures = pd.Series([22.1, 18.4, 25.0, 30.2], index=['Mon', 'Tue', 'Wed', 'Thu'])
print(temperatures)
# Mon    22.1
# Tue    18.4
# Wed    25.0
# Thu    30.2
# dtype: float64

# From a dictionary — keys become the index
population = pd.Series({'Tokyo': 13_960_000, 'Delhi': 30_290_000, 'Lagos': 14_860_000})
print(population)

Accessing elements in a Series is done with two primary indexers. .loc[] uses label-based access — you refer to elements by their index label, exactly as they appear. .iloc[] uses integer position-based access — you refer to elements by their numerical position (0, 1, 2, …), regardless of what the labels are. This distinction becomes critically important when your index contains integers, because direct bracket access s[0] can behave ambiguously. It is always safer and clearer to use .loc[] or .iloc[] explicitly.

print(temperatures.loc['Wed'])    # 25.0  — access by label
print(temperatures.iloc[2])       # 25.0  — access by position (same element here)

# Slicing with loc (inclusive on both ends)
print(temperatures.loc['Mon':'Wed'])

# Slicing with iloc (exclusive on the right end, like Python lists)
print(temperatures.iloc[0:3])

One of the most elegant behaviors of a Pandas Series is index alignment during arithmetic. When you add, subtract, multiply, or divide two Series, Pandas automatically aligns them by their index labels before performing the operation. If a label exists in one Series but not the other, the result for that label is NaN. This behavior is a safeguard against accidentally combining data that belongs to different observations.

sales_q1 = pd.Series({'Alice': 340, 'Bob': 280, 'Carol': 410})
sales_q2 = pd.Series({'Alice': 390, 'Carol': 365, 'Dave': 450})

total = sales_q1 + sales_q2
print(total)
# Alice    730.0
# Bob        NaN   <-- Bob has no Q2 data
# Carol    775.0
# Dave       NaN   <-- Dave has no Q1 data
# dtype: float64

The Pandas DataFrame

A DataFrame is a two-dimensional labeled data structure — a table with named columns and a shared row index. You can think of it as a dictionary of Series objects that all share the same index, or as a spreadsheet or SQL table brought into Python. Each column in a DataFrame is itself a Series, and they can each hold a different data type (integers in one column, floats in another, strings in a third).

The most common way to create a DataFrame is from a dictionary of lists, where each key is a column name and each list contains the values for that column:

data = {
    'name':   ['Alice', 'Bob', 'Carol', 'Dave'],
    'age':    [29, 34, 22, 45],
    'salary': [72000.0, 85000.0, 61000.0, 98000.0],
    'dept':   ['Engineering', 'Marketing', 'Engineering', 'Finance']
}
df = pd.DataFrame(data)
print(df)
#     name  age   salary         dept
# 0  Alice   29  72000.0  Engineering
# 1    Bob   34  85000.0    Marketing
# 2  Carol   22  61000.0  Engineering
# 3   Dave   45  98000.0      Finance

You can also create a DataFrame from a 2D NumPy array, in which case you typically supply column names explicitly. Or you can pass another DataFrame — for example, a subset of columns from an existing one — as the source. Pandas also accepts lists of dictionaries, where each dictionary represents a single row.

# From a 2D NumPy array
arr = np.array([[1, 0.5, True], [2, 1.5, False], [3, 2.5, True]])
df_np = pd.DataFrame(arr, columns=['id', 'score', 'active'])

# From a list of dicts (each dict is one row)
records = [
    {'city': 'Paris',  'country': 'France',  'pop': 2161000},
    {'city': 'Berlin', 'country': 'Germany', 'pop': 3645000},
]
df_cities = pd.DataFrame(records)

Once you have a DataFrame, two attributes give you immediate structural information. df.shape returns a tuple (rows, columns) and is the fastest way to gauge the size of a dataset — knowing you have 1,200,000 rows and 47 columns before you begin analysis is essential for choosing the right approach.

print(df.shape)   # (4, 4) for our employees example
rows, cols = df.shape
print(f"Dataset has {rows} rows and {cols} columns.")

df.dtypes returns a Series mapping each column name to its Pandas data type. This is not merely informational — data types determine which operations are valid. You cannot compute the mean of a column of strings; you cannot do string slicing on an integer column. Common Pandas dtypes include int64, float64, object (typically used for strings), bool, datetime64[ns], and category. Seeing an expected numeric column listed as object is a classic sign that it contains mixed types or hidden non-numeric characters that need cleaning.

print(df.dtypes)
# name       object
# age         int64
# salary    float64
# dept       object
# dtype: object

Inspecting a DataFrame

Before diving into analysis, it is essential to understand the shape, completeness, and statistical character of your data. Pandas provides several methods purpose-built for this initial inspection phase.

df.head(n) returns the first n rows of the DataFrame (default n=5). df.tail(n) returns the last n rows. These are your first look at the raw data — column names, value formats, and any obvious anomalies. You use head() constantly: after loading a file, after a merge, after a transformation, to confirm things look as expected without printing the entire dataset.

large_df = pd.read_csv('transactions.csv')
print(large_df.head())      # first 5 rows
print(large_df.head(10))    # first 10 rows
print(large_df.tail(3))     # last 3 rows

df.info() prints a concise technical summary of the DataFrame to the console. It tells you the total number of rows (the RangeIndex), lists every column with its non-null count and dtype, and reports the total memory usage. The non-null count is particularly valuable: if a column has 10,000 rows but only 8,432 non-null values, you immediately know there are 1,568 missing entries that will need to be addressed before further analysis.

df.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 4 entries, 0 to 3
# Data columns (total 4 columns):
#  #   Column  Non-Null Count  Dtype
# ---  ------  --------------  -----
#  0   name    4 non-null      object
#  1   age     4 non-null      int64
#  2   salary  4 non-null      float64
#  3   dept    4 non-null      object
# dtypes: float64(1), int64(1), object(2)
# memory usage: 256.0+ bytes

df.describe() generates a table of summary statistics for every numeric column in the DataFrame. By default it computes the count of non-null values, mean, standard deviation, minimum, the three quartiles (25th, 50th, and 75th percentile), and the maximum. This single call gives you a statistical fingerprint of your numeric data and is invaluable for spotting outliers (a maximum salary of 9,999,999 when the mean is 75,000 suggests an error), skewed distributions (mean much higher than median), or scale differences between features (relevant before machine learning).

print(df.describe())
#              age         salary
# count   4.000000       4.000000
# mean   32.500000   79000.000000
# std     9.707253   15491.933385
# min    22.000000   61000.000000
# 25%    27.250000   69750.000000
# 50%    31.500000   78500.000000
# 75%    36.750000   87750.000000
# max    45.000000   98000.000000

You can also pass include='all' to describe() to include statistics for non-numeric (object) columns as well. For string columns this shows count, the number of unique values, the most frequent value (top), and its frequency (freq) — helpful for understanding categorical columns at a glance.

print(df.describe(include='all'))

Together, head(), info(), and describe() form the standard opening trio of any Pandas workflow. They answer the three most immediate questions about a new dataset: What does the data look like?, How complete and correctly typed is it?, and What are its statistical properties? Developing the habit of running these three commands every time you load a new dataset will save you from countless downstream errors and surprises.

Putting it all together, a typical Pandas workflow begins by importing the library, loading data from a file, and then immediately inspecting it to understand its structure before any manipulation begins:

import pandas as pd

# Load data
df = pd.read_csv('employee_records.csv')

# Structural inspection
print(df.shape)
print(df.dtypes)

# Content preview
print(df.head())

# Completeness and type summary
df.info()

# Statistical summary
print(df.describe())

# Handle missing values
df['salary'] = df['salary'].fillna(df['salary'].median())
df = df.dropna(subset=['name'])

# Quick visualization
df['salary'].plot(kind='hist', bins=20, title='Salary Distribution')

This pattern — load, inspect, clean, analyze, visualize — is the heartbeat of data work with Pandas, and every step relies on the Series and DataFrame structures introduced here.