Loading and Inspecting Engineering Data

Engineering data analysis begins long before any model is trained or any chart is drawn. The very first step — loading raw data into a Python environment and understanding its structure — is arguably the most consequential. Mistakes made here, such as silently misinterpreted data types, undetected duplicate rows, or a misaligned time index, propagate through every downstream calculation. This topic covers the practical mechanics of reading data into pandas DataFrames, inspecting those DataFrames thoroughly, handling time-series indices correctly, filtering and selecting data with precision, and combining data from multiple sources into a unified, trustworthy dataset.

Reading CSV Files with pandas

The comma-separated values (CSV) format is the most common file type encountered in engineering data workflows. Sensor logs, SCADA exports, lab measurements, and test-bench outputs are routinely saved as CSVs. The pandas function pd.read_csv() is the primary tool for importing this data, and understanding its parameters deeply prevents many subtle bugs.

At its simplest, pd.read_csv() accepts a file path as its first argument:

import pandas as pd

df = pd.read_csv('sensor_log.csv')

The first argument is extremely flexible. It can be a relative or absolute file path string, a URL pointing to a remote file, or even a Python file-like object opened with open(). This means you can read data directly from a web server or from an in-memory buffer without saving it to disk first:

# Reading from a URL
df = pd.read_csv('https://example.com/data/pressure_readings.csv')

# Reading from a file-like object
with open('sensor_log.csv', 'r') as f:
    df = pd.read_csv(f)

One of the most important parameters to understand is dtype. By default, pandas infers the data type of each column by scanning its contents. While convenient, this inference can cause silent errors. For example, a column of equipment IDs like 001, 002, 003 will be read as integers, stripping leading zeros. A voltage column that contains a single erroneous entry like 'N/A' will cause the entire column to be inferred as object (string) rather than float64, causing arithmetic operations to fail or silently produce wrong results. Specifying dtype explicitly removes this ambiguity:

df = pd.read_csv('sensor_log.csv', dtype={
    'equipment_id': str,
    'voltage': float,
    'current': float,
    'status_code': str
})

When you specify types explicitly, pandas will raise an error if a value cannot be converted, alerting you to data quality issues immediately rather than letting them slip through undetected.

The parse_dates parameter instructs pandas to parse specified columns as datetime objects during loading rather than leaving them as plain strings. This avoids a common two-step workflow of loading and then separately converting:

# Parse the 'timestamp' column as a datetime during loading
df = pd.read_csv('sensor_log.csv', parse_dates=['timestamp'])

# Combine separate date and time columns into one datetime column
df = pd.read_csv('sensor_log.csv', parse_dates=[['date', 'time']])

When two column names are passed in a nested list, pandas concatenates them with a space separator and parses the result, creating a single combined datetime column named date_time.

For very large files — gigabyte-scale logs from industrial systems — loading the entire file into memory at once may be impractical. The chunksize parameter enables incremental reading by returning an iterable of DataFrames, each containing the specified number of rows:

chunk_iter = pd.read_csv('large_sensor_log.csv', chunksize=10000)

results = []
for chunk in chunk_iter:
    # Process each chunk independently, e.g., compute the mean pressure
    results.append(chunk['pressure'].mean())

overall_mean = sum(results) / len(results)

This pattern lets you aggregate statistics, filter rows, or transform data without ever holding the entire dataset in RAM. It is a foundational technique for production engineering data pipelines.

Initial Data Inspection

Once data is loaded, the immediate priority is to understand its shape, types, completeness, and statistical properties. Skipping this step is a common source of downstream errors. pandas provides a concise set of inspection tools that together give a complete picture of a DataFrame's health.

df.shape returns a tuple of (rows, columns). This is always the first thing to check because it confirms that the file loaded completely and that the expected number of columns is present:

print(df.shape)
# (52560, 8)  →  52560 rows, 8 columns

If you expected a year's worth of hourly readings (8760 rows) but see 52560, you immediately know something about the data structure warrants investigation — perhaps readings are at 10-minute intervals.

df.dtypes shows the inferred or assigned data type of each column. This is critical for catching type mismatches before they cause errors in calculations:

print(df.dtypes)
# timestamp     datetime64[ns]
# voltage              float64
# current              float64
# status_code           object
# equipment_id          object

If voltage appears as object instead of float64, that is a red flag indicating non-numeric values exist in that column — perhaps a header was repeated mid-file, or a sensor dropout was logged as the string 'ERR'.

df.head() and df.tail() display the first and last five rows by default, though you can pass an integer argument to change the count. Together they let you verify that the file loaded with the correct column headers, that data begins and ends where expected, and that values are in a plausible range:

print(df.head(3))
#            timestamp  voltage  current status_code equipment_id
# 0 2024-01-01 00:00:00   230.1     15.3          OK         E001
# 1 2024-01-01 00:10:00   229.8     15.1          OK         E001
# 2 2024-01-01 00:20:00   231.0     15.4          OK         E001

df.info() provides a compact, all-in-one summary that shows column names, the count of non-null entries in each column, and data types. It is particularly useful for identifying missing data at a glance:

df.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 52560 entries, 0 to 52559
# Data columns (total 5 columns):
#  #   Column        Non-Null Count  Dtype
# ---  ------        --------------  -----
#  0   timestamp     52560 non-null  datetime64[ns]
#  1   voltage       52543 non-null  float64
#  2   current       52560 non-null  float64
#  3   status_code   52560 non-null  object
#  4   equipment_id  52560 non-null  object

Here, voltage has only 52543 non-null entries out of 52560, meaning 17 values are missing. This finding would immediately trigger an investigation into the cause and a decision about how to handle the gaps.

df.describe() computes summary statistics for all numeric columns: count, mean, standard deviation, minimum, 25th percentile, median (50th percentile), 75th percentile, and maximum. For engineering data, these statistics provide immediate sanity checks:

print(df.describe())
#           voltage       current
# count  52543.000    52560.000
# mean     230.021       15.287
# std        1.453        0.812
# min      198.300       -0.500
# 25%      229.200       14.800
# 50%      230.100       15.300
# 75%      230.900       15.800
# max      245.000       18.900

The minimum voltage of 198.3 V is significantly below the 25th percentile of 229.2 V, which immediately suggests outliers or transient fault events worth investigating. A negative current value of -0.5 A might indicate reverse power flow or a sensor offset error. These are findings that df.describe() surfaces in seconds.

To include non-numeric columns in the description (for categorical inspection), pass include='all':

print(df.describe(include='all'))

This adds statistics like the number of unique values and the most frequent value (mode) for string columns, which is useful for auditing categorical columns like status_code.

Working with Time-Series Indices

Engineering data is almost always time-stamped. Setting up the time index correctly unlocks a wide range of powerful pandas functionality for slicing, resampling, and feature extraction. The foundational function here is pd.to_datetime(), which converts string or numeric representations of dates and times into pandas Timestamp objects:

# Converting a string column to datetime
df['timestamp'] = pd.to_datetime(df['timestamp'])

# Converting Unix epoch timestamps (in seconds) to datetime
df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')

# Specifying a format string for non-standard formats
df['timestamp'] = pd.to_datetime(df['timestamp'], format='%d/%m/%Y %H:%M:%S')

Specifying a format string is both faster (pandas skips format inference) and safer (ambiguous dates like 04/05/2024 are interpreted correctly as April 5th rather than May 4th when you write %m/%d/%Y).

Once the column is properly typed, setting it as the DataFrame index with df.set_index() transforms it into a DatetimeIndex, enabling time-based selection and resampling:

df = df.set_index('timestamp')
print(df.index)
# DatetimeIndex(['2024-01-01 00:00:00', '2024-01-01 00:10:00', ...],
#               dtype='datetime64[ns]', name='timestamp', freq=None)

With a DatetimeIndex, you can slice rows using human-readable string labels. This is one of the most ergonomic features pandas offers for time-series work:

# Select all rows in January 2024
january_data = df['2024-01']

# Select all rows on a specific date
day_data = df['2024-03-15']

# Select a specific time range
range_data = df['2024-03-15 08:00':'2024-03-15 17:00']

pandas interprets partial string labels intelligently, so '2024-01' correctly selects every row whose index falls within January 2024, regardless of time-of-day. This is far more readable than constructing boolean masks with explicit start and end timestamps.

The DatetimeIndex also exposes a rich set of accessor attributes through the .dt accessor (for Series) or directly on the index, enabling rapid feature extraction without loops:

# Extract hour-of-day, day-of-week, and month as new columns
df['hour'] = df.index.hour
df['day_of_week'] = df.index.dayofweek   # Monday=0, Sunday=6
df['month'] = df.index.month
df['is_weekend'] = df.index.dayofweek >= 5

These derived features are invaluable for engineering analysis. For example, separating weekday from weekend behavior in power consumption data, or identifying night-shift vs. day-shift operation patterns, requires exactly this kind of temporal decomposition. Other commonly useful attributes include .year, .minute, .second, .quarter, and .is_month_end.

Selecting and Filtering Data

Once data is loaded and its index is configured, you frequently need to work with subsets — specific columns, specific rows, or rows that meet particular conditions. pandas provides several complementary mechanisms for this.

Column selection uses bracket notation. Passing a single column name as a string returns a Series, which is a one-dimensional labeled array. Passing a list of column names returns a DataFrame:

# Returns a Series
voltage_series = df['voltage']

# Returns a DataFrame (note the double brackets)
subset_df = df[['voltage', 'current']]

This distinction matters. A Series has a single data type and behaves like a 1D array. A DataFrame is a 2D table. Many functions behave differently depending on which you pass.

loc[] selects data by label. For a DataFrame with a DatetimeIndex, this means selecting rows by timestamp label and columns by name. The syntax is df.loc[row_selector, column_selector]:

# Select a specific date range and specific columns
subset = df.loc['2024-02-01':'2024-02-28', ['voltage', 'current']]

# Select a single row by exact index label
row = df.loc['2024-03-15 12:00:00']

iloc[] selects data by integer position, ignoring labels entirely. This is useful when you want the first N rows, or every other row, regardless of what the index labels are:

# First 100 rows, all columns
first_100 = df.iloc[:100]

# Rows 50 through 99, first three columns
subset = df.iloc[50:100, :3]

# Last row
last_row = df.iloc[-1]

Boolean masks are arrays of True/False values produced by comparison operations on a column. Passing a boolean mask to df[...] returns only the rows where the mask is True:

# Rows where voltage dropped below 220 V (potential undervoltage events)
low_voltage = df[df['voltage'] < 220]

# Rows where both voltage is high AND current is high
overload = df[(df['voltage'] > 240) & (df['current'] > 18)]

# Rows where status code is a fault condition
faults = df[df['status_code'].isin(['FAULT', 'ERR', 'WARN'])]

When combining multiple conditions, each must be wrapped in parentheses, and you use & (and), | (or), and ~ (not) rather than Python's and, or, not keywords, which do not work element-wise on arrays.

The query() method offers a more readable string-based syntax for filtering, particularly useful when column names are clear and conditions are complex:

# Equivalent to df[(df['voltage'] < 220) & (df['current'] > 15)]
low_v_high_i = df.query('voltage < 220 and current > 15')

# Reference external Python variables using the @ prefix
threshold = 220
low_voltage = df.query('voltage < @threshold')

query() is generally more readable and less prone to parenthesis errors for multi-condition filters. However, it has limitations with column names containing spaces or special characters.

Importing Data from Multiple Sources

Real engineering projects rarely have all relevant data in a single file. You might have temperature readings from one sensor system, pressure readings from another, and maintenance event logs from a CMMS (Computerized Maintenance Management System) — each in separate CSVs. Combining them correctly is essential.

pd.concat() stacks DataFrames together. Setting axis=0 (the default) stacks them vertically, row by row — useful when the same measurements continue across multiple files (e.g., monthly exports):

jan_df = pd.read_csv('readings_jan.csv', parse_dates=['timestamp'], index_col='timestamp')
feb_df = pd.read_csv('readings_feb.csv', parse_dates=['timestamp'], index_col='timestamp')
mar_df = pd.read_csv('readings_mar.csv', parse_dates=['timestamp'], index_col='timestamp')

combined = pd.concat([jan_df, feb_df, mar_df])
combined = combined.sort_index()  # Ensure chronological order after concatenation

Setting axis=1 stacks DataFrames horizontally, column by column — useful when combining different measurement channels that share the same index:

voltage_df = pd.read_csv('voltage_readings.csv', index_col='timestamp', parse_dates=True)
current_df = pd.read_csv('current_readings.csv', index_col='timestamp', parse_dates=True)

combined = pd.concat([voltage_df, current_df], axis=1)

pd.merge() performs relational-style joins, equivalent to SQL JOIN operations. It aligns rows based on matching values in one or more key columns rather than on the index. This is appropriate when combining datasets that share a common identifier but may not be time-aligned:

readings_df = pd.read_csv('sensor_readings.csv')
metadata_df = pd.read_csv('equipment_metadata.csv')

# Inner join: only rows where equipment_id exists in both DataFrames
merged = pd.merge(readings_df, metadata_df, on='equipment_id', how='inner')

# Left join: keep all rows from readings_df, match metadata where available
merged = pd.merge(readings_df, metadata_df, on='equipment_id', how='left')

The how parameter controls join type: 'inner' keeps only matching rows, 'left' keeps all rows from the left DataFrame, 'right' keeps all from the right, and 'outer' keeps all rows from both, filling gaps with NaN.

The join() method is a convenient shorthand for merging on indices rather than columns. It is particularly well-suited to combining time-aligned sensor data where each DataFrame's index is the shared timestamp:

temp_df = pd.read_csv('temperature.csv', index_col='timestamp', parse_dates=True)
pressure_df = pd.read_csv('pressure.csv', index_col='timestamp', parse_dates=True)

# Combine on the shared timestamp index
combined = temp_df.join(pressure_df, how='outer')

Using how='outer' here is important if the two sensors do not record at exactly the same timestamps — it preserves all timestamps from both sources, with NaN where one sensor has no reading at a given moment.

A critical issue that arises after any merge or concatenation is duplicate rows. These can occur when the same event appears in multiple source files, or when a join creates Cartesian products due to non-unique keys. pandas provides duplicated() to detect them and drop_duplicates() to resolve them:

# Check how many duplicate rows exist
num_duplicates = combined.duplicated().sum()
print(f"Duplicate rows: {num_duplicates}")

# Inspect the duplicate rows
print(combined[combined.duplicated(keep=False)])

# Remove duplicates, keeping the first occurrence
combined = combined.drop_duplicates()

# Remove duplicates based only on specific columns
combined = combined.drop_duplicates(subset=['timestamp', 'equipment_id'])

It is important to distinguish between exact duplicates (every column matches) and key duplicates (only the join key matches but other values differ, which may indicate conflicting records rather than simple repetition). The subset parameter helps make this distinction explicit.

Taken together, these techniques — reading with explicit type control, inspecting rigorously, indexing by time, filtering with precision, and combining sources cleanly — form the essential foundation of any robust engineering data analysis workflow. Every subsequent step, from visualization to predictive modeling, depends on the data being correctly loaded and understood at this stage.