Controlled User Testing and Data Collection — Topics & Learning Outcomes

Module Topics

Foundations of Experimental Design for Usability Testing

• What Is Experimental Design in Usability Testing? — Experimental design in usability testing is the systematic planning of a study to ensure results are valid, reliable, and actionable. It defines what will be tested, how it will be measured, and how variables will be controlled.
• Independent and Dependent Variables — In a usability study, the independent variable is the condition being manipulated (such as interface version), while the dependent variables are the outcomes being measured (such as task completion time or error rate).
• Study Validity: Internal and External — Internal validity refers to the degree to which the study accurately measures what it intends to, while external validity refers to how well the findings generalize to real-world contexts.
• Choosing a Study Type: Formative vs. Summative — Formative usability tests are conducted during development to identify and fix problems, while summative tests are conducted after development to evaluate the overall quality of a finished product.
• Ethical Considerations in Usability Studies — Conducting usability testing involves ethical responsibilities including informed consent, participant confidentiality, the right to withdraw, and data handling practices.

Participant Selection and Recruitment

• Defining the Target User Population — The target user population is the group of people for whom the product is designed, and all participant selection decisions should reflect meaningful characteristics of this group.
• Inclusion and Exclusion Criteria — Inclusion criteria specify the characteristics a participant must have to qualify for the study, while exclusion criteria identify characteristics that would disqualify a candidate.
• Screening and Recruitment Methods — Screening involves vetting potential participants against selection criteria, typically using a screener survey or questionnaire administered before the study begins.
• Sample Size Considerations — Determining how many participants to recruit depends on the study type, the diversity of the user population, and the depth of insights needed.
• Scheduling and Logistics — Participant scheduling involves coordinating session times, confirming locations or remote access links, and communicating all logistical details to participants in advance.

Task Scenario Scripting

• What Makes a Good Task Scenario? — A good task scenario is realistic, goal-oriented, and specific enough to be measurable without revealing how the user should complete the task.
• Writing Scenario Context and Prompts — Scenario prompts include a brief narrative context that situates the participant in a realistic use case, followed by a specific action prompt that tells them what goal to accomplish.
• Task Measurability and Success Criteria — Before testing begins, teams must define what constitutes successful task completion and how partial or failed completion will be recorded.
• Piloting and Refining Task Scenarios — Pilot testing involves running one or more trial sessions with non-study participants to identify ambiguities, timing issues, or errors in the task scenarios before live testing.
• Sequencing and the Test Script — The test script is the facilitator's master document that sequences all tasks, instructions, and transitions in the exact order they will occur during each session.

Think-Aloud Protocol

• What Is the Think-Aloud Protocol? — The think-aloud protocol asks participants to verbalize their thoughts, decisions, and reactions continuously as they interact with the product, giving evaluators direct access to the user's mental model.
• Concurrent vs. Retrospective Think-Aloud — Concurrent think-aloud requires participants to verbalize during the task, while retrospective think-aloud asks them to narrate their thoughts after completing the task, often while reviewing a session recording.
• Training Participants to Think Aloud — Most participants are not naturally accustomed to verbalizing their thoughts while working, so a brief training exercise at the start of the session is essential to establish the behavior.
• Facilitator Techniques During Think-Aloud Sessions — During the session, the facilitator must prompt participants who go silent without interfering with their problem-solving or revealing information about the interface.
• Recording and Capturing Think-Aloud Data — Think-aloud sessions should be audio or video recorded so that verbal data can be accurately transcribed and analyzed after the session.

Quantitative Data Collection

• Core Usability Metrics — Quantitative usability metrics provide objective, numerical evidence of how well users can interact with a product, enabling comparison across tasks, sessions, and design iterations.
• Standardized Usability Questionnaires — Post-task and post-session questionnaires provide standardized quantitative measures of perceived usability and user satisfaction that complement behavioral observation data.
• Designing Data Collection Instruments — Data collection instruments such as observation logs, timing sheets, and error tally forms must be designed before testing begins to ensure all relevant data is captured consistently across sessions.
• Timing and Measurement Consistency — For timing data to be meaningful and comparable across participants, start and end points for each task must be precisely defined and consistently applied by the team.
• Aggregating Quantitative Data — After all sessions are complete, quantitative data from individual participants must be aggregated into summary statistics that represent the overall study findings.

Qualitative Data Collection

• Types of Qualitative Data in Usability Testing — Qualitative data in usability testing includes verbal utterances, observable behaviors, emotional reactions, and participant-reported preferences that cannot be meaningfully reduced to numbers.
• Structured Observation and Note-Taking — Effective note-taking during usability sessions requires a structured approach that captures specific, observable behaviors rather than interpretations or evaluations.
• Post-Test Interviews and Debrief — A brief post-test interview or debrief at the end of each session gives participants the opportunity to reflect on their overall experience and raise issues that did not emerge during task performance.
• Affinity Diagramming for Qualitative Synthesis — Affinity diagramming is a team-based method for organizing and synthesizing large volumes of qualitative observations into grouped themes and patterns.
• Documenting Quotes and Critical Incidents — Direct participant quotes and critical incidents — moments of significant struggle, surprise, or delight — are among the most compelling qualitative evidence in a usability report.

Conducting Live User Testing Sessions

• Team Roles During a Testing Session — Each usability testing session should have clearly assigned team roles — typically a facilitator, a note-taker, and one or more observers — to ensure all data is captured without overwhelming any one person.
• Session Environment Setup — The testing environment must be prepared before each session to ensure the product is in the correct starting state, recording tools are functioning, and the space is free from distractions.
• Opening the Session — The session opening establishes rapport with the participant, confirms consent, explains the think-aloud protocol, and sets expectations for how the session will proceed.
• Managing In-Session Challenges — Facilitators must be prepared to handle common session challenges such as participant silence, off-task conversation, technology failures, and requests for guidance.
• Closing the Session and Post-Session Debrief — Closing the session involves thanking the participant, administering any final questionnaires, debriefing on the study's purpose, and distributing any participant incentives.

Compiling Structured Results for Analysis

• Organizing Raw Data After Testing — Before analysis can begin, all raw data from every session must be compiled, labeled, and organized in a consistent format that allows comparison across participants and tasks.
• Identifying Usability Problems and Patterns — Analysis involves reviewing the compiled data to identify recurring usability problems, patterns of behavior, and areas of the product that consistently caused difficulty across participants.
• Structuring the Results Report — The results report is the primary deliverable of the usability study and should present findings in a clear, logical structure that moves from study overview to findings to recommendations.
• Presenting Quantitative Findings — Quantitative findings should be presented using tables, charts, and summary statistics that make numerical patterns immediately visible and interpretable.
• Presenting Qualitative Findings and Recommendations — Qualitative findings are most effectively communicated through a combination of themed summaries, direct participant quotes, and specific critical incident descriptions.

Student Learning Outcomes

By the end of this module, students will be able to:

Course Outcomes (reference)