Aggregating Findings Across Evaluators

When multiple evaluators independently assess an interface — as is standard practice in heuristic evaluation and expert review — each person produces their own separate list of usability problems. No single evaluator, regardless of skill or experience, will catch every problem present in a complex interface. Research consistently shows that even expert evaluators miss a substantial portion of issues when working alone. This is not a flaw in the method; it is simply a consequence of how human attention and pattern recognition work. Different evaluators bring different mental models, different prior experiences, and different sensitivities to different kinds of problems. One evaluator might be highly attuned to navigation inconsistencies while another is more likely to notice error-recovery failures or unclear feedback. Aggregation is the process that harnesses this diversity productively, combining the separate findings of every evaluator into a single, comprehensive master list that none of them could have produced on their own.

The aggregated findings list is not merely a procedural artifact — it is the primary deliverable of the entire evaluation. Design teams, product owners, and developers will use it to prioritize fixes, allocate development effort, and make trade-off decisions about which problems to address first. A well-constructed aggregated list gives stakeholders the clearest possible picture of the interface's usability landscape. A poorly constructed one — with duplicates, contradictory severity ratings, or vaguely merged findings — obscures that landscape and can lead to misallocated effort or overlooked critical issues. Understanding how to aggregate findings rigorously is therefore just as important as knowing how to identify problems in the first place.

Why Aggregation Is Necessary

The fundamental reason aggregation is necessary is that individual evaluators each find only a subset of the total usability problems present in an interface. This is a well-documented phenomenon. In landmark studies of heuristic evaluation, individual evaluators typically identified between 20% and 51% of known usability problems, with the average closer to one-third. The implication is stark: if you rely on a single evaluator's findings, you may be missing two-thirds of the problems. As more evaluators are added, problem coverage increases — but with diminishing returns. A team of three to five evaluators typically achieves a good balance between coverage and practical cost.

Aggregation maximizes problem coverage by ensuring that any issue identified by any evaluator is included in the final deliverable. Even a problem found by only one evaluator deserves consideration. That evaluator may have a domain expertise or perspective that the others lacked. Dismissing a finding simply because only one person noticed it would undermine the purpose of running multiple evaluators in the first place.

The aggregated list is also more credible to stakeholders than any individual report. When a problem appears on three out of four evaluators' lists, that consensus is powerful evidence that the problem is real and significant. When a problem appears on only one list, that fact is also meaningful information — it may indicate a subtler issue, or one that affects only certain user populations, and it should be documented accordingly. The breadth of discovery (how many evaluators identified a given problem) becomes part of the finding's profile and informs how confidently teams can prioritize it.

The Aggregation Meeting

The aggregation process typically takes place in a structured meeting or working session that brings all evaluators together after they have completed their independent assessments. The independence of the individual evaluation phase is critically important: evaluators must complete and document their own findings before seeing anyone else's list. If evaluators review each other's work prematurely, the principle of independent assessment is violated. Findings can become anchored to someone else's framing, and evaluators may unconsciously suppress observations that seem to conflict with a colleague's, or amplify ones that seem to be validated by the group.

The meeting proceeds in a deliberate sequence. First, each evaluator presents their complete findings list without editing or filtering. This is not a time for the group to debate each issue as it is mentioned — the goal of this first phase is to ensure that every finding is on the table. A facilitator (often the evaluation team lead or a UX researcher who did not act as an evaluator) typically manages this process, recording all findings in a shared document or spreadsheet visible to the whole group.

Once all individual lists have been presented, the group moves to the deduplication and merging phase. Findings that appear to describe the same problem are flagged and reviewed together. The group examines whether these findings truly share the same root cause and recommended fix, or whether they are distinct problems that merely resemble each other on the surface. This is nuanced work that requires careful attention, and it is described further in the next section.

Where evaluators have assigned different severity ratings to what is determined to be the same problem, a negotiation takes place. The output of the entire meeting is a single master list: a consolidated record of usability problems with agreed-upon descriptions, consolidated evidence, and settled severity ratings. Each entry also records which evaluators originally identified it.

A well-organized aggregation meeting benefits from some logistical groundwork. Using a shared spreadsheet or digital whiteboard tool before the meeting — where each evaluator enters their findings in a consistent format — can dramatically speed up the process. A standard format might include columns for a problem description, the affected screen or task, the violated heuristic or principle, supporting evidence or quotes, and a severity rating. When all findings arrive in a consistent structure, comparing and merging them becomes much more manageable.

Deduplication and Problem Merging

Deduplication is one of the most intellectually demanding parts of the aggregation process because it requires distinguishing between problems that look similar and problems that actually are the same. Evaluators working independently will often describe the same interface failure in very different language, from very different angles, or in the context of different tasks. These surface differences can make it hard to recognize that two findings are describing the same underlying problem. Conversely, two findings that use nearly identical language may actually be describing different problems that share a common symptom but have different causes and require different fixes.

The correct criterion for merging is not linguistic similarity — it is shared root cause and shared recommended fix. If two findings point to the same design flaw that would be resolved by the same design change, they should be merged. If resolving one finding would leave the other unaddressed, they are distinct problems and must remain separate entries.

Consider an example. Suppose Evaluator A notes: "On the account settings page, the Save button is positioned at the top of the form, which is not where users typically expect to find action buttons." Evaluator B notes: "On the checkout confirmation screen, the Confirm Order button is at the top of the page, opposite to convention." These two findings describe the same design pattern — unconventionally placed action buttons — but they occur on different screens. They may share a root cause (a system-wide layout decision) and the same recommended fix (move primary action buttons to standard positions). If that is the case, merging them into a single finding that references both screens makes sense and correctly signals that this is a systemic problem, not a one-off. A systemic finding often warrants a higher severity rating than an isolated one.

Now consider a different scenario. Evaluator A notes: "Users cannot easily find the account settings because the link is buried in a dropdown under a non-descriptive icon." Evaluator B notes: "The account settings page has no clear navigation path back to the main dashboard." Both findings involve account settings navigation, and it might be tempting to merge them. But the first is a findability problem — users cannot get to account settings — while the second is a wayfinding/return navigation problem — users cannot get back from account settings. These require different fixes and should remain separate findings.

The aggregated list should also document which evaluators identified each problem. This serves two purposes. First, it provides a measure of the finding's robustness: a problem found by four out of five evaluators has a different confidence level than one found by only one. Second, it creates a useful audit trail. If a stakeholder later questions whether a finding is real, being able to say "three independent evaluators identified this issue from different task scenarios" is far more persuasive than "one person noticed this."

Negotiating Severity Across Evaluators

Even when two evaluators agree that a problem exists, they may disagree significantly on how severe it is. One evaluator might rate a navigation failure as a catastrophic problem (severity 4 on a 0–4 scale) while another rates it as a moderate problem (severity 2). These disagreements are normal and expected. Different evaluators have different domain backgrounds, different assumptions about the target user population, and different interpretations of what constitutes a complete blocking failure versus a significant inconvenience. The aggregation process must resolve these disagreements in a principled way.

The negotiation should always begin with each evaluator explaining the reasoning behind their rating before any bargaining begins. This is crucial. An evaluator who rated the problem at severity 4 may have done so because they observed that the problem occurs on the only path to a critical feature — there is no workaround. An evaluator who rated the same problem at severity 2 may have been thinking of an alternate route through the interface that they discovered. When the first evaluator hears this, they may update their understanding of the problem's context. When the second evaluator understands the first evaluator's reasoning about path criticality, they may revise their rating upward. The point is that the reasoning must be shared and examined before any rating is changed.

Consensus should always be grounded in observable evidence about the interface, not in personal preference or social dynamics. It is not acceptable to conclude that a problem is severity 3 simply because the most senior person in the room said so. Nor is it acceptable to settle on a rating because the evaluators want to avoid conflict. The question that should anchor every severity negotiation is: What does the evidence about this interface tell us about how often this problem will occur, how severely it will affect users, and whether they will be able to work around it?

A common but problematic shortcut is to average severity ratings mathematically. If Evaluator A gives a severity of 4 and Evaluator B gives a severity of 1, the arithmetic average is 2.5. But this number may reflect neither evaluator's actual position and tells stakeholders nothing useful about the problem's nature. More importantly, it hides the fact that there is real disagreement that deserves investigation. The discrepancy between a 4 and a 1 is itself meaningful — it suggests that something about the problem's impact or context was understood very differently by different evaluators, and that difference is worth exploring.

When the group genuinely cannot reach consensus after a good-faith discussion of evidence, the right approach is transparency rather than forced agreement. Document the range of ratings (for example, "Severity ratings ranged from 2 to 4 across evaluators") and record the specific rationale that each position rests on. This gives stakeholders the full picture and allows them to make an informed decision about how to prioritize the finding. It also creates an accurate record that can be revisited if user testing later reveals more information about the problem's real-world impact.

The aggregation process, done well, transforms a set of individual perspectives into a shared, evidence-based understanding of an interface's usability problems. The resulting master list is stronger than any single evaluator's report because it reflects diverse points of view, rigorously compared and systematically resolved. When that list is delivered to a design team, it gives them everything they need to make confident, prioritized decisions about what to fix — and why.

Heuristic evaluation is deliberately done by several evaluators working independently, because any single person catches only a fraction of the usability problems in an interface. That design choice creates a follow-up task: once everyone has finished inspecting on their own, the separate lists of findings have to be combined into one clean, agreed-upon set of problems that a team can actually act on. That combining step is called aggregation.

Aggregation usually happens in a structured meeting. Evaluators walk through their findings one at a time, merge duplicates that were described in different words, decide whether two similar observations are really one underlying problem or two distinct ones, settle which heuristic each violation maps to, and negotiate a single severity rating when individual ratings disagree. The output is a deduplicated, prioritized problem list with enough detail (location, heuristic violated, severity, suggested fix) for designers to use.

As you watch the videos below, pay attention to two things. First, how the evaluation process is structured from the start, since consistent note-taking and clear problem descriptions during individual evaluation are what make deduplication possible later. Second, how small numbers of evaluators or test users can still produce trustworthy, decision-ready conclusions — the same logic that justifies why we aggregate across a handful of evaluators rather than relying on one.

NNgroup · 4:50 · 45,581 views · 2y ago · captions — Watch on YouTube

About this video

A short walkthrough from Nielsen Norman Group on how to run a heuristic evaluation systematically: reviewing the interface against a set of heuristics, recording the problems found, and working through the product in an organized way. Expect it to cover the roles of multiple independent evaluators and how their individual findings feed into a combined set of results.

Why this video

It comes from the group that originated the method, and it frames the full process end to end — which is exactly where aggregation fits, immediately after individual evaluations are complete.

NNgroup · 4:35 · 5,527 views · 5y ago · captions — Watch on YouTube

About this video

This video addresses the common objection that a small number of participants cannot represent all users, and explains how limited-scale usability work still supports sound business decisions. It discusses what small-sample findings can and cannot tell you, and how to reason about risk when acting on them.

Why this video

It directly supports the "why aggregation is necessary" subtopic: the value of combining a few evaluators' or users' findings rests on understanding why small samples surface most of the important problems, and it gives you language for defending a merged problem list to stakeholders.

Together these videos reinforce the practical side of the module. The heuristic evaluation walkthrough supports MO2 by showing how violations are identified and recorded against Nielsen's principles (MO1), and it sets up the aggregation step where those individually classified violations are merged into one list. The small-sample video supports MO3 and the negotiation of severity by explaining why findings from a handful of evaluators are credible enough to prioritize and act on, which is the reasoning behind assigning agreed severity ratings and recommending specific fixes.

These videos were chosen from YouTube on October 3, 2026. Each has captions; the player turns them on. Videos on YouTube can be changed or removed — if one no longer plays, tell your instructor. If a player does not appear, use its Watch on YouTube link.