Domain Expertise Evaluation: A Forensic Methodology

Most hiring processes treat domain expertise as self-evident: the candidate either "has it" or doesn't, and the interviewer's gut settles the question. That approach has a documented accuracy problem. Schmidt & Hunter (1998), in their landmark meta-analysis published in Psychological Bulletin, found that unstructured interviews predict job performance at a validity coefficient of roughly .38—and that figure drops further when evaluators are reasoning from impressions rather than evidence. Domain knowledge, assessed without a rubric, is particularly vulnerable to the halo effect and to interviewers overweighting familiarity (candidates who use the same jargon, cite the same frameworks) over actual demonstrated competence.

This guide gives you a forensic procedure: a sequenced method for decomposing domain expertise into verifiable components, gathering evidence against each, and scoring candidates in a way that holds up to scrutiny. It is designed for hiring managers who want a domain expertise evaluation tool that is both structured and honest about its limits.

Step 1: Define the Domain's Depth Map Before You See Any Candidate

What to do

Before reviewing applications, produce a written decomposition of what domain expertise actually means for this role. A depth map has three layers:

1. Foundational knowledge — concepts that any competent practitioner must hold (e.g., for a credit-risk analyst: expected loss modeling, Basel III capital requirements, probability of default estimation).
2. Applied skill — the practitioner's demonstrated ability to translate knowledge into outputs (built a loss-given-default model, presented credit committee recommendations, managed a vintage analysis).
3. Adaptive judgment — the ability to reason under novel or ambiguous conditions where textbook answers don't apply.

What good looks like

A completed depth map is a one-page document, produced collaboratively with a current domain expert inside or adjacent to the team, listing 4–8 competencies per layer with a brief description of what evidence would confirm each. It exists before you see the first resume.

Why this sequencing matters

When evaluators define criteria after reviewing candidates, they unconsciously adjust criteria to match the candidate they already prefer—a well-documented form of post-hoc rationalization (Pager & Shepherd, 2008, Annual Review of Sociology, documented this in a hiring context; the general cognitive mechanism is treated extensively in Kahneman, 2011, Thinking, Fast and Slow). Writing the depth map first is a structural countermeasure.

Step 2: Extract Evidence from the Record — Don't Rely on Claims

What to do

For each layer in your depth map, go through the candidate's resume, LinkedIn profile, published work, portfolio, or any other submitted artifact and separate claims from evidence.

• A claim is an assertion without a verifiable referent: "deep expertise in machine learning pipelines."
• Evidence is a concrete, checkable artifact: "rebuilt the feature-engineering pipeline for a 12-feature credit-scoring model, reducing inference latency from 340ms to 80ms, documented in a 2022 internal technical report."

Use a simple two-column table:

What good looks like

A candidate with strong domain expertise will have a ratio skewed toward evidence. If the record is mostly claims, that is a finding in itself—not disqualifying, but it tells you where to probe in the interview and what to request in a work sample.

For a complementary perspective on reading records systematically, see Analyzing Interview Transcripts for Verifiable Evidence, which applies a similar logic to spoken responses.

Step 3: Design a Domain Probe — Structured, Role-Specific, Scoreable

What to do

Prepare 3–5 domain probe questions calibrated to the depth map's three layers. Each question should have a written scoring rubric before you ask it, with explicit anchors at three levels (novice / proficient / expert).

Effective domain probes are not trivia. They present a scenario or problem the candidate would plausibly face in the role and ask for reasoning, not recall.

Example probe for a senior data scientist role:

"You're handed a training dataset where the positive class is 0.3% of observations. Walk me through how you'd approach model development, and what failure modes would concern you most."

A novice answer addresses class imbalance techniques (SMOTE, weighting) and stops there. A proficient answer addresses those and discusses evaluation metric choice (precision-recall over accuracy, business cost of false positives vs. false negatives). An expert answer does all of the above and probes the data-generating process — why is the event rare, is the 0.3% stable over time, is there label noise — before reaching for a technique.

What good looks like

Scoring anchors are written before the interview. Two independent evaluators score the same response. Inter-rater reliability is checked. Structured interviews with behavioral scoring outperform unstructured formats substantially; McDaniel et al. (1994), in Journal of Applied Psychology, found situational interviews had mean validity of .50 versus .38 for unstructured formats.

Step 4: Map Evidence to Verdict's Six Dimensions

Evaluating domain expertise in isolation misses how it interacts with the rest of the candidate profile. Verdict's evaluation rubric covers six dimensions, and domain expertise contributes to several of them differently:

This mapping prevents the common error of treating domain expertise as a single binary score. A candidate can be genuinely expert in layer 1 (foundational knowledge) but thin in layer 3 (adaptive judgment), which matters enormously for senior roles.

Step 5: Calibrate Against a Benchmark — Your Best Past Hire

What to do

For each open role, identify the one or two people in your organization (or recent alumni) whose performance in a similar role has been objectively strong. Document what their domain-expertise profile actually looked like at the point of hire — not what it looks like now. This is your baseline.

Then score your candidate against that baseline, not against an idealized abstraction.

What good looks like

You have a written benchmark profile (anonymized if needed) that describes what evidence existed at hire for your reference performer. Your candidate scoring table includes a column for "benchmark," so evaluators see concretely whether the candidate clears, meets, or falls short of a proven bar.

This calibration approach is the core of Verdict's content pillar on benchmarking talent against past success. For a broader treatment of how to train evaluation frameworks on historical organizational data, Training AI Models on Historical Organizational Hires covers the underlying methodology in more depth.

Worked Example: Senior Regulatory Affairs Specialist

Depth map excerpt:

• Foundational: FDA 21 CFR Part 11, ICH E6(R2) GCP guidelines, NDA/BLA submission structure
• Applied: Has authored or co-authored at least one regulatory submission that reached agency review
• Adaptive: Can identify regulatory strategy options when the pathway is ambiguous (e.g., 505(b)(2) vs. full NDA)

Candidate record review:

Domain probe response (adaptive layer): Asked about 505(b)(2) strategy, the candidate correctly identified the pathway's reliance on existing safety data and noted the patent certification risk under paragraph IV — unprompted. That response maps to "expert" on the pre-written rubric.

Verdict dimension mapping:

• Capability: Confirmed at foundational and adaptive layers; applied layer partially verified
• Track Record: One strong artifact (BLA filing); CMC leadership partial
• Domain Edge: Paragraph IV litigation awareness is relatively rare at this level — noted as differentiator
• Risk Surface: "Expert in FDA guidelines" claim is unverified; reference check should target this specifically

Pitfalls to Avoid

1. Confusing credentials with competence. A relevant degree or certification is evidence of exposure, not demonstrated performance. Treat credentials as a starting point for inquiry, not a conclusion. Hunter & Hunter (1984), Psychological Bulletin, found that education-based selection has a validity of approximately .10 for job performance — useful but not sufficient.

2. Letting the expert interviewer go unstructured. Subject-matter experts on your panel are valuable but are also prone to evaluating rapport and shared vocabulary rather than genuine depth. Give them the same rubric and scoring anchors as everyone else.

3. Ignoring the depth map's obsolescence risk. Domain expertise degrades and shifts. A candidate with deep expertise in a method that is being displaced by newer approaches presents a different risk profile than their credential list suggests. Include a "currency" check in your depth map — when was this skill last applied in a live, production context?

4. Treating a single strong probe answer as confirmation. One expert-level response is a data point. Domain expertise is a pattern across multiple evidence sources. Require evidence at more than one layer before scoring a candidate as domain-strong.

5. Failing to document. A forensic evaluation that isn't written down is just an opinion. Every scoring decision should have a corresponding evidence note. For a framework on building that paper trail defensibly, see How to Document Hiring Decisions and Build a Paper Trail.

Evaluate Candidates with Verdict

If you want to run this methodology against a real job description — with structured scoring across Capability, Track Record, Trajectory, Influence, Domain Edge, and Risk Surface — Verdict gives you a purpose-built instrument to do exactly that. It won't make the decision for you, but it will ensure you're comparing candidates against a consistent, evidence-cited benchmark rather than a shifting impression. Run a structured evaluation against your own job description and see what the evidence actually shows.