Data Scientist interview scorecard template
A structured scorecard for interviewing a Data Scientist: six weighted competencies, what a 1 and a 5 actually look like, and questions that surface evidence instead of opinions. Print it, or copy it into your ATS.
The Data Scientist scorecard
| Competency | Weight | Score 1 — what it looks like | Score 5 — what it looks like |
|---|---|---|---|
| Framing a problem as a modeling task Whether they can turn a vague business ask into a target variable, a unit of analysis, and a decision the model output will feed. | 18% | Starts from the algorithm rather than the decision, and cannot say what the label was, what a row represented, or who would act on the output. | States the decision first, then the target, the unit of analysis and the cases they deliberately excluded, explaining why each choice narrowed the problem. |
| Experimental design and causal reasoning Whether they can design a test that answers the question asked, since most requests reaching a data scientist are causal and correlational answers mislead. | 20% | Reports A/B test results as a lift number, without a randomization unit, sample size reasoning, or anything they checked before calling the result real. | Names the randomization unit, the metric fixed in advance, the duration and power reasoning, and one confound they found and handled in a specific test. |
| Modeling craft and validation Whether their offline results survive contact with production, which depends on leakage checks, honest baselines and a split that mirrors deployment. | 20% | Reports accuracy on a random split with no baseline comparison, and cannot say whether any feature encoded information unavailable at prediction time. | Describes a leakage or drift they caught, how the metric moved once it was fixed, and the baseline that made the model worth deploying at all. |
| Statistical reasoning under uncertainty Whether they know the limits of their own numbers, which decides whether a stakeholder acts on a result that is really noise. | 16% | Presents point estimates without intervals and treats a p-value threshold as a decision, with no account of how many comparisons were run. | Quantifies uncertainty as a range with the assumption attached, and can name a result they refused to report because the sample would not support it. |
| Production handoff and monitoring Whether the models they build keep working after handoff, since a model that is never deployed or never monitored delivers nothing. | 14% | Work ends at the notebook, the path to production is described as a job for another team, and they do not know how the model performed afterward. | Describes how a model was served, what was monitored after launch, and the retraining or rollback that a specific degradation triggered. |
| Communicating uncertainty to decision makers Whether non-technical stakeholders end up with a correct picture of what the analysis can and cannot support before they commit to it. | 12% | Presents methodology rather than implication, or reduces the result to a single number that hides the caveat that would have changed the decision. | Recalls a case where they told a stakeholder the data could not answer the question, and what they proposed measuring instead. |
Weights sum to 100. Agree them before the first interview, not after — adjusting weights once you have scores is how a panel rationalises a favourite.
Questions that surface evidence
Each one asks for something that already happened, in enough detail to verify. Hypotheticals reward rehearsal, not track record.
- Describe an experiment you designed end to end. What was the randomization unit, how did you decide the sample size, and what did you check before trusting the result?
- Tell us about a model whose offline metrics looked good and whose production behavior did not. What was different, and how did you find out?
- Tell us about a time you had to tell a stakeholder the data could not answer their question. What did you propose measuring instead, and what did they do?
- What is a result you decided not to report, or not to act on? What made you distrust it, and what did you do next?
- Walk us through the features of a model you shipped. Which one mattered most, how did you verify it was available at prediction time, and what did you drop?
Red flags
- Cannot state a baseline for any model they built, only the final metric.
- Every project is described as a modeling win, with no failed hypothesis or discarded approach.
- Uses causal language for observational results without naming a single assumption or confound.
- Cannot say who used the output of their work or what decision it changed.
How to use this scorecard
- Agree the weights with the panel before anyone interviews, and write them down.
- Every interviewer scores every competency independently, adding a note that quotes what the candidate actually said.
- Compare scores before discussing them. Discussing first anchors the panel on whoever speaks loudest.
Build a custom scorecard → · Boolean string to source a Data Scientist →
Frequently asked questions
What is the difference between a data scientist and a data analyst when hiring?
A data scientist is hired for questions that need modeling or experimental design: causal estimates, predictions, A/B tests. An analyst is hired to make existing data answer business questions reliably and repeatedly. Interviewing both against the same rubric will mis-score at least one of them.
What should a data scientist technical interview include?
One experimental design problem, one modeling problem with a validation trap in it, and one case where the honest answer is that the data cannot support a conclusion. Add a walkthrough of a past project in the same session so you can compare stated method against actual practice.
How do I compare data science candidates fairly when their projects are all different?
Score the reasoning rather than the domain: framing, validation, uncertainty and handoff apply to any project. Fix the weights before the first interview and write the evidence beside each score. Verdict applies the same discipline to CVs, tying every dimension score to a cited snippet so candidates stay comparable.
When the stakes are a real hire, use evidence
These tools are quick heuristics. Verdict reads the CV against your job description and scores six dimensions with verbatim quotes as evidence — a hiring document you can defend.