Data journalists often talk about the importance of documenting our analytical processes, so we know what we’ve done and why. Sure, we may record our methods, but sometimes we skip the question that comes first: How do we know that our logic was sound and our results are error-free?

Often, the answer is, “Ask someone else to take a look,” which can get tricky if you’re solo or on a smaller team. Resources like OpenNews’ Peer Data Review or the Lonely Coders’ Club Slack can help. But sometimes you just need a solid process to check your own work.

I developed my own QA process while working at a smaller newsroom with fewer resources, and while working as a data analyst outside of journalism. It’s come in handy on the data team at Chalkbeat, where our three-person team scrapes, analyzes, and visualizes data for reporters across ten states.

Most of our projects are small enough that one data journalist can handle them from start to finish. There’s always a second person who can take a look, but because each person specializes in a different set of technical tools, a robust one-person QA process is also an important part of the team workflow. It catches errors earlier — which frees up time to focus peer review energy on higher-level conceptual concerns or design and accessibility tweaks.

Here’s how I pressure-test my analyses.

Start from what you already know

Isolate your knowns. Turn these into control values — totals, numbers, proportions, or directional trends that should remain consistent throughout your analysis.

For instance, when I analyzed enrollment, achievement, and missing data in Colorado’s virtual schools, I worked from a few knowns: a 2019 state report for online enrollment, statewide enrollment totals for all school types, and performance reports that showed state-identified data limitations. I knew that 1) my filtered virtual enrollment should closely match that 2019 report, but change in 2020 forward 2) that brick-and-mortar enrollment should closely track statewide enrollment 3) that my virtual and brick-and-mortar totals should add up to that state control total 4) that some data would just be missing entirely.

Partially, this is about using what you learn as you first explore your data, in those early evaluations of its shape, its distribution, and its limitations. You can revisit these early details to make sure you’re seeing the same patterns at the end of your analysis that you saw at the start. But it’s also about aligning novel analysis with public knowledge or external information, when available.

• What totals should always add up across your analysis? (For the Colorado online schools story, I made sure I didn’t lose students across filters by referring to the statewide total.)
• What values are known issues (ie, what proportion of missing or null values should you have in which columns)?
• What proportions should be consistent throughout your analysis?
• Does your more granular data match up to your less granular data, if it’s additive? (For education data, I usually check whether school-level values add to match district and state aggregates).
• Do fluctuations in sums or averages indicate any potential errors, duplicates, outliers, or other values that snuck in or out during your analysis?

It’s helpful to pick your own control values by determining the basic trends and bounds of your analysis. But don’t be afraid to look outside of your own work to see how your findings align (or don’t) with existing methods and outcomes. If your work covers a specific area, try to match your analysis to any existing reports that provide regional totals. Even if your analysis is largely providing new information, it should still have some sort of relationship to a larger group or pre-existing, documented trend.

Restructure your data and see what happens

There are a lot of ways to do this, but don’t underestimate the power of pivot tables. The goal here is to treat the pivot table as a shorthand data exploration tool, quickly glimpsing notable trends and aggregates rather than looking for specific numbers.

I’ve mostly used this to catch procedural errors, like a bad SQL join that dropped a few schools with non-standard IDs or a category that got filtered out. It’s also a great way to notice that you accidentally forgot to import 2018 when you combined 10 government spreadsheets into a single dataset. (For instance, on early stages of an analysis of charter school network expansions in New York, a pivot table revealed that the historical enrollment data seemed to use different codes for the same schools, so some schools were getting falsely flagged as “newly opened” or “newly closed” based on an incomplete ID crosswalk.)

Run some quick sums:

• Do you have a similar number of values for each category? This is especially useful for year-over-year analysis — if your original data is weird or bad in a particular year, or the data recording method changed, you might see a significant change.
• Do your pivot table totals line up with the control totals?
• Keep an eye out for data oddities—check your row numbers, watch for repetition of the same values or value counts. Pay special attention to trends and relationships that should be true of basically any data set: for instance, make sure that your unique ID counts are generally lower than your non-unique ID counts, if you have both.
• If your analysis just involves simple pivots, try a few _other _pivots to look for patterns that might merit closer review.

You can also take this a step further by doing a rough data viz in Datawrapper or another quick viz platform.

Retrace your steps with different tools

Pick a single end value—something that you’ve calculated or otherwise manipulated, not a raw number in the data. Then, repeat your analysis for that single value. Don’t use your code or your Excel formulas. Just do the plain, old math. Then, pick another value with significantly different attributes and repeat the process.

When things are simple, I just pick up a calculator, write down all my formulas, and recalculate a few representative values—and a few edge cases. If something gets more complicated and I have enough time to dig in, I try to spot-check or fully replicate my work in another language. I’m most comfortable in SQL, so I might convert into SQL from JavaScript (where I’m less familiar with the data types) or Pandas (which I use a lot, but find less intuitive).

The specific tool isn’t important: This is really about thinking through a problem a bit differently, so that assumptions about nulls, outliers, exceptions, etc., get handled in new ways, and oversights might become more noticeable. Instead of using Excel to create a pivot table or R or Pandas to run built-in methods, can you mirror that logic from scratch?

Try a new method of calculation, with fresh assumptions

This is particularly handy for catching logic errors, which might slip through the cracks of a basic numeric accuracy check.

Can you answer the same question multiple ways? Usually, yes. Try using a different mathematical process to replicate your results. Or, if you’ve made certain assumptions—logical exclusions, buckets or filters, redefinitions—change up those assumptions. Then run your analysis again. Your changed assumptions should generally also change a few outcomes. Look at how the parts move.

Sometimes I’ll use bad assumptions to perform this check. It’s less about whether the assumptions are right, and more about whether or how they change the results of your analysis. (If your “bad assumptions” lead you somewhere more reasonable than you might expect, it can also signal that you need to revisit your parameters.)

In the Colorado online schools story, we had two definitions of “online students”: students who attended virtual schools, as defined by the state, or students who attended an “online program” by a specific federal definition. The state definition excluded some students captured under the federal definition, and the federal definition claimed that some students attending fully virtual schools weren’t actually online students. To check our conclusions, we looked at the trends for both definitions, and also created our own definition that approximated what we thought “real” online enrollment looked like. If any of these had seemed significantly inflated or deflated—or the overall numbers deviated dramatically—we might have had a problem. (Thankfully, they were all fairly similar, with a non-statistically-significant amount of fuzziness.)

As you do this, ask yourself the following:

• How would you expect your results to change based on the new limitations, definitions, or exclusions that you’ve applied?
• Do those changes seem fair and reasonable?
• Do the results seem skewed or misrepresentative? (Should they be, based on your understanding of how your exclusions impacted the analysis?)
• Do the numbers you’d expect to respond to these revisions actually change in the ways you’d expect? If something went up that should have gone down, for instance, it’s time to find out why.

Look at the reported assertions that depend on these results.

• Are those assertions undermined or invalidated by the new methods?
• If there are significant changes in the results, can you reasonably defend why one method is superior to the other?
• If there’s not a strong reason to use one method over the other, do you need to reconsider the presentation or language of your analysis?

In general, if things are changing in a way that doesn’t make sense based on your initial assumptions, it might mean that you’ve made a bad logical assumption somewhere, or you have a math error.

It’s all about shifting perspective

Overall, most of these methods aren’t about finding the right answer, or validating an existing assumption. They’re about reconfiguring your analysis, thinking about it differently. Sometimes the best QA involves putting on your “obsessive naysayer” hat, trying to prove your own results wrong, and then continuing to do that until you’ve identified every error, discrepancy, or logical limitation.

Some of these steps are also commonly recommended at the beginning of an analysis, as initial exploration. Consider revisiting these checks anytime you import new data, change directions, or start a new branch of analysis based on work you’ve already completed. Also, “redoing” some of your initial fact-finding processes is just a good way to ensure that the facts haven’t changed in ways unanticipated or unaddressed by your analysis.

If I’m really stumped, it can also help to talk through the logical assumptions or have someone double-check a key piece of code or formula. A QA process can help you identify the crucial pivot points on which the rest of your analysis hinges—aka, the things most worth finding a peer to review.

Don’t be afraid to ask for help—but when there’s just not time or resources, these tactics can help you find problems and build confidence in your results.

• Share on Twitter
• Share on Facebook