Managing Data Integrity in Times of Crisis
It can be difficult to address key business questions regarding the value of research during a crisis. Dr. Cyrus H. McCandless, VP of Scientific Discovery & Innovation, helps to address the challenges researchers are forced to confront today.
How should I think about data integrity during a crisis?
Review your sampling plans and sources: Data quality and integrity are essential to reliable insights and the clearest possible view of the road ahead. Decisions must be based on the most up-to-date, meaningful and reliable data available.
Now is the time to be a little more thoughtful about how and who you recruit—whether you’re targeting census-matched samples or specific segments—and maybe lean a little less on single-source or in-house managed sample, particularly if you’re concerned about any of the following:
- Your in-house or independently-maintained panel may not have much turnover, may be relatively insulated from the current crisis, or may not include representatives of areas that are more or less affected by coronavirus or the various localized measures taken to manage its spread
- Your panelists are likely to have participated in one or more of your surveys recently
- Your panel was constructed on the basis of a qualification unrelated to your business issues, e.g. panels constructed only of highly-active, established panelists already known to be willing to opt-in to a webcam-based tool
At the same time, we should be more mindful and appreciative than ever of respondents’ time and input. Things are tough all around, and surveys shouldn’t add to our shared feelings of isolation. New panelists who don’t feel valued, respected, or listened-to are less likely to stick around; we should do our best to make panelists feel welcome and appreciated, ensuring that future research efforts continue to have access to this expanded base of panelists.
In other words, be good to each other!
Survey respondents want to know you’re listening. Consider what your next survey script communicates about you and your brand; try to express more empathy and appreciation than the usual “Thank you for your time. Goodbye!”
To what extent should data be reproducible during a crisis?
In times of crisis, evaluating new methodologies or suppliers can be very challenging. We expect consumer sentiment and behavior to change dramatically over relatively short periods of time, making timely research execution the priority.
Reproducibility isn’t about a particular data point, it’s about proving the reliability of a scientific methodology or theory, using respondents who are either a) expected to behave consistently from test to test, or b) expected to change their behavior in ways a theory predicts, in response to events that theory claims to explain.
Once a method or theory has produced consistently useful results across a range of challenges under ‘normal’ circumstances, you know you can rely on those tools to produce accurate data and actionable insights in new and unexpected situations. Knowledge, methods, and techniques that have been proven reliable (whether through replication studies or demonstrated predictive power) don’t become unreliable just because times change, no matter how drastic or unexpected–the laws of gravity don’t change when a meteor strikes and neither do the laws of consumer cognition and behavior! Extreme events are opportunities to discover new facts about human behavior or develop new methods to study it–but circumstances alone don’t change established scientific knowledge.
In times of rapid change, you need to know that your methods and research tools have already been proven valid and reliable; that you can count on their proper application; and that you can count on the scientific expertise applied to analysis and insights. If data from your tracking study wasn’t sensitive to or able to explain known market changes in the past, or if your messaging studies weren’t really helping to make sense of your audience’s responses before now, they’re unlikely to provide reliable insights and allow you to take confident action, with predictable results, in times of crisis. Properly designed, scientifically-proven approaches produce valid predictions and meaningful insights regardless of how the world changes. Ask your vendors for case studies, the scientific evidence behind their methodologies, and evidence of reproducibility and predictive validity from times of relative calm.
Demand the best possible guidance–real science is judged by its ability to produce real answers.
Surviving a crisis in one piece requires swift–but surefooted–action.
We get it–we’re all in crisis mode. Maybe you’re concerned that insights developed during this crisis won’t have a long shelf life, or that there’s more value in studying the tail end of the trend, rather than the peak. But if your company hopes to survive long into the future, this is an extraordinarily rare chance to gather data that will be crucial to continuing your mission through [inevitable] future crises.
What can we gain from continuing research during rare but extreme situations?
There’s tremendous value in research conducted under extreme conditions. Changes in real-world conditions are rarely large enough, rapid enough, or broad enough to help you understand just how far perceptions, attitudes, and behaviors can or will deviate from the norm when different kinds of unexpected market disruptions hit, and learn how to anticipate, recognize, and respond proactively to disruptions.
Even though no one ever wishes for these things to happen, we have to recognize their intrinsic value as learning opportunities. Incidents in the modern era that come close in terms of severity are 9/11 and the 2008 financial crisis. Not only has the world changed significantly since then, but Covid-19/SARS-CoV-2 represents a very different and more uniformly-personal kind of crisis—of a kind that hasn’t occurred since the Spanish Flu of 1918. Unlike 2001 or 2008, no particular group of people is easy to blame if we’re impacted personally, and no particular group of people is immune to the worst impacts. A global pandemic is a very different beast than a financial crisis or an intentional act of terror—in other words, coronavirus has created extreme values in independent variables that are unrelated (or differently-related) to crises your company has navigated in recent years.
To prepare yourself for the recovery we all hope comes sooner than later—and to be prepared for future crises—you need to understand this moment as well as possible, with the best tools at your disposal.
For more information please contact Sentient Decision Science.
