Bad research methods are a problem. They lead to inaccurate findings and the spread of misinformation.
In my field—systematic reviews and meta-analyses—an increasing number of meta-research studies are finding that many, even most, studies have deep methodological flaws.
Nevertheless, bad or superseded practices flourish. People emulate methods they saw in other studies. It’s the easy way to go, but it’s not reliable.
That’s why methods research or meta-research—research on research—is critical to improving the quality of any science, including data science.
In data science, like other areas of science, methods evolve—often quickly. Systematic reviews help make sense of the often-conflicting studies and the heated methodological controversies that result. Some fields generate hundreds, even thousands, of research articles every year, but it’s not easy to find those studies, complicating a systematic review as well. Search for information about a specific data science method, and all the studies that use the method drown out the ones that studied it.
At PubMed Health, we’re trying to help overcome that. We are developing resources to help researchers with the science of meta-analysis and other aspects of systematic reviews in health.
We start by targeting important grey literature to identify empirical studies, systematic reviews, and agencies’ best practices. They feed into a growing collection at PubMed Health, which in turn feeds into PubMed.
Next we collaborate with researchers at AHRQ’s Effective Health Care Scientific Resource Center (SRC) to regularly supply new content for the methodology research filter in PubMed. The resulting collection of meta-research and methods guidance are then searchable in PubMed using the filter sysrev_methods[sb].
Enter that phrase into PubMed’s search box like any other search term to limit the results to those related to systematic reviews. For example, click the following link to see the results of this search:
As data science develops, the science of data science is going to accelerate. We’re going to need to work together to find effective ways to help data scientists keep up. Systematically collecting data meta-science is one way to start.
About the Author:
Hilda Bastian leads the PubMed Health and PubMed Commons projects at the National Center for Biotechnology Information (NCBI) at the National Library of Medicine, NIH. A longtime consumer advocate in Australia who moved into doing, and communicating about, systematic reviews, Hilda became increasingly interested in methods and meta-research. So much so, that she is now doing meta-research towards a PhD in medical science. She also blogs and cartoons about data science and science culture.
* The views of the author are her own and not necessarily those of NCBI.