DataScience@NIH Blog

We are witnessing an evolution in health data. New forms of data – such as wearable devices, patient-generated summaries, and social media feed – are rapidly proliferating as healthcare becomes increasingly digitized. These new types of data hold great promise: they have the potential to complement the clinical chart as a means of documenting the human condition, and can serve as fruitful sources of information for analysis that could improve patient outcomes. Yet, to be useful, these data need to first be organized and shared in ways that maximize their true potential.

The term “scruffy data” is drawn from the field of artificial intelligence which characterizes research solutions as being ‘neat’ (i.e., elegant, clear and provably correct) or ‘scruffy’ (i.e., where intelligence is too computationally intractable to be solved with the sorts of homogeneous system such neat requirements usually mandate).   Biologic data (e.g., genomic data) are generally considered ‘neat,’ whereas patient-generated data, which tend be heterogeneous and messy, are deemed ‘scruffy’. Emerging sources of ‘scruffy’ data were one of the foci on last week’s Partnering for Cures Conference (P4C) in Boston, MA. The goal of this conference was to accelerate patient-centered solutions and create meaningful collaboration in areas such as data sharing, translational research, training for young investigators, and a patient-centric approach to healthcare. Among the participants were senior leaders from NIH including Dr. Patricia Flatley Brennan, Director of the Library of Medicine and interim Associate Director for Data Science, and Carrie Wolinetz, Acting Chief of Staff and Associate Director Science Policy at NIH.

It is estimated that over half of the emerging health data sources, including patient-generated data, are available in free text.  There are valuable insights in the text, but we don’t yet have the tools to analyze and maximize large volumes of free text data.  Leveraging artificial intelligence and machine learning are the next frontier for analyzing the characteristics of insight embodied in patients’ words.  Machine learning capabilities can enable researchers to explore many questions simultaneously to provide a comprehensive interrogation of the problems that the biomedical research community hopes to solve. As explained by Dr. Brennan, “Machine learning will enable us to move beyond the testing of a limited number of hypotheses to “exploring the all of things.”  This opens up new vistas for analysis and understanding. It means we can go from studies that focus on a “‘n’ of one, to an ‘n’ of everything.”

Another theme of the panel discussion on ‘scruffy’ data was the use of health information for medical decision-making by patients and providers. The National Library of Medicine, which hosts over 300 databases of biomedical information, plays a critical role in making health data available to clinicians, researchers and the public. One such source of data is the ClinVar database, which is a freely accessible, public archive of the relationships among human variations and phenotypes. We heard from conference attendees about how this database can serve as a powerful tool in the interpretation of gene variants. We also heard concerns regarding the continuity of personal health data.  For instance, patients can view health information contained in the ClinVar database, but cannot directly re-engage this resource for follow-up information when data are missing or incomplete as only research laboratories or clinical testing labs can submit or update reports. One salient example came from an audience member whose daughter had a genetic condition. When the family looked up the variant, it found records of a patient with a similar genomic footprint who was listed as “28”. But the database did not say whether that individual was 28 years old, or 28 weeks old, or 28 days old. Knowing the answer to this question would have important implications for the family’s and their provider’s chosen course of treatment. Yet to engage the system, patients have to go through third parties, which can be time-consuming and is wholly dependent on the goodwill and record-keeping of the data submitters.

To make health data usable to their fullest extent, they must be gathered, cleaned and organized so they are in a form that can be readily consumed by others. However, it can be time-consuming and expensive prepare data for analysis. It is estimated that between 50-80 percent of the effort in data management is spent in the curation of data.  Patient-generated data presents particular challenges given the range of terminology sets and code profiles, and lack of agreement on which standards are to be used.

Speakers and audience members at the P4C conference shared many exciting new ideas of how to make “scruffy data” more useful and useable, such as: the use of annotation tools allowing database users to comment directly on the data, the creation of federated registries that could allow for the linking of disparate sources of data across distinct registries, and the use of  unique patient identifiers to provide a common element that could tie sources of data to a single patient. Also discussed was the need to address patient research and privacy protections in light of concerns related to patient re-identification through advanced data analytics. These are some of the issues we will be exploring through DataScience@NIH as we seek to explore the new data frontier that is emerging.

The questions moving forward for NIH and the biomedical community will be how do we maximize the value of these rapidly proliferating forms of “scruffy” data and how do we leverage them to their fullest potential to advance biomedical research and patient outcomes. To accomplish this, we need to work closely with the patients who generating the data, the researchers who are analyzing the data, and the clinicians who are using the data for decision-making. As data-intensive research initiatives evolve at NIH (e.g. AllofUs Research Program and the Cancer Moonshoot), where patient-generated data will be more integrated into the data collection, we look forward to updating you on new developments.

---

About the Author:

Elizabeth Kittrie is a Strategic Advisor for Data and Open Science at the National Library of Medicine (NLM) where she is involved in developing the NLM long-range strategic plan and leading data science and open science initiatives.  Prior to joining to the NLM, she served as a Senior Advisor to the Associate Director for Data Science at the National Institutes of Health, where she led open innovation efforts including the Open Science Prize, a collaborative partnership between the NIH, Wellcome Trust and Howard Hughes Medical Institute. Kittrie has also served as Senior Advisor to the Chief Technology Officer of the U.S. Department of Health and Human Services, where she led open government activities across HHS and coordinated the Department’s efforts to develop a common Public Access Policy.  Prior to joining HHS, Kittrie served as the first Associate Director for the Department of Biomedical Informatics at Arizona State University.

Citizen Science is a collaborative approach to research involving the public, not just as subjects of or advisors to the research, but as direct collaborators and partners. It is similar to crowdsourcing, which is defined in two ways: 1. voluntary involvement or contributions solicited from unknown individuals (aka “the crowd”), be they experts or not; and 2. opening a line of scientific inquiry to a group of experts (typically achieved through prizes and challenges).

The NIH Citizen Science Working Group, consisting of 60 staff from across NIH, investigates and shares best practices related to citizen science and crowdsourcing, and engages with other agencies and groups promoting citizen science in other fields. The working group explores how to incorporate citizen science into biomedical research while maintaining NIH’s high level of scientific and ethical standards.

We participate in various open innovation interest groups across the federal government, such as the Federal Community of Practice on Crowdsourcing and Citizen Science (CCS), comprising almost 400 members across more than 40 agencies.  The CCS shares developments in citizen science and crowdsourcing research methodology and plans related workshops. Recently, the CCS collaborated with the Office of Science and Technology Policy to assemble a toolkit on federal citizen science and crowdsourcing and to develop a catalog of federal citizen science and crowdsourcing projects, which launched in the spring of 2016.  Many NIH Citizen Science Working Group members also belong to the newly established Citizen Science Association (CSA), which just held its 2^nd biennial meeting in May.

Even the staunchest citizen science believers at CSA raise questions about data quality. We just don’t let those concerns stop us. Instead, we are working to find answers and uncover solutions.

• How can researchers ensure data collected in a citizen science project are accurate?
• What quality control methods should be in place?
• How do you make sure the project is meaningful for the communities directly impacted?

Questions such as these were proffered throughout the CSA meeting.

Studies on these very issues consistently find “the crowd” is as accurate as, and sometimes more accurate than, individual experts, and yet people still have doubts. They prefer to “leave it to the experts,” believing one must have an advanced degree to truly understand something. And though it can help, formal education is not the only route to expertise.

Experience, time, effort, and immersion can all deepen knowledge. Additionally, humans are quite adept at making inferences, visual perception, and abstract thought, regardless of degree or field of occupation. Groups of people without formal training in a specific discipline can attain the same level of accuracy in image analysis or puzzle solving as one expert. This has been demonstrated over and over again. And when it comes to medicine, are we not all experts in our own health, even if we don’t know the ins and outs of every molecular process?

Perhaps the better question is not of data quality, but of data utility. How useful is the data collected? Can the data be used again in another way to help another project? How does the data benefit the community most impacted by its use? How does the data impact metadata, ethics, or logistics?

The issues surrounding citizen science-derived data (quality, control, utility, sharing, security, etc.) are just part of the open innovation approaches our working group is exploring. We’re interested in hearing from others about what they’re doing, what question or concerns they have, and what solutions have worked for them.

Our outreach efforts will begin right here at NIH.

We’re kicking things off with a (closed) symposium on July 14, 2017 . NIH employees, volunteers, and contractors will explore examples of biomedical citizen science and crowdsourcing in paired talks given by a project leader and a citizen scientist, network with one another to find and establish new collaborations, and hear from open innovation experts from across the federal government on what works (and what doesn’t), services they provide to the larger federal community, and project highlights.

We’re hoping to excite NIH researchers about using citizen science to engage with the public and get them thinking of how they can use it to accomplish their research goals and enliven their grant portfolio.

Seating is limited so register now. The event will be recorded but not broadcast live, with the recording made available via NIH VideoCast archival services. For more information on the NIH Citizen Science Working Group please email CitizenScience@nih.gov.

---

About the Author:

Katrina I. Theisz, MS

Katrina is a Program Analyst in the Structural Biology and Molecular Applications Branch (SBMAB) of the Division of Cancer Biology (DCB), at the National Cancer Institute (NCI), part of the National Institutes of Health (NIH). She serves as the coordinator of the NIH Citizen Science Working Group, As a part of her citizen science related duties Ms. Theisz is an active member in the federal open innovation community, serving as a steering committee member of the Federal Community of Practice on Crowdsourcing and Citizen Science, and participating in the Open Data and Makers communities. She also manages the curation, development, and maintenance of CitSciBio.org, a virtual collaboration space for biomedical citizen science stakeholders. As a member of the DCB Communications and Planning group, she helps oversee the web and social media development of the division. Ms. Theisz received her Master of Science in Mental Health Counseling from Pace University, and her Bachelor of Science in Psychology from York College of Pennsylvania.

Conceptualized in 1813 and later formalized in 1962, the Federal Depository Library program made US federal publications freely accessible by allowing agencies to send documents postage-free to libraries across the country for deposit. It was perhaps one of the government’s earliest attempts at an “open” framework that provided information to the public. While the push for open data is not new, there remains a lack of consensus on how to best leverage secondary use of the data. To aid this endeavor, the research community has recently adopted “FAIR” data principles, pledging to make data “Findable, Accessible, Interoperable, and Re-usable.” But what does it mean to make data FAIR and how can open data be used to its upmost potential?

To answer this question, one must comprehend where the data came from in the first place, which is what many secondary researchers grapple with when encountering data. It has been estimated that up to 90% of a researcher’s time is spent cleaning and deciphering data. Despite this, data science in some areas of research seems more focused on analysis than provenance, creating somewhat of a false dichotomy. Provenance lies at the heart of the scientific process as researchers carefully analyze data (and any associated documentation) to understand the evolution and methodology behind the data. Provenance also relates to other important aspects such as data flow, lineage, and traceability. All these elements elucidate data in a variety of ways and are sometimes used interchangeably.

Unfortunately, there is no universal, formal definition for what constitutes “good” provenance or how to achieve it – it is easier to detect the glaring absence of provenance than it is to agree on a single definition.  To complicate matters further, it is also unclear to what extent investigators have maintained provenance. Ideally, investigators should address data provenance early on in a study; yet provenance is oftentimes an afterthought, rendering the exercise all that more difficult. In such cases, the value of the data is diminished, leading to a data paradox: vast amounts of data with very little insight.

Data provenance should and can be used to overcome the data paradox by distinguishing the novelty from the noise. Performing basic data elucidation such as saving and annotating code, documenting steps taken to manipulate data, and tracking data flow over time can promote a study’s repeatability and reproducibility. Investigators collecting data need platforms to facilitate provenance and data lineage upstream, so that researchers and analysts downstream can understand variable evolution, analyze and explain intricacies, as well as address potential data limitations. Above all else, a keen attention to detail and deep understanding of research data structures are essential for successful and sustained provenance.

As the research community continues to define the data science domain, data provenance will be key to producing FAIR, meaningful data. Much of this is dependent on study investigators not grading back their data, which is why provenance should be viewed as a part of a larger data science infrastructure. Developing this infrastructure requires a holistic approach in which data standards, formats, interoperability, security, privacy, costs, and policy levers are viewed within the data provenance lens. In today’s digital age, simply depositing data ad nauseam and hoping for the best is a disservice to the participants involved as well as the time and money invested with the data. Open data are not enough, as data must be properly tracked, managed, and explained. This not only unlocks the value of the data – it offers an opportunity for a better-informed participant, researcher, and citizen.

---

About the Author:

Dina Mikdadi is a Data Scientist at Booz Allen Hamilton, a consulting and technology firm. Dina possesses a wide range of experience in health and science research, data management, and analytics. She is part of the Health team at Booz Allen working at the intersection of science, analytics and technology. Her other areas of expertise include public health, program evaluation, policy analysis, and statistical programming.  She currently works with the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD).

The NIH Data Sharing Policy calls for the expedited translation of NIH-funded research results into knowledge, products, and procedures to improve human health. Indeed, published studies offer useful, free information for researchers, but these data sets can be hard to find or browse. To address this need, the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) launched its Data and Specimen Hub (DASH) in August 2015.

NICHD DASH shares with the research community data from completed NICHD-funded studies that vary in population, study type, data type, or disease/syndrome. It stores individual-level participant data that have all 18 HIPAA identifiers removed, enabling investigators to browse, access, and analyze information for use in secondary research. DASH not only makes these data freely available in a central repository, it helps NICHD to maximize its investments in studies on HIV/AIDS, pregnancy, infant care, child health, and other topics.

However, not all was smooth as we developed this tool. It was a long road that presented several challenges along the way. Below are a few examples of these issues (and what we did to overcome them), which we hope helps to inform others as they develop their own data-sharing repositories.

Obtaining IRB approval for data sharing
To share study data through DASH, an Institutional Review Board (IRB) or a Privacy Board must formally attest the data to be shared is consistent with the informed consent from participants in the original study. This is challenging for multi-site studies and for completed studies in which the study IRB is no longer active . To address this, the DASH Committee developed guidance and recommendations, including advice for IRBs to consider on how data are internally managed. For example, IRBs may provide an exemption of IRB review when they consider that DASH data are completely de-identified. All data requesters must execute a Data Use Agreement with NICHD and provide local IRB approval in certain cases (if required by the data submitters).

For completed projects in which the IRBs are no longer active, the DASH Committee recommends  investigators consider obtaining approval from the NICHD IRB, the NIH Human Research Protections Program, or an external commercial IRB for the necessary reviews or exemptions. On the other hand, for prospective or ongoing studies, the committee encourages investigators to ensure that language for broad data sharing is included when developing informed consent forms. The committee also suggests  the principal investigators of the original study work closely with their IRB before the study closes to ensure  data sharing is in line with informed consent procedures and data sharing/use agreements.

Ensuring data are organized and ready for meaningful reuse
The concept of data sharing through a centralized repository is still relatively new, and researchers are not necessarily familiar with the process or elements required to facilitate meaningful reuse of their data. Some of the common issues encountered with the data submitted to DASH include missing or incomplete study documentation, missing study schema (data collection schedule and milestones), data stored in a proprietary or unique software format, and incomplete de-identification of the 18 HIPAA identifiers.

The DASH Committee addressed these challenges by providing tools and resources to help researchers prepare data and research documentation for sharing. For example, the committee recommends standard organizational elements for study documentation (such as the study protocol and data dictionary) that are required for accurate interpretation and meaningful reuse of the data. DASH also provides guidance on data organization and formats, data de-identification, and coding and has developed an offline data preparation tool to annotate (for easy search and discovery) and package the data for submission.

Sharing biospecimens linked to data
Linking biospecimens with data is critical for maximizing reuse of biospecimens collected during the study. Unlike data, biospecimens are a finite resource, but sharing biospecimens generates additional challenges around participant privacy, specimen inventory management, and meaningful reuse of the specimens.

The challenges and the solutions associated with obtaining IRB approval for specimen sharing are like that of data as discussed above, but there are other challenges unique to biospecimens. For example, it is imperative that essential information about the specimen, including the type, date collected, quality, and the amount sent to the repository are accurately captured in the biospecimen inventory provided through DASH. Furthermore, for completed studies in which the biospecimens were stored in the biorepository while the study was active and/or prior to any plans to share data, linking the biospecimens with the same de-identified code as in the data has been a challenge.

Such linking must be performed retroactively by the investigators or by DASH staff, and the specimens must be relabeled prior to shipping. For active or prospective studies, investigators can ensure that specimens and data are coded with the same unique, de-identified code for sharing purposes prior to storing the specimens in the biorepository.

Raising awareness of and building support for DASH
Awareness and adoption of DASH by the research community as a data-sharing mechanism continues to require a concerted effort to ensure that stakeholders are aware of this resource. This promotion was spearheaded by a governance committee established during the planning phase of DASH to oversee its implementation, operations, and performance, especially for data submission and access.

The goal of the governance committee is to ensure the repository is effective, sustainable, and impactful. Promoting adoption of DASH requires engaging the research community early to build consensus around unmet needs. This means identifying and developing strategies around potential benefits and challenges relevant to them. The DASH stakeholder communications plan includes tactical approaches to ensure researchers understand the purpose of DASH, know how to use it, and realize its benefits, specifically as a mechanism for NICHD-funded investigators to comply with NIH’s Data Sharing Policy.

Looking ahead
While we are encouraged by DASH’s initial reception by the research community, we realize there is much more to accomplish in developing this tool. Indeed, sharing data helps derive the greatest benefit from research investments, which in turn boosts their potential impact on public health. Secondary use and analysis of the data collected in NICHD-sponsored studies will result in new discoveries that may not even be imagined at the present time. We are confident NICHD DASH will become even more valuable in the months and years ahead as we add more study data from our broad and diverse research portfolio. 

---

About the Authors:

Dr. Diana Bianchi is the director of the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), part of the National Institutes of Health. In this role, she oversees the institute's research on pediatric health and development, maternal health, reproductive health, intellectual and developmental disabilities, and rehabilitation medicine, among other areas.

Dr. Bianchi completed her residency training in pediatrics at the Children’s Hospital, Boston, and her postdoctoral fellowship training in medical genetics and neonatal-perinatal medicine, both at Harvard. She is board-certified in all three specialties and is a practicing medical geneticist with special expertise in reproductive genetics. Her translational research focuses on two broad themes: prenatal genomics with the goal of advancing noninvasive prenatal DNA screening and diagnosis, and investigating the fetal transcriptome to develop new therapies for genetic disorders that can be given prenatally. She can be reached at diana.bianchi@nih.gov.

Dr. Rohan Hazra is the chief of NICHD’s Maternal and Pediatric Infectious Disease Branch at the Eunice Kennedy Shriver National Institute of Child Health and Human Development. He oversees the Pediatric HIV AIDS Cohort Study, a multicenter U.S.-based program that follows perinatally HIV-infected youth and HIV-exposed/uninfected infants, children, and youth. He also is actively involved in pediatric HIV clinical trials and other observational studies in the United States and abroad.

Dr. Hazra's research interests include studying the long-term impact of HIV and its treatment on children, adolescents, and young adults who were infected with HIV as infants. In addition, he continues to be involved in studies evaluating new antiretroviral medications and treatment strategies in HIV-infected children, especially in resource-limited countries. He can be reached at rohan.hazra@nih.gov.

The 2015 report from the Advisory Committee to the NIH Director regarding the future of the National Library of Medicine recommended that "NLM should be the intellectual and programmatic epicenter for data science at NIH and stimulate its advancement throughout biomedical research and application." Over the past two years we’ve taken several steps in this direction and have laid plans for a few others.

Here’s a quick recap of what we’ve done and what’s on the horizon:

1. We’ve created an organizational home for data science, the Data Science Coordinating Unit, headed by Mike Huerta, PhD, NLM Coordinator for Data Science and Open Science Initiatives.
2. We’ve released a new funding opportunity seeking research that makes the benefits of data science accessible to patients and consumers.
3. We’ve expanded our extramural investments in data science research and training.
4. Come October 2017 we’ll accept the deposit of data in support of articles in PubMed Central.
5. We’ve enriched PubMed citations with links to related data sets in local repositories. 
6. We’ve made multiple enhancements to ClinicalTrials.gov.
   • It now includes both trial registration and summary results.
   • Records link to study documents in other systems and to related articles in PubMed.
   • We’ve made it possible to append relevant study documents (e.g., informed consent, study protocol).
   • And, with the help of 18F, we have improved the public interface. (Try it now!)
7. Our NCBI team is working on a cloud-implementation of safe, secure, and persistent storage of whole human genome sequences and other large, important data sets.  They will link dbGaP records, permissions, and access management to long-term storage files.
8. We’re partnering across NIH to develop the NIH and national data science workforce.
9. We’re collaborating with national and international partners, such as the Wellcome Trust and ASAPbio, on data science and open science initiatives, including the effective use of preprints.
10. We’re working with NCBI and the Common Fund to launch the Data Commons Pilot Phase to test ways to store, access, and share FAIR biomedical data and associated tools in the cloud.  Join us at the informational webinar covering the associated NIH Data Commons Pilot Phase Research Opportunity Announcement RM-17-026 Friday, June 23, 2017 from 12-2 PM US EDT.

We have more ground to cover to reach the data-powered future we envision, but through it all, NLM will be working closely with the other NIH institutes and centers to establish an NIH-wide vision of data science and open science research and resources. Watch here for further updates.