NIH Virtual Workshop on Data Metrics

The National Institutes of Health’s Office of Data Science Strategy hosted a virtual workshop on assessing dataset and data resource value and reach on Feb. 19, 2020. 

Virtual Workshop

The goal of this workshop was to discuss core metrics, use cases, and best practices to better understand data usage and impact. The workshop focused on two types of data resources – repositories and knowledgebases – and brought together managers of diverse biomedical data resources to discuss community-supported best practices for data metrics.

NIH-funded data resources (repositories and knowledgebases) use varied approaches to measure use and utility (community value/impact) of the resource itself, and/or the use and utility of specific datasets held by the resource. Data resource managers, funders, and users are interested in understanding the impact and value of data resources and the data they host. This workshop included speakers representing each of these stakeholders to discuss approaches to and values in evaluating research data and its infrastructure.

Some of the questions that participants considered included:

  1. What are the long-term positive or negative consequences of having evaluation metrics for research data?
  2. Are there existing standards or methodologies for assessing research data value and reach?
  3. How might different stakeholders (data resource users, managers, or funders) use data metrics?

NIH Data Science IdeaScale was used to gather input from the community. The agenda and presentations are below, along with recordings of each presentation and panel discussion. A summary of the workshop is also available.

Agenda
TimePresentation
9:00 a.m. – 9:10 a.m. ESTOpening Remarks & Keynote Chair: Dr. Dawei Lin, NIAID, NIH
View Presentation
 
9:10 a.m. – 9:40 a.m. ESTDr. Stefanie Haustein, University of Ottawa, Cautions and lessons around development and implementation of scholarly metrics
View Presentation
 
Session 1: Evaluating and measuring data use and utility
Session chair: Daniella Lowenberg
Presentations (15 minutes + 5 minutes speaker transition time):
 
9:40 a.m. – 9:55 a.m. ESTDaniella Lowenberg, California Digital Library, Introduction to current efforts around the building blocks of data metrics: data usage and citation
View Presentation
 
10:00 a.m. – 10:15 a.m. ESTDr. Valerie Schneider, National Institutes of Health, Perspectives from a data center with multiple repositories
View Presentation
 
10:20 a.m. – 10:35 a.m. ESTDr. Susan Redline, Harvard University, Perspectives from a domain specific data repository: The National Sleep Research Resource
View Presentation
 
10:40 a.m. – 10:55 a.m. ESTDr. Regina Bures, National Institutes of Health, Perspectives from a controlled access data resource
View Presentation
 

Panel Discussion:
11:00 a.m. – 12:00 p.m. - Questions from participants, plus framing questions
Framing questions:

  • What types of indicators have been or could be gathered to measure research data or data repository/knowledgebase value and reach?
  • How does the maturity of a data repository/knowledgebase or the community it represents impact the evaluation of data metrics?
  • How can repositories use existing initiatives and experts around standardized data usage, data citation, and data metrics to responsibly report on data value?

View Panel Discussion

 
12:00 p.m. – 1:00 p.m. ESTLunch Break
 
Session 2: Stakeholder use cases for data usage and utility metrics
Session chair: Dr. Warren Kibbe
Presentations (15 minutes + 5 minutes speaker transition time):
 
1:00 p.m. – 1:15 p.m. ESTSean Coady, National Institutes of Health, Using Metrics to inform the Return on Investment (ROI): A Public Funder’s Perspective
View Presentation
 
1:20 p.m. – 1:35 p.m. ESTDr. Robert Moritz, Institute for Systems Biology, Manager perspective: How do resource managers use metrics to articulate the size, impact and scope of their resource, and the stakeholders of the resource?
View Presentation
 
1:40 p.m. – 1:55 p.m. ESTDr. Brian Byrd, University of Michigan, Community perspective: how do research communities help demonstrate and maximize the utility of a resource and the data it holds. How can metrics promote usage and utility of a resource, and justification for continued support?
View Presentation
 

Panel Discussion:
2:00 p.m. – 3:00 p.m. - presentation Q&A, discussion guided by framing questions
Framing questions:

  • What use cases and questions should all data repositories/knowledgebases have in common around dataset or repository impact?
  • What metrics to evaluate reach and impact do funders, repository managers, data generators and data users want access to?
  • How could established data metrics help in incentivizing researchers to publish data, or influence broader compliance with policy implementations or promotion and tenure processes?

View Panel Discussion

 
3:00 p.m. – 3:10 p.m. ESTClosing Remarks: Dr. Kim Pruitt, NLM, NIH
 
3:10 p.m. – 3:30 p.m. ESTClosing Speaker: Dr. Susan Gregurick, ODSS, NIH
View Closing Remarks and Speaker

This page last reviewed on June 6, 2024