NIH Virtual Workshop on Data Metrics

The National Institutes of Health’s Office of Data Science Strategy hosted a virtual workshop on assessing dataset and data resource value and reach on Feb. 19, 2020.

Virtual Workshop

The goal of this workshop was to discuss core metrics, use cases, and best practices to better understand data usage and impact. The workshop focused on two types of data resources – repositories and knowledgebases – and brought together managers of diverse biomedical data resources to discuss community-supported best practices for data metrics.

NIH-funded data resources (repositories and knowledgebases) use varied approaches to measure use and utility (community value/impact) of the resource itself, and/or the use and utility of specific datasets held by the resource. Data resource managers, funders, and users are interested in understanding the impact and value of data resources and the data they host. This workshop included speakers representing each of these stakeholders to discuss approaches to and values in evaluating research data and its infrastructure.

Some of the questions that participants considered included:

What are the long-term positive or negative consequences of having evaluation metrics for research data?
Are there existing standards or methodologies for assessing research data value and reach?
How might different stakeholders (data resource users, managers, or funders) use data metrics?

NIH Data Science IdeaScale was used to gather input from the community. The agenda and presentations are below, along with recordings of each presentation and panel discussion. A summary of the workshop is also available.

Agenda
Time	Presentation
9:00 a.m. – 9:10 a.m. EST	Opening Remarks & Keynote Chair: Dr. Dawei Lin, NIAID, NIH View Presentation

9:10 a.m. – 9:40 a.m. EST	Dr. Stefanie Haustein, University of Ottawa, Cautions and lessons around development and implementation of scholarly metrics View Presentation

Session 1: Evaluating and measuring data use and utility Session chair: Daniella Lowenberg Presentations (15 minutes + 5 minutes speaker transition time):

9:40 a.m. – 9:55 a.m. EST	Daniella Lowenberg, California Digital Library, Introduction to current efforts around the building blocks of data metrics: data usage and citation View Presentation

10:00 a.m. – 10:15 a.m. EST	Dr. Valerie Schneider, National Institutes of Health, Perspectives from a data center with multiple repositories View Presentation

10:20 a.m. – 10:35 a.m. EST	Dr. Susan Redline, Harvard University, Perspectives from a domain specific data repository: The National Sleep Research Resource View Presentation

10:40 a.m. – 10:55 a.m. EST	Dr. Regina Bures, National Institutes of Health, Perspectives from a controlled access data resource View Presentation

Panel Discussion: 11:00 a.m. – 12:00 p.m. - Questions from participants, plus framing questions Framing questions: What types of indicators have been or could be gathered to measure research data or data repository/knowledgebase value and reach? How does the maturity of a data repository/knowledgebase or the community it represents impact the evaluation of data metrics? How can repositories use existing initiatives and experts around standardized data usage, data citation, and data metrics to responsibly report on data value? View Panel Discussion

12:00 p.m. – 1:00 p.m. EST	Lunch Break

Session 2: Stakeholder use cases for data usage and utility metrics Session chair: Dr. Warren Kibbe Presentations (15 minutes + 5 minutes speaker transition time):

1:00 p.m. – 1:15 p.m. EST	Sean Coady, National Institutes of Health, Using Metrics to inform the Return on Investment (ROI): A Public Funder’s Perspective View Presentation

1:20 p.m. – 1:35 p.m. EST	Dr. Robert Moritz, Institute for Systems Biology, Manager perspective: How do resource managers use metrics to articulate the size, impact and scope of their resource, and the stakeholders of the resource? View Presentation

1:40 p.m. – 1:55 p.m. EST	Dr. Brian Byrd, University of Michigan, Community perspective: how do research communities help demonstrate and maximize the utility of a resource and the data it holds. How can metrics promote usage and utility of a resource, and justification for continued support? View Presentation

Panel Discussion: 2:00 p.m. – 3:00 p.m. - presentation Q&A, discussion guided by framing questions Framing questions: What use cases and questions should all data repositories/knowledgebases have in common around dataset or repository impact? What metrics to evaluate reach and impact do funders, repository managers, data generators and data users want access to? How could established data metrics help in incentivizing researchers to publish data, or influence broader compliance with policy implementations or promotion and tenure processes? View Panel Discussion

3:00 p.m. – 3:10 p.m. EST	Closing Remarks: Dr. Kim Pruitt, NLM, NIH

3:10 p.m. – 3:30 p.m. EST	Closing Speaker: Dr. Susan Gregurick, ODSS, NIH View Closing Remarks and Speaker