Skip to content

A Penn State Nittany Lion statue is seen in the courtyard of Penn State College of Medicine in Summer 2016. The statue is in focus toward the right side of the image. Out of focus in the background, green plants and trees are visible in the courtyard.

Inclusion and Evaluation of Digital Scholarship in the Promotion & Tenure Process at Penn State College of Medicine

Recommendations for inclusion and evaluation of digital scholarship as part of the Promotion and Tenure Process.

Committee

Chair

  • Ann Ouyang, MD

Subcommittees

Forms of Digital Scholarship

  • Ann Rogers, MD, lead
  • Vida Abedi, PhD, MS
  • Munima Nasir, MD

Impact

  • Cynthia Chuang, MD, MSc, lead
  • Vida Abedi, PhD, MS (Public Health Sciences)
  • Jennifer Maranki, MD (Medicine/Gastroenterology)
  • Jennifer McCall-Hosenfeld, MD, MSc (Medicine/General Internal Medicine)
  • Ian Paul, MD, MSc (Pediatrics)
  • Ann Rogers, MD (Surgery)

Quality

  • Priscilla Song, PhD, lead
  • Paul Haidet, MD
  • Munima Nasir, MD
  • Ira Ropson, PhD
  • Peter Waybill, MD

Rationale

Social media and other digital platforms are being increasingly used to disseminate information in science and medicine. Recognizing and evaluating such contributions as scholarship is important to capture the full extent of scholarly activities of faculty members for the Promotion and Tenure process. Current criteria for tenure line and non-tenure line faculty members for the 28 Departments at Penn State College of Medicine lack reference to digital scholarship. Only two Departments currently mention social media as valid evidence of scholarship in the domain of teaching and education, with a comment on the need to provide “evidence of impact.”

This document summarizes the suggestions of the Assessment of Digital Scholarship in Promotion and Tenure Committee. Our goal is to guide individual faculty members and promotion committees on how to include and assess digital scholarship and social media contributions in the dossier for the promotion and tenure of faculty in the Penn State College of Medicine. We acknowledge the assistance of John Russell, Digital Humanities Librarian at Penn State University Libraries, who conducted a preliminary survey of recent literature concerning the evaluation of digital scholarship for promotion and tenure in academic medicine.

Background

The traditional missions of the University are research, teaching and service. In the College of Medicine, patient care is a fourth mission. Faculty are invited to include evidence of scholarship in all relevant missions. Boyer provides useful guidance on what constitutes scholarship in these realms by focusing on the four “functions” of scholarship (ref 1), describing each of these functions as follows:

  • Discovery- original research that advances knowledge
  • Integration – synthesis that brings new insight about information and knowledge across disciplines, across topics within a discipline, or across time
  • Engagement – application and evaluation of knowledge and expertise applied to consequential problems and societal needs of individuals and institutions
  • Teaching – systematic study of teaching and learning process

The current statement of the Dean of the College of Medicine on the criteria for promotion and tenure notes that the faculty member “should have demonstrated accomplishments in: (a) the scholarship of teaching and learning; (b) the scholarship of research and creative accomplishments; (c) service, and the scholarship of service to the University, society, and the profession; and (d) patient care, and the scholarship of patient care, when applicable.” Drawing on “the UniSCOPE Multidimensional Model of Scholarship for the 21st Century” (ref 2), the Dean’s Statement states “The essential element in these accomplishments must be scholarship, which is defined as the thoughtful discovery, transmission and application of knowledge.“ The UniSCOPE model brings the four missions of scholarship (research, teaching, service, and patient care) in alignment with the functions of scholarship (discovery, with transmission and application embodied by Boyer’s concepts of integration, engagement, and teaching).

While scholarship has traditionally been evidenced by publications, new forms of scholarly production and engagement are emerging. Institutions and societies more broadly are recognizing that social media and other digital platforms can advance integration of knowledge, the application of knowledge and transmission of knowledge, thus meeting Boyer’s functions of scholarship as well as the UniSCOPE model.

Assessing the quality and impact of products of scholarship is an important element in consideration for promotion and tenure. Glassick’s criteria (ref 3) are commonly applied in this assessment and can be summarized as:

  • Clear goals
  • Adequate preparation
  • Appropriate methods
  • Significant results
  • Effective presentation
  • Reflective critique

As there is currently almost no mention of digital scholarship in the criteria for promotion in either tenure or non-tenure line faculty for the College of Medicine, this committee was convened to guide faculty and promotion committees on how to include and assess digital scholarship and social media contributions in the dossier for promotion and tenure consideration. Drawing on digital scholarship guidelines developed recently by peer institutions including Duke University School of Medicine, the committee defined digital scholarship as scholarly activity that exclusively uses digital tools to create, share disseminate and advance scientific knowledge (ref 4). The committee was divided into subcommittees to examine types of digital scholarship, assessment of quality of digital scholarship, and assessment of impact of digital scholarship. It was recognized that there is considerable overlap in considering quality and impact.

Types of Digital Scholarship

Current forms of digital scholarship include:

  • Podcasts
  • Blogs/vlogs
  • Educational websites
  • Online journal clubs
  • Channels such as YouTube
  • Facebook or other app-based specialty groups
  • Virtual education websites
  • Tweetorials
  • infographics and other visuals (e.g., Figshare)
  • Open Access Code, Model, or Data Repositories (e.g., GitHub, BioModels, Dryad, ImmPort, etc.)

The subcommittee recognized that this is an expanding field and that broad criteria based on principles is necessary to enable assessment of quality or impact.

Assessing the Quality of Digital Scholarship

While various fields (e.g., emergency medicine [8, 10, 11, 13] Refs 5-8) have attempted to produce guidelines and consensus documents for evaluating the quality of digital scholarship, these quickly become out of date or are not necessarily applicable to other fields. Given the diversity of disciplines and methodologies, as well as the rapidly changing platforms and algorithms for digital media, the quality subcommittee noted the futility of producing a universal set of standardized metrics for assessing the quality of digital scholarship. Instead, the subcommittee recommends applying Glassick’s criteria to document and evaluate the quality of digital scholarship products listed above.

Individual faculty members, in consultation with their department chair and promotion committee, should document in the narrative statement and other dossier materials how each specific digital scholarship product has a clear purpose that addresses important questions in their field, demonstrates adequate preparation, utilizes appropriate methods, achieves significant results, is presented effectively, and offers reflective critique. Documentary evidence of the quality of digital scholarship products might include, but is not limited to, the following:

  • Evidence of peer review (including pre-publication editorial review, post-publication peer commentary, etc.)
  • Evidence of meeting criteria/undergoing critical appraisal from recognized professional organizations
  • Letters of support evaluating the quality of digital scholarship contributions (from both peer collaborators and non-collaborators recognized as experts in the field)
  • Awards
  • Quality evaluation rubrics (e.g., existing tools such as the rMETRIQ score for evaluating online educational resources [refs 6,8])

Given the rapidly changing landscape of social media and other digital platforms, it is especially important for faculty members seeking to have their digital activity considered as a part of their scholarship to provide documentary evidence for the quality of their digital scholarship in the dossier using these guidelines. (ref 9,10)

Assessing the Impact of Digital Scholarship

The impact subcommittee surveyed the available literature, including the Duke University School of Medicine P&T Digital Scholarship Guidelines and available metrics assessing impact of digital platforms (refs 5-11).

The subcommittee felt that as of the current time, there are no comprehensive or reliable standalone metrics for assessing the impact of digital scholarship. Thus, this subcommittee recommends that candidates for promotion and tenure at the PSCOM use multiple strategies to convey impact, which can include the following:

  1. Cite impact using the “Suggested Impact Grid” from the Duke Guidelines. If the candidate’s forms of digital scholarship are not included in the grid, candidates may suggest other criteria to measure the impact of their form of digital scholarship (low, medium, high).  Ref 4,9
  2. Metrics (e.g., Social Media Index Score, Healthcare Social Graph Score, measures of website traffic) have multiple weaknesses, but may be included when applicable.
  3. Narrative descriptions of the impact of digital scholarship should be described in the narrative statement and in the text of the dossier. It is highly recommended that the candidate describe how their digital scholarship addresses components of Glassick’s criteria.
  4. Letter of support from colleagues who can provide testimonials as to the impact of digital scholarship. These letters are similar to the letters of support currently used for assessing teaching or patient care impact.
  5. Other measures of impact (e.g., CME/MOC credit) should also be described, when applicable.

Sample impact grid per guidelines from Duke University School of Medicine P&T Digital Scholarship Guidelines (4):

Adapted from Cabrera el al.7 Abbreviations: AIV=audio/video; DOI=digtal object identifier. *=third party analytics (e.g., Sy111plur.com)

Inclusion in Dossier

Activity Insight is the vehicle for entering all academic activity to be considered in the dossier for promotion and tenure at Penn State University.  Currently there is no easily identifiable place to provide adequate documentation of digital scholarship within Activity Insight.

The committee suggests that an optional Digital Scholarship section be added to the Activity Insight system that would allow faculty members to include a description of their digital scholarship product(s) along with an assessment of quality and impact. Baseline entry fields should include a short description of the digital scholarship product, a longer explanatory field enabling the faculty member to assess the quality and impact of their contribution with respect to Glassick’s criteria. Supplementary entry fields should enable faculty members to include evidence for quality and impact. An example can be found in the Kaiser Permanente School of Medicine CV template (pages 11-15) (ref12)

References

  1. Boyer EL. Scholarship Reconsidered: Priorities of the Professoriate. Lawrenceville, NJ 08648: Princeton University Press; 1990.
  2. D Hyman, et al. Beyond Boyer; The UniSCOPE Model of Scholarship for the 21st Century. J. Higher Education Outreach and Engagement (2001-2) 7: p41
  3. Glassick, CE. Boyer’s Expanded Definitions of Scholarship, the Standards for Assessing Scholarship, and the Elusiveness of the Teaching of Scholarship. Academic Medicine. 2000; 75(9): 877-880.
  4. Digital Scholarship (Duke University)
  5. Husain A, Repanshek Z, Singh M, Ankel F, Beck-Esmay J, Cabrera D, Chan TM, Cooney R, Gisondi M, Gottlieb M, Khadpe J. Consensus guidelines for digital scholarship in academic promotion. Western Journal of Emergency Medicine. 2020 Jul; 21(4): 883. is
  6. Thoma B, Chan TM, Kapur P, Sifford D, Siemens M, Paddock M, Ankel F, Grock A, Lin M, Alexander C, Alkhalifah M. The social media index as an indicator of quality for emergency medicine blogs: a METRIQ study. Annals of emergency medicine. 2018 Dec 1; 72(6): 696-702.
  7. Thoma B, Sanders JL, Lin M, Paterson QS, Steeg J, Chan TM. The social media index: measuring the impact of emergency medicine and critical care websites. Western Journal of Emergency Medicine
  8. Colmers-Gray IN, Krishnan K, Chan TM, Trueger NS, Paddock M, Grock A, Zaver F, Thoma B. The Revised METRIQ Score: A Quality Evaluation Tool for Online Educational Resources. AEM Education and Training: A Global Journal of Emergency Care. 2019; 3(4):387-392.
  9. Cabrera D, Vartabedian BS, Spinner RJ, Jordan BL, Aase LA, Timimi FK. More than likes and tweets: creating social media portfolios for academic promotion and tenure. Journal of graduate medical education. 2017 Aug; 9(4): 421-5.
  10. Cabrera D, Roy D, Chisolm MS. Social media scholarship and alternative metrics for academic promotion and tenure. Journal of the American College of Radiology. 2018 Jan 1; 15(1): 135-41.
  11. Maldonado G, Smart J, Wiechmann W, Kaplan SH, Billimek J, Wray A, Toohey S, Boysen-Osborn M. Frequency of social media and digital scholarship keywords in US medical schools’ promotion and tenure guidelines. Academic Medicine. 2021 Dec 28; 97(1): 105-10.
  12. Kaiser Permanente School of Medicine CV template.

Report prepared by Ann Ouyang MD, Priscilla Song,Ph.D, Cynthia Chuang MD, MSc.