(C) PLOS One
This story was originally published by PLOS One and is unaltered.
. . . . . . . . . .



Institutional dashboards on clinical trial transparency for University Medical Centers: A case study [1]

['Delwen L. Franzen', 'Berlin Institute Of Health At Charité - Universitätsmedizin Berlin', 'Quest Center For Responsible Research', 'Berlin', 'Benjamin Gregory Carlisle', 'Maia Salholz-Hillel', 'Nico Riedel', 'Daniel Strech']

Date: 2023-03

Abstract Background University Medical Centers (UMCs) must do their part for clinical trial transparency by fostering practices such as prospective registration, timely results reporting, and open access. However, research institutions are often unaware of their performance on these practices. Baseline assessments of these practices would highlight where there is room for change and empower UMCs to support improvement. We performed a status quo analysis of established clinical trial registration and reporting practices at German UMCs and developed a dashboard to communicate these baseline assessments with UMC leadership and the wider research community. Methods and findings We developed and applied a semiautomated approach to assess adherence to established transparency practices in a cohort of interventional trials and associated results publications. Trials were registered in ClinicalTrials.gov or the German Clinical Trials Register (DRKS), led by a German UMC, and reported as complete between 2009 and 2017. To assess adherence to transparency practices, we identified results publications associated to trials and applied automated methods at the level of registry data (e.g., prospective registration) and publications (e.g., open access). We also obtained summary results reporting rates of due trials registered in the EU Clinical Trials Register (EUCTR) and conducted at German UMCs from the EU Trials Tracker. We developed an interactive dashboard to display these results across all UMCs and at the level of single UMCs. Our study included and assessed 2,895 interventional trials led by 35 German UMCs. Across all UMCs, prospective registration increased from 33% (n = 58/178) to 75% (n = 144/193) for trials registered in ClinicalTrials.gov and from 0% (n = 0/44) to 79% (n = 19/24) for trials registered in DRKS over the period considered. Of trials with a results publication, 38% (n = 714/1,895) reported the trial registration number in the publication abstract. In turn, 58% (n = 861/1,493) of trials registered in ClinicalTrials.gov and 23% (n = 111/474) of trials registered in DRKS linked the publication in the registration. In contrast to recent increases in summary results reporting of drug trials in the EUCTR, 8% (n = 191/2,253) and 3% (n = 20/642) of due trials registered in ClinicalTrials.gov and DRKS, respectively, had summary results in the registry. Across trial completion years, timely results reporting (within 2 years of trial completion) as a manuscript publication or as summary results was 41% (n = 1,198/2,892). The proportion of openly accessible trial publications steadily increased from 42% (n = 16/38) to 74% (n = 72/97) over the period considered. A limitation of this study is that some of the methods used to assess the transparency practices in this dashboard rely on registry data being accurate and up-to-date. Conclusions In this study, we observed that it is feasible to assess and inform individual UMCs on their performance on clinical trial transparency in a reproducible and publicly accessible way. Beyond helping institutions assess how they perform in relation to mandates or their institutional policy, the dashboard may inform interventions to increase the uptake of clinical transparency practices and serve to evaluate the impact of these interventions.

Author summary Why was this study done? Clinical trials are the foundation of evidence-based medicine and should follow established guidelines for transparency: Their results should be available, findable, and accessible regardless of the outcome.

Previous studies have shown that many clinical trials fall short of transparency guidelines, which distorts the medical evidence base, creates research waste, and undermines medical decision-making.

University Medical Centers (UMCs) play an important role in increasing clinical trial transparency but are often unaware of their performance on these practices, making it difficult to drive improvement. What did the researchers do and find? We developed a pipeline to evaluate clinical trials across several established practices for clinical trial transparency and applied it in a cohort of 2,895 clinical trials led by German UMCs.

We found that while some practices are gaining adherence (e.g., prospective registration in ClinicalTrials.gov increased from 33% to 75% over the period considered), there is much room for improvement (e.g., 41% of trials reported results within 2 years of trial completion).

We developed a dashboard to communicate these transparency assessments to UMCs and support their efforts to improve. What do these findings mean? Our study demonstrates the feasibility of developing a dashboard to communicate adherence to established practices for clinical trial transparency.

By highlighting areas for improvement, the dashboard provides actionable information to UMCs and empowers their efforts to improve.

The dashboard may inform interventions to increase clinical trial transparency and be scaled to other countries and stakeholders, such as funders or clinical trial registries.

Citation: Franzen DL, Carlisle BG, Salholz-Hillel M, Riedel N, Strech D (2023) Institutional dashboards on clinical trial transparency for University Medical Centers: A case study. PLoS Med 20(3): e1004175. https://doi.org/10.1371/journal.pmed.1004175 Academic Editor: Florian Naudet, University of Rennes 1, FRANCE Received: April 28, 2022; Accepted: January 18, 2023; Published: March 21, 2023 Copyright: © 2023 Franzen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. The dashboard is openly available at: https://quest-cttd.bihealth.org/. Code to produce the dashboard is openly available in GitHub at: https://github.com/quest-bih/clinical-dashboard. Code to generate the dataset displayed in the dashboard is openly available in GitHub: https://github.com/maia-sh/intovalue-data/releases/tag/v1.1. Data can be downloaded from the dashboard and are openly available on OSF at: https://osf.io/26dgx/. Raw data obtained from trial registries are openly available on Zenodo at: https://doi.org/10.5281/zenodo.7590083. Data for summary results reporting in the EUCTR are available via the EU Trials Tracker. Funding: This work was funded by the Federal Ministry of Education and Research of Germany (BMBF 01PW18012, https://www.bmbf.de). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: The authors are affiliated to the Charité – Universitätsmedizin Berlin, one of the institutions included in this evaluation and in the dashboard. Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; CTIMP, Clinical Trial of an Investigational Medicinal Product; DOI, Digital Object Identifier; DORA, Declaration on Research Assessment; DRKS, Deutsches Register Klinischer Studien (German Clinical Trials Register); EUCTR, EU Clinical Trials Register; FDAAA, Food and Drug Administration Amendments Act; ICMJE, International Committee of Medical Journal Editors; OA, Open Access; OSF, Open Science Framework; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology; TRN, trial registration number; UMC, University Medical Center; WHO, World Health Organization

Introduction Valid medical decision-making depends on an evidence base composed of clinical trials that were prospectively registered and reported in an unbiased and timely manner. The registration of clinical trials in publicly accessible registries informs clinicians, patients, and other relevant stakeholders about what trials are planned, in progress or completed, and aggregates key information relating to those trials. Trial registration thus reduces bias in our understanding of the existing medical evidence and disincentivizes outcome-switching and selective reporting [1]. For clinical trials to generate useful and generalizable medical knowledge gain, trial results should also be reported in a timely manner after trial completion per the World Health Organization (WHO) Joint Statement on Public Disclosure of Results from Clinical Trials [2]. Disclosure is a necessary but not sufficient component of transparency: Trial results should also be openly accessible and findable, in line with established guidelines [2–6]. However, several studies have shown that clinical trials are often not registered and reported according to these standards [7–11]. Audits of research practices can build understanding of the status quo, inform new policies, and evaluate the impact of interventions to support improvement. Examples include the European Commission’s Open Science monitor [12], the German Open Access monitor [13], the French Open Science Monitor in health [14], and institution-specific dashboards of select research practices [15]. Focusing on trial transparency, the EU Trials Tracker and the Food and Drug Administration Amendments Act 2007 (FDAAA) TrialsTracker [16,17] display up-to-date summary results reporting rates of public and private trial sponsors in a transparent and accessible way. The EU Trials Tracker served as a key resource for initiatives aiming to increase reporting rates of drug trials in the EU Clinical Trials Register (EUCTR) [18,19]. Based on the EU Trials Tracker, results reporting in the EUCTR has increased from 50% in 2018 to 84% (late 2022). Research institutions such as University Medical Centers (UMCs) can incentivize practices for research transparency through their reward and promotion systems [20,21] and by providing education, infrastructure, and services [22,23]. However, internal and external assessments of research conducted at UMCs rarely acknowledge these practices [24,25]. Rather, traditional indicators of research performance such as the number of clinical trials, the extent of third-party funding, and the impact factor of published papers emphasize quantity over quality, which can entrench problematic research practices [26]. Initiatives such as the Declaration on Research Assessment (DORA) and the Hong Kong Principles have called for a change in the way researchers are assessed, and for more recognition of behaviors that strengthen research integrity [20,27]. The establishment of the Coalition on Advancing Research Assessment (CoARA) and the 2022 Agreement on Reforming Research Assessment emphasize this shift towards rewarding responsible research practices to maximize research quality and impact [28]. In turn, the UNESCO Recommendation on Open Science adopted in 2021 affirmed the need to establish monitoring and evaluation mechanisms relating to open science [29]. Audits of transparency practices could empower UMCs to support their uptake by highlighting where there is room for improvement and where to allocate resources. Comparative assessments between institutions could also provide examples of successes and stimulate knowledge transfer. Audits that are based on open and scalable methods facilitate repeated evaluation and uptake at other organizations. Such an evaluation of transparency practices at the level of clinical trials led by UMCs requires reproducible and efficient procedures for (a) sampling all clinical trials and associated results publications affiliated to UMCs and (b) measuring select registration and reporting practices. We previously established procedures for identifying all clinical trials associated with a specific UMC and their earliest results publications [9,11]. In turn, an increasing number of open-source publication and registry screening tools have been developed in the context of meta-research projects aiming to increase research transparency and reproducibility [10,30–32]. The objective of this study was to perform a status quo analysis of a set of established practices for clinical trial transparency at the level of UMCs and present these assessments in the form of an interactive dashboard to support efforts to improve performance. While the general approach of our study is applicable for UMCs worldwide, this study focused on German UMCs.

Discussion Concerns about delayed and incomplete results reporting in clinical research and other sources of research waste have triggered debate on incentivizing individual researchers and UMCs to adopt more responsible research practices [20,22,23]. Here, we introduced the methods and results underlying a dashboard for clinical trial transparency, which provides actionable information on UMCs’ performance in relation to established registration and reporting practices and thereby empowers their efforts to support improvement. This dashboard approach for clinical trial transparency at the level of individual UMCs serves to (a) inform institutions about their performance and set this in relation to national and international transparency guidelines and funder mandates, (b) highlight where there is room for improvement, (c) trigger discussions across relevant stakeholder groups on responsible research practices and their role in assessing research performance, (d) point to success stories and facilitate knowledge sharing between UMCs, and (e) inform the development and evaluation of interventions that aim to increase trial transparency. Trends in trial transparency The dashboard displays progress over time and allows the data to be explored in different ways. While the upward trend for several practices (e.g., prospective registration, OA) is encouraging, there is much room for improvement with respect to established guidelines for clinical trial transparency. For example, less than half (45%) of trials registered in ClinicalTrials.gov or DRKS and completed in 2017 reported results in a manuscript publication within 2 years of trial completion as per WHO and funder recommendations [2,43,44]. We observed a striking difference in the cumulative proportion of summary results reporting of drug trials registered in the EUCTR compared with trials registered in ClinicalTrials.gov and DRKS. The uptake of summary results reporting in the EUCTR likely reflects the combined impact of the EU legal requirement for drug trials to report summary results within 12 months [45], the launch of the EU Trials Tracker and subsequent academic initiatives to increase reporting rates [8,18], as well as media attention [46]. This suggests that audits of compliance with respect to established guidelines and further awareness raising may also have the potential to increase results reporting rates of other types of trials. Actionable areas for stakeholders Some of the practices included in this dashboard can still be addressed retroactively, such as linking publications in the trial registration (realized for 49% of trials with a publication). These constitute actionable areas for improvement that UMCs can contribute to by providing education, support, and incentives. One important way to incentivize UMCs in this regard is to make responsible research practices part of internal and external quality assessment procedures. Other stakeholders such as funders, journals and publishers, registries, and bibliographic databases should complement these activities by reviewing compliance with their policies as well as applicable guidelines and/or laws. Salholz-Hillel and colleagues, for example, outlined specific recommendations for each stakeholder to improve links between trial registrations and publications [10]. UMCs and their core facilities for clinical research can, for example, use the data linked to the dashboard to inform principal investigators about the transparency of their specific trials. We are currently finalizing such a “report card” approach at the Charité - Universitätsmedizin Berlin [47]. Scalability beyond German UMCs The datasets and methods used in this study can be scaled: This has been demonstrated in another European country (Poland) [48] and is currently underway in California, USA [49]. While the generation of the underlying dataset of clinical trials and associated results publications involves manual checks (approximately 10 person-hours per 100 trials), the assessment of transparency practices is largely automated. Institutions in possession of an in-house cohort of clinical trial registry numbers and persistent identifiers (e.g., Digital Object Identifier (DOI)) from matched journal publications, however, could achieve results more quickly. The code to create the dashboard is openly available and can be adapted to other cohorts. Stakeholder and community engagement The uptake of this dashboard approach by UMCs and other stakeholders depends on their respective attitudes and readiness. We previously solicited stakeholders’ views on an institutional dashboard with metrics for responsible research. While interviewees considered the dashboard helpful to see where an institution stands and to initiate change, some pointed to the challenge that making such a dashboard public might risk incorrect interpretation of the metrics and harm UMCs’ reputation [39]. While similar challenges with interpretation and reputation apply to current metrics for research assessment (e.g., impact factors and third-party funding), this stakeholder feedback demonstrates the need for community engagement when introducing novel strategies for research assessment. In this regard, a Delphi study was performed to reach consensus on a core outcome set of open science practices within biomedicine to support audits at the institutional level [50]. A detailed comparative assessment of existing monitoring initiatives and lessons learned could further support these efforts. Updates and further development of the dashboard We are planning regular updates of the registry data for trials already in the dashboard, as well as the inclusion of more recent cohorts of trials with at least 2 years follow-up (e.g., trials completed 2018 to 2021 assessed in 2023). Besides these updates, further transparency practices may be integrated into the dashboard in the future, e.g., dissemination of results as preprints, the use of self-archiving to broaden access to results [51], adherence to reporting guidelines [3], or data sharing [52]. Beyond transparency, other potential metrics could reflect the number of discontinued trials [53] or the proportion of trials that inform clinical practice [54]. The development of such metrics should acknowledge the availability of standards and infrastructure pertaining to the underlying practices [23] and differences between study types and disciplines [27]. Future versions of the dashboard may also display additional subpopulation comparisons, such as different clinical trial registries or UMC particularities [55]. Limitations A limitation of this study is that inaccurate or outdated registry data (e.g., incorrect completion dates or trial status) may have impacted the assessment of transparency practices described in this study. To mitigate this limitation, we updated the registry data with the most recent data we could obtain. The update-related changes suggest no systematic bias in the comparison across UMCs. Another limitation is that the trial dataset may contain more cross-registrations than we identified. For the aforementioned “report card” project, we manually verified 168 trials and found only 2 missed cross-registrations (1%). We therefore believe that missed cross-registrations represent only a small portion of our sample. Moreover, the assessment of each practice in the dashboard applies to a specific subset of trials or publications and comes with unique limitations, largely resulting from challenges associated with manual or automated methods (outlined in more detail in S5 Supplement). More generally, the dashboard focuses on interventional trials registered in ClinicalTrials.gov or DRKS and does not display how German UMC drug trials only registered in the EUCTR perform on established transparency practices (except for summary results reporting in the registry). We are considering including all drug trials in the EUCTR conducted by German UMCs in future developments of the dashboard.

Conclusions UMCs play an important role in fostering clinical trial transparency but face challenges doing so in the absence of baseline assessments of current practice. We assessed adherence to established practices for clinical trial registration and reporting at German UMCs and communicated the results in the form of an interactive dashboard. We observed room for improvement across all assessed practices, some of which can still be addressed retroactively. The dashboard provides actionable information to drive improvement, facilitates knowledge sharing between UMCs, and informs the development of interventions to increase research transparency.

Acknowledgments We would like to acknowledge Tamarinde Haven and Martin Holst for their valuable input that shaped the dashboard. We acknowledge financial support from the Open Access Publication Fund of Charité – Universitätsmedizin Berlin and the German Research Foundation (DFG).

[END]
---
[1] Url: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1004175

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/

You are viewing proxied material from magical.fish. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.