(C) PLOS One
This story was originally published by PLOS One and is unaltered.
. . . . . . . . . .



Validation and implementation of a mobile app decision support system for prostate cancer to improve quality of tumor boards [1]

['Yasemin Ural', 'University Of Cologne', 'Faculty Of Medicine', 'University Hospital Cologne', 'Department Of Urology', 'Uro-Oncology', 'Robot Assisted Surgery', 'Thomas Elter', 'Department I Of Internal Medicine', 'Center For Integrated Oncology Aachen Bonn Cologne Duesseldorf']

Date: 2023-06

Abstract Certified Cancer Centers must present all patients in multidisciplinary tumor boards (MTB), including standard cases with well-established treatment strategies. Too many standard cases can absorb much of the available time, which can be unfavorable for the discussion of complex cases. In any case, this leads to a high quantity, but not necessarily a high quality of tumor boards. Our aim was to develop a partially algorithm-driven decision support system (DSS) for smart phones to provide evidence-based recommendations for first-line therapy of common urological cancers. To assure quality, we compared each single digital decision with recommendations of an experienced MTB and obtained the concordance.1873 prostate cancer patients presented in the MTB of the urological department of the University Hospital of Cologne from 2014 to 2018 have been evaluated. Patient characteristics included age, disease stage, Gleason Score, PSA and previous therapies. The questions addressed to MTB were again answered using DSS. All blinded pairs of answers were assessed for discrepancies by independent reviewers. Overall concordance rate was 99.1% (1856/1873). Stage specific concordance rates were 97.4% (stage I), 99.2% (stage II), 100% (stage III), and 99.2% (stage IV). Quality of concordance were independent of age and risk profile. The reliability of any DSS is the key feature before implementation in clinical routine. Although our system appears to provide this safety, we are now performing cross-validation with several clinics to further increase decision quality and avoid potential clinic bias.

Author summary The quality of therapeutic decisions provided in tumor boards is perhaps the most relevant criterion for optimal cancer outcome. This tool aims to provide optimal recommendations, to assess the quality on a case-by-case basis and furthermore to objectively display the quality of oncological care.

Author summary Everyday clinicians face the difficult task to choose the optimal treatment for their cancer patients due to the emergence of newly available therapeutics and continuously altering treatment guidelines. The resulting flood of information is impossible for clinicians to keep up with. Therefore, clinicians decide as a team, in so called tumor boards, upon the best possible cancer treatment for each patient. Even though the treatment decisions recommended by tumor boards play a critical role for the long-term survival of cancer patients, their accuracy in decision-making has hardly ever been assessed. Unfortunately, current digital tools that have been developed to support clinicians on the process of decision-making, have difficulties to provide treatment recommendations with sufficient accuracy. Therefore, we evaluated the quality of a novel decision-making application by comparing the decision concordance generated by the App with therapeutic recommendations given by a tumor board of a University Cancer Center. For newly diagnosed cancer patients we found that the novel tool matched the decisions made by the tumor board in almost 100% of the cases. These promising results not only show the potential of providing digital support for patient care, but also provide objective quality management while saving board time in favor of discussing more complex cases.

Citation: Ural Y, Elter T, Yilmaz Y, Hallek M, Datta RR, Kleinert R, et al. (2023) Validation and implementation of a mobile app decision support system for prostate cancer to improve quality of tumor boards. PLOS Digit Health 2(6): e0000054. https://doi.org/10.1371/journal.pdig.0000054 Editor: Haleh Ayatollahi, Iran University of Medical Sciences, IRAN (ISLAMIC REPUBLIC OF) Received: April 29, 2022; Accepted: April 27, 2023; Published: June 7, 2023 Copyright: © 2023 Ural et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: Our data was published on the open repository Zenodo (DOI: 10.1371/journal.pdig.0000054.). The full data set can be accessed by everyone using the following link: https://zenodo.org/record/6951736#.Y5rxDOzMLyj. Funding: The authors received no specific funding for this work. Competing interests: I have read the journal´s policy and the authors of this manuscript have the following competing interests: T. E. is the founder and chief medical officer of the company who developed the application "EasyOncology".

Introduction Uro-Oncologists and Oncologists worldwide face the challenge of ensuring that their patients receive the best possible, individualized care for their cancer disease. Keeping up with the fast developments in medicine is very difficult for many physicians, as the rapid growth of scientific knowledge leads to an almost unmanageable variety of new treatment options [1]. The large amount of data is overwhelming and consequently it becomes a demanding task, to decide for the best possible individualized therapy for the patient. Therefore, case discussion in multidisciplinary tumor board (MTB) conferences is one of the most important factors to assure highest quality standards in oncological care. Driven by this assumption, German hospitals are required to present and discuss each cancer patient in MTB in order to get registered as certified cancer center by the German Cancer Society (Deutsche Krebshilfe) [2]. In clinical reality, certification requirements for presenting all routine cases leads to a significant increase in the number of case discussions, wasting valuable time and attention that is needed for the discussion of more complex tumor cases. In the field of evidence-based oncology, it is almost paradoxical that the quality of individual therapeutic decisions of the tumor boards is hardly ever qualitatively assessed. Despite their widespread use in clinical routine, few data is available about the effects of tumor boards on quality of care and long-term survival of cancer patients [3,4]. This results in the need to objectively define and measure the quality of MTB at the level of individual therapeutic recommendations. Without doubt, AI based clinical decision support systems will play a major role in the near future to close the gap between complex data and clinical decision-making [4–6]. Up to this point, systems based on artificial intelligence have been unable to offer reliable assistance in this area, as they are still failing to provide treatment recommendations with sufficient certainty even to standard questions on first-line therapy [7]. For example, one of the leading AI-based systems, Watson for Oncology, matched only 12% (stomach), 80% (colon and breast carcinoma) and 93% (ovarian carcinoma) of the treatment recommendations given by medical experts [8–13]. An important reason for the poor performance of AI systems is the lack of high-quality training datasets. Masses of normalized training data are available for AI image recognition systems, but not for AI applications that are intended to model regular oncology care. In addition to the lack of properly organized and validated training data, another problem is the limited resource of experts initially required for human interpretation and evaluation of AI results. At the end of the day, then, a machine learning tool can only be as good as the data available for training and the trainers who evaluate the AI results. Another approach to provide digital therapeutic recommendations with sufficient certainty could be implemented by developing software based on clinical network expertise. This concept of expert-curated digital decision support systems (DSS) was described in Nature Biotechnology in 2018 and a comparison with approaches of artificial intelligence showed multiple benefits [14]. The main advantage described here is certainly that the expert systems seem to represent clinical reality better than the AI-based systems used so far. The DSS smartphone application EasyOncology (EO), whose digital treatment recommendations are based on continuous matching with real tumor boards, follows this approach and led to the design of this research study. To the best of our knowledge, tumor board decisions have not been validated on a case by case basis by using an expert-curated DSS. In addition, our study allows a direct comparison of the level of concordance reached by an AI-based DSS with an expert-curated DSS with respect to MTB decisions for prostate cancer patients. The aim of this clinical research is to implement the aforementioned technology for validation and quality assurance of a urological tumor board at the University Hospital of Cologne.

Discussion The rapid development of new, innovative oncological treatment options leads more than ever to the requirement of quality-assured therapeutic decisions [17]. In order to give optimal treatment recommendations, physicians usually follow guidelines of medical societies, inform themselves through professional journals, participate in congresses, further their medical education and discuss cases in multidisciplinary tumor boards (MTB). However, who can guarantee that all doctors working in oncology have the time and motivation to handle the information overload? How do they deal with this situation when even current guidelines of the medical associations sometimes fail to mention highly effective and newly approved therapeutics? Who ensures that the expertise of the doctors attending the MTB is actually given and that decisions are not (consciously or unconsciously) influenced by economic motives? Is there any evidence at all that tumor boards really improve the quality of oncology care [3,17–19]? AI-based systems seem to have the potential to support clinical decision making, as they have already impressively demonstrated their outstanding superiority in medical image processing and interpretation for different cancer entities [20–23]. Especially since AI-based systems seem perfectly suited to capture and correlate the immense amount of oncological knowledge, the results of all relevant clinical trials and all published case reports, and, based on this knowledge, to finally generate therapeutic recommendations. As obvious as this sounds, it is almost surprising that AI-based systems have yet failed to establish themselves in clinical oncological routine. So far, most attempts of AI-systems to reliably provide even standard therapeutic recommendations for first-line therapy have been disappointing. For example, Watson for Oncology, the leading application in this field, showed a concordance rate of only 73.6% compared to recommendations made by medical professionals for the first-line therapy of prostate cancer [16]. Watson for Oncology also obtained comparatively low concordance rates in other tumor entities, such as 12% in gastric cancer [13,24], 46.4% to 65.8% in colon cancer [10,11] or 77% in differentiated thyroid carcinoma [25] and others [9,26,27]. When considering the implementation of AI systems, the framework provided by the existing healthcare system must be carefully considered. The effort required to implement systems such as Watson for Oncology in hospitals is enormous. The use of AI systems requires data protection-compliant interoperability between many hospital information systems and the required data must be completely accessible in predefined files. In Germany in particular, data protection requirements of 16 federal states and the non-standardized norms for data processing and storage pose considerable challenges for developers of AI systems. In addition, clinical data continues to be frequently stored in paper files and it should not be forgotten that the exchange of diagnostic reports between clinics, pathologists, oncologists and practices is to this day commonly carried out by a fax machine. Another point of criticism of AI systems is often that the decision-making process is not easy to understand and that one has to trust almost blindly in the correctness of the machine response. Furthermore, the validation efforts that are needed when using AI systems ties up considerable human resources since only medical experts can judge if the recommendations are correct. Until these structural problems are solved, expert-curated solutions offer an alternative, as described in Nature Biotechnology in 2018 [14]. This approach was adopted by medical professionals using the DSS and is the basis of this research. In order to ensure the quality of the recommendations given by the application, a continuous comparison with tumor boards of certified cancer centers was implemented. This comparably simple and resource-saving technical solution proved to be beneficial here, enabling the large number of tumor board recommendations to be effectively compared retrospectively. For prostate carcinoma, our expert curated digital decision support system provides an optimal concordance rate with the therapeutic recommendations of a university tumor board. Yet, the very high concordance rate of our system is probably not surprising, since we evaluated predominantly first-line cases, for which guidelines generally apply. Of course, the degree of complexity increases with each tumor recurrence and additional concomitant diseases. However, this is exactly the specification of our work, which aims to reduce the workload of tumor boards by providing digitalized answers to non-complex routine cases. Despite the fact that this approach achieves better results than other methods published so far, further limitations have to be taken into account. It should be stated as a limiting factor of our work that a high concordance rate is easier to achieve when therapeutic strategies do not show a significant change over a longer period of time, as given during the time period studied, from 2014 to 2018 [27]. The increasing dynamics of diagnostic and therapeutic options in the treatment of prostate cancer thus leads to significantly more frequent and shorter testing intervals of the application, which has been certified as a medical device in the meantime, and to continuous adaptation to best-clinical-practice. By continuously comparing digital and analog recommendations, systemic deficits that lead to deviations usually become quickly apparent, thus enabling the prompt adaptation of the query logic to the dynamic development of therapeutic options. Second limiting bias, a 100% concordance rate is of little value if the recommendation quality of the reference board is not validated. This leads to the necessity of establishing decision networks in order to generate a recommendation basis that is as reliable as possible and provides the basis for the required safety of recommendations. Therefore, cross-validation of different urological cancer centers is ongoing in order to eliminate the single-center bias. Another limitation is certainly that the reference recommendations are based on the German S3 guidelines and thus the methods and results cannot be transferred to other countries uncritically. The main goal of this development is to provide reliable recommendations for standard cases in advance of the tumor board conference with the aim of allowing more time for the discussion of complex cases. It is not in the developers’ interest to achieve 100% agreement between tumor board responses and digital recommendations, as no automated system will be able to consider all the complex clinical circumstances. The agreement of 100% achieved in our analysis in stage III indicates only the simpler decision for active therapy in this risk constellation compared with the early stages of prostate cancer. Rather, a trusted DSS must be able to reliably identify complex clinical constellations, which should then be discussed by experts attending the tumor board. Particularly in the case of complex diseases, medical expertise is irreplaceable and must remain so. However, expertise requires time, and this time should not be spent discussing universally accepted standard procedures.

Conclusion In summary, the study evaluated a decision support system (DSS) for first-line therapy of prostate cancer by comparing its recommendations with those made by a multidisciplinary tumor board (MTB) at a university cancer center. The study found a high level of concordance between the DSS-generated recommendations and those of the MTB, indicating a high level of reliability. Continuous analysis of mismatched cases ensures early adjustment of DSS recommendations to account for changes in best clinical practice. Overall, our results suggest that EO is a promising tool to assist clinicians in providing reliable treatment recommendations for prostate cancer patients. Perspective options It is almost paradoxical in evidence-based driven oncology that the actually relevant quality of individual therapeutic decisions is virtually unknown. The use of intelligent software could ensure the quality of treatment on a case-by-case basis and thus serve as an instrument for quality assurance that can be transparently accessed and compare the quality of oncological care provided by hospitals and medical practices. Based on the smartphone application used in this work for recommendation matching, we developed an interface that enables the necessary inputs in the decision process of tumor boards. Easily integrated into any system, this validated and reliable application could unburden tumor boards from standard cases, thereby allowing more time for discussion of complex cases.

[END]
---
[1] Url: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000054

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/

You are viewing proxied material from magical.fish. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.