Clinical Use Case Design

The intended model aims to predict consultations with PC for patients during their hospitalization. This prediction will facilitate prompt PC availability, potentially enhancing the satisfaction of both patients and caregivers with the healthcare provided. The model will operate as a web-based service, integrated with institutional data flows, and utilized by a subsequent display application overseen by the PC team. For those patients that the PC team deems appropriate, a team member will then contact the patient's corresponding care team. Stakeholders will be engaged when designing the solution (both for buy-in and support and for end-user design) and will include hospital administration, non-PC healthcare teams, medical informatics, the information technology team, patients/caregivers, and funders. Outcomes of enhanced PC referrals would include improved symptom management and pain control, enhanced patient and family satisfaction, better alignment of medical care with patient values and preferences, reduced hospital readmissions and emergency department visits, and enhanced communication and coordination among healthcare providers.[9] Often these might lead to cost savings through reduced hospitalizations, decreased utilization of expensive interventions, improved resource allocation and healthcare efficiency, and enhanced patient and family satisfaction leading to better reputation for the hospital.[10] [11]

Design Thinking Methodology

We will start with involve the users to understand the needs and challenges of the stakeholders involved, including patients, caregivers, healthcare providers, and PC specialists via interviews, surveys, and observations to gain insight into their experiences and pain points. We will ideate solutions via brainstorming over a wide range of workable solutions to the problem, without worrying about feasibility at this stage, via mind mapping, brainstorming sessions, and SCAMPER (Substitute, Combine, Adapt, Modify, put to other uses, Eliminate, Rearrange) to generate ideas,[12] and diverse perspectives and challenges to assumptions will be encouraged at this stage. Next step involves developing a rough prototype of a web application, using wireframes, sketches, or low-fidelity mock-ups. We will perform user testing to get feedback on prototypes using techniques like A/B testing or surveys to get quantitative data on user preferences.[13] We will institute necessary changes based on the feedback to the web application and repeat the prototyping and testing process until we have a final product that meets the needs of users and achieves goals.

Data and Pre-Processing

We will get the necessary permission to collect hospital data for 1 year, for adults (>18 years), and inpatients, excluding patients in the PC unit or waiting to be transferred to an external hospice. The data category will encompass numeric, categorical, or binary formats contingent upon the nature of the variable. Instances consist of patient demographic information, past utilization (quantified by tallying recent instances of inpatient care, intensive care, and primary care interactions before admission), concurrent health conditions (ICD11 diagnosis codes), and dynamic data like lab results and the ongoing length of stay.

Data Ingestion and Preprocessing

In the existing framework of system architecture, whenever a modification is applied to EHRs, the system initiates the creation of HL7 messages. These messages are then disseminated within the organization through the Enterprise Service Bus(ESB) version.[14] The ESB is subject to ongoing surveillance by the Project Data Pipeline. This pipeline includes a rules manager designed to implement clinical enrichment rules onto the messages. The outcomes produced by this rules manager can encompass significant data required for the operation and presentation of the application. Alternatively, if necessary, it can also trigger predictive processing in situations where changes in pertinent variables for monitored patients come into play.

Model Development and Validation

We intend to create a semi-supervised learning framework by utilizing both labeled datasets and unlabeled outcomes.[15] Specifically, our focus is on developing a model that employs a Poisson distribution process to mathematically capture the concept of time-to-event outcomes, particularly in the context of PC consultations.[16] These consultations can be understood as events occurring at varying rates over time. Our approach involves utilizing a classification task with a decision tree algorithm.

To construct the response variable, we will measure the time from admission to a PC consult, taking into account that this measurement is right-censored at the discharge time. The Poisson process will enable us to predict the rate of PC interventions per unit of time. For practical implementation in a clinical setting, we will transform this rate into a 7-day probability of PC consultation. While various ML models can be employed that can leverage Poisson likelihood, our choice is the Gradient Boosting Machine (GBM). The GBM's ability to incorporate a loss function equivalent to the minus log-likelihood for Poisson distributions makes it well-suited for our purpose.

Furthermore, the GBM can effectively handle missing values, which is particularly important considering the presence of missing data in laboratory values and diagnosis codes. Our model will be dynamic, updating predictions as new information becomes available through a carry-it-forward dataset construction approach. However, it is important to acknowledge that assuming constant predictor values between observations might introduce bias or errors.

Upon exceeding a predefined threshold, the model will trigger notifications to the PC team. The threshold will categorize the 7-day probability of PC consultation as “low,” “medium,” or “high.” The “high” category will be calibrated based on the existing capacity of the PC service, aiming to align with the average of around ten new consults daily, including those via the traditional pathway. This calibration will involve utilizing data to establish a threshold for the receiver operating characteristic (ROC) curve.

The labeled dataset will be divided into 80% for training and 20% for testing. We will determine hyperparameters for the GBM model—such as the number of trees, shrinkage, and interaction depth—through cross-validation. This process entails partitioning the dataset into random subsets, using one as the test set while training on the others. Repeating this process for each subset, the average results will guide our final choice of hyperparameters. We will employ 5- to 10-folds, depending on our computational resources. The optimal hyperparameters will be those producing the highest area under the curve (AUC) in the time-dependent ROC curve. This curve will plot AUC and positive predictive value.

Ultimately, the final model will be fitted using all available data and evaluated in real-time using unlabeled data. Its performance will be compared to the cross-validation data, allowing us to assess the AUC and make any necessary adjustments to the model.

User Validation and Clinical Integration

Validation metrics will include the number of PC consults triggered by the ML model, compared with previous years' data, and changes in clinical outcomes guided by the new ML model. End users will be internal stakeholders, to begin with, or external customers if successfully implemented. We will gather feedback from users within the time frame for testing and analyze patterns in the feedback and identify areas where the model can be improved. We will make improvements like adjusting the model parameters, collecting additional data, or making changes to the user interface. We will repeat the process until satisfied with the performance of the ML model.

We will identify key stakeholders who will be impacted by the introduction of the ML model, such as clinicians, administrators, patients, and IT staff. Engage with them early in the process to understand their needs and concerns and involve them in the decision-making process. We will clearly articulate the benefits of the ML model to stakeholders like increasing efficiency, reducing costs, and enhancing the quality of care. We will anticipate and address concerns and potential risks associated with the introduction of the ML model. Communicate clearly about the safeguards and measures in place to protect patient privacy and data security. Provide evidence-based information to address concerns about the accuracy and reliability of the model. We will provide adequate training and support to stakeholders to ensure they are comfortable with the technology and understand how to use it effectively. Offer ongoing support to address any issues or questions that may arise. We will monitor and evaluate the implementation of the ML model to ensure it is meeting the desired outcomes and identify areas for improvement. Share the results with stakeholders to demonstrate the impact of technology and foster ongoing support and engagement.

Ethics, Legal, and Regulatory Considerations

Transparency: It is important to ensure that the ML application is transparent in how it makes its decisions. Patients and healthcare professionals need to be able to understand how the algorithm arrived at its recommendations, so they can trust and act upon them. This holds significant significance in the context of end-of-life choices. To illustrate, comprehending and explaining the course that a decision tree follows to arrive at its conclusion is straightforward, whereas tracing the trajectories of numerous trees, numbering in the hundreds or thousands, becomes notably more challenging.

Autonomy: The ML application should not replace the judgment of healthcare professionals. It should be designed to support clinical decision-making, rather than replace it. It is up to the healthcare professional to make decisions about the care of their patients.

Accountability: It is important to ensure that the ML application is accountable for its decisions. This means that there should be mechanisms in place to monitor its performance and identify any errors or biases that may arise.

Informed consent: Patients should be fully informed about the use of ML in their care and should have the option to opt out if they choose. This requires clear communication about how the ML application works and the implications of its recommendations.

Privacy: The use of personal data is essential for ML algorithms to work, which can potentially reveal sensitive medical information about patients. Therefore, it is important to ensure that patient privacy is protected throughout the development, deployment, and use of the ML application.

Discussion and Conclusion

This concept article aimed to explore a project utilizing the potential of ML-based predictive modelling within EHRs to enhance the referral process for PC in hospitalized patients, allowing for timely and appropriate referrals. The accuracy and performance metrics achieved by these models will suggest if they have the potential to significantly improve the current referral process. Proactive identification of patients who may benefit from PC can initiate early conversations and interventions, leading to better patient outcomes and enhanced quality of life.

One of the key advantages of using ML in this context would be the ability to leverage a wide range of patient data, including demographics, clinical variables, laboratory results, and diagnostic codes. These models can detect patterns and associations that may not be apparent to human clinicians, enabling more accurate predictions. Furthermore, the models can continuously learn and adapt from new data, refining their predictions over time and improving their performance.

However, several challenges and limitations need to be addressed before implementing ML-based predictive modelling for PC referrals in electronic health systems.

Primarily, ensuring data quality and standardization across different healthcare settings is crucial. Discrepancies, absent information, and mistakes within EHRs can result in prejudiced or erroneous forecasts, posing a potential risk to patient well-being. To illustrate, the training of the model will utilize a historical cohort of primary care consultations that is not flawless; specifically, numerous individuals who could have gained from a primary care consultation might not have undergone one. This disparity has the potential to cause oversights by the algorithm, especially if there exists a consistent bias towards primary care consultations under the existing protocols. Mitigation strategies may include effort to get better representative data, using metrics like precision, recall, F1 score, adjust class weights, fairness constraints in model training, regular model audit, and monitoring. Therefore, data preprocessing and quality assurance processes must be robustly implemented.

An additional obstacle pertains to the comprehensibility of ML models. Despite their ability to attain remarkable predictive precision, these models frequently lack transparency in elucidating the fundamental decision-making process. This absence of clarity could impede the establishment of trust and approval from healthcare experts. To illustrate, consider the presumption of carrying forward data for time-dependent predictors, where the assumption that a patient's hemoglobin level remains constant for 3 days solely because a laboratory test was conducted on a Monday and not repeated until Thursday does not hold true. The advancement of interpretable ML techniques, capable of offering insights into the drivers behind predictions, is a domain that demands further attention.

Additionally, ethical considerations must be carefully addressed when implementing ML models in healthcare. Patient privacy, data security, and informed consent are of utmost importance. Safeguards must be in place to protect sensitive patient information and ensure compliance with privacy regulations. It is crucial to strike a balance between the potential benefits of predictive modelling and the ethical responsibilities associated with its use. Should our solution become part of a clinical workflow, obtaining approval from the Food and Drug Administration (FDA) could be necessary. For instance, the US FDA categorizes medical devices, including software, into four different classes, depending on the level of risk they pose to both the patient and the user.[17]

Despite these challenges, the potential impact of ML-based predictive modeling in enhancing PC referrals within EHRs is significant. Future research should focus on prospective validation of the models in real-world clinical settings to assess their performance in diverse patient populations. Additionally, studies should explore the impact of implementing these models on clinical workflow, patient outcomes, and healthcare resource utilization.

Furthermore, collaboration between clinicians, data scientists, and policymakers is essential for successful implementation. Engaging healthcare providers in the development and validation of these models can help address concerns, foster trust, and ensure that the technology aligns with clinical needs and workflows. Moreover, policymakers should establish guidelines and regulations that promote the responsible and ethical use of ML in healthcare.

In conclusion, the use of ML-driven predictive modeling shows significant potential in improving the process of referring patients for PC during their hospitalization. Detecting individuals who could gain from PC at an earlier point can result in better patient results and more effective utilization of resources. Nevertheless, it is crucial to meticulously address concerns related to data accuracy, comprehensibility, and ethical implications. By tackling these obstacles and delving deeper into the possible advantages, we can make strides in incorporating ML into EHRs, thereby enhancing the provision of PC for those requiring it most.

Conflicts of Interest

None declared.

Acknowledgments

Artificial intelligence for Clinician Champions Certificate Program, The Michener Institute of Education at UHN.

Patient Consent

None Declared.

Disclosure of Funding Support

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References