Eligible patients for this study were those who participated in our previous study [3] of adult patients visiting the ED between February 4, 2015, and March 13, 2017; had their blood samples genotyped through their participation in the Mass General Brigham (MGB) Biobank [5] (88% self-reported White); and had nonmissing information on suicide attempt(s) within 6 months following their ED discharge. In total, 333 patients with genetically identified European ancestry met the eligibility criteria and demonstrated a suicide attempt prevalence of 8.4% (n=28) at the 6-month follow-up (n=178, 53.5% self-reported male and n=324, 97.3% self-reported White). Although our previous study [3] also examined suicide attempts at 1 month after ED discharge, the event rate within this window was too low to permit stable estimates. The study sample differed significantly from the original cohort [3] by age (P<.001), self-reported race (P<.001) and ethnicity (P=.06), insurance type (P=.001), and patterns of health care utilization (P<.001; see Multimedia Appendix 1 [3]). Details on recruitment, informed consent process, and data collection can be found in Boutin et al [5] (for the MGB Biobank study) and Nock et al [3] (for the suicide prediction study).

The primary outcome was any suicide attempt within 6 months of the ED visit based on either follow-up surveys or a review of linked EHRs [3]. For the latter, we used the International Classification of Diseases, Ninth Revision (ICD-9) and International Classification of Diseases, Tenth Revision (ICD-10) to identify qualifying diagnostic codes for suicide attempts that we previously validated [6,7].

We extracted the predicted probabilities from the best-performing ensemble model from our previous work [3] for 6-month suicide attempts. This model incorporated patient-reported surveys, a previously developed EHR-based suicide risk score, and sociodemographic characteristics (eg, age, sex, income, education, race and ethnicity, and employment status). In addition, we generated PRSs for depression, bipolar disorder, schizophrenia, suicide attempt, and externalizing traits derived from the largest available European ancestry genome-wide association study of these phenotypes using a Bayesian polygenic risk scoring method called “PRS-CS” (see Multimedia Appendices 2 and 3) [8]. We subsequently residualized individual disorder PRSs for biological sex, age, genomic chip, and the top 20 principal components for population stratification to adjust for potential confounding.

We first established the baseline model by fitting our previously validated suicide risk score and calculated patient risk stratification accuracy (measured using the area under the receiver operating characteristic curve [AUC], area under the precision-recall curve [AUPRC], and positive predictive value [PPV]). We then added each PRS to the baseline model to evaluate whether adding individual disorder PRSs would improve the AUC, AUPRC, or PPV. Lastly, we incorporated all 5 PRSs to examine whether incorporating multiple neuropsychiatric PRSs would increase the predetermined metrics more than adding individual disorder PRSs to the baseline model alone.

In addition to fitting logistic regression models, we used the SuperLearner stacked generalization approach that combines predictions across a range of algorithms, including those that can capture nonlinear relationships (see Multimedia Appendix 4) [9]. We used 10-fold stratified cross-validation in a 70% training sample (n_train=235) to develop the models and evaluated the models in a 30% holdout sample (n_holdout=98). There were no significant differences in sample characteristics and feature distributions between the train and holdout samples (all P>.05; see Multimedia Appendix 5). All statistical analyses were conducted using R software (version 4.1.2; R Foundation for Statistical Computing).

The study procedures were approved by the Institutional Review Boards of Harvard University and MGB (protocol code 2010P000246, approved on February 18, 2010). Additionally, the MGB Biobank study was conducted in accordance with the Declaration of Helsinki and approved by the MGB Institutional Review Board (protocol code 2009P002312, approved on April 29, 2010), with no compensation provided to participants. This study involves secondary analyses using de-identified data from the original studies, which is covered under the initial consent and IRB approval, without requiring additional consent.

The baseline model for 6-month suicide attempts had an AUC of 0.84 (95% CI 0.70-0.98; see Figure 1 and Multimedia Appendix 6). Models that included individual disorder PRSs alone had modest or poor AUC, with the schizophrenia PRS having the highest AUC (0.58, 95% CI 0.41-0.76), followed by the bipolar disorder PRS (0.56, 95% CI 0.39-0.73). When individual disorder PRSs were added to the baseline model, the logistic regression and the ensemble models that included the schizophrenia PRS and clinical risk score had the highest AUC (0.86, 95% CI 0.73-0.99), followed by ensemble models each including the suicide PRS and externalizing disorder PRS, but these provided only a modest numerical increase in AUC compared to the baseline model alone (see Figure 1). In general, there was no improvement in AUC when adding the PRS for depression or bipolar disorder to the clinical risk score. However, we observed a numerically higher AUC when the depression PRS was incorporated using an ensemble approach than using logistic regression. The ensemble model that included the clinical risk score and all 5 PRSs had the same AUC (0.86, 95% CI 0.72-0.99) as the ensemble model including the schizophrenia PRS and clinical risk score and had nearly the same AUC as the logistic regression including the same set of features.

We examined precision-recall curves to see how PPV varies across levels of sensitivity with the goal of explaining the best-performing model, which included the clinical risk score and schizophrenia PRS (see Figure 2). All models that included the clinical risk score were comparable in identifying 40% to 50% of suicide attempt cases within 6 months after ED discharge, indicating a higher sensitivity than the models only including individual disorder PRSs (see Multimedia Appendix 7). Specifically, shown in Figure 2, the baseline model had a higher PPV (26%-50%) than the other models when the sensitivity was in the 0.05 to 0.35 range. The models including the clinical risk score with or without PRSs had the same PPV (13%-26%) when the sensitivity was in the 0.4 to 1.0 range, and the model with the schizophrenia PRS alone had a lower PPV (12%-18%). AUPRC was 0.42 for the baseline model but reached 0.45 when the schizophrenia PRS was added, which is consistent with the observed improvement in AUC with the same model configuration.

We found modest evidence suggesting that the integration of the PRS for schizophrenia (but the PRSs for not the other related phenotypes) might enhance the prediction of short-term risk for suicide attempt in patients discharged from the ED; both the AUC and AUPRC were numerically, although not significantly, higher when the schizophrenia PRS was added to the baseline clinical model. The improved predictive performance is likely explained by the higher heritability and statistical power of the schizophrenia PRS compared to the other PRSs examined in this study (see Multimedia Appendix 8). However, while heritability provides a compelling explanation, it does not fully account for the schizophrenia findings, as the predictive power of PRSs is also influenced by factors such as genetic architecture and heterogeneity in phenotype ascertainment. Furthermore, given the high dimensionality of the phenotypic features in the suicide prediction model, the addition of 1 or more PRSs is expected to have only a modest effect on prediction accuracy.

Nevertheless, the nonsignificant improvement in performance we observed should be interpreted in light of our limited study sample size and statistical power of neuropsychiatric PRSs. Of the PRSs we examined, only the schizophrenia PRS was well powered (88%) to detect an association with suicide attempt in the holdout sample.

Future studies utilizing larger biobank samples will enable a more robust and well-powered evaluation of the potential utility of PRSs in enhancing patient risk stratification in high-risk clinical settings. For instance, larger samples could facilitate the training of separate, context-specific baseline models using EHR and survey data from patients with schizophrenia or bipolar disorder, followed by the integration of the respective PRSs into each model. Such an approach would provide a more nuanced understanding of the clinical relevance of PRSs and their potential role in improving risk stratification and patient outcomes.

In conclusion, we did not observe a substantial benefit of adding psychiatric PRSs to EHR and survey-based prediction models of suicide attempt in an ED setting. Given the importance of optimizing risk stratification to inform suicide prevention, further studies in large, diverse samples are warranted to clarify the value of incorporating genomic risk factors.