You are here

Article Text

Article menu

Download PDFPDF

XML
Systemic sclerosis
Original research
New risk model is able to identify patients with a low risk of progression in systemic sclerosis
  1. Nina Marijn van Leeuwen1,
  2. Marc Maurits1,
  3. Sophie Liem1,
  4. Jacopo Ciaffi2,
  5. Nina Ajmone Marsan3,
  6. Maarten Ninaber4,
  7. Cornelia Allaart1,
  8. Henrike Gillet van Dongen5,6,
  9. Robbert Goekoop7,
  10. Tom Huizinga1,
  11. Rachel Knevel8 and
  12. Jeska De Vries-Bouwstra1
  1. 1Department of Rheumatology, Leiden University Medical Center, Leiden, The Netherlands
  2. 2Medicine and Rheumatology Unit, IRCCS Istituto Ortopedico Rizzoli, Bologna, Emilia-Romagna, Italy
  3. 3Cardiology, Leiden University Medical Center, Leiden, Netherlands
  4. 4Pulmonology, Leiden University Medical Center, Leiden, The Netherlands
  5. 5Department of Rheumatology, Medisch Centrum Haaglanden, Den Haag, Zuid-Holland, The Netherlands
  6. 6LangeLand Hospital, Zoetermeer, Zuid-Holland, The Netherlands
  7. 7Rheumatology, HagaZiekenhuis, The Hague, Zuid-Holland, The Netherlands
  8. 8Rheumatology, Brigham and Women's Hospital, Boston, Massachusetts, USA
  1. Correspondence to Nina Marijn van Leeuwen; n.m.van_leeuwen{at}lumc.nl

Abstract

Objectives To develop a prediction model to guide annual assessment of systemic sclerosis (SSc) patients tailored in accordance to disease activity.

Methods A machine learning approach was used to develop a model that can identify patients without disease progression. SSc patients included in the prospective Leiden SSc cohort and fulfilling the ACR/EULAR 2013 criteria were included. Disease progression was defined as progression in ≥1 organ system, and/or start of immunosuppression or death. Using elastic-net-regularisation, and including 90 independent clinical variables (100% complete), we trained the model on 75% and validated it on 25% of the patients, optimising on negative predictive value (NPV) to minimise the likelihood of missing progression. Probability cutoffs were identified for low and high risk for disease progression by expert assessment.

Results Of the 492 SSc patients (follow-up range: 2–10 years), disease progression during follow-up was observed in 52% (median time 4.9 years). Performance of the model in the test set showed an AUC-ROC of 0.66. Probability score cutoffs were defined: low risk for disease progression (<0.197, NPV:1.0; 29% of patients), intermediate risk (0.197–0.223, NPV:0.82; 27%) and high risk (>0.223, NPV:0.78; 44%). The relevant variables for the model were: previous use of cyclophosphamide or corticosteroids, start with immunosuppressive drugs, previous gastrointestinal progression, previous cardiovascular event, pulmonary arterial hypertension, modified Rodnan Skin Score, creatine kinase and diffusing capacity for carbon monoxide.

Conclusion Our machine-learning-assisted model for progression enabled us to classify 29% of SSc patients as ‘low risk’. In this group, annual assessment programmes could be less extensive than indicated by international guidelines.

  • scleroderma
  • systemic
  • outcome assessment
  • health care
  • autoimmunity

Data availability statement

Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as online supplemental information.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/rmdopen-2020-001524

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Key messages

    What is already known about this subject?

    • Systemic sclerosis (SSc) is a severe autoimmune disease characterised by large clinical heterogeneity ranging from rapidly progressive to mild and indolent for a longer period of time.

    • Currently, evidence-based guidelines on the frequency and extent of annual assessment are lacking.

    What does this study add?

    • In our monocentric, high-quality, prospective SSc cohort 48% of the patients did not show any sign of disease progression during 5 years of follow-up.

    • With the use of machine learning methods, a prediction model was developed that can identify SSc patients with mild and stable disease.

    How might this impact on clinical practice or further developments?

    • Implementation of the prediction model in clinical practice enables tailormade annual assessment of SSc patients and diminishes burden of extensive evaluation in SSc patients with mild and stable disease.

    Introduction

    Systemic sclerosis (SSc) is a heterogeneous disease. The spectrum of the disease ranges from rapidly progressive, with generalised fibrosis of the skin and the vital organs to a more indolent form developing over an extended period of time.1 The amount of patients with progression of SSc is substantial and progression occurs most often in early disease.2 3 It is important to note that around 50% of patients will never show any signs of progression. To accurately assess the trajectory of the disease, several studies addressed identification of risk factors of future skin and organ progression in different SSc subpopulations.4 5 Existing prediction models in SSc are often based on a subset of SSc patients, and do not capture the whole population.2 6 Prediction of the disease course remains challenging in the individual patient which raises the questions whether personalised prediction in the heterogeneous SSc population is actually feasible.

    For physicians in clinical practice, it is important to have clear guidance regarding intensity and frequency of follow-up, not only to identify disease progression timely but also to limit excessive diagnostics in mild SSc patients. Currently, no evidence based international guidelines for follow-up of SSc exist, except for the European Society of cardiologie/ European Respiratory Society (ESC/ERS) guideline recommending annual echocardiography for detection of pulmonary arterial hypertension (PAH).7 In 2019, an international standard for longitudinal follow-up describing points to address in annual assessment of patients with SSc was developed based on Delphi-expert consensus. Overall, 55 items were identified including clinical assessments, laboratory measurements, imaging and functional investigations.8 Whether the identified items are sufficient to identify disease progression timely in all patients is yet to be determined. Moreover, in some patients with mild disease, annual follow-up might even be more concise and assessing 55 tools on an annual basis might not be necessary. Of note, previous prediction studies concercing prevalent SSc cohorts might have underestimated progression in SSc by failing to capture the early rapid progressors.9 On the other hand, with the introduction of the ACR/EULAR 201310 criteria additional cases with less severe disease might be identified.10 Together, these observations provide the rationale for the design of data-driven recommendations that describe tailormade systematic assessments for individual SSc patients in line with their individual disease course.

    Our prospective SSc cohort includes both mild and severe patients who undergo annual assessment, as the healthcare system in The Netherlands is characterised by high accessibility. Starting from 2009 all patients fulfilling Leroy criteria for early SSc have been included.11 12 In the current study, we included detailed information on disease progression in our prospective cohort and we addressed an important limitation that is often encountered when searching for predictive factors in large datasets from SSc patient registries: the high incidence of missing data. Therefore, in our prediction model, we only included patients with complete data available on at least three visits.

    With this study, we aim to develop a tailormade model to guide annual assessment in individual SSc patients, with a special focus on patients with a low risk of disease progression in whom annual extensive investigations may be considered redundant. To address this we: (1) determined the proportion of patients without disease progression, (2) applied machine learning to build a prediction model in the patients with complete data available at ≥3 time points to predict lack of progression. Additionally, (3) we evaluated a second prediction model including the variables from the Delphi consensus guideline and compared the performance of this model to the machine-learning-assisted model in order to assess which investigations are minimally needed to identify patients with a low risk of disease progression.

    Method

    Patient selection

    In the Leiden University Medical Center, all SSc patients, with a range of disease severity from mild to very severe, undergo annual extensive screening during a 1–2 day healthcare programme (combined care in systemic sclerosis (CCISS)). This includes a detailed physical examination, modified Rodnan skin score (mRSS) assessment,13 laboratory testing (with autoantibody screening at baseline), ECG, pulmonary function test (PFT), optional echocardiography (mandatory at baseline visit), optional 24-hour Holter ECG monitoring (mandatory at baseline visit), optional cardiopulmonary exercise tests and optional high-resolution CT (HRCT) (mandatory at baseline visit). Patients are requested to complete various questionnaires at every visit.14–22 Additionally, at every visit, blood and serum samples are collected and stored in the Leiden Scleroderma Biobank.

    For the first part of the study (ie, numbers of disease progressors), SSc patients who fulfilled the (1) ACR 1980 and/or LeRoy (from 2009 to 2013) criteria or the (2) ACR/EULAR 2013 and/or Leroy criteria and had at least two assessments were included.12 23 24 For the second part of the study, including the machine-learning-assisted prediction model, we analysed SSc patients who fulfilled the ACR/EULAR 2013 criteria and had at least three complete visits (a third visit was necessary for our primary outcome). This ensured that included patients had complete data available of at least three time points. A complete visit consisted of at least a physical examination (including mRSS), laboratory testing, a PFT, ECG, HRCT and a transthoracic echocardiography. The strict inclusion criteria were necessary to limit the amount of missing data on important organ systems and only patients who underwent complete screening of organ systems were included in the model in order to minimalise the likelihood of missing any important organ involvement.

    Outcomes

    Disease progression was defined as progression in one or more organ systems; pulmonary, cardiac, gastrointestinal (GIT), skin, renal and/or myositis (online supplemental table S1 for detailed explanation). For pulmonary, PAH, skin and renal crisis, progression was defined as described previously.2 10 25–27 Cardiac progression, gastrointestinal progression and myositis were each defined using a combination of variables and based on consensus among authors. Use of immunomodulatory medication was recorded at every visit and included: cyclophosphamide, methotrexate (MTX), myocophenolate mofetil (MMF), azathioprine, corticosteroids, hydroxychloroquine (HCQ) and stem cell transplantation. Patients included in clinical trials (Resolve (lenabasum), Senscis (nintedanib), FocuSSced (Tocilizumab), autologous stemcell transplantion (ASTIS), rituximab, ASTIS (stem cell transplantation) were also captured. Use of biologicals outside of trials was observed in <0.5% of the patients, therefore these were not depicted separately but were included in the primary outcome.28–32 The primary endpoint in the prediction model was defined as progression in ≥1 organ system, and/or start of immunosuppression (IS) or death between the two most recent visits.

    Predictors

    The included predictors in the Machine-Learning-Assisted model were selected based on the predictors identified by experts,8 additional predictors were selected based on clinical expertise and current literature. In order to prevent exclusion of too many patients due to missingness, we dropped four variables (out of 94) with a missingness percentage >5% (nailfold videocapillaroscopy, of which annual collection started in 2013, and three variables derived from the UCLA GIT questionnaire, namely faecal soiling, diarrhoea and distension/bloating, of which annual collection started in 2013). This resulted in 90 independent variables (100% complete in n=248 patients) to predict progression at the final assessment. The 90 variables included in our model—all 100% complete—are described in online supplemental table S2. A timeline of the study can be found in online supplemental figure S1.

    Statistics

    For the first part of the study, we used descriptive statistics to evaluate the number of disease progressors in SSc during 10 years of follow-up. For the second part of the study, the development of the prediction model, we applied a machine learning approach known as ‘elastic net regularisation’. The elastic net performs simultaneous regularisation and variable selection in order to reduce variance with minimal risk of bias.33 Independent variables were all the variables collected during follow-up visits (predictors) until the prediction visit (event visit=primary outcome). Disease progression on any organ system during follow-up was also included as independent variable in different manners: (1) progression between third to event visit and the prediction visit (dichotomous), (2) progression developed before the prediction visit (dichotomous), (3) the amount of times progression occurred between baseline visit and the prediction visit (quantitative) and (4) in how many organ systems progression occurred over time (quantitative). Including progression as an independent variable mimics the decision making of the physician in clinical care, where decisions regarding follow-up are made based on previous information (including progression occurring years before the current visit). Given the extensive amount of information from previous visits we examined whether these data could predict the development of progression at the final (event) visit. The dependent variable in the model was defined as progression in ≥1 organ system, and/or start of IS at the last recorded visit, or death after a complete baseline visit. All patients without any progression during follow-up were identified as ‘non-progressors’. In order to preserve the maximum number of patients, we filtered variables on 95% call rate prior to deleting incomplete cases, variables with more >5% missingness were checked in the Eletronic Patient Dossier (EPD) and in case of true missingness deleted from the dataset. To develop the model using leave-one-out cross validation and independently validate the final model’s performance, the included patients were randomly split in a training (75%) and a test (25%) set. The model was developed and optimised on the larger training set and subsequently applied to the test set with the lambda, alpha and coefficients set to the identified optima. The chosen predictive variables were entirely based on the training data. Risk probability scores and Area Under Curve-Receiver Operator Characteristics (AUC-ROC) were created (based on test data) to identify optimal cutoffs for the risk on disease progression.

    Due to the detrimental impact of undertreatment in SSc, we opted to maximise negative predictive value (NPV) with a constraint on sensitivity, which minimises the likelihood of missing progression. However, a benefit of the probabilistic nature of a prediction model is the flexibility of the case cut-off point; by sliding across the ROC curve one can choose to prioritise any preferred performance metric. Cutoffs for risk probability scores were defined based on the test characteristics (maximise NPV) and the distribution of probabilities plots in the test set. After cut-off selection, we ran a post hoc analysis to assess the missed progressors.

    Lastly, using logistic regression we built a prediction model including 51 (out of the 55) variables from the Delphi consensus guideline.8 We had to exclude leg oedema, urine analysis, liver function test and New York Heart Association class due to missingness in >5%. The performance, and the risk probability scores of the model including the Delphi variables as predictors were compared with the model derived from machine learning based on the AUC-ROC curve and the probability plots using descriptive analyses.

    All analyses were performed in R V.3.5.0. The ‘glmnet’ package was used for elastic net regularisation and leave one out cross-validation was implemented through the ‘caret’ package.

    Results

    Patient population

    For the first part of the study, we included 492 SSc patients who completed at least two visits in our cohort (figure 1). Seventy-nine per cent was female (n=389) with a mean age of 55 years (SD 14), a median disease duration since first non-Raynaud’s symptom of 3.2 years years (IQR 1–10) and the median mRSS was 4 (IQR 0–6) (table 1).

    View this table:
    Table 1

    Baseline characteristics of the included patients separated in non-progressors and progressors

    Figure 1
    Figure 1

    Flow chart description: flow charts of inclusion process. Cut-off time point for inclusion: 1 July 2019. *92 patients had to be excluded due to missing data/incomplete data eventhough they had three or more visits. Of these, the majority did not show progression based on clinical and laboratory assessment, 6 min walking distance and pulmonary function testing. SSc, systemic sclerosis.

    Progressors versus non-progressors in SSc cohort

    In n=492 SSc patients (2109 time points, range of follow-up 2–8 years), disease progression during the follow-up was observed in 52% (n=257) after a median of 4.9 years (IQR 2–7) (figure 2). Pulmonary (23%) and cardiac progression (29%) occurred most often, death (all-cause) occurred in 12% of the patients (n=60). We confirm that patients with dcSSc, interstitial lung disease (ILD) and Anti-topoisomerase antibody (ATA) at baseline were more likely to experience disease progression somewhere during the disease course (table 1). Forty-eight per cent of the SSc patients (n=235) did not show progression during follow-up (median 3.5 years (IQR 2–6).

    Figure 2
    Figure 2

    Progressors in SSc cohort description: organ progression in SSc cohort, progression was not always limited to one organ domain. Ttwenty-five per cent of the patients showed organ progression on more than one organ domain. SSc, systemic sclerosis.

    Patient selection machine-learning-assisted model

    Of the 248 patients that could be included for development of the predition model, 80% was female (n=220) with a mean age of 53 years (SD 14), a median disease duration since first non-Raynaud sign or symptom of 3.5 years (IQR 1–9) and median mRSS of 4 (IQR 1–6). The baseline characteristics of these patients are shown in online supplemental table S3. Comparison of baseline demographic and clinical characteristics between these patients and the 492 patients with two assessments available showed that the patients included for the prediction model development were more often ATA positive and had higher prevalence of ILD. Other characteristics were not significantly different between the groups (online supplemental table S4).

    Machine-learning-assisted prediction model

    After leave-one out cross validation, the final model consisted of 90 variables. The machine-learning-assisted model identified 10 independent variables predictive for disease progression (online supplemental table S5). The identified predictors were: previous use of cyclophosphamide (β 0.94) or corticosteroids (β 0.43), previous gastrointestinal (GI) progression (β 0.34), a cardiac event in medical history (β 0.31), PAH (β 0.30), start of immunosuppressives (β 0.21), previous cardiac progression (β 0.08), mRSS (β 0.01), CK (β 0.0006) and DLCO (β −0.004).

    The machine-learning-assisted model had an AUC-ROC of 0.77 in the training set (n=185). The mean (SD) probability score for risk of progression in non-progressors was 0.23 (0.05), and in progressors 0.31 (0.11) (online supplemental figure S2). The AUC-ROC of the model in the validation set (n=63) was 0.66 (figure 3 ROC curve and distribution of probability plot). In this set, the mean (SD) probability score for risk of progression in non-progressors was 0.24 (0.06) and in the progressors 0.29 (0.13). Based on expert opinion, the distribution of probabilities plot and the test characteristics of the validation set (maximise NPV), we identified two cut-offs to identify patients with low (<0.197), intermediate (0.197–0.223) and high risk (>0.223) of progression (table 2). The cut-off <0.197 gives a perfect sensitivity but poor specificity in detecting patients at low risk. This means that there are no patients with progression in the group with a low risk score. However, due to the low cut-off, there will be a number of patients who will be falsely classified as intermediate/high risk patients. A third threshold (>0.627) corresponding to maximal specificity to find progressors is also presented (table 2).

    View this table:
    Table 2

    Test characteristics of data-driven prediction model

    Figure 3
    Figure 3

    ROC curve and distribution of probability plot. ROC curve and distribution of probability plot of the validation set in progressors and non-progressors.

    Disease progression and probability scores in machine-learning-assisted model prediction model

    Our primary outcome was progression at the event visit, which occurred in 60 out of 248 patients (24%). Progression was identified in all subdomains: disease subset (n=3), skin (n=4), lung (n=14), cardiac (n=28), GI (n=15), renal (n=2), PAH (n=4), myositis (n=6), start of IS therapy (n=6), and all-cause death (n=11; detailed overview of cause of death is shown in online supplemental file S1). In the validation set (n=63), 22% (n=14) showed progression during the event visit, while 78% (n=49) did not show progression. With guidance of the Machine-learning assisted model prediction model 28 patients were identified as high risk for progression which was correct in 32% (n=9); 18 patients were identified as low risk which was correct in 100% (due to our strict cut-off), which means that 29% of the patients in the validation set (18 out of 63) had a low-risk score and indeed did not show progression. In summary, 29% of the patients in our population can be identified as low risk and in these annual organ assessment could be less extensive. Of the patients with intermediate risk according to our model (n=17), five showed progression.

    Progressors stratified for treatment initiation

    To evaluate the clinical relevance of the probability scores we performed an additional analyses in the organ progressors group (patients with organ progression from the validation set) by stratifying them for immunosuppressive treatment initiation after the data collection closure. We handsearched the electronic patient files of the organ progressors to collect data on IS treatment initiation (started after data collection closure 01.07.2019). Our results showed that patients with organ progression at the most recent visit (primary outcome), for which medication was started, were more likely to score higher on the probability risk score. There was one patient with lung progession who also started treatment with a risk score just above the cut-off for low risk (figure 4). In the non-progressors with a low-risk score (n=18), we identified n=8 patients who never had any IS treatment during their disease course, n=9 did use IS medication somewhere during follow-up (HCQ: n=3 due to artralgia, polyartritis, or synovitis, MTX: n=6 due to limited skin involvement), and 1 patient is still on MMF treatment because of minimal skin (mRSS 2) and lung involvement (minimal interstitial changes on HRCT, without pulmonary function abnormalities).

    Figure 4
    Figure 4

    Probability risk scores of the progressors stratified for treatment initiation and organ domain. Patients with cardiac progression and treatment (n=3): 1 trifascicular block with pauses >3 s for which pacemaker implantation, severe tricuspid insufficiency, 2 new right bundle branch block, decrease in LVEF <50%, increase dyspnoea,3 clinical cardiac involvement; supraventriculair arrhythmias 2%, diastolic dysfunction grade 1, elevated troponin T and CK, progressive dyspnoea). Patients with cardiac progression without treatment (n=2): 1 LVEF <54%, 2 suptraventricular arrhythmias >2% on 24 hours Holter ECG monitoring. ILD progression with treatment (n=3): 1 mild fibrotic changes with a decrease in FVC (73% to 58%) and in DLCO (97%–76%), 2 increase in fibrotic changes, decline FVC (52%–42%) and decline in DLCO (48%–28%), 3 progressive ILD and decline in FVC and DLCO (n=3). ILD progression without treatment (n=1): 1 presence of ILD with bronchiectasis, honeycombing and an increase in reticular opacities, no clinical symptoms, with FVC decline (101%–90%). Skin progression with treatment (n=2): 1 mRSS increase from 10 to 17, 2 increase mRSS 10 to 23. Gastrointestinal progression with treatment (n=1): 1 weight loss >10% in 1 year and Hb decline. One patient developed renal crisis, one patient died due to lung carcinoma (also had supraventriculair extrasystoles >2 s and in increase in fibrotic changes on HRCT). DLCO, diffusing capacity of the lungs for carbon monoxide; FVC, forced vital capacity; HRCT, high-resolution CT; ILD, interstitial lung disease; mRSS, modified Rodnan skin score.

    Delphi score versus machine-learning-assisted model

    The most recently published guideline on follow-up in SSc is based on expert opinion by Delphi consensus which advises to yearly measure 55 variables. Based on these 55 independent variables, we built a prediction model (Delphi Model) in order to assess the ability of the Delphi items to identify patients at risk for progression, and compare the performance of the ‘Delphi model’ with the model derived from machine learning. The AUC-ROC of the Delphi model in the validation set was 0.65 (figure 5 ROC curve and distribution of probabilities plot). The mean (SD) probability score for the risk of progression in non-progressors was 0.14 (0.30), and in the progressors 0.42 (0.46). Of the 54 patients in this validation set, 13 developed progression, of whom six patients had a probability risk score below 0.007. The results of the Delphi model, shown in the distribution of probabilities of the validation set, made it difficult to identify cut-offs for low, intermediate and high-risk patients on progression. Therefore, a cut-off based on an NPV of 1.0, as we used in the machine-learning-assisted model, is not feasible in this model.

    Figure 5
    Figure 5

    ROC curve and distribution of probabilities plot of the Delphi model. ROC curve and distribution of probabilities plot of the Delphi model stratified for progression.

    The coefficients that were significant in the final model of this prediction set can be found in online supplemental table S6. The included predictors with the largest coefficients were: previous use of corticosteroids (β 6.66), previous use of iloprost (β 15.1), previous use of bosentan (β −18.0), current use of MMF (β 5.98) or cardiac event in the past (β 5.39).

    Discussion

    Our newly developed prediction model was able to identify SSc patients with a low risk for disease progression in whom less extensive annual evaluation can be justified. We confirm that SSc is a severe and heterogeneous disease with overall progression occurring in 52% of the patients somewhere during follow-up. In total, 235 patients did not experience disease progression during 3.5 years (IQR 2–6) of follow-up.

    With the use of machine learning, we developed a prediction model and we managed to include 248 SSc patients with complete data on 90 variables on at least three visits. These patients had a median follow-up of 5.4 years (IQR 3.2–7.5). Although the overall accuracy of the model was moderate, it performed very well in identifying patients with a low risk for disease progression (29% NPV 1.0). For these patients we can adjust annual evaluation using a less extensive diagnostic programme.

    To identify patients at low risk, we calculated probability scores with the machine-learning-assisted prediction model. The cut-off for low-risk patients was very strict, since we did not want to miss any organ progression, with none of the progressors scoring under the low risk cut-off (NPV 1.0). Twenty-nine per cent of the SSc patients were identified in the low-risk group and extensive follow-up might not be necessary in this patient group. The machine-learning-assisted model could therefore significantly reduce healthcare costs without substantial risk to our patients. The assessments that are necessary to identify progression with our model are predominantly: use of IS medication in the past, presence of PAH, mRSS, DLCO and cardiac and GI involvement/progression. Based on this observation showing a diverse group of characteristics that identify risk of progression, we conclude that in all patients with a new diagnosis of SSc complete organ assessment is necessary to guide future follow-up.

    To build the machine-learning-assisted model, we used ‘elastic-net regularisation’, a variable selection method that allows to address multicollinearity. It provides a more reproducible prediction than multiple regression, especially when predictors are highly correlated. Elastic net regularisation has been shown to robustly maintain predictive accuracy even with a large number of predictors relative to the number of observations. We note that the variables in the final model are predictors, and can therefore not be interpreted as having a causal relationship with progression. Furthermore, since we used a regularisation method, variables that play an important causal role could have been dropped from the model when other variables had a similar or stronger association. Even though the CCISS care pathway is highly standardised and in accordance with international guidelines, we cannot rule out that factors related to the local healthcare situation have influenced the results. It is therefore important to validate this model in different healthcare systems. We did not calibrate the probabilities of the machine-learning-assisted model model, whereby the probabilities are slightly different from the real risk. This was acceptable since we used a cut-off to identify patients at low risk for progression and not the full range of probabilities.

    One of our secondary aims was to compare the machine-learning-assisted model with a model based on the Delphi guideline including the selected tools, to evaluate if assessment of these 55 tools in every patient on a yearly basis might be redundant for a part of the SSc population. The prediction model based on the Delphi variables (including 51 expert opinion variables in the final model) had a similar AUC-ROC as the Machine-learning assited model model (with only 10 out of 90 variables in the final model based on data driven selection). By using the identified variables for annual follow-up selected by experts to predict disease progression, the discrimination of probability scores between progressors and non-progessors improved but identification of low-risk patients was more difficult, and physicians need to collect 51 variables. Forty-six per cent of the patients that exhibited progression had a risk-probability close to 0 (<0.007) according to the prediction model based on the Delphi model. The machine-learning-assisted model was very well suited to identify patients at low risk as 29% had a probability below 0.197 and all these patients were non-progressors. The comparison between the two models demonstrates that the combination of all Delphi variables cannot directly be used to predict patients at (low) risk for progression. Clearly, the machine-learning-assisted model as constructed in our study is useful to identify patients who are at low risk for disease progression and who therefore may not need intensive follow-up evaluations. Important sidenote, in both models, only patients who underwent complete evaluation for organ involvement and disease progression at least twice were included. Given the severe and heterogeneous nature of SSc, which is underlined by the fact that 52% of patients experienced disease progression during follow-up, in our opinion, annual extensive evaluation is justified in newly diagnosed SSc patients during the first 2 years. After two evaluations, our current data show that one could consider to apply the probability scores for risk on progression and identify patients in whom follow-up evaluations can be less extensive.

    There are some limitations to be acknowledged. First, the clinical variables collected in this study ideally reflect disease activity, disease status and organ damage, to predict disease outcome. However, in SSc, uniform and validated definitions are lacking for some of the organ systems, which should be taken into account as a general limitation. For example, GI involvement was based on presence of gastric antral vascular ectasia (GAVE) or anaemia AND weight loss, although anaemia and weight loss can have other reasons outside of GI involvement. Second, evaluating progression of GAVE and/or PAH is difficult to capture from a clinical database. We chose to classify PAH patients as non-progressors based on stable PFT, which might have missed some patients. With respect to GAVE we are reasonably convinced that patients with clinically relevant GAVE are correctly identified based on the fact we included haemoglobin in our dataset. Vascular manifestations and arthritis were included as predictors in our model and were not included as outcome variables as we aim to evaluate tailored annual internal organ involvement assessment. However, we acknowledge that these symptoms can have a large impact on patients’ well-being; therefore, this should be taken into account when applying our prediction in clinical practice. In our local clinical practice, these symptoms are evaluated and treated by the treating rheumatologist during regular outpatient clinical visits (outside of the SSc care pathway). Second, the follow-up duration might not be sufficient to capture all progressors. Although median follow-up duration is short (5.4 years), 54% of patients had a disease duration of >10 years since first non-RP at the end of the observation period. The follow-up period between progressors and non-progressors was different; however, the proportion of progressors is similar among groups stratified for follow-up duration (data not shown). importantly, we had to exclude 244 patients for our final model, as we defined 100% complete data on at least three visits as a prerequisite (for both independent and dependent variables) to predict as accurate as possible (figure 1). When building the model, we preferred to overestimate progression instead of missing important organ progression. With that in mind, we used these strict inclusion criteria, and, as a consequence, the possibility of selection bias must be considered. The patients included in the model were more often ATA positive and more often had ILD. Therefore, the population used to build the progression model probably had more severe disease, and as such the observed selection does not interferes with the primary aim. Of the 244 patients that had to be excluded for development of the prediction model, 81% was still in follow-up as part of the CCISS cohort, and the majority (89%) did not show progression based on clinical assessment, including PFT and 6 min walking test (online supplemental figure S3). The large time frame of the study might also be a limitation as IS treatment might have changed over the years. We evaluated the use of IS therapy for three time frames (2009–2012, 2013–2016, 2017–2019) and found a similar percentage of patients starting IS medication, which makes a large impact on primary outcome unlikely. Another limitation of the study is intrinsic to the heterogeneity of the disease. Included predictors might act as risk and protective factors at the same time, as our primary outcome was aggregated disease progression. We tried to built different models for every organ system, however, the occurrence rate of progression was too low to create single reliable models. Finally, we did not have access to a prospective and independent validation sample with all 90 variables available to further test the model’s validity. While a completely held‐out test sample is statistically equivalent to a prospective sample from the same population, separate validation from a truly prospective sample could further examine the model’s generalisability. Previous studies have looked at predictors individually, but the unbiased, data‐driven approach, which is a major strength of our work, could contribute to tailor future directions for research and clinical practice. For instance, accurate prediction of patient outcome can be used to inform treatment planning decisions, where modelling the likelihood of disease progression is a critical outcome of interest to healthcare systems, providers and stakeholders. Future development of these tools with larger training samples can improve prediction of patient outcome, even to the point where differential predictions of outcome for the personalisation of monitoring of SSc might be possible.

    A next step is to validate this model in clinical practice. Therefore, we have designed the trial ‘From a pragmatic model to a pragmatic study: a non-inferiority randomised trial’. We aim to start inclusion in 2021. The aim of this trial is to evaluate whether annual assessment in patients who underwent extensive evaluation at least twice and are categorised as low-risk patients based on our machine learning derived model, can be less extensive without jeopardising healthcare utilisation, quality of life, disease perceptions and disease course. In addition, an online tool will be developed to calculate risk scores for SSc patients (work in progress).

    In the end, achieving equality of assessment worldwide will most likely increase the standard care for SSc. However, until now there were no existing evidence-based guidelines for standardised follow-up of patients with SSc. This study showed that disease progression somewhere during follow-up occurs in 52% of the SSc patients, with a high variety between organ systems. Without the use of a prediction model these findings justify the annual complete organ assessments, at least for the first 5 years since first non-RP symptom. While identifying SSc patients at risk for progression remains difficult, our prediction model facilitates the stratification of low, intermediate and high-risk patients. In conclusion, SSc patients with a low risk at progression can be identified with the use of the machine-learning-assisted model and allows us to confidently identify a subset of patients who can safely reduce their visit frequency.

    Data availability statement

    Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as online supplemental information.

    Ethics statements

    Ethics approval

    The cohort study was designed in accordance with the ethical principles of the Declaration of Helsinki. All patients gave written informed consent. Collection and analysis of data have been approved by the local ethics committee (Leiden CME number B16.037).

    References

    1. ↵
      1. Allanore Y,
      2. Simms R,
      3. Distler O, et al
      . Systemic sclerosis. Nat Rev Dis Primers 2015;1:15002.doi:10.1038/nrdp.2015.2pmid:http://www.ncbi.nlm.nih.gov/pubmed/27189141
      OpenUrlCrossRefPubMed
    2. ↵
      1. Becker M,
      2. Graf N,
      3. Sauter R, et al
      . Predictors of disease worsening defined by progression of organ damage in diffuse systemic sclerosis: a European scleroderma trials and research (EUSTAR) analysis. Ann Rheum Dis 2019;78:1242–8.doi:10.1136/annrheumdis-2019-215145pmid:http://www.ncbi.nlm.nih.gov/pubmed/31227488
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Khanna D,
      2. Denton CP
      . Evidence-based management of rapidly progressing systemic sclerosis. Best Pract Res Clin Rheumatol 2010;24:387–400.doi:10.1016/j.berh.2009.12.002pmid:http://www.ncbi.nlm.nih.gov/pubmed/20534372
      OpenUrlCrossRefPubMed
    4. ↵
      1. Ledoult E,
      2. Launay D,
      3. Béhal H, et al
      . Early trajectories of skin thickening are associated with severity and mortality in systemic sclerosis. Arthritis Res Ther 2020;22:30.doi:10.1186/s13075-020-2113-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/32070422
      OpenUrlPubMed
    5. ↵
      1. Fasano S,
      2. Riccardi A,
      3. Messiniti V, et al
      . Revised European scleroderma trials and research group activity index is the best predictor of short-term severity accrual. Ann Rheum Dis 2019;78:1681–5.doi:10.1136/annrheumdis-2019-215787pmid:http://www.ncbi.nlm.nih.gov/pubmed/31422354
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Wu W,
      2. Jordan S,
      3. Becker MO, et al
      . Prediction of progression of interstitial lung disease in patients with systemic sclerosis: the SPAR model. Ann Rheum Dis 2018;77:1326–32.doi:10.1136/annrheumdis-2018-213201pmid:http://www.ncbi.nlm.nih.gov/pubmed/29875097
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Diab N,
      2. Hassoun PM
      . Pulmonary arterial hypertension: screening challenges in systemic sclerosis and future directions. Eur Respir J 2017;49:1700522. doi:10.1183/13993003.00522-2017pmid:http://www.ncbi.nlm.nih.gov/pubmed/28495698
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Hoffmann-Vold A-M,
      2. Distler O,
      3. Murray B, et al
      . Setting the International standard for longitudinal follow-up of patients with systemic sclerosis: a Delphi-based expert consensus on core clinical features. RMD Open 2019;5:e000826. doi:10.1136/rmdopen-2018-000826pmid:http://www.ncbi.nlm.nih.gov/pubmed/30997150
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Hao Y,
      2. Hudson M,
      3. Baron M, et al
      . Early mortality in a multinational systemic sclerosis inception cohort. Arthritis Rheumatol 2017;69:1067–77.doi:10.1002/art.40027pmid:http://www.ncbi.nlm.nih.gov/pubmed/28029745
      OpenUrlPubMed
    10. ↵
      1. Boonstra M,
      2. Ninaber MK,
      3. Ajmone Marsan N, et al
      . Prognostic properties of anti-topoisomerase antibodies in patients identified by the ACR/EULAR 2013 systemic sclerosis criteria. Rheumatology 2019;58:730–2.doi:10.1093/rheumatology/kez005pmid:http://www.ncbi.nlm.nih.gov/pubmed/30690619
      OpenUrlPubMed
    11. ↵
      1. Meijs J,
      2. Schouffoer AA,
      3. Ajmone Marsan N, et al
      . Therapeutic and diagnostic outcomes of a standardised, comprehensive care pathway for patients with systemic sclerosis. RMD Open 2016;2:e000159. doi:10.1136/rmdopen-2015-000159pmid:http://www.ncbi.nlm.nih.gov/pubmed/27042333
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. LeRoy EC,
      2. Medsger TA
      . Criteria for the classification of early systemic sclerosis. J Rheumatol 2001;28:1573–6.pmid:http://www.ncbi.nlm.nih.gov/pubmed/11469464
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Clements P,
      2. Lachenbruch P,
      3. Siebold J, et al
      . Inter and intraobserver variability of total skin thickness score (modified Rodnan TSS) in systemic sclerosis. J Rheumatol 1995;22:1281–5.pmid:http://www.ncbi.nlm.nih.gov/pubmed/7562759
      OpenUrlPubMedWeb of Science
    14. ↵
      1. Clements PJ,
      2. Wong WK,
      3. Hurwitz EL, et al
      . The disability index of the health assessment questionnaire is a predictor and correlate of outcome in the high-dose versus low-dose penicillamine in systemic sclerosis trial. Arthritis Rheum 2001;44:653–61.doi:10.1002/1529-0131(200103)44:3&lt;653::AID-ANR114&gt;3.0.CO;2-Qpmid:http://www.ncbi.nlm.nih.gov/pubmed/11263780
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Ware JE,
      2. Kosinski M,
      3. Keller S
      . SF-36 physical and mental health summary scales: a user’s manual: Health Assessment Lab, 1994.
    16. ↵
      1. Newnham EA,
      2. Harwood KE,
      3. Page AC
      . Evaluating the clinical significance of responses by psychiatric inpatients to the mental health subscales of the SF-36. J Affect Disord 2007;98:91–7.doi:10.1016/j.jad.2006.07.001pmid:http://www.ncbi.nlm.nih.gov/pubmed/16904752
      OpenUrlCrossRefPubMed
    17. ↵
      1. Mouthon L,
      2. Rannou F,
      3. Bérezné A, et al
      . Development and validation of a scale for mouth handicap in systemic sclerosis: the mouth handicap in systemic sclerosis scale. Ann Rheum Dis 2007;66:1651–5.doi:10.1136/ard.2007.070532pmid:http://www.ncbi.nlm.nih.gov/pubmed/17502364
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Schouffoer AA,
      2. Strijbos E,
      3. Schuerwegh AJM, et al
      . Translation, cross-cultural adaptation, and validation of the mouth handicap in systemic sclerosis questionnaire (MHISS) into the Dutch language. Clin Rheumatol 2013;32:1649–55.doi:10.1007/s10067-013-2321-4pmid:http://www.ncbi.nlm.nih.gov/pubmed/23881437
      OpenUrlCrossRefPubMed
    19. ↵
      1. Dolan P
      . Modeling valuations for EuroQol health states. Med Care 1997;35:1095–108.doi:10.1097/00005650-199711000-00002pmid:http://www.ncbi.nlm.nih.gov/pubmed/9366889
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Lamers L,
      2. Stalmeier P,
      3. McDonnell J, et al
      . Kwaliteit van leven meten in economische evaluaties: Het Nederlands EQ-5D-tarief. Ned Tijdschr Geneeskd 2005;149:1574–8.
      OpenUrlPubMed
    21. ↵
      1. Meijs J,
      2. Pors D,
      3. Vliet Vlieland TPM, et al
      . Translation, cross-cultural adaptation, and validation of the UCLA scleroderma clinical trial Consortium gastrointestinal tract instrument (SCTC GIT) 2.0 into Dutch. Clin Exp Rheumatol 2014;32:S-41-8.pmid:http://www.ncbi.nlm.nih.gov/pubmed/24984228
      OpenUrlPubMed
    22. ↵
      1. Khanna D,
      2. Hays RD,
      3. Park GS, et al
      . Development of a preliminary scleroderma gastrointestinal tract 1.0 quality of life instrument. Arthritis Rheum 2007;57:1280–6.doi:10.1002/art.22987
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. MAOaP L
      . Preliminary criteria for the classification of systemic sclerosis (scleroderma). Subcommittee for scleroderma criteria of the American rheumatism association diagnostic and therapeutic criteria Committee. Arthritis Rheum 1980;23:581–90.doi:10.1002/art.1780230510pmid:http://www.ncbi.nlm.nih.gov/pubmed/7378088
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. van den Hoogen F,
      2. Khanna D,
      3. Fransen J, et al
      . 2013 classification criteria for systemic sclerosis: an American College of rheumatology/European League against rheumatism collaborative initiative. Ann Rheum Dis 2013;72:1747–55.doi:10.1136/annrheumdis-2013-204424pmid:http://www.ncbi.nlm.nih.gov/pubmed/24092682
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Galiè N,
      2. Humbert M,
      3. Vachiery J-L, et al
      . 2015 ESC/ERS guidelines for the diagnosis and treatment of pulmonary hypertension: the joint Task force for the diagnosis and treatment of pulmonary hypertension of the European Society of cardiology (ESC) and the European respiratory Society (ERS): endorsed by: association for European paediatric and congenital cardiology (AEPC), International Society for heart and lung transplantation (ISHLT). Eur Respir J 2015;46:903–75.doi:10.1183/13993003.01032-2015pmid:http://www.ncbi.nlm.nih.gov/pubmed/26318161
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Butler E-A,
      2. Baron M,
      3. Fogo AB, et al
      . Generation of a core set of items to develop classification criteria for scleroderma renal crisis using consensus methodology. Arthritis Rheumatol 2019;71:964–71.doi:10.1002/art.40809pmid:http://www.ncbi.nlm.nih.gov/pubmed/30614663
      OpenUrlPubMed
    27. ↵
      1. Distler O,
      2. Assassi S,
      3. Cottin V, et al
      . Predictors of progression in systemic sclerosis patients with interstitial lung disease. Eur Respir J 2020;55:1902026.doi:10.1183/13993003.02026-2019pmid:http://www.ncbi.nlm.nih.gov/pubmed/32079645
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Distler O,
      2. Highland KB,
      3. Gahlemann M, et al
      . Nintedanib for systemic sclerosis-associated interstitial lung disease. N Engl J Med 2019;380:2518–28.doi:10.1056/NEJMoa1903076pmid:http://www.ncbi.nlm.nih.gov/pubmed/31112379
      OpenUrlCrossRefPubMed
    29. ↵
      1. Boonstra M,
      2. Meijs J,
      3. Dorjée AL, et al
      . Rituximab in early systemic sclerosis. RMD Open 2017;3:e000384. doi:10.1136/rmdopen-2016-000384pmid:http://www.ncbi.nlm.nih.gov/pubmed/28879049
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. van Laar JM,
      2. Farge D,
      3. Sont JK, et al
      . Autologous hematopoietic stem cell transplantation vs intravenous pulse cyclophosphamide in diffuse cutaneous systemic sclerosis: a randomized clinical trial. JAMA 2014;311:2490–8.doi:10.1001/jama.2014.6368pmid:http://www.ncbi.nlm.nih.gov/pubmed/25058083
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Khanna D,
      2. Lin CJF,
      3. Furst DE, et al
      . Tocilizumab in systemic sclerosis: a randomised, double-blind, placebo-controlled, phase 3 trial. Lancet Respir Med 2020;8:963–74.doi:10.1016/S2213-2600(20)30318-0pmid:http://www.ncbi.nlm.nih.gov/pubmed/32866440
      OpenUrlPubMed
    32. ↵
      1. Spiera R,
      2. Hummers L,
      3. Chung L, et al
      . Safety and efficacy of Lenabasum in a phase II, randomized, placebo-controlled trial in adults with systemic sclerosis. Arthritis Rheumatol 2020;72:1350–60.doi:10.1002/art.41294pmid:http://www.ncbi.nlm.nih.gov/pubmed/32336038
      OpenUrlPubMed
    33. ↵
      1. Zou H,
      2. Hastie T
      . Regularization and variable selection via the elastic net. J Royal Statistical Soc B 2005;67:301–20.doi:10.1111/j.1467-9868.2005.00503.x
      OpenUrl

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • NMvL and MM contributed equally.

    • Presented at This paper has been presented as an abstract at the ACR 2020 conference.

    • Contributors NMvL, MM, RK and JDV-B conceived and planned the research idea. NMvL, JC, MN, NAM and SL collected the clinical data, NMvL, MM, RK, SL, TH and JDV-B, contributed to the content of the analyses plan. NMvL, MM, SL, JC, RK, MN, NAM, CA, RG, HGvD, JDV-B and TH performed the analyses and contributed to the interpretation of the results. NMvL took the lead in writing the manuscript. All authors read and critically reviewed the manuscript prior the submission.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.