You are here

Article Text

Article menu

Download PDFPDF

Systemic lupus erythematosus
Multicriteria decision analysis process to develop new classification criteria for systemic lupus erythematosus
  1. Sara K Tedeschi1,
  2. Sindhu R Johnson2,
  3. Dimitrios T Boumpas3,
  4. David Daikh4,5,
  5. Thomas Dörner6,
  6. Betty Diamond7,
  7. Søren Jacobsen8,
  8. David Jayne9,
  9. Diane L Kamen10,
  10. W Joseph McCune11,
  11. Marta Mosca12,
  12. Rosalind Ramsey-Goldman13,
  13. Guillermo Ruiz-Irastorza14,
  14. Matthias Schneider15,
  15. Murray Urowitz2,
  16. David Wofsy4,
  17. Josef S Smolen16,
  18. Raymond P Naden17,
  19. Martin Aringer18,
  20. Karen H Costenbader1
  1. 1 Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, Massachusetts, USA
  2. 2 Department of Medicine, Toronto Western Hospital, Mount Sinai Hospital, Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
  3. 3 Departments of Internal Medicine and Rheumatology, Clinical Immunology and Allergy, University of Crete, Heraklion, Greece
  4. 4 Department of Medicine, University of California San Francisco, San Francisco, California, USA
  5. 5 Department of Medicine, VA Medical Center, San Francisco, California, USA
  6. 6 Department of Medicine/Rheumatology and Clinical Immunology, Charite University Hospitals, Berlin, Germany
  7. 7 Center for Autoimmune, Musculoskeletal and Hematopoietic Diseases, Feinstein Institute for Medical Research, Manhasset, New York, USA
  8. 8 Copenhagen Lupus and Vasculitis Clinic, Center for Rheumatology and Spine Diseases, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
  9. 9 Department of Medicine, University of Cambridge, Cambridge, UK
  10. 10 Department of Medicine, Medical University of South Carolina, Charleston, South Carolina, USA
  11. 11 Department of Medicine, University of Michigan, Ann Arbor, Michigan, USA
  12. 12 Department of Clinical and Experimental Medicine, University of Pisa, Pisa, Italy
  13. 13 Department of Medicine/Division of Rheumatology, Northwestern University, Chicago, Illinois, USA
  14. 14 Autoimmune Diseases Research Unit, Department of Internal Medicine, Biocruces Health Research Institute, Hospital Universitario Cruces, University of The Basque Country, Bizkaia, Spain
  15. 15 Rheumatology Department, University Hospital Dusseldorf, Dusseldorf, Germany
  16. 16 Department of Rheumatology, Medical University of Vienna, Vienna, Austria
  17. 17 Department of Medicine, McMaster University, Hamilton, Canada
  18. 18 Department of Medicine III, University Medical Center and Faculty of Medicine Carl Gustav Carus at the TU Dresden, Dresden, Germany
  1. Correspondence to Dr Sara K Tedeschi, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA 02115, USA; stedeschi1{at}bwh.harvard.edu

Abstract

European League Against Rheumatism and are jointly supporting multiphase development of systemic lupus erythematosus (SLE) classification criteria based on weighted criteria and a continuous probability scale. Prior steps included item generation, item reduction and hierarchical organisation of candidate criteria using an evidence-based approach. Our objectives were to determine relative weights using multicriteria decision analysis (MCDA) and to set a provisional threshold score for SLE classification. An SLE Expert Panel (8 European, 9 North American) submitted 164 real, unique cases with a wide range of SLE probability in a standardised format. Using the candidate criteria, experts scored and rank-ordered 20 representative cases. At an in-person meeting, experts reviewed inter-rater reliability of scoring, further refined criteria definitions and participated in an MCDA exercise. Based on expert consensus decisions on pairwise comparisons of criteria, 1000minds software calculated criteria weights and rank-ordered the remaining 144 cases based on their additive scores. The score of the lowest-ranked case for which complete expert consensus was achieved defined the provisional threshold classification score. Inter-rater reliability of scoring cases with the candidate criteria was good. MCDA involved 74 pairwise decisions and was repeated for the arthritis and mucocutaneous domains when the initial ranking of some cases did not match expert opinion. After criteria weights and additive scores were recalculated once, experts reached consensus for SLE classification for all cases scoring>83. Using an iterative process, the candidate criteria definitions were refined, preliminary weights were calculated and a provisional threshold score for SLE classification was determined.

  • systemic lupus erythematosus
  • clinical research
  • methodology

https://doi.org/10.1136/annrheumdis-2018-214685

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    A multinational effort to develop new classification criteria for systemic lupus erythematosus (SLE) for clinical research, jointly supported by the European League Against Rheumatism (EULAR) and American College of Rheumatology (ACR), is underway. The overarching goal is to develop a system that identifies potential participants for clinical research studies, requiring a degree of homogeneity among subjects while simultaneously dealing with the extreme heterogeneity of SLE.1 The aim was to design a system with the maximum combination of sensitivity and specificity for SLE, retaining face validity. While the classification criteria are not intended for diagnosis or clinical care, it is acknowledged that the only available ‘gold standard’ for the presence of SLE is expert clinician opinion.

    A 12 member Steering Committee was formed with input from EULAR and ACR leadership to oversee a four-phase process.2 In Phase 1, items were generated using a Delphi exercise,3 early SLE cohort4 and SLE patient survey5; and antinuclear antibody (ANA) was evaluated as a potential entry criterion.6 7 During Phase 2, the list of potential criteria was narrowed using nominal group technique.8 9 Phase 3 began with a literature review for test performance characteristics of candidate criteria and data-driven organisation of criteria into domains.1 This report outlines the latter part of Phase 3: criteria weighting and threshold score identification through a consensus-based multicriteria decision analysis (MCDA) approach.10–12 The goal was to develop a criteria system producing a continuous measure of the relative probability that a case (ie, particular combination of clinical features) could be characterised as SLE, and a provisional threshold score above which a case could be definitely classified as SLE for clinical research.13 14 Phase 4 involves the determination of the final threshold, followed by validation of the classification system.

    Methods

    An international panel of SLE experts collected and rank-ordered patient case scenarios, participated in an in-person consensus meeting and held post-meeting email and telephone discussions.

    SLE expert panel

    The Steering Committee invited 6 additional experts (3 European, 3 North American) to form a 17 person SLE Expert Panel (‘SLE experts’) to assist with this phase and establish external validity of the criteria development process. SLE experts were senior clinicians focused on SLE, many of whom direct SLE clinics at their institutions, and senior clinical investigators with expertise in SLE.

    Development of patient case scenarios

    Each of the 17 SLE experts submitted 10 deidentified real cases based on adult patients from his/her own cohort in a standardised online form using REDCap (Research Electronic Data Capture), a secure, web-based application for research studies.15 Each expert was asked to submit five cases with ‘definite’ or ‘likely’ SLE and five cases in which they considered but ultimately did not diagnose SLE and/or diagnosed a condition mimicking SLE such as rheumatoid arthritis, other inflammatory arthritis, Sjögren’s syndrome, antiphospholipid antibody syndrome or viral infection. ANA≥1:80 was required of all cases. The REDCap form included three options for each clinical and laboratory criterion: yes (present), no (absent) and unknown.

    Rank ordering and scoring of cases

    From 164 deidentified cases, three authors of this manuscript (KHC, RPN, SKT) chose a representative sample of 20 reflecting a range of possible SLE cases. Each case was abstracted into standardised paragraph format. Laboratory tests that had not been performed were treated as unknown. SLE experts were asked to rank the cases based on their confidence that the case should be classified as SLE. This exercise introduced SLE experts to the challenge of assessing the relative influence of individual criteria in pointing towards or away from SLE.

    SLE experts then scored the 20 cases using a standardised REDCap form reflecting the draft SLE classification criteria as of September 2016, based on the Phase 2 nominal group technique exercise16 and subsequent work by the Steering Committee.1 The REDCap form included 10 domains; each domain included 2–6 options. Experts were provided written instructions for scoring and a list of proposed definitions for each criterion. The instructions specified that within each domain, criteria were ordered from least to most supportive of SLE and if multiple criteria were present in one domain only the single criterion furthest down the list (ie, most supportive of SLE) should be scored. The instructions specified that a criterion should not be scored if a cause more likely than SLE existed (eg, other autoimmune disease, malignancy, medication). Criteria did not need to occur simultaneously and could occur before or after the detection of ANA≥1:80 as long as another explanation more likely than SLE did not exist.

    In-person consensus meeting, November 2016

    During a 1.5-day in-person meeting, RPN and AH moderated discussions among SLE experts leading to consensus decisions. Goals of this meeting included achieving full consensus on criteria definitions, calculating criteria weights via a MCDA exercise and establishing a provisional threshold score for SLE classification.

    1. Review of case scoring and criteria refinement. Experts reviewed a summary of the REDCap scoring exercise. Discrepancies in scoring individual cases were discussed in depth to understand the underlying reasons. Criteria definitions were discussed in the context of these discrepancies and refined based on consensus agreement.

    2. MCDA to determine weights. The MCDA exercise is based on the PAPRIKA method (Potentially All Pairwise RanKings of all possible Alternatives),17 as implemented by 1000minds software (http://www.1000minds.com). This method and software have been used extensively since 2010 for developing classification criteria.10 11 18 Experts voted on a series of pairwise decisions about hypothetical cases, each defined by two criteria from two domains. For example, hypothetical case A: ‘oral ulcers’ (mucocutaneous domain) and ‘acute pericarditis’ (serositis domain) versus hypothetical case B ‘alopecia’ (mucocutaneous domain) and ‘pleural effusion’ (serositis domain). Experts were asked to decide whether they would more likely classify hypothetical case A or B as SLE, presuming all else was equal about the cases. Voting was conducted anonymously, but where opinions diverged cases were discussed until full consensus was reached. Consensus opinion was based on the specificity of each manifestation for SLE and how much its presence would increase the likelihood of SLE (although specificity for some manifestations has not been formally evaluated, as discussed in Ref. 1). Such pairwise-ranking questions were repeated with different pairs of hypothetical cases—always involving trade-offs between different combinations of criteria, two at a time—until enough information about expert preferences had been collected to determine relative criteria weights for all criteria. Each time experts ranked a pair, all other cases that could be pairwise ranked via the logical property of ‘transitivity’ were identified and eliminated. For example, if experts ranked hypothetical case A over B and B over C, then by transitivity A is also ranked over C (and experts are not asked to choose between A and C). This procedure ensures the number of pairwise-ranking questions posed is minimised, and experts end up having pairwise ranked all possible cases defined on two criteria at a time. Consensus decisions were entered into 1000minds software, which uses linear programming techniques to derive weights for each criterion.17

    3. Assessment of the face validity of the weights. Criteria weights were summed to produce an additive score for each case. Only the highest-weighted criterion in each domain was counted towards the additive score, as specified in the instructions (Box 1). The remainder of the 164 cases were scored and arranged in rank order from highest to lowest score. SLE experts reviewed a spreadsheet listing the criteria present in each case and anonymously voted whether they would classify each as SLE. For cases where expert opinion differed, RPN facilitated discussion to achieve full consensus about case classification. Cases were discussed in descending rank order (confidence that the case should be classified as SLE) until agreement on classification could not be reached.

    4. Determination of an upper threshold score. The score of the last case for which the group achieved consensus on classification as SLE was the initial threshold.

    5. Review of cases below the threshold. The cases with scores immediately below the initial threshold were individually reviewed. The threshold thus functioned as a way to focus the discussion on these ‘borderline’ cases, and the individual criteria present in each of these. SLE experts reached consensus that several of these cases should have been classified as SLE. Experts discussed discrepancies between expert opinion and the initial weights assigned to some of the criteria.

    6. Weighting and upper threshold revision. The MCDA exercise was repeated once for those criteria whose calculated weights were inconsistent with expert opinion. Weights for all criteria were recalculated using 1000minds and additive scores were recalculated. SLE experts again anonymously voted on classifying each case as SLE, followed by discussion facilitated by RPN to achieve consensus. The score of the last case for which expert consensus was achieved was the provisional full consensus upper threshold score. Phase 4 involves further refinement of the upper threshold score.

    Box 1.

    Provisional SLE classification criteria organisation and definitions

    Opening statements:

    • A history of a positive ANA by Hep 2 immunofluorescence ≥1:80 is required for consideration of a person for SLE classification.

    • For each criterion, do not score if a cause more likely than SLE exists (such as infection, malignancy, medication, rosacea, endocrine disorder, other autoimmune disease).

    • Occurrence of a criterion on at least one occasion is sufficient.

    • Criteria need not occur simultaneously.

    • At least one clinical criterion must be present.

    • Within each domain, only the highest weighted criterion is counted towards the total score.

    Clinical domains and criteria

    Constitutional

    • Fever:>38.3°C with no other source identified.

    Haematological

    • Leucopaenia: WBC<4000/mm3.

    • Thrombocytopaenia: Platelets<100 000/mm3.

    • Autoimmune haemolysis: (1) evidence of haemolysis, such as reticulocytosis, low haptoglobin, elevated indirect bilirubin, elevated LDH and (2) positive Coomb’s (direct antiglobulin) test.

    Neuropsychiatric

    • Delirium: characterised by (1) change in consciousness or level of arousal with reduced ability to focus and (2) symptom development over hours to <2 days and (3) symptom fluctuation throughout the day and (4) either (4a) acute/subacute change in cognition (eg, memory deficit or disorientation) or (4b) change in behaviour, mood or affect (eg, restlessness, reversal of sleep/wake cycle and so on).

    • Psychosis: characterised by (1) delusions and/or hallucinations without insight and (2) absence of delirium.

    • Seizure: primary generalised seizure or partial/focal seizure, with independent description by a reliable witness. If EEG is performed, abnormalities must be present.

    Mucocutaneous

    • Non-scarring alopecia, observed by a clinician*

    • Oral ulcers, observed by a clinician*

    • Subacute cutaneous lupus (SCLE) or discoid lupus (DLE): SCLE is characterised by annular or papulosquamous (psoriasiform) cutaneous eruption observed by a clinician,* usually photodistributed. If skin biopsy is performed, typical changes must be present.26 DLE is characterised by erythematous-violaceous cutaneous lesions with secondary changes of atrophic scarring, dyspigmentation, often follicular hyperkeratosis/plugging (scalp), observed by a clinician,* leading to scarring alopecia on the scalp. Lesions have a preference for the head and neck, especially the conchal bowl, but may be found in nearly any location. If skin biopsy is performed, typical changes must be present.26

    • Acute cutaneous lupus: Malar rash (localised) or maculopapular rash (generalised) observed by a clinician,* with or without photosensitivity. If skin biopsy is performed, typical changes must be present.26

    Serositis

    • Pleural or pericardial effusion: imaging evidence (such as ultrasound, X-ray, CT scan, MRI) of pleural or pericardial effusion or both

    • Acute pericarditis: ≥2 of: (1) pericardial chest pain (typically sharp, worse with inspiration, improved by leaning forward), (2) pericardial rub, (3) EKG with new widespread ST-elevation or PR depression, (4) new or worsened pericardial effusion on imaging (such as ultrasound, X-ray, CT scan, MRI)

    Musculoskeletal

    • Synovitis in ≥2 joints: characterised by joint swelling and tenderness, observed by a clinician*

    Renal

    • Proteinuria>0.5 g/24 hours: on 24 hours urine collection or spot urine protein-to-creatinine ratio representing >0.5 g protein/24 hours

    • Renal biopsy with Class II or V lupus nephritis, per International Society of Nephrology/Renal Pathology Society (ISN/RPS) 2003 classification27

    • Renal biopsy with Class III or IV lupus nephritis, per International Society of Nephrology/Renal Pathology Society (ISN/RPS) 2003 classification27

    Immunological domains and criteria

    Antiphospholipid antibodies

    • Anticardiolipin IgG (>40 GPL units) or anti-β2GP1 IgG (>40 units) or lupus anticoagulant positive

    Complement proteins

    • Low C3 or low C4

    • Low C3 and low C4

    SLE-specific antibodies

    • Anti-dsDNA antibody

    • Anti-Smith antibody

    • *Direct observation may include physical examination or review of a photograph.26

    Determining a lower threshold score

    SLE experts attempted to set an upper threshold for definite SLE classification and a lower threshold for very low probability for classification. Individuals with scores falling between these two thresholds might be candidates for inclusion in observational studies or SLE prevention trials. Due to insufficient time at the November 2016 meeting, the lower threshold was addressed in emails, secondary exercises and conference calls in the next 2 months. SLE experts were asked to rate the cases that fell below the upper threshold score as ‘probable SLE’, ‘possible SLE’ or ‘unlikely SLE’. The score of the case for which≥70% indicated ‘unlikely SLE’ was assigned as the lower threshold.

    Results

    At the in-person meeting, SLE experts agreed that classification as SLE means a patient is appropriate for inclusion in SLE clinical research—and that classification as SLE should not guide clinical decisions about SLE diagnosis or treatment. Experts agreed that the threshold score should have high specificity for SLE, ensuring a high degree of homogeneity among classified patients and facilitating comparisons across clinical studies. SLE experts reached consensus that patients with overlap syndromes could be classified as SLE if they met SLE classification criteria, allowing clinical investigators to decide whether to include or exclude patients with overlap syndromes in specific research studies.

    Review of scoring and criteria refinement

    There was considerable inconsistency between SLE experts using the REDCap form to score cases. Each expert scored a total of 200 items (20 cases, 10 domains); all 17 experts scored 127/200 (64%) domains exactly the same. Reasons for discrepant data entry included human error in data entry, not following the instructions, variability in interpreting the candidate criteria based on context and different interpretations of criteria definitions (see online supplement 1 for details).

    Review of the rank-ordering exercise

    There was agreement on the cases that the majority of SLE experts ranked the highest and lowest, but a spectrum of ranking for cases in between (figure 1). This reflected the different relative weights that individual experts attached to particular criteria.

    Figure 1
    Figure 1

    Rank-ordering exercise results. Seventeen SLE experts ranked each of 20 cases in order of most likely to least likely SLE. More than 85% of experts rated 5 cases as definite or probable SLE and 5 cases as possible or unlikely SLE. Expert opinion varied for the remaining 10 cases. SLE, systemic lupus erythematosus.

    MCDA exercise to determine consensus weights using 1000 minds software. SLE experts anonymously voted on 74 pairs of hypothetical cases. Sometimes it was agreed that hypothetical cases A and B were equally likely to be SLE. For a handful of pairwise comparisons, consensus could not be reached and the decision was to defer that comparison and approach their relativity from other pairwise comparisons. Significant changes to the criteria during this stage included:

    • Mucocutaneous and musculoskeletal domains. SLE experts decided that observation by a clinician should be required for consistency with other clinical domains. The definition of clinician-observed was broadened to include physical examination or review of a photograph.

    • Neurological domain. Due to disagreement over whether seizure or cranial neuropathy was more specific for SLE (the SLICC19 and ACR 198220 manuscripts did not present the specificity of these individual items), and because the prevalence of cranial neuropathy is very low in SLE (and none of the 164 patient cases had cranial neuropathy), the group reached consensus to remove cranial neuropathy.

    • Renal domain. SLE experts decided that Class VI lupus nephritis was not specific for SLE based on clinical experience and lack of published data, and agreed on removing Class VI nephritis. Importantly, since historical manifestations are included in the scoring system, previous evidence of class II, III, IV or V lupus nephritis would be fully accounted for. These steps resulted in the updated definitions depicted in Box 1.

    Face validity of the weights and initial upper threshold score

    The additive score ranged 0–201 for the 164 cases. SLE experts reviewed the cases in order from highest to lowest score and reached consensus on classifying the 69 highest-scored cases as SLE. The group was unable to reach full consensus for a case with a score of 70; this patient had oral ulcers, leucopaenia, low C3 or C4 and positive anti-dsDNA. The last case for which experts reached consensus (17/17 votes) for classification as SLE had a score of 71, and an initial upper threshold score was set as >70.

    Revising criteria weights and provisional upper threshold score

    The experts reviewed cases scored 60–70. Many had arthritis and most experts had voted to classify them as SLE. Therefore, the group felt that the weight assigned to arthritis was too low. After reviewing the specific criteria present in these cases, the mucocutaneous domain was reorganised based on expert consensus: acute cutaneous lupus was assigned the most influential position because it is most specific, and subacute cutaneous lupus and discoid lupus were grouped together and less influential than acute cutaneous lupus. Anonymous voting was repeated for pairwise comparisons including arthritis and mucocutaneous criteria. 1000minds software recalculated relative weights for all criteria and rescored all cases using the revised weights.

    After this second round of MCDA, arthritis received a greater weight than prior, now identical to the weight of pleural or pericardial effusion. Acute cutaneous lupus was assigned the same weight as acute pericarditis and anti-dsDNA (table 1). The group repeated the anonymous voting exercise and reached consensus about the 82 highest-scored cases. Experts were unable to reach full consensus for the same case that determined the initial threshold. As that case now had a score of 83 using the revised weights, a 100% specific provisional consensus threshold was set as >83. Provisional criteria weights resulting from the MCDA exercise are shown in table 1.

    View this table:
    Table 1

    Provisional SLE classification criteria weights determined by a multicriteria decision analysis exercise

    Lower threshold score

    SLE experts individually rated the 82 cases below the upper threshold score; the distribution of expert opinion is shown in figure 2. The score of the case for which ≥70% indicated ‘unlikely SLE’ was 27. Only 7 of 52 unique cases (13.5%) included in this exercise would be classified as ‘unlikely SLE’ based on this lower threshold, and the remaining 86.5% would potentially be candidates for inclusion into observational or preventive studies. Through a series of telephone calls and emails, it became clear that expert opinion varied considerably concerning the cases below the upper threshold. Additionally, the terms ‘probable’, ‘possible’ and ‘unlikely’ were not being uniformly interpreted. The SLE experts decided against assigning a lower threshold because it would exclude only a few cases from clinical studies.

    Figure 2
    Figure 2

    Exercise to consider a lower threshold. SLE experts anonymously labelled unique cases falling below the preliminary upper threshold as probable, possible or unlikely SLE. Among 82 cases below the upper threshold, 52 had unique combinations of criteria. Rows represent unique cases. Columns represent ratings from each SLE expert. SLE, systemic lupus erythematosus.

    Discussion

    In Phase 3 of this SLE classification criteria development project, we applied a consensus-based, data-driven MCDA approach to assign criteria weights and identify a threshold score for SLE classification among adults for clinical research. This exercise resulted in provisional criteria weights that have face validity and are additive, providing a continuous measure of increasing likelihood for SLE based on combinations of criteria. While full consensus of the 17 SLE experts was reached for cases scoring >83 points, it became evident that expert opinions varied for cases with mid-range or low scores. Many cases with scores just under 83 were still considered SLE by the majority of experts, but in an additional exercise focusing on cases below the threshold for definite SLE, very few were deemed ‘unlikely SLE’ by ≥70% of experts.

    This stage was largely based on the items resulting from the Phase 2 nominal group technique exercise16 and evidence from our literature review of the sensitivity and specificity of the individual candidate criteria.1 These efforts followed rigorous data-driven and expert-guided criteria development methodology in order to ensure high face and content validity of the items, and high discriminant validity of the criteria set.21 22 However, our literature review also revealed knowledge gaps about the sensitivity and specificity of some of the newly proposed criteria, thus expert consensus opinion was critical for decision making.

    Consistent with developing other systems of classification criteria,23 24 there were significant discrepancies in ranking 20 cases regarding likelihood of SLE classification. Discussions centred on two aspects: (1) the precision and thus specificity of clinical and serological manifestations and (2) attribution of manifestations to SLE versus other connective tissue diseases. Some experts expressed concern about misinterpretation of rosacea as acute cutaneous lupus and about false positive anti-dsDNA via ELISA, each of which would reduce the specificity of the proposed classification system. To address these concerns, SLE experts agreed to include detailed definitions for each criterion to mitigate the risk of misinterpreting clinical signs and symptoms. Because particular laboratory assays (eg, Farr method for anti-dsDNA) are not uniformly available in all clinical settings, SLE experts decided that the testing method would not be specified, enabling SLE classification in a wide range of clinics.

    The attribution of manifestations to SLE was discussed at length. For some cases, SLE experts were uncertain about how to interpret particular findings when SLE and another disease, such as primary antiphospholipid syndrome or Sjögren’s syndrome, seemed equally likely. It became apparent that not all these decisions could be made with certainty and that SLE experts from different centres could reach opposing conclusions. The criteria system allows for SLE classification in patients with overlap syndromes (eg, SLE with secondary Sjögren’s) as long as manifestations are considered to be equally or more likely due to SLE than the other condition.

    The decision to exclude Class VI lupus nephritis was unanimous, given the lack of specificity of this end-stage finding. The discussions leading to the consensus elimination of mononeuropathy and cranial neuropathy were of greater interest. It was first mentioned that the specificities of these entities differed and that mononeuropathy is not specific for SLE. The group reached full consensus to eliminate mononeuropathy; cranial neuropathy was initially retained. The group then discussed that cranial neuropathy is a very rare presenting sign in SLE25 and none of the 164 cases had cranial neuropathy. Experts reached a unanimous decision that the low prevalence of cranial neuropathy in SLE warranted its elimination.

    Using a data-driven approach based on literature review1 combined with an expert-driven MCDA process based on real patient cases, this third phase of the SLE classification project has led to precisely defined criteria with individual weights derived through consensus decisions by 17 international SLE experts. The individual criteria weights have face validity, and taken together they depict current expert understanding of SLE. The provisional threshold sets a high bar for SLE classification (100% specificity), and Phase 4 will consider the appropriate balance between specificity and sensitivity before finalising the threshold. The provisional classification criteria and threshold resulting from Phase 3 are being refined and validated in a large, distinct set of patient cases to finalise the project.

    Acknowledgments

    EULAR and ACR jointly provided support for this effort. The Steering Committee includes Martin Aringer, Sindhu Johnson, Thomas Dorner, Dimitrios Boumpas, Marta Mosca, Josef Smolen, David Wofsy, Diane Kamen, Karen Costenbader, David Daikh, Rosalind Ramsey-Goldman, David Jayne. The authors would like to thank Amy Turner for her outstanding organisational support of this project and Alison Hendry for her expertise and assistance during the ACR 2016 meeting. This work was previously presented at the EULAR 2017 Annual Congress and the ACR 2017 Annual Meeting.

    References

    1. 1.↵
      2. Tedeschi SK ,
      3. Johnson SR ,
      4. Boumpas D , et al
      . Developing and refining new candidate criteria for systemic lupus erythematosus classification: an international collaboration. Arthritis Care Res 2018;70:571–81.doi:10.1002/acr.23317
      OpenUrl
    2. 2.↵
      2. Aringer M ,
      3. Dörner T ,
      4. Leuchten N , et al
      . Toward new criteria for systemic lupus erythematosus-a standpoint. Lupus 2016;25:805–11.doi:10.1177/0961203316644338
      OpenUrlCrossRefPubMed
    3. 3.↵
      2. Schmajuk G ,
      3. Hoyer BF ,
      4. Aringer M , et al
      . Multicenter Delphi exercise to identify important key items for classifying systemic lupus erythematosus. Arthritis Care Res 2018;70:1488–94.doi:10.1002/acr.23503
      OpenUrl
    4. 4.↵
      2. Mosca M ,
      3. Costenbader KH ,
      4. Johnson SR , et al
      . Brief report: how do patients with newly diagnosed systemic lupus erythematosus present? A multicenter cohort of early systemic lupus erythematosus to inform the development of new classification criteria. Arthritis Rheumatol 2019;71:91–8.doi:10.1002/art.40674
      OpenUrl
    5. 5.↵
      2. Leuchten N ,
      3. Milke B ,
      4. Winkler-Rohlfing B , et al
      . Early symptoms of systemic lupus erythematosus (SLE) recalled by 339 SLE patients. Lupus 2018;27:1431–6.doi:10.1177/0961203318776093
      OpenUrl
    6. 6.↵
      2. Leuchten N ,
      3. Hoyer A ,
      4. Brinks R , et al
      . Performance of antinuclear antibodies for classifying systemic lupus erythematosus: a systematic literature review and meta-regression of diagnostic data. Arthritis Care Res 2018;70:428–38.doi:10.1002/acr.23292
      OpenUrl
    7. 7.↵
      2. Leuchten N ,
      3. Bertsias G ,
      4. Smolen J , et al
      . ANA as an entry criterion in SLE classification. Arthritis Care Res 2018.doi:10.1002/acr.23558
    8. 8.↵
      2. Nair R ,
      3. Aggarwal R ,
      4. Khanna D
      . Methods of formal consensus in classification/diagnostic criteria and guideline development. Semin Arthritis Rheum 2011;41:95–105.doi:10.1016/j.semarthrit.2010.12.001
      OpenUrlCrossRefPubMed
    9. 9.↵
      2. Johnson SR ,
      3. Khanna D ,
      4. Daikh D , et al
      . Use of consensus methodology to determine candidate items for systemic lupus erythematosus classification criteria. J Rheumatol 2018. doi: 10.3899/jrheum.180478. [Epub ahead of print 15 Dec 2018].doi:10.3899/jrheum.180478.
    10. 10.↵
      2. Johnson SR ,
      3. Naden RP ,
      4. Fransen J , et al
      . Multicriteria decision analysis methods with 1000Minds for developing systemic sclerosis classification criteria. J Clin Epidemiol 2014;67:706–14.doi:10.1016/j.jclinepi.2013.12.009
      OpenUrlCrossRefPubMed
    11. 11.↵
      2. Neogi T ,
      3. Jansen TL ,
      4. Dalbeth N , et al
      . 2015 gout classification criteria: an American College of Rheumatology/European League against rheumatism collaborative initiative. Arthritis Rheumatol 2015;67:2557–68.doi:10.1002/art.39254
      OpenUrl
    12. 12.↵
      2. Neogi T ,
      3. Aletaha D ,
      4. Silman AJ , et al
      . The 2010 American College of Rheumatology/European League against rheumatism classification criteria for rheumatoid arthritis: phase 2 methodological report. Arthritis Rheum 2010;62:2582–91.doi:10.1002/art.27580
      OpenUrlCrossRefPubMedWeb of Science
    13. 13.↵
      2. Tedeschi SK ,
      3. Johnson SR ,
      4. Boumpas DT
      . Multicriteria decision analysis for development of new systemic lupus erythematosus classification criteria (Abstract). Ann Rheum Dis 2017;76(Suppl 2).
    14. 14.↵
      2. Tedeschi SK ,
      3. Johnson SR ,
      4. Boumpas D
      . Multicriteria decision analysis for the development of new systemic lupus erythematosus classification criteria (Abstract). Arthritis Rheum 2017;69(Suppl 10).
    15. 15.↵
      2. Harris PA ,
      3. Taylor R ,
      4. Thielke R , et al
      . Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377–81.doi:10.1016/j.jbi.2008.08.010
      OpenUrlCrossRefPubMedWeb of Science
    16. 16.↵
      2. Johnson SR ,
      3. Khanna D ,
      4. Cervera R
      . Use of nominal group technique to determine candidate items for sle classification criteria development [abstract]. Arthritis Rheum 2016;68(Suppl 10).
    17. 17.↵
      2. Hansen P ,
      3. Ombler F
      . A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. JMCDA 2008;15:87–107.doi:10.1002/mcda.428
      OpenUrl
    18. 18.↵
      2. Shiboski CH ,
      3. Shiboski SC ,
      4. Seror R , et al
      . 2016 American College of Rheumatology/European League against rheumatism classification criteria for primary Sjögren's syndrome: a consensus and data-driven methodology involving three international patient cohorts. Ann Rheum Dis 2017;76:9–16.doi:10.1136/annrheumdis-2016-210571
      OpenUrlAbstract/FREE Full Text
    19. 19.↵
      2. Petri M ,
      3. Orbai A-M ,
      4. Alarcón GS , et al
      . Derivation and validation of the systemic lupus international collaborating clinics classification criteria for systemic lupus erythematosus. Arthritis & Rheumatism 2012;64:2677–86.doi:10.1002/art.34473
      OpenUrlCrossRefPubMedWeb of Science
    20. 20.↵
      2. Tan EM ,
      3. Cohen AS ,
      4. Fries JF , et al
      . The 1982 revised criteria for the classification of systemic lupus erythematosus. Arthritis Rheum 1982;25:1271–7.doi:10.1002/art.1780251101
      OpenUrlCrossRefPubMedWeb of Science
    21. 21.↵
      2. Singh JA ,
      3. Solomon DH ,
      4. Dougados M
      . Development of classification and response criteria for rheumatic diseases. Arthritis Rheum 2006;55:348–52.
      OpenUrlCrossRefPubMedWeb of Science
    22. 22.↵
      2. Johnson SR ,
      3. Goek ON ,
      4. Singh-Grewal D , et al
      . Classification criteria in rheumatic diseases: a review of methodologic properties. Arthritis Rheum 2007;57:1119–33.doi:10.1002/art.23018
      OpenUrlCrossRefPubMedWeb of Science
    23. 23.↵
      2. Aletaha D ,
      3. Neogi T ,
      4. Silman AJ , et al
      . 2010 rheumatoid arthritis classification criteria: an American College of Rheumatology/European League against rheumatism collaborative initiative. Arthritis Rheum 2010;62:2569–81.doi:10.1002/art.27584
      OpenUrlCrossRefPubMedWeb of Science
    24. 24.↵
      2. van den Hoogen F ,
      3. Khanna D ,
      4. Fransen J , et al
      . 2013 classification criteria for systemic sclerosis: an American College of Rheumatology/European League against rheumatism collaborative initiative. Arthritis Rheum 2013;65:2737–47.doi:10.1002/art.38098
      OpenUrlCrossRefPubMedWeb of Science
    25. 25.↵
      2. Hanly JG ,
      3. Urowitz MB ,
      4. Sanchez-Guerrero J , et al
      . Neuropsychiatric events at the time of diagnosis of systemic lupus erythematosus: an international inception cohort study. Arthritis Rheum 2007;56:265–73.doi:10.1002/art.22305
      OpenUrlCrossRefPubMedWeb of Science
    26. 26.↵
      2. Elman SA ,
      3. Joyce C ,
      4. Nyberg F , et al
      . Development of classification criteria for discoid lupus erythematosus: results of a Delphi exercise. J Am Acad Dermatol 2017;77:261–7.doi:10.1016/j.jaad.2017.02.030
      OpenUrlPubMed
    27. 27.↵
      2. Weening JJ ,
      3. D'Agati VD ,
      4. Schwartz MM , et al
      . The classification of glomerulonephritis in systemic lupus erythematosus revisited. Kidney Int 2004;65:521–30.doi:10.1111/j.1523-1755.2004.00443.x
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Handling editor David S Pisetsky

    • Contributors SRJ, DJ, JSS, RPN, MA and KHC were responsible for planning this work. SKT, SRJ, DTB, DD,TD, BD, SJ, DLK, WJM, MM, RR-G, GR-I, MS, MU, DW, JSS, RPN, MA and KHC conducted this work. SKT, SRJ, DTB, DD,TD, BD, SJ, DLK, WJM, MM, RR-G, GR-I, MS, MU, DW, JSS, RPN, MA and KHC contributed to manuscript preparation.

    • Funding This work was jointly supported by the European League Against Rheumatism and the American College of Rheumatology. SKT’s work on this project was supported in part by the Lupus Foundation of America Career Development Award and NIAMS L30 AR070514. SRJ was supported by a Canadian Institutes of Health Research New Investigator Award. SJ was supported by a grant from the Danish Rheumatism Association (A-3865).

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.