Article Text
PDF
Abstract
Background Rheumatoid factor (RF) and anti-citrullinated protein/peptide antibodies (ACPA) are important biomarkers for diagnosis of rheumatoid arthritis (RA). However, there is poor harmonisation of RF and ACPA assays. The aim of this study was to refine RF and ACPA interpretation across commercial assays.
Materials and methods Six total RF isotype-non-specific assays, 3 RF IgM isotype-specific assays and 9 ACPA immunoglobulin G assays of 13 different companies were evaluated using 398 diagnostic samples from patients with RA and 1073 disease controls.
Results Using cut-offs proposed by the manufacturer, there was a large variability in diagnostic sensitivity and specificity between assays. Thresholds of antibody levels were determined based on predefined specificities and used to define test result intervals. Test result interval-specific likelihood ratios (LRs) were concordant across the different RF and ACPA assays. For all assays, the LR for RA increased with increasing antibody level. Higher LRs were found for ACPA than for RF. ACPA levels associated with LRs >80 were found in a substantial fraction (>22%) of patients with RA.
Conclusion Defining thresholds for antibody levels and assigning test result interval-specific LRs allows alignment of clinical interpretation for all RF and ACPA assays.
- rheumatoid factor
- rheumatoid arthritis
- anti-citrullinated protein antibodies
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Key messages
What is already known about this subject?
There is poor harmonisation of rheumatoid factor (RF) and anti-citrullinated protein/peptide antibodies (ACPA) assays with large variability in sensitivity and specificity.
What does this study add?
Variability in diagnostic sensitivity between RF and ACPA assays can be reduced by defining thresholds that are based on predefined specificity.
Thresholds of antibody levels based on predefined specificities were used to define test result intervals.
Test result interval-specific likelihood ratios (LRs) were concordant across RF and ACPA assays.
How might this impact on clinical practice or further developments?
Defining thresholds for antibody levels and assigning test result interval-specific LRs aligns clinical interpretation of RF/ACPA assays.
Introduction
Rheumatoid arthritis (RA) is a chronic, systemic autoimmune disease that affects approximately 1% of the population. It is characterised by joint inflammation that can lead to cartilage and bone damage if left untreated and can potentially result in disability, reduction of quality of life and increased mortality.1 Early identification of RA (even in the preclinical phase) is critical as early intervention results in better long-term disease control and can prevent joint damage.2–8 Identifying a person with RA can be challenging, even for expert physicians.9 10 Clinical signs and symptoms alone are usually insufficient to diagnose RA. The presence of rheumatoid factor (RF) and anti-cyclic citrullinated protein/peptide antibodies (ACPA) in serum supports the diagnosis of RA.11 However, there is poor agreement among the currently available RF and ACPA assays, which may have an impact on RA diagnosis and classification of a patient.12–14 This poor agreement not only relates to heterogeneity in test characteristics (nature of antigen, assay set up, affinity/avidity and antibody detection methodology),15 but also, and largely, to heterogeneity in the way the cut-off is set. Differences in cut-off values among assays were clearly demonstrated by the fact that the international standard for RF (W1066) scored either negative, equivocal or positive, depending on the assay used.13 Thus, there is an evident need to harmonise clinical interpretation of autoantibody tests used for RA diagnosis.
The aim of this international multicentre study is to harmonise interpretation of commonly used commercially available RF and ACPA assays. In order to overcome differences in the arbitrary units of the different assays, we applied the concept of likelihood ratio (LR), which is a unit-independent method to express test results.16–18 The LR is defined as the fraction of patients with a particular test result divided by the fraction of controls with the same test result. For example, a test result with an LR of 10 indicates that this test result is 10 times more likely to be found in patients with the disease than in (disease) controls, whereas a test result with an LR of 0.1 is 10 times less likely to be found in patients with the disease than in (disease) controls. The LR allows to convey immediately clinically relevant information related to the antibody level. The higher the antibody level, the higher the LR for disease.13 19–21 Based on the LR, the post-test probability of disease can be estimated (post-test odds=pre-test odds×LR).20
Materials and methods
Patients and samples
Serum samples were obtained from 11 European hospitals: Division of Rheumatology, Medical University of Vienna (Austria), University Hospital of Leuven (Belgium), University Hospital of Ghent (Belgium), OLV Hospital of Aalst (Belgium), National Institute of Rheumatology and Physiotherapy of Budapest (Hungary), Centre Hospitalier de Luxembourg (Luxembourg), University Medical Centre of Ljubljana (Slovenia), Sahlgrenska Academy Hospital of Gothenburg (Sweden), University Hospital of Linköping (Sweden), University Hospital of Basel (Switzerland) and Kantonsspital of Aarau (Switzerland). An overview of the patients with RA and controls recruited at each institute is given in online supplemental table 1.
Supplemental material
The RA cohort (n=398) consisted of consecutive newly diagnosed patients with RA. A patient with synovitis was considered to have RA when no alternative diagnosis could better explain the symptoms, and when the treating rheumatologist initiated methotrexate treatment or other disease-modifying antirheumatic drugs. Demographic data (age and sex) and the clinical data included in the ACR/EULAR RA classification criteria (number and type of joints affected, duration of symptoms, C reactive protein and erythrocyte sedimentation rate) were recorded. Descriptive characteristics of the RA cohort are provided in online supplemental table 2.
The disease control cohort (n=1073) consisted of (a) a rheumatologic disease control group (RDCG) (ie, consecutive patients consulting a rheumatology clinic for the first time but in whom RA was eventually excluded; the exact composition is listed in online supplemental table 1) (n=656); (b) specific disease control groups (DCG) (ie, patients with established diagnoses22–30 of antineutrophil cytoplasmic antibody (ANCA) associated vasculitis (n=24) with arthritis, osteoarthritis (n=25), psoriatic arthritis (n=25), reactive arthritis (n=20), spondyloarthritis (n=25), systemic lupus erythematosus (SLE) (n=50) and primary Sjögren’s syndrome (pSS)) (n=48) and (c) a healthy control (HC) group (n=200) (anonymised blood donors from the Red Cross-Flanders (2008) (n=50) and healthy individuals recruited by OLV Hospital Aalst (n=150)). Sample collection complied with the World Medical Association’s Declaration of Helsinki (as revised in 2013). Due to the unavailability of reagents or an instrument in Belgium at the timepoint of the study, the analyses of, respectively, ACPA Bio-Rad and RF/ACPA Siemens Healthineers (hereafter referred by Siemens) were performed after the 19 analyses batches of the other analysis runs. These analyses were performed on a smaller cohort of samples consisting of 362 patients with RA, 594 RDCG, 206 DCG and 196 HC samples. An overview of the demographic features of the different study cohorts (RA and controls) is included in online supplemental table 3.
RF and ACPA assays
All samples were analysed with six total RF isotype-non-specific assays, three RF IgM isotype-specific assays and nine ACPA immunoglobulin (IgG) assays. An overview of the different RF and ACPA assays and their specific test characteristics is provided in online supplemental table 4. The tests were performed by the OLV Hospital Aalst according to the manufacturers’ instructions. Results were expressed in the arbitrary units of the respective assay.
For RF, the RF EliA IgM on Phadia 250 (Thermo Fisher Scientific, Sweden), RF‐II (Roche Diagnostics, Germany) and Diagam RF (Diagam, Belgium) on a cobas c 501 analyser (Roche Diagnostics), AUTOZYME RF IgM (Cambridge Life Science, UK) on a QUANTA-Lyser two instrument (Inova Diagnostics, San Diego, California, USA), Alegria RF IgM on Alegria instrument (Orgentec, Germany), RF on Alinity c (Abbott, Germany), RF on Vitros 4600 (Ortho-Clinical diagnostics, Raritan, New York, USA), RF on AU680 (Beckman Coulter, Brea, California, USA), and RF on Atellica CH-930 (Siemens, UK) were evaluated.
For ACPA, CCP EliA IgG on Phadia 250, anti-CCP IgG on a cobas e 601 analyser, IMMUNOSCAN CCPlus (Svar Life Science, Sweden) on a QUANTA-Lyser 2 instrument, CCP IgG on IDS-iSYS (Immunodiagnostic Systems, UK), anti-CCP hs on Alegria (Orgentec), anti-CCP IgG on Alinity i, anti-CCP ELISA (IgG) on Euroimmun Analyzer I-2P (Euroimmun, Germany), anti-CCP IgG on BioPlex 2200 (Bio-Rad Laboratories, Hercules, California, USA) and anti-CCP IgG on Atellica IM 1300 (Siemens, UK) were included. All included ACPA assays were CCP2 tests (online supplemental table 4).
Statistical analysis
Diagnostic performance was evaluated by sensitivity, specificity and (test result interval-specific) LR. Statistics were performed using MEDCALC (V.17.1, Ostend, Belgium) and Analyse-it for Microsoft Excel (V.4.90).
Results
Performance characteristics of RF and ACPA
For all methods, RF and ACPA levels were significantly higher in patients with RA than in controls, except for RF in pSS (online supplemental table 5, supplemental figure 1A,B, supplemental table 6, supplemental figure 2A,B).
Based on the cut-off values proposed by the manufacturer, the sensitivity of the RF assays ranged from 51.8% to 74.4%, the specificity from 72.4% to 93.9% and the LR for RA from 2.7 to 9.0 (inter-manufacturer variation (coefficient of variation, CV%) in LR for RA was 29.5%; table 1). The sensitivity of the RF IgM assays (range 51.8%–62.3%) was comparable to the sensitivity of total RF assays (range: 51.8%–74.4%).
Performance characteristics of RF assays at the cut-off defined by the manufacturer. 95% CIs are presented within brackets
Based on the cut-off values proposed by the manufacturer, the sensitivity of the ACPA assays ranged from 57.8% to 64.6%, the specificity from 94.9% to 97.8% and the LR for RA from 12.6 to 26.6 (CV% of LR for RA was 21.3%; table 2).
Performance characteristics of ACPA assays at the cut-off defined by the manufacturer. 95% CIs are presented between brackets
At a threshold corresponding to a specificity of 92.5% and 97.5%, the sensitivity of the RF tests ranged from 49.5% to 61.6% and 22.6% to 39.2%, respectively, with a corresponding range in LR from, respectively, 6.7 to 8.2 (CV% of LR for RA was 7.5%) and 9.0 to 15.6 (CV% of LR for RA was 15.7%) (online supplemental table 7A). A substantial fraction (>22%) of the patients with RA had total RF and RF IgM results exceeding the 97.5% specificity threshold.
At a threshold corresponding to a specificity of 97.5% and 99.0%, the sensitivity of ACPA tests ranged from 58.8% to 63.6% and 41.5% to 50.5%, respectively, with a corresponding range in LR from, respectively, 23.4 to 25.3 (CV% of LR for RA was 2.4%) and 40.4 to 49.3 (CV% of LR for RA was 7.5%) (online supplemental table 7B). A substantial fraction of the patients with RA (41.5%–51.7%) had ACPA values exceeding the 99% specificity threshold.
The range of sensitivities associated with a predefined specificity was narrower than the range of sensitivities associated with the cut-offs proposed by the manufacturer.
Test result interval-specific LR
Next, we evaluated how the LR for RA increases with increasing antibody level. In order to determine test result intervals in a consistent manner across assays, we defined intervals that are delimited by thresholds that correspond to predefined specificities (90.0%, 92.5%, 95.0% and 97.5% for RF and 90.0%, 97.5%, 99.0% and 99.8% for ACPA). Online supplemental table 7 shows the sensitivities and LR associated with the various predefined specificities. Online supplemental table 8 shows the sensitivities and LRs after recalculations based on different compositions of the control population. The thresholds that correspond to a high specificity (eg, 97.5%) were lower for an HC population compared with a diseased control population.
For each test result interval, the interval-specific LR for RA was calculated, as well as the fraction of patients and controls that had a result within this interval. The results are presented for RF and ACPA in, respectively, tables 3 and 4 and illustrated for one RF and ACPA assay in, respectively, figures 1 and 2.
Box whisker plots of the likelihood ratio of serum rheumatoid factor (RF) for the different test result-specific intervals, delimited by thresholds that correspond to predefined specificities (90.0%, 92.5%, 95.0% and 97.5%). See table 3 for further details. Results are shown for the Beckman Coulter RF assay, but similar results were obtained for the other RF assays included in the study.
Box whisker plots of the likelihood ratio of serum anti-citrullinated protein/peptide antibodies (ACPA) for the different test result-specific intervals, delimited by thresholds that correspond to predefined specificities (90.0%, 97.5%, 99.0% and 99.8%). See table 4 for further details. Results are shown for the Orgentec ACPA assay, but similar results were obtained for the other ACPA assays included in the study.
RF test result-specific LRs
ACPA test result-specific LRs
The LR for RF test results that were below the 90.0% specificity threshold varied between 0.39 and 0.51, depending on the assay. The LRs of the RF test result interval delimited by thresholds corresponding to a specificity of 90.0% and 92.5% (found in 2.3%–7.8% of patients with RA) ranged from 0.93 to 3.77 (which is close to 1; most of the corresponding 95% CIs included 1). The LRs further increased with increasing antibody levels (figures 1 and 2). It ranged from 2.30 to 4.15 and from 4.89 to 9.29 for the test result intervals delimited by thresholds corresponding to 92.5%–95.0% specificity and 95.0%–97.5% specificity, respectively. A substantial fraction (>22% for IgM or total RF) of the patients with RA had RF results exceeding the 97.5% specificity threshold, with LRs ranging from 8.99 to 15.58.
The LR for ACPA test results that were below the 90.0% specificity threshold varied between 0.30 and 0.39, depending on the assay. The LR of the ACPA test result interval delimited by thresholds corresponding to specificities of 97.5% (found in 3%–10% of the patients) ranged from 0.46 to 1.54 (which is close to 1; 95% CI included 1). The LR increased with increasing ACPA antibody levels (figures 1 and 2). It ranged from 7.08 to 12.97, from 23.93 to 37.07 and from 79.98 to 106.90 for the ACPA test result intervals delimited by thresholds corresponding to 97.5%–99.0% specificity, 99.0%–99.8% specificity and >99.8% specificity, respectively. For some assays (Roche ACPA, Bio-Rad ACPA and Siemens ACPA), the 99.8% specificity threshold could not be determined and the LR for values exceeding the 99.0% specificity threshold ranged from 40.44 to 47.60. A substantial fraction of the patients with RA (41.5%–51.7%) had ACPA values exceeding the 99.0% specificity threshold.
The LR depends on the composition of the patient and control population included. We recalculated the test result interval-specific LRs for different compositions of the control population (HCs; consecutive rheumatologic disease controls; and all control groups excluding HCs, pSS and SLE groups). The results are summarised in online supplemental tables 9 and 10 for, respectively, RF and ACPA.
Discussion
Even though many studies have attempted to discover and validate novel biomarkers for RA,6 ACPA and RF remain the most well-established and widely available RA markers for diagnosis and classification.31 32 However, there is great variability among commercially available RF and ACPA assays.12–14 This is mainly related to differences in the way companies define cut-off values and to the differences in the patient and control populations used to establish the cut-off values. Importantly, the variability among RF and ACPA assays affects patient RA classification according to 2010 ACR/EULAR criteria13 33 and jeopardises RA diagnosis in routine rheumatological practice.11
In order to overcome these shortcomings, we established an international cohort of samples obtained from RA patients at diagnosis and from controls and defined test result-specific LRs at predefined specificity thresholds. In that way, we could align interpretation across companies. Test result-specific LRs bear more clinically relevant information than the dichotomous information associated with a single cut-off value. Our study demonstrated remarkably similar LRs across the different companies at predefined specificity thresholds (table 4). Reporting test resul-specific LRs helps with the interpretation of autoantibody tests (and thus creates added value).16 17 A test result with an LR of 1 indicates that this test result is equally likely to be found in (disease) controls as in patients with the disease. Generally, an LR >10 is considered to support a diagnosis and an LR <0.1 to rule out a diagnosis.34 It is important that laboratory professionals and clinicians become more familiar with the concept of LRs and that they develop an intuitive feeling for the clinical relevance of an LR.18 There is increasing awareness that efforts should be undertaken to harmonise interpretation of RF and ACPA test results. Rönnelid et al recently stated that ‘complementing harmonised cut-offs with information about test result-specific likelihood ratios … will increase the richness and information value of autoantibody data …’.35 The concept of test result-specific LRs has recently also been proposed to harmonise interpretation of proteinase-3 and myeloperoxidase ANCA.36–38
The companies that participated in the study support the concept of test result-specific LRs and will evaluate options for providing that information to their customers, for example, by the organisation of scientific seminars, white papers or product flyers. Laboratory professionals are encouraged to report the LR associated with a particular test result obtained with a specific assay (manufacturer). Data presented in online supplemental tables 9 and 10 allow to report the LR that best fits with the patient population served by the laboratory (eg, tertiary centre vs primary care) (through selection of the LR derived from the most appropriate control group).
When test result-specific LRs are applied, relevant differences in clinical significance between low antibody levels versus higher antibody levels become apparent. The higher the antibody level, the higher the LR and the higher the likelihood for disease.20 21 Moreover, when test result-specific LRs are applied, differences in the clinical value between high levels of RF and ACPA become apparent as well. For example, a high antibody level, that is, a value exceeding the 97.5% specificity (for RF) and 99% specificity (for ACPA) threshold was associated with, respectively, an LR of 7.0–15.6 for RF and an LR of 40.4–49.3 for ACPA, depending on the test kit used. Cut-off values for RF and ACPA are not aligned across manufacturers. Relatedly, cut-offs defining ‘high antibody positivity’ based on three times the manufacturer’s cut-off (as applied in the 2010 classification criteria11) are arbitrarily chosen and not aligned as well.13 In RA diagnosis, serology plays a crucial role in the diagnostic assessment of patients with less typical symptoms, that is, with a lower pre-test probability.11 Using test result-specific LRs on a nomogram, one can calculate the post-test probability of disease.11 32 34 Consequently, the use of harmonised LRs directly impacts clinical decision-making.
In a subsequent study, we show the impact of using specificity-based thresholds on RA classification. In that study, we also show that RA classification can be improved by taking into account the nature of the antibody (RF vs ACPA), the antibody level and single or double positivity (manuscript in preparation).
In conclusion, defining thresholds for antibody levels and assigning test result-specific LRs helps to harmonise interpretation of RF and ACPA immunoassays in routine clinical practice. The LR is dependent on the antibody level, and the antibody type (higher LR for ACPA than for RF). Applying the same criteria on the same cohort of patients and controls is a powerful tool to align interpretation across assays from different companies. It is the shared societal responsibility of laboratory professionals, in vitro diagnostics and clinicians to harmonise interpretation of laboratory test results such that they can be reliably applied in routine diagnostic protocols.39
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and the study was approved by the institutional ethical boards of the University Hospitals Leuven (B322201939877) and OLV Ziekenhuis Aalst. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
We thank all participating diagnostic companies for in-kind support with assays, technical training and constructive discussions. Furthermore, we are very thankful to the laboratory technicians of all participating laboratories for their most appreciated assistance in the performance of the rheumatoid factor/anti-cyclic citrullinated protein/peptide antibodies analyses.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Correction notice This article has been updated since it was first published. The corresponding authors have been updated.
Contributors LVH, XB, BVC, DS, GS and PV contributed to the study conception and design. Patient samples were selected by LVH, BVC, DS, DA, PV, CB, EN, RP, SC, CD, IH, LBernosconi, FB, CW, RM, DK, CS, AK, RJ, BJ, EK and PJ. Reference material was provided by PSG and by National Institute for Biological Standards and Control (LS). Material preparation and data collection was performed by SVDB, LBogaert, LVH, AVL, GV and JR; data analysis by LVH and XB. Manuscript was prepared by LVH, XB, DS and GS and commented by all authors. The final manuscript was read and approved by all authors. XB is guarantor of the study.
Funding All participating diagnostic companies in-kind supported with rheumatoid factor/anti-cyclic citrullinated protein/peptide antibodies assays and provided technical training and support free of charge (see competing interests).
Competing interests XB, LVH, GS and DS have received speaker fees from and have been a consultant for Thermo Fisher Scientific. LBernasconi and IH have received speaker fees from Thermo Fisher Scientific. DA and PJ are editorial board members of RMD Open. All participating diagnostic companies in-kind supported with RF/ACPA assays and technical training: Thermo Fisher Scientific, Uppsala, Sweden; Cambridge Life Science, Ely, UK; Ortho-Clinical Diagnostics, Raritan, New Jersey, USA; Diagam, Ghislenghien, Belgium; Roche Diagnostics, Mannheim, Germany; Svar Life Science, Malmö, Sweden; Immunodiagnostic Systems, Tyne and Wear, UK; Orgentec, Mainz, Germany; Abbott, Wiesbaden, Germany; Euroimmun, Lübeck, Germany; Bio-Rad Laboratories, Hercules, California, USA; and Siemens Healthineers, Sudbury, UK.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.