Abstract
Summary
Recent studies have evaluated the performance of FRAX® in independent cohorts. The interpretation of most is problematic for reasons summarised in this perspective.
Introduction
FRAX is an extensively validated assessment tool for the prediction of fracture in men and women. The aim of this study was to review the methods used since the launch of FRAX to further evaluate this instrument.
Methods
This covers a critical review of studies investigating the calibration of FRAX or assessing its performance characteristics in external cohorts.
Results
Most studies used inappropriate methodologies to compare the performance characteristics of FRAX with other models. These included discordant parameters of risk (comparing incidence with probabilities), comparison with internally derived predictors and inappropriate use and interpretation of receiver operating characteristic curves. These deficits markedly impair interpretation of these studies.
Conclusion
Cohort studies that have evaluated the performance of FRAX need to be interpreted with caution and preferably re-evaluated.
Similar content being viewed by others
References
Hippisley-Cox J, Coupland C (2009) Predicting risk of osteoporotic fracture in men and women in England and Wales: prospective derivation and validation of QFractures Scores. Br Med J 339:b4229
Chen P, Krege JH, Adachi JD et al (2009) Vertebral fracture status and the World Health Organization risk factors for predicting osteoporotic fracture risk. J Bone Miner 24:495–502
Sornay-Rendu E, Munoz F, Delmas PD, Chapurlat RD (2010) The FRAX® tool in French women: how well does it describe the real incidence of fracture in the OFELY cohort. J Bone Miner Res 25:2101–2107
Bolland MJ, Siu AT, Mason BH et al (2010) (2011) Evaluation of the FRAX and Garvan fracture risk calculators in older women. J Bone Miner Res 26:420–427
Ensrud KE, Lui LY, Taylor BC et al (2009) A comparison of prediction models for fractures in older women: is more better? Arch Intern Med 169:2087–2094
Tremollieres FA, Pouilles JM, Drewniak N, Laparra J, Ribot CA, Dargent-Molina P (2010) Fracture risk prediction using BMD and clinical risk factors in early postmenopausal women: sensitivity of the WHO FRAX tool. J Bone Miner Res 25:1002–1009
Donaldson MG, Palermo L, Schousboe JT, Ensrud KE, Hochberg MC, Cummings SR (2009) FRAX and risk of vertebral fractures: the fracture intervention trial. J Bone Miner Res 24:1793–1799
Hillier TA, Cauley JA, Rizzo JH (2011) The WHO absolute fracture risk models (FRAX): do clinical risk factors improve fracture prediction in older women without osteoporosis? J Bone Miner Res 26:1774–1782
Kayan K, Johansson H, Oden A et al (2009) Can fall risk be incorporated into fracture risk assessment algorithms: a pilot study of responsiveness to clodronate. Osteoporos Int 20:2055–2061
Pluskiewicz W, Adamczyk P, Franek E et al (2010) Ten-year probability of osteoporotic fracture in 2012 Polish women assessed by FRAX and nomogram by Nguyen et al.—conformity between methods and their clinical utility. Bone 46:1661–1667
Kanis JA, Oden A, Johansson H, McCloskey EV (2008) Expressing fracture risk (letter). Osteoporos Int 19:593–594
Kanis JA, Oden A, Johnell O et al (2007) The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women. Osteoporos Int 18:1033–1046
Kanis JA on behalf of the World Health Organization Scientific Group (2008) Assessment of osteoporosis at the primary health-care level. Technical Report. WHO Collaborating Centre, University of Sheffield, UK. Available at http://www.shef.ac.uk/FRAX/
Papaioannou A, Morin S, Cheung AM, Scientific Advisory Council of Osteoporosis Canada (2010) 2010 clinical practice guidelines for the diagnosis and management of osteoporosis in Canada: summary. CMAJ 23(182):864–873
Dawson-Hughes B, National Osteoporosis Foundation Guide Committee (2008) A revised clinician’s guide to the prevention and treatment of osteoporosis. J Clin Endocrinol Metab 93:2463–2465
Grossman JM, Gordon R, Ranganath VK et al (2010) American College of Rheumatology 2010 recommendations for the prevention and treatment of glucocorticoid-induced osteoporosis. Arthritis Care Res (Hoboken) 62:1515–1526
National Osteoporosis Foundation (2008) Clinician’s guide to prevention and treatment of osteoporosis. National Osteoporosis Foundation. www.nof.org, Washington, DC
Pencina MJ, D’Agostino RB, Sr RB, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Statist Med 27:157–172
Kanis JA, Johnell O, Oden A et al (2006) The use of multiple sites for the diagnosis of osteoporosis. Osteoporos Int 17:527–534
Blake GM, Patel R, Knapp KM, Fogelman I (2003) Does the combination of two BMD measurements improve fracture discrimination? J Bone Miner Res 18:1955–1963
Leslie WD, Lix LM, Tsang JF, Caetano PA, Program Manitoba Bone Density (2007) Single-site vs multisite bone density measurement for fracture prediction. Arch Intern Med 167:1641–1647
Leslie WD, Lix LM, Johansson H, Oden A, McCloskey E, Kanis JA (2010) Spine-hip discordance and fracture risk assessment: a physician-friendly FRAX enhancement. Osteoporos Int. Oct 20. [Epub ahead of print] PubMed PMID: 20959961
Abrahamsen B, Vestergaard P, Rud B et al (2006) Ten-year absolute risk of osteoporotic fractures according to BMD T score at menopause: The Danish Osteoporosis Prevention Study. J Bone Miner Res 21:796–800
Baron JA, Barrett J, Malenka D et al (1994) Racial differences in fracture risk. Epidemiol 5:42–47
Jatrana S, Blakely T (2008) Ethnic inequalities in mortality among the elderly in New Zealand. Aust N Z J Public Health 32:437–443
Levine S, Makin M, Menczel J, Robin G, Naor E, Steinberg R (1970) Incidence of fractures of the proximal end of the femur in Jerusalem: a study of ethnic factors. J Bone Joint Surg Am 52:1193–1202
Wittich A, Bagur A, Mautalen C et al (2010) Epidemiology of hip fracture in Tucuman, Argentina. Osteoporos Int 21:1803–1807
Elffors I, Allander E, Kanis JA et al (1994) The variable incidence of hip fracture in southern Europe: the MEDOS Study. Osteoporos Int 4:253–263
Jonsson B, Gardsell P, Johnell O, Redlund-Johnell I, Sernbo I (1992) Differences in fracture pattern between an urban and a rural population: a comparative population-based study in southern Sweden. Osteoporos Int 2:269–273
Finsen V, Benum P (1987) Changing incidence of hip fractures in rural and urban areas of central Norway. Clin Orthop Relat Res 218:104–110
Bulajic-Kopjar M, Wiik J, Nordhagen R (1998) Regional differences in the incidence of femoral neck fractures in Norway. Tidsskr Nor Laegeforen 118:30–33
Chevalley T, Herrmann FR, Delmi et al (2002) Evaluation of the age-adjusted incidence of hip fractures between urban and rural areas: the difference is not related to the prevalence of institutions for the elderly. Osteoporos Int 13:113–118
Matković V, Kostial K, Simonović I, Buzina R, Brodarec A, Nordin BE (1979) Bone status and fracture rates in two regions of Yugoslavia. Am J Clin Nutr 32:540–549
Madhok R, Melton LJ 3rd, Atkinson EJ, O’Fallon WM, Lewallen DG (1993) Urban vs rural increase in hip fracture incidence. Age and sex of 901 cases 1980–89 in Olmsted County, U.S.A. Acta Orthop Scand 64:543–548
Kaastad TS, Meyer HE, Falch JA (2008) Incidence of hip fracture in Oslo, Norway: differences within the city. Bone 22:175–178
Kanis JA, Oden A, Johnell O, Jonsson B, de Laet C, Dawson A (2001) The burden of osteoporotic fractures: a method for setting intervention thresholds. Osteoporos Int 12:417–427
Melton LJ, Crowson CS, O’Fallon WM (1999) Fracture incidence in Olmsted County, Minnesota: comparison of urban and with rural rates and changes in urban rates over time. Osteoporos Int 9:29–37
Singer BR, McLauchlan CJ, Robinson CM, Christie J (1998) Epidemiology of fracture in 15,000 adults. The influence of age and gender. J Bone Joint Surg 80B:234–238
Lippuner K, Johansson H, Kanis JA, Rizzoli R (2010) FRAX® assessment of osteoporotic fracture probability in Switzerland. Osteoporos Int 21:381–390
Sanders KM, Pasco JA, Ugoni AM et al (1998) The exclusion of high trauma fractures may underestimate the prevalence of bone fragility fractures in the community: the Geelong Osteoporosis Study. J Bone Miner Res 13:1337–1342
Melton LJ (1995) Epidemiology of fractures. In: Riggs BL, Melton LJ (eds) Osteoporosis: etiology, diagnosis and management, 2nd edn. Lippincott-Raven, Philadelphia, pp 225–227
Johnell O, Gullberg B, Kanis JA (1997) The hospital burden of vertebral fracture. A study of national register sources. Osteoporos Int 7:138–144
Nguyen ND, Frost SA, Center JR, Eisman JA, Nguyen TV (2008) Development of prognostic nomograms for individualizing 5-year and 10-year fracture risks. Osteoporos Int 19:1431–1444
Nguyen ND, Frost SA, Center JR, Eisman JA, Nguyen TV (2007) Development of a nomogram for individualizing hip fracture risk in men and women. Osteoporos Int 18:1109–1117
Gillespie LD, Robertson MC, Gillespie WJ et al. (2009) Interventions for preventing falls in older people living in the community. Cochrane Database of Systematic Reviews 2009, Issue 2. Art.No.: D007146.doi:10.1002/14651858.CD007146.pub2
McClung MR, Geusens P, Miller PD et al (2001) Effect of risedronate on the risk of hip fracture in elderly women. Hip Intervention Program Study Group. N Engl J Med 344:333–340
Langsetmo L, Nguyen TV, Nguyen ND et al (2010) Independent external validation of nomograms for predicting risk of low-trauma fracture and hip fracture. CMAJ. doi:10.1503/cmaj.100458
Sk S, Nguyen ND, Center JR et al (2010) Prognosis of fracture: evaluation of predictive accuracy of the FRAX algorithm and Garvan nomogram. Osteoporos Int 21:863–871
Van Geel ACM, van den Bergh JPW, Dinant GJ, Geusens PP (2010) Individualizing fracture risk prediction. Maturitas 65:143–148
Geusens P, Van Geel T, Van Den Berg J (2010) Can hip fracture prediction in women be estimated beyond bone mineral density measurement alone. Ther Adv Musculoskelet Dis 2:63–67
Van Geel T, Huntjens K, Bours S et al (2010) Comparison of FRAX and Garvan case finding algorithms in patients presenting with a fracture. Bone 47:S188
Collins GS, Mallett S, Altman DG (2011) Predicting risk of osteoporotic and hip fracture in the United Kingdom: prospective independent and external validation of QFractureScores. BMJ 342:d3651. doi:10.1136/bmj.d3651
Cummins NM, Poku EK, Towler MR, O'Driscoll OM, Ralston SH (2011) Clinical risk factors for osteoporosis in Ireland and the UK: a comparison of FRAX and QFractureScores. Calcif Tissue Int 89:172–177
Fujiwara S, Hamaya E, Goto W, Masunari N, Furukawa K, Fukunaga M, Nakamura T, Miyauchi A, Chen P (2011) Vertebral fracture status and the World Health Organization risk factors for predicting osteoporotic fracture risk in Japan. Bone 49(3):520–525
Conflicts of interest
None.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Derivation and assumptions for Table 4
We assume that the risk variable (the score) before addition has a normal distribution with the same standard deviation for the group without the event as for those with the event. Without loss of generality, it can be assumed that the standard deviation is 1.
The area under the ROC curve will not be constant if we follow individuals for a long time. Here, we assume that the follow-up is short, so only a very small proportion of individuals have had an event. A hazard function describes the momentary risk and a hazard ratio is defined at every moment though the ratio is often assumed to be constant.
The area under the ROC curve equals the probability that a randomly selected individual among those with event has a larger value on the risk variable than a randomly selected individual without event. Let us denote the normally distributed risk variable (score) for an individual with event by X 1 and the corresponding variable for an individual without event by X 2. Then
where Δ is the mean difference between X 1 and X 2 and Φ is the standardized normal distribution function. The above relation implies
where Φ −1 is the inverse to standardized normal distribution function.
The new 0–1 risk variable is denote by Y 1 for those with event and Y 2 for those without event, P(Y 1 = 1) = p 1 and P(Y 2 = 1) = p2. The following notations are used:
a | p 1–p 2 |
b | p 1⋅(1−p 1) + p 2⋅(1−p2) |
c | Φ −1(AUC)⋅√2 |
d | 2 |
The area under the ROC curve for the new predictor will be
where z is a constant making the area under the ROC curve as large as possible.
The expression above equals
which attains its maximum when \( (c + z\;\cdot a)/{(d + {z^{{2}}}\;\cdot b)^{{ \frac{\hbox{$\scriptstyle 1$}}{\hbox{$\scriptstyle 2$}} }}} \) is as large as possible and that expression attains its maximum when the square is as large as possible. To find the z corresponding to the maximum, we take the derivative of
and put the derivative to zero. The solution is
By putting z equal to the value given by expression 2 and calculating the expression 1, we obtain the area under the ROC curve when the new risk variable is used together with the original one.
Rights and permissions
About this article
Cite this article
Kanis, J.A., Oden, A., Johansson, H. et al. Pitfalls in the external validation of FRAX. Osteoporos Int 23, 423–431 (2012). https://doi.org/10.1007/s00198-011-1846-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00198-011-1846-0