Skip to main content

Advertisement

Log in

Pitfalls in the external validation of FRAX

  • Opinion Paper
  • Published:
Osteoporosis International Aims and scope Submit manuscript

Abstract

Summary

Recent studies have evaluated the performance of FRAX® in independent cohorts. The interpretation of most is problematic for reasons summarised in this perspective.

Introduction

FRAX is an extensively validated assessment tool for the prediction of fracture in men and women. The aim of this study was to review the methods used since the launch of FRAX to further evaluate this instrument.

Methods

This covers a critical review of studies investigating the calibration of FRAX or assessing its performance characteristics in external cohorts.

Results

Most studies used inappropriate methodologies to compare the performance characteristics of FRAX with other models. These included discordant parameters of risk (comparing incidence with probabilities), comparison with internally derived predictors and inappropriate use and interpretation of receiver operating characteristic curves. These deficits markedly impair interpretation of these studies.

Conclusion

Cohort studies that have evaluated the performance of FRAX need to be interpreted with caution and preferably re-evaluated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Hippisley-Cox J, Coupland C (2009) Predicting risk of osteoporotic fracture in men and women in England and Wales: prospective derivation and validation of QFractures Scores. Br Med J 339:b4229

    Article  Google Scholar 

  2. Chen P, Krege JH, Adachi JD et al (2009) Vertebral fracture status and the World Health Organization risk factors for predicting osteoporotic fracture risk. J Bone Miner 24:495–502

    Article  Google Scholar 

  3. Sornay-Rendu E, Munoz F, Delmas PD, Chapurlat RD (2010) The FRAX® tool in French women: how well does it describe the real incidence of fracture in the OFELY cohort. J Bone Miner Res 25:2101–2107

    Article  PubMed  Google Scholar 

  4. Bolland MJ, Siu AT, Mason BH et al (2010) (2011) Evaluation of the FRAX and Garvan fracture risk calculators in older women. J Bone Miner Res 26:420–427

    Article  Google Scholar 

  5. Ensrud KE, Lui LY, Taylor BC et al (2009) A comparison of prediction models for fractures in older women: is more better? Arch Intern Med 169:2087–2094

    Article  PubMed  Google Scholar 

  6. Tremollieres FA, Pouilles JM, Drewniak N, Laparra J, Ribot CA, Dargent-Molina P (2010) Fracture risk prediction using BMD and clinical risk factors in early postmenopausal women: sensitivity of the WHO FRAX tool. J Bone Miner Res 25:1002–1009

    Article  PubMed  Google Scholar 

  7. Donaldson MG, Palermo L, Schousboe JT, Ensrud KE, Hochberg MC, Cummings SR (2009) FRAX and risk of vertebral fractures: the fracture intervention trial. J Bone Miner Res 24:1793–1799

    Article  PubMed  Google Scholar 

  8. Hillier TA, Cauley JA, Rizzo JH (2011) The WHO absolute fracture risk models (FRAX): do clinical risk factors improve fracture prediction in older women without osteoporosis? J Bone Miner Res 26:1774–1782

    Article  PubMed  Google Scholar 

  9. Kayan K, Johansson H, Oden A et al (2009) Can fall risk be incorporated into fracture risk assessment algorithms: a pilot study of responsiveness to clodronate. Osteoporos Int 20:2055–2061

    Article  PubMed  CAS  Google Scholar 

  10. Pluskiewicz W, Adamczyk P, Franek E et al (2010) Ten-year probability of osteoporotic fracture in 2012 Polish women assessed by FRAX and nomogram by Nguyen et al.—conformity between methods and their clinical utility. Bone 46:1661–1667

    Article  PubMed  CAS  Google Scholar 

  11. Kanis JA, Oden A, Johansson H, McCloskey EV (2008) Expressing fracture risk (letter). Osteoporos Int 19:593–594

    Article  Google Scholar 

  12. Kanis JA, Oden A, Johnell O et al (2007) The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women. Osteoporos Int 18:1033–1046

    Article  PubMed  CAS  Google Scholar 

  13. Kanis JA on behalf of the World Health Organization Scientific Group (2008) Assessment of osteoporosis at the primary health-care level. Technical Report. WHO Collaborating Centre, University of Sheffield, UK. Available at http://www.shef.ac.uk/FRAX/

  14. Papaioannou A, Morin S, Cheung AM, Scientific Advisory Council of Osteoporosis Canada (2010) 2010 clinical practice guidelines for the diagnosis and management of osteoporosis in Canada: summary. CMAJ 23(182):864–873

    Google Scholar 

  15. Dawson-Hughes B, National Osteoporosis Foundation Guide Committee (2008) A revised clinician’s guide to the prevention and treatment of osteoporosis. J Clin Endocrinol Metab 93:2463–2465

    Article  PubMed  CAS  Google Scholar 

  16. Grossman JM, Gordon R, Ranganath VK et al (2010) American College of Rheumatology 2010 recommendations for the prevention and treatment of glucocorticoid-induced osteoporosis. Arthritis Care Res (Hoboken) 62:1515–1526

    Article  Google Scholar 

  17. National Osteoporosis Foundation (2008) Clinician’s guide to prevention and treatment of osteoporosis. National Osteoporosis Foundation. www.nof.org, Washington, DC

    Google Scholar 

  18. Pencina MJ, D’Agostino RB, Sr RB, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Statist Med 27:157–172

    Article  Google Scholar 

  19. Kanis JA, Johnell O, Oden A et al (2006) The use of multiple sites for the diagnosis of osteoporosis. Osteoporos Int 17:527–534

    Article  PubMed  CAS  Google Scholar 

  20. Blake GM, Patel R, Knapp KM, Fogelman I (2003) Does the combination of two BMD measurements improve fracture discrimination? J Bone Miner Res 18:1955–1963

    Article  PubMed  Google Scholar 

  21. Leslie WD, Lix LM, Tsang JF, Caetano PA, Program Manitoba Bone Density (2007) Single-site vs multisite bone density measurement for fracture prediction. Arch Intern Med 167:1641–1647

    Article  PubMed  Google Scholar 

  22. Leslie WD, Lix LM, Johansson H, Oden A, McCloskey E, Kanis JA (2010) Spine-hip discordance and fracture risk assessment: a physician-friendly FRAX enhancement. Osteoporos Int. Oct 20. [Epub ahead of print] PubMed PMID: 20959961

  23. Abrahamsen B, Vestergaard P, Rud B et al (2006) Ten-year absolute risk of osteoporotic fractures according to BMD T score at menopause: The Danish Osteoporosis Prevention Study. J Bone Miner Res 21:796–800

    Article  PubMed  Google Scholar 

  24. Baron JA, Barrett J, Malenka D et al (1994) Racial differences in fracture risk. Epidemiol 5:42–47

    Article  CAS  Google Scholar 

  25. Jatrana S, Blakely T (2008) Ethnic inequalities in mortality among the elderly in New Zealand. Aust N Z J Public Health 32:437–443

    Article  PubMed  Google Scholar 

  26. Levine S, Makin M, Menczel J, Robin G, Naor E, Steinberg R (1970) Incidence of fractures of the proximal end of the femur in Jerusalem: a study of ethnic factors. J Bone Joint Surg Am 52:1193–1202

    PubMed  CAS  Google Scholar 

  27. Wittich A, Bagur A, Mautalen C et al (2010) Epidemiology of hip fracture in Tucuman, Argentina. Osteoporos Int 21:1803–1807

    Article  PubMed  CAS  Google Scholar 

  28. Elffors I, Allander E, Kanis JA et al (1994) The variable incidence of hip fracture in southern Europe: the MEDOS Study. Osteoporos Int 4:253–263

    Article  PubMed  CAS  Google Scholar 

  29. Jonsson B, Gardsell P, Johnell O, Redlund-Johnell I, Sernbo I (1992) Differences in fracture pattern between an urban and a rural population: a comparative population-based study in southern Sweden. Osteoporos Int 2:269–273

    Article  PubMed  CAS  Google Scholar 

  30. Finsen V, Benum P (1987) Changing incidence of hip fractures in rural and urban areas of central Norway. Clin Orthop Relat Res 218:104–110

    PubMed  Google Scholar 

  31. Bulajic-Kopjar M, Wiik J, Nordhagen R (1998) Regional differences in the incidence of femoral neck fractures in Norway. Tidsskr Nor Laegeforen 118:30–33

    PubMed  CAS  Google Scholar 

  32. Chevalley T, Herrmann FR, Delmi et al (2002) Evaluation of the age-adjusted incidence of hip fractures between urban and rural areas: the difference is not related to the prevalence of institutions for the elderly. Osteoporos Int 13:113–118

    Article  PubMed  CAS  Google Scholar 

  33. Matković V, Kostial K, Simonović I, Buzina R, Brodarec A, Nordin BE (1979) Bone status and fracture rates in two regions of Yugoslavia. Am J Clin Nutr 32:540–549

    PubMed  Google Scholar 

  34. Madhok R, Melton LJ 3rd, Atkinson EJ, O’Fallon WM, Lewallen DG (1993) Urban vs rural increase in hip fracture incidence. Age and sex of 901 cases 1980–89 in Olmsted County, U.S.A. Acta Orthop Scand 64:543–548

    Article  PubMed  CAS  Google Scholar 

  35. Kaastad TS, Meyer HE, Falch JA (2008) Incidence of hip fracture in Oslo, Norway: differences within the city. Bone 22:175–178

    Article  Google Scholar 

  36. Kanis JA, Oden A, Johnell O, Jonsson B, de Laet C, Dawson A (2001) The burden of osteoporotic fractures: a method for setting intervention thresholds. Osteoporos Int 12:417–427

    Article  PubMed  CAS  Google Scholar 

  37. Melton LJ, Crowson CS, O’Fallon WM (1999) Fracture incidence in Olmsted County, Minnesota: comparison of urban and with rural rates and changes in urban rates over time. Osteoporos Int 9:29–37

    Article  PubMed  Google Scholar 

  38. Singer BR, McLauchlan CJ, Robinson CM, Christie J (1998) Epidemiology of fracture in 15,000 adults. The influence of age and gender. J Bone Joint Surg 80B:234–238

    Google Scholar 

  39. Lippuner K, Johansson H, Kanis JA, Rizzoli R (2010) FRAX® assessment of osteoporotic fracture probability in Switzerland. Osteoporos Int 21:381–390

    Article  PubMed  CAS  Google Scholar 

  40. Sanders KM, Pasco JA, Ugoni AM et al (1998) The exclusion of high trauma fractures may underestimate the prevalence of bone fragility fractures in the community: the Geelong Osteoporosis Study. J Bone Miner Res 13:1337–1342

    Article  PubMed  CAS  Google Scholar 

  41. Melton LJ (1995) Epidemiology of fractures. In: Riggs BL, Melton LJ (eds) Osteoporosis: etiology, diagnosis and management, 2nd edn. Lippincott-Raven, Philadelphia, pp 225–227

    Google Scholar 

  42. Johnell O, Gullberg B, Kanis JA (1997) The hospital burden of vertebral fracture. A study of national register sources. Osteoporos Int 7:138–144

    Article  PubMed  CAS  Google Scholar 

  43. Nguyen ND, Frost SA, Center JR, Eisman JA, Nguyen TV (2008) Development of prognostic nomograms for individualizing 5-year and 10-year fracture risks. Osteoporos Int 19:1431–1444

    Article  PubMed  CAS  Google Scholar 

  44. Nguyen ND, Frost SA, Center JR, Eisman JA, Nguyen TV (2007) Development of a nomogram for individualizing hip fracture risk in men and women. Osteoporos Int 18:1109–1117

    Article  PubMed  CAS  Google Scholar 

  45. Gillespie LD, Robertson MC, Gillespie WJ et al. (2009) Interventions for preventing falls in older people living in the community. Cochrane Database of Systematic Reviews 2009, Issue 2. Art.No.: D007146.doi:10.1002/14651858.CD007146.pub2

  46. McClung MR, Geusens P, Miller PD et al (2001) Effect of risedronate on the risk of hip fracture in elderly women. Hip Intervention Program Study Group. N Engl J Med 344:333–340

    Article  PubMed  CAS  Google Scholar 

  47. Langsetmo L, Nguyen TV, Nguyen ND et al (2010) Independent external validation of nomograms for predicting risk of low-trauma fracture and hip fracture. CMAJ. doi:10.1503/cmaj.100458

  48. Sk S, Nguyen ND, Center JR et al (2010) Prognosis of fracture: evaluation of predictive accuracy of the FRAX algorithm and Garvan nomogram. Osteoporos Int 21:863–871

    Article  Google Scholar 

  49. Van Geel ACM, van den Bergh JPW, Dinant GJ, Geusens PP (2010) Individualizing fracture risk prediction. Maturitas 65:143–148

    Article  PubMed  Google Scholar 

  50. Geusens P, Van Geel T, Van Den Berg J (2010) Can hip fracture prediction in women be estimated beyond bone mineral density measurement alone. Ther Adv Musculoskelet Dis 2:63–67

    Article  Google Scholar 

  51. Van Geel T, Huntjens K, Bours S et al (2010) Comparison of FRAX and Garvan case finding algorithms in patients presenting with a fracture. Bone 47:S188

    Article  Google Scholar 

  52. Collins GS, Mallett S, Altman DG (2011) Predicting risk of osteoporotic and hip fracture in the United Kingdom: prospective independent and external validation of QFractureScores. BMJ 342:d3651. doi:10.1136/bmj.d3651

  53. Cummins NM, Poku EK, Towler MR, O'Driscoll OM, Ralston SH (2011) Clinical risk factors for osteoporosis in Ireland and the UK: a comparison of FRAX and QFractureScores. Calcif Tissue Int 89:172–177

    Google Scholar 

  54. Fujiwara S, Hamaya E, Goto W, Masunari N, Furukawa K, Fukunaga M, Nakamura T, Miyauchi A, Chen P (2011) Vertebral fracture status and the World Health Organization risk factors for predicting osteoporotic fracture risk in Japan. Bone 49(3):520–525

    Google Scholar 

Download references

Conflicts of interest

None.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to J. A. Kanis.

Appendix

Appendix

Derivation and assumptions for Table 4

We assume that the risk variable (the score) before addition has a normal distribution with the same standard deviation for the group without the event as for those with the event. Without loss of generality, it can be assumed that the standard deviation is 1.

The area under the ROC curve will not be constant if we follow individuals for a long time. Here, we assume that the follow-up is short, so only a very small proportion of individuals have had an event. A hazard function describes the momentary risk and a hazard ratio is defined at every moment though the ratio is often assumed to be constant.

The area under the ROC curve equals the probability that a randomly selected individual among those with event has a larger value on the risk variable than a randomly selected individual without event. Let us denote the normally distributed risk variable (score) for an individual with event by X 1 and the corresponding variable for an individual without event by X 2. Then

$$ {\text{AUC}} = P\left( {{X_{{1}}}{ > }{X_{{2}}}} \right) = \Phi (\Delta /\surd {2}), $$

where Δ is the mean difference between X 1 and X 2 and Φ is the standardized normal distribution function. The above relation implies

$$ \Delta = {\Phi^{{ - {1}}}}\left( {\text{AUC}} \right)\cdot\surd {2}, $$

where Φ −1 is the inverse to standardized normal distribution function.

The new 0–1 risk variable is denote by Y 1 for those with event and Y 2 for those without event, P(Y 1 = 1) = p 1 and P(Y 2 = 1) = p2. The following notations are used:

a

p 1p 2

b

p 1⋅(1−p 1) + p 2⋅(1−p2)

c

Φ −1(AUC)⋅√2

d

2

The area under the ROC curve for the new predictor will be

$$ P({X_{{1}}} + z\;\cdot {Y_{{1}}} > {X_{{2}}} + z\;\cdot {Y_{{2}}}), $$

where z is a constant making the area under the ROC curve as large as possible.

The expression above equals

$$ \Phi ((c + z\;\cdot a)/{(d + {z^{{2}}}\;\cdot b)^{{ \frac{\hbox{$\scriptstyle 1$}}{\hbox{$\scriptstyle 2$}} }}}), $$
(1)

which attains its maximum when \( (c + z\;\cdot a)/{(d + {z^{{2}}}\;\cdot b)^{{ \frac{\hbox{$\scriptstyle 1$}}{\hbox{$\scriptstyle 2$}} }}} \) is as large as possible and that expression attains its maximum when the square is as large as possible. To find the z corresponding to the maximum, we take the derivative of

$$ ({c^{{2}}} + {2}\;\cdot a\;\cdot c\;\cdot z + {a^{{2}}}\;\cdot{z^{{2}}})/(d + b\;\cdot{z^{{2}}}) $$

and put the derivative to zero. The solution is

$$ z = \frac{{{a^2} \cdot d - b \cdot {c^2}}}{{2 \cdot a \cdot b \cdot c}} + {(\frac{d}{b} + {(\frac{{{a^2} \cdot d - b \cdot {c^2}}}{{2 \cdot a \cdot b \cdot c}})^2})^{{ \frac{\hbox{$\scriptstyle 1$}}{\hbox{$\scriptstyle 2$}} }}} $$
(2)

By putting z equal to the value given by expression 2 and calculating the expression 1, we obtain the area under the ROC curve when the new risk variable is used together with the original one.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kanis, J.A., Oden, A., Johansson, H. et al. Pitfalls in the external validation of FRAX. Osteoporos Int 23, 423–431 (2012). https://doi.org/10.1007/s00198-011-1846-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00198-011-1846-0

Keywords

Navigation