The Boards of Medical Specialties are collegial bodies formed by prominent medical professionals who cultivate the same subject and who establish a certification and recertification process. The Mexican Board of Rheumatology, AC (CMR), was founded on February 10, 1975. It currently consists of 12 advisors. Since its inception, the CMR has certified 730 adults and 49 pediatric rheumatologists through an examination, which consists of a theory and a practical part. Until 2012, the practical evaluation was carried out on a real patient, in the form of a single case.

The certification exam of any medical specialty is considered a high-stakes exam, so boards must do everything possible to ensure the assessment tools are made with an appropriate technical framework and educational professionalism.1 The examination of a real case or a single case evaluates medical performance in real settings, verifying the physicians capabilities for obtaining helpful information, making appropriate decisions, integ scoring previous knowledge and discussing views. However, it is poorly reliable and reproducible.2 The objective structured clinical examination (OSCE) is a highly objective assessment tool that allows to examine skills and a large number of skills, leading some authors to consider it the gold standard for assessing clinical competencies.3–6 Since 1975, the OSCE has been used in Canada, the United States, Australia and Europe to evaluate undergraduate clinical skills. In Mexico, it is used since 1996 as a summative exam at the Faculty of Medicine of the Autonomous University of Mexico and as the professional practice examination phase of the undergraduate course since 2002.7 In addition, the OSCE has also shown good results in evaluating12 general physicians,8,9 surgeons, pediatricians,10 and rheumatologists11 musculoskeletal ultrasound skills.

Since its inception, the CMR has recognized that clinical competence is a crucial training aspect of any future rheumatologist. Two years ago, the CMR implemented an OSCE in order to apply it to applicants as part of their annual certification process. This paper describes the process and examines the validity (construct) and reliability of the OSCE.

The paper describes the design, development and implementation of an OSCE to assess clinical skills in the Rheumatology Board Annual certification applied by the CMR. In recent years, it has been proposed that the OSCE is a more objective tool for assessing clinical skills, especially when compared with other modalities, as single clinical cases. However, its limitations are recognized, so it must be subjected to rigorous scrutiny and pilot testing as a prerequisite step to its application.3

OSCE was designed by a collegial panel of experts, who applied international recommendations for its development,15 among which the prior establishment of a table of specifications, validation of the stations by a group of experts and the application of a pilot test stand out. The OSCE assessment was performed with reference to a criterion, as a standard was set in advance, as recommended by the current trend of competency-based education (www.conacem.org).

The construct validity of the OSCE was analyzed in several ways. First, the performance of the OSCE with the theory part of the certification exam was correlated. In the literature, significant correlations between the OSCE and theory evaluations have been described, with values ranging between 0.59 and 0.71,16–18 although there is little information in the field of rheumatology.11,12 A recent report published by Kissin et al. described a correlation of 0.52 (P≤0.001) between a n9 station circuit designed to assess musculoskeletal ultrasound skills and its corresponding theory part12; conversely, Raj et al. found no significant correlation between the theory and practical knowledge, when the latter was assessed by 2 stations for exploration of the hand and knee applied to medical students.19 We found a correlation of 0.44 in 2013, with no correlation in 2014. This could be related to a worse performance of the candidates in the theory test in 2014 and/or the fact that both assessments (theory and practice) measured different knowledge. It should be noted that finding a high correlation between the 2 assessments indicates that only a small gain in information concerning the knowledge of the applicant is obtained when both instruments are applied; a plausible hypothesis is that, in situations of high correlation between both, the OSCE explores (like the theory test) “knowledge” and “know how” and shuns the “show how” of the Miller pyramid.20 In relation to this, Matsell et al.10 identified a key element in construct validity for an OSCE when it is based on the high correlation with a written evaluation. From a theory point of view, the OSCE assesses a wide variety of skills/clinical competencies, so the authors propose stations categorized into 4 major domains (knowledge, problem solving, clinical skills and patient management) and correlating stations for each domain with the corresponding theory part. This proposal, which has great theoretical foundation, will be included in the forthcoming implementation of OSCE.

Regarding construct validity, the experience construct was also analyzed; a valid assessment should show how more experienced candidates perform better.21 All applicants had similar training in years. However, the same circuit was applied to 3 newly certified rheumatologists, under similar test day conditions. In general, the latter had higher scores than the applicants.

Finally and indirectly, construct validity was analyzed in 2014 comparing the pass percentage of those passing the theory test (and the average score) in those who passed or did not pass the OSCE. Our results support the validity of the instrument.

With respect to reliability, no evaluation was performed, although 3 ways have been proposed to increase the reliability of the OSCE and all of them were implemented: short station design to allow evaluation of multiple clinical situations, evaluation of the stations through predetermined checklists and integration of circuits with different stations, evaluators and17 simulated patients. In our case, patients selected were real patients (except for 2 actors); however, they received training booklets designed to represent the condition of each station and reviewed by an expert panel. All this increases the reliability of the OSCE without affecting its validity, as when simulated patients or actors are used. In addition, 9 stations were applied in 2 consecutive years and showed excellent stability between the 2 groups of aspirants and internal consistency between certified rheumatologists in the pilot test was excellent. Finally, while a low interstation correlation was found, this indicates that each station evaluated different skills or competencies and therefore discarded redundancy of our instrument. It is important to emphasize the need to include an adequate number of stations to not affect the reliability of the OSCE.3,22 In the one similar exercise (that we know of), the exercise circuit consisted of 12 stations.11

Probably one of the biggest benefits of an OSCE (and other similar evaluations) is the ability to identify deficiencies in the training of residents and eventually modify the curriculum. It is therefore essential to have an optimal assessment tool. While it is true that the performance of our instrument was adequate, we must not forget that the OSCE is ultimately an approximation to the real problems but not a real physician-patient encounter. Our experience is limited to two years, but it is recommended that any instrument be used for 3–5 consecutive years, allowing proper calibration thereof. Furthermore, the OSCE was not reproduced in full in consecutive years. With regard to the assessment of the stations, no scientific method was applied to establish the passing score for each station,23 but some previous criteria were established. The OSCE applied to certified rheumatologists (pilot) was developed on a different day than the OSCE applied to applicants, although an effort was made to reproduce the conditions of the final evaluation. The implementation of OSCE required financial and human resources; when performing the exercise to define the costs, these amounted to 120000 pesos (about $9200), a figure which is not excessive but much greater than what is usually invested in the practical assessment of residents and could affect its long-term viability.

In conclusion, the OSCE designed by the CMR showed adequate construct validity and reliability, so it can be integrated as an instrument for assessing clinical skills in the board examination in order to obtain a certification as a rheumatologist. Certification boards have a social responsibility and also a responsibility to the medical community, documenting that specialists are competent to exercise their practice. It is therefore necessary to incorporate rigor, methodological quality and transparency to the certification processes. In parallel, society must require greater professionalism. The effort of the whole of society, the medical community, patients and the competent authorities must have a common goal to help increase the quality of care our population receives.

The authors declare no conflict of interest.