The Mexican Accreditation Council for Rheumatology annually certifies trainees in Rheumatology using a multiple-choice test and an objective structured clinical examination (OSCE). Since 2015, candidate's communication skills (CS) have been rated by both patients and by physician examiners and correlated with results on the OSCE. This study compared the CS from candidates to annual accreditation in Rheumatology as rated by patients and by physician examiners, and assessed whether these correlated with candidate's performance in the OSCE.
Material and methodsFrom 2015 to 2017, 8 areas of CS were evaluated using a Likert scale, in each OSCE station that involved a patient. Both patient and physician evaluators were trained annually and their evaluations were performed blindly. The associations were calculated using the Pearson correlation coefficient.
ResultsIn general, candidates were given high CS scores; the scores from patients of the candidate's CS were better than those of physician examiners; within the majority of the stations, both scores were found to correlate moderately. In addition, the scoring of CS correlated with trainee performance at the corresponding OSCE station. Interestingly, better correlations were found when the skills were rated by the patients compared to physician scores. The average CS score was correlated with the overall OSCE performance for each trainee, but not with the multiple-choice test, except in the 2017 accreditation process, when a weak correlation was found.
ConclusionsCS assessed during a national accreditation process correlated with the candidate's performance at the station level and with the overall OSCE.
El Consejo Mexicano de Reumatología certifica anualmente reumatólogos mediante una prueba teórica y un examen clínico objetivo estructurado (ECOE). Desde el año 2015, se evalúan las habilidades de comunicación (HC) de los candidatos. Los objetivos fueron comparar las HC evaluadas por el paciente (HCP) y por el médico (HCM) y correlacionarlas con el desempeño de los candidatos en el ECOE.
Material y métodosDurante los años 2015, 2016 y 2017, se evaluaron las HC en las estaciones dinámicas, mediante una escala de Likert aplicada a 8 áreas. Pacientes y evaluadores fueron entrenados cada año para calificar a los aspirantes, lo cual se realizó el día del ECOE, de manera ciega, por ambos. Se calcularon coeficientes de correlación de Pearson.
ResultadosEn general, a lo largo de los 3 años, los candidatos obtuvieron puntajes altos en las HC. Los pacientes puntuaron mejor a los candidatos que los evaluadores médicos. Las HCP y las HCM correlacionaron entre sí (de leve a moderado) en la mayoría de las estaciones. El puntaje de las HC de cada candidato correlacionó con su desempeño en la estación correspondiente; se encontraron mejores correlaciones con las HCP. El promedio de las HC de cada candidato correlacionó con el desempeño global en el ECOE, pero no así con la prueba teórica (salvo en el año 2017, cuando hubo una correlación baja).
ConclusionesLas HC evaluadas durante un examen de certificación en Reumatología correlacionan con el desempeño de cada candidato en cada estación y en el ECOE global.
Accreditation councils of the medical specialties are committed to society to guarantee that certified specialists have the necessary skills and competencies for practicing their profession. This implies developing and applying evaluation tools, establishing cutoff points that determine whether or not a candidate should be accredited and, in short, making decisions that are of crucial relevance for the candidates, health and educational authorities and society in general.1–3
The function of the Mexican Accreditation Council for Rheumatology (CMR) is to annually certify all the candidates who have completed their training in rheumatology in any of the national centers accredited for that purpose. The process of certification implies a multiple-choice test utilizing case reports and a practical evaluation. Until 2012, the latter was based on “the long case” with a real patient, widely recognized for its utility in training evaluation, but it is not highly recommended for the evaluation of competencies of “high consequences”.4,5 For this reason, since 2013, the CMR implemented an objective structured clinical examination (OSCE) to evaluate clinical skills in the annual certification examination. Throughout the years, the OSCE has been found to have an adequate performance.6,7
The OSCE consists of a circuit of stations, some of which include real patients. The latter are thus known as “dynamic” stations.8 Each station (static or dynamic) is assigned a rheumatologist (examiner) who evaluates the candidates using a checklist and scale of general performance. Since 2015, the dynamic stations have an additional instrument for evaluation, designed and adapted to determine the communication skills (CS) of the candidates with respect to the patient.
Effective CS has an impact on patient satisfaction9 and on the outcome.10,11 Nevertheless, in many countries, formal training in CS is not included in the academic curriculum and is exceptional once undergraduate education is completed; as with other aspects of clinical competencies, CS declines over time and the lack of practice.12 In addition, the teaching model and the training techniques applied in CS are based on a classical model of teacher-trainee in which teaching lacks definite planning, structure and objectives. In our usual clinical work, it is common to confront situations in which communication with the patient and those around him or her is complex; the resident in training is often alone and is not being observed by a specialist with greater experience; thus there is not an adequate feedback. A subject that has been a motive for debate in the area of CS is who should evaluate these skills, the patients or an examiner. It is interesting that it has been demonstrated that standardized patients can efficiently evaluate the CS of trainees and physician examiners.13,14
In the present study, we report the experience of the CMR relative to the evaluation of CS of candidates for accreditation in rheumatology in 2015, 2016 and 2017. The specific objectives were:
- 1.
To describe and compare the CS of candidates for accreditation in rheumatology evaluated by the patient and by a physician examiner.
- 2.
Correlate the CS of the candidates with their performance in the OSCE.
Mexico has 16 centers certified for the training of specialists in rheumatology. All of the candidates for accreditation have completed at least 4 years of training and have a letter of recommendation from their respective professors of the courses offered in the training centers. Each year, the certification examination is to be held on 2 consecutive days; the OSCE is given on the first day and the multiple-choice test on the second. Both tests are developed by specifically designated committees and follow a strict process that has previously been described.6,7
Table 1 shows, in each year (2015, 2016 and 2017), the number of candidates, the number of reagents in the multiple-choice test, the total number of stations and the dynamic stations (in which the CS of the candidates were evaluated).
Characteristics of the Accreditation Examination of the Mexican Accreditation Council for Rheumatology and Number of Candidates Each Year.
| 2015 certification | 2016 certification | 2017 certification | |
|---|---|---|---|
| No. of candidates | 43 | 37 | 38 |
| No. of reagents in the MCT | 222 | 200 | 300 |
| Total no. of stations | 15 | 15 | 17 |
| No. of dynamic stations | 11 | 11 | 12a |
Communication skills were evaluated in each of the dynamic stations corresponding to the circuit of each year.
We adapted 2 similar formats for the CS, one directed at the patient and the other for the examiner. Both had identical contents and only the wording was changed. Each format included a single Likert scale: from 1=unsatisfactory to 9=superior, which was utilized to evaluate 8 areas of CS of the candidates (Table 2 and Anexo A, Supplemental material).
Communication Skills of the Candidates Evaluated During the Accreditation Examination.
| 1. | Reception and presentation of the patient |
| 2. | Explication to the patient of what the candidate will do during the encounter |
| 3. | Understanding on the part of the patient the explication concerning the candidate |
| 4. | Perception (of the patient) as to the candidates interest in the comments made by the patient |
| 5. | Visual contact between the candidate and the patient |
| 6. | Empathy of the candidate with respect to the patient |
| 7. | Care (on the part of the candidate) in performing the examination and/or procedure |
| 8. | The candidate perceives that the patient feels pain during the examination and/or procedure |
All of the patients were trained during an session designed to apply the format and to score each candidate. Likewise, all of the examiners were also trained. The evaluation of each candidate was scored by the patient and the examiner on the day of the multiple-choice test, independently (blindly), immediately after the participation of each trainee; each assigned a single score (from 1 to 9) to each candidate.
In addition, each examiner assigned to a station had 2 tools, a checklist and a scale of overall performance corresponding to the station; at the end of the encounter with each trainee, the examiner assigned a score for each tool. The score for each station was established exclusively by the score of the candidate according to the checklist.7
Statistical AnalysisWe applied descriptive statistics and Student's t-test. Each candidate was given a score for the multiple-choice test and another (overall) for the OSCE, and the scores for all of the stations included each year were averaged.
We correlated the CS of each candidate, evaluated by the patient and the examiner using the Pearson correlation coefficient. Likewise, the CS was correlated with the performance in each station, in the OSCE and the multiple-choice test.
The analyses were performed using the SPSS statistical package for PC (v. 20; Chicago, IL, United States).
ResultsCommunication Skill Score Given by the Examiner and by the PatientIn general, and consistently over 3 years, the candidates were given high scores in CS. In the majority of the stations, the patients gave higher scores to the trainees than did the examiners, as is shown in Table 3, which summarizes the OSCE data of 2015.
Correlation of the Score for Communication Skills of the Candidates Provided by the Examiner and by the Patient During the 2015 Objective Structured Clinical Examination.
| Stationsa | CS score (mean±SD) provided by the examinerb | CS score (mean±SD) provided by the patientb | rc | P |
|---|---|---|---|---|
| No. 2 | 6.1±1.6 | 8.0±0.5 | 0.22 | .05 |
| No. 3 | 5.6±1 | 7±1 | 0.05 | NS |
| No. 5 | 7.5±1.2 | 6.9±1.4 | 0.15 | NS |
| No. 6 | 7.6±0.9 | 5.8±0.6 | 0.12 | NS |
| No. 8 | 7.1±1.2 | 7.6±1 | 0.7 | ≤.001 |
| No. 9 | 7.2±1.1 | 7.5±1 | 0.6 | ≤.001 |
| No. 10 | 8.4±3.2 | 8.6±0.5 | 0.01 | NS |
| No. 11 | 7.8±0.8 | 7.8±1 | 0.62 | ≤.001 |
| No. 13 | 6.2±1.4 | 6.5±0.9 | 0.43 | ≤.001 |
| No. 14 | 7.0±1.6 | 7.9±1.4 | 0.59 | ≤.001 |
| No. 15 | 7.3±0.5 | 6.8±1.4 | 0.29 | .05 |
In 2015, there was a slight to moderate correlation, but significant, between the evaluation of the candidate's CS scored by the patient and that of the examiner in most of the stations (Table 3); in later years, there was a moderate correlation between the two evaluations in 40%–50% of the stations.
Impact of Communication Skills in the Performance in the StationsIn general, over a period of 3 years, the CS score obtained by each trainee correlated with the performance of the candidate in the station. It is interesting that the correlations were more frequently found to be positive and higher when the CS were scored by the patient than by the examiner, as is shown in Table 4, which summarizes the data from 2017.
Correlation Between the Communication Skills Evaluated by the Patient and by the Examiner and the Performance of the Candidate in Given Stations (2017).
| Station/category of the disease | Competencies evaluated | CSa/CPc r (P) | CSb/CPc r (P) |
|---|---|---|---|
| Gout | Problem solution prescription | 0.602 (<.001) | 0.399 (<.01) |
| Rheumatoid arthritis | Problem solution prescription | 0.503 (<.001) | 0.117 (.10) |
| Low back pain | Medical record | 0.542 (<.001) | 0.127 (.10) |
| Rheumatoid arthritis | Clinical examination | 0.360 (<.01) | 0.025 (.4) |
| Rheumatoid arthritis | Medical record | 0.530 (<.001) | −0.047 (.4) |
| Gout | Medical record | 0.515 (<.001) | 0.363 (<.01) |
| Disorders affecting soft tissue | Clinical examination | 0.291 (<.01) | 0.162 (.06) |
| Spondyloarthritis | Clinical examination | 0.038 (.4) | 0.119 (.1) |
| Antiphospholipid antibody syndrome | Problem solution | 0.760 (<.001) | 0.407 (<.001) |
| Sjögren's syndrome | Medical record | 0.271 (<.01) | 0.118 (.1) |
| Inflammatory myopathy | Clinical examination | 0.471 (<.001) | 0.330 (<.01) |
| Scleroderma | Medical record | 0.790 (<.001) | 0.274 (<.01) |
CP, candidate performance; CS, communication skills.
The average of the CS of each trainee (evaluated by the patient or by the examiner) correlated with the overall performance of the candidate in the OSCE (especially in 2015 and 2017); however, this did not occur with the multiple-choice test, except in 2017, when there was a correlation, although notably lower than that observed with the OSCE. Table 5 summarizes the data from each year.
Correlation (and P Value) Between the Mean Communication Skills (CS) of Each Candidate (Evaluated by the Examiner) and the Multiple-choice Test (MCT) and Between the Mean CS of Each Candidate and the Total Objective Structured Clinical Examination (OSCE) Score.
| Certification | CS/MCT | CS/OSCE |
|---|---|---|
| 2015 | r=0.22 (P=.10) | r=0.47 (P=.042) |
| 2016 | r=0.18 (P=.32) | r=0.32 (P=.06) |
| 2017 | r=0.38 (P=.01) | r=0.67 (P<.001) |
The CMR is committed to Mexican society to guarantee that the accredited specialists have the knowledge, skills and competencies to attend to the health problems that are associated with our specialty. For this, the CMR develops and applies evaluation tools that enable the identification of eventual deficiencies in the training of our specialists; the information derived from the analysis of the evaluations is annually shared with the professors who are going to give the training courses and, jointly, strategies are proposed for the purpose of improving suboptimal areas and, therefore, the quality of our specialists.
The present report evaluates the CS of the candidates for accreditation in rheumatology over 3 consecutive years. Communication skills constitute a basic competence in establishing communication between the physician and the patient that favor a solid, effective and lasting relationship between the two. The benefits of a relationship with these characteristics are also extended to the patient's family and other support networks that integrate the health systems.10 In recent decades, international consensuses have been developed in the area of physician-patient communications,11 as well as guidelines to be applied in medical schools12 and to serve as references for professional practice,13 all of which stresses the importance of this subject.
In general, the candidates for the examination of certification in rheumatology had a good performance in CS in dealing with the patients; however, this was not homogeneous, as is expected and has been described in other populations.14 The patients gave higher scores to the trainees than to the examiners and both evaluations presented significant correlations (low and moderate) between them in more than half of the OSCE stations. Communication skills correlated with the performance of the candidates in the stations, with the overall performance in OSCE, but that was not observed with the multiple-choice test (except in 2017). The literature provides evidence that efficient CS correlated with the clinical competencies of those being evaluated10,15,16 and that the general performance of physicians improved after training involving CS17,18; in fact, there are educational interventions that have improved confidence in CS in those who had completed the program even months after having participated.10 Even more relevant is the fact that efficient CS have been associated with desired outcomes for the patients, such as better adherence to drug therapy and diet, better pain control and a better physical, functional and psychological well-being.9,19–23 Finally and within the physician-legal framework that is applied when the patients develop complications or undesirable outcomes, early and sincere communication with the patients has been shown to be crucial.24–28 It is important to point out that the OSCE has been demonstrated to have a good performance in evaluating CS and identifying specific deficits,28 combined with its adequate performance in the accreditation process.5,6
One relevant question about CS is whether the patient or an external physician should evaluate it. Interestingly and intuitively, we encountered a better correlation with the performance of the trainees in the stations (and with the OSCE) when the CS were evaluated by the patient. The best correlations were obtained in the last year, which could indicate a certain degree of apprenticeship gained by the patients (and, in general, all the participants in the OSCE), who are trained year after year to score the candidates. We should point out that the performance of the trainees was established by the examiner, using the checklist. Our results confirm published studies in which standardized patients adequately evaluate the CS of surgical residents with different years of training.16,17,24,25
The present report has certain limitations. Communication skills were evaluated utilizing an adapted tool, that included a Likert scale in which 8 domains were jointly applied. This tool was subjected to a process of validation; although both users (patient and examiner) were trained in its application, the OSCE has also been shown to be a suitable tool for the evaluation of CS.17 In each process of accreditation, 2 consecutive OSCE were applied due to limited space. This could have affected the reproducibility of the results29; nevertheless, the physical space, patients and examiners were the same in both circuits. Finally, the validity and reproducibility of the results of an OSCE are determined by its length, that is, by the number of stations included. Over 3 years, we included a maximum of 12 dynamic stations, and the results obtained may not be solid enough for making decisions, especially if the latter have important consequences.30
Currently, physicians are not independent actors in health care, but perform more like leaders or members of multidisciplinary teams3; thus, they must acquire and perfect certain professional attributes, the first of which includes CS. Our work is to establish the bases for a specific proposal: integrate residency in rheumatology into a formal, continued education directed at CS. We want to stress that training in CS is favored in the contextual setting of routine practice, with an adequate feedback31,32 and an environment that requires that those involved have protected time for teaching.18,33,34
We are what we repeatedly do. Excellence, then, is not an act but a habit.
(Aristotle, 400 BC)
Ethical DisclosuresProtection of human and animal subjectsThe authors declare that no experiments were performed on humans or animals for this study.
Confidentiality of dataThe authors declare that no patient data appear in this article.
Right to privacy and informed consentThe authors declare that no patient data appear in this article.
Conflicts of InterestNone of the authors has any financial relationship that could be considered a conflict of interest in relation to this article.
The following are the supplementary data to this article:
Please cite this article as: Pascual-Ramos V, Flores-Alvarado DE, Portela-Hernández M, Maldonado-Velázquez MR, Amezcua-Guerra LM, López-Zepeda J, et al. Las habilidades de comunicación de los aspirantes a la certificación en Reumatología se asocian con su desempeño en el examen clínico objetivo estructurado. Reumatol Clin. 2019;15:97–101.
