Reumatología Clínica (English Edition) Reumatología Clínica (English Edition)
Reumatol Clin 2017;13:139-44 - Vol. 13 Num.3 DOI: 10.1016/j.reumae.2016.04.010
Original Article
Readability, Relevance and Quality of the Information in Spanish on the Web for Patients With Rheumatoid Arthritis
Legibilidad, relevancia y calidad de la información en español en la Web para pacientes con artritis reumatoide
Jose Dionisio Castillo-Ortiza,, , Jose de Jesus Valdivia-Nunoa, Andrea Ramirez-Gomeza, Heber Garagarza-Mariscala, Carlos Gallegos-Riosa, Gabriel Flores-Hernandeza, Luis Hernandez-Sancheza, Victor Brambila-Barbaa, Jose Juan Castaneda-Sancheza, Zalathiel Barajas-Ochoaa, Angel Suarez-Ricoa, Jorge Manuel Sanchez-Gonzalezb, Cesar Ramos-Remusa,b
a Unidad de Investigación en Enfermedades Crónico-Degenerativas, Guadalajara, Mexico
b Vicerrectoría Académica, Universidad Autónoma de Guadalajara, Zapopan, Mexico
Received 08 March 2016, Accepted 22 April 2016
Abstract
Background

Education is a major health determinant and one of the main independent outcome predictors in rheumatoid arthritis (RA). The use of the Internet by patients has grown exponentially in the last decade.

Objective

To assess the characteristics, legibility and quality of the information available in Spanish in the Internet regarding to rheumatoid arthritis.

Material and methods

The search was performed in Google using the phrase rheumatoid arthritis. Information from the first 30 pages was evaluated according to a pre-established format (relevance, scope, authorship, type of publication and financial objective). The quality and legibility of the pages were assessed using two validated tools, DISCERN and INFLESZ respectively. Data extraction was performed by senior medical students and evaluation was achieved by consensus.

Results

The Google search returned 323 hits but only 63% were considered relevant; 80% of them were information sites (71% discussed exclusively RA, 44% conventional treatment and 12% alternative therapies) and 12.5% had a primary financial interest. 60% of the sites were created by nonprofit organizations and 15% by medical associations. Web sites posted by medical institutions from the United States of America were better positioned in Spanish (Arthritis Foundation 4th position and American College of Rheumatology 10th position) than web sites posted by Spanish speaking countries.

Conclusions

There is a risk of disinformation for patients with RA that use the Internet. We identified a window of opportunity for rheumatology medical institutions from Spanish-speaking countries to have a more prominent societal involvement in the education of their patients with RA.

Resumen

La educación es un determinante mayor de salud y uno de los predictores independientes de desenlace en artritis reumatoide (AR). El uso del Internet por pacientes ha crecido en forma exponencial en la última década.

Objetivos

Evaluar las características, legibilidad y calidad de la información disponible en Internet en idioma español en relación con la AR.

Material y métodos

Se buscó la frase AR en Google. Se evaluaron las primeras 30 páginas de resultados de acuerdo con un formato diseñado ex profeso (relevancia, autoría, tipo de publicación, enfermedad discutida e interés financiero); además se evaluaron la calidad y la legibilidad de las páginas, con las herramientas DISCERN e INFLESZ, respectivamente. La extracción de datos se realizó por médicos pasantes y la evaluación fue por consenso.

Resultados

Se obtuvieron 323 resultados, pero solo el 63% de ellos fueron relevantes; el 80% de estos fueron sitios de información (71% discutían exclusivamente AR, 44% terapia convencional y 12% terapias alternativas). Un 12,5% tenía interés financiero. El 60% de los sitios fueron creados por organizaciones no lucrativas y 15% por asociaciones médicas. Las asociaciones médicas de Estados Unidos de América se posicionan mejor en español (Arthritis Foundation en la posición 4 y el American College of Rheumatology en la 10) que los sitios web de países de habla hispana.

Conclusiones

Existe riesgo de desinformación para los pacientes con AR que utilizan la Web. Se identifica además áreas de oportunidad para instituciones médicas de países de habla hispana para tener un mayor involucramiento social en la educación de sus pacientes.

Keywords
Rheumatoid arthritis, Internet, Web, Education
Palabras clave
Artritis reumatoide, Internet, Web, Educación
Introduction

Rheumatoid arthritis (RA) is a disease that can be prolonged for decades and has a multidimensional impact, ranging from joint destruction to the development of comorbidities and high direct and indirect costs.1,2 Formal education of RA patients is of the utmost importance, from the perspective of determinants of health3 or as a predictor of the outcome.4 Information is the platform for acquiring knowledge that will be transformed into behavior that will have an influence on the form with which a person faces disease, makes choices concerning caring for his or her health and adheres to treatment.5,6

A physician provides the scientific information that a patient may need by means of a process of filtering and dosing.7 At the present time, what a patient requires of the physician is greater attention, an improvement in communication, sufficient time during visits and not to have to wait too much to be attended to.8 However, these requirements are frequently not met.9,10 Although physicians continue to be the primary source of information, the Internet has become an important option for patients in recent years, at least in countries like the United States of America (US). For example, 81% of the adults in the US utilize the Internet and 72% mentioned that they had looked for health information over the past 12 months.11 In fact, users of the Internet also visit websites that are specific for health professionals; of the millions of searches carried out each year in the web page of the National Library of Medicine, 33% are performed by the general public.12

There are more than 40 publications concerning the quality of the information available on the web related to different diseases and at least 15 on RA; all are in English and analyze websites in English. The conclusions in most of these publications focus on whether the information on the Internet can increment the autonomy of patients, but it can also increase their anxiety and be the source of misleading information due to the heterogeneity and low quality of the information13,14; moreover, patients can become the victims of persons with commercial interests.15,16

However, to the best of our knowledge, there are no publications analyzing the characteristics of the information available in the web about RA in Spanish. This is important if we take into account the fact that Spanish is the second language in numbers of first-language speakers in the world17 and that the utilization of the Internet in Spanish has also increased; it is used by 88.3% of Spaniards18 and by 50.2% of Mexicans,19 which add up to approximately 103.8 million users. If we consider it true that the prevalence of RA is between 0.5% and 1% of the general population, then there would be between 518 thousand and 1 million RA patients as potential users of the Internet only in those 2 countries. This, and because of the chronicity of RA, it can be assumed that there is a large number of potential users of the Internet who speak Spanish as their first language and search for information on their disease. In the present study, we analyzed the quality of the contents, the legibility and the relevance of the websites with information on RA in Spanish.

Material and Methods

This is a cross-sectional study to audit web pages in the Internet to assess the characteristics of the information related to RA in Spanish. We used the methods of a previous publication from Canada because the detailed description of their methodology enabled us to reproduce it.20 Briefly, we did an extensive search in Spanish, without filters or restrictions, using the phrase “artritis reumatoide” via the Google search engine, as it is the engine most widely used in the world.21 The search was conducted on a single day and the resulting web pages were stored in Portable Document Format (PDF) for later assessment. All of the pages were evaluated after being visited and the information had been collected from their associated sites. The web pages from the first 30 pages of results were identified and accessed (hits) (10 hits per results page). Two sixth-year medical students evaluated each web page separately; discrepancies and/or controversies were resolved by consensus, together with 2 of the participating rheumatologists (DCO and CRR). We considered a website to be a collection of several related web pages that had a common Internet domain. The evaluation of the pages was performed in accordance with the following domains: relevance, type of publication, scope, contents, authorship and financial objectives, in terms of the definitions provided20:

  • a.

    Relevance. Web pages were considered relevant if the content included some clinical aspect of RA—etiology, symptoms, signs, diagnosis, treatment and/or prognosis. If the information did not deal with a clinical aspect but could influence the conduct of the patient with respect to the disease, it was considered of doubtful relevance. Only the relevant or those of doubtful relevance continued in the process of evaluation.

  • b.

    Type of publication. The web pages were classified as advertisements, information, links, new articles, part of a publication, attempts to recruit patients for an investigation, chat rooms, case reports and support group posts.

  • c.

    Scope. We established whether the web pages dealt only with aspects of RA or discussed other autoimmune diseases, and used “arthritis” as a general term or in reference to several diseases (fibromyalgia, juvenile idiopathic arthritis, scleroderma, etc.).

  • d.

    Contents. The web pages were categorized according to their contents: the inclusion of general information (clinical or epidemiological data on RA) or the mention of conventional therapy or some type of alternative therapy.

  • e.

    Authorship. The information from the websites was classified as being from a lucrative industry or a for-profit organization, a nonprofit organization, individuals with no clear affiliation and medical institutions (universities, hospitals, medical associations).

  • f.

    Financial objectives. The websites were cataloged depending on whether they contained explicit information on products being marketed, requested donations or required the payment of a membership fee to accede. They were classified as being direct or indirect product marketing, as requiring a membership fee, as requesting donations or none of the above.

Aside from the procedures described in the reference study,20 others were added to assess the “content quality” domains of the web pages, the legibility and possible variations in the results depending on the Internet Protocol (IP) address utilized. The content quality was evaluated using the DISCERN tool.22 DISCERN is an instrument designed to rate the quality of the content of a publication on the Internet, but cannot be utilized to evaluate the scientific quality or accuracy of the evidence on which the publication is based. DISCERN consists of 15 specific questions plus a question that measures the general quality. Each of the first 15 questions represents in itself a specific quality criterion; the rating is 1 (if the response is “no”), 2–4 (up to a certain point) or 5 (if the response is a definite “yes”). As our study did not evaluate information on specific treatments and each of the 15 questions represents a separate quality criterion, questions referring to treatments were excluded and we utilized only the following items: 1 (The objectives are clear?); 2 (Those objectives are achieved?); 4 (Is there a clear declaration of any sources of information used to compile the data, aside from the author or the product?); 6 (Is it objective and not biased?); and 7 (Is data provided on additional sources of support and information?); question 16 was also included (an intuitive summary of the responses to the 15 previous questions/a subjective evaluation of the general quality).

Legibility was evaluated using the INFLESZ program, which measures the syntactic complexity of a written text, whether it be a complete file or a fragment of a work.23 The score obtained with the INFLESZ scale ranges from 0 to 100. A text is considered to be very difficult to read if the score is from 0 to 40 (scientific or university publications), somewhat difficult to read with a score of 40.1 to 55 (high school level, specialized journals, etc.); normal if the score is from 55.1 to 65 (newspapers or sports news), quite easy with a score is 65.1 to 80 (grade school level) and very easy if the score is over 80.1 (grade school level, comic books, novels, etc.). For the analysis of legibility, we excluded figures, tables, footnotes, links and other pages and references.

We analyzed web pages published by hospitals, institutions and societies or medical associations (medical institutions) separately, using the procedures described above.

To evaluate the possible variation in the results depending on the IP address, we did the same search utilizing 3 different IP addresses: from Guadalajara, Mexico (189.169.108.15), from San Andrés, Colombia (104.224.35.115) and from Madrid, Spain (185.25.87.186).24

Statistical Analysis

Dichotomized variables were expressed as proportions. Chi-squared test was used to compare the general characteristics, quality and legibility of the web pages in each group according to the order in which they appeared, taking the pages from medical associations, societies and institutions together with the rest of the relevant pages. A P value <.05 was considered to indicate statistical significance. The analysis was done with the STATA (version 12.0) software package.

Results

We found 323 web pages, 121 (38%) of which were not evaluated (10 because they were in another language and 111 because we considered them irrelevant). Table 1 shows the type of publication, the characteristics of the disease being dealt with and the content of the 202 relevant web pages. In all, 80% of the pages provided general information exclusively on RA; 13% of the pages included an alternative therapy, either as the main topic or in association with conventional therapy. We identified 186 websites, nearly all of which had been created by nonprofit institutions, although only 15% had been posted by a medical institution (Table 2).

Table 1.

Features of the Relevant Web Pages.

Type of publication  n=202 
Advertisements, n (%)  33 (16) 
Information, n (%)  161 (80) 
Links, n (%)  38 (19) 
News articles, n (%)  21 (10) 
Research results, n (%)  9 (5) 
Recruitment of patients for investigation, n (%)  1 (<1) 
Chat rooms, n (%)  5 (2) 
Case reports, n (%)  5 (2) 
Support group posts, n (%)  5 (2) 
Disease dealt with
Rheumatoid arthritis, n (%)  147 (73) 
Arthritis, n (%)  11 (5) 
Autoimmune diseases, n (%)  2 (1) 
Several (arthritic diseases and others), n (%)  42 (21) 
Content
General information, n (%)  181 (90) 
Conventional therapy, n (%)  89 (44) 
Alternative therapy, n (%)  25 (13) 
Unclear, n (%)  4 (2) 
Table 2.

General Characteristics of the Relevant Websites.

Authorship  n=186 
For-profit organization, n (%)  35 (19) 
Nonprofit organization, n (%)  93 (50) 
Individuals with no clear affiliation, n (%)  44 (24) 
Medical associations, universities and hospitals  26 (15) 
Financial objective  n=186 
Direct product marketing  2 (1) 
Indirect product marketing  15 (8) 
Membership required  1 (<1) 
Donations requested  2 (1) 
Unclear  35 (19) 
None of the above  131 (70) 

The evaluation of the content quality showed that 61% of the web pages clearly declared their objectives, but only 48% fulfilled them. Around a third of the pages clearly provided the sources of the information utilized, only 22% offered unbiased information and more than half failed to name additional sources of information. The general quality of a fourth of the pages was considered to be high (Table 3). With respect to the legibility, more than half of the pages were very difficult (9%) or somewhat difficult (52%) to read, whereas a fourth of the pages had a normal level of legibility in accordance with the INFLESZ SCALE (Table 4).

Table 3.

Evaluation of the Quality of the 202 Web Pages.

Quality as defined by the question  No
n (%) 
Partially
n (%) 
Yes
n (%) 
1. The objectives are clear?  59 (29)  20 (10)  123 (61) 
2. Are the objectives fulfilled?  65(32)  41 (20)  96 (48) 
4. Is there a clear declaration of any sources of information used for the publication, aside from the author?  114 (57)  17 (8)  71 (35) 
6. The information is clear and unbiased?  102 (50)  56 (28)  44 (22) 
7. Is data provided on additional sources of support and information?  116 (57)  14 (7)  72 (36) 
General quality of the publication  Low  Moderate  High 
16. Based on all of the responses, what is the general quality of the publication as a source of information concerning the choice of treatment?  67 (33)  85 (42)  50 (25) 
Table 4.

Legibility of the 202 Relevant Web Pages (INFLESZ).

INFLESZ scale  Web pages
n (%) 
0–40  18 (9) 
40.1–55  105 (52) 
55.1–65  53 (26) 
65.1–80  16 (8) 
>80.1  10 (5) 

0–40: very difficult to read; 40.1–55: somewhat difficult to read; 55.1–65: normal level of legibility; 65.1–80: quite easy to read; >80.1: very easy to read.

Table 5 shows the position of the web pages in Spanish belonging to medical institutions, according to the place in which they appeared, the content quality (DISCERN) and legibility (INFLESZ). Among the first 300 pages that appeared in Google, only 26 (8.6%) were created by medical institutions: 17 (65%) from the US, 7 (27%) from Spain, one from Mexico and one from Venezuela. Among the first 10 websites, 3 were from medical institutions; they included the Arthritis Foundation (US) in 4th place, the Spanish Society of Rheumatology in 8th and 9th positions, and the American College of Rheumatology in 10th place. The web page of the Mexican College of Rheumatology appeared in 118th place, but was classified as irrelevant, and was excluded from the analysis. The only Mexican page included in the analysis was in 31st and 122nd place. The content quality was rated good in 11 pages (42%) and poor in 10 (38%). Concerning the evaluation of legibility, 8 (31%) were quite easy or easy to read, 9 (35%) were normal and 9 (35%) were somewhat difficult or very difficult to read.

Table 5.

Characteristics of the Web Pages of Medical Institutions.

Web page  Position in Google search  DISCERNa  INFLESZb 
Arthritis Foundation (US)  74.3 
Spanish Society of Rheumatology (SP)  8, 9  5, 1  57, 35 
American College of Rheumatology (US)  10  71.9 
Clínica DAM (private medical specialties in Madrid) (SP)  12  60.9 
Centers of Disease Control and Prevention (US)  18  61.1 
Medical Center for Rheumatic Diseases – ARTRICENTER (MEX)  31, 122  61.1 
      62.7 
Cleveland Clinic (US)  39  62.7 
University of Chicago Kids Hospital (US)  44  66.0 
American Academy of Orthopaedic Surgeons (US)  52  69.7 
Clínica Universidad de Navarra (SP)  56  48.7 
University of California San Francisco Medical Center (US)  69  55.1 
Universidad de las Islas Baleares (SP)  105  44.8 
Mayo Clinic (US)  111  54.7 
New York Presbyterian Hospital (US)  166  57.9 
Medical Center for Rehabilitation Premium-Madrid (SP)  175  44.4 
Hospital Universitario Marqués de Valdecilla, Santander (SP)  180  42.9 
Venezuelan Society of Rheumatology  198  56.4 
Austin Heart (US)  217  48.9 
Miami University (US)  218  54.9 
National Institute of Arthritis and Musculoskeletal and Skin Diseases, NIH (US)  238  62.6 
The Medical Center of Plano (US)  247  49.1 
Children's Healthcare of Atlanta (US)  248  55.6 
Yale Medical Group (US)  256  59.2 
Wesley Medical Center (US)  265  54.6 
The University of Iowa (US)  269  48.5 
Hospital General Universitario de Elda Virgen de la Salud (SP)  279  40.5 

MEX, Mexico; NIH, National Institutes of Health; SP, Spain; US, United States of America.

a

General quality using DISCERN – 1: low quality; 2–3: moderate quality; 4–5: high quality.

b

Legibility score using the INFLESZ scale – 0–40: very difficult to read; 40.1–55: somewhat difficult to read; 55.1–65: normal level of legibility; 65.1–80: quite easy to read; >80.1: very easy to read.

The use of different IP addresses changed the information retrieved only in the web pages labeled by Google as “advertisements”. The pages retrieved were the same, although the order in which they appeared changed slightly when different IP addresses were utilized (data not shown).

Discussion

The importance of this cross-sectional study designed to audit web pages is based on 4 premises: a) there is an upward trend in the utilization of the Internet as a source of health information; b) the information obtained via Internet can influence decision making25; c) there has not been an earlier study similar to this one; and d) the high number of potential users (between 500 thousand and a million RA patients only in Spain and Mexico).

The relevance (specificity) of the web pages obtained via Google is low (62%), and is similar using different search engines and in other languages.20,26 The majority of the relevant pages are websites providing information specifically on RA. Regarding treatment, nearly half of the relevant pages refer to conventional therapy, and 13% to alternative therapies. Although half of the web pages are posted by nonprofit organizations, only 15% were created by universities, hospitals, associations or medical societies. These results could vary, however, if another search word or strategies and filters were used.

The content quality of the web pages is irregular; the majority express their objectives, but less than half of them fulfill them, and they do not state the source of the information (references); the quality is rated good in only 1 page in every 4. However, we should bear in mind that DISCERN evaluates the quality of a publication in terms of its content, but not the scientific quality or accuracy of the evidence on which the message it provides is based. On the other hand, there are 2 features that could result in a better or a poorer rating of the relevant pages. The first is the exclusion of several of the DISCERN questions related to the specific evaluation of treatment; the second is the fact that the quality was evaluated by sixth-year medical students. In this respect, we consider that the view of medical students may be “closer” to that of the patients than that of rheumatologists, whose evaluation could be contaminated with assessment bias (scientific validity). Therefore, the rates reported here should be considered subjective and taken from the perspective of the evaluators.

Legibility is a key aspect in preparing any document for patients; it is the whole of the typographical and linguistic characteristics of the written text that make it easy to read and understand. Our results show that the majority of the texts are directed at persons who have attended school for over 12 years; more than half are very difficult or somewhat difficult to read. Difficulties in terms of legibility have also been reported for web-based texts in Spanish concerning other nonrheumatic diseases.27,28 In agreement with our results and with the average formal education in Mexico (11.9 years), more than half of the pages on RA will not be understandable to an average Mexican. The same phenomenon occurs in English. Thus, it has been recommended that the texts have a legibility equivalent to that obtained in 6 years of schooling.28,29

It is commendable that universities, hospitals, societies and medical associations (medical institutions) have a predominant position in the large volume of information on RA on the Internet. The position of these institutions can be evaluated, at least partly, considering the potential users, the place they occupy in the pages that appear with the search engine, the content quality and legibility of the text. The number of potential users (RA patients) is high. On the other hand, the place in which a website appears during the search for information is related to user traffic. In a recent study, the percentage of traffic received at the first position in Google was measured in the US and Canada; the website in the first position contributed 33% of the traffic, compared with 18% for the second position. The cumulative loss in traffic when the sites that appear on the first page in Google are compared with those of the second is even more notable; the sites listed in the first 10 results in Google generated 92% of all of the traffic in an average search, and when we changed to the second page in Google, the traffic fell by 95%.30 In our study, we found 26 medical institutions that posted their web page in Spanish (within the first 300): 17 (65%) from the US, 7 (27%) from Spain, one from Venezuela and one from Mexico. The Arthritis Foundation was in 4th place, with good content and a legibility that was “quite easy to read”; the Spanish Society of Rheumatology occupied places 8 and 9 (repeated), with good content in its first appearance and “normal” legibility; and the American College of Rheumatology was in 10th place, with good content and “quite easy” to read. It is notable that the only Mexican web page which occupied the 31st and 122nd places (repeated), with good content and “normal” legibility, has no rheumatologists among its medical staff. These results correspond to a certain point of time and the position in Google can vary slightly, depending on the IP address.30 However, it is striking that medical institutions in which Spanish is not the official language are in a better position with respect to Spanish-speaking users than the medical institutions of countries in which the predominant mother tongue is Spanish.

In short, RA patients can find a vast amount of information on their disease, although it is irregular in terms of relevance, content quality and legibility. The medical institutions, in general, are also scattered with respect to their position; nevertheless, 3 are within the first 10 positions, although 2 are from the US. There is an area of opportunity so that medical institutions in countries in which Spanish is primarily the mother tongue can take the lead to provide useful information, with quality, that is easy to read and in harmony with the sociocultural characteristics of their patients.

Ethical DisclosuresProtection of human and animal subjects

The authors declare that no experiments were performed on humans or animals for this study.

Confidentiality of data

The authors declare that no patient data appear in this article.

Right to privacy and informed consent

The authors declare that no patient data appear in this article.

Funding

The authors received no external funding for performing this study.

Conflicts of Interest

The authors declare they have no conflicts of interest.

References
1
L. Klareskog,A.I. Catrina,S. Paget
Rheumatoid arthritis
2
E. Loza,J.A. Jover,L. Rodriguez-Rodriguez,L. Carmona
Observed and expected frequency of comorbid chronic diseases in rheumatic patients
Ann Rheum Dis, 67 (2008), pp. 418-421 http://dx.doi.org/10.1136/ard.2007.078659
3
World Health Organization (WHO). Health impact assessment (HIA). Available from: http://www.who.int/hia/evidence/doh/en/ [accessed 09.06.15]
4
L.F. Callahan,T. Pincus
Formal education level as a significant marker of clinical status in rheumatoid arthritis
Arthritis Rheum, 31 (1988), pp. 1346-1357
5
C. Ramos-Remus,A.L. Salcedo-Rocha,R.E. Prieto-Parra,F. Galvan-Villegas
How important is patient education?
Baillieres Best Pract Res Clin Rheumatol, 14 (2000), pp. 689-703 http://dx.doi.org/10.1053/berh.2000.0107
6
J.G. Anderson
Consumers of e-health: patterns of use and barriers
Soc Sci Comput Rev, 22 (2004), pp. 7
7
M. Sanchez-Mendiola
El enfermo ante la información científica: ¿Es el médico guía u obstáculo?
CONAMED, 10 (2005), pp. 4
8
A.R. Jadad,C.A. Rizo,M.W. Enkin
I am a good patient, believe it or not
9
K. Nanjid,C. Ochir,S. Nyamjav,P. Mendsaikhan
Assessment of changes in health care needs
World Hosp Health Serv, 50 (2014), pp. 27-30
10
D.V. Canyon
Crisis management teams in health organisations
J Bus Contin Emer Plan, 5 (2012), pp. 365-372
11
Pew Research Center: Internet, Science & Technology. Highlights of the Pew Internet Project's research related to health and health care. Available from: http://www.pewinternet.org/fact-sheets/health-fact-sheet [accessed 12.06.15]
12
F.B. Wood,B. Lyon,M.B. Schell,P. Kitendaugh,V.H. Cid,E.R. Siegel
Public library consumer health information pilot project: results of a National Library of Medicine evaluation
Bull Med Libr Assoc, 88 (2000), pp. 314-322
13
P. Hartzband,J. Groopman
Untangling the Web — patients, doctors, and the Internet
N Engl J Med, 362 (2010), pp. 1063-1066 http://dx.doi.org/10.1056/NEJMp0911938
14
H.S. Wald,C.E. Dube,D.C. Anthony
Untangling the Web — the impact of Internet use on health care and the physician-patient relationship
Patient Educ Couns, 68 (2007), pp. 218-224 http://dx.doi.org/10.1016/j.pec.2007.05.016
15
G. Eysenbach,A.R. Jadad
Evidence-based patient choice and consumer health informatics in the Internet age
J Med Internet Res, 3 (2001), pp. E19 http://dx.doi.org/10.2196/jmir.3.2.e19
16
G. Eysenbach,J. Powell,O. Kuss,E.R. Sa
Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review
JAMA, 287 (2002), pp. 2691-2700
17
Ethnologue The World Languages (summary by language size). Available from: http://www.ethnologue.com/statistics/size [accessed 12.06.15]
18
Instituto Nacional de Estadistica. Población que usa Internet (en los últimos tres meses). Availabe from: http://www.ine.es/ss/ [accessed 15.02.16]
19
Asociación Mexicana de Internet. Estudio de Hábitos de los Usuarios de Internet en México. Available from: https://www.amipci.org.mx/es/noticiasx/2241-alcanza-internet-el-51-de-penetracion-entre-los-usuarios-potenciales-de-mexico-amipci [accessed 15.02.16]
20
M.E. Suarez-Almazor,C.J. Kendall,M. Dorgan
Surfing the Net — information on the World Wide Web for persons with arthritis: patient empowerment or patient deceit?
J Rheumatol, 28 (2001), pp. 185-191
21
Market share statistics for Internet technologies. Realtime Web analytics with no sampling. Available from: https://www.netmarketshare.com/search-engine-market-share.aspx?qprid=4&qpcustomd=0 [accessed 15.02.16]
22
Quality criteria for consumer health information. The DISCERN instrument. Available from: http://www.discern.org.uk/discern_instrument.php [accessed 10.06.15]
23
Universidad de Alicante. Herramientas de análisis de la legibilidad. Available from: http://accesibilidadweb.dlsi.ua.es/?menu=hr-legibilidad [accessed 09.10.15]
24
Virtual Private Network. Available from: https://www.hidemyass.com [accessed 15.05.15]
25
Al-Salebi FA, Reynolds M. The important characteristics to make a good Website. Available from: https://people.rit.edu/fxa4314/737/research/WebSiteResearchPaperAlsalebiFaisal.pdf [accessed 15.05.15]
26
H.A. Kim,Y.D. Bae,Y.I. Seo
Arthritis information on the Web and its influence on patients and physicians: a Korean study
Clin Exp Rheumatol, 22 (2004), pp. 49-54
27
M. Bea-Munoz,M. Medina-Sanchez,Florez-Garcia M.T. Readability
Internet accessibility of informative documents for spinal cord injury patients in Spanish
An Sist Sanit Navar, 38 (2015), pp. 255-262
28
R.L. Rhee,J.M. Von Feldt,H.R. Schumacher,P.A. Merkel
Readability and suitability assessment of patient education materials in rheumatic diseases
Arthritis Care Res (Hoboken), 65 (2013), pp. 1702-1706
29
M.A. Graber,C.M. Roller,B. Kaeble
Readability levels of patient education material on the World Wide Web
J Fam Pract, 48 (1999), pp. 58-61
30
Chitika. The value of Google results positioning. Available from: http://www.chitika.com/google-positioning-value [accessed 22.12.15]

Please cite this article as: Castillo-Ortiz JD, Valdivia-Nuno JJ, Ramirez-Gomez A, Garagarza-Mariscal H, Gallegos-Rios C, Flores-Hernandez G, et al. Legibilidad, relevancia y calidad de la información en español en la Web para pacientes con artritis reumatoide. Reumatol Clin. 2017;13:139–144.

Corresponding author. (Jose Dionisio Castillo-Ortiz josedco006@hotmail.com)
Copyright © 2016. Elsevier España, S.L.U. and Sociedad Española de Reumatología y Colegio Mexicano de Reumatología
Reumatol Clin 2017;13:139-44 - Vol. 13 Num.3 DOI: 10.1016/j.reumae.2016.04.010