Transparency and Quality of Health Apps: The HON Approach.

We are well into the 21st century and the Internet has been around long enough that there are adults who have not known a world without this wonderful tool. And just as time has gone since the beginnings of the Internet, so too has it developed, probably above and beyond the wildest dreams of its founders. These developments, though mostly positive, also have their share of the not so positive. One of these challenges is the difficulty in maintaining accuracy and quality of all the information, data gathered, aggregated or automatically generated being displayed on the Internet on Web websites or via mobile application, and this is a concern in the health domain. In this paper, we attempt to discuss in detail, some of the latest developments along with the challenges each of them entail and proposed Code of Conduct for health apps and connected objects.

Accessing Reliable Health Information on the Web: A Review of the HON Approach.

Accessing online health content of high quality and reliability presents challenges. Laypersons cannot easily differentiate trustworthy content from misinformed or manipulated content. This article describes complementary approaches for members of the general public and health professionals to find trustworthy content with as little bias as possible. These include the Khresmoi health search engine (K4E), the Health On the Net Code of Conduct (HONcode) and health trust indicator Web browser extensions.

Health On the Net's 20 Years of Transparent and Reliable Health Information.

The Health On the Net Foundation (HON) was born in 1996, during the beginning of the World Wide Web, from a collective decision by health specialists, led by the late Jean-Raoul Scherrer, who anticipated the need for online trustworthy health information. Because the Internet is a free space that everyone shares, a search for quality information is like a shot in the dark: neither will reliably hit their target. Thus, HON was created to promote deployment of useful and reliable online health information, and to enable its appropriate and efficient use. Two decades on, HON is the oldest and most valued quality marker for online health information. The organization has maintained its reputation through dynamic measures, innovative endeavors and dedication to upholding key values and goals. This paper provides an overview of the HON Foundation, and its activities, challenges, and achievements over the years.

Accessing Quality Online Health Information: What Is the Solution?

The majority of the adult population in both Europe and North America have access to the internet. Over 70% state that they have used the internet to look for health information and the majority started their search at a search engine. Given that search engines list sites according to popularity and not quality, it is imperative that users have a means of discerning trustworthy and honest information from non-reliable health information. The HONcode, a set of eight quality guidelines, ensures access to standardized trustworthy health information which can be used as a tool to guide consumers.

Automated Detection of Health Websites' HONcode Conformity: Can N-gram Tokenization Replace Stemming?

Authors evaluated supervised automatic classification algorithms for determination of health related web-page compliance with individual HONcode criteria of conduct using varying length character n-gram vectors to represent healthcare web page documents. The training/testing collection comprised web page fragments extracted by HONcode experts during the manual certification process. The authors compared automated classification performance of n-gram tokenization to the automated classification performance of document words and Porter-stemmed document words using a Naive Bayes classifier and DF (document frequency) dimensionality reduction metrics. The study attempted to determine whether the automated, language-independent approach might safely replace word-based classification. Using 5-grams as document features, authors also compared the baseline DF reduction function to Chi-square and Z-score dimensionality reductions. Overall study results indicate that n-gram tokenization provided a potentially viable alternative to document word stemming.

User-oriented evaluation of a medical image retrieval system for radiologists.

PURPOSE: This article reports the user-oriented evaluation of a text- and content-based medical image retrieval system. User tests with radiologists using a search system for images in the medical literature are presented. The goal of the tests is to assess the usability of the system, identify system and interface aspects that need improvement and useful additions. Another objective is to investigate the system's added value to radiology information retrieval. The study provides an insight into required specifications and potential shortcomings of medical image retrieval systems through a concrete methodology for conducting user tests. METHODS: User tests with a working image retrieval system of images from the biomedical literature were performed in an iterative manner, where each iteration had the participants perform radiology information seeking tasks and then refining the system as well as the user study design itself. During these tasks the interaction of the users with the system was monitored, usability aspects were measured, retrieval success rates recorded and feedback was collected through survey forms. RESULTS: In total, 16 radiologists participated in the user tests. The success rates in finding relevant information were on average 87% and 78% for image and case retrieval tasks, respectively. The average time for a successful search was below 3 min in both cases. Users felt quickly comfortable with the novel techniques and tools (after 5 to 15 min), such as content-based image retrieval and relevance feedback. User satisfaction measures show a very positive attitude toward the system's functionalities while the user feedback helped identifying the system's weak points. The participants proposed several potentially useful new functionalities, such as filtering by imaging modality and search for articles using image examples. CONCLUSION: The iterative character of the evaluation helped to obtain diverse and detailed feedback on all system aspects. Radiologists are quickly familiar with the functionalities but have several comments on desired functionalities. The analysis of the results can potentially assist system refinement for future medical information retrieval systems. Moreover, the methodology presented as well as the discussion on the limitations and challenges of such studies can be useful for user-oriented medical image retrieval evaluation, as user-oriented evaluation of interactive system is still only rarely performed. Such interactive evaluations can be limited in effort if done iteratively and can give many insights for developing better systems.

Automated Detection of HONcode Website Conformity Compared to Manual Detection: An Evaluation.

BACKGROUND: To earn HONcode certification, a website must conform to the 8 principles of the HONcode of Conduct In the current manual process of certification, a HONcode expert assesses the candidate website using precise guidelines for each principle. In the scope of the European project KHRESMOI, the Health on the Net (HON) Foundation has developed an automated system to assist in detecting a website's HONcode conformity. Automated assistance in conducting HONcode reviews can expedite the current time-consuming tasks of HONcode certification and ongoing surveillance. Additionally, an automated tool used as a plugin to a general search engine might help to detect health websites that respect HONcode principles but have not yet been certified. OBJECTIVE: The goal of this study was to determine whether the automated system is capable of performing as good as human experts for the task of identifying HONcode principles on health websites. METHODS: Using manual evaluation by HONcode senior experts as a baseline, this study compared the capability of the automated HONcode detection system to that of the HONcode senior experts. A set of 27 health-related websites were manually assessed for compliance to each of the 8 HONcode principles by senior HONcode experts. The same set of websites were processed by the automated system for HONcode compliance detection based on supervised machine learning. The results obtained by these two methods were then compared. RESULTS: For the privacy criterion, the automated system obtained the same results as the human expert for 17 of 27 sites (14 true positives and 3 true negatives) without noise (0 false positives). The remaining 10 false negative instances for the privacy criterion represented tolerable behavior because it is important that all automatically detected principle conformities are accurate (ie, specificity [100%] is preferred over sensitivity [58%] for the privacy criterion). In addition, the automated system had precision of at least 75%, with a recall of more than 50% for contact details (100% precision, 69% recall), authority (85% precision, 52% recall), and reference (75% precision, 56% recall). The results also revealed issues for some criteria such as date. Changing the "document" definition (ie, using the sentence instead of whole document as a unit of classification) within the automated system resolved some but not all of them. CONCLUSIONS: Study results indicate concordance between automated and expert manual compliance detection for authority, privacy, reference, and contact details. Results also indicate that using the same general parameters for automated detection of each criterion produces suboptimal results. Future work to configure optimal system parameters for each HONcode principle would improve results. The potential utility of integrating automated detection of HONcode conformity into future search engines is also discussed.

General vs health specialized search engine: a blind comparative evaluation of top search results.

This paper presents the results of a blind comparison of top ten search results retrieved by Google.ch (French) and Khresmoi for everyone, a health specialized search engine. Participants--students of the Faculty of Medicine of the University of Geneva had to complete three tasks and select their preferred results. The majority of the participants have largely preferred Google results while Khresmoi results showed potential to compete in specific topics. The coverage of the results seems to be one of the reasons. The second being that participants do not know how to select quality and transparent health web pages. More awareness, tools and education about the matter is required for the students of Medicine to be able to efficiently distinguish trustworthy online health information.

Observations and lessons learnt from non health professionals evaluating a health search engine.

This article presents the results of one of the stages of the user-centered evaluation conducted in a framework of the EU project Khresmoi. In a controlled environment, users were asked to perform health-related tasks using a search engine specifically developed for trustworthy online health information. Twenty seven participants from largely the Czech Republic and France took part in the evaluation. All reported overall a positive experience, while some features caused some criticism. Learning points are summed up regarding running such types of evaluations with the general public and specifically with patients.

User tests for assessing a medical image retrieval system: a pilot study.

Content-based image retrieval (CBIR) has often been proposed to assist medical decision making in complement to textual information search. However, applications of this novel technology have rarely reached the end users. The study presented in this paper describes the design and setup for performing pilot user tests in order to assess a medical information retrieval system that supports CBIR with the goal of having more detailed tests with an updated system. Five individuals with medical education participated in the study at the University hospitals of Geneva. They were recorded and observed while interacting with the system, and then provided feedback on the usability of the system. Participants seemed to understand the concept and practical uses of the new tools, and needed 10-15 minutes to feel confident with the system. The results of this pilot study will be used for improving the system functionalities as well as an input for designing a new iteration of larger-scale user tests among radiologists.

Key-phrase based classification of public health web pages.

This paper describes and evaluates the public health web pages classification model based on key phrase extraction and matching. Easily extendible both in terms of new classes as well as the new language this method proves to be a good solution for text classification faced with the total lack of training data. To evaluate the proposed solution we have used a small collection of public health related web pages created by a double blind manual classification. Our experiments have shown that by choosing the adequate threshold value the desired value for either precision or recall can be achieved.

Partial nephrectomy online: a preliminary evaluation of the quality of health information on the Internet.

UNLABELLED: What's known on the subject? and What does the study add? Patients are highly likely to access the Internet for health information, and studies have reported that inaccurate or low-quality information may alter patients' expectations and negatively impact informed decision-making. In a unique collaboration with the Health On the Internet (HON) Foundation, we evaluated the top 20 search results for the urology search term 'partial nephrectomy,' and identified the highest and lowest scoring criteria to increase awareness of areas of concern and improvement. OBJECTIVE: To further evaluate the quality of information available on the Internet with regard to the management of localized renal cancer, we evaluated websites providing information on 'partial nephrectomy' in conjunction with the Health On the Internet (HON) Foundation. Many patients now utilize the Internet as a resource to provide further information on disease, treatments and outcomes, and health information on the Internet is largely unregulated. Inaccurate information may contribute to unrealistic expectations and dissatisfied patients. PATIENTS AND METHODS: A google.com search identified the top 30 websites for the search term 'partial nephrectomy'. The HON Foundation evaluated each website according to the eight principles for Health on the Internet code of conduct (HONcode) certification and reported the overall frequency of certification, as well as individual website compliance with each of the principles. RESULTS: Overall, seven (23.3%) of 30 websites met the requirements of HONcode certification and an additional two (6.7%) websites were under review to maintain their certification based on updating their resources. The remaining 21 (70%) websites did not meet the standards for certification. The lowest performing criteria included proper citation of medical information and a clear distinction of advertising from editorial content. CONCLUSIONS: The low rate of HONcode compliance for these websites illustrates the poor quality of information that patients may encounter when researching options for nephron-sparing surgery, which may have a significant impact on patient decision-making and treatment choices. Physicians should be aware of the quality of Internet resources and how to best use these tools to help guide patients to websites with valid information.

Online health information search: what struggles and empowers the users? Results of an online survey.

The most popular mean of searching for online health content is a general search engine for all domains of interest. Being general implies on one hand that the search engine is not tailored to the needs which are particular to the medical and on another hand that health domain and health-specific queries may not always return adequate and adapted results. The aim of our study was to identify difficulties and preferences in online health information search encountered by members of the general public. The survey in four languages was online from the 9th of March until the 27th of April, 2011. 385 answers were collected, representing mostly the opinions of highly educated users, mostly from France and Spain. The most important characteristics of a search engine are relevance and trustworthiness of results. The results currently retrieved do not fulfil these requirements. The ideal representation of the information will be a categorization of the results into different groups. Medical dictionaries/thesauruses, suggested relevant topics, image searches and spelling corrections are regarded as helpful tools. There is a need to work towards better customized solutions which provide users with the trustworthy information of high quality specific to his/her case in a user-friendly environment which would eventually lead to making appropriate health decisions.

Trustworthiness and relevance in web-based clinical question answering.

Question answering systems try to give precise answers to a user's question posed in natural language. It is of utmost importance that the answers returned are relevant to the user's question. For clinical QA, the trustworthiness of answers is another important issue. Limiting the document collection to certified websites helps to improve the trustworthiness of answers. On the other hand, limited document collections are known to harm the relevancy of answers. We show, however, in a comparative evaluation, that promoting trustworthiness has no negative effect on the relevance of the retrieved answers in our clinical QA system. On the contrary, the answers found are in general more relevant.

Evolution of health web certification through the HONcode experience.

Today, the Web is a media with increasing pervasiveness around the world. Its use is constantly growing and the medical field is no exception. With this large amount of information, the problem is no longer about finding information but assessing the credibility of the publishers as well as the relevance and accuracy of the documents retrieved from the web. This problem is particularly relevant in the medical area which has a direct impact on the well-being of citizens and in the Web 2.0 context where information publishing is easier than ever. To address the quality of the medical Internet, the HONcode certification proposed by the Health On the Net Foundation (HON) is certainly the most successful initiative. The aims of this paper are to present certification activity through the HONcode experience and to show that certification is more complex than a simple code of conduct. Therefore, we first present the HONcode, its application and its current evolutions. Following that, we give some quantitative results and describe how the final user can access the certified information.

Results of the 10 HON survey on health and medical internet use.

The Internet is increasingly being used as a means to search and communicate health information. As the mission of Health on the Net Foundation (HON) is to guide healthcare consumers and professionals to trustworthy online information, we have been interested in seeing the trend of the attitudes towards Internet use for health purposes since 1996. This article presents the results of the 10th HON survey conducted in July-August 2010 (in English and French). It was hosted on the HON site with links from Facebook and Twitter and from HONcode certified web sites. There were 524 participants coming mainly from France (28%), the UK (18%) and the USA (18%). 65% of participants represented the "general public", while the remaining 35% were professionals. Information quality remains the main barrier users encounter while looking for health information online; at the same time, 79% believe they critically assess online content. Both patients and physicians consider the Internet to be helpful in facilitating their communication during consultations, although professionals are more sceptic than the general public. These results justify the continuing efforts of HON to raise public awareness regarding online health information and the ethical, quality and transparency issues, and to educate and guide users towards trustworthy health information.

Improving the transparency of health information found on the internet through the honcode: a comparative study.

This study aims to show that health websites not asking for HONcode certification (Control sample websites A) do not respect elementary ethical standards such as the HONcode. The HONcode quality and ethical standards and the certification process have been developed by the Health on the Net Foundation to improve the transparency of the health and medical information found on the Internet. We compared the compliance with the 8 HONcode principles, and respectively the respect of principles 1 (authority), 4 (assignment), 5 (justification) and 8 (honesty in advertising and editorial policy) by certified websites (A) and by health websites which have not requested the certification (B). The assessment of the HONcode compliance was performed by HON evaluators by the same standards for all type of sites. Results shows that 0.6% of health websites not asking for HONcode certification does respect the eight HONcode ethical standards vs. 89% of certified websites. Regarding the principles 1, 4, 5 and 8, 1.2% of B respect these principles vs. 92% for A. The certification process led health websites to respect the ethical and quality standards such as the HONcode, and disclosing the production process of the health website.

Enriching the trustworthiness of health-related web pages.

We present an experimental mechanism for enriching web content with quality metadata. This mechanism is based on a simple and well-known initiative in the field of the health-related web, the HONcode. The Resource Description Framework (RDF) format and the Dublin Core Metadata Element Set were used to formalize these metadata. The model of trust proposed is based on a quality model for health-related web pages that has been tested in practice over a period of thirteen years. Our model has been explored in the context of a project to develop a research tool that automatically detects the occurrence of quality criteria in health-related web pages.

[Certification of health-related websites in France]. / Certification des sites dédiés à la santé en France.

The 2004 statute that created the French National Authority for Health (HAS, Haute Autorité de Santé) required it to establish a procedure for the certification of health-related web sites. The HAS established a procedure based on the HONcode certification scheme set up by the Health On the Net Foundation, with which HAS has a partnership agreement. The HONcode includes eight principles that govern the quality of online heath information and its presentation (quality of the production process). The collaboration between HAS and HON has already led to improvements in a large number of web sites in France and to their certification. The main advantages of certification for site publishers are better site quality and enhanced credibility rather than a larger audience. Quality certification has little impact on the choice of site by Internet users as they tend to use search engines to find health-related information. Future development of the procedure should work to increase the value of certification both by improving the quality of sites and in signaling quality to Internet users.

Supervised approach to recognize question type in a QA system for health.

Many attempts have been made in the QA domain but no system applicable to the field of health is currently available on the Internet. This paper describes a bilingual French/English question answering system adapted to the health domain and more particularly the detection of the question's model. Indeed, the Question Analyzer module for identifying the question's model has a greater effect on the rest of the QA system. Our original hypothesis for the QA is that a question can be defined by two criteria: type of answer expected and medical type. These two must appear in the step of detection of the model in order to better define the type of question and thus, the corresponding answer. For this, questions were searched on the Internet and then given to experts in order to obtain classifications according to criteria such as type of question and type of medical context as mentioned above. In addition, tests of supervised and non-supervised classification were made to determine features of questions. The result of this first step was that algorithms of classification were chosen. The results obtained showed that categorizers giving the best results were the SVM. Currently, for a set of 100 questions, 84 are well categorized in English and 68 in French according to the type of answer expected. This figures fall to less than 50% for the medical type. Evaluations have showed that the system was good to identify the type of answer expected and could be enhanced for the medical type. It leads us to use an external source of knowledge: UMLS. A future improvement will be the usage of UMLS semantic network to better categorize a query according to the medical domain.