Microsoft word - $asq5923513_file000001_87571220.doc

HEALTH INFORMATION PORTALS: HOW CAN WE IMPROVE Rosetta Manaszewicz, Monash University rmanasz@optusnet.com.au Monash University PO Box 197, Caulfield East, 3145, Australia Health information portals (HIP) are now common place. Governments and other health agencies are using HIPs extensively to reduce costs and distribute information more effectively. Generally, HIPs are not very technically sophisticated specifically in terms of options for improving searching. There are many ways in which searching and retrieving relevant information can be improved. This paper presents an exploratory study which investigated five HIPs. Each HIP offered a range of features and functionality to assist with searching. Our research explored the features and functionality of each HIP. Through usability evaluations we compared the response of users to each HIP and identified users’ preferences for improved searching. We found that HIPs with improved search functionality and other features that assisted searching were better received by the users. Users regarded these portals as easier to understand, easier to use, required fewer steps in retrieving information and were more likely to say they would return. Comments from users are provided to illustrate further the importance of providing effective functionality. The paper concludes with recommendations for Health Information Portal builders on what is needed to improve the user search experience. Keywords: Health information portals, functionality, usability. INTRODUCTION
Governments worldwide are grappling with increased demands for health information and pressures on health systems. Coupled with this is an explosion in the quantity of health information available particularly through the Internet. Many governments and health agencies are disseminating health information via the Internet. Reasons for doing so include mitigating against the problems of poor quality information and helping defray costs in information distribution. Research findings by Kennedy (2003) suggest a strong correlation between health outcomes and the level to which patients are informed, that is, when patients have more information relating to their health this leads to improved health outcomes. Despite the best efforts of the various agencies distributing information, health consumers frequently report that the Internet based health information they find is not useful and are not meeting user needs (Vermaas and Wijngaert 2005). Information relevance is most important to health consumers (HON 2006) yet finding appropriate, relevant and timely information is often difficult (Zeng et al. 2004). Information is often incomplete (HON 2006) and difficult to read (Sillence et al. 2004). Further, most health Websites and portals swamp users with information (Burstein et al. 2005). There are however, a variety of ways in which users can be assisted in overcoming the identified problems in searching and retrieving relevant information. To date there has been limited research exploring users’ perceptions of HIPs and what encourages use, this research sought to fill this gap. The exploratory study reported in this paper investigated the features and functionality of five non commercial Health Information Portals (HIP). Through usability evaluations in a laboratory setting we explored users’ responses to each of the HIPs, their expectations when searching for health information, in particular how they responded to their search experience and the extent to which this was influenced by the features and functionality of the HIP and their willingness to use the same source in the future. The research found that HIPs with improved search features/ functionality, were better received by the users. Users also rated these portals as easy to use and as a result, were more likely to return to them. HEALTH INFORMATION PORTALS
This section discusses mechanisms for improving searching and information retrieval from health websites or HIPs and factors that determine success. For the purposes of this paper a HIP is defined as an internet based system which assists users find relevant internet based health information resources. A HIP would generally facilitate information retrieval from other websites through a range of features and functionality but not store or provide information specifically. Health websites should enable health consumers to easily find relevant information. A number of features a health information website or portal might have to support searching were identified by Luo and Najdawi (2004): “a catalogue of health information” - a mechanism for organizing edited health information; a search engine; “a personalization system” described as allowing health consumers to customize the interface based on their preferences and; “a network of communities” where health consumers can exchange information with others (Luo and Najdawi 2004). Other technologically possible features include 1. Providing a level of information differentiation to improve searching such as information organized based on a specific disease, drugs, for children, personal stories. 2. Spellchecking, many medical terms are unfamiliar to health consumers and difficult to spell. 3. Parsing where users can enter a question or sentence and the search examines both the key word/s and the context. 4. An ontology and/or thesaurus helps users identify specific medical search terms and narrow searching. Other factors that will determine how successful a HIP will be include the users’ ability to find relevant information (Josefsson 2006; Pew/Internet 2006). One study reported that 37% of users were unable to find relevant health information (Zeng et al. 2004). Through observations, Zeng et al (2004) found a major reason was “the consumers’ use of simple search strategies (browsing or short text queries) that did not characterize their information needs well.” Zeng et al (2004) conclude from their research that there are three ways searching can be improved; health consumers making queries more specific, improved search functionality and limiting information retrieved. Research by Pew/Internet (2006) found 25% of users were overwhelmed by the amount of information retrieved and 22% were frustrated by their inability to find relevant information. Easy to use websites are also critical to success (Klein 2007). The 2006 HON survey (2006) found ‘Easy to use’ was ranked highest as helpful for searching. Factors such as the quality of the navigation, how long it takes to retrieve information and how easy the text is to read on the screen all contribute to ease of use (HON 2006). Research also suggests that these elements will impact on the overall user experience, influencing whether or not users will return to a Website (Fisher et al. 2007). There are many examples of health information portals. Non-commercial providers are the focus of this research as previous research has found health consumers trust these sites are more (HON Survey 2006). Non-commercial providers include hospitals, governments and patient/advocacy groups. The choice for the average health consumer is therefore vast. The question this poses is how does the health consumer choose between these sources? This is no trivial decision if the patient is facing a life threatening illness and information and answers in relation to treatment are critical (Josefsson 2006). The purpose of the research reported in this paper was to examine the features and functionality offered by HIPs in relation to searching, the impact this has on the user’s experience and to identify what users want. Whilst the authors recognise the importance of information quality, quality is a complex issue and was outside the scope of this research. The questions we sought to answer were: • What different features / functionality are provided for users of HIPs to improve searching? • What searching features / functionality do users want? • What impact do different features / functionality have on users’ overall experience, including searching and their intention to return to a HIP? RESEARCH APPROACH
Five generic non-commercial, HIPs were identified. Portals from Europe (Health on the Net), North America (Medline Plus and Mayo Clinic) and Australia (Better Health and HealthInsite) where the research was conducted, were identified. The sites were selected based on the following criteria: • Government sponsored. The HON Survey (2006) identified that 79% of health consumers preferred a government agency to be responsible for online health information provision. • The health information provided is generic that is not specific to a disease, age group etc. This was to ensure the relevance of the task to all users. • HIPs were selected based on the level of features / functionality available. We looked for HIPs with a greater number of features / functionality to test their effectiveness. Note only two Australian HIPs met the criteria. The Australian portals are the two key government sponsored portals, Medline Plus claims to be the largest medical library in the world (Medline Plus 2009), the Mayo clinic is the largest not for profit practice in the world (Mayo 2009) and HON describes itself as one of the most respected HIPs in the world (HON 2006). It should be noted that in 1999 (not in subsequent years) the HON survey (1999) asked users which Website most closely met their needs. The results found Medline Plus was rated as the highest, Mayo Health System (now Mayo Clinic) was listed number eight and HON number 10. From previous work we identified a number of features/functionality that can assist users searching for information (Fisher et al. 2007). We examined the home page of each HIP for evidence of features/functionality. Users are most likely to use features/functionality visible from the home page. Table 1 details the features/functionality and the method used to assess each. All had a search engine. Feature/functionality How the feature or functionality was determined
Personalisation
Examined if the portal asked for information to help in retrieving information relevant to the user. Did the HIP start an initial dialog to identify user’s personal needs eg ask the gender of the user, age or information type? Analysed information differentiation offered. Were users given topics to search within? Could the user retrieve other types of information eg personal stories? Spell checking and ‘sounds like’ indexing tested using misspelt words. The sentence "What are the side effects of Ventolin?" was used to search each HIP to test if returned results contained both the phrase ‘side effect’ and ‘Ventolin’ Checked for an ontology and/or thesaurus. For example was a list of possible search words or terms available for users to access? Were newsletters, forums or individual feedback available? Table 1 – Features and functionality for HIPs Usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (Bevan 2001). The research examined users’ response to the five HIPs, through a usability evaluation - in particular aspects of the HIPs that related to searching and retrieving information. The usability instrument was based on an instrument developed and tested previously by one of the authors (Fisher et al. 2004). A full copy of the instrument can be found at http://www.sims.monash.edu.au/staff/jfisher/. The instrument is based on key items identified in the literature as important in websites designed for information distribution. The key items are how easy information is to read including terminology and instructions on using the Website (Becker and Mottay 2001; Cox and Dale 2002; Kunst 2002). The quantity of information presented (Bruce 1999; Zeng et al. 2004). The design and presentation of the text (Becker and Mottay 2001; Sutcliffe 2002) and how easy it is to use including locating information and navigation (Nielsen 1999; Fu and Salvendy 2002; Hargittai 2003). The usability evaluation involved 223 users. The instrument consisted of Likert scale type statements, questions requiring a response from 1 scored lowest to 5 highest, questions requiring a Yes/No response (scored Yes 2, No 1) and questions requiring a free text response. Numerous usability instruments exist however each evaluation conducted needs to be designed specifically for the goals of the research and adjusted to meet those goals (Dumas and Redish 1994, 185). The instrument contains questions and statements similar to those in other studies (Zhang et al 2000; Nel et al. 1999). The users were students, studying Human Computer Interaction, as part of their studies and during a class they participated in the usability test. 223 users evaluated the portals resulting in 411 usable evaluations. 33% of participants were female and 67% male and 91% described themselves as very experienced with Internet searching. The use of students for usability evaluations such as this is in line with other studies (Nel et al. 1999; Zhang et al. 2000). Abdinnour-Helm et al. (2005) argue that students can be appropriate providing they are similar to Web users generally and are likely to perform the tasks being investigated. The majority of users (91%) were aged between 21 and 30 years. This age group are likely to have searched for health information (Fox 2006). The users were asked to ‘Think of a health issue that is important to you, a friend or a family member. Using each of the health websites, search for information on that topic’. Users were provided with examples of information they might search for (exercise, diet, medication, a specific health condition). Each user conducted a search on two HIPs; one was an Australian portal and one international portal. Once the search was completed users completed the questionnaire and reviewed the next HIP. Each user spent about 15 minutes searching each HIP. The quantitative data were analysed using SPSS. A Factor Analysis was conducted to explore the relationship between the identified elements and to assess the degree to which factors were measuring the same or a similar concept (Bryman and Cramer 1992) and descriptive statistics to compare results for individual HIPs. The qualitative data was analysed for the research themes relating to searching, using a meta-matrix approach which is described by Miles and Huberman (1994, pg 93) as “essentially the 'crossing' of two lists, set up as rows and columns.” Users were asked if they had searched for information on the internet, 10% said they had never used the internet to search for health information, 70% indicated they occasionally search, 18% search often. Users were also asked how they searched. Users could tick more than one option. Most users, 185 (97%) had used a search engine, 7% used an Australian Website and 2% used an international Website. Next each portal is described briefly and the features and functionality available. Features and functionality
Medline Plus is the National Library of Medicine (NLM) located in Bethesda,
Maryland, USA. It describes itself as bringing “together authoritative information from NLM, the
National Institutes of Health (NIH), and other government agencies and health-related organizations.”
Table 2 presents an analysis of features and functionality of Medline Plus.
Feature / Functionality
750 available health topics organised according to diseases, diagnoses, demographics and wellness. Searchable topics on the different parts of the body, diagrams are provided also. Yes. Dictionary is also provided to help with spelling and definitions. A medical encyclopaedia and dictionary are provided. Within these users are able to search alphabetically with links to definitions and specific topics. Current health news items are available. There are interactive tutorials including videos on particular health topics, videos on surgery and other procedures. Table 2 Features and functionality available on Medline Plus Mayo Clinic: () Website describes its website as providing “information and
services from the world's first and largest integrated, not-for-profit group medical practice.” Table 3
presents the analysis of features and functionality of the Mayo Clinic.
Feature / Functionality
Diseases / Treatments listed alphabetically. Users can search by clicking on a letter. Patient stories are provided for some topics. Subscription to an e-mail newsletter is available Table 3 Features and functionality available on Mayo Clinic Health on the Net Foundation (HON) ( Swiss based says it is “the leading organization
promoting and guiding the deployment of useful and reliable online medical and health information,
and its appropriate and efficient use.” Table 4 presents the analysis of features and functionality.
Feature / Functionality
Differentiates between information for patients/individual from medical persons. Searching was possible based on age and gender. Table 4 Features and functionality available on HON HealthInsite (, established by the Australian Commonwealth Government
and aims “to improve the health of Australians by providing easy access to quality information about
human health.” Table 5 presents the analysis of HealthInsite features/functionality.
Feature / Functionality
The advanced search allows uses to indicate if the information is for a ‘child, youth, adult easy, adult medium or professional person’. Preference for other document types can also be indicated for example data, images, document, multimedia, statistics. Wide range of health topics organised under different headings such as diseases, wellbeing and stages of life. A-Z is search available. Yes but only if the user used ‘Advanced Search’ Monthly newsletter available on subscription Table 5 Features and functionality available on HealthInsite Better Health ( established by the Victorian Department of Human
Services, Australia. Described as “a consumer health information Website for the Victorian
community.” Table 6 presents the analysis of features/functionality for better Health.
Feature / Functionality
Has personal stories and limited number of health topics. An additional category of ‘healthy eating’ is provided. Some information is organised according to gender. Fact sheets organised by category or A-Z, hot topic of the week and some podcasts. Latest updates are available for those who subscribe. Table 6 Features and functionality available on Better Health Users were asked “Thinking about what you need to help you find the most relevant information please indicate on a scale of 1 (not important) to 5 (very important) how important each of the following features are”. This question was asked after users had searched using both portals. 200 users answered the question. Table 7 describes how each feature/functionality was described to participants and responses. Feature/ functionality
Description provided to participants
User rating
A list of terms to help you refine your search Spell check: for example offering a list of options if a word is Choice of information type for example being able to choose medical /scientific information or information in simple language or personal stories. Provided newsletters, e-mail, feedback from a professional. Personalisation for example asking your preference for information, your age, gender any other information about your search that relates to you Table 7 User views of features and functionality Usability evaluation results
The usability evaluations explored more items than those presented in this paper. The results are confined to reporting on aspects influencing the users’ search experience. This includes factors such as ease of use, information quantity, terminology, design and presentation of text, navigation and information on searching. Each user evaluated two HIPs and all users evaluated one of the Australian HIPs. 76 users evaluated Medline, 66 evaluated Mayo Clinic, HON was evaluated by 65 users, 82 users evaluated Better Health and 122 evaluated HealthInsite. The HIPs were randomly assigned hence the difference in the number of users who evaluated each of the different portals. 4.2.1 Quantitative user responses to individual Health Information Portals To understand the relationship between the different elements a factor analysis on nine of the variables
was undertaken. Tables 8 and 9 present the questions/statements put to users and the results.
Communalities
The text was displayed in a way that was easy to read The language used was easy to understand All the information I required to complete the task was on the Website The number of steps required to get to the information I wanted was acceptable It was easy to find information on the topic that was relevant for me I understood the terminology used on the Website I found the search function useful in helping me locate relevant information Extraction Method: Principal Component Analysis. Table 8 Factor Analysis Communalities Extraction Method: Principal Component Analysis. Rotated Component Matrix
It was easy to find information on the topic that was relevant for me All the information I required to complete the task was on the Website The number of steps required to get to the information I wanted was acceptable I found the search function useful in helping me locate relevant information The text was displayed in a way that was easy to read The language used was easy to understand I understood the terminology used on the Website Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. Rotation converged in 4 iterations. Table 9 Factor Analysis Extraction Method: Principal Component Analysis. . The Kaiser-Meyer-Olkin value, measuring sampling adequacy was .783. This should be greater than .6 for a satisfactory factor analysis (Pallant 2001). Bartlett’s test of sphericity was .000. The factor analysis indicated 69% of the variance can be explained by three components: Factor 1 relates to how easily users were able to find relevant information, Factor 2 relates to the presentation of the text and Factor 3 is the language used. Questions requiring a Yes/No response were asked of the users. Table 10 presents the results, both the percentage of the Yes / No responses and the average response is provided. Bold indicates a Yes result higher than the overall average (questions 1-3) or No (questions 4 and 5) lower than the average. Health On
Table 10 Responses to each HIP Table 11 details the responses to the Likert scale statements and questions. The average for each HIP is presented and the average for all HIPs. Bold is again used to indicate better than average results. Health On
experience using this site? The site was easy to use All the information I required to complete the task was on the Website I found the search function useful in helping me locate relevant information The size of the text was easy to read The text was displayed in a way that was easy to read The language used was easy to understand I understood the terminology used on the Website It was easy to find information on the topic that was relevant for me The number of steps required to get to the Table 11 Response to scale questions for individual HIPs Qualitative user responses to the individual Health Information Portals To better understand the users’ responses to the questions in Tables 11 and 12, open ended questions were asked. Qualitative comments relating to searching and finding information were analysed for two questions: “What was the best feature or part of the Website?” and “What was the worst feature or part of the Website?” Table 12 contains some responses where the comments. Best Feature
Worst Feature
Medline Plus
Easy to find information by using the search field It categorises information well such his symptoms, Some time more detailed information is there. You just Mayo Clinic
Website is well structured and information for non- There is no ‘back to search results’ link medical people is presented in quite general terms Multiple ways of searching, list of problems, easy Some diseases can’t be found by looking based on the to understand headings and select them by the letter Topics can be accessed by alphabetical links Limited information available on certain topics Alphabetical list of diseases, treatments and services. Good search engine.
Health on the Net (HON)
Types of searches. Search also checks for similar Some search results don’t match the category it is Search results categorised into groups. Search is Hard to find particular health issue if you are not sure Better Health
There is a search function. There are headings and Too many clicks. Not all relevant information at the Search was chaotic returns irrelevant information Search engine didn't provide information if people entered the wrong spelling. The menu provided information by health topic, All. Don't know where to go other than the search Health Insite
It provided an advanced search so users can easily Hard to read the results, confused between ‘Related HealthInsite topics’ and ‘ Resources found’ Search and A-Z health topics is really easy to use Too many links to click on before searching – the The ‘Conditions and Diseases’ part has useful No content of its own, I can just use Google instead of information to help me understand the diseases this site, unless I want Australian sites. Too crude, not useful information for a health Website Table 12 Qualitative comments from users DISCUSSION
Fox and Rainie (2002) report that 53% of 18-29 year olds had searched the internet for health information. By 2006 this had grown to 77% (Fox 2006). We found 90% of our 18-30 year old users, have searched for health information, 97% using a search engine not a HIP. This is an internet savvy generation and the Internet is an important source of health information for this demographic. Our research demonstrates a link between the features users want, the features available on the portals we explored and the usability evaluation results. The two areas we examined and found impact on the user’s search experience are discussed next. Features and functionality available to improve searching: Of the five HIPs analysed we found Medline Plus had the most extensive range of features and functionality. Providing an ontology or thesaurus to help users refine their search was the feature users wanted most followed by spell checking and differentiated information access. Only Medline Plus included an ontology and thesaurus. All, except HealthInsite had spellchecking however, only two assisted searching through parsing (HON and Medline Plus). All offered some form of differentiated information access. Only one HIP (HealthInsite) offered a personalised search but this was not available from the home page. Users qualitative responses highlight further the importance of features to assist searching. It was evident in the responses users gave when asked about the best and worst features; many mentioned how useful the ‘dictionary’ was or a list of search terms or topics organised alphabetically. Users commented on the lack of spell checking, terminology and what they needed to improve their search. A number of users mentioned that the information was too broad suggesting that because some of the portals did not have well differentiated information that they could not refine their searches adequately enough. Even if a portal only provided an alphabetical search users commented that it was useful. HIP design: The quality of the design includes how easy a HIP is to use and search, the way information is displayed and how frustrated users were in using the HIP. Tables 10 and 11 detailed users’ responses to a range of questions and statements relating to design. Medline Plus was ahead of the overall average on all items. Mayo Clinic was ahead on ten, HON, seven, HealthInsite five and Better Health, two. Users’ responses suggest a strong link between the quality of the HIP design, the features and functionality to assist searching and the overall reaction a user had to the HIP. The more features and functionality the HIP had the better the user experience. Medline Plus was the portal most users preferred, fewer users were frustrated and almost all said they were able to find information on the topic. Users also found it was the easiest to use, the search function was the most useful, the text display was regarded as the best and users said they were able to retrieve relevant information most easily from this HIP. Mayo Clinic was rated second by the users on most items; it also provides a high level of information differentiation and spell checking but does not offer an ontology or thesaurus. By contrast the most poorly rated portal was Better Health, it provided minimum assistance to users for searching in terms of features and functionality. It had some information differentiated but this was limited to the information organised around a small number of topics. Better Health rated most poorly in terms of users’ ability to find information and to find enough information, 65% of users were frustrated at some point. Users indicated they did not like Better Health, they did not have enough information to complete the task, could not find relevant information and there were too many steps needed to retrieve information. The number of negative comments users made can be an indication of dissatisfaction. User frustration was evident when presented with poor search results and lack of information. The number of users who said there was a ‘worst feature’ have been counted and expressed as a percentage of the total number of comments made for that portal. Note not all users provided a best or worst feature. There were very few negative comments from users on Medline Plus. 42 (68%) users commented on the worst feature eight users actually said there were no ‘worst features’. This portal attracted the fewest negative comments on the question of worst feature. A number of users mentioned specifically that the Mayo Clinic had a variety of ways of searching. In particular users mentioned the alphabetical search and how easy it was to search for a particular health issue. 77% (52 users) commented on the worst feature and of these none said there was no worst feature. Many of the users commented negatively on the quality of the search function of the HON Portal and many generally did not find it easy to use. 89% (58 users) made a comment on the worst feature, none said there was no worst feature. Fourteen users commented on the poor quality of the search function. 65% (72) users of the Better Health site commented on the worst feature, of those five said there was no worst feature. Most of the negative comments from users focused on the poor quality of the search function and the overall usability of the portal. For HealthInsite, 90% (110) of users comment on the worst feature. Only one user said there was no worst feature. LIMITATIONS AND CONCLUSION
We acknowledge the fact that, most of the users were under the age of 30 and maybe less concerned about health matters than older users. It would therefore be useful to compare these results with users over 30. However, little previous research has reported on the extent to which people under 30 have searched for health information. It is not surprising that within our group more than 97% had searched for health information using a search engine. As it is likely this demographic will turn to the internet first for health information they are an important group to consider when designing a HIP. More statistical testing is needed to provide a deeper understanding of the impacts of the factors. IMPROVING THE USER SEARCH EXPERIENCE
A key indicator of success is whether a user would use the HIP again. Most users (87%) would use Medline Plus again, 75% would use Mayo Clinic, 65% HON, 70% HealthInsite and 52% Better Health. One factor, the search experience, is likely to have influenced a user’s preparedness to return this includes finding information, finding information easily, the number of steps it takes to retrieve information and finding enough information. Medline Plus was ahead of all the other HIPs on all these items. Better Health however was last on three of the four (users found HON slightly less easy to use than Better Health). From this research it suggests Medline Plus offers users a better search experience, Medline Plus also provides the widest range of features and functionality to assist users in their search. If HIP designers want to ensure users are satisfied with their search experience and will return consideration has to be given to improving the search experience. An analysis of the data helped us to identify features/functionality that should be included in a HIP. Table 13 describes those features/functionality and are in priority order. Feature/functionality Description
Ontology and
Provide easy to access list of medical words and search terms and alternative words. Include a wide range of topics and then sub categories and provide alternative ways to access information. Users may not know the exact term or phase they are searching for therefore an alphabetical list of health topics is a useful feature one users like and want. Many users cannot spell medical terms. Spell checking is essential. HIPs must avoid returning nothing at all if the term is misspelt. Providing a quality parsing facility where the search takes into account phrases rather than one word in a user’s question. This assists users with making queries more specific. Users should be able to ask questions as part of a search. Include pictures particularly to help explain what is presented, provide the ability to search within results, support multiple languages, newsletters and discussion boards can be useful. Personalisation can be used to limit the quantity of information retrieved and ensure a higher degree of relevance of information to individual users. This makes for better searching outcomes and reduces the quantity of retrieved information. Table 13 Recommended features and functionality for a HIP If the designers and sponsors of health information portals want to attract a wider audience, in particular those under 30, and draw health consumers away from search engines such as Google then attention has to be paid to the elements that improve users’ search experience. As argued by Kunst et al (2002) the Internet has “the potential to facilitate but also to jeopardise health care provision”. How HIPs are designed particularly how easily users can search and find relevant information is critical. Including key features and functionality such as an ontology or thesaurus, differentiated information access and spell checking are important for ensuring good search results and a good search experience. Until the design of HIPs incorporate what is needed to improve the search experience health consumers will continue to be dissatisfied with their searching results and the benefits if HIPs will not be fully realised. References
Abdinnour-Helm, S., B. Chapparro and S. Farmer (2005). Using the End-User Computing Satisfaction Instrument to Measure Satisfaction with a Web Site. Decision Sciences 36(2): 341-364. Becker, S. and F. Mottay (2001). A Global Perspective on Web Site Usability. IEEE Software Bevan, N. (2001). International standards for HCI and usability. International Journal of Human -- Bruce, H. (1999). Perceptions of the Internet: What people think when they search the Internet for information. Internet Research: Electronic Networking Applications and Policy. 9(3): 187-199. Bryman, A. and D. Cramer (1992). Quantitative Data Analysis for Social Scientists. London, Burstein, F., J. Fisher, S. McKemmish, R. Manaszewicz and P. Malhotra (2005). User centred quality health information provision: benefits and challenges. 38th HICSS, Hawaii, IEEE., Cox, J. and B. Dale (2002). The quality factors in Web site design and use: an examination. International Journal of Quality and Reliability Management 19(7): 862 -- 888. Dumas, J. and J. Redish (1994). A Practical Guide to Usability Testing. Norwood, Ablex Publishing Fisher, J., A. Craig and J. Bentley (2007). Moving from a Web presence to e-commerce: The importance of a business–web strategy for small business owners. Electronic Markets 17(4). Fisher, J., F. Burstein, K. Lynch, K. Lazarenko and S. McKemmish (2007). Health information websites: is the health consumer being well served? AMCIS. Colorado USA, Colorado State University Fisher, J., J. Bentley, R. Turner and A. Craig (2004). A usability instrument for evaluating websites - navigation elements. OZCHI,, Woollongong, University of Woollongong Fox, S. (2006). Online Health Search 2006. Washington, Pew Internet & American Life Project: 15. Fox, S. and L. Rainie (2002). Vital Decisions: How Internet users decide what information to trust when they or their loved ones are sick. Washington, D.C, Pew Internet & American Life Project: 1 - 43. Fu, L. and G. Salvendy (2002). The contribution of apparent and inherent usability to user's satisfaction in searching and browsing task on the Web. Ergonomics 45(6): 415 -- 424. Hargittai, E. (2003). Serving citizens' needs: Minimising online hurdles to accessing government information. IT& Society 1(3): 27 -- 41. MedlinePlus (2009) About Medline Plusccessed March 2009 HON Survey. (1999). 5th HON Survey on the Evolution of Internet Use for Health Purposes. HON Survey. (2006). HON's Survey of Health and Medical Internet Users. Retrieved November Huntington, P., D. Nicholas and P. Williams (2003). Characterising and profiling health Web user and site types: going beyond "hits". Aslib Proceedings 55(5/6): 277 -- 287. Josefsson, U. (2006). Exploring E-patents' Heterogeneity: Towards Personalized E-health Applications. ECIS. Goteborg, IT University of Goteborg. Kennedy, I. (2003). Patients are experts in their own field. BMJ 326: 1276.
Klein, R. (2007). An empirical examination of patient-physician portal acceptance." European Journal
of Information Systems 16(6): 751 -- 760. Kunst, H., Groot, Diederic, Latthe, Ballayi M., Latthe, Manish, Khan, Khalid S. (2002). Accuracy of information on apparently credible Websites: survey of five common health topics. British Medical Journal 324: 581-582. Luo, W. and M. Najdawi (2004). Trust-building measures: a review of consumer health portals. Communications of the ACM 47(1): 108 - 113. Mayo (2009) About Mayo Clinic Accessed March 2009 Miles, M. B. and M. A. Huberman (1994). Qualitative Data Analysis. London, Sage. Nielsen, J. (1999). "User Interface Directions for the Web." Communications of the ACM 42(1): 65-73.
Nel, D., R. van Niekerk, B. J and T. Davies (1999). Going with the Flow: Web sites and customer
involvement. Internet Research: Electronic Networking Applications and Policy 9(2): 109-116. Pallant, J. (2001). SPSS Survival Manual. Crows Nest, Allen & Unwin Pew/Internet, A. L. P. (2006). Online Health Search 2006. S. Fox, Pew Research Centre: 15. Sillence, E., P. Briggs, L. Fishwick and P. Harris (2004). Trust and mistrust of online health sites. Conference on Human Factors in Computing Systems, Vienna, Austria, ACM Press. Sutcliffe, A. (2002). Assessing the Reliability of Heuristic Evaluation for Web site Attractiveness and Vermaas, K. and L. Wijngaert (2005). Seeking health information on the Internet. 13th European Conference on Information Systems, Regensburg, University of Regensburg. Williams, P., D. Nicholas, P. Huntington and F. McLean (2002). Surfing for health: user evaluation of a health information Website. Part two: the fieldwork. Health Information and Libraries Journal 19(4): 214 -- 225. Zeng, Q., S. Kogan, R. Plovnich, J. Crowell, E. Lacroix and R. Greenes (2004). Positive attitudes and failed queries, an exploration of the conundrums of consumer health information retrieval. International Journal of Medical Informatics 73: 45 -- 55. Zhang, Z., K. Keeling and R. Pavur (2000). Information quality of commercial web site homepages: an explorative analysis. ICIS, Brisbane, Academy for Information Management.

Source: http://is2.lse.ac.uk/asp/aspecis/20090042.pdf