Chapter 7 Qualitative methods for effectiveness evaluation: When numbers are not enough Introduction Methods of collecting qualitative information 7.2.1 Interviews and focus groups 7.2.2 Questionnaires with open-ended questions 7.2.3 Observations 7.2.4 Document analysis Using qualitative methods in evaluation 7.3.1 Identifying implementation and intermediate outcomes 7.3.2 Verifying and complementing quantitative outcome measures 7.3.3 Eliminating threats to internal validity 7.3.4 Identifying unintended outcomes 7.3.5 Developing quantitative measures Selecting a sample for qualitative studies Qualitative data management and analysis Ensuring good quality data Qualitative Methods Chapter 7 7.1 Introduction
The types of questions used in interviews willdepend on the purpose of the data- gathering.
Qualitative methods play an important role in
They could be about any of the following:
safety intervention evaluation. Although in most
knowledge (e.g., What did you learn about in
situations, numbers are necessary to prove
effectiveness, qualitative methods can yield
experience (In what ways, if any, have things
We first describe four methods of gathering
practices (In what way, if any, has the training
qualitative information: 1) interviews and focus
program influenced your safety practices on
groups; 2) questionnaires with open-ended
questions; 3) observations; and 4) document
analysis. We identify and illustrate several
different ways in which these types of data can be
beliefs (What do you think the company’s
used in an effectiveness evaluation. We follow
goals are in providing you this program?)
with some details of how to select study subjects,
feelings (How do you feel about participating
analyze the collected data, and ensure good
A good interviewer is sensitive to the mood andfeelings of the interviewee(s), listens well, and
Methods of collecting qualitative data
encourages them to elaborate on the topicdiscussed. Better interviews will result from
2) Questionnaires with open-ended questions
interviews and has practiced with the interview
questions. There are a number of approaches for
7.2 Methods of collecting qualitative Structured interviews
Structured interviews contain a standardizedmeans of getting information. The same set of
7.2.1 Interviews and focus groups
carefully worded and ordered set of questions
are used with each respondent. This technique
information is through in-depth interviewing.
reduces the level of skill required for the
This involves open-ended questions, where
interviewer to do a good job and curtails the
interviewees can answer questions on their own
influence of any particular interviewer on the
terms and in as much detail as they like. This is
results. Structured interviews are useful where
in contrast to the typical questions found on
several people are conducting the interviews or
employee surveys, that prompt for yes/no,
if the interviewers are inexperienced. On the
multiple choice or very short answers. For
other hand, there is less opportunity to learn
example, a truly open-ended question asks “what
do you think about the new safety program?”.
In contrast, only a limited range of answers is
allowed if you ask, “how useful was the newsafety program?” or “was the new programuseful?”
Chapter 7 Qualitative Methods Semi-structured interview
information, since extreme or false views tend tobe challenged by others in the group. A skilled
A semi-structured approach to interviewing
facilitator can guide the group’s dynamics so that
the participants stay on topic and people who are
standardization and flexibility. Here, an
either shy or have less popular opinions are
interview guide is used, which is basically a
checklist of the issues explored during theinterviews. There is no set order to the topics,
Exert some caution in selecting individuals for a
and specific questions are not necessarily worked
focus group. First, this format is not advisable if
out ahead of time. However, before ending the
sensitive information of either a personal or
interview, the interviewer makes sure all the
organizational nature is sought. People might be
items have been covered, through the natural
reluctant to speak up and could be vulnerable to
course of the conversation. Any topics not yet
repercussions if they do. For similar reasons, and,
covered can then be addressed. As with the
depending on the subject of the interview, you
structured interview, this method ensures that
should probably group people together with
the same type of interview data are gathered
similar positions within the organizational
hierarchy. In particular, consider separating laborand management; and supervisors and those
Unstructured interview
they supervise. In some cases, you might want togroup men and women separately.
The unstructured interview is more like aninformal conversation, where questions are
Guidelines for obtaining good interview data
conversation. Although certain topics arecovered, there are no predetermined questions.
1. Let the subject(s) know at the outset how long
The data varies with each person interviewed.
the interview will last, its purpose and general
This makes the analysis more challenging. As
outline. Explain how confidentiality will be
well, more opportunity exists for an interviewer’s
bias to influence the results. The strength of this
2. Obtain consent (preferably by signing a consent
approach though is that the interviewer can tailor
form) for participating before starting the
the approach and line of questioning to each
3. Start off the interview with non-controversial
questions that require minimal recall. More
Focus group interview
sensitive topics, including questions onknowledge, should be covered once a rapport
A focus group is like an interview with a small
group of people rather than only one person. A
4. Create an atmosphere of having a conversation.
semi-structured approach is most useful. About
You do not want people to feel as if they are
six to ten people can be interviewed together and
the interviews usually last from one-and-one-half
5. Ask clear, truly open-ended, questions.
to two hours. This allows time for participants to
discuss about eight to ten questions.
7. Be attentive. Indicate interest through your
The focus group technique is a highly efficient
8. Tape record the interview in order to have a
way to collect data. You receive the opinions of
detailed record for analysis. Record important
several people at the same time. The social
setting provides a measure of validation for the
Qualitative Methods Chapter 7 7.2.2 Questionnaires with open-ended
Observational data is especially helpful in
questions
evaluating safety programs as an externalevaluator. An understanding of the physical and
social environment will be increased. You will
questionnaires - even with truly open-ended
catch issues that might go unreported during the
questions - to be the most effective way to gather
interviews because the insiders are too close to
qualitative information. It is assumed that many
their situations. As well, people might not speak
people do not want to take the time to write out
freely during interviews in fear of reprisal from
a response. As well, it cannot be sensitive to
co-workers or management. Finally, an on-site
interviewee differences, since everyone gets the
same question. The depth of responses is limited
intervention activities are occurring as described.
because there is no opportunity to follow up onan interviewee’s statement with other questions.
If you are an internal evaluator planning to useobservations, be aware that one’s view of things
is influenced by one’s background and position
questionnaire to measure the quantitative
within the organization. Thus, if observations
objectives in the project, you can then quite
are going to play a large role in an evaluation,
economically expand the breadth of the results
consider bringing in an external, more neutral
by including a few open-ended questions. These
observer. Similarly, you might have to choose
can be useful for gauging participant reactions,
between being an observer or a participant, or
identifying program barriers, bringing out
something in between. The more you participate,
unintended consequences of the intervention,
the more first-hand your knowledge will be. The
disadvantage is that it becomes more difficult to
quantitative measures. Furthermore, the results
maintain “objectivity” and your presence could
of this initial screen can help you decide on the
nature and extent of any follow-up qualitativestudies.
Tailor the length and frequency of observations toyour requirements. This can range from a single
7.2.3 Observations
two-hour site visit to verify programimplementation to a full-time, year-long presence
Another way of collecting qualitative data is to
to fully understand, for example, a change in
actually go on-site and observe what is going on.
safety climate. Field notes are the primary means
Depending on your needs for the evaluation,
of recording observational information. This can
everything can be captured, including the
be supplemented with photographs or videos,
physical environment, social organization,
although such methods are often obtrusive.
program activities, as well as behaviors and
Good field notes require a selectivity that can
interactions of people. Or you can take a more
focus on the important details, yet not severely
narrow focus. The type of observational data
used in qualitative analyses can be different thanthat used in quantitative analyses. In the latter,
7.2.4 Document analysis
specific observations are always being sought:e.g., whether a particular procedure is being
Documents of interest in workplace safety
done correctly or if a particular work-site
intervention evaluations can include material
condition is observed. In contrast, for the
containing policies or procedures related to the
purpose of qualitative analysis, specific types of
intervention, safety records, committee minutes,
observations might not be defined beforehand.
correspondence, memoranda, or reports. Theycan suggest topics to include in interviews or
Chapter 7 Qualitative Methods
questionnaires and offer evidence of intervention
important addition to an evaluation, since it is
implementation, barriers to implementation, or
not usually possible to quantitatively measure
other events in the workplace that could threaten
every pertinent intermediate effect of the
the evaluation’s internal validity.
intervention. It can be difficult to anticipate themall and measure them quantitatively. You
Be aware that documents are never more than a
especially want to find out the extent to which
partial reflection of reality. Some are normative;
the intervention was implemented as planned.
such as procedures documents. They tell what
Document analysis, observations and interviews
should be done, not whether it actually is done.
can be used to check on program activities.
Some documents are descriptive - e.g., minutesof meetings. However, they can reflect one
Example of how qualitative methods can
person’s view (e.g., the minute-taker or chair of
be used to identify intermediate outcomes
the meeting), more than the collective view. Ways to use qualitative methods in
Let us return to an earlier example34 where an
effectiveness evaluation
intervention consisted of a workplace- basedincident investigation team assisting
Interviews, questionnaires, observations and
supervisors in their investigation of incidents.
documents are used alone or in combination
towards several purposes in safety intervention
measures (frequency and severity of injuries)
evaluations. Here, we elaborate on five ways in
which they can contribute to an effectiveness
report and percentage of incidents generatingcorrective action proposals). Interviews
Ways to use qualitative methods in
helped fill in the picture further of how the
effectiveness evaluation
intervention could have led to the observeddecrease in injuries and their severity. The
safety representatives found the incidentinvestigation teams helpful and felt that better
2. Verifying and complementing quantitative
3. Eliminating threats to internal validity
intermediate outcome - has been identified as
and severity of injuries were decreased. 7.3.1 Identifying implementation and 7.3.2 Verifying and complementing intermediate outcomes quantitative outcome measures
Qualitative data can help elucidate the steps
between the intervention and the final outcome,
quantitative measures. Through an approach of
including implementation and intermediate
“triangulation”, two or more different
outcomes. They can identify results not captured
methodological approaches can measure the
in the quantitative measures. This can be an
same thing in order to establish consistency. You
Qualitative Methods Chapter 7
might undertake a broad-based safety initiative
7.3.4 Identifying unintended outcomes
to change the “safety climate” in the workplace. Certainly, you could use a safety climate
Interviews and, possibly, observations are useful
questionnaire, which typically consists of close-
ways to identify unintended outcomes.
ended questionnaire items, to assess a change in
safety climate. Also valuable are open-ended
assessed quantitatively, such as an increase in an
questionnaire items or interviews completed by
untargeted type of injury, others would be better
key personnel regarding observed changes in the
detected through qualitative inquiry.
workplace atmosphere concerning safety. If themethods are consistent in their portrayal of
Interviews are an especially good at gauging the
change in safety climate, then a “cross-
reactions of intervention participants and others
validation” of the methods has been achieved
and you can present your conclusions with more
supervisors, union leaders and managers. Their
reactions and those of others involved with theintervention, are important, since a poor
Sometimes the methods are complementary in
response by an influential individual or group of
that they might measure different aspects of the
individuals at a work-site could have a big effect
on the program. It might explain the lack of
interviews might detect aspects of change missed
success of a promising intervention. Unintended
by a questionnaire containing only close-ended
outcomes can also be more positive. In one
evaluation, for example, interviews with workersand foremen showed that several people believed
7.3.3 Eliminating threats to internal validity
that the recent decrease in the number of laborgrievances could be attributed to the improved
Interviews with key officials can provide
information crucial for addressing potential
7.3.5 Developing quantitative measures Example of how qualitative information
Data collected using qualitative methods in the
helps reduce threats to internal validity
planning stage of the evaluation can provide thebasis for the subsequent development of relevant
In the evaluation example just discussed on the
quantitative measurement instruments. Here are
previous page, interviews and analysis of safety
committee minutes revealed the followinginformation which helped eliminate threats tointernal validity. The workplace physical plan,products, production techniques and activities, aswell as the safety-related policies, purchases andactivities (apart from the creation of the incidentinvestigation committee) had remained constantover the six-year evaluation period - suggestingno evidence of history threats. There was also noevidence for an instrumentation or reportingthreat, since there were no changes in the incidentreporting criteria, nor in safety-related policies,purchases and activities (apart from the creation ofthe committee). Chapter 7 Qualitative Methods Examples of how qualitative studies can help develop quantitative instruments
1) Interviews, observations and document analysis can lead to the development and inclusion of
certain items on questionnaires. For example, say that opinions expressed in interviews had arepeating theme that safety is for sissies. If your intervention is in part designed to change thisattitude, then it would be a good idea to develop a questionnaire that includes questions whichmeasure such safety attitudes.
2) People have used the details of incident records, a qualitative information source, to develop
workplace-specific checklists of work practices or work-site conditions used in quantitativeassessment. They review records to find which unsafe practices and conditions are associated withincidents. Interventions are then developed which encourage the practice of safer alternatives. Aswell, checklists of these safe practices and work-site conditions are developed and used inevaluation. Quantitative measurement consists of making (random) observations and recordingwhether the safe or unsafe version of the practice or work-site condition was observed.
3) Menckel and Carter35 described a new safety initiative in which a group assisted workplace
supervisors in their investigation of incidents within their division. Preliminary interviews anddocument analysis showed that there was often a long delay between incident occurrence and itsformal reporting. As a result, corrective safety measures were correspondingly delayed in theirimplementation. Thus, one of the ways evaluators chose to measure the effect of a new workplaceincident investigation group was by how long it took for incidents to be reported.
7.4 Selecting a sample for qualitative
occupational safety program. Then compare
quotes across groups to see if there are differencesin experiences which might influence the
Once you have decided to use qualitative data
intended goals of the program. Furthermore, you
collection methods as part of the program
might separately ask male and female workers
evaluation, you need to decide from whom, or
about any problems in participating in the
about what, data should be collected. This might
program. Again, comparisons can be made to
include collecting data from specific employee
see if both females and males similarly received
work groups, manager groups, female or male
workers, or different classifications of workers. Additionally, you might want to collect data
We describe eight different purposeful sampling
about a particular event, occurrence, or incident.
Rather than surveying the entire work force, use
Extreme or deviant case sampling
what is called purposeful sampling. Here, oneselects information-rich cases to study in-depth.
Identify unusual or special cases. It is possible
They are purposefully selected so that the
investigator can learn, in detail, about issues of
conditions (good or bad) rather than the many
central importance to the program. For example,
possibilities which fall in the middle. For
example, survey data collected after a safety
workgroups about their experience in a particular
program is over might show one or two people
Qualitative Methods Chapter 7
who have made big changes. A follow-up with
Criterion sampling
an interview could validate the responses as wellas discover what in the program motivated them
to make such big changes. By limiting the focus
predetermined important criterion. Even if all
to extreme cases, this approach to sampling is
employees at the work-site receive the training,
you might interview only those most exposed tothe particular hazard targeted by the training. Heterogeneity sampling/maximum variation
They may reveal major system weaknesses that
sampling
Identify cases with differing characteristics (e.g.,
Politically important case sampling
age, gender, education, job classification) toprovide diverse points of view. Any common
Identify, and select (or not) politically sensitive
patterns emerging from the variant cases can
individuals. You might want to interview a
capture the core experiences and shared aspects
union steward who supports the program, and
thereby can enrich the information obtained. Homogenous sampling Convenience sampling
Identify a small group of information-rich cases
- similar in terms of background, employment
participants for qualitative data collection lies
level, experiences, etc. and explore the issue of
with picking cases that are easiest to obtain and
interest in depth. It might be of interest to
those most likely to participate. This is also the
separate groups of management and then labor
least desirable method. The problem is that in
and compare their opinions about a safety
the end, it is difficult to know exactly who was
interviewed and if their opinions are consistentwith others possibly affected by the program. Typical case sampling
Identify “typical” individuals to describe the
7.5 Qualitative data management and
benefits of the program. Cases are selected with
analysis
the co-operation of key informants such asprogram staff. This information can be used to
help “sell” the program to others reluctant to
qualitative data. The process is described here
in very general terms and appears as a sequenceof steps, which in actual practice can occur
Critical case sampling
simultaneously or may even be repeated. First,all raw information, if not already in a written
Find individuals who could dramatically make a
form, is converted to text. Thus, taped interviews
point about the program. They may be identified
by asking a number of people involved with the
summarized using words, etc. This body of
program. A good bet are the leaders in the group
textual material is reviewed to identify important
who could provide suggestions about how to
features and, possibly, summarize them. A
coding system of keywords, or some other datareduction technique, is developed to facilitate thisprocess. The data, either in summarized form ornot, is then reviewed to identify patterns. These
Chapter 7 Qualitative Methods
patterns are concerned with the following:
However, researchers generally try to reduce
similarities or differences among groups or
their effect on their research by using concepts
events; repeated themes; and relationships
and methods agreed upon by other researchers.
Ways to guard against bias include the following:outlining explicit methods for data collection and
data analyses; adhering to these methods;
having more than one researcher collect data;
regarding the data. Depending on the scope of
having a second, non-biased person summarize
and/or draw conclusions from the data; and
trustworthiness of these generalizations by
letting the data speak for themselves and not
testing them with the results of further data
forcing them into a framework designed by the
collection or comparing them with existing
Appropriate sampling
Success at the data analysis stage requires thatgood data management practices are observed
Someone reading your evaluation wants to be
from the beginning of data collection. Use
sure that the right sample has been selected for
systematic methods for collecting, storing,
the stated purpose. For example, you could not
retrieving and analyzing data. People have
developed various techniques to help highlight,
organize or summarize data. A useful reference
intervention, if either management or employee
representatives are not represented. Thus, the
Huberman 1994]. This reference also reviews the
various software developed to assist in both the
explicit and justified with respect to the study’s
data reduction and pattern recognition stages of
Validation by subjects 7.6 Ensuring good quality data
One of the best ways to determine whether or
Concerns about reliability and validity apply to
not you “got it right” in your study, is to check
qualitative data, just as they do to quantitative
with the subjects you are studying. This involves
confirming the accuracy of the data collected, the
qualitative investigation wants to know that the
reasonableness of the method used to summarize
stated methods have been used consistently
it, and the soundness of the conclusions. Of
throughout the study (reliability concerns). They
course the potential biases of the subjects
also want to know that there are no hidden biases
consulted must be kept in mind when weighing
in the data collection, the data analysis nor the
conclusions drawn (validity concerns). Thorough methods of drawing conclusions
The following contains considerations andsuggestions for ensuring that good quality data is
Avoid drawing conclusions too soon. This can
be caused by researcher bias or pressure to comeup with answers quickly. In contrast, well-
Minimizing evaluator bias
grounded conclusions require time for at leastsome of the following activities: 1) reviewing
The product of a study no doubt bears the
collected data to identify anything which has
personal mark of the people conducting it.
been overlooked; 2) searching for evidence which
Qualitative Methods Chapter 7
contradicts preliminary conclusions, either by
7.7 Summary
reviewing data already collected or by gatheringnew data; 3) confirming important data or
conclusions through “triangulation”, i.e., finding
gathering qualitative information: interviews;
agreement when using a different data source,
questionnaires with open-ended questions;
methodology or researcher; and 4) exploring
observations; and document analysis.
alternative explanations for patterns observed in
Qualitative data can be used in several ways to
complement quantitative methods: identifyingimplementation and intermediate outcomes;
Conduct a pilot study
verifying and complementing quantitativeoutcomes; eliminating threats to internal validity;
Conducting a pilot study or trial run with your
proposed research methods is often of great
developing quantitative measures. In contrast to
value. Feedback from those involved in the pilot
quantitative methodology, qualitative methods
study can be used to refine a sampling strategy,
interview guide, other data collection procedures,
sampling strategies. We briefly discussed
and even procedures for data management.
methods of analysis and methods to ensure goodquality data. Key points from Chapter 7
• Use interviews and focus groups, questionnaires with open-ended questions, observations, and
document analysis to enrich your evaluation.
• Use qualitative methods for one or more purposes:
identify implementation and intermediate outcomes
verify and complement quantitative measures
• Use one of several purposeful sampling strategies.
• Collect and analyze data in ways which enhance their reliability and validity.
(form according to EEC Directive 93/112/EC)NAME : LITHIUM, THIONYL CHLORIDE (Li-SOCl2)1 - IDENTIFICATION (of the product and the supplier)LS 14250, LS 14250 C, LS 14500, LS 14500 C, LS 17500, LS 26500, LS 26500 C, LS 33600, LS 33600 C, LSH 26180, LSH 14, LSH 20Solution of lithium tetrachloroaluminate** Lithium tetrachloroaluminate is a combination of lithium chloride (LiCl) and aluminum chlo
Eur J Vasc Endovasc Surg xx, 1e7 (xxxx)doi:10.1016/j.ejvs.2006.04.033, available online onChronic Venous Disease Treated by Ultrasound GuidedAim. To report the outcome of a series of patients with chronic venous disease due to incompetence of saphenous trunksmanaged by ultrasound guided foam sclerotherapy (UFS). Patients and methods. A group of 808 patients comprise this series. CEAP clinical