Evaluating Helmet Campaigns
Summary: Campaign evaluations are important to your helmet program and to others who will follow. We are great
believers in field observation rather than telephone sampling.
Introduction
One of the worst problems of helmet promotion campaigns is evaluation. Thorough evaluations are
necessary for those who come next and need to know about the best practices that produce the best results. But those who
have struggled to put together a campaign and see it through are ready to relax, and are already convinced by their
experience that they have done good. So the evaluation phase can be neglected, and the lessons learned are lost. Even
worse, the people in the next state are denied the ability to promote their campaign or law with hard numbers and
demonstrated results. So we have begun to assemble some material so you don't have to reinvent this wheel.
Our approach is firmly oriented toward scientific observation, and toward field observation methods. Anyone who has
worked with bicycle riders and surveys knows the futility of asking a rider how far they have ridden recently, or almost
anything else about their riding, and expecting an accurate answer. Ask a room full of second-graders if they have bike
helmets and if they use them, and a forest of hands will pop up to give you the answer they know you want. Just ask a
teacher. The State of Georgia passed a helmet law, and before they passed it they called people and asked if their kids
were using helmets. Then after the law was passed they called again and -- surprise! -- more people answered that their
kids were obeying the law and using helmets. The only rider surveys we accept are those based on ride logs kept daily by
the hardest-core riders, and those are few and far between.
From these assumptions, our bias is toward the field observation method: actual field counts of who is and who is not
wearing a helmet. If all other things were equal, careful counts before and after your campaign or before and after your
law passed should show whether or not your actions were generating results. All things may not be equal, but at least you
have to try.
Don't miss this one!
For starters, we highly recommend the article on
Measuring Community Bike Helmet Use Among
Children from
Public Health Reports (Vol. 116, No. 2, pp.113-121). It details the ins and outs of using
various measurement techniques. The authors share our bias toward actual observation to determine helmet use and go on to
say that it can be used to evaluate the risk of injury. We don't agree with all of their findings, but If you are doing
helmet counts you want to take the time to read this excellent article first.
The first group of materials below comes from a New York State program that produced a helmet evaluation campaign manual
in 1990. The original manual was showing its age, so we have replaced it with newer material from the current program
contact: The New York State Department of Health, Bureau of Injury Prevention, email:
injury@health.state.ny.us Following that is a summation of the points in a recent
article by Les Becker et al in Injury Prevention, which discusses how to choose observation points and how to conduct the
observations. It is less detailed than the New York materials, but has more information on methods of selecting
observation points. Finally, we have added a chart prepared by the Centers for Disease Control summarizing evaluation
efforts from several localities.
New York State Material
The New York State Department of Health (NYSDOH), Bureau of Injury Prevention (BIP), Division of Chronic Disease Control
and Prevention, supports a statewide bicycle helmet safety project including helmet distributions, education and
awareness for the public and medical professionals, and evaluation research. The project has been funded through various
grant awards from the National Centers for Disease Control and Prevention (CDC), the New York State Governor's Traffic
Safety Committee (GTSC), and the NYS Developmental Disabilities Planning Council (DDPC), as well as from private
donations from insurance companies. It is coordinated with statewide legislation requiring helmets for all children under
14 years of age while bicycling or inline skating. The State has developed an evaluation method for this program, which
is presented below.
Evaluation Research
In addition to process evaluation, including documentation of the number of helmets distributed,
educational presentations conducted, and educational/promotional materials requested, various strategies have been
developed to ascertain the effectiveness of the overall project and determine if there has been a change in bicycle
helmet ownership and usage. A school-based survey was administered each year between 1989 and 1993 through a sample of
Parent Teacher Associations in upstate counties and in a sample of schools in NYC. This survey has documented an increase
in ownership and usage of bicycle helmets, most significantly in younger children.
A protocol for conducting an observational study to determine helmet usage has been developed by Bureau of Injury
Prevention staff and distributed within NYS and nationally. Pre- and post- law observation studies were conducted
statewide to investigate any changes in bicycle helmet usage due to implementation of the bicycle helmet legislation. In
addition, the NYSDOH BIP, under cooperative agreement with the CDC, is conducting a bicycle helmet evaluation project to
determine which strategy, or combination of strategies, is associated with increasing bicycle helmet usage.
Method for Taking Observations
- Observational Study Design
- Form for Recording Observations
- General Guidelines for Taking Observations
- Instructions for Observers
- Slides from Observer Training Course
Observational Study Design
Bicycle helmet observations are conducted in six counties across NYS, three times a
year, Spring, Summer and Fall. Each county has two observers who are trained on the proper methods of conducting
observations, and each county has three observational sites (school, park, playground, etc.). Observers conduct a
combination of stationary and driving observations at the three sites. Most often each observer picks one site each and
they share the third site. Stationary observations are usually 30 minutes in duration and driving observations are
usually 60 minutes although variations by county/site are possible.
An example of how the observations may be conducted is listed below:
Site One Site Two Site Three
----------------------------------------------------------------------
Observer 1. 30 min stationary+ 1. 30 min stationary
A 60 min driving + 60 min driving
2. 30 min stationary +
60 min driving
----------------------------------------------------------------------
Observer 1. 30 min stationary 1. 30 min stationary
B + 60 min driving + 60 min driving
2. 30 min stationary
+ 60 min driving
----------------------------------------------------------------------
Observations are conducted on non-rainy, relatively warm days. For each site, the observers conduct the observations
at the same time of day and observe from the same vantage point during the stationary observation and follow the same route for
the driving observation (determined ahead of time during site visits).
Bicycle Helmet Observation Form
COUNTY:
PLACE OBSERVED: Stationary / Driving
DATE: TIME: Begin End
WEATHER: Overcast / Partly Cloudy / Sunny Approx. Temp.
OBSERVER'S NAME:
Alone or Together with (name)
===================================================================
-------------------------------------------------------------------
BOYS Helmet No Helmet
-------------------------------------------------------------------
|
| 10+
|
|
------------------------------------------------------
|
| Under 10
|
|
------------------------------------------------------
------------------------------------------------------------------
GIRLS
-------------------------------------------------------------------
|
| 10+
|
|
------------------------------------------------------
|
| Under 10
|
|
------------------------------------------------------
===================================================================
Straight line = Correctly worn helmet
Squiggly line = Incorrectly worn helmet
Top of box = Sure of age
Bottom of box = Unsure of age
General Guidelines for Driving Observations
Very Important: For safety concerns please identify a person, other
than yourself, to drive while you are conducting the observations.
This
driver can be different every time and no training is required. The driver will be reimbursed at $8.00 per
hour. If a driving observation is conducted in association with a stationary observation and the driver accompanies the
observer on a stationary observation, the driver's time IS reimbursable. Please be sure the driver fills out the driver
reimbursement form.
Mileage is reimbursable at $0.31 per mile while on the driving route only. Mileage to and from the driving route
is not reimbursable at this time. Please be sure the owner of the automobile fills out the mileage reimbursement
form.
Plan the
driving route ahead of time based on your knowledge of the community. If a driving route has been
pre-established, please try to follow this route as closely as possible. Please submit a map with the driving route
clearly marked for each driving observation, even if the route did not change. Please put the date you conducted the
driving observation on the map.
Plan an
overall loop where the route begins and ends at the same place.
Focus on
residential areas rather than business districts or rural areas. Observe for the
time allotted
based on your instruction packet. If you finish the route before the allotted time is over, restart the route and
continue until you reach the time limit. Mark on the map where you stopped. A substantial overlap or shortfall may
require modifications the next time to lengthen/shorten the route.
Do not observe while on highways/main streets where traffic is traveling greater than 35 mph. If you must travel these
roads to get from one residential area to another, stop the clock and resume once you get to the next area.
If you encounter a traffic jam or construction delay, stop the clock and resume once you are clear.
Avoid double counting children whenever possible. If restarting a route, try to notice if you are observing the same
children. Do not count those you know you saw in the beginning. Only observe as you drive one-way on a dead end street.
Observing while coming back out will count the same children seen on the way in. Stop the clock if it is a very long dead
end and resume when you get back to the main route.
Do not observe within a 2-3 minute (4-5 block) radius of the stationary site. This should eliminate observing children
who are riding to and from the site and who may be counted while you are doing the stationary observation.
If you have any questions, please contact the New York State Department of Health, Bureau of Injury Prevention, email:
injury@health.state.ny.us.
THANK YOU FOR TAKING THE TIME TO REVIEW THESE INSTRUCTIONS. YOUR HELP IS GREATLY APPRECIATED!
Instructions: Bicycle Helmet Observational Study
1. Observations should be completed between the pre-determined
dates. If you have trouble completing them by the end of the time period, please let us know.
2. Conduct stationary and/or driving observations for each site based on the specific instructions provided in your
observation packet. (Note the date and time started and stopped, and whether it was a stationary or driving observation
on the observation forms.)
3. Remember to observe for the specified number of minutes only. Stationary observations are usually 30 minutes. The
length of time for the driving observation may vary by county/site. Please refer to your observation packet.
4. If you do not observe any riders, please call us immediately. You may be asked to do another observation if your
schedule/weather/time permits. You will be paid if you are asked to do this extra observation.
5. Observe on non-rainy, relatively warm days. (Note the weather on the observation forms.)
6. Observe the same type of day (weekday/weekend) and time of day at each site.
7. When no reliability assessments are being conducted, all observations should be completed separately from the other
observer. Please be sure to coordinate schedules so you do not inadvertently show up on the same day. (Check the
appropriate 'alone' or 'together' space on the observation forms.)
8. Make one mark per child in one box noting presence or absence of a helmet, gender (boy/girl), age (10 and older or
under 10).
9. When noting age, if you are sure about the age of the child, mark in the upper half of the box; if you are NOT sure
about age, mark in the bottom half of the box.
10. If the child is wearing a helmet, note whether the helmet is used properly by marking a straight line for a correctly
worn helmet and a squiggly line for an incorrectly worn helmet. (Diagram shows a correctly worn helmet as level on the
head, not tilted back or forward or to the side, with the chin strap fastened snugly, not hanging down way below the
chin.)
11. If the child is not wearing a helmet, the mark will always be a straight line. There is no way for a helmet to be
worn incorrectly (squiggly line) if the rider is not wearing one! 12. Please return final site maps (if applicable),
observation forms and reimbursement forms in the enclosed pre-addressed postage paid envelopes once all observations are
completed. Please keep a photo copy or some other record of the observations in case forms get lost in the mail. Please
limit number of pages to 4-5 per envelope due to postal weight regulations.
If you have any questions, please contact the New York State Department of Health, Bureau of Injury Prevention, email:
injury@health.state.ny.us.
THANK YOU FOR TAKING THE TIME TO REVIEW THESE INSTRUCTIONS. YOUR HELP IS GREATLY APPRECIATED!
Helmet Observation Training
This is based on slides prepared to go with training conducted by Patricia O'Connor, PhD,
consultant to the NY State Department of Health.
Slide 1
Purposes of Research: I
Slide 2
Purposes of Research: II
- Discover
- Demonstrate
- Refute
- Replicate
Slide 3
Essential Characteristics of Research
Slide 4
Research Design
Slide 5
Types of Research
- Surveys
- Interviews
- Observations
Slide 6
Observational Studies are Scientific when they:
- Serve a research purpose
- Are planned deliberately
- Are recorded Systematically
- Are subjected to reliability and validity checks
Slide 7
Now to practice and to establish reliability of observation:
- Food Court area: Coat on/off, gender, younger/older
- Outside cars parking: Seat belt used/not, gender, younger/older
- Mall area: glasses/not, gender, younger/older
- Mall area: carrying bag/not, gender, younger/older
- Mall area: climbing stairs: hold rail/not, gender, younger/older
Slide 8
Training Observation Form (above)
Additional Forms
The New York State material includes forms for driver reimbursement, mileage reimbursement and
observer reimbursement. There is a bicycle helmet application form, and methods of calculating income to determine who
qualifies for low-income helmet assistance.
Maryland's Community-Based Approach
Staffers of the Maryland Department of Health published an article in
Injury Prevention, 1996; 2:283-285 by Les R.
Becker et al, on experience in the State of Maryland gathering numbers before and after their helmet laws were passed.
(The numbers will be available eventually.) Their conclusions:
Selecting Sites for Observation
The article details two methods: asking club cyclists where to find the most
riders with some map study thrown in, and asking community informants where to find the most young bicycle riders. The
second method, using community informants, yielded more cyclists per hour and more observations per dollar spent than
asking club cyclists.
Training Observers and Conducting Observations
For the Maryland studies, high school and college students were
trained as observers a day or two before their sessions. After about 30 minutes of training they were able to identify
and number passing cyclists while recording their age group, gender, race and helmet use on a form. Observations were
made from vehicles and from fixed points between 9 AM and 4 PM on summer days. The sites selected by knowledgeable
community members generated from 50 to 450 percent more observations of riders in the under-16 age group than those
selected using a combination of bicycle club member recommendations and studying maps for such features as parks and
playgrounds. The study authors were unable to say if the increased size of the sample made the results more
representative of the target group.
Other Evaluations: Results of Some Evaluations of Helmet Laws
This chart was produced by the Centers for Disease Control in Atlanta, a part of the U.S. Public Health Service. It
attempts to summarize the data from some evaluations of bicycle helmet laws
Centers for Disease Control
Evaluation of Legislation and Community Programs
to Increase the use of Bicycle Helmets - Selected Locations
Helmet-use rates are for children except for
Victoria, Australia, which included adults.
Helmet use Rates
Pre- Post-
Location Years Program type Program Program
Evaluated
Victoria, March 1983- Community campaign 6 percent 36 percent
Australia March 1990
Comments: Included education, mass media publicity,
support by professional associations and community groups,
involvement of bicycling groups, and $10 government rebate
for helmet purchases.
d.o. March 1990 Helmet legislation 36 percent 73 percent
March 1991 introduced
Comments: Hospitalizations for bicycle-related
head injuries also decreased by 37 per cent,
Howard 1990-1991 Helmet legislation, 4 percent 47 percent
County, community campaign
Maryland
Comments: Activity prompted by bicycling deaths of two
children. Use determined by observation. School-based
survey showed 11 percent and 37 per cent.
Montgomery 1990-1991 Community campaign 8 percent 19 percent
County,
Maryland
Comments: Use determined by observation. Use determined
by school-based survey was 8 per cent and 13 per cent
Baltimore 1990-1991 No specific helmet 19 percent 4 percent
County, promotion activities
Maryland
Comments: Served as control county. Use determined by
observation. Use determined by school-based survey was
7 percent and 11 percent.
Seattle, 1987-1988 Community campaign 5 percent 14 percent
Washington
Comments: Included education of parents by physicians;
advertising in newspapers, on television, and on radio;
school presentations; and discount coupons for helmets.
d.o. 1980-1990 Community campaign 14 percent 33 percent
Comments: Follow-up evaluation of bicycle helmet campaign.
d.o. 1990-1993 Community campaign 33 percent 60 percent
Comments: Follow-up evaluation of bicycle helmet campaign.
Bicycle-related head Injuries decreased approximately 67
per cent among children 5-14 years of age who were members
of a health maintenance organization.
Portland, 1987-1988 No specific helmet 1 percent 4 percent
Oregon promotion activities
Comments: Control community, use determined by observation.
Barrie, 1988-1989 Educational program 0 percent 0 percent
Ontario
Canada
Comments: Use determined by limited number of observations.
d.o. 1980-1989 Educational program 0 percent 22 percent
and helmet subsidy
Comments: Use determined by limited number of observations.
Source:
Injury-Control Recommendations: Bicycle Helmets
Morbidity and Mortality Weekly Report, Vol 44, No. RR-l February 17, 1995
Centers for Disease Control and Prevention (CDC)
U. S. Public Health Service, Department of Health and Human Services
Harborview Injury Prevention Center
Rivara, Thompson, Patterson, Thompson
PREVENTION OF BICYCLE-RELATED
INJURIES: Helmets, Education, and Legislation Annual Review of Public Health, Vol. 19:293-318 (Volume publication
date May 1998) Frederick P. Rivara, Diane C. Thompson, Matthew Q. Patterson, and Robert S. Thompson. Portion of the
abstract: "Helmets have been shown to reduce bicycle-related head injuries for cyclists of all ages involved in all types
of crashes including those with motor vehicles. Helmet use has been promoted using educational campaigns, helmet
subsidies, and legislation. Careful evaluation of these strategies has shown that these interventions increase helmet use
and decrease the incidence of bicycle injuries."
Centers for Disease Control
Evaluation of Legislation and Community Programs
to Increase the use of Bicycle Helmets - Selected Locations
Helmet-use rates are for children except for
Victoria, Australia, which included adults.
Helmet use Rates
Pre- Post-
Location Years Program type Program Program
Evaluated
Victoria, March 1983- Community 6 percent 36 percent
Australia March 1990 campaign
Comments: Included education, mass media publicity,
support by professional associations and community
groups, involvement of bicycling groups, and $10
government rebate for helmet purchases.
Victoria, March 1990 Helmet 36 percent 73 percent
March 1991 legislation
introduced
Comments: Hospitalizations for bicycle-related
head injuries also decreased by 37 per cent,
Howard 1990-1991 Helmet 4 percent 47 percent
County, legislation,
Maryland community campaign
Comments: Activity prompted by bicycling deaths of
two children. Use determined by observation. A
school-based survey showed 11 percent and 37 per
cent.
Montgomery 1990-1991 Community 8 percent 19 percent
County, campaign
Maryland
Comments: Use determined by observation. Use
determined by school-based survey was 8 per cent
and 13 per cent
Baltimore 1990-1991 No specific 19 percent 4 percent
County, helmet promotion
Maryland activities
Comments: Served as control county. Use determined
by observation. Use determined by a school-based
survey was 7 percent and 11 percent.
Seattle, 1987-1988 Community 5 percent 14 percent
Washington campaign
Comments: Included education of parents by
physicians; advertising in newspapers, on
television, and on radio; school presentations;
and discount coupons for helmets.
Seattle, 1980-1990 Community 14 percent 33 percent
Washington campaign
Comments: Follow-up evaluation of bicycle helmet
campaign.
Seattle, 1990-1993 Community 33 percent 60 percent
Washington campaign
Comments: Follow-up evaluation of bicycle helmet
campaign. Bicycle-related head injuries decreased
approximately 67 per cent among children 5-14 years
of age who were members of a health maintenance
organization.
Portland, 1987-1988 No specific 1 percent 4 percent
Oregon helmet
promotion
activities
Comments: Control community, use determined by
observation.
Barrie, 1988-1989 Educational 0 percent 0 percent
Ontario program
Canada
Comments: Use determined by a limited number of
observations.
Barrie, 1980-1989 Educational 0 percent 22 percent
Ontario program and
Canada helmet subsidy
Comments: Use determined by a limited number of
observations.
Source:
Injury-Control Recommendations: Bicycle Helmets
Morbidity and Mortality Weekly Report, Vol 44, No. RR-l February 17, 1995
Centers for Disease Control and Prevention (CDC)
U. S. Public Health Service, Department of Health and Human Services
This chart is part of the source document,
the rest of which
is available from CDC
Swedish Study
Swedish authorities published a paper in 2003 evaluating helmet campaign strategies. The overall
conclusion is that a considerable increase in helmet use by cyclists could be achieved by noncompulsory measures, but the
use levels are not as high as those achieved by legislation.
Dutch Study
A similar study in the Netherlands turned up some different conclusions about helmet use. We have
the abstract and a link to the original study in Dutch.
More Studies
The best list of studies and references on educational campaign evaluations is the page titled
"Bicycle
Injury Interventions Programs to Increase Helmet Use: Education" now on archive.org and somewhat outdated, but
originally from the Harborview Injury Prevention Center site.
In 2016 a study was published titled "Challenges in the accurate surveillance of booster seat and bicycle helmet usage by
children: lessons from the field". The abstract says: "..This paper examined the challenges that confront efforts to
collect surveillance data relevant to child traffic safety, including observation, interview, and focus group methods.
Strategies to address key challenges in order to improve the efficiency and accuracy of surveillance methods were
recommended. The potential for new technology to enhance existing surveillance methods was also explored." The article is
available free on the
International Journal of Environmental Research
and Public Health site.