ECUR 809 Program
Evaluation - Module 3
PILOTING A SURVEY
For
this assignment I chose to survey the staff at my school following a staff /
student dodgeball tournament with the following question:
Does participation in fun events with students raise staff morale?
The survey I designed was comprised of 9 “Yes / No”
questions, 3 “qualitative” questions, 12 “choose the number that best describes
your feeling” and 15 “never, occasionally, usually, always, N/A”
questions. The instructions I included
on the cover page asked staff to give me feedback on the following questions:
a)
Were the
survey questions relevant to the main question posed?
b)
Was the survey easy or difficult to fill out or
follow?
c)
Did you feel as though your viewpoint was
accurately represented with the questions posed?
Feedback from
Survey Participants
A.
Directions
Survey participants commented that
my survey hit on all of the main reasons why the SRC hosts the dodgeball event
and why it has become a tradition at the school. Some participants felt that some survey
questions did not apply to the main question, and in fact there were too many
questions that pertained to one’s level of physical fitness. Others felt that the survey could have been
directed to one or two selected students for their feedback.
Some participants commented that there were
more questions related to student morale than teacher morale. Also, specifically questions 7, 14, and 15
had directions applied to them that did not make sense (i.e. “usually,
sometimes, never…”). One participant
commented that one of the qualifiers, “I’m a changed person” was way above VERY
SATISFIED and felt unsure of how much MORE satisfied she needed to be in order
to choose that answer.
B.
Formatting
Some participants commented that
there could have been more space provided for a written answer when the survey
asked for one. They felt somewhat
limited by the space provided. One
participant commented that the survey was easy to follow and that their
viewpoint(s) were accurately represented with the type of questions posed.
C.
Design
Feedback on the design of the
survey included the fact that too many questions were focused on the level of
physical fitness of the participant and that was not the main question posed
for the purpose of the survey. Also,
there was no room to comment on “other” reasons for participation in the
dodgeball tournament other than if you were fit enough to participate.
D.
General
Comments
Survey participants were willing to provide
feedback for my survey and felt that the time it took to complete the survey
did not interfere with their work in any way.
I placed the survey in the participant’s mailboxes at school and
provided a little treat as incentive. I
think the participants understood that this was a light-hearted look at what
they had just participated in and was non-threatening with no long term
consequences. One participant commented
that he “didn’t know exactly what the point of the survey was”. Also, I was horrified to find a spelling
error on the first page which made me want to crawl into a hole! I felt that the spelling error might have
jeopardized the quality of the survey in some way, or at least how the
participants felt about the survey as a whole.
General Survey
Notes
I felt
that my original survey had a clearly defined question and that the survey was
designed with that question in mind. The
survey I originally designed was easy to read, the directions were easy to
follow and it was relatively quick to fill out.
I delivered the survey immediately after the event (dodgeball tournament)
so that the information was timely.
One thing I needed to improve with the survey
directions was that there were a few ambiguous questions that did not have much
relevance to “staff morale” but rather alluded more to the physical fitness of
the participants. In retrospect, some of
the questions I posed may have been too personal in nature and could have been
phrased in a more general fashion. I
think I may have also tried to be too “familiar” with my phraseology in an
effort to inject humor, which was not necessary.
I felt that the questions I used had distinct phraseology and were not
too long. I tried to use a variety of
question types so that the survey had some flow and the questions did not get
tiresome. I did not use a very
flattering format for the original survey.
I think that more time could have been spent in the design of a more
appealing “look” to the entire survey (more polished or professional looking).
The piloting phase of this survey
taught me some valuable lessons in survey construction. I can see where I may have had some biases
show in my original questions and where I should have toned down the “tongue-in-cheek”
phraseology of my questions.
QUESTIONS:
1.
Organization – who to be sampled next time? Layout could have been better.
2.
Questionnaire design – was the design of the
survey logical?
3.
Sampling – is the survey directed at the right
population for accurate data?
4.
Data Processing – how was the data decoded?
5.
Analysis – was there sufficient data collected
to offer input into program evaluation?
I learned that a good survey depends on “how much attention
is given to the problems that may arise.”
This factor determines the QUALITY of the survey, not the size, scope,
or prominence of the survey. (It also
really helps to give out candy as incentive!)
ORIGINAL COVER
LETTER
March
13, 2014
Dear
________________________________,
I was hoping to take a few minutes
of your time with this survey. I am
currently taking a class in Program Evaluation and, as part of this class, I am
required to develop a survey. The goal
of this assignment is to help me understand how to construct good questions and
collect relevant data.
As a member of one of the Staff
Dodgeball Teams, I have developed a survey asking this question:
Does participation in fun events
with students raise staff morale?
If you have a few spare moments,
please fill out the accompanying survey.
I am interested in your feedback on the following points:
a)
Where the survey questions relevant to the main question posed?
b) Was
the survey easy (of difficult) to fill out or follow?
c) Did
you feel as though your viewpoint was accurately represented with the questions
posed?
Please include ANY comments that
you feel would have made this a more effective surveying tool or any question(s)
you feel would be good to add, etc.
Thanks!
ORIGINAL SURVEY
Does participation in fun events with
students raise staff morale?
Yes / No Questions
Did you participate in the Staff /
Student dodgeball tournament during lunch hour? Y/N
Were you in any way coerced into
participating in the dodgeball tournament?
If so, explain how? Y/N
Did you meet with your team members
or communicate in any way about the following details:
a) Date
and time of game(s) Y/N
b) Rules
for the game (s) Y/N
c) Team
Uniform Y/N
d) Strategy Y/N
Did you feel a sense of team
“pride” on the day of the event? Y/N
Did you feel a sense of
“pride” on the days following the
event? Y/N
Did winning “matter”? Y/N
Comment briefly on your fitness
level upon entering this tournament.
Choose the number that best describes your level of satisfaction with
playing on a staff team:
1=I hated it 2=I tolerated it 3=
I liked it 4= It was lots of fun 5=I’m a changed person
How did you feel about the following factors involving staff engagement
in the dodgeball tournament:
1= Not Satisfied 2= Somewhat Satisfied 3= Satisfied 4=Very
Satisfied 5= I’m joining the National
Team
___ pre-game prep
___during the game (team support
and cheering)
___post game revelry
___walking into your class after
the win / loss
___sitting at home after 10:00 p.m.
after the win / loss
Answer the following statements by selecting N( Never),
O( Occasionally), U( Usually),
A(Always), N/A (Not Applicable)
1.
Students like to see teachers in fun events. N O U A N/A
2. Students
will participate in a more assertive N O U A N/A
manner when there is a teacher team in play.
3. Students
are more likely to participate in N O U A N/A
classes in which their teachers are actively
involved in fun activities.
4. Teachers
are encouraged to participate in fun N O U A N/A
events with students.
5. Teachers
have an increased sense of well- being N O U A N/A
and morale when involved in fun events.
6. Teachers’
public image increases in a positive way
N O U A N/A
when they engage in fun activities with
students.
7. Teachers
should not participate in fun events because
N O U
A N/A
it is just awkward to watch.
8. Teachers
have a great sense of fair play. N O U A N/A
9. Students
learn most about team play when they participate
in fun events in which teachers are involved. N O U A N/A
10. Students
should be encouraged to mock teachers as it builds
teacher’s self esteem. N O U A N/A
11. The
school is a better place when teachers and students
participate in fun events. N O U A N/A
12. Team
uniforms raise team morale. N O U A N/A
13. Teachers
are sore losers. N O U A N/A
14. Based
on this event, I will participate in more staff team N O U A
N/A
events.
15. If
I were not physically fit, I would never participate in staff
events (like dodgeball). N O U A N/A
Answer the following questions based on your current level of fitness
upon entering this dodgeball tournament:
1=couch potato
2= walk the dog at least twice a week
3=I have taken Zumba classes
4= qualified for Iron Man (twice)
5= P.E. Staff at MMGCI
___How I would describe my daily
fitness routine
___How I would describe my comfort
level with participation in school sports
___How I would describe my desire
to have fun and be a good sport
___How I would describe my athletic
ability
___How I would describe my
self-esteem (pre- dodgeball experience)
___How I would describe my
self-esteem (post- dodgeball experience)
NEW COVER
LETTER
March
13, 2014
Dear
________________________________,
I was hoping to take a few minutes
of your time with this survey. I am
currently taking a class in Program Evaluation and, as part of this class, I am
required to develop a survey. The goal
of this assignment is to help me understand how to construct good questions and
collect relevant data.
As a member of one of the Staff
Dodgeball Teams, I have developed a survey asking this question:
Does participation in fun events with
students raise staff morale?
If you have a few spare moments,
please fill out the accompanying survey.
I am interested in your feedback on the following points:
a)
Were the survey
questions relevant to the main question posed?
b) Do you have any comments on the
layout and flow of the survey?
c) Did you feel as though your
viewpoint was accurately represented with the questions posed?
Please include ANY comments that
you feel would have made this a more effective surveying tool or any
question(s) you feel would be good to add, etc.
Thank-you for taking the time to
complete this survey. I appreciate your
time and valuable insight!
NEW AND
IMPROVED SURVEY
In order to evaluate the effectiveness of the SRC
programming for students and staff, please read the following survey questions
and choose the BEST response that most accurately reflects your personal
experience. This survey hopes to answer
the question:
Does participation in fun events with students raise staff morale?
Yes / No Questions
Please circle Y for
yes and N for no to the following questions:
1.
Did you participate in the Staff / Student
dodgeball tournament during lunch hour?
Y / N
2.
Were you in any way coerced into participating
in the dodgeball tournament? If so,
explain how? Y / N
3. Did you meet with your team members or
communicate in any way about the following details:
a) Date and time of game(s) Y
/ N
b) Rules for the game (s) Y
/ N
c) Team Uniform
Y / N
d) Strategy
Y / N
4. Did you feel a sense of team “pride” on the
day of the event? Y / N
5. Did you feel a sense of “pride” on the days
following the event? Y / N
6. Did winning “matter”? Y
/ N
Comment briefly on the following
question:
7. What was your level of fitness upon entering this
tournament?
Choose the number
that best describes your level of satisfaction with playing on a staff team:
1=I hated it 2=I tolerated it 3= I liked it 4=
It was lots of fun 5=Terrific Experience
Read the following statements
and choose the number that best describes your level of satisfaction with staff
engagement in the dodgeball tournament:
1= Not
Satisfied 2=
Somewhat Satisfied 3=
Satisfied 4=Very Satisfied
___ pre-game prep
___during the game (team support and cheering)
___post game revelry
___walking into your class after the win / loss
___sitting at home after 10:00 p.m. after the win / loss
|
Answer the following statements by choosing N (Never), O
(Occasionally), U (Usually), A (Always), and N/A (Not Applicable):
|
||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
1. Students like to see
teachers in fun events.
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
2. Students will
participate in a more assertive
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
manner when there is a teacher team in play.
|
||||||||||
|
3. Students are more
likely to participate in classes
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
in which their teachers are actively involved in
|
||||||||||
|
fun activities.
|
||||||||||
|
Teachers are encouraged to participate in fun events
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
with students.
|
||||||||||
|
Teachers have an increased sense of well-being
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
and morale when involved in fun events.
|
||||||||||
|
Teachers' public image increases in a positive way
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
when they engage in fun activities with students.
|
||||||||||
|
Teachers have a great sense of fair play.
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
Students learn most about team play when they
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
participate in fun events in which teachers are
|
||||||||||
|
involved.
|
||||||||||
|
The school is a better place when teachers and
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
students participate in fun events.
|
||||||||||
|
Team uniforms raise team morale.
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
Based on this event, I will participate in more
|
N
|
O
|
U
|
A
|
N/A
|
|||||
|
staff team events.
|
||||||||||
*Any questions that were not
related to staff morale or were about fitness level were eliminated (#7, #10,
#13, #15)
All New:
Please comment briefly about
your experience in the dodgeball tournament held at MMGCI.
Based on your experience,
comment about the likelihood of your participation in this or similar events at
MMGCI.
Would you recommend this
experience to other colleagues who did not participate? Comment why or why not.
