Planning a Program Evaluation: Worksheet

 

 

Steps in Program Evaluation

Manage the evaluation   § Human Subjects Protection § Timeline § Responsibilities § Budget		1. Engage stakeholders
		2. Focus
		§ Describe program-logic model § Define purpose § Determine use/users § Determine key questions § Select indicators § Determine design	3. Collect data
			§ Identify sources § Select method(s) § Pilot test § Set schedule § Determine sample	4. Analyze & interpret
				§ Process data § Analyze § Interpret data § What did you learn? § What are the limitations?	5. Use
					§ Share findings and lessons learned § Use in decision making § Determine next steps
Standards of evaluation: · Utility     · Feasibility     · Propriety     ·Accuracy

 

          Engage Stakeholders

 

Who should be involved?

School Administration;  Teachers; Students

How might they be engaged?

Focus Groups; Information Sessions; Surveys; e-mail communication

 

          Focus the Evaluation

 

What are you going to evaluate?  Describe program (logic model).

The evaluation will focus on input factors such as who has invested in the Outdoor Education program in terms of money, equipment, research, and training.  The outputs include group work, presentations, trips, Leadership classes, and Projects students are engaged in throughout the term.  Along with these student outputs, there will be some analysis of Parent / Guardian investment and what their desired outcomes might be for their student. The outcomes of the Outdoor Education Program will be examined based on what results or changes occurred for the individuals, groups, or the community as a result of being involved in the Outdoor Education Program.  More specifically, the social, economic, civic, and environmental outcomes will be examined.  External factors such as the weather play a critical role in the delivery of this program and can adversely affect the input, output, and outcomes.  As well, the desire to continue offering this program is an external factor at play.  The primary assumption for the Outdoor Education Program is that students will learn in the best environment possible.

What is the purpose of the evaluation?

To determine the effectiveness of an Outdoor Education Program as a vehicle to teach in the best environment possible in order to develop lifelong learning in which the curriculum is taught with a focus on social justice and an  understanding of the interconnectedness of humans to their environment while developing relationships of all program participants.

Who will use the evaluation? How will they use it?

Who/users	How will they use the information?
School Administrators	Increase enrollment; effective programming
Teachers	Community Partnerships; benefits to students
Students	Realize potential; pride in accomplishments

What questions will the evaluation seek to answer?

1. Is learning happening in the best environment possible?

2. Is the curriculum being delivered in 5 key areas?

3. What lifelong learning opportunities are being fostered?

What information do you need to answer the questions?

What I wish to know	Indicators – How will I know it?
Best Environment?	5 Outdoor Trips
Curricular Connections?	Examine courses based on outcomes
Lifelong Learning?	Community connections

When is the evaluation needed?

2014 – 15 school year

What evaluation design will you use?

Logic Model

          Collect the information

 

What sources of information will you use?

Existing information:	Course Registration Handbook for MMGCI; website
People:	Erin McVittie and Mitch Lowe (program teachers); students
Pictorial records and observations:	Sask Ed Curriculum; observation; surveys; focus group

 

What data collection method(s) will you use?

ü	Survey	ü	Document review
ü	Interview	ü	Testimonials
ü	Observation	ü	Expert panel
	Group techniques		Simulated problems or situations
	Case study		Journal, log, diary
	Tests		Unobtrusive measures
	Photos, videos		Other (list)

 

Instrumentation: What is needed to record the information?

Laptop

 

When will you collect data for each method you’ve chosen?

Method	Before program	During program	Immediately after	Later
Interview		ü
Observation		ü
Document Review		ü

 

 

 

Will a sample be used? 

 

No	ü
Yes		If yes, describe the procedure you will use.

Pilot testing: when, where, how? 

          Analyze and Interpret

 

How will the data be analyzed?

Data analysis methods:	Data analysis will be based on the learning outcomes of the program
Who responsible:	Evaluator

 

How will the information be interpreted—by whom?

The information will be shared with program teachers and school administrators by the program evaluator.

 

What did you learn? What are the limitations?

The limitations include time, location students are learning, and comfort level of program participants to share information.

 

 

               Use the Information

 

How will the evaluation be communicated and shared?

To whom	When/where/how to present
School Administration	Teacher Learning Group time; informal discussion; formal report
Program Teachers	Before or after school; informal discussion; formal report
Students	Focus group

Next steps?

 

          Manage the evaluation                 Standards

ü	Human subject’s protection	ü	Utility
ü	Management chart	ü	Feasibility
ü	Timeline	ü	Propriety
ü	Responsibilities	ü	Accuracy
ü	Budget