Who should evaluate programs?
You of course! But you are not alone. You will be able to find help from the following sources:
Allison Nichols is the Evaluation Specialist. Her office is 814 Knapp Hall in Morgantown. Her phone number is 304-293-8643 and email is email@example.com. She is able to help you plan an evaluation, design questionnaires, analyze data, and write evaluation reports.
Extension employs specialists in a host of subject areas. These specialists are experts in evaluating programs within their subject areas. They will know how similar programs have been evaluated, where to find sources in the literature, and how to find other groups of people interested in evaluating programs such as yours. Be sure to talk to an appropriate subject specialist before designing your evaluation.
You may want to work on an evaluation with other people who are doing the same or similar program. Contact your unit director or specialist to find out if a team is working on an evaluation project related to your work.You may want to work on an evaluation with other people who are doing the same or similar program. Contact your unit director or specialist to find out if a team is working on an evaluation project related to your work.
Many Extension units throughout the country have produced helpful evaluation materials. We will discuss these resources under the Where question.
Why should you evaluate your programs?
To document impact
You and your stakeholders will want to know if participants have benefited from the program.
To document participation
You and your stakeholders will want to know the extent to which participants have been involved in your program. This information will include how many attended, how often and how long they attended, and the types of people who attended (age, sex, income level, educational level, occupations, family roles, etc.)
To improve programming
You, your team members, and your stakeholders will also want to know which part of your program has worked and which part has not worked. This information will help you make changes to the program.
To identify new programming needs
You will want to know if you need to build upon current programming or provide new programs in your county to meet new or existing needs.
To meet WVU Extension requirements
Faculty must evaluate their programs for several reasons:
- The National Institute for Food and Nutrition (NIFA), which is part of USDA in Washington, DC, requires that we submit information each year to support the impact of our programs. WVU Extension must rely on Extension educators at the county level as well as specialists and program coordinators at the state level, to collect data to document program success. The reports determine future funding.
- Each year in December, Extension faculty members at WVU, like their counterparts in academic departments, submit accomplishment files that are used to determine annual salary increases and are used to evaluate the faculty member for promotion and/or tenure. Faculty members write program narratives that document their efforts as well as outputs and outcomes. Information gathered through program evaluation help to document an individual’s effort and outcome.
- External grant funding supports many of our Extension programs. Grantees must submit regular reports that contain evaluation information. Your help is needed to gather and, at times, analyze program evaluation data that is then included in reports to funding agencies.
What are some good evaluation tools or methods?
Start with an evaluation question
All good evaluations begin with a question rather than a method. It is tempting to decide ahead of time that you will hand out questionnaires or conduct focus groups to conduct an evaluation, but that is usually not the best approach.
Some evaluation questions include:
- Are participant needs being met?
- Have participants learned anything new? (change in knowledge or skill)
- Have participants changed their attitudes? (change in attitudes or motivation)
- Have participants used what they have learned? (change in behavior)
- Has an organization changed its policies? (change in organizational behavior)
- Has a social problem been reduced or eliminated? (environmental or contextual change)
Pick a method or methods that will best help you get the answer to your question
- Program success might be best measured using a questionnaire or interview.
- Knowledge or skill gain might be best measured using a pre- and post-questionnaire or skill assessment.
- Attitudes might also be measured using pre- and post-questionnaires.
- Behavior might be measured by observation or examining employment records or interviewing employers.
- Contextual or environmental change might be measured in a number of ways including interviews, content analysis of newspapers or other records, or observations.
See how others have measured your question
You are probably not the first one to ask a question about your program or type of program. You can get a lot of help by reading what others have published. If you base your evaluation in the literature, you may also have a better chance of getting it published in a journal. If you can find an instrument that someone has used and validated, you should probably use it rather than create a new one.
If a program satisfaction survey is what you want, we have a client evaluation form that you can use. You can find the client evaluation on the WVU Extension Service website on the evaluation page.
Try designing an evaluation instrument
Go ahead, take a stab at designing an evaluation instrument. Don’t forget there are people out there to help you. Always remember that you should pilot all instruments to be sure you are communicating well and are getting the answers you want.
When should you evaluate your programs?
“ASAP” is not necessarily the right answer to this question. Process evaluations should probably start right away so that you can document how you planned and implemented a program. This would mean collecting minute meetings, schedules, drafts of educational materials, numbers of participants, etc. This information will assist you in understanding what you did right or wrong and help others implement your program. You should probably not conduct an impact evaluation, however, that is finding out how participants have benefited, until the program is well established.
To help you know how to plan your program and when and what to evaluate, we suggest you design a logic model. A logic model is a framework for looking at the parts of a program and understanding what you can expect the program outcomes to be. The components of a logic model include: the situation or the context in which your program is operating, the inputs or what you and your colleagues bring to the program, the outputs or the activities and participants, the short-term outcomes or knowledge, skill, and attitude change, the medium outcomes or behavioral changes, and the long-term outcomes or changes to the environment or social context. The components are listed linearly to show how they impact one another. To learn more about logic models call the Evaluation Specialist or visit http://www.uwex.edu/ces/lmcourse/ to take an online logic model course at the University of Wisconsin Extension website.
When do I need Institutional Review Board approval?
The WVU Institutional Review Board has the responsibility of protecting individuals who are the subjects of research. Most evaluation projects do not need IRB approval, however, if the intent is to disseminate the research in a research journal or publication, you will need to have IRB approval. Types of IRP applications include the following: Exemptions are granted for research projects that include adults (18+), do not include sensitive subjects, and can be conducted without knowing the identity of the subject. Expedited reviews are for research projects that include children/youth, sensitive subjects, and require the researcher know the identity of the subjects. There is one more review called “full board review” that is used when the subjects face significant risks while participating. Extension projects rarely go through full board review. Currently, WVU uses the BRAAN system for submitting and approving IRB proposals. (https://ecomp.wvu.edu/Default.aspx) You can find instructions on how to access the system and complete a protocol on the WVU-ES website in the evaluation section. This process can take from four to six weeks.
Research Ethics Test
- All researchers, whose names appear on an IRB application as an investigator, must pass the Research Ethics Test. (https://www.citiprogram.org/default.asp)
- Once you pass the online test, your name will appear on a list of approved investigators. You should take the test as soon as possible so that you will be prepared to participate on an Extension research project.
Where can you go to find help with evaluation?There are many books and publications on evaluation and Dr. Nichols will be able to refer you to the ones that will be helpful for you.
For more information please contact:
Allison Nichols, Ed.D. Evaluation Specialist
West Virginia University Extension Service
814 Knapp Hall P.O. Box 6031
Morgantown, WV 26506-6031