Evaluation Toolkit® Now Available to Personal Finance Educators
Date: May 2007
An innovative tool to evaluate the effectiveness of financial education programs now is accessible to individuals and organizations involved in financial literacy activities. The new resource, called the NEFE Financial Education Evaluation Toolkit® , was developed by university researchers through a $137,000 grant from the National Endowment for Financial Education® (NEFE®).
The toolkit consists of two online components: a database and a facilitator’s manual. The database allows users to quickly and easily customize an evaluation tool for their financial education program. The manual guides educators in designing the measurement tool, and collecting and analyzing the resulting data, which then can be used to improve their programs’ effectiveness, provide accountability to stakeholders, and support funding requests.
Within the free toolkit’s database, educators can customize an evaluation instrument by selecting various options. For example, they can choose a post-program evaluation only; pre- and post-evaluations; stages-to-change evaluation (i.e., document the process of behavior change over time); or train-the-trainer evaluation. In addition, educators can select specific aspects they want to evaluate, such as knowledge growth, increases in skills or confidence, actions taken, and changes in financial behavior and/or financial position.
Users also can select, add, or edit knowledge questions, behavior statements, open-ended questions to elicit qualitative information, and demographic questions. In addition, the database spans a wide range of financial topics, which allows the evaluation tool to be tailored to subjects covered by many different education programs. Educators will find questions and statements related to decision-making, cash-flow management, savings and investments, credit and debt management, homeownership, and retirement, among others. These options are available to help the instructors adapt the tool to their specific evaluation needs.
“A financial education program without evaluation is like an explorer without a compass,” says Ted Beck, NEFE president and CEO. “Evaluation should be integrated into a program’s design from the beginning, yet research has found that a lack of understanding about how to measure a program’s impact, combined with limited time and resources, have prevented educators from giving evaluation the attention it needs. The result is that we often do not know if education efforts are, in fact, improving consumers’ financial situation.”
The research Beck refers to was conducted in 2004 under NEFE’s grant to the University of Georgia. Koralalage S.U. Jayaratne, Ph.D., who at the time was the evaluation specialist in the Cooperative Extension Family and Consumer Sciences department at the University of Georgia and is currently the state leader for program evaluation and an assistant professor at North Carolina State University, led the research team.
Other researchers included Angela Lyons, Ph.D., assistant professor, at the University of Illinois at Urbana- Champaign; Lance Palmer, Ph.D., CPA, assistant professor, at the University of Georgia; Jimmy Hansen, Cooperative Extension Service, at the University of Georgia; Pamela Turner, Ph.D., assistant professor, at the University of Georgia; Joan Koonce, Ph.D., associate professor, at the University of Georgia; Erik Scherpf, doctoral student, at the University of Illinois at Urbana- Champaign; and C.W. Copeland, doctoral student, at the University of Georgia. The study involved eight focus group sessions and a national online survey of financial professionals and educators.
Study participants said that the most common barriers and challenges associated with program evaluation included not having enough time; limited staff and financial resources; difficulty in motivating participants to complete an evaluation; lack of existing evaluation materials and resources; and lack of attention paid to evaluation. Participants also said that an ideal evaluation tool should be flexible, adaptable, quick and easy to use, and provide guidance on how to design an evaluation instrument, gather and study data, and effectively present the results. Using the study’s findings as a guide, the research team began work on an evaluation tool that would address each of the barriers and challenges identified, and incorporate the qualities of an ideal instrument. During 2005 and 2006, the team developed an evaluation database and manual, and tested both with financial educators in Georgia.
After modification, the NEFE Financial Education Evaluation Toolkit was launched on the NEFE Web site early in 2007. The program’s developers will introduce it during the next several months at workshops and professional conferences across the nation.
“We believe this is the most significant evaluation tool yet to be made available to the financial literacy community,” Beck says. “The toolkit’s flexible design allows it to be customized for different organizations, audiences, and topics; and it can be used on a national, regional, or local level. This is a huge step toward a more standardized, consistent approach to program evaluation, which ultimately will improve the outcome of financial education efforts.”
Click here to access the Financial Education Evaluation Toolkit®.
Evaluation Toolkit® Now Available to Personal Finance Educators
Date: May 2007
An innovative tool to evaluate the effectiveness of financial education programs now is accessible to individuals and organizations involved in financial literacy activities. The new resource, called the NEFE Financial Education Evaluation Toolkit® , was developed by university researchers through a $137,000 grant from the National Endowment for Financial Education® (NEFE®).
The toolkit consists of two online components: a database and a facilitator’s manual. The database allows users to quickly and easily customize an evaluation tool for their financial education program. The manual guides educators in designing the measurement tool, and collecting and analyzing the resulting data, which then can be used to improve their programs’ effectiveness, provide accountability to stakeholders, and support funding requests.
Within the free toolkit’s database, educators can customize an evaluation instrument by selecting various options. For example, they can choose a post-program evaluation only; pre- and post-evaluations; stages-to-change evaluation (i.e., document the process of behavior change over time); or train-the-trainer evaluation. In addition, educators can select specific aspects they want to evaluate, such as knowledge growth, increases in skills or confidence, actions taken, and changes in financial behavior and/or financial position.
Users also can select, add, or edit knowledge questions, behavior statements, open-ended questions to elicit qualitative information, and demographic questions. In addition, the database spans a wide range of financial topics, which allows the evaluation tool to be tailored to subjects covered by many different education programs. Educators will find questions and statements related to decision-making, cash-flow management, savings and investments, credit and debt management, homeownership, and retirement, among others. These options are available to help the instructors adapt the tool to their specific evaluation needs.
“A financial education program without evaluation is like an explorer without a compass,” says Ted Beck, NEFE president and CEO. “Evaluation should be integrated into a program’s design from the beginning, yet research has found that a lack of understanding about how to measure a program’s impact, combined with limited time and resources, have prevented educators from giving evaluation the attention it needs. The result is that we often do not know if education efforts are, in fact, improving consumers’ financial situation.”
The research Beck refers to was conducted in 2004 under NEFE’s grant to the University of Georgia. Koralalage S.U. Jayaratne, Ph.D., who at the time was the evaluation specialist in the Cooperative Extension Family and Consumer Sciences department at the University of Georgia and is currently the state leader for program evaluation and an assistant professor at North Carolina State University, led the research team.
Other researchers included Angela Lyons, Ph.D., assistant professor, at the University of Illinois at Urbana- Champaign; Lance Palmer, Ph.D., CPA, assistant professor, at the University of Georgia; Jimmy Hansen, Cooperative Extension Service, at the University of Georgia; Pamela Turner, Ph.D., assistant professor, at the University of Georgia; Joan Koonce, Ph.D., associate professor, at the University of Georgia; Erik Scherpf, doctoral student, at the University of Illinois at Urbana- Champaign; and C.W. Copeland, doctoral student, at the University of Georgia. The study involved eight focus group sessions and a national online survey of financial professionals and educators.
Study participants said that the most common barriers and challenges associated with program evaluation included not having enough time; limited staff and financial resources; difficulty in motivating participants to complete an evaluation; lack of existing evaluation materials and resources; and lack of attention paid to evaluation. Participants also said that an ideal evaluation tool should be flexible, adaptable, quick and easy to use, and provide guidance on how to design an evaluation instrument, gather and study data, and effectively present the results. Using the study’s findings as a guide, the research team began work on an evaluation tool that would address each of the barriers and challenges identified, and incorporate the qualities of an ideal instrument. During 2005 and 2006, the team developed an evaluation database and manual, and tested both with financial educators in Georgia.
After modification, the NEFE Financial Education Evaluation Toolkit was launched on the NEFE Web site early in 2007. The program’s developers will introduce it during the next several months at workshops and professional conferences across the nation.
“We believe this is the most significant evaluation tool yet to be made available to the financial literacy community,” Beck says. “The toolkit’s flexible design allows it to be customized for different organizations, audiences, and topics; and it can be used on a national, regional, or local level. This is a huge step toward a more standardized, consistent approach to program evaluation, which ultimately will improve the outcome of financial education efforts.”
Click here to access the Financial Education Evaluation Toolkit®.