Financial education program evaluation can be defined as a process by which the educator systematically compares learner achievements with the program goals and objectives to determine the success or failure of the educational program.

Learner Achievements

There are three possible scenarios of learner achievements.

The first scenario is simply achieving goals and objectives up to planned levels. The second scenario is exceeding goals and objectives beyond planned levels. If the program results in either of these two achievements, it is considered a success.

The third scenario is not reaching the planned levels of goals and objectives. If the program results in this third scenario, it is considered a failure.

Financial educators conduct evaluation to document program outcomes and to identify the contributing factors to those outcomes. Therefore, the program evaluation must generate two sets of information. First, it documents the results or outcome of the program, which are useful to justify the continuation or the termination of the program. Sometimes, this process is referred to as the impact evaluation or summative evaluation.

Second, evaluation reveals the factors that contribute to program outcomes. These contributing factors can be either positive (strengths) or negative (weaknesses). The identification of a program's strengths and weaknesses is helpful for improving the program by eliminating weaknesses and building on strengths. This information keeps the educator informed about the actions needed to improve the program in achieving desired results. This part of evaluation is sometimes called formative evaluation.

Financial Education Impact

The extent to which people changed or benefited as a result of their participation in educational programs refers to the financial education impact. Depending on the time taken to manifest these changes, program impact can be classified into three broad categories: immediate impact, intermediate impact, and long-term impact.

Immediate impacts include changes in participants' knowledge, attitudes, and aspirations that can be measured soon after completion of the program. Intermediate impacts include changes in participants' financial behavior, such as the adoption of appropriate financial management practices. Normally, intermediate impacts take one to six months to manifest. Long-term impacts are improved economic conditions, including getting out of debt and buying a house. Generally, long-term impacts take more than six months to manifest.

Impact Indicators

An impact indicator can be defined as a reasonable, useful, and meaningful measure of intended participant outcome. For example, impact indicators can include the percentage of participants who improved their financial management knowledge, financial management behavior, economic condition, and so on. These impact indicators are useful to document the outcome of a financial education program, which allows objective criteria for assessing the worth or merit of the program.

Impact indicators are closely associated with the program/activity objectives. Therefore, impact indicators are used to determine whether programs/activities have achieved their objectives.

Predicting Program Impact Using the Logic Model

The rational prediction of impact from program activities and resources is called logic modeling, and it is useful in planning impact evaluation. The logic model links cause and result by using "if" and "then" words. It depicts the programming process in graphical form to help clarify what should be done at each stage in order to reach the next stage.

The following diagram illustrates the application of the logic model in the financial education programming process. It starts with resources, or "inputs," needed for planning the financial education program. The resources include time and money. If the necessary resources are provided, then financial educators will be able to develop and deliver educational programs; these are called program outputs. Then, if planned programs are delivered, planned impacts will take place.

The logic model can be used to predict the type of program that should be implemented. Therefore, the logic model has two important applications. First, it is useful to plan impact evaluation. Second, it is useful to plan programs to achieve desired impact.

Diagram 1 elaborates the application of the logic model in predicting the financial education program impact.

Diagram 1. Logic Model Illustration

The logic model indicates that the impact is the result of output, and output is the result of input. If a financial educator does not spend the necessary resources and deliver educational programs, there will not be any impact. The logic model shows that if a program is to be evaluated, the educator must first be sure that the educational materials and programs have been developed and delivered to the target group of participants as planned for in the program.

Program output and impact are directly related to the resources or input spent on the program. For example, if Program A receives more resources and Program B receives fewer resources, generally Program A will have greater potential to create more output and impact than will Program B. This condition is due to the fact that, with more input, the educator can provide more educational materials and learning opportunities.

Output data include the number of educational materials developed and delivered; the number of educational programs, workshops, and seminars presented; and number of participants reached.

Following this train of thought, if participants are provided more learning opportunities, they may be able to benefit from higher levels of impact. This condition implies that input and output determine the impact of financial education programs. The logic model indicates that the first step of impact evaluation is to ascertain whether sufficient resources have been used to provide educational opportunities for the target participants as planned in the program.

However, there are variables that can affect the impact of a program. For example, participants' motivations for learning and changing can differ, and they are important determinants of the impact of any educational program. In addition, it is important to understand whether the target audience will voluntarily participate, or if their participation is mandated in some way. Knowing this will help the educator evaluate how much time should be spent on increasing motivation before specific behaviors are suggested.

It is also important for the financial educator to remember that output data are not the impact of financial education programs. Since output data such as educational activities and programs lead to impact, output data is included in evaluation reports. However, it is important to identify impact data from output data to assess the effectiveness of financial education programs in terms of bringing about desired socio-economic changes.

Impact Hierarchy

Impact indicators can be organized in a hierarchical manner based on the order that change is observed in the participant as shown in Diagram 2.

Diagram 2. Impact Hierarchy

The Significance of Impact Hierarchy

Hierarchy of impact helps the financial educator understand the program outcome process. For example, when an educational program is presented, the first measurable outcome is the participants' perceived levels of satisfaction with the program. This perceived satisfaction is partly the result of the participants' interaction with the educational program. Depending on their interaction with the program, as well as the quality of the program and how well it met their needs, participants may be satisfied, indifferent, or dissatisfied with the program.

If participants are satisfied with the program, then there is a great potential to create the next level of impact: learning. This includes changing participants' knowledge, attitudes, skills, and aspirations toward the planned direction of the program. If the program is effective, these outcomes take place just before the program ends.

The degree of change in the participants varies with the educational program, type of participants, and the participants' socio-economic environment. The participants' behavior change can take place over a period of time. The behavior-changing period varies from one month to a few months after the program.

If participants adopt appropriate financial management behaviors, there is a potential for achieving the next level of impact. That is the end result of behavior changes. Depending on the program emphasis, participants will be able to achieve their financial goals. For example, if a participant who attends a homebuyer education program learns to save for a home, and actually saves money, then he or she would be able to buy the home.

The lower the level on the impact hierarchy, the easier it is to document, but the weaker the evidence to justify the educational program. On the other hand, the higher the level on the impact hierarchy, the stronger the evidence for the justification of the program, but the more difficult it is to document the results. These two factors indicate that the educator needs to take a middle ground for the impact evaluation between the strength of evidence and the practicality of collecting data. The rule of thumb is to document the highest possible impact with the available resources for evaluation.