Share:

Scale Validation

Cultural Sensitivity Project

For the past two years, NEFE has led a collaboration on a scale validation project with the intent of addressing a major gap in the literature: the lack of rigorous testing and validation of one of the most utilized survey assessments in the financial services industry. It is important to understand which scales work best and for whom, especially when survey results are used to make decisions about individuals and communities. Some studies in personal finance use confirmatory factor analysis to assess the measure fit of scales, but little work has been done to assess measure invariance, meaning there are little to no differences in interpretation across sub-groups. This makes it difficult to interpret study findings or to discern what a study is contributing to the body of knowledge. Invariance testing is crucial to studying how different scales apply across demographics and other consumer subsets. Researchers have faced barriers in making this happen due to lack of training, funding or other reasons. This motivated NEFE to explore solutions that will circumvent this obstacle and help move the field forward.

The two-year collaborative research project with Mission Measurement started with approximately 90 scales, filtered by the following criteria:

  • Created using samples of adults aged 18 years and older.
  • Produced after the year 2000.
  • Has at least two measurable items within the scale.

NEFE, with assistance from a panel of seven financial literacy experts, narrowed down the 90 scales to 11 final scales, based on the presence of some validity testing, rigorous development, or those that are pervasive and commonly used. The project’s data originated from the 11 chosen scales and demographic questions, with the sample attempting to capture the diversity of the U.S. adult population to ensure sufficient counts of 12 demographic sub-groups to support the planned analyses.

Two methods were used in the assessment of these existing scales. Confirmatory factor analysis examined how well the hypothesized factor structure (i.e. the set of items for each scale or subscale and the factor loadings for each item) fits the new data collected. Scale reliability (assessed using Cronbach’s Alpha) was utilized to measure whether the scale could be expected to provide consistent measures over time. The results showed that, of the 11 scales, five exceeded all fit statistics and were labeled “favorable”; three were labeled “reasonable” because they exceeded most fit statistics; and three were labeled “problematic” because they failed to exceed the thresholds for several fit statistics. Following those assessments, invariance was calculated through multigroup confirmatory factor analysis, which helped indicate whether the questions are interpreted similarly by sub-groups of the sample. Scales that perform well in the confirmatory factor analysis and reliability analyses may not provide valid and reliable measures of the construct for all demographic subgroups, especially those traditionally marginalized. Invariance testing helps us understand if a given measure is being interpreted in the same way across different cultural backgrounds and other types of demographics, which allows researchers to make more accurate comparisons across groups.

Researchers have a responsibility to ensure reliability with the construct being tested to study the populations and sub-groups in their study, and it is especially important when investigating traditionally marginalized groups who were not likely to have been represented in the original scale development process. This study is a great step for assessment of existing scales, but it does require additional research so that more groups can be included to support scale development and selection. The next steps could include further demographic characteristics for these scales and an evaluation of further scales not included in the study with confirmatory factor analysis and measurement invariance to specify their issues and identify solutions. Additionally, NEFE is in the process of collaborating with the panel to produce a consensus paper sharing recommendations emerging from this project and a set of methodological studies to address specific topics raised during the project.

Back to Top