Chapter 1 What are the different types of model fit?
There are two types of global model fit in CFA: exact fit and approximate fit. Exact fit is a test of model fit in that it compares a test statistic to a probability distribution to calculate a \(p\)-value, while an approximate fit index can be thought of as an effect size measure that quantifies the degree of misfit in the model. Both are derived from the amount of overall misfit in the model, where misfit is defined as the difference between the model-implied variance-covariance matrix (e.g., the user’s path diagram) and the data-generated variance-covariance matrix (i.e., the observed relationships in the user’s data).
The most commonly used test of exact fit is the \({\chi}^2\) test, although others can be used as wellsee 5. Tests of exact fit are concerned with the presence of misfit (of any kind) anywhere in the variance-covariance matrices. The \({\chi}^2\) test is a test of exact fit because the null hypothesis is that the model-implied variance-covariance matrix exactly matches the data-generated variance-covariance matrix. If the \({\chi}^2\) test is significant, researchers have evidence to suggest that the model does not exactly fit the data. As such, the \({\chi}^2\) test is the strictest way to evaluate model fit and test psychological theory6.
The most used approximate fit indices are the standardized root mean square residual (SRMR), root mean square error of approximation (RMSEA), and the comparative fit index (CFI). There are no probability distributions or p-values associated with these indices, and thus they are not tests of fit but rather a way to evaluate the amount of misfit in the model (e.g., to determine if the misfit is trivial or substantial). In other words, approximate fit indices are different from tests of exact fit because they are concerned with quantifying the degree of misfit in the model. The SRMR is derived from the residual correlation matrix (i.e., the difference between the model-implied variance-covariance matrix and the data-generated variance-covariance matrix) and can be thought of as the average magnitude of the residuals. The RMSEA is derived from the \({\chi}^2\) statistic but differs in that it includes a parsimony correction (i.e., it rewards simpler models with stronger theory that restrict more paths to 0). The CFI is the ratio of the model-reported \({\chi}^2\) and the baseline \({\chi}^2\) and can thus be thought of as the relative improvement in model fit. Lower values of SRMR and RMSEA are indicative of better fit while higher values of the CFI are indicative of better fit7.
The last type of model fit is localized area of fit. In contrast to an overall evaluation of global fit, local fit is an investigation of each cell of the variance covariance matrix to diagnose any areas of strain in the model. This is often done to probe the source of misfit after finding a significant \({\chi}^2\) statistic or a value of an approximate fit index that is indicative of substantial misfit. Still, good global fit can mask local misfit and thus it may be prudent to check regardless of global model fit. Although local fit is discussed at length in introductory textbooks for factor analysis7,8, it is often not presented in journal articles.
Briefly, local fit can be investigated by probing the residual correlation matrix for extreme cases or by consulting modification indices for modifications that would substantially reduce the \({\chi}^2\) statistic8. Even if an extreme value is found, the model should not be changed unless the revisions are supported theoretically. There are several reasons for this. The first is that the modifications suggested by the software are derived statistically and may substantially alter the theory behind the initial model. Secondly, the misfit may be sample-specific and thus may not generalize to other samples. Third, there is no guarantee that a modification will resolve misfit and could instead lead a researcher down a rabbit hole akin to p-hacking. Fourth, it is difficult to know if misfit is due to an issue with the theory about the internal structure or one or more of the items. As such, any modifications should be justified theoretically and qualitatively.