Generalizability Theory

For G-Theory computations using Facets, see Table 5

 

The aims of a Facets analysis and G-Theory are different. G-Theory operationalizes a "raters are rating machines" philosophy, so we want the "machines" to operate in exactly the same way. Facets operationalizes a "raters are independent experts" philosophy, so we want the "experts" to concur on their overall verdicts, but we expect them to disagree on the details.

 

In G-Theory, the aim is to reduce the unwanted variance so that the observed scores become the "truth". Of course, there is always measurement error, and some raters are always more lenient or more severe than other raters, Consequently the observed scores are always influenced by the "luck of the draw" (Shavelson & Webb, 1991). In Facets, the aim is to adjust for unwanted variance so that the estimated measures (after adjustment) become the "truth".

 

Usable judging plans (experimental designs) for Facets and G-Theory differ. G-Theory accepts nesting, such as raters within test center. Facets demands partial crossing, such as by having raters at all test centers rate the same videotaped performances. G-Theory demands judging designs for which the variance terms are estimable. Facets accepts deliberate and accidental missing data.

 

Shavelson, R. J., & Webb, N. M. (1991). Generalizability Theory: A Primer. Thousand Oaks, CA: Sage.