Testlets are groups of locally dependent items, such as the questions about the same topic. Testlets are obvious in theory, but obscure in practice. If the local dependency of all the items is about the same across testlets, then the testlet effect disappears. So, "testlets" are not the problem, the problem is unevenness of local dependency across the testlets. Table 23.99 - inter-item dependency - tends to be too local. It rarely gives us a good picture of dependency within testlets or across testlets.
Suggestion: in each item label, put a testlet code. Then from the individual-item analysis, output Winsteps Table 28 - item subtotals by testlet. Testlets with average item mean-squares much below 1.0 have noticeably greater dependency (of some type) than the other testlets. These testlets would be the obvious candidates to become PCM items. However, there are usually many reasons for local dependency across items within a dataset, so we would want to see the Table 28 findings confirmed by Table 23.1, etc. Each more dependent testlet should also be the dominant feature in a Contrast.
A simple testlet example:
1. Analyze exam1.txt - note down the person measure S.D.
2. choose two items with high local dependency - see Table 23.99
3. create one testlet item (0-1-2) from the two items (0-1)
4. Analyze exam1.txt without the two items and with the testlet item.
The person raw scores must be the same as in 1.
The person measure S.D. must be slightly less than in 1.
The change in person measure S.D. shows the effect of the local dependency on the person measures.
Testlet remediation:
1 combine the testlet's locally dependent items into one polytomous "super" item <- supported by Winsteps
2. the random-effects testlet model i <- not supported by Winsteps
How are you conceptualizing the testlets?
A. If the testlets are conceptualized as groups of separate items, then please put a testlet-code in column 1 of the item label. You can then obtain item-level summary statistics in Winsteps using ISUBTOTAL=1 and Table 27. You can also obtain person-by-testlet measures using DPF=1 and Table 31, and person-group-by-testlet using DPF=1, DIF=person group code column and Table 33.
B. If two or more items are highly dependent (so that their residuals are highly correlated in Table 23.99) , then it may be better combine them into one "testlet" item. Testlet Response Theory is technically complex and requires special software, but we can approximate it by combining the dependent items into partial-credit items.
From a Winsteps analysis of the separate items, Output files menu, RFILE= to Excel (with headings). In Excel, sum the separate items into new combined-item columns. Save the Excel file In Winsteps, Excel/RSSST menu, Excel option. Click on the saved Excel file. Select the combined items from the list of Excel columns. Format a Winsteps control and data file. (You can also select the original uncombined items, but will need to weight them zero with IWEIGHT= in the new analysis.) Analyze the new Winsteps control and data file with ISGROUPS=0 (partial-credit model)
Cross-plot the person measures from the original analysis and the partial credit analysis. Are there any differences? Which analysis produces more meaningful results?
How to combine items into super-items or testlets |
||
1. |
If your data are already in columns in an Excel worksheet, jump to 8. |
|
2. |
Place your data in a Winsteps control and data file. Our example is the Knox Cube Test data, exam1.txt |
|
3. |
Launch Winsteps with your control and data file. Analyze your data Verify that all is correct |
|
4. |
For the scored responses, Output Files menu RFILE= or for the original responses, Output Files menu IPMATRIX= |
|
5. |
Output File dialog box: Excel Permanent file OK |
|
6. |
File name dialog box: Enter name for testlet Excel file
Our example: Exam1-testlet.xls
Save |
|
7. |
Excel displays with the data.
Check that the item labels are in Row 1. |
|
8. |
Insert columns for the testlets, super-items or combined items
In our example, we will have a super-item item for each length of tapping pattern: Two, Three, ..., Seven |
|
9. |
Sum the responses to the relevant items into the super-item columns |
|
10. |
Save or “Save As” the Excel worksheet |
|
11. |
Winsteps menu bar Excel/RSSST Selection box: Excel |
|
12. |
Select the saved Excel worksheet |
|
13. |
The Excel column labels display with the items and super-items |
|
14. |
Copy-and-paste the super-item column labels under “Item Response Variables” Other items can be copied, especially if you plan to weight them zero with IWEIGHT= Also copy the person column labels under “Person Label Variables” |
|
15. |
Click on “Construct Winsteps File” |
|
16. |
Save the Winsteps control and data file with a suitable name. |
|
17. |
The new Winsteps control and data file displays. It contains the superitems. CODES= contains the scores in the superitems.
|
|
18. |
Remove the ; before GROUPS = 0 because we expect each super-item to have its own structure (partial credit model). |
|
19. |
Save the Winsteps control and data file |
|
20. |
Click on “Launch Winsteps” |
|
21. |
The analysis of the super-items is performed |
|
22. |
Appendix: Impact of the super-items on measurement.
Output PFILE=pf.txt from the super-item analysis, then in the standard analysis, Scatterplot the person measures against pf.txt
In this example, the logit spread of the person measures is less for the super-items than for the standard items. This indicates that there is local dependency among the standard items that comprise each super-item. |
|
23. |
In the plot in 22. all the measures have shifted. The super-items are automatically correctly weighted. However, the mean difficulty of the items changes when separate items are collapsed into super-items. In this situation, either (i) set the mean ability of the persons to be the same in both analyses, UPMEAN=0 ; anchor the mean person ability at 0 logits or (ii) anchor an unchanged item at the same logit difficulty in both analyses, 1 0 ; anchor item 1 at 0 logits * |
|
