As a beginning Hong Kong language tester who has yet to achieve a name in the field, I was delighted to be invited to take part in the Dynamic Placement Test (DPT) Standard Setting event. I had never attended a standard setting before, but I understood the concept: it’s a mini-conference where a group of assessment specialists analyse and discuss the items in a test. I was worried that I might not be able to make myself useful but thankfully, Clarity and the test developers had eased my worries.
The sessions were sequenced in a logical way — from a concise introduction to CEFR, then a quick CEFR terms familiarization activity, and finally to several key sessions in which we analysed, discussed, and evaluated selected DPT test items with the test developer and item writers on a clever real-time online platform. The event was well-scheduled with a social event at the start, so I found myself comfortable with and confident of working with experts sitting next to me to improve the validity and reliability of the test.
Among all the things that I have enjoyed at the event, there are two that stand out. The first is diversity. It is amazing to have gathered over 20 experts coming from a variety of disciplines including language testing, language teaching, computational linguistics, and computer-mediated learning. There we all got to contribute collectively to the development and validation of DPT. I learned that each and every single individual always has something unique to bring to the table, be it his/her expertise or his/her knowledge about particular group(s) of test-takers, which offers new insights. For example, even for a novice like me, I was able to raise issues about both intended and unintended washback effects of the test items, which are seldom brought up but are nonetheless crucial in test validation.
The other thing that I appreciate is the open discussions we had in which we justified our votes on the CEFR level of each item, put forth arguments, and, at times, defended and argued for our views. Before this event, I used to think standard setting is all about reaching agreements. Yet, it was not until the end of the three days when I realized that what should be valued is not only convergence but also divergence. Indeed, even in cases where ideas, beliefs, or results do not converge, we should be happy because after all, one of the primary goals of standard setting is to expose problematic practices or items and to fix them using the appropriate method(s).
Overall, it really is a joy to have been a part of the DPT standard setting. It was great to work with and learn from this lovely group of people. I cannot wait to see how Clarity and telc will address the issues we have raised, such as method(s) of standard setting (i.e. Angoff Method vs Bookmark Method) and minor problems with a few response types and test items. I am confident that it will not be long before DPT makes further improvements and achieves a real positive impact on the world.
You can contact Chi Lai Tsang at c.tsang.16@alumni.ucl.ac.uk
Further reading:
A first for Clarity: Dynamic Placement Test Standard Setting and Benchmarking
Do you know you’re being tested?
Putting a test “to the test” — how to make it valid and reliable
