This manifesto is the result of the symposium “Language Testing in Austria: Towards a Research Agenda” held in Klagenfurt on October 19th to 20th 2018. The working groups discussed desiderata for future research with reference to an adapted version of Weir’s (2005) validity framework. This manifesto is an attempt to synthesise and summarise the suggestions made by the participants. The aim of the manifesto is to be a stimulant for future research. It does not aim to be all‑encompassing or prescriptive in any sense.
This list provides a structured overview of the suggestions made.
Research into genre-specific features of the text types (Textsorten) required in writing. Identifying such features is an important contribution to establishing contextual validity, because knowledge of these features is a prerequisite for defining criteria for correctness. This should be given high priority in L1 German and also in the modern foreign languages.
Research into difficulty-generating features of items and tasks
Theory based validity
Cognitive strategy use (think‑aloud, eye‑tracking, mouse‑click protocols)
More detailed descriptions of the competence models underlying individual subtests of Matura (particularly Language in Use) and educational standards tests.
Developing competence level descriptions for the classical languages in analogy to CEFR
Developing competence level descriptions for L1 in analogy to CEFR
Scoring validity & Standardsetting
Research into difficulty-generating features of items
Research into causes of discrepancies among expert judgements of item focus and difficulty
Research into causes of discrepancies between estimated and empirical item difficulty
Rating validity & Benchmarking for writing and speaking
Rating scale validation
Attempts to synthesize rating scales with aquisitional sequences identified in SLA
Research into rater cognition
Research into difficulty-generating features of tasks
Research into rater reliability by multiple rating of a sample of matura performances
Research into washback effects on teaching, learning, classroom testing, textbooks
Use research designs which include a wide range of test stakeholders (learner, teachers, parents, government and official bodies, …).
Research into language independent competences (in the receptive and the productive skills), e.g. correlation of English Writing with German Writing within BIST and within Matura.
Research into convergent and discriminant validity of subtests within Educational Standards and of subtests within Matura (particularly Language in Use with aspects of Reading)
Research into the predictive validity of BIST for Matura and Matura for success in tertiary‑level studies.
Promote assessment literacy
Promote a deeper understanding of test constructs in the teaching profession
Exploring the formative assessment potential of test systems
Cooperation in development and research at tertiary level
Development and research in the testing of young learners (Volksschule)
Research into the relationship among socio‑economic variables and test performances
Research into factors complicating the implementation of testing systems
Create new and make existing corpora in all languages available for research purposes