Department of German

CHRONOLOGY OF ASSESSMENT-RELATED ACTIVITIES DURING THE IMPLEMENTATION PHASE OF THE CURRICULUM "DEVELOPING MULTIPLE LITERACIES," 

YEAR ONE, AY 1997 - 1998

We realized right from the beginning that our curricular and pedagogical shift inherently required an equally dramatic shift in assessment practice. For the first three semesters of the implementation of the curriculum we dealt with that need primarily on an informal basis, through various level-specific decisions on how to assess student learning outcomes during a course and at its conclusion. Typically, these decisions were arrived at by instructors who collaborated in various ways, a process that redirected our assessment practices, improved their quality, and provided some first important feedback. The following activities deserve particular mention:

• At the end of the first semester of implementation of the new curriculum, in the fall of 1997, students answered a questionnaire about their perceptions of our efforts.

• At the end of the first year, language performance data were gathered in courses at levels I-III in a coordinated fashion. This information about student learning at different levels of instruction is currently being evaluated along several dimensions.

Thus, while classroom- and course-based assessment practices in fact increasingly reflected our experience with teaching and learning within such a curriculum, we also became aware that we were in need of much better assessment criteria, greater uniformity in assessment practices, and, in general, more elaborated knowledge regarding assessment in a task- and content-based curriculum.

In particular, the following complex categories required further elaboration:

• Assessment of learning outcomes at the various instructional levels and throughout the revised curriculum. This typically takes the form of formative and summative assessment at particular instructional and curricular points

• Assessment of the curriculum renewal project as a whole and its washback effects on the entire project

• Global assessment practices that are somewhat independent of though related to the curriculum renewal project (e.g., placement examination, required proficiency examination of students in the School of Foreign Service, study abroad examination)

YEAR TWO, AY 1998 - 1999

Our ability to tackle these matters in a systematic fashion was greatly enhanced, during the spring semester of 1999 and over the summer, due to the presence of John Norris, a Ph.D. Candidate in Applied Linguistics at the University of Hawai’i, with a specialization in educational measurement and task-based assessment in language programs. His stay with us was made possible through exemplary cooperation with the National Foreign Language Resource Center at the University of Hawai’i. Not only did he offer us elaborate knowledge on a range of task-based assessment issues within a curricular context; with an undergraduate and Master’s degree in German, and one year of graduate study in Germany as part of a Fulbright exchange, he was also able to ascertain, first hand, what was happening in our curriculum and in our classes -- and readily participated in our conversations, in English and German, about all these matters!

In particular, he engaged and led us in the following activities in support of the development and implementation of a task-based performance assessment framework for the entire undergraduate curriculum:

• A series of faculty development workshops (January 27, April 15, and May 21, 1999)

These laid out issues in task-based assessment, provided feedback on current assessment practices in the Department (based on Norris’ observation of classes and meetings with faculty teaching at all levels), identified areas in need of review, suggested matters that required immediate attention, and coordinated the process of establishing guidelines for administration, scoring, and feedback in Levels I - IV.

• Creation of use specifications for assessment in Levels I - IV

Following guidelines and instructions presented by Norris, committees of instructors at each of the four program levels developed use specifications for assessment that address, at some level of detail, the following global matters pertaining to intended uses of assessment:

- A general description

- Who uses the assessment?

- What is assessed?

- What is the purpose?

- Who and/or what is impacted?

- Summary of recommendations for instruments and procedures.

• Creation of a policy document regarding assessment practices for the entire Department

This document describes the intended use of assessment in the German Department in order to clarify its role and nature for those who perform the assessment (e.g., teachers and the German Department as a whole) and for those who use the outcomes of assessment (e.g, teachers, students, various administrative units within the university, outside constituents), and to delineate its use for evaluating the curriculum as a whole, particularly in its sequenced courses in Levels I-III and in the small group of Level IV courses.

• Creation of a series of questionnaires on assessment practices and perceptions in courses at Levels I - IV

Responses to these questionnaires which were sent to both faculty (response rate 100%) and students (response rate 69%) addressed general issues in assessment practice at each of the four program levels; questioned the use of particular assessment types at particular program levels; and focused on grading practices in German program classes. In addition, students were asked to respond to two short sections dealing with the placement exam and the proficiency exam, a graduation requirement for students in the School of Foreign Service. These high return rates enabled relatively trustworthy interpretations about student and teacher attitudes towards assessment practices in general, specific test types at each of the curricular levels, and other issues like grading, placement testing, and proficiency testing.

• Final program-wide meeting for Spring semester, 1999

Curricular committees revised test use specifications in response to general policy statements and survey results; assessment practices were reviewed and suggestions for academic year 1999-2000 made in order to better implement assessment in intended ways; plans were instituted for developing new placement exam.

• Creation of a new placement test

During the summer of 1999, our efforts concentrated on developing a new placement test which would better correspond with the curriculum, while still being administrable within placement exam time constraints. Developed by Norris with the help of several graduate students, the exam is comprised of reading comprehension, listening comprehension, and C-test sections, all based on authentic texts sampled from level-appropriate materials. Pilot versions of the placement exam, self-assessment, and background information form were administered to the students participating in our summer study abroad program in Trier, Germany. Norris analyzed the pilot test results for item and test qualities, revised the instruments accordingly and, together with Byrnes and Pfeiffer, decided on its final version and set preliminary decision cut-scores for placement into variouscurricular levels.

• Creation of various self-assessment and background information instruments

In addition, Norris developed a student self-assessment instrument based on a range oftasks associated with the different curricular levels and incorporating oral proficiency self-assessment developed by the Center for Applied Linguistics.He also developed a background information form for gathering German language learning experience information from incoming students.

YEAR THREE, AY 1999 - 2000

• C-tests and self-assessments administered during first week of classes to all students (except level I).

• Data collection initiated for the use of assessment-based information in curricular evaluation efforts and the validation of curricular assessment system.

• German Speaking Test (Simulated Oral Proficiency Interview from CAL)administered to group of 29 volunteers from several curricular levels for use in criterion-related comparisons of students' oral abilities across curricular levels.

• Five graduate students began training in German Speaking Test (GST) rating process (using the ACTFL Guidelines); training completed by early November. Graduate students rate 29 volunteer GST tapes.

• C-tests administered to all students during final week of the semester in order to investigate cross-curricular validity of the placement system as well as change in student abilities over the course of instruction. In addition, the C-test and self-assessment were administered to all level I students at end of the fall semester.

• Final assessment systems clarified for each of the integrated curricular levels. All final assessment elements recorded and collected, including oral interviews, final written essays, and final exams; data to be analyzed in order to validate the use of such assessment for the purposes of curricular outcomes evaluation

• GST to be administered during first few weeks of spring semester to all students at levels I-III, and to a sample of students at level IV, in order to provide a point of comparison for curricular evaluation and to assess oral abilities of students according to a widely accepted oral proficiency assessment system. GST tapes to be rated by trained graduate student raters.

• Writing instruction and assessment practices to be reviewed and revised during program-wide workshop.
• Oral assessment practices to be reviewed and revised during program-wide workshop.
• Guidelines for administration and scoring criteria to be developed for assessing written and oral performance within each of the curricular levels as well as across curricular levels.
• Reliability analyses to be conducted on oral and written performance assessment. Data from oral interviews, written essays, and final exams to be analyzed in on-going validation of their use for curricular evaluation (i.e., in order to determine what aspects of the curriculum are being assessed within each, how consistently each is being administered and scored, how scores may be interpreted, how scores may be used in evaluating theachievement of curricular objectives).
• Further survey data to be gathered concerning teacher and student perspectives on the general consequences associated with assessment in the German program.
• Quantitative data to be entered and analyzed (Norris). All data to be integrated into validity judgments about the assessment system. Implications for achievement of curricular objectives to be inferred. Outcomes of various analyses to be reported to teachers at the end of Spring semester, 2000.

• Follow-up data to be gathered at conclusion of spring semester to track change across entire year of program (including C-tests, self-assessments, and all final assessment system data).
• Curricular committees and coordinators to review the achievement of curricular objectives based on data provided by assessment system; evaluate and summarize achievement; recommend changes in implementation of curriculum.
• Norris to provide final summary of assessment practices and the use of assessment as a tool for evaluating curricular outcomes, including recommendations for further development.

December 4, 1999