Nathan’s Evaluation Plan

This evaluation plan is to designed to look at aspects of the new online component that will be added to the C.E.E programme prior to students’ arrival in Dunedin. It involves both formative assessment, and feedback once the students arrive in Dunedin for the onsite part of the programme.

1. Testing the Activities

• Testing activities with current C.E.E students
Currently, students on the C.E.E programme have been involved in similar activities during their stay in Dunedin. I have given them these exercises while I was in Japan, so that they were largely working unsupervised, and only received brief feedback on my return. The results have been encouraging, with many students giving good powerpoint presentations (videos have been recorded to use for feedback and evaluation purposes) and peer and teacher feedback provided. Students were given an opportunity to watch these videos and discuss their problems and what they need to work on. In early January 2011, these students will do a further presentation based on a similar theme, and they will then be asked to provide some feedback on their experiences (through interviews).

• Testing with 2011 intake students - online
Based on this experience, the 2011 intake will complete similar (or modified) activities in Feb – Mar 2011, before they arrive in Dunedin, and the feedback and evaluation methods above will be used before proceeding with a more formal introduction of the online component to the C.E.E programme.

2. Ongoing Evaluation Methods

• Peer Feedback
I believe that peer feedback is an important part of learning, and that it is an appropriate method of evaluation in a collaborative activity such as the one I am proposing. I have tried this kind of peer feedback with current C.E.E students, and found it to be quite successful, provided students are given a detailed framework within which to work. A sample can be found here… Although comments tend to be (overly?) encouraging, some students still pointed out some quite relevant advice on how to improve – here is a summary of some of the feedback provided on the powerpoint presentation from December 2010…

• Ongoing teacher feedback
Students will be provided with language pointers and assistance during the project so that they can develop throughout the project, not simply at the end. We will monitor not only their time on the site, but also their response to the feedback and whether this is reflected in their posts. At present this is done with the paper journals, but I believe this will be more effective with a blog, as feedback can be provided without the need for students to hand in journals, and review will be easier too.

• Kanazawa teacher feedback
As special part of this programme is that staff at Kanazawa Technical College will be interacting with the students during this online learning period, so they will be able to provide direct and honest feedback both to the students and to us.

3. Data Gathering

• Portfolio assessment
Students’ collaborative production will be assessed, and its continued use and updating monitored.

• Otago Polytechnic Colleague feedback
I will get other teachers involved in providing feedback – in particular Jean Gordon, who is involved with the C.E.E programme. Later feedback may be obtained from other C.E.E teachers and Japanese support staff as appropriate. I would also welcome feedback and exchange with Marc Doesberg and staff at Foundation studies who may be working on similar pre-departure or online courses for Korean and Chinese students.

• Host family feedback
Ultimately, one of the main goals of this pre-departure course is to improve communication between students and host families by getting students to think about who they are and where they come from, and how they can explain this in English. A large part of the feedback, therefore will come from host families, in the form of a survey to be carried out several months into the Otago Polytechnic part of the C.E.E programme, perhaps to coincide with a face-to-face feedback session with the host families to iron out any cultural or social issues that have arisen during the first few months of the homestay programme.

• Student interviews
Students will be asked to reflect on their experiences and the outcomes several months after arriving in Dunedin. I like the idea of doing this through an informal interview, either with myself or another staff member. In my experience, surveys tend to get very superficial results, and this is especially so with international students. Asian students in general, and Japanese in particular, are reluctant to criticize their teachers, so comments and assessments tend to be very positive, but informal discussions with people they feel comfortable with often bring out more honest opinions – obviously there is still the possibility of being led to give certain answers that would need to be watched for. Interviews could be recorded provided students felt comfortable with the technology, and these video recordings assessed by independent staff if necessary.

• KTC staff feedback
There is also the scope to work together with staff from KTC who teach the students after they return from Otago Polytechnic, to get further feedback and to better integrate the activities with what is being done at KTC prior to, and after the C.E.E programme at Otago Polytechnic.

4. Criteria for Judging Success:

• completion of the activity; time spent and responses to online work

• achievements of outcomes based on tests and activities built into moodle / online learning environment

In terms of English ability and cultural awareness, short tests and activities to check the understanding of the material covered will be introduced and questions asked during the coursework. Overall English progress throughout the C.E.E programme has tended to be measured using the TOEIC (Test of English for International Communication) test. Benchmark tests and progress scores during the year at OP have been recorded for the past few years, and these will provide a guide as to overall English progress, but they will be of only limited use in evaluating the short online component of the course prior to arrival in Dunedin.

• Feedback from students and staff both at KTC (English teachers) and at Otago Polytechnic

• Effectiveness in improving communication between students and host families.
This will be very difficult to measure objectively, and will vary with a student’s natural willingness to communicate, confidence, and previous language ability. Some idea of the effectiveness should be possible, however, based on surveys and discussions with host families, especially those host families who have hosted KTC students previously, and who are therefore familiar with communication problems of these students on arrival at Otago Polytechnic

• The ability of the students to interact with other New Zealanders, both students and members of the public.
Again, this will be difficult to quantify, and may require several years close observation and recordings to achieve reliable data as to their effectiveness.

5. Reviewing Evaluation Data

• A full review
A review will be undertaken after the trial period (2011) is complete, to see how effective the online learning activities have been. A report will be prepared to be presented to stakeholders, both in ABE, KTC and host families who are interested to show how effective this new component has been, and what can be done to improve outcomes, and the activities / delivery methods will then be adjusted or replaced in consultation with Kanazawa teachers and former students, before the programme becomes a formal part of the C.E.E programme in 2012.