As part of the materials design module on the course we have been asked to design a course-book evaluation check-list. I paired up with Helen Lee and together we compiled our ideas to make a worksheet. Helen put in the majority of the work and I added a couple of bits and pieces and together we formatted the design.
My reading to get a better idea of design and exemplars on course-book evaluation grids focused on McGrath. I; (2002) Materials Evaluation and Design for Language Teaching, Edinburgh University Press
Unfortunately I haven’t got the book to hand and cannot page reference the section I read (which will teach me to note down references as I go!) but the Mcgrath has a large section that concentrates on different examples of course-book evaluation grids. We didn’t base ours on any one particular grid but the areas are common to most of the examples in McGrath and the check-list is detailed. Helen took blank copies into a local EFL school and teachers looked at some books and evaluated them. The results were positive with the teachers commenting that it helped them to focus and evaluate the course-book. My reflection of this is that there are a lot of questions in the questionnaire, all relevant, and that at the end of completing the form teachers can look down the answer column and see the number of Yes/No/NA that they’ve written, look at the Nos and decide if they still think they would use the book more often than not.
I also read chapter 11 by Frances Amrani entitled The process of evaluation: a publishers view in Tomlinson, B (2011): Materials Development in Language Teaching (Cambridge University Press) and below are some notes on evaluating material and the process and events that happen.
1. There is a stage called Post evaluation, evaluation – this is where the material needs to be evaluated at a reasonable time after its initial use as teacher and student needs may change. This highlight to me the whole process of evaluation is an on-going issue which I hadn’t thought about beforehand. It is often easier for people to remember the bad about something new they have tried than the good. I imagine this is what publishers want to hear so they can improve on the current version. The whole process of evaluation is an on-going procedure throughout the development of material from initial idea to post production.
2. Focus groups – these are mainly used by publishers to identify teachers’ reactions to new materials and to allow further discussion. However, teachers might not say what they really think about the material, what they say they would do with the material is sometimes not what happens in the actual classroom. For changes to be made to materials there needs to be several focus groups all saying the same thing which is then backed up by proven expert opinion.
3. Editorial visits and classroom observations – allows publishers to get first hand views of their materials being used – however, the presence of an observer in the class can affect the way the teacher plans and executes the use of the material. In my experience, teachers rarely d.t.b….this means “do the book” exactly as it stands and teachers may feel under pressure if observed to do it as it is without any adaption.
4. Questionnaires – these are popular way of evaluating material with the results usually analysed through a specific analysis programme. May not be taken as seriously by all teachers possibly due to the time factor as it takes them away from their own planning! It is probable that a face to face meeting with a publisher is more effective and so the questionnaire results could be skewed. I believe that a way to combat this is to invite written comment about a book on a website forum, this allows teachers time to think about the material.
5. Centre pull-out – Some centres pull out of the pilot which skews the number of representatives in that particular sector
6. Frank honesty? – Some reviewers are not as honest as they should be when evaluating a product – say what they think needs to be heard because this is the easier option. In addition to this if a centre does not have many resources anything new is a bonus and the evaluation may be less objective.
7. Criticism only – Some reviewers criticise without giving ideas on improvement
8. Completed questionnaires – Teachers may not answer the questions fully and not take the questionnaires as seriously the need to
Other issues (not Tomlinson)
9. Impartiality – Teacher’s attitudes may be affected by other different types of course-book/material, therefore they become less impartial and are not able to give a balanced view.
The reading gave me the opportunity to reflect on the evaluation process of a course-book and I learnt a lot about the stages. I particularly agree with Tomlinson’s post evaluation, evaluation stage which is not given enough focus in a teaching setting due to time constraints. Generally, what happens is a course-book is shown to teachers, rejected or accepted depending on their comments and that is the end. There is usually very little time for post reflection which in the long term is detrimental to the book and to the students because if it needs adapting to meet their needs this goes by unnoticed and the plethora of teaching ideas on adaption is once again not shared.