J. J. Gibson criticized his fellow psychologists for studying what was accessible and easy to measure rather than what was important (Reed, 1988). Recently I have applied his admonition to my teaching evaluations. Although my course content is revised frequently to include current information in the field, my method of teaching evaluation has remained largely the same, consisting of a standard student evaluation of teaching (SET) questionnaire composed of multiple-choice questions with space left at the end for written comments. My SET averages have been remarkably stable, confirming that I am a good but not outstanding teacher. Typically my overall rating averages 4.1 on a five-point scale, with a small standard deviation. Written comments from students praise my enthusiasm and knowledge of the subject matter and criticize my examination questions and grades as not reflecting what they have learned during the semester.

I have reached what Erikson ( 1950) called the stage of generativity in my teaching, which includes the desire to sum up what has been accomplished and note what was left undone. Freed from pressures toward further career advancement, one can confront difficult questions of purpose and meaning, what some have called "the big fuzzies." As a college teacher, the important questions for me were not "Were my courses well organized?", "Was I accessible to students?" or even "How could I have raised my overall SET mean above 4.1?" but the more difficult and vague question, "Did my students learn anything in these classes that made a difference in their lives?" The long-term answer to this question would require a follow-up investigation among graduates that would be highly unusual and inefficient at the level of a single course. A short-term answer, involving contemporaneous students, is more practicable.

My goal in becoming a psychology instructor was to teach students aspects of the field that would make a difference in their lives. Almost four-decades of using SET has convinced me that the customary rating scale with quantifiable multiple-choice categories does not adequately address awareness changes, as expressed in phrases such as "opening the students' eyes," "establishing connections between laboratory and life," and "changing the way students look at the world. " If these changes are not measured on the SET, it is legitimate to ask whether they can be assessed on examinations. At least in the large lecture classes I teach, I believe the answer is no. First, examinations test knowledge rather than opinion, and typically focus on abstract principles rather than aspects of the student's life. Second, my large lecture classes rely heavy on multiple choice questions which seem less suitable than essays for assessingchanges in awareness. For these reasons, I find it necessary to employ an additional procedure, independent of SET and examinations.

At the first class session, I announce that the course has two objectives, the first involving content and the second awareness. The first objective relates to the subject matter of the field, specifically the major theories, concepts, methods, and research findings which are covered in lectures and the textbook. The second objective is to influence the way that students look at the world. This is attempted through examples that link the subject matter to students' life experiences, class exercises, demonstrations and assignments, and through A-V demonstrations that show real-world examples of the concepts discussed in class. The combination of examination grades and SET, supplemented with less formal measures, including class size, attendance, participation, and office hour visits, have been sufficient to determine if the first objective has been fulfilled. To evaluate achievement of the second goal, I make the following announcement on the last day of the abnormal psychology course, after the students have filled out the standard SET.

At the first class session, the instructor stated that the course has two goals. The first is to teach the concepts, terminology and research findings of abnormal psychology. The second objective is increased self-awareness and understanding of other people. On a piece of paper, can you describe the degree to which the course accomplished the second objective, in terms of increasing your self-awareness and understanding of other people. Use specific examples if you want, and if the course did not, you can indicate this.

Student responses to this question indicated that the abnormal psychology class had altered awareness in these and other respects:

• Increased empathy for the mentally ill, the homeless, and family members.
• Learned about effects of deinstitutionalization.
• Realized that many of those with mental disorder don't want to change
• We are not always in control of our actions.
• More respect for individual differences.
• Realized how fortunate I was.
• Laws relating to mental illness.
• Drug information.
• More aware of euphemisms.
• Mental disorder can affect anybody.
• Disorders can range from mild to severe.
• Used to think that the mental illness was due to faulty character; now see it as a response to something beyond individual's control.
• I have become more tolerant.
• Became aware of causes and treatments for mental disorder.
• Able to explain these things to others.
• Can assist friends and family members.

A slightly different open-ended question is used on the last day of my environmental psychology class, following the standard SET:

At the first session, you were told that the course has two objectives. The first is to teach the theories, methods, concepts, terminology, and research findings in environmental psychology. The second objective is to increase your awareness of the immediate environment, to open your eyes to things not previously noticed. On the piece of paper in front of you, can you indicate the extent to which the class has changed the way you see and experience the world.

• All students in the classes reported changes in awareness in these and other respects:
• Pay a lot more attention to seating arrangements and how people congregate in public locations.
• Look for landmarks and other distinguishing features of cities.
• I notice form, shapes, colors more, and the presence of plants, what makes a room successful.
• More aware of problems connected with high-rise buildings.
• Learned about security problems (poor defensible space) when building are hidden from public view.
• Sidewalk project made me look down. Really notice things in the concrete.
• More aware of space use, especially personal space.
• In the natural environment I notice little things that I never would have before.
• My mother being in a hospital had me paying particular attention to her room. I immediately went out and bought flowers, plants, and nature posters to brighten up the dull and depressing room. Also went back to her hope for possessions to personalize the room.
• Wonder why drab classrooms exist. When I walk through buildings, I notice the lighting, paint color, and temperature.
• Aware of poor furniture arrangement in most airports, hospitals, etc. and notice benches and building ledges not meant to be sat on.
• More conscious of the trash and litter. I make more of an active effort to pick it up.
• Find myself telling people that trees raise property values and about the proper design of parks and plazas.
• By noticing small details, I take life slower and notice the many "little things" around me.
• Pay more attention to nonverbal cues, how people claim space. I ask myself why certain spaces are used more than others and what design elements can be improved.

Discussion

The SET forms I use did not ask the wrong questions. I remain committed to delivering clear lectures, encouraging class discussion, being accessible, and the other issues covered by the multiple-choice questions. I rely on the SET to tell me if I have met student expectations in such matters. Yet I have come to realize that I also want to know whether or not the course was meaningfully connected to the student's life, and not just another step on the path to a degree and a career. The proposed procedure is a supplement to a traditional SET, not a substitute. I am convinced that if the awareness question is not asked directly, the instructor will remain in the dark as to whether changes have occurred and what they were. What I have done is a variant of the familiar parent question, "What did you learn in school today?" The parent making this query is less interested in hearing about specific mathematical operations or verb conjugations, than in knowing whether the child has learned anything useful and important in the classroom.

As a teacher of research methods and co-author of a methods textbook, I am acutely aware of the problems in tabulating and interpreting answers to open-ended questions. I am also familiar with techniques of quantitative and qualitative content analysis. If I wanted, I could use these responses to construct multiple-choice questions, asking students the degree on a 5-point scale that the course increased their awareness of personal space, color, lighting, chair arrangements, territoriality, and the other concepts covered in the course. Yet I doubt that formal tabulation and quantification of responses are necessary or desirable. This is not a research project or a procedure required by the institution. The written responses are for my guidance only. They tell me, anonymously, in the students own words, and separate from the grading and faculty review, what students learned in the class that was connected to their lives. This information goes beyond anything I had obtained in the standard SET or from examination answers. Indeed there is a fundamental difference between a student knowing something in the abstract, and able to answer correctly on an examination, and making connections in their own lives. The latter is not, and should not, be graded. Although it can be quantified, as a major, moderate, slight, or no change in awareness, there is little reason to do this, and it would take extra time and effort. For me as an instructor, it is sufficient to read the essays in their raw form and take notes on key points to be included in the next course offering. It takes five minutes to administer the single open-ended question and less than 30-minutes to read through the replies.

Finding out that students learned things they considered important and useful in the class has helped to prevent teaching burnout, to maintain my enthusiasm and commitment in large classes where I know few students by name. How long could I continue to be a "4.1 instructor"? I have learned that no matter what I do, I am not likely to raise my overall instructor mean above 4.3. For me personally, the awareness questionnaire has told me that the students learned some valuable and important things (both in their terms and mine), independent of my SET instructor rating of 4.1 and the class average of 81.3 on the final examination.

References

Erikson, E. H. (1950). Childhood and society. New York: Norton.

Reed, E. S. (1988). James J. Gibson and the psychology of perception. New Haven, CT. Yale University Press.