Up
Department
Program
Course
Resources

REPORT FROM THE COMMITTEE ON ASSESSMENT
May, 1989

Alex Joncas, Karen Schwalm, Carol Sunshine
Department of English
Glendale Community College, Glendale, AZ

View the Final Report from the Assessment Committee about Spring, 1989 assessment activities. 

Responses to the assessment questionnaire last fall revealed that the assessment experience of faculty members in the English Department varied. Some of us had only been involved in the kind of evaluation we did in our individual classes; some of us had participated in scoring sessions (holistic, primary trait or a combination, both formal and informal); some had taught in programs (or had specific knowledge of programs) where formal assessment had been mandated. Not surprisingly, our attitudes toward assessment were affected significantly by our experience. When this committee first met, we found that our backgrounds mirrored that variation within the department, so we began by learning as much as we could about assessment. After extensive reading, far-ranging discussion, and some limited local experimentation, we know significantly more now than we did last fall about the reasons for assessment, its attractions and drawbacks, and the various forms it may take. What follows is a report of our findings, followed by a recommendation from the committee to the department.

What is assessment?

Teachers evaluate student writing continually in the classroom. That is, they ask students to write, they read what has been written, they diagnose areas for improvement, and then they read additional drafts, noting whether improvement has occurred. Evaluation is at the heart of what teachers do; they are experts at it! Assessment differs from this familiar evaluative activity most frequently by moving beyond the individual classroom in a variety of ways. Thus while evaluation within the classroom focuses on the educational progress of the individual student, assessment concentrates on improving the writing of groups of students without regard to classroom boundaries. Evaluation within the classroom occurs in a conversation between teacher and student; in assessment, conversation occurs among colleagues. Classroom evaluation enables an individual student to improve; assessment activities encourage teachers to discuss what they mean by good writing, thus benefiting all students.

Assessment provides answers to questions, and who asks the questions determines what kind of assessment activities occur. When administrators or legislators ask questions, the kind of information collected during assessment and the way it is reported must, of necessity, answer their questions. This is called summative assessment, and it answers questions that "outsiders" pose and reflects their concerns. When teachers ask questions, the kind of information collected (and the manner in which it is collected and reported), answers their questions and reflects their concerns as practitioners. This is called formative assessment. Whether assessment reports "out" or whether it reports "in" is a crucial decision; what writing teachers have discovered in programs across the country is that the kinds of questions outsiders pose often hinder instruction. Teachers believe that the best kind of assessment serves instructional purposes only if it coincides with instructional goals. That is, a well-designed assessment program should answer questions that teachers raise about teaching and learning.

In addition to determining who asks the questions, assessment designers must decide which areas of instruction will be investigated. With respect to writing instruction, assessment can focus on the students, on instructors, on courses, on programs, or on combinations of these. Some assessments focus only on student behavior; others emphasize teaching performance. Some assessments look at the content and range of a particular course; others attempt to evaluate whole programs. Of course, as the area of investigation broadens, assessment becomes more complex. It is more difficult to assess the impact of a college education on "society" than it is to assess the effect of prewriting on the drafts produced by a group of basic writers. Again, valid and useful assessments can emphasize any one or more of these areas.

Finally, assessment efforts can produce individual or aggregate results, depending on the purposes of the assessors. Individual reports can be used to make individual decisions about placement or promotion; these barrier tests help teachers make decisions about the educational progress of students. But aggregate results can be equally useful in describing the characteristics of groups: what can students entering English 101 do?

A taxonomy such as this can help teachers locate assessment activities and make decisions about them. No one assessment procedure serves all purposes and answers all questions, but to be useful to writing teachers, assessment ought to foster the discussion of what they mean by good writing.

Why assess?

One way of learning about the drawbacks or benefits of assessment procedures is to set up pilot projects, and the committee did that in the form of a midterm essay this semester. (A detailed report of that activity is attached.) One of the most astonishing events during our holistic grading session was watching each other come to quick judgments about student essays.

After a single reading, we regularly agreed which papers were better than others. But what was even more interesting was to listen to ourselves make these implicit judgments explicit, to describe why we felt one paper was better than another. For example, we all agreed that paper x was better than paper y because it was better developed; only in conversation did we discover for ourselves what we meant by that: examples and details supporting and illustrating the main idea, not just examples alone. This kind of discussion is at the heart of writing assessment. We make explicit those values upon which we base our judgments.

In articulating these judgments for ourselves, (learning about our own values in the process), we also articulate our judgments for our students. Despite our efforts to the contrary, students often believe that each teacher defines good writing differently, and they move from class to class in search of a teacher whose definition coincides with how they write. While we may applaud their admirable attention to audience, while we admit that different teachers have different course requirements, nevertheless, we all recognize particular features common to good writing. Making our judgments explicit and public helps those students to see the common features in what we all do differently.

However, we also make these judgments about good writing explicit for those outside our department: teachers at other institutions, administrators, legislators. The assessment procedure itself, the assignment sheets or questions, the scoring guides, and the tabulated results document our systematic attention to a question that is at the heart of what we teach: what are the features of good writing? And realistically, such documents are of immense interest to administrators. We have not felt significant pressure to assess in Arizona, but if the experiences of other community colleges across the country are any indication, we will.

Why not assess?

Many members of the department have voiced objections to assessment, and we have encountered similar objections in many of the reports we have read of assessments at other institutions. Writing assessment is fraught with many dangers. Some people feel any assessment will abridge the rights of the individual instructor to operate freely in an academic environment. Others feel that the performance of their students (who may have come to them ill-prepared) will reflect badly on themselves as teachers. Others are afraid that some teachers will "teach to the test." Finally, some are concerned with the additional time such an assessment may require. These are all real and serious concerns, but we feel that we can allay some of them at the outset.

Assessment brings up again in a different way questions about academic freedom raised by multiple sections of the same course. Faculty members, of course, have the right and the responsibility to determine how they accomplish their goals in the classroom. While multiple-section courses are not a new phenomenon, in the past, individual institutions handled variations among sections of the same course differently because they saw such variation as an internal issue. During the past twenty years, however, the frequency with which students transfer between institutions before completing a degree has increased significantly, and institutions have developed a number of mechanisms for demonstrating to students, colleagues and administrators that these multiple sections are indeed the same course, can be listed under the same course number, and can be considered as equivalent at other campuses within the district and at other institutions within the state. First is the catalog description, a contract between the institution and the student. Second is the competency document, authored in this department but adopted (with variations) across the district. Third are articulation agreements hammered out between colleges and universities statewide. Fourth, and finally, is a general adoption of textbooks, allowing for differences among campuses and choices within departments. All of these decisions restrict instructor freedom slightly, and some people may object to them. But they make it possible for students to take English 101 anywhere in Arizona, to accumulate courses at a number of campuses in the state, and eventually to transfer those courses to or from a university, reasonably confident that what they learned from one instructor is roughly equivalent to what a friend learned from a different instructor teaching the same course. Assessment within the department further demonstrates to students that, in addition to a common description of the course, common goals, and common texts, equivalent standards of good writing are being applied to the work they produce. All of these features in common make it possible to offer multiple sections of a single course to large numbers of students.

The second objection to assessment is that our students' achievement will be used as a measure of teaching effectiveness, when, in fact, student achievement depends on multiple factors, of which teaching is only one. This is particularly true of students at community colleges.   In fact, such "value-added" assessment is one of the worst forms of assessment, and instructors are quite justified in objecting to it; however, it is often the kind of assessment suggested by "outsiders" who don't understand educational issues. Teachers have a right to be apprehensive when the assessment information gathered is linked to individual instructors. It need not be. If our purpose is to improve instruction in general, we can determine average scores (as we did in our practice session). Individual instructors can compare their students with the department average if they so wish, but they do not need to reveal that information to anyone else. In fact, most institutions assess in this fashion. The way to ensure that assessment of the writing program does not become assessment of the individual teacher is to design the assessment in that fashion and retain control of the information collected.

The third objection to assessment is that some teachers will "teach to the test," improving their students' scores at the expense of their students' education. Whether this occurs or not depends on the kind of assessment procedure that is employed. If administrators choose a proprietary test (from ACT or some other outside organization), this may be a real danger, especially if they were to choose a multiple-choice test. Such tests are now being used, and more are being prepared; it would be a natural move for those campuses using the ASSET for placement to consider using a similar test for assessment. Almost every national writing organization has stated that writing skill should be assessed using direct rather than indirect measures, that is by having students write, not by having them answer multiple choice questions about writing skills. In addition, many of the benefits we found from our holistic assessment this semester would be lost if we sent our students papers off to someone else to be scored. If we develop our own assessment activities, they will be ours, representing our goals for English 101, recognizing our students, abilities. If we design appropriate assessment activities, "teaching to the test" should be indistinguishable from "teaching the course."

Finally, many of us feel that we are already worked to our limit. The semester isn't long enough to do what we feel must be done; we already put in many hours outside the classroom responding to student papers, doing the work of the department, campus, or district, and participating in other professional activities. We must be concerned with the quality of our own personal and professional lives. There is no way around it: assessment worth doing takes time. It takes time to design the assessment activity; it takes time to work out the mechanics of administration; it takes time to establish evaluation standards, to read and score student essays, to interpret the results. Some of these activities we would be doing anyway for our own courses: writing test questions and grading the responses. This semester, for instance, we used an in-class essay in the middle of the term for our assessment, and we intend to use the final exam period for another such experiment. An assessment activity, thus, can be aligned with any one of several already existing writing occasions: a paper assignment, an examination, an in-class essay. Collaborative assignment writing, sampling procedures and holistic scoring strategies can free up time which can then be spent designing worthwhile assessment activities and setting evaluation standards, activities that we found so illuminating that they were worth the time devoted to them.

How to assess?

There is no rule that says that, in a program this large and this diverse, all instructors must assess in exactly the same manner. In f act, it might be important to resist the temptation to produce a single statistic which can be used to identify writing competency in the program. Three or four such numbers, identifying different accomplishments, might be more useful descriptors of our program over a period of time.

Furthermore, by offering multiple options, faculty members could evaluate differently from year to year if they so wished, learning different things about their students and their own teaching. Assessment activities do affect instruction; by designing varied assessment procedures, the committee hoped to account for the variety of instructional modes operating in sections of English 101, so that each instructor could choose a mode of assessment that was compatible with the structure and aims of his or her own classes, and could work with colleagues with whom. he or she felt comfortable.

What follows are four options for writing assessment. We have tried to describe the important features of each option and to clearly delineate the advantages and disadvantages of each, but we have also left certain decisions up to those instructors who choose that particular option. The procedures themselves will become more clearly defined in use. Each option has a different purpose and produces a different kind of assessment information collectively, however, they can produce a picture of the teaching and learning that occurs in this department in English 101.

Option # I -- Portfolio Assessment

Purpose: To evaluate student writing across a number of occasions or purposes, and to encourage students to see their writing as always open to revision.

Description: Each student compiles a portfolio containing three pieces of writing assigned in English 101, two representing required modes as stipulated in the competencies and one representing one of the optional modes. Each instructor warrants that the writing was done for class and represents the work of the student. The portfolios are read anonymously by another faculty member and a judgment is made based on the three pieces of writing: passing or not passing. How individual instructors use this judgment is left up to them. in some programs, if a portfolio does not pass, the student may not pass the class, but a student whose portfolio passes may still receive a non-passing grade if other requirements have not been met.

Advantages: Portfolios can present a more accurate picture of student writing ability because they contain pieces written in different modes, on different subjects, for different purposes. They emphasize the process approach to writing; students can continue to revise their writing throughout the semester, even after the instructor has graded the paper. Thus, an early paper can benefit from insights gained at the end of the semester. Portfolios can significantly change the relationship between instructor and student from evaluator to coach.

Disadvantages: Some instructors who do not emphasize multiple drafts in their classes might find constant revision difficult to handle. Instructors would be required to warrant that they had seen multiple drafts of student papers to guard against plagiarism. Portfolios do not assess the student's ability to write extemporaneously. The evaluation can be a terrible burden: portfolio assessment is usually done in small programs. However, a midterm reading is often scheduled so that students can get a preliminary reading on one or two papers, reducing their anxiety. This also eliminates some of the reading load at the end of the semester.

Evaluation: Those instructors choosing this option would agree on the criteria for evaluation, i.e. what writing competency should a student be able to demonstrate upon completing English 101, a combination of student writing performance and difficulty of assigned task. The critical question for this group might revolve around the description of IC', papers. Usual outcomes: Most portfolios pass. This is to be expected; there is no point in students submitting 'D" papers.

Appeals: In some programs, if a student portfolio has not passed the final reading, the instructor (not the student) may request additional readings. If a single instructor will agree to pass the portfolio, it passes.  In actual practice, requests for second readings are rare. If the evaluation is advisory, appeals are irrelevant.

Reporting: In addition to reporting the percent of passing portfolios, instructors might tabulate the kinds of assignments being made for each one of the modes in English 101 and report these to the department. Some analysis of the reasons for failing portfolios could be included. Student and instructor attitudes toward the process might be collected.

Option # 2 -- A Common Assignment

Purpose: To evaluate revised student writing responding to a single assigned task that requires the student to choose an appropriate mode of development and develop an appropriate thesis.

Description: Clusters of three or four instructors would design a final writing assignment for their students that, in their judgment would allow students to demonstrate acceptable levels of writing proficiency at the end of English 101. For example, the assignment might specify the subject matter for writing but require each student to demonstrate use of more than one mode of development in his or her paper, to generate an intention, and to determine an appropriate audience. Students would follow the normal classroom procedure for writing papers, submitting drafts, soliciting peer reviews, revising in any way they see fit. Papers would be handed in without drafts and unidentified as to instructor or student. A sample would be pulled to establish the scoring standards, and the remaining papers would be scored holistically by two readers other than the instructor. Instructors would agree in advance how to use the evaluations in figuring their grades.

Advantages: This assessment activity requires students to apply skills taught during the semester. Students would still be able to revise their papers over several class periods, and the instructor would shift from being evaluator to being coach. Students who have done all their writing on computers would not have to change their behavior at the end of the semester for evaluation. Students who work more slowly would not be penalized.

Disadvantages: Instructors might find it difficult to design an effective assignment. Assessment would be based on a single extent, even though it was spread over time. The amount of help each student receives might differ significantly. Evaluation of papers attempting quite different tasks might be difficult.

Evaluation: Those instructors choosing this option would agree on the criteria for evaluation, i.e. what writing competency should a student be able to demonstrate at the end of English 101, a combination of student writing performance   and   f difficulty of assigned task. The critical question for this group might also revolve around the description of "C" papers.

Usual Outcomes: The distribution of grades would probably follow the distribution of grades in individual classrooms.

Appeals: Because these papers would be holistically scored, anchor papers would have to be established. If a paper received divergent scores (separated by more than one point) an additional reading would be required. Reporting: In addition to a percentage of passing and failing papers, instructors choosing this option could provide the department with a copy of the assignment, the scoring guide used and typical papers at each grading level.

Option 3 -- A Common Examination

Purpose: To evaluate student extemporaneous writing ability performed under conditions typically found in college classes (where they are writing about new information and can prepare for the examination).

Description: A committee of interested faculty choosing this option would select a topic, find appropriate readings, and write questions that elicit responses in one of the modes required by the competencies. One week before the final exam period (the last week of class), students would receive a packet of readings on the topic. They would be encouraged to read the passages carefully and think about the issue discussed. A series of prewriting activities could be designed to go along with the readings, or those could be left to instructor discretion. At their regular exam period, students would be randomly assigned one question to answer which would require them to employ one of the required modes outlined in the competencies. Faculty would grade their own essays without making any marks on the papers. All essays would be collected at the end of the exam week and held for evaluation at the beginning of the following semester. At that time, the papers would be shuffled, a random sample would be pulled and all instructors electing this option would establish the anchors. Another sample of papers (representing all instructors participating) would be pulled and those same instructors would score that sample of papers. After the results have been tabulated, individual instructors could retrieve their own students, papers and if they so desired, compare the scores to their own grades.

Advantages: The authority of the individual faculty member to be the sole determiner of grades would be retained. Choosing topics about which students do not already possess information can neutralize the effects of prior knowledge on writing ability; all students will have access to an appropriate body of information. Because the topics would be new to the student, and reading would be required, the essay would more closely match the kind of writing expected of college students. A picture of the abilities of our students could be drawn without reading all the papers.

Disadvantages: This option tests both reading and writing skills which might require some instructors to change what they do in their classes. While students should be able to plan ahead of time and will have sufficient time to do some revising during the exam period, they still are writing extemporaneously. There might be great variation in what individual instructors do with the preparation time. Students are writing in only one mode, even though they would have to prepare for several. Students would not be able to choose the question they answer. The assessment would occur after-the-fact, and its effect on teaching and learning might be more difficult to define.

Evaluation: Those instructors choosing this option would agree on the criteria for evaluation, i.e. what writing competency should a student be able to demonstrate at the end of English 101, a combination of student writing performance and difficulty of assigned task. The critical question for this group might also revolve around the determination of passing scores.  Because of the difficulty of scheduling large-scale scoring at the very end of the semester, group scoring would have to wait until the beginning of the next semester, and faculty might not have the same attitude toward achieving consensus as they would have when operating under time pressure.

Usual Outcomes: Student writing performance on the exam would cover the full range of student writing performance in our classes.

Appeals: Because the essays would be scored after grades had been determined, this activity would have no affect on student grades.

Reporting: In addition to the percentage of passing and failing papers, instructors in this group could provide the department with a copy of the essay materials, of the scoring guide, and of typical or anchor papers.

Option # 4 - Individual Reporting

Purpose: To provide a mechanism for individual instructors who do not wish to undertake any of the other options to assess their own courses.

Description: Individual instructors would design their own procedures for evaluating their students' writing at the end of the semester. No more specific description is possible because the options are unlimited.

Advantages: Instructors who feel that they have effective evaluation procedures operating in their own classes would not have to change what they are now doing.

Disadvantages: There would be no way to separate assessment from evaluation of the individual instructor, and any criticism of the assessment procedure might be construed as criticism of the instructor. The benefits of collaboration would be lost, and additional time would be required to prepare the evaluation documents.

Evaluation: No changes would occur in student evaluation.

Usual Outcomes: Unknown.

Appeals: Students who wish to appeal their grades would follow current procedures.

Reporting: Instructors choosing this option would present to the department individually the same information   presented by groups of instructors choosing other options: a description of the evaluation procedures, a copy of the assignment or test employed, a copy of evaluation standards, a report of grade distributions, and copies of typical student papers receiving particular grades.

Committee Proposal

We propose that the department undertake, over the next three years, a pilot assessment of English 101 governed by the following conditions:

  • that faculty (both regular and visiting) choose one of the four previously described assessment activities each year.
  • that one day during the week of accountability before each semester begins be devoted to assessment activities (portfolio criteria, assignment writing, holistic scoring), and that no other campus or district activities impinge on that time.
  • that no assessment figures be collected that could be used to evaluate individual faculty members unless that faculty member so chooses (as in option #4).
    that individual clusters keep track of the time spent developing assessment assignments, scoring papers, and interpreting results and report that information to the department with their other assessment results.
  • that an assessment committee made up of four instructors be established with course release to coordinate efforts of faculty members choosing the different options, to read assessment research so as to answer procedural questions raised by colleagues, to accumulate and synthesize the reports from different groups and individuals, and to guide the evaluation of the assessment program itself.
  • that at the end of three years, the entire department reevaluate the undertaking to see if it should make assessment a permanent part of English 101, with or without adjustments in any of the procedures.

    • Bibliography

    Assessment Update: Progress, Trends, and Practices in Higher Education. San Francisco:
         Jossey- Bass, 1989.

    Ballard, Ronald L. "Toward an Appropriate Exit Examination for English Composition." A Report
         Funded by Hagerstown Junior College. July 9, 1986. ERIC ED274405.

    Barritt, Laren, Particia L. Stock, and Francelia Clark. "Researching Practice: Evaluating 
         Assessment Essays." College Composition and Communication 37.3 (1986): 315-327.

    Breland, Hunter M., Roberta Camp, Robert J. Jones, Margaret M. Morris, and Donald A. Rock.
          Assessing Writing Skill. Research Monograph No. 11. New York: College Entrance 
          Examination Board, 1987.

    Brossell, Gordon. "Rhetorical Specification in Essay Examination Topics." College English 45
        (1983):165-73.

    Brossell, Gordon, and Barbara Hoetker Ash. "An Experiment with the Wording of Essay Topics."
          College Composition and Communication 35.4 (1984): 423-425.

    Cooper, Charles R. "Holistic Evaluation of Writing" in Evaluating Writing: Describing, Measuring,
          Judging. Ill.: National Council of Teachers of English, 1977. Urbana,  2-31.

    Cooper, Charles R., and Lee Odell. Evaluating Writing: Describing, Measuring, Judging. Urbana,
         Ill.: National Council of Teachers of English, 1977.

    Davis, Barbara Gross, Michael Scriven, and Susan Thomas. The Evaluation of Composition   
          Instruction. 2nd ed. New York:      Teachers College Press, 1987.

    Dovell, Patricia, and Diane Buhr. "Essay Topic Difficulty in Relation to Scoring Models."
          Paper presented at the annual meeting of the American Educational Research Association,
          San Francisco, April 1986. ERIC ED267127.

    Ewell, Peter T., and Robert P. Lisensky. Assessing Institutional Effectiveness: Redirecting the
    Self-Study Process    . Boulder, Colo.: Consortium for the Advancement of Private Higher Education, 1988.

    Prederiksen, Norman. "The Real Test Bias: Influences of Testing on Teaching and Learning."
          American Psychologist 39.3      (1984): 193-202.

    Greenberg, Karen L., Harvey S. Wiener, and Richard A. Donovan, eds. Writing Assessment:
          Issues and Strategies. New York: Longman, 1986.

    Hoetker, James, and Gordon Brossell. "A Procedure for Writing Content-Fair Essay Examination
          Topics for Large-ScaleWriting Assessments." College Composition and Communication
          37.3 1986': 328-335.

    Hutchings, Pat. "Linking Assessment and Teaching."  Update 1.1 (1989): 8-14. Assessment Update.

    Livingston, Samuel A. "The Effects of Time Limits on the Quality of Student-Written Essays."
          Paper presented at the annual meeting of the American Educational Research Association,
          New York, April, 1987. ERIC ED286936.

    Lloyd-Jones, Richard. "Primary Trait Scoring." In Evaluating Writing.  Ed. Charles Cooper and
          Lee Odell. Urbana, Ill. The National Council of Teachers of English, 1977. 32-66.

    MacDonald, Susan Peck. "Problem Definition in Academic Writing." College English 49.3 (1987):
          315-331. Magner, Denise K. "An Expert on 'Assessment'  Wants the Concept to Become
          Universal." The Chronicle of Higher Education 1 Feb. 1989: A10.

    McIntyre, Chuck. "Assessment in Community Colleges." Assessment Update 1.1 (1989): 12-13.

    Notes from the National Testing Network in Writing. Vol. 8. New York: Instructional Resource
          Center, 1988.

    Ruth, Leo, and Sandra Murphy. "Designing Topics for Writing Assessment: Problems of Meaning."
         College Composition and Communication 35.4 (1984): 410-422.

    Steele, Joe M. "Trends and Patterns in Writing Assessment."  Paper presented at the Third Annual  
         Conference on the Assessment of Writing, San Francisco, March, 1985. ERIC ED268146.

    White, Edward M. "Holisticism."  College Composition and Communication 35.4 (1984): 400-409.

    White, Edward M. Developing Successful College Writing Programs. San Francisco: Jossey-Bass,  1989.

    White, Edward M. Teaching and Assessing Writing: Recent Advances in Understanding, Evaluating, and
        Improving Student Performance.  San Francisco: Jossey-Bass, 1985.