Abstract
This paper describes the kinds of evaluation employed in the creation and management of a credit course in technical writing developed at the University of Waterloo.  From September 1995 to April 1998, sections of this course have been offered entirely on the Web to students across Canada at 4-month intervals.  The course uses SGML converter technology in the creation and maintenance of its materials and in students' preparation and submission of assignments.  Evaluation includes examination of students' records  of system use and access, assignment preparation and a variety of electronic communications, as well as the electronic marking and measurement of their course assignments.  We attempt to assess group performance against perceptions and to incorporate student requests into our design and expectations.  In addition to the above methods, we present students with a series of optional on-line evaluations after significant assignments and at the conclusion of their final report at the end of the course.  All student responses in this process remain anonymous.

Evaluation Procedures within the University Community and Their Online Variations
Generally, the University of Waterloo distributes course evaluations to students to obtain responses on the success of every course.  Instructors distribute evaluations to on-campus students during the last scheduled class, while distance-education students are mailed the evaluation at the end of the term. In both cases the responses are anonymous, and the professor does not receive the evaluation results until after the final marks are registered. Each faculty administers a variation of the form specific to its academic needs.  Both the number and the range of the questions are limited.  For example, the distance-education evaluation is made up of nine questions dealing with presentation of course material, the course's ability to maintain student interest, the course organization, value of readings, fairness in grading, instructor feedback, and an overall evaluation of both the instructor and the course. Students may respond to these questions in the five categories of 'excellent', 'good', 'satisfactory', 'fair', and 'poor'. In addition, there are three 'comment' style questions dealing with the strengths and weaknesses of the course, and a general view of the course. In this way this form is specific to distance-education needs.

By comparison, our online technical writing course incorporates the evaluation process throughout the course, allowing for a two-way dialogue to which the instructor can react, and the students witness responses to their suggestions. Instead of a single evaluation at the end of the term, students have the option to complete several evaluations throughout the course.  These occur at times when their new skills and our grading of their work enable them to understand both their performance and our learning objectives in light of applied instruction. In total, the students can respond to over one hundred questions. They receive the evaluations after each assignment is submitted but before the return of their graded work online. Such timing provides for more honest responses because the students are not influenced by their assignment marks.  Evaluation responses are completely confidential.  They are sent via email to a designated computer account from which authorship cannot be traced.

These evaluations solicit information on most aspects of users' learning experience, participation, support and their sense of what the course provides, with its relevance to their expectations about their own training and understanding of the processes of technical documentation.  We have synthesized the ranges of questions from five faculty models and resolved them to the new conditions of the electronic version of the course.

We have developed these evaluation procedures to elicit a comprehensive view of both real activity and student opinion about their learning process.  We make modifications in content, course administration and requirements in light of the results of each term's survey and we try to show students the immediate good effects of their responses by announcing changes to procedures and materials. 

These have given us clear evidence that students:

Course Design and Structures
as Background for the Evaluation Process
The 4-month university course in technical writing, which we offer entirely on the Web as credit and certificate learning through the Department of English at the University of Waterloo.  Our colleague, Dr. Katherine Schellenberg, has provided the extensive statistical planning and analysis which now form the bases for our evaluation methods. The course consists of a web site with:

We use these same tools to develop online materials for this and other courses and we incorporate appropriate student materials, (with their permission), in new aspects of the work.

Students complete five technical documents in a sequence of increasing complexity.  They provide all other members of the course with a current resume and proposal letter, from which, by a process of inquiry and selection, all members form themselves into groups of three to complete the central 50% of the exercise. 

They work together to produce portions of a manual, on which they then conduct usability tests.  They complete the course with an extended Report on an aspect of their learning experience, often related to the application of online techniques to other areas of their training and work.  Students create all assignments in SGML and then convert them into HTML for online display to classmates and markers.  A graded version of each assignment is returned to the student under a password for privacy. 

Members are encouraged, in chats and by tutorials, to look at their own work in the contexts of others' submissions and the instructor's remarks internally in their documents.  Students retain and may distribute their materials as proof of their abilities in SGML and the creation of interactive learning.  We provide references on students' request to potential employers and recommend members to companies seeking technical writers with these skills.

By the completion of the course each participant has experienced the major communications tools used in the creation and exchange of Web-based technical documents.  Each has worked with and understood the mark-up and conversion issues surrounding SGML, RTF and HTML displays.  Most have dealt with some of the requirements for full multi-media expression on CD-ROM, the Web or on Intranets for internal distribution.  This is 'Technical Writing' in a very current and complete sense and our students have been trained in it, individually and in groups, with all the resources our databases and course layout can provide.  In the near future we plan to add optional services in audio and video interchange,  XML document creation and Java authoring.

In effect, we have made a course in which the course materials and techniques are learned and used by participants even as they complete their writing assignments.  By the conclusion of the course many members have the full capacity to create SGML-based interactive projects for inter- and Intranet expressions. for their own and their employers' uses.  Most course materials have been available to the public at our web site and we continue to respond to inquiries and applications from individuals and companies on the Web.  At the time of writing we are preparing a commercial version of our work, with certificate status for participants and an extended range of topics related to online learning and information exchange, to launch in fall of 1998.

Getting here from nowhere:
How We Built the Present Model
In the fall of 1995, after a hectic eight months of development, primarily by students we offered the first completely Web-based course in technical writing to some 120 students anywhere in Canada registered through Waterloo.  The project spanned five time zones, over some 3,500 miles, from Newfoundland to Vancouver.  In the preceding months we'd designed and built the course data, communications structures, interface and software necessary for students to convert the SGML files of their five assignments directly into HTML documents for the Web and into RTF as Word files for print.

We launched the project knowing we had insufficient online help resources and we then supported individuals and groups at every stage of the assignment editing and submission process directly from email responses, phone calls and instructor handling of submissions files.  We knew the risks we took and we had been prepared from the outset to pay the price in both instructor overload and student frustrations that the launch process, as we had designed it, cost.

It was during this term that we compiled a series of questions in the form of an online evaluation. We planned to present the students with the evaluation once they had submitted final assignment.  However, the political process of obtaining approval to present the students with the online evaluation proved longer than expected.  By the time approval was received the course was long over.  Although we were somewhat upset by this lost opportunity, we realized that we had a working model that could be developed and improved for the next large section offering.

In September 1996, we offered a vastly modified course to a similar class size -- 125 students across Canada.  We had used the intervening months to improve the accuracy and scope of the database on technical writing and the instruction materials for users.  We still had huge problems handling, and helping students handle, their assignment submissions.  However, the changes we had made were efficacious as student morale and responses demonstrated. Nevertheless, the transfer and grading processes for assignments still exacted an unacceptable price, in contact hours for repair and completion of student submissions, from the instructors and markers.  We spent the next eight months of 1997 addressing the registration-privacy-submission and conversion problems, with minimal efforts devoted to information and content issues which, our users now informed us, they deemed to be stable and complete.

Evolving the Evaluation Process:
Emerging Standards for Student Response
The evaluation process also underwent a major change from the previous fall.  Instead of the original, single evaluation, the questions were broken out into four evaluations.  Each contained a set of "constant" questions, allowing us to categorize the responses into groups; for example, students identified themselves as 'on-' or 'off-campus', working, program year, etc.

Each evaluation focuses on a different aspect of the course. The first evaluation determined the student's education level and computer experience. It also sought to determine the student's perception of on-campus/off-campus advantages at the start of the course.  The second evaluation dealt with methods of communication used in the course: Webchat, email, the newsgroup and the "Instructor's Comments". The third evaluation also dealt with communications, but from a different perspective. It asked about their actual environments for using the course --when, where, what limits and problems, what advantages, when did they chat, and with whom, to what effect? We asked students how they felt they had developed in social, as well as technical, ways. Did they start to avoid certain communications? Did Webchat evolve into a social game as they began to use it?  What subtle, social issues emerged? The fourth evaluation was a comprehensive view, covering aspects of the course from beginning to end.

Brian edited the raw data from each evaluation into a Web page for general viewing by the students, instructors, and support team.  All course participants saw the results, commented on them, among themselves and in the newsgroups, and made suggestions to enable the course planners to incorporate good ideas almost immediately into the course structure.  The support team received indications of problems of which they normally were not aware. The course instructors received comments on various assignments while the course was still in progress. The students received instruction by seeing how the class in general felt about various issues and they saw their own views reflected in the context of the group and the course overall.

The number of responses for the evaluations varied: 38 for the first evaluation; 41 for the second; 16 for the third; 28 for the fourth. In sum, we felt the number of responses was encouraging.  There was a drop in the number of responses for the third evaluation and this may have been because we asked too many questions, too often, at a time when members were busy.  It may also have been that participants were satisfied with both the course progress and the amount of influence they saw they were having in its evolution.

While the evaluation process, along with the accompanying feedback, provided the perception that the instructors and support staff were indeed listening and responding to the students concerns, we conducted no further analysis of the raw data at that time.  The next step was to incorporate the results into a system where we could analyse it and where we could recognize trends.

In the fall of 1997, because of demands in other areas and strain on our resources, we had to test our new registration-submission software, the Course Administration Tool, directly on the entire class, while their work was in progress.  Instructors had to give simultaneous support for the old and new submissions procedures running concurrently.  Even under these stressful conditions, the integrated courseware performed at a level far above previous course offerings.  Students experienced early successes at each stage of the preparation and entering of assignments and the instructors experienced relief from a myriad of complex, detailed questions by naïve, frustrated users.  We were able to save our bacon, in effect, by handling, on an individual basis, all the technical problems our converter and distribution software had been  designed to do automatically.  Our instructor-student ratios remained unacceptably high and the dominant evaluation question "How much does this cost?" still had an unacceptable answer.

We resolved this question, in 1997 in a Canadian academic environment, with students providing their own machines and Web access. It is $350 - $400 per student, per 4-month course of about 150 hours of student involvement.  This does not include development and marketing costs both of the course materials and the courseware tools.  It does include all instructor time for answering email, handling the news and chat groups, marking all assignments and providing support for tutorials.  We include some personal online sessions and help in the forming of the 3-person groups, the administration of the marks and final certification procedures. In addition, we provide technical support for our servers and software, some advice on communications, editing and conversion strategies.  In effect, we believe this to be the amount to cover the operation of a fully developed course.  It does not include hardware and communications, software development and the upgrades necessary to incorporate and describe new Web-based software and tools.

By December 1997 we had achieved our objectives: a comprehensive, cost-efficient, interactive course which provided learning for users in the models of technical writing and the process of SGML document construction.  Our participants experienced self-directed learning, across all the resources of the Web, the conditions of remote development of group projects, using online communications tools, and the effectiveness of electronic marking and tutorials for their assignments.

Making Evaluations into Research: Analysis of the Online Models
The on-line evaluation project also evolved. We conducted a review of the materials to make sure we were asking appropriate questions in the correct manner and we were able to apply validity to the process overall.  Doctor Schellenberg devised a technique after the fact to format the data in such a way that it could easily be entered into SPSS where the information could be more closely analyzed.

The inclusion of a pseudonym in each of the evaluations also gave us the ability to track a student from one evaluation to another without compromising his or her identity.  For the most part this worked well, with the few exceptions where students forgot their original pseudonym. 

Time constraints obliged us to combine the first and second evaluations.  This resulted in 57 questions to the students.  The final evaluation contained only 30 questions. We employed considerable coaxing in the form of repeated, personal emails to get students to respond to this last evaluation. We received 56 responses to the first evaluation and 69 responses to the second, a slightly lower than participation rate than we expected, considering the class size of 123.  By asking so many questions, we may have reduced the number of participation.  Students may regard it as simply "too much work to respond" since evaluations have no effect on their course marks and it is obvious that most student issues are addressed and members' concerns are heard and acted upon. Some may well have felt that they were represented in the broad base of views put forward by their fellows. We have had no indications of further evaluation needs from the class by email requests or inquiries.

Top Level Analysis: What We Found
The evaluations are composed of five parts:
1. "Background information", about the students' profiles, academic levels and programs
2. Questions about users' perceptions of the course, ease of access to the materials, value of content, and  communication choices.
3. Work Group questions focusing on how they viewed their contribution and the contribution of others to the 50% of shared assignment work.
4. Assignment questions providing students' perceptions of the relationships and value of assignments and marking procedures to their understanding of writing concepts.
5. General questions such as the overall workload of the course, hours committed the value of work groups and time spent in private versus group activities.
We were able to make several observations from the basic SPSS data.  From the "background information" section of the evaluation process we discovered that there was an even balance between the number of arts/humanities students and science/technology students.  The data also revealed that 38.2% of the students had no previous online chat experience. 
Online communications rates among the most pleasant of our surprises.  We were aware of the general fears expressed by naysayers that electronic learning would stunt human communications, dumb down social interchanges to machine-derived comments and make for a boring or depersonalized learning experience. 

Quite the opposite occurred.  Class members worked on assignments in relative privacy, using the entire range of course materials conveniently, whenever they needed.  They contacted each other confidently, politely and often via email.  They exchanged nested and attached materials as needed, viewed each others' work in an open and congenial way and generally conducted themselves, across five time zones, like professionals.  They found information easily and quickly among FAQs and in the newsgroups, identified specialists and topic leaders for general assistance -- and they also played.   Many continue their academic and social contacts across terms and some dozen have furthered their studies in our specialized section of advanced training in the subsequent terms.  Four now develop materials in the new course structure as part of the instructor group.

Results showed us that initially 35.2% had never used newsgroups; 83.6% had never used Standard Generalized Markup Language, (SGML), which is a requirement to complete all assignments in the course.  40.7% of the class had no prior programming experience-a somewhat predictable number considering the balance of arts and technology students.

The evaluation questions "about the course" also offered insight into student views of the pedagogy as participants.  On our seven-point scale, from 1,"very poor", to 7, "excellent", 25% of the students regarded the set-up instructions for the course as "excellent", while only 6.3 percent felt the instructions were 2 "poor". 35.4% gave a 6 on the scale; 22.9% gave a 5; 4.2% gave a 4; 6.3% gave a 3.  No rating of "very poor" was used by any of the 125 participants.

When asked about the ease of access to the WWW course material, 76.4% of the class gave a rating of either 1 (very easy) or 2 (easy).  Only 3.6% rated the access as 7 (very hard) or 6 (hard).  This is a major comment on the ease and general availability of Web access in just two years from our 1995 fall offering.  Then we found that a major component of our work was support for users' connection and familiarization with our basic communications and editing tools.  We have moved from an initial student installation requirement of nine pieces of software to only three for all course functions in those two years.  Moreover, these programs have become both easier to install and more effective to use in each new expression of the products.

The questions dealing with students' communication choices showed the following:  92.6% of the students read the newsgroup at least once a week -- 20.4% read the newsgroup more than 5 times a week.  96.4% of the students read the "Instructor's Comments" at least once a week; of that, 40% read them more than 5 times a week.  It is obvious that online communications between student/instructor instructor/students and students/students played a major role in the course. 

Often, students would respond to another students' queries in a news item before the instructor had a chance to reply.  The reply was then available for all other students to view and it eliminated the need for the instructor to reply individually to students with the same problem.

For the second and third assignments students formed groups of three and worked together to complete the task for a shared grade.  The method of group formation was based on the results of their first assignment in which they created a resume and proposal letter.  Each student could freely view all other students' resumes and letters, and decide if he or she wanted that person as a member of their group.  The selection process varied widely as every member sought to find two complimentary people with whom to complete one half of the course.  The ways in which they solved the important and complex group formation questions show a great deal about how workers  interact online.

Here are some of those activities and conclusions. When asked if they read the "class names list" in the group selection process, 40% read all the names, and 20% read most of the names.  Only 3.6% did not read the list at all.  Of the student who did read the list, 69% did not recognize any of the names, and 25% recognized between one and four other students.  We anticipated the high number of students who did not recognize any students, since 69% were distance education students at a remove from the University and each other.

89% of students used email (as one would expect) to contact their peers and to form the three-member groups, followed by face-to-face contacts (3.6%, mostly on campus), and phone (7.3%).  From the various contacts, 27.3% of the students accepted the first offer to join a group, while 14% turned down one offer, 36.4% turned down two offers, and 21.8% turned down three or more offers before joining a group.  The majority of student found the experience of forming groups to be very interesting, as well as being fair.

Evaluation on the fly: Short-term, quick responses in the field
Throughout the design and construction of the online course, we have attempted kinds of user evaluation at each stage.  Some of these have been subjective, casual and inconsistent, used for smaller issues and short-term needs.  Others were based in the experiences of the course developer-instructors, many of whom were former members of the class.  Some of our assessments were short-term and specific, the need to verify the effectiveness of a part of the technology itself.  A University staff member developed our initial course-wide survey while she was yet a student in the course.

Conclusions and a Sense of Vindication
This project has had something of the sense of a grand experiment about it.  Using undergraduate expertise and academic credits, we designed a course which parallels many of the experiences a technical writer undergoes in a first work situation.  Even the major metaphors of an office, bookshelves and project work groups simulate working conditions for our University's Co-op students who make up a majority of the course membership.  The involvement of professionals in several commercial writing departments as tutors and advisors adds to this authenticity, as do the large numbers of employees participating to upgrade their skills and acquire new technical experience.  The assignments also follow a pattern of seeking a job through resumes, working in a group to create online documentation for software and creating a multimedia report which analyses their experiences of new technology over a protracted period.  Course contacts remain for many an added bonus in new academic and work situations where a former partner can provide information and insights in a new situation by email.

Our evaluation procedures form a part of this comprehensive set of services by showing users how the managers of an online learning environment can incorporate individuals' ideas and the group's opinions into major changes in course design and operation, sometimes within days of the assessment of a survey.

The most important priority for future evaluations is to increase the participation rate to incorporate all members' opinions.  This is necessary primarily because non-participants can be anticipated to reflect less enthusiasm for the course overall, and that impression must be captured if we hope to improve those aspects of the experience which their silence reflects.

As the amount of data from each of these evaluations increases with each new offering, we will start to see trends develop, and we will be able to modify the course in ways to anticipate and reflect them.  Getting the entire course design right, particularly in the context of the operating the course was an act of courage or folly three years ago.  Something of the openness we required of our students, as they made their assignments available for all to read, has been captured now in the candor -- and trust -- they show in their considerations of the exercise in thinking they undergo.  This is an encouragement all round.  We, as instructors, feel we are working with thorough, current information which our learners provide because they feel we care and that we use it to provide a more effective experience for them.  We test all areas of the course and share the results of that assessment by talking openly about our experiences and what we can do to improve them for everyone.  Brian Cameron's synthesis and posting of each evaluation as it comes in, with our comments and the actions we take to optimize activities from that information add a dimension students have not encountered in other classes.

Our plans now include Kathy Schellenberg's continuing detailed analysis of our results, with a posting of these into the commencement of our September sections.  This will be a way of saying "Welcome" and letting the neophytes see they are valued and effective from the outset.  We hope in this way to begin to collect earlier data than in our previous offerings and simultaneously to provide a fuller explanation of what we expect, but of what their predecessors felt and found.

Evaluations, in the form of student responses, have also become a central focus of our development of course materials.  As it is possible to make electronic expressions interactive, so too is it possible to make learning more immediate to users, by incorporating their views, opinions and activities into the conduct of the course.  They take an increasing responsibility and exert a stronger effect on what they learn.  Linguists know that language study is descriptive, not prescriptive.  By analogy we hope to begin to show our fellow learners that the way they see technical writing influences what technical writing is as an experience and it is possible for a group to influence itself by its work and by its assessments of that work.

This is the next stage of our research, to have user evaluations and user performance demonstrate to this same audience our students, how they perform as writers in their assignments and how their views influence that activity of writing. All members undertake, as part of their third assignment, a group exercise in usability through which they devise an objective test for their own documents.  It is at this point that the scales fall off, when they come to see what makes a technical writer's focus -- the audience whose experiences of their work they witness for the first time on the other side of their egos. 

We believe we may now have a way to have students see clearly the results of their class evaluations and the ways in which their perceptions influence what they expect to learn. From this point they begin their understanding of the online experience.  So we plan to add an early explanation of the evaluation objects and processes with previous examples of class views of the course.  We will then invite members to respond to a general outline of learning objectives, including what they expect to bring to the course in motivation, commitment and expectations. This self-evaluation will form the base for their assessments of their subsequent performance and their emerging sense of their development as technical writers.

About the authors
Dr. Paul Beam
Department of English
The University of Waterloo
pdbeam@watarts.uwaterloo.ca

Brian Cameron
Manager of Technical Support
Information Systems & Technology
The University of Waterloo
hesse@ist.uwaterloo.ca