Tolentine Hall, Room 103
800 Lancaster Avenue
Villanova, PA 19085
There are probably more studies of student ratings than of all of the other data used to evaluate college teaching combined. Although one can find individual studies that support almost any conclusion, for a number of variables there are enough studies to discern trends. In general, student ratings tend to be statistically reliable, valid, and relatively free from bias, probably more so than any other data used for faculty evaluation. Nevertheless, student ratings are only one source of data about teaching and must be used in combination with multiple sources of data if one wishes to make a judgment of all of the components of college teaching. Further, student ratings are data that must be interpreted. We should not confuse a source of data with the evaluators who use the data to make a judgment. Cashin, Idea Paper no. 20, 1988, see Cashin 1995 for updated information.
--Office of Vice President for Academic Affairs, June, 2006
You will soon be receiving the packets with the materials for administering Villanova's Course And Teacher Survey (CATS). This guide is intended to answer some of the questions that faculty members have about student surveys. Your chair or program director can also advise you about CATS. If you have additional suggestions or questions about either this guide or the CATS process, please contact Dr. Randy Weinstein, Associate Vice Provost for Teaching and Learning, 610-519-5858.
Generally, we feel that student perceptions are a useful component of understanding teaching effectiveness. The CATS results can give you as a faculty member an additional source of information about how students perceived your classes. They can also give useful information to your chair who can, in turn, make useful suggestions for improving your teaching effectiveness. Ultimately the CATS results are used as one component of the overall evaluation of your teaching, which will be conducted by your colleagues, and your chair. It should be stressed, however, that the CATS results are intended to be a component of this process. Some people inaccurately describe the CATS surveys as "student evaluations." This is incorrect. At Villanova, only faculty and administrators evaluate teaching effectiveness. While student perceptions are taken into account in the process of forming those evaluations, the evaluations are made by faculty members and administrators, not by students.
The current policy calls for administration of CATS in most courses every semester during the fall and spring semesters. Classes with fewer than three students are not surveyed, and sometimes department chairs will determine that certain classes should not be surveyed. CATS surveys are not done in the summer unless specially requested by deans.
When you receive the packet, you will see detailed instructions to you as to how to administer the surveys. Please read them carefully. The general idea is that they should be administered in the last two weeks of the semester, and at the beginning rather than at the end of the class. You should allow the students ten or fifteen minutes (more if there are supplemental questions) to complete the surveys, you should not be in the room when the surveys are being filled out, and you should ask a student to collect them, put them in a sealed envelope and return them to your departmental office. At that point the surveys go directly to the Office of Planning and Institution Research (OPIR) where they are processed.
CATS is intended to be a minimum survey, and it is assumed that faculty, departments, and colleges may want to add additional questions. Click here to see a copy of the instrument itself. As you will see, the form includes space for additional questions and for open-ended responses.
There are opportunities for 26 user defined questions on the reverse side of the instrument, lettered from A to Z. You are free to prepare your own additional questions. Some faculty members ask students to comment on the value of individual assignments or texts, or on other specific aspects of their course. These questions could also be open-ended, with students writing comments on a separate sheet. You may, if you wish, send your additional questions to OPIR (by November 1 for the fall or by March 15 for the spring) and they will put your questions in the packet. If you prefer, you may just give your own questions out on the day you administer the instrument. Tell the students to answer the questions on your sheet in the spaces provided on the back of the CATS form. When you get the results back, you can correlate the tabulated responses with the questions that you asked. Your college or department may also have supplemental questions, so please check this with your chair before you start making up your own questions.
The relationship between student ratings and grades has been extensively studied by social scientists, and there is little evidence for the common view that students trade high student ratings for high grades or punish instructors with high standards. Indeed, the results of Villanova's study completed in 2011 are reported below and indicate grading has an inconsequential impact on CATS scores. At Villanova, some of the most lenient graders get low scores and some of the most demanding teachers get high scores. On the other hand, students do not respond well to grading practices that they perceive as arbitrary and unfair, and they will definitely express their resentment about what they perceive as unfair grading practices when it comes time to fill out the CATS forms. Rather than trying to give high grades, new faculty members should focus on giving students greater clarity about expectations. One experienced faculty member put it this way: "Students don't expect to get all As and Bs, but they would like to think that courses are arranged so that a student who put the effort into it could get a top grade. They hate courses where it seems impossible to do well, or where they don't really understand what is expected of them."
A month or two after the end of the semester, you'll get an e-mail telling you that the CATS results are available on Novasis (the same system where you post grades). You can view your CATS through the "Faculty CATS Report" link in the Advising Tools portal under th Faculty tab in MyNOVA. The Novasis reports are only available to you, and require the same password system that is used in entering grades. A few weeks after that, the filled out forms themselves (including the results to the open-ended questions) and printed reports will be sent to your chair. The chair will review the forms and then return them to you. Unfortunately, it will be a number of weeks before you get to see the open-ended material.
The CATS reports go to you, your chair, your dean, and to the VPAA. Only you and your chair will see the results of the open-ended questions. New full-time faculty members go through a formal process of evaluation once a year, and teaching is a big part of that evaluation. Typically chairs have a faculty committee to recommend on the evaluation, and those committees usually have access to CATS reports.
According to a policy passed by the University Senate, the CATS reports may, with the permission of the faculty member, be published on-line. When you see the NOVASIS version of your reports, you are given the option to make the reports available on-line by selecting "Yes" under the "CATS on Novasis" column. If you do so, the reports will be available to Villanova students, staff, and faculty who have valid NOVASIS usernames and PINs. Some students review the published CATS results in deciding which classes to take. The decision to publish your CATS scores is purely up to you, and you may change your decision as often as you want. The Office of Academic Affairs encourages you to make your CATS results public so students can use the information to learn about courses and faculty. The CATS offer our students a comprehensive view superior to the other sources they often use to learn about our courses.
After you get your results back, you might want to discuss the results with your chair or program director. The chair can tell you how your CATS surveys will be regarded by your department. Often new faculty members remark that the results they get in the first semester or two are lower than what they saw at their previous institution. After a few semesters, the results frequently improve.
Your chair can tell you what information is used by your department in evaluating teaching effectiveness. Usually departments look at things such as syllabi, tests, peer observations, and grading patterns. Realistically, chairs and your faculty colleagues also draw on a wide range of information when they come to think about teaching effectiveness. Especially at registration time, chairs spend a great deal of time talking to students about courses, and they hear a lot about student perception of teaching effectiveness. The same is true of other colleagues. The result is that when it comes time to do a formal evaluation of teaching, a lot of factors are taken into account, not just CATS reports.
New faculty members should concentrate on teaching effectiveness rather than student survey results. The question they should ask themselves is: "What can I do to be a more effective teacher?" Trust us, the ratings will take care of themselves. There are a number of strategies for improving teaching effectiveness.
Having said that, we have analyzed the CATS results from a large sample of students and we can make a few generalizations about how to improve ratings. This is based on a pretest consisting of about 800 students.
Improving scores on overall teaching effectiveness. There are six factors that are, in the perception of a large number of Villanova students, most closely associated with the overall rating on teaching. The results suggest that improvements in the students' perception of these factors will have the greatest impact on how they assess your overall performance as a teacher. For those who are interested in the statistics, we list them with numbers attached, but the general idea is that the factors are listed in order of relevance. Remember, that we are discussing student perceptions here, so that it is not enough, for example, for you to organize and plan the course effectively from your own perspective. Students must also perceive that the course is well organized and planned. Sometimes it is helpful to spend a little bit of time helping students perceive things that you yourself may be aware of.
Explanation of variance of responses of overall evaluation of quality of instruction (question 28)
|Question||Percentage of variation explained by adding this question (R square)||Correlation with question 28|
|17||Explains course material clearly||.540||.735|
|11||Interacts effectively with students||.615||.606|
|26||I learned a great deal in this course||.649||.629|
|8||Organizes and plans the course effectively||.661||.631|
|10||Makes goals of the course clear||.667||.595|
|13||Is available for help outside the classroom||.673||.474|
Improving scores on the course itself. The results also give some suggestions about which factors students associate with a high overall score on the value of the course itself. Some of these factors overlap with those related to high effectiveness on the part of the teacher, others are different. Again, improvements in these factors are likely to generate higher scores on the overall rating of the course.
Explanation of variance of responses of overall evaluation of value of the course (question 29)
|Question||Percentage of variation explained by adding this question (R square)||Correlation with question 29|
|26||I learned a great deal||.562||.750|
|23||I found the course intellectually stimulating||.597||.701|
|13||Instructor is available for help outside of class||.611||.406|
|21||Employs test and graded materials relevant to course content||.622||.521|
|10||Makes goals of the course clear||.633||.155|
In most departments, salary recommendations are based on the annual evaluations, and CATS data are a component in forming the evaluations of teaching.
The CATS reports figure into Rank and Tenure decisions in two ways. First, the Rank and Tenure committees will look at the past departmental annual evaluations. In so far as CATS reports are a component of past teaching evaluations, they influence this aspect of Rank and Tenure. The Rank and Tenure committees also look at a summary report of the applicant's CATS scores. Generally, the committee only looks at the results for questions 22, 23, 26, 28 and 29.
A committee of faculty and administrators designed the CATS form and circulated the draft to the entire faculty for comments and revisions. Villanova began using the CATS in 1997. Another committee reviewed and revised the CATS form in 2003-2004. Villanova began using the new CATS form in the fall semester of 2004.
In using and interpreting student ratings, committees, department chairs and other administrators should be guided by these principles:
In addition to the principles listed above, administrators and peer committees are required to follow the recommendations listed in the next section summarizing Villanova’s study examining the effect demographic and situational variables have on CATS scores.