CATS: A Guide for Faculty Members

Course and Teacher Surveys (CATS) at Villanova University

There are probably more studies of student ratings than of all of the other data used to evaluate college teaching combined. Although one can find individual studies that support almost any conclusion, for a number of variables there are enough studies to discern trends. In general, student ratings tend to be statistically reliable, valid, and relatively free from bias, probably more so than any other data used for faculty evaluation. Nevertheless, student ratings are only one source of data about teaching and must be used in combination with multiple sources of data if one wishes to make a judgment of all of the components of college teaching. Further, student ratings are data that must be interpreted. We should not confuse a source of data with the evaluators who use the data to make a judgment. Cashin, Idea Paper no. 20, 1988, see Cashin 1995 for updated information.

--Office of Vice President for Academic Affairs, June, 2006

You will soon be receiving the packets with the materials for administering Villanova's Course And Teacher Survey (CATS). This guide is intended to answer some of the questions that faculty members have about student surveys. Your chair or program director can also advise you about CATS. If you have additional suggestions or questions about either this guide or the CATS process, please contact Dr. Randy Weinstein, Associate Vice Provost for Teaching and Learning, 610-519-5858.

Why does Villanova survey students about teaching effectiveness?

Generally, we feel that student perceptions are a useful component of understanding teaching effectiveness. The CATS results can give you as a faculty member an additional source of information about how students perceived your classes. They can also give useful information to your chair who can, in turn, make useful suggestions for improving your teaching effectiveness. Ultimately the CATS results are used as one component of the overall evaluation of your teaching, which will be conducted by your colleagues, and your chair. It should be stressed, however, that the CATS results are intended to be a component of this process. Some people inaccurately describe the CATS surveys as "student evaluations." This is incorrect. At Villanova, only faculty and administrators evaluate teaching effectiveness. While student perceptions are taken into account in the process of forming those evaluations, the evaluations are made by faculty members and administrators, not by students.

Back to Top

How often and in which classes are the surveys conducted?

The current policy calls for administration of CATS in most courses every semester during the fall and spring semesters. Classes with fewer than three students are not surveyed, and sometimes department chairs will determine that certain classes should not be surveyed. CATS surveys are not done in the summer unless specially requested by deans.

Back to Top

How are the surveys administered?

When you receive the packet, you will see detailed instructions to you as to how to administer the surveys. Please read them carefully. The general idea is that they should be administered in the last two weeks of the semester, and at the beginning rather than at the end of the class. You should allow the students ten or fifteen minutes (more if there are supplemental questions) to complete the surveys, you should not be in the room when the surveys are being filled out, and you should ask a student to collect them, put them in a sealed envelope and return them to your departmental office. At that point the surveys go directly to the Office of Planning and Institution Research (OPIR) where they are processed.

Back to Top

What questions are asked?

CATS is intended to be a minimum survey, and it is assumed that faculty, departments, and colleges may want to add additional questions. Click here to see a copy of the instrument itself. As you will see, the form includes space for additional questions and for open-ended responses.

Back to Top

May I add questions of my own?

There are opportunities for 26 user defined questions on the reverse side of the instrument, lettered from A to Z. You are free to prepare your own additional questions. Some faculty members ask students to comment on the value of individual assignments or texts, or on other specific aspects of their course. These questions could also be open-ended, with students writing comments on a separate sheet. You may, if you wish, send your additional questions to OPIR (by November 1 for the fall or by March 15 for the spring) and they will put your questions in the packet. If you prefer, you may just give your own questions out on the day you administer the instrument. Tell the students to answer the questions on your sheet in the spaces provided on the back of the CATS form. When you get the results back, you can correlate the tabulated responses with the questions that you asked. Your college or department may also have supplemental questions, so please check this with your chair before you start making up your own questions.

Back to Top

What is the relationship between the grades I give students and the ratings they give me on the CATS forms?

The relationship between student ratings and grades has been extensively studied by social scientists, and there is little evidence for the common view that students trade high student ratings for high grades or punish instructors with high standards. Indeed, the results of Villanova's study completed in 2011 are reported below and indicate grading has an inconsequential impact on CATS scores.  At Villanova, some of the most lenient graders get low scores and some of the most demanding teachers get high scores. On the other hand, students do not respond well to grading practices that they perceive as arbitrary and unfair, and they will definitely express their resentment about what they perceive as unfair grading practices when it comes time to fill out the CATS forms. Rather than trying to give high grades, new faculty members should focus on giving students greater clarity about expectations. One experienced faculty member put it this way: "Students don't expect to get all As and Bs, but they would like to think that courses are arranged so that a student who put the effort into it could get a top grade. They hate courses where it seems impossible to do well, or where they don't really understand what is expected of them."

Back to Top

When will I get the results?

A month or two after the end of the semester, you'll get an e-mail telling you that the CATS results are available on Novasis (the same system where you post grades). You can view your CATS through the "Faculty CATS Report" link in the Advising Tools portal under th Faculty tab in MyNOVA. The Novasis reports are only available to you, and require the same password system that is used in entering grades. A few weeks after that, the filled out forms themselves (including the results to the open-ended questions) and printed reports will be sent to your chair. The chair will review the forms and then return them to you. Unfortunately, it will be a number of weeks before you get to see the open-ended material.

Back to Top

Who else will see the results?

The CATS reports go to you, your chair, your dean, and to the VPAA. Only you and your chair will see the results of the open-ended questions. New full-time faculty members go through a formal process of evaluation once a year, and teaching is a big part of that evaluation. Typically chairs have a faculty committee to recommend on the evaluation, and those committees usually have access to CATS reports.

Back to Top

Will the results be available on the world wide web?

According to a policy passed by the University Senate, the CATS reports may, with the permission of the faculty member, be published on-line. When you see the NOVASIS version of your reports, you are given the option to make the reports available on-line by selecting "Yes" under the "CATS on Novasis" column.  If you do so, the reports will be available to Villanova students, staff, and faculty who have valid NOVASIS usernames and PINs. Some students review the published CATS results in deciding which classes to take. The decision to publish your CATS scores is purely up to you, and you may change your decision as often as you want.  The Office of Academic Affairs encourages you to make your CATS results public so students can use the information to learn about courses and faculty.  The CATS offer our students a comprehensive view superior to the other sources they often use to learn about our courses.

Back to Top

When should I consult with my chair?

After you get your results back, you might want to discuss the results with your chair or program director. The chair can tell you how your CATS surveys will be regarded by your department. Often new faculty members remark that the results they get in the first semester or two are lower than what they saw at their previous institution. After a few semesters, the results frequently improve.

Back to Top

What other information will be considered when my teaching is reviewed?

Your chair can tell you what information is used by your department in evaluating teaching effectiveness. Usually departments look at things such as syllabi, tests, peer observations, and grading patterns. Realistically, chairs and your faculty colleagues also draw on a wide range of information when they come to think about teaching effectiveness. Especially at registration time, chairs spend a great deal of time talking to students about courses, and they hear a lot about student perception of teaching effectiveness. The same is true of other colleagues. The result is that when it comes time to do a formal evaluation of teaching, a lot of factors are taken into account, not just CATS reports.

Back to Top

What can I do to improve my ratings?

New faculty members should concentrate on teaching effectiveness rather than student survey results. The question they should ask themselves is: "What can I do to be a more effective teacher?" Trust us, the ratings will take care of themselves. There are a number of strategies for improving teaching effectiveness. 

  • One of the most useful steps is to draw on the resources of the Villanova Institute for Teaching and Learning (VITAL). VITAL offers a host of services ranging from confidential consultations to public workshops.
     
  • Generally speaking, teaching effectiveness does improve over the first few semesters, as faculty members get a clearer sense of student learning styles.
     
  • Faculty members may also want to ask advice of recognized good teachers in the department, and may ask permission to observe classes.

Having said that, we have analyzed the CATS results from a large sample of students and we can make a few generalizations about how to improve ratings. This is based on a pretest consisting of about 800 students.

Improving scores on overall teaching effectiveness. There are six factors that are, in the perception of a large number of Villanova students, most closely associated with the overall rating on teaching. The results suggest that improvements in the students' perception of these factors will have the greatest impact on how they assess your overall performance as a teacher. For those who are interested in the statistics, we list them with numbers attached, but the general idea is that the factors are listed in order of relevance. Remember, that we are discussing student perceptions here, so that it is not enough, for example, for you to organize and plan the course effectively from your own perspective. Students must also perceive that the course is well organized and planned. Sometimes it is helpful to spend a little bit of time helping students perceive things that you yourself may be aware of.

Back to Top

Explanation of variance of responses of overall evaluation of quality of instruction (question 28)

  Question Percentage of variation explained by adding this question (R square) Correlation with question 28
17 Explains course material clearly .540 .735
11 Interacts effectively with students .615 .606
26 I learned a great deal in this course .649 .629
8 Organizes and plans the course effectively .661 .631
10 Makes goals of the course clear .667 .595
13 Is available for help outside the classroom .673 .474

Improving scores on the course itself. The results also give some suggestions about which factors students associate with a high overall score on the value of the course itself. Some of these factors overlap with those related to high effectiveness on the part of the teacher, others are different. Again, improvements in these factors are likely to generate higher scores on the overall rating of the course.

Explanation of variance of responses of overall evaluation of value of the course (question 29)

  Question Percentage of variation explained by adding this question (R square) Correlation with question 29
26 I learned a great deal .562 .750
23 I found the course intellectually stimulating .597 .701
13 Instructor is available for help outside of class .611 .406
21 Employs test and graded materials relevant to course content .622 .521
10 Makes goals of the course clear .633 .155

Will the CATS surveys be used in determining salary?

In most departments, salary recommendations are based on the annual evaluations, and CATS data are a component in forming the evaluations of teaching.

Back to Top

Will the results be used in decisions regarding rank and tenure?

The CATS reports figure into Rank and Tenure decisions in two ways. First, the Rank and Tenure committees will look at the past departmental annual evaluations. In so far as CATS reports are a component of past teaching evaluations, they influence this aspect of Rank and Tenure. The Rank and Tenure committees also look at a summary report of the applicant's CATS scores. Generally, the committee only looks at the results for questions 22, 23, 26, 28 and 29.

Back to Top

Who designed the instrument, can it be changed?

A committee of faculty and administrators designed the CATS form and circulated the draft to the entire faculty for comments and revisions.  Villanova began using the CATS in 1997.  Another committee reviewed and revised the CATS form in 2003-2004.  Villanova began using the new CATS form in the fall semester of 2004.

Back to Top

General Principles for the use of student ratings for faculty evaluation and development: (Issued November 25, 1996)

In using and interpreting student ratings, committees, department chairs and other administrators should be guided by these principles:

  1. Student ratings are a valuable source of data about effective teaching. (Research shows that they are highly reliable and stable, and are relatively valid against a variety of indicators).
  2. Student ratings must be supplemented with other sources of information about teaching, such as: syllabi; tests; grade distributions; faculty self-evaluations; and reports of class observations by colleagues.
  3. Student ratings are only one tool for helping to improve teaching. Ratings can be a valuable source of information that aids in instructional improvement. They may help to identify particular aspects of a course that merit attention (e.g., test and exams, organization, student interaction, etc.). However, while serving this "diagnostic" function, standard rating forms are not very "prescriptive" in terms of specific suggestions to the instructor for making adjustments. In this regard, the role of one's peers or chairperson, confidential interviews with students, responses to open-ended questions, and teaching workshops are probably more valuable. Faculty members should be encouraged to do their own confidential surveys during the course of the semester.
  4. Administrators and peer committees who rely on student ratings have a responsibility to be educated in the interpretation of these data. Extensive research has been done on student ratings and those who use these data should be aware of the strengths and limitations of student ratings. Among other things, the research suggests the following principles:
  • Standardized procedures should be carefully developed and scrupulously followed for the administration of student rating forms. For example, the forms should be anonymous, the instructor should not be in the room when the forms are administered; the forms should be distributed toward the end of the semester; and students should not be unduly rushed when completing them.
  • When using student ratings for evaluative purposes, look primarily at data that include multiple classes or sections across several semesters, and from classes that have at least 10 raters and two-thirds of the class present. Data on a few classes or a few students are likely to be unreliable.
  • Look at the results in terms of relevant comparisons; the absolute numbers in student ratings are of only limited value. Department chairs should furnish faculty members with reports that allow faculty members to compare their own results to that of the department as a whole.
  • When using student ratings for evaluative purposes, focus primarily on "global" evaluative items about overall instructor performance and overall course satisfaction.  While these questions are appropriate for evaluative purposes, they are of much less values for developmental purposes.  Chairs and others who work with faculty members to help them improve their teaching should rely primarily on the more diagnostic questions, such as questions 8-21 on the CATS form.
  • Allow faculty to submit comments on their student evaluations, and take into account special factors (e.g., whether the course is a new or redesigned course or is team-taught). 

In addition to the principles listed above, administrators and peer committees are required to follow the recommendations listed in the next section summarizing Villanova’s study examining the effect demographic and situational variables have on CATS scores.

Back to Top

The Effect of Demographic and Situational Variables on Undergraduate CATS Scores (Final Report June 2012)

In 2010-2011, the Academic Policy Committee (APC), the Office of Planning and Institutional Research (OPIR) and the Office of Academic Affairs (OAA) partnered to complete a study investigating the effect of demographic and situational variables on CATS scores in undergraduate courses.   APC and OPIR chose to assess the effects of 13 variables that were either suggested by Villanova faculty or have been frequently investigated in the extensive published literature on student surveys of faculty teaching.

OPIR contracted with Dr. Luke Keele, an Associate Professor of Political Science at Pennsylvania State University, and an expert in regression and semi-parametric analyses, to perform a “hierarchical linear modeling” (HLM) form of regression analysis using five years of CATS data (2005-2010).  This dataset consisted of about 330,000 completed CATS forms from approximately 17,000 undergraduate course sections.  Use of HLM regression allowed for the “nesting” of faculty in their departments for comparison purposes.

Dependent Variable

Question 28 (Q28) “Rate the Overall Quality of the Instructor” is featured as the dependent variable.  The Departmental average score on Q28 ranges from 3.90 to 4.78 with a university wide average of 4.31 and a standard deviation of 0.92.

Dr. Keele also used Question 29 “Rate the Overall Value of the Course” as a dependent variable and found the results consistent with the results for Question 28, so his report featured Question 28 only.

Independent Variables

APC asked Dr. Keele to test the influence of 13 variables on Q28:

  1. The student’s self-reported workload
  2. Class Size (under 30 or over 30 students)
  3. The student’s self-reported GPA
  4. The average grade given in the class
  5. The ratio of self-reported GPA to class GPA
  6. The student’s class rank (freshman, sophomore, junior or senior)
  7. The age of the faculty member
  8. The time of day when the course was scheduled
  9. The type of course (required or elective)
  10. The type of instructor (tenured, full-time non-tenured, part-time)
  11. The faculty member’s number of years of teaching experience at Villanova
  12. The gender of the student or faculty member
  13. The race and ethnicity of the faculty member

APC and OAA’s Summary of Key Findings and Recommendations for Interpreting the CATS

Based on Dr. Keele's analysis, the Office of Academic Affairs and the Academic Policy Committee strongly recommend:

  1. Faculty and administrators should always view CATS scores as only one indicator in the evaluation of faculty teaching and, recognize that these data are based on student perceptions.  Remember Villanova’s rationale for surveying students about teaching effectiveness:
  2.  "Generally, we feel that student perceptions are a useful component of understanding teaching effectiveness. The CATS results can give you as a faculty member an additional source of information about how students perceived your classes. They can also give useful information to your Chair who can, in turn, make useful suggestions for improving your teaching effectiveness. Ultimately the CATS results are used as one component of the overall evaluation of your teaching, which will be conducted by your colleagues and your Chair. It should be stressed, however, that the CATS results are intended to be a component of this process. Some people inaccurately describe the CATS surveys as "student evaluations." This is incorrect. At Villanova, only faculty and administrators evaluate teaching effectiveness. While student perceptions are taken into account in the process of forming those evaluations, the evaluations are made by faculty members and administrators, not by students."
  3.  In view of the substantive differences across academic departments, faculty members’ scores should be compared primarily to departmental benchmarks (e.g., average scores).
  4. Small differences, such as two tenths of a point, from departmental and college means on Questions 28 and 29 should be interpreted as inconsequential differences in order to appropriately mitigate the small differences attributed to some of the demographic and situational variables, especially the differences described in items 4 and 5 presented below in the “Detailed Results of the Statistical Analysis.”
  5. The class size, the rigor of grading, the time of day, the type of course, the class rank of the student and the student’s GPA do not have a consequential impact and therefore should not be emphasized when interpreting CATS scores on Questions 28 and 29.
  6. Faculty should not be concerned about asking students to spend 3 or more hours per week on course work outside of class, because higher workload is not associated with lower CATS scores on Questions 28 and 29.
  7. The age of the instructor, the number of years of teaching experience, and the type of full-time faculty member (tenure-track, tenured, or non-tenure track) do not have a consequential impact on CATS scores on Questions 28 and 29.  Department Chairs should work with all faculty members to develop effective teaching, especially part-time faculty members who have slightly lower CATS scores.

Detailed Results of the Statistical Analysis

he following eight variables had an inconsequential effect of 0.06 or less in either direction (higher or lower than the average score) on Q28:

  1. Class Size (under 30 or over 30 students)
  2. The student’s self-reported GPA
  3. The Student’s class rank (e.g., freshman, sophomore, junior or senior)
  4. The age of the faculty member
  5. Time of day class is taught (e.g., before 9:00 or before 10:00)
  6. The type of course (required or elective)
  7. The ratio of self-reported GPA to the class GPA
  8. The faculty member’s number of years of teaching experience (e.g., less than 15 years and more than 25 years)

The following five variables had small effects around 0.1 to 0.21 in either direction (higher or lower than the average score) on Q28:

  1. As students report working more than 1 to 2 hours per week outside of class, the CATS score increases by 0.13 for 3 to 4 hours, 0.11 for 5 to 6 hours, or 0.07 for 7 to 8 hours.
  2. A higher average grade in the course (e.g. A-) will increase the CATS score by 0.1 and lower average grade in the course (e.g. B-) will decrease the CATS score by 0.09.
  3. In comparison to tenured faculty members: the full-time, non-tenure track faculty members score 0.04 lower, the tenure-track faculty members score 0.07 lower, and part-time faculty score 0.21 lower.
  4.  Female faculty members have a lower CATS score than male faculty by 0.04.  Male students award slightly higher CATS scores than female students (a difference of 0.08).  All students tend to rate female faculty slightly lower than male faculty.
  5. African American and White faculty score the same.  Hispanic and Asian faculty score slightly lower; however, the gender differences within each group mainly explain the slightly lower scores.

10/5/12

Office of the Provost

Tolentine Hall, Room 103
800 Lancaster Avenue
Villanova, PA 19085
Phone: 610.519.4525
Fax: 610.519.6200

Meet Our Staff