As a culmination of our 2021-2022 ACAD Fellowship project, we recently held an interactive session during the 2023 ACAD conference to discuss the prevailing views of peer review of teaching (PRT) and potential biases associated with their use for faculty evaluation. The guiding question for our project was whether peer review of teaching—where typically tenured faculty members observe and evaluate untenured colleagues’ teaching and pedagogy—should be used as the gold standard. In response to increasing concern about biases in student evaluations, peer observation has emerged as the go-to practice for the evaluation of faculty teaching. Based on our individual and institutional experiences, we have been persuaded that biases can arise in peer reviews of teaching and our end goal is for academia—faculty, departments, and institutions—to focus their attention on how to mitigate these biases. In this article, we summarize the results of a survey on institutional approaches to peer observation of teaching and a scenario-based exercise that engaged audience members as part of our general session presentation at this year’s ACAD conference.
Background
A vast literature highlights potential biases in student evaluations of faculty members’ teaching in institutions of higher education (Basow, 1995; Basow & Martin, 2012; Basow & Silberg, 1987; Kreitzer & Sweet-Cushman, 2021). Notably, studies have shown more negative evaluations by students when instructors belong to minoritized groups based on gender or race/ethnicity, although biases in evaluations are likely to extend to other instructors with other minoritized identities such as gender identity and gender expression, sexual orientation, nationality, religious affiliation, etc. As a means to counter the influence of biases in student evaluations of teaching on faculty evaluation processes (e.g., tenure and promotion), institutions have been encouraged to incorporate peer review of teaching in the evaluation of faculty for tenure and promotion (Stark & Freishtat, 2014; Stewart & Valian, 2018). The underlying assumption regarding peer review of teaching is that it is likely to be less subject to biases than student evaluations because it relies on experienced and seasoned faculty to conduct the observations. Unfortunately, there are no data to support this assumption; indeed, anecdotal evidence suggests that untenured faculty from historically marginalized and minoritized groups—e.g., based on race/ethnicity, gender, sexual orientation, gender identity and their intersections—question the putative objective nature of how these peer observations of their teaching are conducted and/or utilized. Whereas colleges and universities have endorsed the influence of explicit and implicit biases that can influence faculty hiring, there has been little movement in considering the influence of such biases in the evaluation of faculty for tenure and promotion.
Today, peer review of teaching has become an important element of faculty tenure and promotion evaluation and numerous institutions have established procedures for the peer review of teaching. However, the assumption that peer review of teaching is free from biases has not been substantiated, and some institutions recognize this (e.g., University of South Carolina, Center for Academic Innovation and Faculty Support, “Peer Observation of Teaching,” 2020). Likewise, Jordan Troisi (2020) has summarized a number of possible biases that can influence peer review of teaching. These include power play and positionality; the “sage” bias; lack of awareness of innovative teaching practices; biases that operate in other contexts such as hiring; duplication/replication of one’s own teaching practices; and disciplinary differences.
Survey Project
For our ACAD Fellows Project, we conducted a study of how institutions approach peer observation of teaching in faculty evaluation (if they do) and sought input on any concerns about the way such observation is conducted and/or used. Specifically, our study goal was to understand better how peer review of teaching is structured, administered, and utilized in faculty review and evaluation. We developed a semi-structured survey questionnaire that included closed and open-ended questions that focused on institutional practices and policies related to peer review of faculty teaching. Content areas of the survey related to whether the peer observation of teaching practice exists at the institution; whether training is provided for such peer observation; whether guidelines exist for peer observation of teaching, and, where the responsibility rests for developing, approving, and communicating guidelines; the level of clarity and prevailing culture of the use of guidelines; and open-ended responses about any campus concerns surrounding peer observation of teaching and potential biases therein.
With indispensable support from ACAD Executive Director, Laura Best, we conducted an online survey on the practices and processes related to peer observation of teaching. After receiving approval from the Institutional Review Board of Lafayette College, ACAD members received an email invitation to participate in our survey; Laura Best also posted the survey link on ACAD’s listserv. After a 3-week data collection window in October 2022, we received 190 usable responses (44 clicks on the survey link were recorded but no responses were made).
Sample Description: Respondents’ roles varied within academic administration, including directors; assistant and associate deans; assistant, associate, and vice provosts; provosts/vice presidents for academic affairs and chief academic officers; department and division leaders; and vice presidents and a chancellor. Six respondents indicated that they were in acting or interim roles and seven respondents were faculty members (these were possibly former administrators, who are eligible for ACAD membership). Approximately two-thirds of the respondents (66.3%) were full-time administrators and 83% were in positions with responsibilities for advancing and supporting faculty in teaching enhancement.
In terms of the types of institutions represented in the survey, 64% identified their institution as a “4-year plus” institution and about 25% as “4-year only.” The remaining institutions were described as “2-year and 2-year plus.” The majority of institutions (60%) were private and about 25% were religiously affiliated. The total student enrollment among the surveyed institutions ranged from 460 to 100,000 students, with an average of 8,124 students; the total full-time undergraduate student population ranged from 175 to 75,000 students, with an average of 5,798 students.
Major Findings and Conclusions: Of the total 190 survey responses, 154 (81%) indicated that peer observation of teaching was conducted at their institutions. Of these 154, 105 (68%) said that their institutions had established guidelines for the practice of peer observation of teaching. This group was asked follow-up information; 6 respondents did not complete the remaining questions, resulting in a sample size of 99 respondents for the next several questions on the survey. In general, respondents considered the established guidelines to be clear (n=72)[1] but less than half (n=47) indicated the use of the established guidelines for the peer observation of teaching was mandatory. The development and approval of these guidelines generally involved some combination of a faculty committee, department committee, and/or the provost’s office and the guidelines were communicated mostly via the faculty handbook exclusively or in combination with the department or provost’s website. Twenty-two respondents indicated that there was no policy for revising the guidelines for peer observation of teaching. Of the remaining 77 respondents, 17 checked that there was a known periodic basis for revising the guidelines while 53 indicated that the policy for revising the guidelines was ad hoc in nature and 7 indicated that they did not know; of these 77, 21 respondents indicated that the guidelines were revised on a 1-5 year cycle, 47 checked that there was no particular cycle, and 2 did not know. In terms of institutional education or training for peer observation of teaching, 43 respondents indicated yes, 51 checked no, and 4 respondents did not answer the question.
We asked all 154 respondents who had indicated that the peer observation of faculty occurred at their institutions to share if there had been any concerns on their campus about potential biases in how such observation was conducted and utilized. In terms of how the observation is conducted, 56 said yes (36%), 90 checked no or don’t know (58%), and 8 did not respond to the question. Six themes emerged from the 56 respondents who expressed concerns about potential bias in their PRT process. The themes were forms of bias (“Racial and gender bias, disciplinary bias”), inconsistent procedures and/or guidelines (“Without established guidelines and instruments consistent across the entire university, biases have become apparent”), lack of training (“Those doing reviews are not trained and evaluate on their own experience, not evidence-based principles”), too subjective (“Occasionally, process thought to be too subjective, which might open the door for biases”), defining peers in the process (“The “peer” reviews are conducted by the Department Chairs”), and general awareness about bias (“We’ve acknowledged that biases exist in student and peer observations, but we haven’t yet arrived at ways to mitigate bias”).
With regard to how the guidelines are used, 49 responded yes (32%), 97 checked no or don’t know (63%), and 8 did not respond. Five themes emerged from the 49 respondents who shared concerns about how PRT is used in faculty evaluations. The themes were no training about best practices or implicit bias (“Biases not based on fact or direct experience”), external influences (“non-tenured faculty worried that a peer might be hard on them if they do not like them”), lack of trust/“conflict of interest” (“Only one voice in the peer review process”), How they are used (“subjective use of observations”), and thoughts/efforts to reform (“Brought attention to studies showing biases in the use of faculty evaluations”).
Based on the results of the survey we conclude that more awareness about bias in PRT is necessary at institutions that use this process as part of their faculty evaluations. The majority of the respondents indicated that PRT takes place at their institutions, and most have established guidelines. However, the majority of respondents did not know or had no concerns about bias in PRT and how it is used in faculty evaluations.
Interactive Exercise and Discussion
To build awareness about potential biases that can operate in peer reviews of teaching, we conducted an interactive discussion during which audience members read two scenarios describing peer observation of teaching with instructions to identify potential biases and discuss possible strategies that could serve to mitigate such biases. During the discussion, audience members observed that peer review without established guidelines, evaluator training, or contextualizing information (syllabus, lesson plan, statement of teaching philosophy), is likely to produce uneven expectations for teaching effectiveness and to invite subjectivity or bias into the process. It was also observed that a conversation between the instructor and the evaluator, either before or after the review, would help to clarify for the evaluator the instructor’s pedagogical approach and strategies for classroom management. Preparatory or follow-up discussions would also serve as a mechanism for continuous improvement of both the instructor and evaluator. A final observation was that lack of clear, uniform, and transparent guidelines for peer review of teaching may invite bias during a committee review of candidate materials. It was stated that without standardized procedures for committee review of peer evaluations, the use of these materials for the purposes of promotion and tenure may be vulnerable to subjective interpretation and biases.[2]
Closing Thoughts
In closing we asked attendees and ourselves to consider possible actions to support equity-minded reform of our processes and policies related to peer-observation of teaching, especially as they may relate to promotion and tenure. On an individual level, how might we better equip ourselves to communicate objective standards and expectations for teaching effectiveness and to recognize their own potential biases? On an institutional level, how can we develop and communicate expanded definitions of teaching effectiveness? What policies or processes might be reviewed in order to reveal hidden expectations and biases or to identify areas for improvement in the peer-observation process?
Next Steps
A number of areas are available for future exploration, including collecting rubrics used for peer evaluation of teaching, considering them through the lens of our data on potential and perceived biases. Another potential step is the development of training for peer reviewers and evaluators, providing basic guidelines and best practices based on our study.
Notes
[1] For convenience, only raw frequencies are reported here due to the sample size being close to 100.
[2] Scenarios were adapted, with permission, from Brewster et al., 2022.
References
Basow, S. A. (1995). Student evaluations of college professors: When gender matters. Journal of Educational Psychology, 87(4), 656.
Brewster, K., Hamming, J., Pierce, L., & Smith, T. (2022, January 27). Interrupting Bias in Peer Evaluation of Teaching. Toolkit for Conversations & Considerations about Faculty Evaluation. Associated Colleges of the South. https://www.acsouth.edu/faculty-evaluation/fe-toolkit/taking-a-closer-look-at-peer-evaluation-of-teaching/
Basow, S. A., & Martin, J. L. (2012). Bias in student evaluations. In M. E. Kite (Ed.), Effective evaluation of teaching: A guide for faculty and administrators (pp. 40—49). Society for the Teaching of Psychology.
Basow, S. A., & Silberg, N. T. (1987). Student evaluations of college professors: Are female and male professors rated differently? Journal of educational psychology, 79(3), 308.
Kreitzer, R. J., & Sweet-Cushman, J. (2021). Evaluating student evaluations of teaching: A review of measurement and equity bias in SETs and recommendations for ethical reform. Journal of Academic Ethics, 1-12.
Stark, P., & Freishtat, R. (2014). An evaluation of course evaluations. Science Open Research doi:10.14293/s2199-1006.1.sor-edu.aofrqa.v1.
Stewart, A. J., & Valian, V. (2018). An inclusive academy: Achieving diversity and excellence. Cambridge, MA: The MIT Press.
Troisi, J. (2020, January 10). Potential biases in peer review of teaching. https://blogs.rollins.edu/facultyevaluation/wp-content/uploads/2021/03/Implicit-Bias.pdf
