Teacher Ratings in Manhasset Selection Process
Each student invited to Phase 2 of the advanced math selection process is rated by their 5th grade teacher. Teachers are asked to complete a Renzulli-Hartman rating scale form. This form uses a Likert scale to convert opinions into numeric values of 1-60 that are called teacher rating scores. The district revalued those scores into rubric points of 1, 2, 3 or 4.
The ratings used in Manhasset were purported to be the respected Renzulli scale scores, but they are not. Using the Renzulli scale form without following the Renzulli protocol and procedures does not result in accurate ratings for students. Click here to understand the limitations of the procedures applied.
Educational Research on Teacher Ratings
Our district began utilizing ratings scale in 2021 under the new policy that assumed that the teacher rating process was accurate and reliable. Unfortunately, educational research says that teacher ratings scales are biased and inaccurate and should NOT be used for advanced placement selection.
In August 2023, a study conducted by NWEA (the educational nonprofit that develops the assessments our children take 3x per year) and the University of Connecticut (home of the famed Renzulli ratings) and published through Brown University concluded the following:
-
"Up to 25% of a student’s rating score can be attributed to the teacher doing the rating." The research found that "the rating scales are not easily comparable across teachers."
-
It is “impossible to set a cut score for admission to a [gifted/advanced] program... that is equitable." I
Therefore, it is impossible to meaningfully incorporate teacher ratings scales into a sorting rubric the way that Manhasset has.
Other researchers have corroborated the finding that teacher ratings are not a fair way to select students for gifted/advanced services. In 2023, Rambo-Hernandez, Brigandi, Amin and Spillane published "Who Gets Identified? The Consequences of Variability in Teacher Ratings and Combination Rules for Determining Eligibility for Gifted Services for Young Children" in the Journal for the Education of the Gifted 2023, Vol. 46(2) 111–139. The following are direct quotes from that research:
-
"The lack of consistency between teacher ratings makes the determinations of who gets access to services largely dependent on who rated the student. Simply put, this is unfair to students and not defensible."
-
"Consider this example: One teacher consistently rates her students using the high end of a scale, and another teacher consistently rates her students using the lower end of a scale. If this between-teacher variability is ignored and the students with the highest absolute scores are identified for gifted services, students from the first teacher are much more likely to be identified for services than students in the second teacher’s classroom."
-
"who rated the student has a large impact on who is likely to receive [advanced placement] services. Perhaps, this is problematic enough to support that teacher ratings should not be considered when making decisions about services. If teacher ratings are going to continue to be a mainstay of gifted education identification practices, our recommendation is to use them to widen the pool of potential students—but never to narrow it."
Similarly, in October 2023, an analysis of teacher ratings by The Thomas B. Fordham Institute, an educational institution focused on promoting educational excellence in America via quality research, analysis, and commentary, concluded the following:
-
The variation in teacher ratings weakens the usefulness of teacher ratings in identifying students for advanced placement programs.
-
The teacher rating problem could be due to lack of high-quality training and poor implementation, but more likely "the ratings themselves are inherently flawed and doomed to teacher error and bias."
-
When administrators insist on using them, teacher ratings should be used exclusively as a means to include students who aren’t otherwise identified as advanced based on test scores; no students with sufficiently high scores should be denied advanced placement because of a low teacher rating.
-
And educators should be effectively trained on how to use these systems.
-
"Another option is to not use them at all. "
What Went Wrong in Manhasset?
The district did not implement the Renzulli protocols properly. They relied on poorly collected information from teacher ratings that have been proven biased and inaccurate in educational research. Therefore, the teacher ratings used in Manhasset are inaccurate and meaningless. Why?
-
Some teachers don't really rate at all; they give all students the max score.
-
Some grade too harshly relative to the majority.
-
Administrators provide NO checks and balances on the ratings given.
Administrators were advised about these flaws in August 2023 and the recommendations from informed parents were ignored.
Teacher Ratings - Distribution of Medians
Similar to findings in educational research, teacher ratings reflect teacher grading patterns, not student behaviors. The table below shows the median ratings by teacher for the 2023 cohort.
The chart shows that
-
Most teachers (7 out of 10) gave more than half of their students a '4'. That is the "norm". But, there are extreme graders on the high and low end - with each accounting for about 1/3 of the population.
-
Easy graders: Three teachers gave all students in their class a '4' . These teachers accounted for 32% of the pool. Two of these teachers impossibly rated all 19 students at the highest possible level on every criteria.
-
Hard graders: Three teachers graded more harshly (Dr. S, Mrs. A and Teacher 2). Their median rating was a '2' or '1'. One of those teachers (Mrs. A) did not give any of her 11 students a '4'. These students entered the process with 2-3 points less in their rubric than others and were required to score 10 or 30 points higher on the secure exam, simply because they had hard graders. These teachers accounted for another 32% of the pool.
There is no checks and balances on teacher ratings given. Teacher ratings are taken at face value by administrators and incorporated blindly into a rubric.
-
No one tells the teachers what the norm is. A teacher's personal tendencies drive their rating behavior.
-
Lenient graders are not forced to "redo" their ratings nor are they subjected to a committee review as required in a proper Renzulli rating protocol or as an attentive administrator might require.
-
The ratings for harsh graders are not adjusted to make them comparable to others (i.e., by adding 2-3 points to the low graders scores to align grading tendencies)
-
Feedback is only collected from one teacher. Parents seeking additional teacher feedback are denied. In a neighboring district (Port Washington), at least two ratings are collected over 2 years. Under the Renzulli protocol, additional teachers can nominate students and students can seek recommendations from other teachers.
Teacher Rating - Distribution of Each Teacher's Scores
The chart below shows the wide variation in teacher rating behavior that cannot be attributed to student behavior. Munsey teachers are shown on the left. Shelter Rock teachers on the right. High graders are listed first. Low graders are listed last.
Two low graders in Munsey have been identified as Mrs. A and Dr. S.
Students Impacted by Teacher Rating
For the majority (64%) of students invited into the process, the teacher rating they receive meaningfully changes the secure exam score needed for admission ranging from a 50% to an 80% due to the rubric applied. This radically changes their probability of admission.
Before the selection process was changed in 2021, teacher ratings did not play a role in placement at all. Unless teacher ratings can be done accurately, they should be eliminated.
Sources
Data shown on this site came from Freedom of Information Law requests (FOIL) requests made between August 2023 and January 2024. Sources for the information are provided below by clicking on orange label below. If you would like to see additional sources not shown here, please email us at manhassetmath@gmail.com
Teacher ratings by school and by teacher
About Manhasset Excellence
Manhasset Excellence aspires to ensure excellence in Manhasset Schools by promoting flexibility and transparency in advanced placement programs. We hope to educate the Manhasset community about the selection process for the 6th grade advanced math program in hopes that a better understanding about the policy and its implementation will allow parents and the administrators to work together to establish better placement processes for all subjects in the future. Information presented is estimated based on available information from the Board, administrators, the math department and parent feedback.