Renzulli-Hartman is a respected rating scale for teacher input.  Click here for more information about Renzulli, the protocol and the research on Renzulli scales.

 

Manhasset represented that they would use a Renzulli rating scale in its selection process.  Click here for the policy presented.  But, the Renzulli Hartman scale collected in Manhasset is not a legitimate Renzulli Hartman rating.  Administrators used one of the Renzulli forms from the Renzulli manual without applying the Renzulli process. They didn't follow the protocols or procedures of a legitimate Renzulli process. The result was inaccurate measures of student behavior. The ratings say more about the teachers than the students.   
​

Renzulli Protocol

​The Renzulli research offers a recommended Renzulli protocol that Manhasset did not follow.  That protocol notes that 1/2 of the population is chosen by being in the top 1/2 of the pool on standardized tests.  The other half is composed of recommended students. 

The completion of the form is the recommendation and students recommended are automatically admitted.  The actual score from the tally form is not meaningful or comparable.  It is only used to inform future instruction.  Not every student even gets rated.  Those who score well enough don't get ratings (so that you can never get the contradiction of a high scoring student with a low rating).  Only those recommended get a rating.  (Manhasset did the opposite and required a rating for every student.)

Easy graders would be forced to regrade students and their ratings would be subject to committee review.

Multiple teachers were eligible to recommend students to compensate for harsh graders.  Students could seek recommendations.

This method states that Renzulli scores would never be used to disqualify a student. Manhasset did not implement the full Renzulli protocol and used its ratings to disqualify students.  Click here for disqualified "Unlucky Students".
 

Renzulli Scale Form as a Survey

Manhasset chose to use the Renzulli scale form outside of the Renzulli protocol.  In that case, it is being used as a survey form.  It is a simple Likert scale form.  The validity of the data collected is dependent on proper data collection methods as outlined in the Renzulli manual.  Manhasset did not follow the procedures described in the manual.  
 

• 10 different teachers filled out the form for each of their students (~10 each)
   

• Students were only rated by one teacher
   

• Teachers were not trained properly.  The district mistakenly assumed all teachers would be unbiased and would rate students similarly.  There are easy and hard grading teachers, but their scores are not recalibrated. Click here to see how differently teachers rate students. 
   

• The manual calls for a significant number of observations to be collected.  It recommends several hundred, and 100 at a minimum.  Manhasset collected 78.  Statistically, it is impossible to adjust for teacher differences due to the insufficient number of data points. .
   

• Teachers never rated the same student. There was no way to control for bias given the number of students observed per teacher and the rating procedure used. The inability to calibrate teachers' rating of the same subject made it impossible to adjust for teacher behavior differences more broadly.
   

• The administrators provided no checks and balances on the ratings as the Renzulli protocol would have required.  
   

• Norms should have been developed by random sampling of students.  There was no random sampling.  Manhasset forced teachers to evaluate only the top 78 in the district introducing significant sample bias.  They used this data to determine a "normal" score and ranked students based on it.

​

Manhasset did not use teacher ratings to evaluate students on the margin or those that were tied for the last few spots. 

 

Instead, they took the inaccurate scores from this biased sample and used them as the primary means of choosing 63% of the students in the pool for the advanced math program. 

​

Despite significant variation in teacher rating behaviors, the district used these unadjusted ratings to force rank students and allowed poor teacher rating scores to outweigh objective test score data for placement.  
 

Students who possessed more content knowledge based on actual math test performance were denied while those with less content knowledge were admitted.  (See "Unlucky Students" here.)  

​

​​