RATING SCALES BES 
month, and a lack of such change indicates a weakness in 
the scale. If a judge rates a person the same on successive 
occasions it may be an indication that he has learned noth- 
ing new about the subject’s personality in the interval; or 
that he did find reason to vary his estimate, but wished to 
avoid detection of his initial misjudgment; or, again, it may 
mean that the scale afforded him no adequate means of 
expressing an altered judgment. 
Agreement between raters, as shown by correlation of 
their ratings as well as by similarity of means and disper- 
sion of ratings assigned to the same group of workers, is 
probably a better criterion of the value of a rating method 
than agreement between the ratings made by a single judge 
at different times. If the former agreement exists, it is an 
indication that the scale calls attention to universally noted 
characteristics and makes them the basis for the rating, pro- 
vided the judges have had equal opportunity to judge the 
subject, and no judges are included toward whom the sub- 
ject would display an exceptional attitude. The data from 
which Hayes and Paterson draw their conclusions are rat- 
ings by foremen on their workmen. No matter how closely 
foremen agree in rating their subordinates, a different story 
might be told if the subordinates were rated by their wives 
and their fellow-workers. Agreement among raters is to be 
expected only if they bear the same social and industrial 
relationship to the subjects. Such agreement may be con- 
sidered a good indication of the relative worth of a rating 
method. The practical difficulty most often encountered is 
in finding two or more judges who know the men well enough 
to rate them accurately. 
A fourth way of determining the relative merit of a rating 
method has to do with the normality of the distribution of 
ratings. This method is based on the assumption that the 
distribution of accurate measures of any ability in a large 
sampling of the general population takes the form of the 
normal probability curve. The fact that ratings on an 
¥3z