BIGENTUM DES INSTITUTT FUR WELTWI XT -CHAFT © i= BIELICOTHEK Ne. T Apb43 533/86 ws slit A MANUAL FOR DEVELOPING SCIENTIFIC METHODS OF VOCATIONAL SELECTION WALTER VAN DYKE,BINGHAM DIRECTOR, PERSONNEL RESEARCH FEDERATION: EDITOR, JOURNAL OF PERSONNEL RESEARCH; FORMERLY PROFESSOR OF PSYCHOLOGY, CARNEGIE INSTITUTE OF TECHNOLOGY; CONSULTANT IN INDUSTRIAL PSYCHOLOGY MAX FREYD RESEARCH ASSOCIATE, PERSONNEL RESEARCH FEDERATION; FORMERLY INSTRUCTOR IN PSYCHOLOGY, UNIVERSITY OF PENNSYLVANIA La i, Y . A -l 2 3 CHICAGO & NEW YORK A.W. SHAW COMPANY LONDON, A. W. SHAW AND COMPANY, LIMITED BY AND 1926 COPYRIGHT, 1926, BY A. W. SHAW COMPANY PRINTED IN THE UNITED STATES OF AMERICA TO James McKEeeN CATTELL PREFACE Tu1s manual outlines a procedure for developing, evalu- ating, and installing measurement methods in vocational selection. It explains how to construct vocational tests and how to determine their reliability and validity. The con- struction and validation of rating scales, interest question- naires, application forms, interview methods, and similar devices used in determining aptitudes also come within its scope. So complex is the task of occupational analysis and indi- vidual placement, so numerous are the independent variables, that to some it has seemed futile to expect the ideals of science and the methods of measurement to be applicable. Others have gone to an opposite extreme, seeing in the pre- cision of test technique a quick relief from the uncertainties of personal impression. Neither of these views is justified. Measurement in vocational selection has gradually shown itself to be a feasible though slowly attainable goal. The techniques of experiment and statistical method have had, here as elsewhere, their notable triumphs. But science is not an easy mistress. She opens no broad highroad to suc- cess in selecting employees. Only a pseudo-science, ignorant of statistical requirements and the standards of proof, could pretend to teach how to read character at sight or claim 100% accuracy in vocational selection. The search for genuinely scientific means of gaging abil- ities, demonstrably sound in principle and at the same time practicable, economical, useful in the routine of hiring and placement, is in itself a rather complicated process. This process of research, this adaptation of the universal prin- ciples of inductive logic or scientific method to the particular task of discovering, developing, and testing new methods of personnel selection, forms the subject-matter of this manual. 8) 1 EMPLOYMENT PSYCHOLOGY No pretense is made at a comprehensive summary of the results of selection studies. For these, the reader must turn to sources mentioned at the end of the vo!.me. The technique of investigation here formulated is based on the accumulated knowledge of many specialists who have contributed to the literature of research in vocational selec- tion, as well as on the authors’ experience. Although sea- soned investigators may discover in it little that is new to them, they will find it valuable as a check list, and as a guide to the experimental and statistical method in voca- tional selection to place in the hands of their less experienced assistants. The treatment presupposes familiarity with per- sonnel administration, psychology, mental test technique, and statistical methods. The intention has been to include what would be useful to industrial research workers and to executives who want to know at first hand what is really involved in a thoroughgoing program for developing im- proved means of selection for a given occupation. The book is planned also for use in connection with college courses in vocational psychology. While this manual deals in particular with the scientific methods which may be employed in vocational selection, it obviously has bearings also on vocational guidance. Vocational selection has for its aim the selection of work- ers with the greatest ability for a given occupation. Its primary emphasis differs from that of vocational guidance, which is concerned with the selection of the occupation in which a given individual’s abilities can be put to the most effective use. One focuses on the job, the other on the indi- vidual. Vocational selection, then, means a choice from among a group of individuals with reference to a restricted set of abilities; vocational guidance requires a choice from among an assortment of occupations and professions with reference to the relative strength of all the individual's abilities and opportunities. Proper vocational guidance will greatly reduce the problem of vocational selection. The latter, however, takes precedence in research, since scien- ry PREFACE a tific vocational guidance cannot function without a knowl- edge of the abilities required for the different occupations and of the methods for measuring these abilities. Studies carried on with the technique here outlined will therefore be of value not only to employer and applicants, but also to those who give vocational and educational coun- sel, since these researches add to available knowledge of means for measuring vocational aptitude and of the require- ments for success in the occupations studied. We are happy to acknowledge our obligations to numerous investigators in industry and in the universities, both here and abroad. Many are mentioned in the text or in the bibliography. In particular we think of the pioneers in per- sonnel research who were at the Carnegie Institute of Tech- nology for shorter or longer periods between 1915 and 1923, during the presidency of Dr. A. A. Hamerschlag: Professors Scott, Miner, Thurstone, Whipple, Gordon, Charters, Yoakum, Strong, Stevenson, Lovelace, Schoen, Kenagy, Craig, and their associates and students, including, among many others, Doctors Beatty, Ruml, Hansen, Robinson, Bills, Moore, Ream, and Manson. They have shared in the evolution of this manual. For reading the manuscript and offering sound suggestions, we are grateful to Professors E. L. Thorndike and A. W. Kornhauser. For permission to reproduce tables or charts, we are indebted to Doctors Thurstone, Kelley, Scott, and Moede, the Adjutant General of the United States Army, the Life Insurance Sales Research Bureau, the Dennison Manufac- turing Company, the General Electric Company, the Mac- millan Company, the World Book Company, the Popular Science Monthly, and the Journal of Applied Psychology. WarLTter VAN DYKE BINGHAM Max FrevD New York, July 26, 1926 Vil CONTENTS PREFACE Ses a sense EE tests sass ssc TYGER eew THE PROBLEM OF SELECTION OF EMPLOYEES ......c0se Personnel selection. The 11 steps in research procedure in em- ployment psychology. Considerations in the chrice of job or occupation to be studied. T JOB ANALYSIS «vvvuvannnnsnsnsss-snsoannns sssssssees 13 Outline for job analysis. Sources of information in making a job analysis. Procedure. Illustrative job analysis. Important points. III CRITERIA OF VOCATIONAL SUCCESS ++eeseasssnnesasanes Importance of the choice of criteria. Thirteen suggested criteria of vocational success. General considerations in the selection of criteria. Iv CHOICE OF WORKERS TO BE STUDIED ....vevsesnsasesss 49 Considerations in the choice of workers for study. Some dis- turbing variables to be eliminated. Newly hired versus old employees. \ ANALYSIS OF THE WORKER vescessvnccasccrsrssesesss 58 Topics for a man analysis. Analysis of abilities. Innate and acquired abilities. General and specific abilities. Relative im- portance of abilities. VI SELECTION OF EXAMINATIONS evscersnccnnnaosrsuvsvns J Principles governing the selection and construction of tests: Ob- jectivity ; reliability ; simplicity, analogy of test and ability to be measured ; resemblance to vocational tasks. Sources of material: The vocational apparatus; former studies of the vocation; general test publications: miscellaneous sources of information. 1 2 uy 30 ir 76 ix EMPLOYMENT PSYCHOLOGY v PSYCHOLOGICAL TESTS... ir st dea, What is meant by a psychological test. Classification of tests ac- cording to purpose, ability measured, apparatus or material used, technique, restrictions on response. Verbal tests. VIII PsycHoLocIcAL Tests (Concluded) o................. 10+ Performance tests. Technique of test construction. Order of items. Length of test. Time limits. Directions and fore-exercise. Directions for scoring the test. Omnibus tests. Alternative forms. Arrangement in series. RATING SCALES es ee ee Ee — Methods of estimating abilities: Order of merit; grouping; the Scott form of man-to-man comparison scale; the scale of alter- natives or two steps; per cent scales; multiple-step scales; linear scales; descriptive scales; graphic rating scales. bi RATING Scares (Concluded) ........................ I,2 Statistical and non-statistical bases for the comparison of rating methods. How to make a graphic rating scale. Obtaining the ratings. Statistical treatment of ratings. QUESTIONNAIRES: THE PERSONAL HisToRY RECORD AND THE INTEREST ANALYSIS |, ces cre riots 143 Definition of a questionnaire. Uses of a questionnaire in an in- vestigation of this sort. Characteristics of a good questionnaire. Obtaining the data. XII TEST ADMINISTRATION + vonsnensneesss annum I55 Preliminary details in test administration. The examiner's atti- tude. The testing room. Procedure in testing. XIII VALIDATION OF THE MEASURING INSTRUMENTS . ......... It, Tabulation. Meaning of reliability. Measures of reliability. Cor- rection for attenuation. Meaning of validity. Measures of va- lidity. Group comparisons. Differentiation of groups by test scores. Differentiation of groups by questionnaire items, xX /T1 87 Qu IX SP 7 XT 32 CONTENTS XIV VALIDATION OF THE MEASURING INSTRUMENTS (Concluded) 174 Correlation: Product-moment formula; method of rank differ- ences; fourfold table method; Pearson biserial r; correlation ratio. Coefficient of mean square contingency. Coefficient of as- sociation. Probable errors of measures of correlation. Inter- pretation of the size of the correlation coefficient. XV PREDICTION OF VOCATIONAL SUCCESS +:+::tcsssscrveeese 2 J Prediction based on group comparison. Critical scores and critical sections. XVI PrEDICTION OF VOCATIONAL Success (Concluded) ...... 197 Prediction based on correlation. Regression line and regression equation. Standard error of estimate. Percentile curves. XVII PREDICTION BY COMBINED SCORES +1::vcvvevnuannennss 2 Total scores based on group comparisons. Weighting critical sec~ tions. Total scores based on the establishment of correlation. Multiple correlation coefficient and multiple regression equa- tion. Other methods of combining scores. XVIII Economic VALUE OF THE EXAMINATIONS .............. 2I} Necessity for comparison of old and new methods of selection. Relative predictive accuracy. Relative economic value. XIX THE EXAMINATIONS AT WORK +ovsuser=z-tl:zrrrtns es BEE Installation: the examiner; the forms; the room; the procedure; the examiner as interviewer. Maintenance: modifications of procedure and standards. REFERENCES vais vss nv evs nism vsnassnensrsrerinsss Bley TABLES .ncusrt ens names onniinessnsvnrsess savas es vi 240 DDE a xe vies ins sans sien sne veins dentine sven eewsenl. 205 Xi {QC 20% 2927 PROCEDURES IN EMPLOYMENT PSYCHOLOGY I THE PROBLEM OF SELECTION OF EMPLOYEES Personnel selection. The 11 steps in research procedure in employment psychology. Considerations in the choice of job or occupation to be studied. THE right man in the right place is an age-old ideal of industry, education, and government. Economic losses suf- fered by industry from misfits in employment are matched by the losses borne by the employees themselves, the dissatis- factions of misplaced workers engaged day after day on tasks that are too easy, too hard, or otherwise unrewarding, dis- tasteful, or exasperating. Correct adjustment of workers and their work is recognized as a prime responsibility of management, an obligation to society no less than to the stockholders and to the employees themselves. It is a duty of science to help in effecting this adjustment of workers and their work, in so far as scientific method can be made to yield measures of abilities which are more relia- ble than unaided judgment. To speed the perfecting of such useful instruments for measuring abilities is the aim of this manual. THE PROBLEM OF PERSONNEL SELECTION To select the best man for a particular opening admittedly calls for an understanding of the requirements of the job, coupled with the means of finding out which applicant most closely meets these requirements. Usually the interviewer or employment supervisor responsible for the preliminary sifting of applicants has at hand brief but concise employ- ment specifications based on job analyses. He also has some acquaintance with the job in question through observation EMPLOYMENT PSYCHOLOGY 1% : = LB — k ¥ g& a 20 % % mm vm DC = 11,988 Replies to Appointments Appointments Men Hired Men Making Advertisement Made Kept Good Figure 1: The problem of selection of salesmen in one firm. Data from O. R. Johnson (81). and through conversation with workers and supervisors regarding it. If he has kept good records, he has probably checked his opinions as to the education, age, experience, skill, physique, intelligence, and special aptitudes most desired in an applicant for the post, by going over his data as to employees who have left the job dissatisfied, or who have been transferred or dropped as unsatisfactory. He knows how wasteful it is to engage men too dull to learn the duties expected of them within a reasonable time, or too bright and ambitious to be satisfied with the work and the wage the job will pay. He seeks dependability as well as skill, application no less than talent. Health and strength may, for certain occupations, outweigh brilliance. Some- np THE PROBLEM OF SELECTION - times social acceptability, either within the group of fellow- workers or with the public, is essential. To know the outstanding requirements of each job, whether of inspector, elevator operator, millwright, salesman, or accountant, and to be able to discover and rightly appraise indications of ability to succeed in this job as they are revealed in the facts of the applicant’s personal history and of his behavior in examination and interview—these are the difficult duties of the specialist in employment. Initial selection of employees for particular positions in industry, business, and the civil service is, however, no more vital than the selection of applicants for special vocational training. The number of college graduates annually seeking admission to such a professional school as the Harvard Med- ical is much larger than the number of available places, making it necessary for the faculty to give a great deal of time and consideration to the selection of those who are admitted. The dental faculty of another university, realiz- ing that men they had taught for two or three years fre- quently had to be dropped for lack of fundamental muscular coordinations essential to the acquisition of skill, asked a psychologist to prepare a battery of motor coordination tests as an aid in selecting first-year students. The works man- ager of a prosperous and progressive factory is debating with his staff—while these words are being written—as to the advisability of introducing a two- or three-year course of instruction and factory experience for the purpose of developing supervisors. “How will you pick your young men for this training?” he asks of his educational director. “Tell me how you are going to select the boys who have it in them to make good supervisors.” Since the educational director has at present no plan of demonstrated worth for selecting future supervisors from among the factory em- ployees and the young high-school and college graduates of the community, he proposes to make a thorough study of the personal histories, the personalities and the abilities of those young men now in supervisory positions in the com- 5 EMPLOYMENT PSYCHOLOGY pany who are making good, and on this groundwork to build a dependable plan of selection of recruits to be developed into the future supervisory force. Preliminary sifting of applicants for apprentice training is a live problem in trade and continuation schools, and also within the ranks of a few of the trade unions. Art schools and conservatories of music have their difficulties in dis- covering the really talented candidates. Selection for voca- tional training, whether on the level of trade or profession, is even more difficult at times than the choice of the best applicant to step at once into a vacant position in industry. It requires an estimate of the applicant’s ability to learn and to develop as well as of his ability to do. Considerations governing selection of the particular job, occupation, or profession to be investigated will be reviewed in the final pages of this chapter. But first it will be well to outline the plan.of this manual, which follows the steps involved in carrying out a research on methods of selection for a vocation. STEPS IN RESEARCH PROCEDURE After deciding upon the occupation for which an improved method of selection is to be devised, the first step is to make a job analysis, to ascertain the precise duties and activities of the occupation, just what the worker does, what tools he uses, and so forth. The investigator will also want to know what the conditions of work are, the hours, the sources of applicants, the relation to other jobs in the organization, the opportunities for promotion, and the most frequent reasons for leaving. The investigator should then in consultation with the responsible executives decide upon a criterion of success in the occupation. This is necessary in order to have some measure with which to compare the results of the experi- mental procedures in selection. The next step is to choose persons to serve as subjects 6 THE PROBLEM OF SELECTION for the investigation. This choice will be determined in part by practical considerations and in part by the criterion of vocational success which has been agreed upon. The investigator then proceeds to determine at least pro- visionally the abilities most essential for success. This step is facilitated by having for observation the workers who have been chosen as subjects for the investigation. In the light of his analysis the investigator selects or devises psychological tests, questionnaires, rating scales, or other examinations which give promise of measuring these essential abilities. Under carefully controlled conditions the investigator next applies these means of measuring abilities to the subjects chosen for the investigation. At or before this stage of the research the reliability of the examinations should be determined. A test or rating scale which does not give the same results upon repeated trials is as unreliable for purposes of vocational selection as an elastic tape would be for measuring stature. After the reliability of the examinations has been ascer- tained, their validity as indicators of success in the vocation is determined. The investigator validates a measuring instrument by comparing his measurements with the cri- terion of vocational success. A device is valid and useful which yields measures so closely related statistically to the criterion of success that, knowing a person’s score, his degree of success in the occupation may be at least roughly foretold. The value of the examination depends upon the closeness with which it predicts the vocational success of the applicant. The next step in the research is to find what combination of the several examinations which have proved to be valid, yields a total score with a maximum predictive value. The investigator will then have to justify this battery of tests or other aids in selection by comparing its predictive accuracy with that of the methods of selection previously In use. EMPLOYMENT PSYCHOLOGY If the new methods prove to be more efficient and are recommended for adoption, he installs them as a part of the employment procedure. He also assures himself from time to time that they are being used properly, and checks their predictive accuracy frequently in order to adjust them if necessary to changes in industrial demands or type of applicant. Not every research in vocational selection will require that all of these 11 steps be included in the program. The investigator will, however, be more apt to avoid pitfalls if he examines his outline of proposed procedure critically with each of these steps in mind, and assures himself that pro- vision has been made for every stage of investigation essen- tial to the sound progress of his particular study. We are primarily concerned with the scientific measure- ment of aptitudes for an occupation. The method outlined may also be used -in developing differentiating tests for placement. It can be used in distinguishing between several occupations which differ in the degree of difficulty of their work. The method may be employed, for example, in validating tests for placing office help at either typewriting, stenography, or secretarial work; or at filing or comptometer work. The examinations of which this manual treats in greatest detail are psychological tests. The accuracy with which a test measures an ability to succeed in a job varies with the extent to which it calls out the same or similar types of adjustments as the job. If there is a close relationship between test scores and success at the job, disturbing vari- ables being held constant, the assumption may be made that the relationship is caused by the fact that the two call out the same ability or abilities. A causal connection between two variables is merely the observed concomitant variation of the two. A causal rela- tionship between test performance and success at the job may be inferred if the correlation between them is high. Even this modicum of theory may be omitted. If the corre- Q THE PROBLEM OF SELECTION lation between test performance and success at the job is high, one may be used as the sign of the other. The test performance, requiring at most a few hours, then becomes a prognosis of what the applicant will be able to accomplish at the job over a long period of time. First are discussed the grounds for selecting a job or occupation to be studied in an organization, the analysis of this occupation, and the selection or construction of tests and other instruments of measurement; then, the technique of trying out the tests and determining their validity as indicators of probable success; while a final chapter deals with the installation and use of the new aids in selection. CHOICE OF JOB OR OCCUPATION TO BE STUDIED The range and variety of occupations inviting the atten- tion of the investigator is great. Care may well be exer- cised in deciding on the ones whose study by the scientific method gives most promise of valuable results. Not every occupation can profitably be made a subject of scientific study. This the investigator must realize if he wishes to avoid disappointment and failure. The scientific method requires the satisfaction of certain well-defined requirements. Where these conditions are absent, the in- vestigator must acknowledge that the scientific method can- not function. Other methods must suffice until conditions are made right for the introduction of scientific methods. Even where conditions permit the use of scientific proce- dure, economic considerations may not warrant such a study. Before deciding on an investigation of selection methods in an occupation, there is need of answers to these five ques- tions: 1. Does a problem of selection actually exist in this occu- pation? Every scientific investigation aims to solve a prob- lem. It may happen that the problem here defines itself concretely in the form of excessive labor turnover, low pro- 0 EMPLOYMENT PSYCHOLOGY duction, or costly breakage in one department. If no such definite and limited problem has been made obvious, but if the management wishes to raise the general level of effi- ciency in its employment department, then the investigator will find it advisable before he proceeds with his investiga- tion to ask the cost department to help him find the job where increased efficiency in selecting employees would bring about the greatest economic saving. A New England investigator, a novice in research, was given an assignment in the personnel department of a pub- lic service corporation to “try out this psychological test business, and see if there is anything in it for our company.” He asked a psychologist, “What tests do you recommend that we should try?” But he could not state whether the heaviest losses due to poor selection were being suffered in the office or in the boiler-room, among the executives or among the sales representatives. It was a new idea to him that his own cost accountant might be able to advise him better than a psychologist as to the most promising point of attack in his research. Later he assured himself that the number of clerical workers hired and the cost of training them were such that a 5% increase in accuracy of selection would more than pay for the cost of administering a scien- tific program. An association of employing printers, in cooperation with the typographical unions of New York City, was considering a research to develop a scientific technique of selecting boys to receive training as compositor apprentices. The research worker’s inquiry, however, as to the effectiveness of the present informal but carefully administered methods of selec- tion, revealed the astonishing fact that only 2% of the boys selected proved to be unsatisfactory students in the appren- tice school. As this was the only available criterion of good selection, it was decided, wisely, to defer research on psycho- logical tests for selection. Better tests of progress and of trade proficiency were the immediate need. 10 THE PROBLEM OF SELECTION 2. Are there more applicants than jobs? If the number of available applicants is less than the number of jobs that must be filled, then a research in selection is beside the point. Exceptions to this rule are found in those instances where greatly improved selection has decreased the number of employees required to do the work or has attracted to the employment window a larger and better group of applicants. 3. Are there enough employees doing the same kind of work to make possible a reliable study? Statistical method demands a considerable number of cases for exactness of conclusions. If only a few employees are doing the same sort of work, the findings of an investigation will have only a very low reliability when used to select from among future applicants. 4. Is there a valid and reliable criterion of success at this work? Without some dependable indication of a man’s actual success, it will be impossible to find out whether or not the proposed measurements of abilities distinguish between successful and unsuccessful employees. 5. Is the investigator assured of cooperation? A reliable study is a practical impossibility without cooperation of both workers and management. We shall assume that the investigator has selected for study a job, an occupation, or a profession, after assuring himself that a real problem of selection exists there; that the excess of applicants over vacancies is great enough to promise considerable savings as a result of improvements in selection; that the number of cases available for study is large enough to warrant statistical reliability in the findings; that a dependable criterion of success is to be had; and that the research will not be seriously hampered by lack of active cooperation. The job chosen may be a highly responsible post such as branch manager or design engineer. Or choice may have rested on an important but more restricted occu- pation, such as that of chemical analyst, draftsman, accoun- I1 EMPLOYMENT PSYCHOLOGY tant, or nurse. It may be in an office, perhaps as pay clerk or comptometer operator; in a store, as cashier or assistant buyer; in a trade school, as apprentice compositor; in a uni- versity where standards of admission are being fixed for entrance to law school; or in a civil service commission which is preparing examinations for selection of policemen, plumbers, hospital attendants, highway engineers, and income-tax adjusters. Whatever the occupation selected for study, scientific method imposes much the same program of successive steps in research procedure. Having fixed on his particular problem, the investigator must get his bearings by making a careful analysis of the job to be studied. The next chapter deals with the methods of carrying out a complete job analysis. 12 IT 13 JOB ANALYSIS Outline for job analysis. Sources of information in making a job analysis. Procedure. Illustrative job analyses. Important points. JoB analysis, according to Tead and Metcalf, is the “sci- entific study and statement of all the facts about a job which reveal its content and the modifying factors which surround it.” (183,! p. 255). We shall use the term in this sense, applying it to the analysis not only of the actual work involved, but also of such factors as the conditions under which the work is carried on, its place in the complete organization, training required, and the like. It is a safe assumption that the accuracy and success of a study in vocational selection will be directly proportional to the completeness with which the foundation is built in the job analysis. In too many instances the psychologist has been content to construct tests and other measuring instru- ments after only a superficial observation of the work processes. Such a procedure may overlook economic and social considerations, either of the worker or of the job in relation to the organization of the company, which may far overshadow in importance a test of ability in the selection of workers. It may be a simple affair to test ability as a longshoreman. Some ordinary physical measurements might suffice. But what a wealth of other factors bring a man to this occupation, make him a satisfied worker, and keep him at the job! The abilities tested may be the least important factor. A great life insurance company developed and installed tests of clerical ability which met with only mod- [7 Numbers within parentheses throughout the text refer to authors or books listed in the Bibliography, on pages 235 to 245. EMPLOYMENT PSYCHOLOGY erate success in practice because one point bearing on sta- bility and satisfaction on the job had been overlooked. The work of the office was necessarily so distributed that the heaviest load fell on the first days of the week, often requir- ing several hours of overtime on those days. No matter how competent the employee as judged by tests or experi- ence, she would not remain long with the company unless she felt willing to accede to these overtime demands. A complete job analysis would have brought out this fact. In most large organizations a set of employment specifica- tions based upon a job analysis will probably be available to the investigator. Such an analysis usually lists only the most outstanding facts about the job. Rarely is the infor- mation complete enough to serve as a basis for test con- struction. Such specifications may serve as a point of departure, but whether they are available or not, it will be a practical necessity for the investigator to make an analysis himself in order to see with his own eyes just what the job involves. He brings to the problem a somewhat different point of view and for his own particular purposes he should study and evaluate all the phases of the job. Job analyses have other uses as well as for purposes of employment. They may serve as a basis for improving methods and conditions of work, for establishing a training course for new workers, for wage setting, for determining lines of promotion, and for defining responsibilities. While it is impossible to prepare in advance a compre- hensive list of items to be considered in a job analysis which will serve for every type of job or every use, the following check list will be found suggestive. It will ordinarily have to be shortened and adapted to the particular job under study. The heart of the analysis is “5. Exact Operations.” Similar lists are given by Scott and Clothier (166), Tead and Gregg (182), Meine (112), Toops (202), Kingsbury (92), and others. 14 JOB ANALYSIS OUTLINE FOR JOB ANALYSIS I. IDENTIFICATION OF THE JOB Name of the occupation. Identifying symbol. Alter- native names. Names and locations of departments in which the work is carried on. Similar occupations from which or to which workers could be transferred. 2. NuMBER EMPLOYED Present force. Anticipated requirements. 3. TyPE oF Work Statement of the duties, functions, and responsibili- ties connected with the job. This should be a broad identifying statement, and should not cover the method of carrying out these duties nor the abilities required. TEcuNICAL EQUIPMENT Tools. What kind of tools? Who supplies them? Who is responsible for their upkeep? Machines. What type? In what condition? Who is responsible for upkeep? Materials. What kind? What variations? Descrip- tion of materials. 5. Exact OPERATIONS The exact duties and the ways in which they are carried out. This should be in narrative form and in great detail. Operations should be listed numerically in sequence. Begin each item with an active verb. Give the amount of time devoted to each operation and its relative importance. Describe the exact motions in form that may be suggestive of test construction. What parts of the body are used? Are movements standardized? What repairs or adjustments to the machines or equip- ment does the worker have to make? 6. ConDITIONS OF WORK Location. Factory, office, inside, outside, overhead, underground. Workroom. Ventilation, temperature, humidity, illu- mination. Time. Permanent, temporary, day, night, hours of labor per day, hours per week, overtime, peak loads, uni- formity of work, rest pauses, lunch hour. vacations. I5 16 EMPLOYMENT PSYCHOLOGY Posture. Standing, sitting, stooping, walking, climb- ing, reaching, lifting, kneeling. Speed. Quick, moderate, slow, variations in speed. Necessity for turning out work extremely fast to meet an emergency. Accuracy. Coarse, fine, exacting. Possibility of loss through inaccuracy. Automaticity. Varied, routine, monotonous, mixed. Health hazards. Nerve strain, eye strain, physical strain (heavy, medium, light), moisture, heat, dust, fumes, acids, exposure to weather. Accident hazards. Slippery floors, unguarded ma- chinery, pointed tools, rapidly moving objects, liability to burns. Disagreeable features. Dirt, noise, oil, and others included above. Vibration. Fatigue. What are the most frequent or serious grievances? Agreeable features. Cleanliness, prestige, companion- ship. Social features. Does the employee work alone or with others? Is he permitted to talk? How close is the supervision over him? Standards of output. What standards must be met? How are quotas set? Is inspection uniform and equi- table? 7. Pay Method. Monthly, weekly, biweekly, daily, hourly, piece-rate, guaranteed hourly minimum plus bonus, check or cash. Rate. Average earnings, range of pay, starting pay, maximum obtainable. How often are advances in pay given? Bonuses and premiums. When given? How much? On what basis? Penalties. For absence, tardiness, infringement of rules, spoilage. 8. NON-FINANCIAL INCENTIVES AND REWARDS Social rewards for good work. Incentives to speed and high quality of output. Prestige connected with the job. JOB ANALYSIS 9. TRAINING Is training given by the company? In vestibule school or on the job? What does it cover? How long does it last? How much does it cost to train a worker? What percentage of new workers complete the training? How is the worker paid during training? 10. BROADER SociAL AspEcTs oF THE JoB What are the ambitions and ideals of the workers? What are their standards of success? What employee service does the company make available? Hospital, insurance, sick benefits, pensions, loans, employee stock ownership, employee representation in management, social work, night classes, recreation, house organ, hous- ing. What is the psychological effect of the job on the worker? What sort of home life do the workers have? What are their living conditions in general outside of the factory? Fatigue, recreation, temperance, marriage. Social status of fellow workers. 11. RELATION TO OTHER JOBS Organization charts, showing lines of promotion, lines of authority, lines of routine procedure. Comparison with other jobs in regard to salary, opportunity for pro- motion, prestige, and so forth. Is the acquired skill an asset outside of the company? 12. EMPLOYMENT CONDITIONS Selection. How are workers selected? How efficient is this method? What other methods have been used? What would improved methods of selection accomplish? Sources of supply. Unions. Promotion and transfer. What opportunities for either? Adequate personnel records? Periodic rating and consideration for promotion or wage readjustment? Turnover. What is the turnover? What are the most frequent causes of leaving? 13. RECORDS What records of output, rate of progress, quality, spoilage, materials, time, and so forth, are available? Application blanks, references, ratings, and so forth. 14. PROBLEMS What are the main personnel problems which this job has raised? 17 EMPLOYMENT PSYCHOLOGY It is customary to list also under the heading of “Job Analysis” all the abilities and characteristics of the em- ployees which are of importance in selecting men for the job. In a scientific investigation the analysis of these abili- ties is facilitated if it is made after the criterion of success has been decided upon and the workers chosen for measure- ment. We shall, therefore, devote a later chapter to this topic. SOURCES OF INFORMATION IN MAKING A JOB ANALYSIS Working at the job. Undoubtedly the best method of obtaining the information sought is for the investigator to spend considerable time at the job, mingling with the work- ers and obtaining their points of view as well as observing the motions which the job calls for. It is sometimes prefera- ble if his identity and purpose be kept from both workers and executives immediately in charge. He should go through the entire employment process, including the train- ing courses, and if possible join in the social life of the employees after he is hired. This procedure will give him the opportunity of analyzing the motions he makes as well as the motions of the other workers. It will place him in a position to make important notes on the abilities, social life, personal history, personality, aspirations, and attitudes of the employees, as well as on the conditions of work and the quality of training and supervision. Observation. The investigator may draw valuable con- clusions from merely observing the men at work, without doing the work himself. He should aim to keep himself in the background in order to avoid making the workmen self- conscious. Photographs and motion pictures of the work are valuable additions to his written observations. Interviews with workers. Another source of information is through personal interviews with workers of all degrees of proficiency. The investigator should try to find the reasons 18 JOB ANALYSIS for their success or failure. He will gain important infor- mation only if he is able to differentiate between true rea- sons and those which are untrue but satisfying. The practical considerations to be kept in mind in analyz- ing the worker at his work are well brought out by Uhrbrock (207): As the interest in job analysis grew, it became evident that certain qualifications were necessary for success in the work. In the first place, the man who is selected for the work should be mature and make a good impression on the group whose work he is to analyze. He need not necessarily be what is known as a “good mixer,” but he should at least command confidence and respect. His job is to get “job facts” from the men, as well as to observe their work. The job analyst must be able to “sell the idea” of making a complete record of the elements that go into the job. By “job” is meant man and equipment functioning smoothly in the performance of work. Qualities that he must possess are patience and tact. He will find it helpful if he speaks the language of the shop. In analyzing the work of printers one soon finds out that those who set type are not “composers” but “compositors.” Among the pattern-makers the man who does not have a union card is a “skate.” To refer to him in any other terms is to put oneself beyond the pale. The telegrapher refers to the day-letter as a “blue” and the night-message as a “nite.” To know such terms helps in establishing points of contact in talking with the workers. The successful job analysts keep the men informed as to the purpose of their work. They talk freely about the use of the material in working out training programs, or how the items that make up the job can be evaluated by a salary adjustment com- mittee. In fact, this is one of the successful ways of getting the men to talk. Each one likes to feel that his particular work is the keystone that holds up the arch. If the job analyst can con- vince the man that the facts he gathers will be put to some prac- tical use, he will make his work easier and more effective. The men are averse to the presence of non-producers—‘“dead-heads” as they call them. They feel that each man in the concern should carry his own weight. In most cases this attitude is not expressed openly. The cooperation of the men will be in proportion to their understanding of the work and purpose of the job analyst. Interviews with supervisors. Information obtained from IQ EMPLOYMENT PSYCHOLOGY workers should be checked by going over it with the work- ers’ supervisors. Essential points may be developed which the workers themselves had overlooked or thought to be too trivial or too obvious to be mentioned. It is also advisable to consult executives, foremen, and principals who have risen from the ranks, even though their judgments are often char- acterized by a tendency to emphasize certain points at the expense of others. When interviewing these men the investi- gator will naturally avoid any appearance of questioning the adequacy of their analyses. His success in his enterprise, depending. in part on his ability to get along with the men in the firm, must rest on mutual respect. The investigator should give assurance that any informa- tion or records entrusted to him will be for his own use and not be made public. He should be as agreeable as pos- sible with both worker and executive. He should keep the attitude of one who is there to learn. Above all, he should avoid telling the workers how to do their work. Records. Office records on output, employment, turn- over, and personal history must, of course, be consulted. Previous studies. Another source of information is the conclusions of other investigators who have analyzed the same or similar vocations. These conclusions will be found in the trade literature, in psychological periodicals, or in such general works as Parsons (126). Attention is also directed to publications on job analysis soon to be issued by the American Council on Education. PROCEDURE IN MAKING THE JOB ANALYSIS There is no standard procedure for carrying out a job analysis. The method pursued will vary from job to job. In many cases it will be impossible to make use of all the sources of information described above. In a general way two types of work can be distinguished, requiring different approaches. One type of work is mechan- 20 JOB ANALYSIS ical and routine. The other type is characterized by variety and calls for the use of intelligence in meeting new situ- ations. Many jobs combine both types of work. Routine work is illustrated by the following quotation from Link describing the duties of a shell inspector (101, pp. 24-25). The work of inspecting shells was done at a table like an upturned shallow box. Upon this hollow table was dumped a large box of brass shells, not yet loaded, and all of exactly the same kind. The work of each girl was to inspect these shells and throw out those that were defective. In doing this a girl would first gather up a large handful of shells, as many as could be piled in one hand, being careful to have all of them pointing in the same direction. Then she would put both hands around the shells and turn them all up so as to expose their insides. She would then look down into every shell for dents, scratches, stains, and other very minute defects. When any such defect was dis- covered, the shell was skilfully extracted from the pile and thrown into one of three or four appropriate scrap boxes. The entire handful was then turned over, and the head of every shell exam- ined for various defects. The shells were then held in a hori- zontal position on the left hand, and allowed to roll from the pile into the right hand. Each shell, in the process of rolling from one hand into the other, exposed its lateral surface and was closely scrutinized for scratches, oil dents, stains, and other defects. The good ones were taken in the right hand and dropped into a pocket at the right side of the table, through which they fell into a box below. The second type of work is illustrated by the following analysis of the duties of an assistant buyer in a department store (112, pp. 32-34). ASSISTANT BUYER The duties as listed here are arranged in the order of importance. (The masculine pronoun is used throughout for uniformity.) A. DUTIES—SUMMARY 1. Regular and Primary Responsibilities [. Merchandising the stock.—He is responsible under the supervision of the buyer and within the rules to so merchandise the reserve and forward stocks 21 EMPLOYMENT PSYCHOLOGY as to produce the maximum of sales and the minimum of losses. II. Operating and clerical routine.—He is responsible for the routine, operating and clerical work of the buying division of his sections. III. Public demand.—He shall keep the buyer advised of the nature of the public demand and the abil- ity of the stock to meet it. IV. Publicity.—He shall assist the buyer in handling the publicity for the section. V. Market conditions.—He shall, so far as in his power, keep the buyer advised of market condi- tions, especially local, and possible jobs, and shall assist in sorting jobs for retail. 2. Secondary and Occasional Responsibilities I. In absence of buyer..—He shall represent the buyer. (See Buyer’s Duties.) II. Cooperation in giving service and in educational work—He shall cooperate with the floor super- intendent in giving service to customers, and he shall assist the educational department, the buyer, and the floor superintendent in the training of sales people, stock people, and automatic markers. III. Meetings.—He shall take active part in the assis- tant buyers’ meetings. B. Duties IN DETAIL I. Regular and Primary Responsibilities in Detail I. Merchandising the stock (a) Know at all times the retail stock forward. (6) Have forward enough merchandise to force maximum sales. (¢) Maintain, as far as the rules and stock limi- tations permit, an adequate variety of styles, materials, sizes, and colors to meet the demand at various prices. (d) Allow no merchandise to remain in reserve longer than the allotted time, except by spe- cial approval of the basement merchandise manager. 22 JOB ANALYSIS (e) Push slow-selling forward stock to avoid undue automatic losses. r) By constant study of the automatic mark-down merchandise and of the relative speed at which the merchan- dise moves. Jv concentrating through the floor superintendent the efforts of sales- veople on the slower-moving stock. I'y advantageous stock arrangement, ‘+ ly suggesting various forms of pub- licity. (5) By changing location or by mark-outs. (f) Follow up the merchandise on order so as to make sure that it will arrive at the time agreed upon. (g) Analyze the merchandise tape as issued day by day. (%) Study carefully the 50-A report. II. Operating and clerical routine (1) Operating routine: (a) Inspect merchandise on arrival to verify price, styles, materials, colors, sizes. "' See that merchandise is properly marked and placed into forward or reserve stock. .¢) See that the bill is properly checked and entered on invoice register and in the bill book, retailed and signed. (To be done by 3 p.m.) (d) Verify stock record of merchandise charged to his department at least weekly. (e) See that all mark-downs, either vol- untary or automatic, are properly taken, entered, and signed. (f) See that transfers, consignments, and returns are properly made out, signed, and entered in bill book and stock record. 22 24 EMPLOYMENT PSYCHOLOGY (g) Assist the buyer in the handling of the merchandise correspondence. (%) See that no merchandise is held un- duly in the receiving room. (¢) Make all possible checks to account for and prevent stock shortage. (j) Make summaries of the forward stock and submit weekly to basement mer- chandise manager. (%k) Arrive in department by 8:15 a.m. (2) Clerical routine: (a) See that the daily sales record is kept. (5) See that the sales report is made out and turned in. (c) Keep such permanent records of auto- matic mark-downs as are of value for future buying. (d) Keep under the direction of the buyer a record of merchandise lots of con- sequence placed on sale and enter the selling results of each. (e) Keep the reserve stock cards up to date and submit weekly to basement merchandise manager. (f) Make such stock analyses as are called for by the buyer, division man- ager, or basement merchandise man- ager. III. Public demand: He shall keep the buyer advised of the demand of his public as to: (1) Prices. (2) Styles. (3) Colors. (4) Materials. (5) Sizes. IV. Publicity: (a) He shall obtain from the basement advertising office his space assignment. (6) He shall assist the buyer in planning for the use of space. JOB ANALYSIS (c) He shall see that the advertising copy and merchandise reach the basement advertising office in ample time, and shall check the advertising profits. V. Market conditions: The assistant shall constantly be on the look- out for jobs of merchandise and keep the buyer fully advised. These may be in the shape of present actual, probable, or possible future jobs. He shall obtain his information from: (a) Newspapers. (6) Trade publications. (c) Contact with manufacturers, especially local. (d) Correspondence. 2. Secondary and Occasional Responsibilities in Detail I. In absence of buyer: He shall represent the buyer during his absence. II. Cooperation in giving service and in educational work: (a) Service. He shall give at all times all pos- sible cooperation to the floor superintendent in service to customers with special regard to technical knowledge in: (1) Seiling (advice to salespeople). (2) Instruction of the salespeople in merchandising. (3) Adjustments where this knowledge may be of assistance. (6) Education. He shall cooperate with the education department, floor superintendent, and the buyer in the training of the people of his department. (1) Salespeople. (a) Instruction on merchandise from the technical side. (6) Advice when requested in handling difficult customers. (¢) Methods of increasing sale of slow-selling merchandise. 25 EMPLOYMENT PSYCHOLOGY (2) Stock people. (a) Instruction on merchandise from the technical side. (0) Assistance in training. (3) Merchandise clericals: He shall as- sist in their training and efficiency. (4) Automatic markers: He shall assist in their training and efficiency. III. Assistant buyers’ meetings: He shall take an active part in all assistant buy- ers’ meetings, and shall study such problems as the buyers’ meetings have under discussion. The printer’s job is an illustration of a sort of work which is composed of a number of routine jobs and at the same time calls for the use of judgment. The United States Army has developed a number of excellent educational manuals for the training of technicians. These manuals describe in great detail the different job operations and are the best examples of complete work analyses of the sort the investi- gator should aim to make. (Such a job analysis, when made, may also serve as a basis for a training manual.) The following quotation from Educational Manual No. 18, Print- ing—Hand Composition (142) describes one of the routine duties of the printer. U. S. ARMY Operation Sheet No. 15’ E. AND R. SPECIAL SCHOOL PRINTING TRADES—HAND COMPOSITION DisTtrRIBUTING TYPE References: DEVINNE, Modern Book Composition, pp. 93-95. HENRY, Printing for School and Shop, pp. 33-35, Sec. 41. Directions: 1. Place type, properly cleaned, on brass galley and dampen slightly with wet sponge, first removing any cuts or ornaments having wood bases. If the matter to be distributed consists of 1 Reproduced by permission of the Adjutant General. 206 JOB ANALYSIS Si i - ARG he FRIES RT HR 130 FE BI HAL 77 sR HRN ER NEAR HO RE ER] / AR RR FER ERAT 3 Bhs. (6 FERRIS ‘NOR {IH aut ¥ Figure 2 several different sizes and faces of type, assemble all lines of one size and face together before beginning. 2. Before distributing compare the face and nick of the type to be distributed with that in the case in which it is to be placed. 3. Pick up only as much type as can be easily handled, holding it in the left hand, nicks upward, as shown in Figure 2. Take only two or three lines at a time until skill in handling has been acquired. 4. Beginning at the right-hand end of the top line, grasp a few letters, preferably a complete word, between the thumb and the first and middle fingers of the right hand, separate the letters and place them in the proper boxes. In taking off a word the type is slid forward onto the thumb by the middle finger, held in place by the first finger, and the two fingers then moving alternately backward and forward over the thumb release the type, letter by letter, allowing it to fall gently into the boxes (Figure 3). Pro- ceed slowly at first, taking care that a letter is not released until the hand is directly over the box in which it belongs. 2% 28 EMPLOYMENT PSYCHOLOGY (of - ds \ 5: - ol oi? =< Jt Se 7 EY Figure 3 5. When distributing sizes larger than 12 point lay the letters crosswise of the case, and avoid dropping large letters face down- ward. Watch carefully the sizes of spaces, as it is important that they be placed in the proper boxes. In analyzing work involving monotonous repetition, a description such as the one quoted from Link will be the end result. There will be no question of definition of duties, since the worker will be doing the same thing over and over. The best procedure for the investigator is apparently to observe the man at work and make notes on what he does, supplementing this by spending some little time in carrying on the work himself. In exceptional instances the technique of micro-motion study as developed by the Gilbreths can be employed to advantage (62). When dealing with more complicated and varied kinds of jobs, the first step is to obtain from a worker a list of all the things he does. This list is made the basis of interviews with several other workers, resulting in its modification and extension. The revised list is then discussed with other men who rank just below and just above this man. It often hap- pens that some of the duties listed belong to their jobs instead of the one under investigation. When the duties JOB ANALYSIS have been correctly listed and o.k.’d by both workers and supervisors, the next step is to determine their relative importance and the amount of time spent on each. This information is obtained by again interviewing the worker and by watching him at work. Salesmen have been studied in this way by going with them and making a time study of the different phases of their work. The third step is to find out just how the worker accomplishes each of his tasks. This again involves personal interviews with workers and observation of them at work. Either method of analysis should be supplemented by all the additional information which may be gathered from the company records and from the literature. All information should be placed in writing as soon as obtained, and should be concrete, concise, and in minute detail. IMPORTANT POINTS IN JOB ANALYSIS Confine yourself to the statement of facts. Do not overlook anything. Before deciding not to investi- gate a point thoroughly, first make sure that it is not worth investigating. Do not tell the worker how to do his work. Talk to the worker in his own language. Do not confuse the work processes with the abilities necessary to carry them on. Do not confine the analysis to the work done by the ablest man. Do not argue with worker or supervisor regarding any questionable point in the analysis. The job analysis serves as a point of departure for sev- eral other preliminary steps which will be treated succes- sively: the choice of a criterion of success, the selection of men to be the subjects of the investigation, and the analysis of abilities required for success. The first of these will be the subject of the next chapter. 20 I~ CRITERIA OF VOCATIONAL SUCCESS Importance of the choice of criteria. Thirteen suggested criteria of voca- tional success. General considerations in the selection of criteria. A criTERION of accomplishment is something which may be used as a measuring stick for gaging a worker’s relative success or failure. Such a measure of the worth of an employee to his con- cern should consist of more than the mere opinion of his supervisors. A good criterion of success is objective, fac- tual, reliable. It answers with definiteness such questions as these: Who are the. most valuable workers, and who the least valuable in a selected department? What is the order of merit within a list of salesmen? Which of the executives are outstanding successes and which could most readily be spared? Unless the records of factory or office yield dependable answers to such questions, it is impossible to determine quantitatively the results of improved procedures of select- ing and developing personnel; but where adequate measures of occupational success are to be had, the way is open for the trial of scientific personnel methods and the determina- tion of their validity. An executive can, for example, check one method of hiring with another, and learn definitely which pays best. He can find the answers to questions as to which of two methods of supervision, or of remuneration, is most effective. He has a measuring stick which is indis- pensable in quantitative studies of many vital personnel problems. From the management’s point of view, the successful em- ployee, in contrast to the unsuccessful, does more work, 1 20 CRITERIA OF VOCATIONAL SUCCESS 21 does it better, with less supervision, with less interruption through absence from the job. He makes fewer mistakes and has fewer accidents. He offers a larger number of good original suggestions looking toward improvement of condi- tions or of processes. He ordinarily learns more quickly, is promoted more rapidly, and stays with the company. His quantity and quality of output, rate of advancement, length of service, and so forth, are aspects of vocational success, each of which can be measured, expressed in nu- merical terms, and used as a criterion against which to check the validity of predictions based upon employment tests, personal history items, interest questionnaires, or in- terviews. Dependable measures of actual accomplishment, of suc- cess or failure at the job, are needed for any scientific in- vestigation in selection of personnel. Many a study of methods of selecting people for positions has led to ambigu- ous conclusions because of the inadequacy or unreliability of the criterion by which the methods were judged. All too often a research has passed through the laborious and ex- pensive stages of making the job analysis, constructing in- genious tests, and giving the tests to numerous employees, before the investigator discovered that no adequate and reliable measure of relative individual achievement on the job was to be had. The salesmen for a corporation doing a business of national scope were given a battery of tests at a series of sales conventions with the thought that the value of the tests would then be ascertained by checking the Scores against the auditor’s records of commissions earned. Later it was found that commissions were not a fair cri- terion of sales ability in that concern because of gross dif- ferences of territory and inadequate bases for quota setting. Ratings of the value of the salesmen to the company, made by the branch sales-managers and the home-office execu- tives, did not agree. No other criterion of success was available except length of service with the concern, and that measure was not considered a good one by the interested ¥ | EMPLOYMENT PSYCHOLOGY executives. The investigation had then to be abandoned for lack of a sound criterion. Early consideration, then, of available criteria of voca- tional success in the occupation being studied will enable an investigator to avoid serious pitfalls. The selection of subjects for the investigation, the determination of the abili- ties essential to success in the vocation, and the choice of tests will depend in part on the criterion of success which is adopted. If there were no problem of personnel there would be no need to make an investigation of the type we are describing. The motive for making the investigation usually comes from a large turnover among workmen, excessive breakage, low output, high cost of training new employees, or some similar problem of management. When the investigator is faced with such a problem, the criterion by which the man- agement will judge his efforts is the improvement of exist- ing conditions. The investigator is obliged to adopt as his criterion the variable which has occasioned the study. Vocational success as seen through the eyes of the man- agement may be quite different from the ideals of success which motivate the worker. The college student who sells some article from house to house in order to earn money to finish his course is not to be compared with the man of less than average intelligence who does this to support his family. Perhaps one standard of success among secretaries is matrimony. Toolmakers not infrequently shift from fac- tory to factory in order to broaden their experience and increase their trade skill; and among some groups of tool- makers it is a matter of professional pride not to have worked long for one concern. Their ideal of vocational success is a well-rounded versatility. The employer’s ideal is competency in making the particular sorts of tools he needs, with the particular equipment he provides. It does no harm for the investigator to canvass the workers as to their ideals of vocational success in order to see if these are in agreement with the ideals of the management. A lack of 22 CRITERIA OF VOCATIONAL SUCCESS harmony in vocational ideals may turn out to be the crux of the employment problem. Studies of the professions bring out more clearly the social importance of the problem of vocational success. What constitutes success as a doctor, a journalist, or a clergyman? Foster (53), in his study of the careers of the Harvard Class of ’94 as one means of estimating the merit of the plan of requiring a specified amount of concentration and scattering of elective courses, used an original and appropriate method of gaging success. The alumni whom he studied were engaged in a great variety of occupations, each perhaps with a different standard of success. His first problem was to find a basis for comparing all these men with each other. The criterion of success which he adopted was that the alumnus be rated successful by at least two of the three judges who knew the class intimately, each judge having been asked to designate “those men who had achieved the kind of success which he would be glad to have Harvard College promote, if possible, by the administration of the curriculum.” Such an approach although relying on subjective opinion makes at least a beginning in the critical consideration of success in the professions. In a larger sense, vocational accomplichment is a function of our civilization. Standards vary from one age to another and from one continent to another. The ancient Greek or the modern Hindu cannot be judged by the same standards as the Canadian. Even such closely related stocks as the English and the American show important differences in their vocational ideals and aspirations. Such considera- tions will, however, lead the investigator far afield. Atten- tion must be directed toward the criteria which the business or industry itself can provide and which it will consider sound. SUGGESTED CRITERIA OF SUCCESS Mention will here be made of 1 3 kinds of criteria of vo- cational accomplishment by which psychological tests and 33 EMPLOYMENT PSYCHOLOGY other measurements of ability may be evaluated. If more than one reliable criterion may be had, it is important that ihe measurements be checked against each of them sepa- rately. Wechsler (214), for example, in his study of taxi- cab drivers, discovered that one of his tests predicted the number of accidents the driver would have and another the wages he would earn. The use of only one of these criteria would have made his study half as valuable. 1. Time required to train the employee. In many occu- pations, preliminary training is given the new employee before He is placed at work or during the early stages of his employment while he has but little proficiency. The cost of this supervision or special training is usually con- siderable, and other things being equal, the sooner the new employee acquires the necessary information and skill the greater the saving to the firm. Where the employee is re- quired to reach a certain level of proficiency before being allowed to enter active service, the criterion may be the time required to reach the prescribed level. Some employers guarantee to workers or salesmen a minimum rate of pay, to which is added a bonus or a piece-rate payment when the employee’s performance warrants it. Usually the employee is costing the concern more than he earns until this stage of proficiency is reached. In this case, the time required to learn the job well enough to be earning a bonus is a use- ful criterion of initial vocational success. 2. Standing in corporation schools. If the preliminary training takes place in a vestibule school, apprenticeship course or other corporation school, the grades received in this school may be used as a criterion for testing the effec- tiveness of selective methods. The instruction is ordinarily of the practical sort which concerns itself with the particu- lar operations required for the job, and the criterion often becomes the rating of the teacher on the skill which the student displays in these operations at the end of the course. When possible, actual measure of this skill should be ob- 34 CRITERIA OF VOCATIONAL SUCCESS 5 tained at the conclusion of the training period and used in place of the instructors’ estimates. 3. Quantity and quality of output. In spite of the pre- cautions their use requires, measures of quantity and quality of output are on the whole the most useful criteria. These measures may be in terms of the quantity and quality of work turned out per unit of time, or of the amount sold (when salesmen are being compared), provided, of course, the workmen are doing or making or selling the same thing, have had comparable experience, and are not limited or restricted in their output by management or working con- ditions. These measures of production should be expressed as the average amount produced over an extended period of time. In developing improved tests for the selection of post- office clerks and railway mail clerks, the Director of Re- search of the United States Civil Service Commission, Dr. L. J. O'Rourke (124), gave much attention to the selection of criteria—measures of actual ability of the men on the job. He arranged with the Chicago post-office to weigh the first-class mail distributed daily by a representative group of 124 clerks during a six-month period. The average num- ber of pounds distributed by each clerk, together with the time in minutes required to do it, constituted his first cri- terion of ability. A second criterion was the record in a monthly examination or test to determine how accurately and quickly each clerk can distribute into his own distribu- tion case. To these objective measures of ability were added a third criterion—foremen’s ratings. The combined criteria gave a satisfactory measure of actual ability as a mail sorter, and made possible the validation of the new type of civil service test by means of which 60,000 to 80,000 applicants a year are now examined, more easily, accurately, and fairly than before, with much less expense to the gov- ernment and with measurable improvement in the average ability of applicants selected. When salesmen are being compared according to the 3c EMPLOYMENT PSYCHOLOGY number of units sold, the investigator must bear in mind contributory variables, such as difficulty of the territory assigned to the salesman, possible prejudice against him in that part of the country, market for the product in his ter- ritory, extent to which that market has been covered, amount and character of competition, time the salesman has been on the job, and his ability to sustain his sales record. Some executives have theories as to the type of person who ought to be able to sell for their company (such as tall men, or blonds, or college men) and use special in- centives with salesmen of this type. All these factors tend to reduce the reliability of a salesman’s production as a true measure of his ability. The contributory variables are scarcely less important with workers engaged in routine mechanical tasks. As the raw materials supplied to them vary in quality, it becomes harder or easier to do the work rapidly and accurately. Output may be limited by speed of machines or flow of materials. Interruptions in the flow of work, breakdowns, and other contingencies are difficult to allow for. Stand- ards of inspection are sometimes relaxed or stiffened. Heat- ing, lighting, or ventilation may be radically altered. The supervisor may be having trouble at home which makes him unreasonable in his demands on the workers. Any of these complicating circumstances may initiate a tremendous upset in group morale which shows itself in marked varia- tions of quantity and quality of output even though the financial incentive remains the same. The investigator must watch all such possible variables and hold them con- stant or make appropriate allowances. Foremen have been compared as to the amount which each is able to produce in his department, but this is ordi- narily difficult because of differences in size and nature of work of departments. The average production per man has sometimes been used as the measure of the foreman’s ability. 4. Performance in standardized examinations. Often it 26 CRITERIA OF VOCATIONAL SUCCESS an will be found that usable measures of quantity and quality of output of individual workers are not to be had. The investigator may save time in the long run by pausing to devise and standardize performance examinations—typical sample jobs by means of which he can determine how ably the workers can do their work. The scores made on these standard examinations will serve as criteria of their voca- tional ability and success. The investigator must be sure the piece of work selected for the examination is repre- sentative; that variables, such as working conditions, ma- terials, tools, and incentives can be held constant; and that he can get a measure of the performance in terms either of objectively definable gradations of quality of the product, or of time required to make it. The duties of an office worker are often so varied that the records of the manage- ment yield no suitable measures of her ability. To get one measure of her success she may then be examined and compared with the other workers in actual performance by use of standard tests in typing, taking dictation, comput- ing, filing, or whatever her chief duties are. Standard tasks have similarly been used as examinations of factory workers, draftsmen, dentists, bacteriologists, and many other semiskilled, skilled, and professional workers. A valuable lesson is to be learned from the experience of investigators of college personnel problems, particularly in studying the selection of students for admission, their classification and assignment, and the prediction of their relative success in different types of college work. The criteria first used were the marks or grades the students got in their courses. The student was deemed to have succeeded or failed according to whether he received enough passing marks to enable him to stay in the college or course he had chosen. But the available examination marks were usually found not to furnish a sufficiently sound criterion of what the students were accomplishing. It be- came necessary to devise the new-type examination, much more searching and comprehensive, much more definite and 37 EMPLOYMENT PSYCHOLOGY exacting in its demands, and at the same time much fairer to the students than the old type of school examination. It has taken some years to develop the technique of con- structing the new-type examinations, but the result has been that those university departments which use them are now able to measure the relative success of their students more objectively and reliably. In so doing, they provide the investigator of educational personnel problems a vastly better criterion than he formerly had. When psychologists turned aside for a time from the devising of psychological tests for students, and developed this improved technique for measuring actual achievement in school work, they not only did a service to education, they made possible later advances in psychological research. So also in industrial research, time may be gained in the long run by pausing to develop good standard examinations of actual skill and accomplishment on the job. These are almost certain to provide the best single criterion of vocational success, where the records do not already furnish suitable measures of individual output. 5s. Accidents and loss due to breakage or claims. These are good criteria for use with persons who must handle materials which are fragile or easily lost. Freedom from accidents has also been used to measure success as a street-car motorman or taxicab driver. Such a criterion may be combined with measures of quality and quantity of output. 6. Salary. A man’s worth to a firm (and accordingly his vocational success) is apparently shown by the salary which that company is willing to pay him. Some oi the variables which tend to make this criterion unreliable are the employee’s length of service with the firm, his family relationship to some of the high executives, his ability to impress the management with his worth, his enterprise or his lack of tact in demanding a salary increase. Many of these variables need not be isolated if the investigator 28 CRITERIA OF VOCATIONAL SUCCESS wishes to measure the individual’s success in terms of abil- ity to get along in the world. The investigator must take into account such matters as the cost of living in the various localities in which the men are employed, the relative size of the budgets allowed to each department, and similar varia- bles. 7. Commissions and bonuses. Commissions and income from piece-work are more flexible indexes of occupational ability than salaries. With compensation on this basis a man’s income is usually proportional to his output. The investigator using this criterion will be confronted most forcibly with the problems of interest and uniform motiva- tion. Earnings should be averaged over a long period of time, and the cautions mentioned under “quantity and quality of output” should be observed. Bonuses when added to commissions exaggerate differences in ability but do not change the rank of workers in the criterion. 8. Length of service or stability on the job. The firm may meet with little difficulty in locating men who after instruction are capable of complying with the minimum requirements of vocational efficiency, but the turnover may be so great that the cost of training new men is an undue item of expense. The problem for the investigator then becomes one of measuring a certain stability of tempera- ment, or contentment in that type of occupation, as well as ability to do the required work. If he accepts the challenge to measure stability of temperament or to pre- dict contentment on the job in question, he must not use as subjects those who leave the company for reasons be- yond their control. The Scovill Manufacturing Company has used to excellent effect in its studies of employment tests for factory operatives the criterion of length of ser- vice. For the purposes of the research an unsuccessful employee is defined as one who leaves the company within six months of hiring for any reason other than lay off, death, or illness not connected with the occupation. An employee 30 EMPLOYMENT PSYCHOLOGY who can and does hold down his job for six months is classified as a success at that job. Satisfaction in the job is an asset to be sought. If it can be shown that there is a close relationship between length of service and efficiency on the job the problem reduces itself to simpler terms. If there is no relationship between the two, and if tests select the efficient, the problem of ap- propriate incentives to remain with the company confronts the management. Frequently the problem finds its solution in an analysis of the circumstances under which the men leave the em- ploy of the company rather than in an analysis of their temperament. Check lists of reasons for leaving and sug- gested remedies are given by Bezanson (8) and Benge (7). Too much credence should not be given to the employee’s own reasons for leaving, as it may be difficult for him to formulate them, he may wish to leave an opening for a return to the company, or he may rationalize, that is, find a plausible reason in place of the real one which he does not care to admit even to himself. The foreman’s reasons for a large turnover in his department should also be ac- cepted with reservations. Length of service as a measure of success may have nega- tive value in an occupation which is considered merely as a training ground. Here brevity of service before transfer or promotion indicates vocational accomplishment. 9. Advancement in the firm. Advancement may be measured in two ways, either of which may be considered a criterion of success. The investigator may choose as his criterion the mere fact that the worker is promoted, or he may choose the rapidity with which the worker is promoted. In either case he selects as prominent successes in the occu- pation those who have been promoted because of the ability they have shown in that occupation, regardless of whether or not they have been promoted to positions requiring the same abilities. But if the management has advanced them not because of the ability they have displayed in that par- 40 CRITERIA OF VOCATIONAL SUCCESS 4° ticular occupation but because it considers that occupation as a source of supply for men for more important jobs, the investigator must consider the abilities of these men with reference to these more important jobs rather than to the source-of-supply job. This situation is illustrated where men are required to spend a certain amount of time as shop workmen in preparation for positions as minor execu- tives, personnel men, salesmen, or designers. 10. Degree of responsibility. Another criterion is the responsibility which the men are required to shoulder. This may be measured in terms of the number of subordinates they have, the value of the product they handle, or, more particularly, the nature of the supervision they are required to give. 11. Membership in professional societies. Membership in professional societies, entrance to which is based on merit, is a legitimate criterion of success. This should in- clude local organizations as well as the national engineer- ing societies, honorary fraternities, and mention in “Wko’s Who” or “American Men of Science.” 12. Trade status. If a classification of workers into levels of skill is recognized by the unions or by the manage- ment, these gradations of trade status may be used as a criterion. The best example of the use of this criterion is in the validation of the trade tests developed by the United States Army during the World War, when groups of men known by their employers or fellow-tradesmen to be nov- ices, apprentices, journeymen, or experts in the trade in question were chosen to serve as subjects in determining the diagnostic or differentiating value of each item in the proposed trade test. 13. Ratings. Ratings may be made by immediate su- periors, by teachers in corporation schools, or by fellow workmen. Because ratings are less objective and reliable than most of the preceding measures of success, they should not be used if these others are to be obtained. The whole x} EMPLOYMENT PSYCHOLOGY subject of rating scales is taken up in Chapters IX and X. Ratings are more often useful as criteria of success of executives and salesmen than of operatives engaged in rou- tine tasks requiring manual skill. Objective criteria of manual skill are more likely to correlate with objective per- formance tests, and ratings are more likely to correlate with tests of personality. Ratings may be made on the single characteristic of suc- cess in the occupation; or they may be distributed over a combination of several abilities which are deemed neces- sary to success in the occupation, and the sum used as a judgment on vocational success. But the safer procedure in obtaining a criterion of success is to ask for ratings on success, using the ratings on component abilities required for success at a later time as checks on the investigator’s analysis of these abilities. If ratings are adopted as criteria, use should be made of a final rating which is the unweighted average of the independent judgments of at least three intelligent unbiased men who are thoroughly acquainted with the persons to be rated and who are given plenty of time in which to make their judgments. The reliability of this criterion should be determined by asking each judge to rate the men again at a later date. Executives who are skeptical about psychological tests sometimes insist on the use of their personal estimate of their subordinates as a measure of success. In this case one compensation attaches to the use of ratings as criteria, namely, that if a correlation is established between criterion and tests, the proof of the value of the tests is brought home in a very personal way to the executive who makes the rat- ing. GENERAL CONSIDERATIONS If the investigator believes that any one of the available criteria taken alone is inadequate to express the employee’s vocational accomplishment, there is no reason why several 42 CRITERIA OF VOCATIONAL SUCCESS 43 of them should not be combined or averaged. The series of measures will have to be adjusted before averaging in order to avoid giving undue weight to any one of the criteria. The investigator may weight each of the contributing cri- teria in accordance with its importance as he judges it. Statistical precautions to be borne in mind are discussed in Chapter XVII. A test may predict one sort of vocational accomplish- ment and not another. Hence, when more than one cri- terion is available, the better procedure is to correlate the tests with each of the several criteria independently. If the criteria are themselves not closely related, the probability of finding a high correlation between a test and a criterion is increased. If several criteria are used independently it will be necessary to make for each of them a list of essential abilities before constructing tests. The care demanded in the choice of a criterion of success is well demonstrated in Snow’s study of tests for the selec- tion of taxicab chauffeurs (170). His problem was to cut down the number of accidents by devising tests that would predict relative freedom from accidents. So he turned to the accident record for his criterion. He found that acci- dents could be classified according to the following types: 1. A type due to recklessness. (In this article we distinguish between carelessness and recklessness. Carelessness is a general word; it implies a lack of care for one reason or another, such as driving without the brakes in proper condition. But recklessness connotes a more or less deliberate lack of care. It is the type of behavior for which we tend to say there is no excuse.) The reck- less accident is one such as is caused by speeding, passing traffic lights, or running on the left side of the street. 2. A type due to carelessness. This is one which results, for example, when the driver goes out with a car which has a faulty steering apparatus or too much play in the wheels. He does not do so deliberately, but merely does not take the pains to learn the condition of his car. 3. A type due to emotional instability. This is the type of accident in which another car suddenly careens around the corner. The driver could avoid an accident by immediately applying the 3% x 44 EMPLOYMENT PSYCHOLOGY brakes. However, he loses his presence of mind and there is a collision. 4. A type in which the driver has an accident because of a lack of foresight. He is never conscious of the possible movements of other vehicles or pedestrians. He can drive all right, does not take chances, does not get abnormally excited in emergencies, but merely lacks the ability quickly to size up a situation which is at all out of the ordinary. This is the driver who, for example, because there is no obstructing traffic, comes speeding down a street, beside which several small children are unconcernedly playing. One of them steps in front of him and is injured. In sizing up the street situation, this driver should have taken into account the children on the sidewalk. In traffic he may run into the rear of the car ahead because he does not foresee from the movements of the other cars and the entire situation the possibil- ity of that car stopping suddenly. When “riding the meter” (riding in front with the driver) with all types of drivers I discovered that good drivers invariably possessed this ability to see ahead, to feel what the other person is going to do. s. A type due, for lack of a better mame, to physiological defects. Here is the driver whose reaction time varies consider- ably from the average, who is easily distracted, has an abnormally short range of observation, or poor ability to estimate the speed of objects as well as the distance between moving objects, or who has little muscular resistance. These five types are those with which the psychologist is con- cerned. In addition, there are a sixth, seventh, and eighth type of accident. 6. A type due to physical defects, or disease of various forms. We had nothing to do with the prevention of this type of accident inasmuch as the medical staff of the Yellow Cab Company passes on all applicants. 7. A type due to a temporary prepossessing idea. The driver who has domestic difficulties or sickness in his family is subject to these accidents. His mind at moments instead of being on his driving is occupied by these difficulties. T he welfare department aims to prevent such cases. 8. A type which is unavoidable. This is the accident due to such causes as a sudden breaking of the steering gear, or unusually bad street conditions, such as icy pavements on which, no matter how much care is taken, an accident is likely to occur. There are others in this class, too—accidents which it is not possible for LL CRITERIA OF VOCATIONAL SUCCESS 23 the driver to avoid. The car maintenance department attempts to minimize those due to mechanical defects. If the purpose of the investigator is to reduce the number of accidents by better selection, he must use as many cri- teria as there are types of accidents. He has to construct tests independently for measuring the ability required to avoid each type of accident. It is clear that one test or a battery of tests cannot be expected to predict accidents of all types. One test may predict accidents due to emotional instability and another test accidents due to some physio- logical defect. Snow used each of the first five types of accidents as his criteria for the evaluation of tests. But he was still faced with difficulties. To quote his own words: The accident data on each driver are obtained in this way: When a driver has an accident it is recorded against him. All accidents, however, are not equal in the degree to which the driver is the cause. There are several variables. The driver contributes to some accidents to a greater degree than to others, For instance, a driver hits a telegraph pole which is rotted; the pole breaks and no one is injured. Another driver has the same sort of accident, except that the pole he strikes is not at all rotted and his passenger is injured. The consequences are different, but the cause is the same. Then there is the factor of time—the length of service of the man, and his previous experience. One driver in his first two months has, say, two accidents, while another has but one acci- dent in three months, then has three in the next two weeks. In such a case, one must remember that some drivers have accidents relatively often at the start, but relatively seldom after a short period. The reason may be that these men have never driven before, or that it takes them longer to adjust themselves to the new conditions, but once they do learn they make very good drivers. On the other hand, some men have relatively no accidents at the start, but later develop a tendency toward them. Perhaps they are quick learners, or have had previous experience in driv- Ing, and once the newness of the position is worn off, they soon let up and become less careful. But how is one to classify and evaluate accidents under such conditions? There is also a weather factor. A driver who during a heavy ic 2 EMPLOYMENT PSYCHOLOGY rainstorm knocks the rear wheel off another car is obviously not so much the cause of the accident as one who has the same sort of accident on a bright, clear day. Likewise, icy pavements are often the cause of accidents which are inexcusable in better weather. What is careful and slow driving in one case will not be so in the other. That, too, must be considered. There are even other complicating factors, such as a careless pedestrian stepping in front of the cab. Here the driver is not the cause, yet technically he has had an accident. There may be some unknown defect in the car mechanism which may cause an accident. Again the driver is not the cause, yet technically he has had an accident. These, then, are the variables: To what degree is the driver the cause of the accident? Should this be modified by the time factor, or by the weather conditions? These are very pertinent questions which must be dealt with in establishing criteria with which to standardize the tests, to test them (correlate them with actual fact), and to prorate them (determine their weight in establishing a final score). Even if we determine whether or not the driver is the cause, it is impossible to discriminate finely the extent to which he is the cause. This cannot be expressed numerically, nor can it even be accurately expressed with a rating scale. The legal department of the company, since it gets all the original information—reports from the passengers and witnesses, as well as from the drivers—attaches one of the three following opinions to each accident: gross negligence, unavoidable, question- able. Tt is this judgment we use in determining to what extent the driver is the cause of the accident. Snow’s success in applying the psychology of vocational selection to the problem of choosing cab operators rests not wholly on his ingenuity in devising and adapting tests. His critical scrutiny of his criteria and his strict adherence to them in planning his research and in evaluating its results have likewise been important. A criterion may be expressed in terms of a rough twofold or threefold classification of the workers, or in terms of such fine units of measurement that each man is distin- guished from the others in accomplishment. If the criterion, for example, is the amount sold in the course of a year, the men may be separated into two groups: those who sold 46 CRITERIA OF VOCATIONAL SUCCESS IY a given amount or more, and those who sold less than that amount. Or the men may be ranked in order of their sales, in which case there may be as many groups as there are men. Use of fine distinctions in vocational accomplishment is not recommended unless the criterion is very reliable. (For methods of ascertaining reliability, see Chapter XIII.) It is quite as important to determine the reliability of a criterion as the reliability of a test. Once the criterion has been selected, care should be taken throughout the study to keep it free from the influence of extraneous factors. Some of the disturbing variables to be guarded against have already been mentioned. If ratings are used as criteria, they should be made without a knowl- edge of the tests to be used, otherwise the rater is apt to have in mind success in these tests rather than vocational success when he judges the workers. The criterion must be strictly adhered to in selecting the subjects for measure- ment and in determining the abilities necessary for success in the vocation. The foregoing criteria are standards or measures of the vocational success of the individual worker. In some in- dustrial laboratories in Germany criteria such as these are applied not to the workers separately, but to a whole de- partment. Tests are constructed to measure the abili- ties which the investigator deems important, and without waiting for validation, applicants who score high in these tests are hired. When the men selected by this method form a considerable part of the working force of the depart- ment, the investigator determines how much the efficiency of the department as a whole has improved. This is mea- sured by changes in the number of men required to do the work, changes in the output of the department, reduction in breakage, reduction in cost of training and turnover, and so forth. This method has little to recommend it scientifically. It makes unwarranted assumptions regarding the ability of = EMPLOYMENT PSYCHOLOGY psychologists to devise tests for definite abilities. It does not enable the investigator to separate the wheat from the chaff in his series of tests. It leaves room for grave doubts whether the tests are responsible for any improvement that is observed. Correct scientific procedure requires that each test be validated separately on individual workmen of known vocational efficiency. Group criteria become sec- ondary and may be used later to demonstrate to the manage- ment the economic savings effected by use of the tests. The choice of a criterion of occupational success will vary with the use to which it is to be put as well as with the extent and dependability of available records. Mea- sures of output are by all odds the best criterion if the worker’s output is conditioned mainly by his own ability and persistence, and not by factors outside of his control. Time required to learn the job is a good criterion where training is expensive. Length of service, as expressed in terms of ability and willingness to hold down the job for at least six months (or some other suitable period) is a clear-cut criterion of success which has proved extremely useful in selection studies. In jobs where high proficiency is essential, the worker’s measured performance in a well- standardized trade test or searching proficiency examina- tion is the best gage of his actual ability. Ratings or super- visor’s estimates of relative success are, on the other hand, the least dependable of criteria, to which recourse will be had reluctantly, and only when the management cannot provide a more reliable and objective measure of occupa- tional accomplishment. When the criterion or criteria have been selected and have been approved by the executive most interested in the installation of the tests, the next step is the selection, on the basis of these vocational criteria, of men to serve as subjects for the investigation. 48 IV 40 CHOICE OF WORKERS TO BE STUDIED Considerations in the choice of workers for study. Some disturbing variables to be eliminated. Newly hired versus old employees. THE criterion of vocational accomplishment having been settled upon, the investigator proceeds with the selection of workers or applicants whose abilities he is to measure. The choice must be made with due regard to the adopted cri- terion, so that a sufficient number of cases representing high, low, and average degree of success on the job will surely be included. REPRESENTATIVE SAMPLING The criterion which investigator and management have jointly agreed to use—as described in the previous chapter —may be output, length of service, rate of advancement, commission or bonuses earned, or a combination of these and other measures of actual vocational accomplishment. If this criterion is not kept in mind when the decision is made regarding the particular individuals whose abilities are to be the subject of investigation, the study is in danger of leading to inconclusive results. It is obvious that no relationships between test scores and actual abilities on the job can be established if all the workers tested are equally good at the job. In order to learn how well the tests or other measures differentiate the successes from the failures, good and poor workers must both be included. The sub- jects should be representative samplings within the criterion. Equal numbers of successes and failures should be chosen if possible; or if the criterion is threefold—successful, aver- EMPLOYMENT PSYCHOLOGY age, or failure—there should be an appropriate proportion of subjects in each group. If the criterion is a continuous variable, such as quantity of output or average school mark, a representative dis- tribution of workers throughout the range of this criterion should be selected. If all the men engaged in the occupa- tion were to be judged by the criterion which has been fixed upon, the resulting distribution would resemble the normal probability distribution; that is, a great many workers would cluster about the average of the group, while fewer and fewer cases would be found as the lower and higher extremes of ability were approached. The problem of the investigator is to discover useful methods of measuring those abilities of the workers which are related to the criterion, so that those who stand high in the mea- surement of the ability will as a general rule also stand high in the criterion, the measure of actual accomplishment on the job. It is important then to obtain test records on workers known to represent various degrees of vocational accomplishment expressed in terms of the criterion which has been adopted. If only successful, competent workers are available at the time, the alternatives are to test also a wide sampling of the population not engaged in that occupation, to see whether significant differences can be found between these highly selected workers and the general run of the work- ing population; or to accumulate over a long period of time test records on new employees as they are hired. The investigator should bear in mind when choosing his subjects, that a wide range of vocational ability will tend to increase the correlations with the tests, and that a large number of cases will tend to increase the reliability of these correlations. It is impossible to make a generalization re- garding the number of cases necessary. Investigators have generally hesitated to publish results where less than 30 cases are used. But it is to be borne in mind that the num- ber of cases required depends on the reliability of both Jo) CHOICE OF WORKERS TO BE STUDIED 51 criterion and tests and on the conditions of the investiga- tion. The investigator must make certain that all of his sub- jects have achieved their standing in the criterion by the exercise of those abilities considered necessary for success at the job, and not through chance circumstances or special advantages, DISTURBING VARIABLES In selecting the workers to be measured, extreme varia- tion in age or any other gross disturbing variables which are apt to have an effect on test performance should not be allowed to complicate the problem. These variables should be controlled by spreading them evenly throughout the dis- tribution of classes of vocational accomplishment, or by eliminating as subjects those who represent extremes in the disturbing variables. Examples of a few of these variables may be cited. Sex. If men and women are both employed in the occu- pation, it will be necessary to examine the results for sex differences, and if need be, to evaluate the tests separately for the two sexes. The importance of this point will be made obvious if the reader will look into the literature on sex differences in test performance (see 73, p. 112). In such a test as card sorting only 24% of the men reach the median score of the women, whereas in speed of movement (finger and arm) 71% of the men reach the median score of the women, Age. Differences in the ages of the subjects should not be so great as to have an influence on their comparative test performances. If proper sampling does not permit a limited range of ages, then the data should be examined to see if some correction should be made for great differences in age. Sensory defects. If sensory acuity is a factor in success in the vocation, it should be determined in a special exam- : EMPLOYMENT PSYCHOLOGY ination and should not be allowed to intrude itself on tests for special abilities. The most general sensory disturbances with which the investigator will have to deal are those relating to vision and hearing. The job specifications will inform the examiner of the occupational demands in the sensory field. If he is testing applicants he will not waste his time testing those who fall short of this standard. If he is testing employees, it might be presumed that all will come up to the standard. But this is a hazardous assumption. Even in some well- equipped personnel departments the data obtained regard- ing the visual acuity of applicants are inadequate. Some industrial physicians do not record corrected vision but measure only the acuity of the unaided eye. The informa- tion they do record is indispensable as a precaution against fraudulent claims for subsequent injury to eyesight. But what the employment manager wants to know is, how clearly the worker can see with the glasses he ordinarily wears. He also wants to know how well the worker can see without eye-strain, not at 20 feet, but at working dis- tance from his blue-print, drill press, micrometer, or draft- ing board, and under conditions of illumination similar to those prevailing on the job. If information of this char- acter is not already available, the investigator will ordinarily do well to see that it is obtained. He may find that differ- ences in keenness of vision, hearing, touch, or muscular sensitivity have much to do with relative achievement on the job in question. On one factory job where 140 young women are tending noisy machines, about 8o are seasoned and successful opera- tors. In the top quarter of this group, 50% of the girls have better than normal vision in both eyes; in the lower quarter, only 6%. And yet on casual inspection there is nothing to indicate that on this job more than on the great majority of factory operations, superior vision would con- tribute greatly to an experienced worker’s higher average earnings and output. 32 CHOICE OF WORKERS TO BE STUDIED 53 Motor defects. Organic or functional motor disturbances, such as tics, should receive the same attention as sensory disturbances. If occasionally a successful employee is found who suffers from such disturbances, he should be recognized as an exception to the general rule of health among the employees. If the subjects of the experiment are to be representative of the great number, such an excep- tional individual may safely be ignored. Literacy. Needless to say, if the tests require a language response, no one who is illiterate should be chosen as a subject. Practical literacy means more than ability to write one’s name. It means command of the English lan- guage roughly equivalent to the completion of five years in the public schools. Where many of the workers are for- eigners or otherwise not sufficiently familiar with written English, the investigator must either exclude them from among his subjects by a preliminary test of reading ability, or else make special adaptations of any test forms or direc- tions which call for reading or for verbal written response. The latter procedure is necessary in many American fac- tories, where a large fraction of the competent employees are apt to be unfamiliar with written English. While differences in literacy, motor defects, age, and sex have been emphasized, the investigator will be alert for other variables, such as extremes of health or education. The point cannot be stressed too much that if these gross disturbing variables have an influence on test performance they should be nullified by elimination. If this step is impossible, then the method of partial correlation may be employed to make the required correction; but it must be borne in mind that the necessity for the use of partial cor- relation in making corrections is often an indication of a poorly planned experiment. Use is sometimes made of multiple and partial correlation to obtain the best combina- tion of scores; but this, however, is a different matter from the elimination of gross variables. Multiple and partial correlation will be explained in Chapter XVII. rT, ¥ EMPLOYMENT PSYCHOLOGY NEWLY HIRED VERSUS OLD EMPLOYEES Measurements of abilities may be evaluated by compar- ing them with the vocational accomplishment of seasoned employees, or by obtaining these measurements on new workers at the time of hiring, and postponing the valida- tion until their vocational accomplishment has been de- termined. Either method is justifiable, but there are sev- eral points in favor of the use of newly hired employees. The most important is that by examining the men at the time of application the influence of special training in the firm may be eliminated. The abilities measured will be those possessed by applicants on whom the tests will be used in future if they prove valid. In other words, the validation of the tests on new employees approaches nearer the situation under which the tests are eventually to be used. A second point is that the range of ability among applicants will be “greater than among old employees and this will tend to increase correlations of tests with the criterion and give a truer picture of the value of the tests. Indeed, in some occupations the elimination of the ill- adapted goes forward so persistently during the early weeks of training or of employment on the job that the body of experienced workers constitutes a highly selected group, much too uniform in ability to serve satisfactorily for an experiment in determining the correspondence between test performance and relative proficiency in the occupation. In the third place, foremen and managers frequently object to the disturbance and expense of taking men off their work to be tested. This objection does not hold with applicants at the time of hiring. Finally, there may be too few old employees on whom to validate the tests. If applicants are used, cases may be accumulated until sufficient data are at hand. The most important point in favor of testing old em- ployees is that the delay incident to the testing of new workers is avoided. It may require one or two years to S4. CHOICE OF WORKERS TO BE STUDIED 53 accumulate a sufficient number of cases of new employees. Moreover, with old employees the criterion of success may be obtained immediately, whereas nothing can be known about the newly hired workers until they have been at work for a considerable period of time. Meanwhile interest in the investigation may grow cold. In testing old employees it is important to avoid the measurement of technical training acquired since entering the service of this company. If, however, some of the tests are shown to measure abilities acquired by experience on this job, they may nevertheless be used on future ap- plicants on the assumption, subject to verification, that an applicant who comes to the firm already well equipped with these abilities, regardless of when or where he acquired them, will be more acceptable because he will require a shorter period of training than one who lacks these abilities. Difficulties in this regard will be diminished if the in- vestigator uses only newly hired workers as subjects and compares their standings in tests as obtained at the time of application with their vocational success as later demon- strated, or if he uses old employees as subjects but elimi- nates from his tests material of a technical or informational sort. Scott (165) suggested the inclusion among a group of applicants of a “ringer” whose identity is unknown to the investigator. The ringer is known by the management to be either an exceptionally good worker or an exceptionally poor one; and it is assumed that in one case he should test high and in the other case low, compared with the group of ap- plicants. Of course, no conclusions can be drawn from comparisons of a group with but one or two individuals, and this procedure has nothing to commend it except the added zest which the management finds in the experiment. As an extension of this method Scott proposed comparing test scores of a group of proved experts and a group of applicants. Those applicants who test the same as the experts are then considered favorable material. The un- - EMPLOYMENT PSYCHOLOGY certain point about this procedure is that nothing is known of the tested ability of the proved failures. They might make the same scores as the experts! No differentiation in test performance between experts and failures is proved. We may then conclude that the best way to choose sub- jects for testing is to select workers already known to be competent or incompetent, provided the extremes of ability as well as the middle ranges are available in sufficient num- bers, and provided the group is in other respects sufficiently homogeneous, or sufficiently large to make possible a statis- tical control of disturbing variables. Where the number of experienced employees is too few, or too uniform in ability to yield a sufficient range in the criterion of vocational success, measurement of new workers at the time of employment is indicated. Except for the disadvantage of protracting the investigation, this procedure will commend itself for the reasons already stated. Applicants should be hired for experimental pur- poses regardless of their test scores. GENERAL CONSIDERATIONS No man should take the tests under compulsion. If any- one fails to respond to persuasion, the investigator will do well to let the matter rest. When the Jthers have been tested, the obstinate person is apt to ask for an examination. Any person should be rejected as a subject who is already familiar with the tests or other measuring instruments. Occasionally a supervisor or other minor executive whose men are being examined will ask that he also be given an examination. The investigator will find it advisable to re- fuse all special requests, with the explanation that since the tests are still in the experimental stage and nothing is definitely known of their value, the scores will be meaning- less. The scores made by individual workers during the experi- mental period of test development and standardization 56 CHOICE OF WORKERS TO BE STUDIED 57 should be kept strictly confidential. They should not be made available to employment interviewers or to supervisors who might attach unwarranted significance to them. It is standard practice in some factories such as those of the Gen- eral Electric Company, to tell an employee his score if he asks it, but to tell no one else without his permission. Only after validation of the test is complete, is use of it made in the employment office. The investigator has at this point chosen a number of workers representing different levels of vocational accom- plishment to serve as subjects for his study. He should go over the tentative list of abilities which is one result of his job analysis and make a further check of its correspondence with the criterion and with the subjects he has now chosen. When this has been done, he must consider the methods whereby these or other significant abilities may be measured in order to determine their relationship to the criterion of success. Our attention then turns to a consideration of the bases for choosing the measuring instruments whose relia- bility and value in forecasting success at the job are to be ascertained. r ANALYSIS OF THE WORKER Topics for a man analysis. Analysis of abilities. Innate and acquired abilities. General and specific abilities. Relative importance of abilities. AN investigator when commencing a study in vocational selection first gets his bearings by making a complete job analysis. This is a comprehensive survey which gives in detail the duties of the worker and the conditions surround- ing the job. Such a ground-work of information enables the investigator to select criteria of success which are not only reliable but also satisfactory to the management. With these criteria in mind the investigator then chooses workers or new employees to serve as the subjects of his investiga- tion. These workers he will examine in various ways by means of tests, rating scales, and questionnaires. The measurements or test scores will eventually be compared with the known vocational success of the workers in order to determine the validity of the examinations for selecting potential successes. For his guidance in constructing the examinations the in- vestigator should assemble a list of abilities and personal characteristics which he considers important for success in the vocation and which he hopes to be able to measure by his examinations. The construction of such a list is usually considered a part of the job analysis. It issues in a set of personnel specifications. In this type of investigation, how- ever, the analysis of abilities and characteristics plays an unusual réle and is a step distinct from the job analysis. It differs from the usual job specifications in these respects: it is much more minute and complete; it is made with a view to a definite limited conception of what constitutes accom- Vv £8 ANALYSIS OF THE WORKER plishment in the vocation; it is made with a group of workers for observation who are well chosen with regard to this con- ception of success; and it is made to serve as a basis for the construction of scientific examinations rather than for the immediate use of the employment office. A warning is here in place. The list of essential abilities which the investigator assembles is a tentative one, for the purpose of suggesting lines of approach in the construction of examinations. He cannot state dogmatically that the list is correct, nor can he assert that each examination he later constructs will yield accurate measures of these particular abilities. If he knew all this there would be no neces- sity for carrying through the rest of the investigation. Un- fortunately, such omniscience is denied to most men. The scientific investigator must patiently seek examinations whose records bear such a close statistical relationship to success in the vocation that they may be used in the em- ployment office in selecting applicants with assurance that future vocational success is closely predicted. The investi- gator cannot choose these examinations casually. Such a procedure he would find extremely wasteful. He must have guides to follow. Consequently he does not decide on the examinations to be tried until he has analyzed the workers on the job and assembled a list of points in which he has reason to think the good workers differ from the poor ones. This list he uses as suggestive material in constructing examinations. The validation of the examinations as selec- tive instruments is ultimately accomplished by comparing their results with the criterion of vocational accomplish- ment, not with the list of abilities. The process of analyzing the worker is similar to that of analyzing the job, but the investigator focusses his atten- tion on the worker rather than on the work. He selects for observation those workers who are later to be examined by means of the tests or other measuring devices he develops. To give an outline which will fit all vocations is as im- 50 EMPLOYMENT PSYCHOLOGY possible as in the case of job analysis. This chapter offers only a suggestive classification of the most important points to be covered by the analysis as a basis for selection of tests, rating scales, and questionnaires. The items are grouped under the following eight headings: 1. Personal data 2. Personal history ~. Previous experience _. Education and training -. Physical characteristics 5. Social characteristics 7. Interests 8. Abilities Helpful suggestions may be obtained from the various per- sonality schemata and catalogs of human abilities. These should be consulted not so much to obtain an indication of what is required for the job under study as to make sure that all the most important items have been considered. The number of these lists is legion, and only a few can be men- tioned here: Allport (1), Baade, Lipmann, and Stern (4), Blackford (14), Bridges (18), Cimbal (34), Kirby (94), Link (101), Mendenhall (29), Meyer (113), Parsons (126), Partridge (127), Roget (150), Stern (171), Terman (184), Viteles (209), Webb (213), Wells (217), and Yerkes and LaRue (227). One of the most complete lists of topics which the investigator may wish to consider is found in Davenport’s Trait Book (43). TOPICS FOR A MAN ANALYSIS 1. Personal data. Under this heading are included many jtems commonly found on the application blank, such as age, sex, race or nationality, marital status, number of de- pendents, home ownership, savings and investments, and membership in social and professional organizations. Such information about the workers chosen for study can often be obtained with little trouble from the company records. 60 ANALYSIS OF THE WORKER A number of investigators have found significant correla- tions between a properly weighted total score on items of this sort and success in the occupation studied. By such means it has been possible to select employees more accu- rately for life-insurance selling and for certain types of specialty selling (89). 2. Personal history. A man’s vocational tendencies are determined in great part by his early personal history, and predictions of future success should rest in part upon his past accomplishment. Below are some suggestive questions: Where was the worker born? What was the native country of his parents? What was his father’s occupation? Under what conditions was the worker brought up? How many brothers and sisters did he have? Which of his parents did the worker prefer? Which of his parents did the worker take after? Who was the dominant member of the family? What was the financial and social status of the worker's family? What occupation did his parents wish him to enter? Was his childhood such that if he had it to live over again, he would choose more cheerful conditions? Freyd, among others, has shown the worth of such topics in the differentiation of occupational groups (58). The investigator will want to use his own judgment in the selection of questions on personal history. He will be limited by the difficulty of obtaining such personal informa- tion from the worker, but the difficulties are not insurmount- able. Two groups of personal history items are so valuable in this connection that each deserves special mention: the worker’s previous occupational experience, and his educa- tion. 3. Previous experience. The worker’s occupational career should be recorded in sufficient detail to show not only the 61 EMPLOYMENT PSYCHOLOGY sequence of jobs but the time employed at each, and the length of periods of unemployment. It is essential that the work be described in enough detail to permit accurate identification of the job held. “Sales- man,” “machine operator,” “teacher,” “electrician” are not definite enough. A worker who had once been employed with the title of electrician was found on further question- ing to have had no duties more technical than turning the electric lights on and off in a motion-picture theater. “Just what did you do?” is the question which should be repeatedly pressed for precise answer. When it is possible to learn how well the worker succeeded at these various occupations, the way is opened to a study of the predictive value of different sorts of experience, with reference to success in the occupation being investigated. Even without any index of relative success in previous jobs, some useful facts may be uncovered with reference to pre- ferred types of previous experience. Even so simple a measure as average length of time on previous jobs has been found to have some validity in forecasting stability on routine work. 4. Education and training. Data on these subjects are easily obtained either from the records of the employment office or by questionnaires. They should cover informal as well as formal education, night-school and extension work as well as regular courses, avocational as well as vocational education. School marks and other evidences of school accomplishment may be worth getting if the investigator knows the standards of the schools and of the instructors concerned. Rate of progress through school is almost as accurate a measure of intelligence as a good mental alertness test. For this reason the application blank or questionnaire should include the items “age at leaving school,” and “grade reached.” This information is unreliable unless checked in personal interview. It is impossible to secure it from many 62 ANALYSIS OF THE WORKER iv older workers or from those whose schooling has been in other countries. But where it can be obtained, it gives a good clue to mental retardation or acceleration. 5. Physical characteristics. Items ordinarily covered in a routine medical examination sometimes prove to be of voca- tional importance. Any good medical examination blank will be suggestive in this regard. Height, weight, strength, health, blood pressure, sensory discrimination, physical im- pairments, and medical history are the minimum. The investigator may, if disposed, try, either directly or from photographs, some of the determinants in the Black- ford system, the Merton system, or any other system of in- ferring abilities from facial or bodily features He should be warned in advance, however, that sincere attempts of scientific investigators have hitherto uniformly failed to dis- close statistically significant and rewarding correlations be- tween these phrenological or physiognomic signs and valid criteria of character and ability. 6. Social characteristics. When the securing of cooperation or the arousal of friendly attitudes in others is a necessity in the job, the appearance of the worker may not be neglected in the analysis. No man, however unprejudiced he may try to be, is entirely free from conditioned responses to the physical appearance of others. The term social character- istics is used here to mean those physical characteristics as well as those habits and modes of behavior which make a worker a socially acceptable or unacceptable person. Al- though this topic logically belongs under headings 5 and 8, its importance justifies Separate treatment. Questions of importance are: How does the worker impress others socially? How do people like his clothes, his voice, his manner? Has he emotional balance? Has he a healthy outlook on life? What peculiar or abnormal beliefs and reactions does he display? 62 EMPLOYMENT PSYCHOLOGY Is oe Wapt-inded or alive to what is going on around im? Is he good-natured? Is he a fault-finder? Is he talkative? Data of this sort may be gathered by psychiatric exami- nations or by aid of rating scales. Use may be made of ratings on personal appearance, voice, carriage, and other aspects of what is ordinarily called personality or social effectiveness—characteristics which have a strong influence on other people’s attitudes but which are not yet readily measurable by psychological tests. =. Interests. Both vocational and avocational interests should by all means receive consideration in analyzing the workers. The reliability of interests as indicators of special aptitude and likelihood of success is small in early youth, but with mature people there is close relation between in- terests and vocational abilities. It is consequently im- portant to know the hobbies of the worker, his recreations, his ambitions, his attitudes, as well as his relative interest in this and other occupations. The analysis of interests has proved of great significance in almost every vocational study in which it has been attempted. It is most illuminating to an investigator who is at home in the atmosphere of the shop or office and who can win the confidence of the workers, to try to discover their special interests in their work and particularly their pride in de- tails of craftsmanship. There is vastly more of this type of interest even in highly specialized routine production than the casual observer is inclined to suppose. A persistent search for these points of pride and satisfaction in accom- plishment yields clues to essential skills, aptitudes, and tem- peramental characteristics. 3. Abilities. The number of abilities is almost infinite and it is impossible here even to suggest a check list of names to help the investigator in his search for the abilities HA ANALYSIS OF THE WORKER he exhibited by the workers he is studying. However, the name is not the essential thing. In fact, readiness to name an ability is sometimes a hindrance to its correct description and complete analysis. The investigator need not hesitate to make the description of the ability several pages long, pro- vided he states exactly what the worker must be able to do. Activities imply the abilities which make them possible. ANALYSIS OF ABILITIES In analyzing the abilities required to succeed in the oc- cupation, the investigator should think of the criterion of vocational success he has chosen and should select and de- scribe the abilities with reference to that criterion. Some cautions to be borne in mind when observing the worker in this connection are given in Chapter II. The investigator should obtain his information about activities and abilities by doing the work himself, by observing the workers, by in- terviewing workers or supervisors, and by consulting the literature. When dealing with workers he does well to talk in their language. He gains nothing by arguing with them or airing his own views about the abilities under study. The list of abilities required on the job should be made both definite and objective. It is not enough to make gen- eral statements that mechanical ability, quickness, or memory are necessary; the statements should be in terms of the specific behavior that is called for. There are many different sorts of memory, with no necessary relationship to each other. A person may be able to remember music ex- cellently and yet be unable to remember the names of his acquaintances. So too with abilities like tact or honesty. Does the secretary whose tactfulness is being defined have to write tactful letters of appeal or does she have to turn away angry collectors? Does an employee whose integrity is being defined have to be honest in the sense of not mis- representing goods or in the sense of not making off with the ne EMPLOYMENT PSYCHOLOGY company’s cash? The more definite the description of the abilities, the more likely is the analysis to be correct. Fur- thermore, when the abilities are stated in terms of actual behavior the investigator can see his way more clearly to the construction of suitable tests. The abilities should, where possible, be paired with their respective job operations. Shell inspecting, the description of which was quoted on page 21, requires the following abilities (101, p. 25): 1. Good eyesight. The defects to be detected were often so minute as to be indistinguishable to any but the best of eyes. It took the experimenter almost a minute to see one of the most common defects which these girls were required to notice in an instant. Any weakness of the eyes or marked difference between the two would be likely to show bad results in the inspection. 2. Keen visual discrimination. Good eyesight is not sufficient. The inspector, looking at a whole handful of shells, must, with a few glances, be able to recognize those which are defective and remain oblivious to those which are not. 3. Quick reaction; that is, the ability to extract, as quickly as seen, the defective shell and toss it into the appropriate box. 4. Accuracy of movement, required in picking out the right shell from the closely held handful. This requires a very peculiar kind of deftness, and, in order to facilitate it, many of the girls allow their finger nails to grow to an unusually long and sharp point. 5. Steadiness of attention. The least wavering of the eyes or letting up of the attention is likely to allow some bad shells to slip by or to lengthen the operation. One way to make the essential abilities stand out is to analyze the difficulties of the job. By finding out what errors are most frequent, what annoyances are most discon- certing, or what sorts of poor work are proving most ex- pensive, the investigator quickly brings to a focus his search for the really necessary abilities. The difficulty analysis as used by Charters (30) has as its main purpose the disclosure of specific needs for train- ing but it is serviceable also in the present connection. The 66 ANALYSIS OF THE WORKER investigator must carefully train his interviewers to secure, from a number of workers in the occupation, accounts of the difficulties of the job. These data are analyzed and tabu- lated for frequency. Such studies have been useful in re- vising curricula for training secretaries, librarians, teachers, home-makers, pharmacists, retail salespeople, specialty salesmen, and various types of executives. The difficulty analysis supplements the duty analysis. It is sometimes fol- lowed by a listing and evaluation of the abilities deemed most desirable in the worker. These abilities as listed are often found to be stated in terms too abstract to be very useful in test construction; but the process as a whole is often helpful and suggestive for research in vocational selec- tion as well as in training. INNATE AND ACQUIRED ABILITIES The distinction between innate and acquired abilities may be disregarded in analyzing abilities and constructing tests. Most of the workers with whom the investigator deals are adults, and even in the earlier stages of a person’s develop- ment his innate and his acquired abilities are practically. indistinguishable. It has even been maintained that the distinction is entirely hypothetical. Certain it is that nearly all abilities are modifiable—subject to adaptation and im- provement with practice. The question as yet unanswered is the extent to which some of these abilities are specifically determined by native endowment. F ortunately for the in- vestigator in vocational selection, he needs to know only whether a person has the abilities in question, or the capa- city to develop these abilities. Whether these capacities were defined and limited in the germ plasm or were pro- duced through environmental influences of infancy and childhood is no present concern of his. What he requires is a measure of the applicant’s capabilities at the time of em- ployment. Just what can this man do? If he has previously 67 EMPLOYMENT PSYCHOLOGY worked at the occupation, he has already available some skill, trade knowledge, and judgment which will shorten his period of training for this particular job. These abilities should be measured—tested by means of an adequate sam- pling of typical standardized occupational tasks, information items, and judgment problems. If, on the other hand, the applicant has had no experience in this vocation, his ability to handle a sample of the work, and particularly the rate at which he acquires this new ability, may be an indication of his future success at the job. The investigator will, of course, interpret all measurements in the light of his infor- mation as to the worker’s previous education, experience, and opportunities to learn; he may even find it advantageous to get the facts about the interests and occupational accom- plishments of parents and grandparents. But he need not trouble himself as to whether the observed relation between parental success and achievement of offspring is due to the forces of heredity or of environment. The distinction be- tween native and acquired abilities may safely be ignored. GENERAL AND SPECIFIC ABILITIES A similar haze envelops the distinction between specific abilities and general abilities, but the controversies over this subject cannot be so promptly dismissed. An appreciation of the problem, at least, is helpful to an understanding of much current writing on vocational selection. Every one believes in specific abilities. A person’s achievements in occupation or profession are conditioned by the excellence of a large number of readily distinguish- able characteristics, each independent, to some degree at least, from all the others. The question which has long concerned psychologists is, whether a man’s ability as a whole is essentially a summation of these specific abilities. Are there not certain common factors involved in all of a man’s behavior? Are there not at least a few general abilities? 68 ANALYSIS OF THE WORKER One of the alleged common factors is the ability to learn, the ability to profit by experience, the ability to modify behavior in order to achieve a purpose, the ability to shape one’s acts in the light of an end to be sought. This ability is usually called intelligence. No one questions that intel- ligence, as so defined, is the outstanding characteristic of the animals higher in the evolutionary scale; or that it is most indispensable in those human beings whose professional ac- complishments are superlative. The controversy is as to whether this all-important ability called intelligence is a general common factor conditioning in greater or smaller degree all of a person’s acts of adjustment, or whether it is simply a convenient abstract term used to refer to a sum total of many specific abilities, similar in some regards but independent of each other. The literature on this topic is of appalling extent, and includes much statistical as well as theoretical material. The issue has been beclouded somewhat by the tendency to identify the alleged common factor, intelligence, with the ability measured by some well-known test such as Army Alpha or the Binet-Simon series. These tests, like all so- called mental alertness tests or intelligence scales, aim to provide a sampling of performance typical of what the per- son examined would do under a wide variety of circum- stances. To do this, it is inevitable that tasks be presented whose performance calls for certain specific abilities in addi- tion to the more universal modes of response which it is the real purpose of the test to disclose. One’s score in an intel- ligence test is then first of all a measure of specific ability, of ability to do precisely the things required of him in that test. So far as these responses are identical, wholly or in part, with elements of responses needed in other situations, the test becomes a partial measure of ability to meet these other situations also. Even such a limited sampling of verbal problems as Army Alpha gives something of an indi- cation of a young man’s capacity to succeed in a professional 60 EMPLOYMENT PSYCHOLOGY school or in a clerical position. There are abilities tapped by the examination which undoubtedly are components of relative success in a wide variety of activities and occupa- tions. Are these identities to be explained solely by the hypothesis that all these activities and occupations present situations which call out some of the very same specific abilities as the examination? Or is it better to assume that some one general ability is demanded in greater or smaller degree by all these situations? This question is a debatable one. Spearman and his fol- lowers, while insisting on the importance of specific abilities, hold that they have demonstrated the presence among these abilities of a general common factor. Indeed they main- tain that there is not only a single common factor, intelli- gence, determining in part our responses; but that other pervasive determinants can be distinguished and measured, such as character or perseverance. Output of energy is still another characteristic which seems to some psychologists to be general—to be, in other words, to some extent typical or characteristic of a person’s behavior irrespective of the direction of his activities. Psychologists in developing the most dependable measures of such characteristics as mental alertness, ability to learn, or general intelligence, have sought to assemble a wide sampling of varied tasks. They have included in their bat- tery those tests which correlate highly with the criterion selected, but which intercorrelate least with each other. In this way they have tried to reduce to a minimum the contri- bution to the final score made by the several specific abili- ties. They have aimed to disclose the more pervasive ten- dencies of adjustment, the more general ability. Some of them have called the ability measured by these tests, general intelligence. Others call it conceptual intelligence, or mem- tal alertness, or, merely, ability to do this kind of test! It is fortunate that the investigator in vocational selection does not have to wait until this controversy about general 0 ANALYSIS OF THE WORKER oI ability is ended before he begins to make practical use of so-called general tests of mental alertness, output of energy, and the like. He will ordinarily find that the more specific he makes his own tests, the closer they will predict ability to do the work in question. But the varied samplings of ability made by the standard intelligence tests undoubtedly have their predictive value also. A worker who can do his tasks well is nevertheless apt to become discontented and restless if he is able to do work much more difficult and exacting than his job requires. Dissatisfaction and labor turnover traceable to such maladjustments are sometimes predictable from intelligence test data. Ability to learn certain kinds of occupational tasks is likewise significantly correlated with intelligence test score. For these reasons the investigator will not hesitate to include appropriate general tests in his program of examinations, even though he may be skeptical of the theory of the general common factor, regarding which controversies still rage. At the same time, no shrewd investigator will limit him- self to any single test for an assumed general ability. If he is interested in intelligence, he will use one test for intelli- gence in manipulating mechanisms, another for intelligence in dealing with people, a third for intelligence as applied to ideas, words, concepts, written symbols. Indeed he will choose his test according to the kind of mechanical con- trivances to be manipulated, the social level or type of situation in which social intelligence is to be demanded, or the particular realms of abstract ideas in which clear, quick, and accurate thinking is to be sought. He will go as far in the direction of measuring specific abilities as the nature of his problem permits. RELATIVE IMPORTANCE OF ABILITIES Before attempting to construct an examination for any particular item in his analysis of the abilities required, the i] . EMPLOYMENT PSYCHOLOGY investigator naturally wants good evidence that it is really an important item. The relative importance of any ability varies with the frequency with which the job demands its exercise and with the proportion of the worker’s time which he has to devote to the work for which the ability in question is necessary. Ability to dictate his correspondence crisply would, on this score, rank low among the abilities desired in an accountant. The table of frequencies of duties performed by secretaries as found by Charters and Whitley (33) brings into clear perspective the relative importance of these abilities. FREQUENCY OF DUTIES OF SECRETARIES Some Dyin She Lees 10% Some le Yost 10% Take dictation: letters Take dictation: sales quotas and Answer telephone x graphs : Meet callers Address a meeting of employees : Attend buyers’ or salesmen’s meet- Make engagements and appoint- ings ments Organize committees Compose letters Check cash-register returns Clean and oil typewriter Make blue-prints The investigator should determine the relative importance of the items he has listed with reference to the selected standards of success. For instance, certain items will assume importance if length of service is the standard by which they are judged, whereas these items may lose in importance if output or other considerations have been chosen as indices of success of workers. The priority of the several items with regard to one cri- terion of success may be roughly determined by getting supervisors to rank the items in order of importance. Another method is to obtain ratings on workers with refer- ence to these items and to compare these ratings with stand- ings of the workers in the criterion of success. Ratings on some points will be found to correspond more closely with 72 ANALYSIS OF THE WORKER 73 the criterion. These points should be emphasized in the later selection and construction of examinations. As a matter of caution it should be borne in mind that the ratings, as well as the original analysis, are open to errors of judgment. If, however, the investigator has good cooperation and the raters are carefully trained in the use of the scale, the results will be suggestive. In addition, it may later be found that the scale is reliable and valid enough to warrant its use by interviewers of applicants for the job. Cook and Manson (36) carried through an analysis of abilities necessary in effective retail selling, which serves well to illustrate the value of sound procedure. The first step was to obtain, by carefully planned interviews with customers, the data for constructing a list of traits important for retail salespeople. Some questions were framed to bring out desirable traits, others undesirable ones. The answers, recorded verbatim, were later translated into trait terms. The traits most often mentioned, in order of frequency, were: Interest in customer; merchandise information; display of merchandise; interest in work; courtesy; alertness: appear- ance; and ability to size up customer. For 5 of these traits, graphic rating scales were constructed. A scale for rating the “total sale” furnished the criterion against which could be checked the several ratings of the salespeople. After training in the use of these scales, 8 professional shoppers rated 75 sales episodes, and 7 representative customers rated another 75 sales situations. The regression equation was then used to determine the proper weight of each of the 5 traits. It was found that the 2 qualities weighted heaviest by the professional shoppers are alertness (speed with which the sale progresses) and interest in customer; but the order of importance as determined by the representative custom- ers was: (1) Interest in the customer, (2) display of mer- chandise, (3) alertness, (4) merchandise information, (5) courtesy. The customer does not want to be hurried. Courtesy, apart from its expression through interest and the b Fa y EMPLOYMENT PSYCHOLOGY other qualities, is of little importance; in fact, effusiveness is a detriment. The interview method and the rating scale method both proved useful in determining the relative im- portance of the abilities needed in retail salespeople. We have emphasized the importance of making a com- prehensive analysis of the worker and assembling a list of items with reference not only to activities and abilities but also to every phase of human variability which is subject to measurement in one way or another and whose share in vocational accomplishment may be determined. In addi- tion to the items listed above, the investigator may at his own discretion study such variables as the quality of the worker’s references, judgments from photographs, self- ratings by the worker, his motives in applying for the posi- tion, his ambitions, and the like. Although some resistance may be anticipated on the part of both workers and man- agement to such minute inquiry, nevertheless if the examina- tion is skilfully constructed and tact is employed in its presentation the most important items in the analysis can be covered. When this is judiciously done, items of real significance for vocational prediction are apt to be discov- ered. The measurement of intelligence, under the great impetus of the results obtained in the United States Army during the World War, played a dominant part in vocational measurement for a few years. But farsighted investigators have been discovering usefulness in other methods of mea- suring vocational determinants also, such as the statistical evaluation of personal history and interest questionnaires. Meanwhile, slow but steady progress is being made in the measurement of what may prove to be the most potent of all the vocational determinants—traits of personality. The greatest progress in vocational measurement will beyond doubt come from the application of the experimental method of psychology to the material of the psychiatric diagnosis and the physiological examination. 74 ANALYSIS OF THE WORKER The broad topic of man analysis has been treated apart from the chapter on job analysis, but this has been done for convenience of exposition only. The job apart from the worker who does it is an abstraction, and so, too, is the worker apart from the job. The actual unit is the worker- in-his-work. This is most obvious when our attention is focused on high managerial positions. An executive post, no matter how minutely the duties are specified, varies with the man who fills it. The executive, also, who has been, let us say, a superintendent, is a somewhat different person when promoted to the general managership. The investi- gator should never forget to think of the job in terms of the workers who perform it, and of the workers as related to their own specific occupations. The investigator is now faced with the task of choosing or constructing examinations for the experimental measure- ment of the workers he has decided to study. In the selec- tion or construction of these examinations he will be guided by the estimated importance of the abilities and character- istics he has listed. The general principles governing the selection of examinations will be considered in the next chapter. 78 V1 6 SELECTION OF EXAMINATIONS Principles governing the selection and construction of tests: Objectivity; reliability ; simplicity; analogy of test and ability to be measured; resem- blance to vocational tasks. Sources of material: The vocational apparatus; former studies of the vocation; general test publications; miscellaneous sources of information. THE investigator has at this point chosen a criterion of success in the profession, occupation, or job being studied, has selected those who are to serve as subjects for his in- vestigation, and has drawn up a tentative list of abilities considered essential to success in the occupation. He is now faced with the task of choosing or constructing suitable examinations for the abilities deemed essential to success. Later he must test the validity of these examinations. Mea- sures obtained with them will have to be compared with the status of the workers in the criterion of actual vocational accomplishment, in order to determine which examinations if any have sufficient validity in predicting success in this vocation to warrant their use in selection. Serviceable types of measuring devices from among which a choice will be made are psychological tests, rating scales, and certain forms of questionnaires, such as application blanks and interest analyses. Subsequent chapters will treat in detail of the construction and use of each of these types of examination. Here attention is focused on basic gen- eral principles governing the selection and construction of measuring instruments. While these principles apply par- ticularly to psychological tests, several of them are equally pertinent to other kinds of device for gaging abilities essen- tial to vocational success. SELECTION OF EXAMINATIONS 77 What, then, are the characteristics of a good test? And where should the investigator search for suitable test materials? PRINCIPLES GOVERNING THE SELECTION AND CONSTRUCTION OF TESTS Objectivity. Foremost among the characteristics of a scientific test is that it yields objective measures. Opinion, individual judgment, and the personal equation of the examiner who gives or scores the test should be reduced to a minimum. Research procedure in vocational selection has for its purpose the provision of something better than mere hunches or unanalyzed general impressions as a basis for decisions and recommendations. The outstanding char- acteristic of the unscientific methods is their lack of ob- jectivity. An interviewer who relies on subjective impres- sions or indefinable intuitions is dependent upon his alert- ness, his previous experience, and his social sensitivity, He never knows when or how much he can trust his hunches. Moreover, interviewers differ widely in their ability to arrive at a fair estimate of an applicant through general impres- sion. Their judgments tend to be at variance because of personal prejudices, differences in ability to judge, and failure to agree upon the vital point most essential for the job. A few rare employment interviewers seem to be very successful although relying on their unanalyzed subjective impressions. But no business concern wishes to be depen- dent upon a rarely gifted person who cannot readily be re- placed. The purpose of the investigator is to bring to light the abilities and characteristics of a worker which the suc- cessful interviewer judges in the applicant, and to measure them accurately by a method which can then be used with equal success by any carefully trained examiner. For in- tuitive opinion he substitutes objective tests. EMPLOYMENT PSYCHOLOGY Objectivity is more readily attainable in tests for routine work and for ability to learn than for measures of social characteristics and such intangible abilities as resourceful- ness and originality. It is not contended that employment procedure can dispense with subjective estimates and im- pressions, particularly with reference to the typical emo- tional tendencies which are grouped under the broad cate- gories of personality and character. It is maintained, how- ever, that wherever objective measures can be made, they are more uniform and dependable. Such measures of a person’s likes and dislikes and of his tact, energy, seli- control, and honesty in selected situations, are conceivable. Reliability. Measures must be not only objective but reliable. Thus, an interviewer who has asked an applicant about her training and experience can make a subjective estimate of her probable ability to learn to operate a comp- tometer. If, knowing that she must have some arithmetical ability, he gives her some problems to do, he gets an ob- jective measure of that ability, but not a very reliable one unless the problems have been well selected, previously graded as to difficulty, timed, and standardized. An Edison questionnaire may be an objective measure of range of in- formation, but it is certainly a very unreliable one. What are the characteristics of a test which is at once objective and reliable? A reliable test when repeated on a representative group of people after a period in which they have had equal prac- tice in the ability (or no practice) should yield measures which preserve the same relationships between them. In other words, the test correlates highly with itself. More- over, the measurements obtained with it by different exam- iners are the same, provided they adhere strictly to a uni- form procedure. Of course, if one examiner varies from the standard practice by giving more specific instructions to the applicant or by allowing him more time to work on the test, the results cannot be compared with those obtained by other ~8 SELECTION OF EXAMINATIONS examiners. The test so administered becomes an unreliable measure of the ability in question. The material of which the test is composed, the conditions under which it is given, the directions, the fore-exercises, the time limits, and the method of scoring do not vary in a reliable test. The pro- cedure depends as little as possible upon the skill of the examiner, and the whole matter proceeds almost automati- cally. This principle applies also to rating scales and ques- tionnaires. The investigator should aim at maximum reliability in the construction of his tests and should find out just how reliable or unreliable they really are as instruments for measuring the abilities they purport to measure. Mathe- matical procedures for determining reliability are outlined in Chapter XIII. Simplicity. Other things being equal, the simpler test is the more desirable. A test requiring no complicated appara- tus or involved technique has these advantages: It is inex- pensive to construct; it requires less time to administer; it does not demand as much technical training of the exam- iner; it does not frighten the applicant. One advantage of an employment test is that it gives immediately a prognosis of ability which otherwise would require months on the job to disclose. The test record should have the practical advantage that it is much easier to obtain than the criterion. Simplicity should, therefore, be constantly before the investigator as a desired characteristic in his tests. Analogy of test and ability to be measured. The investi- gator may wisely be guided in his selection of test materials by their apparent relationship to the abilities required for success. It is impossible to say in advance whether or not a certain test measures a certain vocational ability. It is probable, however, that a test which seems to measure that ability will be more valid than one which seems to bear no resemblance to it. If the abilities have been analyzed into 70 EMPLOYMENT PSYCHOLOGY samples of the actual behavior involved, the selection or construction of tests by analogy is comparatively simple. Hollingworth gives four methods of test construction, in brief as follows: The method of the vocational miniature involves the re- production of the entire work or an important part of it on a small scale. Telephone operators, for example, have been examined by placing them before a miniature switchboard and requiring them to make calls and responses just as if they were before the standard apparatus. This method does not demand an analysis of the abilities necessary for success at the job, but if such an analysis had been made and such a test used, the test would be classifiable as one based on the psychological requirements for the job, since presumably the miniature apparatus measures these requirements in their totality. One difficulty with this method is the neces- sity for the construction of special apparatus. A source of error is that the miniature apparatus may require finer coordinations and adjustments than are demanded by the standard equipment. The sampling method selects an actual sample or piece of the work to be performed and measures the skill with which the applicant carries out the task. This method is identical with certain types of trade testing. It finds a ready use in testing office employees because of the adaptability of some of the office tasks to standardized test procedure. Such a standard task may also serve as a criterion for aptitude tests. In the analogy method, tests are used “which bear real or supposed resemblance to the sort of situation met by the worker in the given occupational activity. The material is new, but the attitude and endeavor of the worker seem to be much the same” (72, p. 111). This method makes use of the analyzed requirements for the job and attempts to devise measures for these specific requirements. The tests do not employ the apparatus of the job in its totality, in fo SELECTION OF EXAMINATIONS part, or by a small reproduction, but measure what seem to be the more elemental or fundamental psychological abili- ties required on the job. Even though at the present state of progress a psychologist may hesitate to state definitely just what ability is measured by a certain test and may find it difficult to name the abilities involved in a given voca- tional task, it is nevertheless true that he can describe cer- tain types of behavior which are apparent to every one and he can construct tests which call for the same or closely similar types of behavior. A great deal of confusion has been thrown about this matter by assuming that the abili- ties necessary to vocational success are subjective and that any test which fails to reproduce the internal situation can- not be a just measure of the ability. The investigator must not forget that what he seeks is an objective test whose measures bear a significant relationship to vocational suc- cess. He cannot claim that the test which he constructs by the analogy method actually measures the ability for which it is intended; but he may safely assume that a test so con- structed is more likely to correlate with the criterion of success than tests chosen at random. The random method does not rest upon observation or analysis of the occupation, but uses a large variety of tests chosen at random, in the hope that one or more of them will correlate with vocational success. This is a wasteful and feeble method. In the published examples of the use of this method, most of the tests which proved significant would have been chosen if the method of analogy had been used; that is, they were tests which bore an apparent rela- tionship to the job. Regarding this method, Hollingworth says (72, p. 113): It is not pretended that the test is a miniature of the work of such calling, nor that it is a fair sample of such work, nor even that it involves precisely the same mental functions that come into play in such work. The test records and ability in the par- ticular type of work show high positive correlation, which means that an individual who is good or medium or poor in the one is, Rr EMPLOYMENT PSYCHOLOGY as a mere matter of fact, also found to be good, medium, or poor in the other. Hence, without further analysis, the one may be used as the sign of the other. The statements contained in the last two sentences may apply with equal truth to any of the four methods of test construction. The correlation with vocational success is the final indication of the value of the test. Of the four methods just described, the one which has found widest use is that of building tests which call for operations analogous to the operations observed in the job. Even those who resort to the random method rarely choose their tests entirely by chance. Resemblance to vocational tasks. The preceding section dealt with the materials going into the test. The present section deals with the manner of presenting this material to the applicant so that he will want to do his best. The test situation ought to bear at least a superficial resemblance to the job for which it is intended, as regards its items and mode of presentation. If the analogy method has been em- ployed in developing or selecting the tests, they are apt to be immediately suggestive of the vocational task, but some- times the analogy is apparent only to the examiner. Diffi- culties arise if the applicant cannot see any relationship be- tween the tests he is required to undergo and the work for which he is applying. The usual written intelligence exam- ination seems childish and irrelevant to the business execu- tive. To forestall this objection, the Bureau of Personnel Research of Carnegie Institute of Technology devised in 1922 a test for use with business executives (58a). The content of this test is taken from commerce and finance, but its form is that of the usual intelligence tests. With groups of business men, such a test correlates highly with, and is a useful substitute for, an intelligence test of the ordinary type. Reluctance of the applicant to take the tests may be avoided by providing the motivation in the title, the direc- Q2 SELECTION OF EXAMINATIONS §; tions, or the choice of items. A test of “imagination” in- cluded in a battery of tests for salesmen gets readier accep- tance when labeled “foresight.” The directions may call attention to the analogy between job and test or they may set up a hypothetical situation in which the applicant is to place himself. The items may often be so worded or con- structed as to refer directly to the job without injury to the purpose of the test. To repeat, the investigator should make it his aim not only to construct his tests with some regard to their analogy with the vocational requirements, but should also motivate the applicant by so wording or arranging his test material or directions as to make their relevancy to the job apparent. SOURCES OF MATERIAL The more profitable sources of material are listed in this section. In addition, the investigator may run across sug- gestive material in newspapers and magazines, or by brows- ing in stores or factories. His originality will be shown by his success in adapting these materials to his needs. We shall deal here with some of the sources of completed tests, or of material from which tests, questionnaires, and rating scales may be devised. The vocational apparatus. The equipment or materials used by the worker on the job may often be adapted to test purposes. If the apparatus is costly or cumbersome it will be possible to test only one man at a time. Reproductions of small parts of the equipment may be used for test pur- poses, permitting the testing of groups. Sometimes pictures may be substituted for the actual apparatus in testing. Sources for illustrations are text-books or manuals of the trade, trade journals, trade catalogs, or photographs, dia- grams, and sketches specially prepared. Former studies of the vocation. It goes without saying that the investigator should be familiar with all the previous 3 1X . EMPLOYMENT PSYCHOLOGY research work on the vocation in question so that he can use positive findings and avoid whatever proved worthless. Published studies of this sort are to be found in psychologi- cal journals, trade magazines, volumes of research, and in miscellaneous publications, reference to which may be found in bibliographies. His acquaintance with other research workers often enables the investigator to learn of results obtained in unpublished studies. General test publications. These need not relate to the vocation under study but may cover the general field of tests and their evaluation. No matter how well acquainted he is with the field of psychological tests in general, a review of the literature will assure the investigator that he has not overlooked any applicable tests. Since psychological tests are an outgrowth of standard laboratory experiments in psychology, it will repay effort to refer to some of the laboratory manuals. Among these are Titchener (200), Myers (122), Sanford (161), Lang- feld and Allport (100), Schultze (163), and Scripture (167), as well as numerous untranslated German volumes. Manuals of psychological tests constitute a second source of information. Some of the best known are those of Whip- ple (220), Franz (54), Woodworth and Wells (225), Dewey, Child, and Ruml (44), Pintner and Paterson (139), Giese (61), Downey (48), Ballard (35), Terman (185), Burt (22), Trabue and Stockbridge (205), Stern (174), and Lipmann (103). Considerable test material may also be found in Link (ror), Burtt (23), and Partridge (127), and in the report on “Psychological Examining in the United States Army” (228). A number of volumes on educational measurements and some of the studies published in the Psychological Monographs and the Archives of Psychology contain valuable test material. Material may also be found in the catalogs of the C. H. Stoelting Company (Chicago), World Book Company (Yonkers), and E. Zimmermann (Leipzig and Berlin). 24 SELECTION OF EXAMINATIONS 85 A third source of information is through journals which publish test researches, a list of which is given in the Ap- pendix. Bibliographies of tests are suggestive. A number of agencies are at work at present on such compilations. Bib- liographies have been made by Mitchell and Ruger (116), the World Book Company ( 226), Coss and Outhwaite (37), Whipnle (220), Muscio ( 121), Manson (105), Rossi and Rossi (153), Doherty and Others (47), and Lipmann (102). The Journal of Educational Psychology, the Journal of Per- sonnel Research, and the Psychological Bulletin publish short abstracts and bibliographies. In the latter journal Kornhauser (97) has made a very complete abstract. All- port (2), May and Hartshorne (109), Symonds (180), and Cady (24) have summarized the literature on tests of per- sonality and character. The most comprehensive bibliog- raphy is the yearly Psychological Index. Cody (35) and Kemble (88) have published books on tests without giving much information on the value of the examinations de- scribed, but the materials are suggestive of tests for office workers or factory operatives. Miscellaneous sources of information. Material for rat- ings on personality and appearance may be found in Perrin (136), Hepner (69), Wells (217), Terman (184), Holling- worth (71), and Freyd (58); and for anthropological measurements in Whipple (220), Yerkes and LaRue (227), Davenport (43), and Wilder (221), as well as in physical examination blanks used by physicians. For neurological and psychiatric examinations, Meyer (113), Franz (54), Kirby (94), and Cimbal (34) give complete outlines. For material for questionnaires on personal and family history, interests and attitudes, and so forth, the investigator may refer to Moore (118), Freyd (58), Meyer (113), Yerkes and LaRue (227), Stern (171), Miner (115), Allport (1), Healy (68), and Kemble (88). The investigator may be advised not to limit himself to EMPLOYMENT PSYCHOLOGY the well-known psychological tests, but to use any promising measuring device which may be made to conform to the principles set down in this and the preceding chapter. Proof of relationship between a series of measurements and suc- cess in an occupation makes that series of measurements, regardless of its origin, a matter of interest and value. Specialists in the selection of employees have installed sys- tems with a total disregard for scientific methods of validat- ing their results. Scientists who have evaluated the work of these persons have found most of it worthless. But some of these systems and tests show originality and, if applied with scientific care, might reward the investigator. If, on the other hand, they prove worthless, the investigator who has checked them scientifically instead of dismissing them with- out a hearing has a weapon with which to combat their advocates. The following chapters deal with the details of the con- struction of psychological tests, rating scales, and question- naires. 86 rr PSYCHOLOGICAL TESTS What is meant by a psychological test. Classification of tests according to purpose, ability measured, apparatus or material used, technique, restrictions on response. Verbal tests. A PSYCHOLOGICAL test is a means of gaging an ability by measuring performance in a standard task (or tasks of graded difficulty), performed under standard conditions. The conditions which should be definitely specified and con- trolled include the incentive for the person to do his best, as well as more obvious conditions such as standard equip- ment and materials, and uniform preliminary instruction and fore-exercise. A good test yields an accurate objective measure of typical behavior in a way to reveal the extent to which the person differs from other individuals with respect to the ability tested. An unfortunate tendency has appeared in some quarters to use the term “tests” broadly to cover all kinds of psycho- logical measuring devices, including rating scales and ques- tionnaires. But a rating scale is a very different sort of instrument, to meet a different need. It yields a record of the rater’s subjective estimate of the person’s ability or trait, whereas a test yields an objective measure of actual performance. One is a systematized record of opinion; the other a measurement of fact. A questionnaire calls for factual data about either per- formance or opinion. It does not undertake, however, to get a precise measure under standardized conditions of what the person can do now, but rather to find out what he has done, what his personal history has been, what his interests, tastes and ambitions are, and the like. av 87 EMPLOYMENT PSYCHOLOGY It has been customary to draw various lines of distinction between psychological tests and psychological experiments. Some of these distinctions have broken down in practice. It is not strictly correct to state that a test dispenses with the typical precautions in control of conditions deemed essential in a labora- tory experiment; nor that only the experiment utilizes instruments of precision. Nor is it always strictly true that an experiment is performed to try out an hypothesis and establish a generalization while a test is limited in its purpose to the measurement of the individual. Terman (186) in his presidential address before the American Psychological Association effectually disposed of that tradition by assembling the general principles of psychology which have been ‘developed by means of research with tests. In the foregoing chapter several features of a good voca- tional test were outlined. It should be simple in conception and operation. It should be constructed so as to yield an objective quantitative measure which does not vary with the person who administers or scores it. Its material, the conditions under which it is given, the directions, and the method of scoring should be rigidly specified and adhered to, so that scores made by persons examined at different times and places will be comparable. It should be built on the theory that it measures one or more of the abilities neces- sary to the job. It should have an apparent relevance to the job requirements to insure the applicant’s interest. Chapter V discusses the ways of arriving at the abilities which the tests should aim to measure. In constructing a test for one of these abilities, the investigator will find it profitable to refer to the sources of test material listed in Chapter VI. He will also find it highly worth while to con- sult all relevant material on test construction and applica- tion gathered by other psychologists, especially where they have studied the same or similar vocations. Above all, he should hesitate to develop a new test if he can find one already available. Unless he makes a decided improvement on an existing test he will do little more than add to the growing profusion of insufficiently evaluated and hence use- less psychological tests. 88 PSYCHOLOGICAL TESTS Regarding the technique of constructing tests, quite a body of knowledge has accumulated, but no very complete statement of it has found its way into the literature. Per- haps most writers have neglected to record their technique because they consider it merely the application of common sense or a modification of well-known laboratory practice. Some treatment of test construction is, however, to be found in Monroe (117), Toops (202), Burtt (23), Symonds (180), May and Hartshorne (109), Chapman (28), Pater- son (129), Whipple (220), McCall (104), Pressey (141), Scott and Clothier (166), and O’Rourke (124), and in “Psychological Examining in the United States Army” (228). When in 1917 a little group of seven psychologists met at Vineland to construct the army mental tests, they formu- lated a number of criteria to guide them in the adaptation of test materials to military purposes. Some of these cri- teria of a good test are widely applicable, while others are limited by the special purposes for which these tests were designed. They are (228, p. 299): (1) adaptability for group use, (2) validity as a measure of intelligence, (3) range of intelligence measured, (4) objectivity of scoring, (5) rapidity of scoring, (6) unfavorableness to coaching, (7) unfavorableness to malingering, (8) unfavorableness to cheating, (9) independence of schooling, (10) minimum of writing in response, (11) interest and appeal, and (12) economy of time. The report from which these criteria were abstracted enlarges upon them very little, although the account is the most complete one to be found on the develop- ment of a group intelligence examination. Some of these criteria of a good test may be subsumed under the headings (a) reliability and (&) validity. Tests intended for industrial use should be characterized also by (c) freedom from ludicrous or silly items, (d) apparent relevance to the purpose, and (e) convenience of administer- Ing. EMPLOYMENT PSYCHOLOGY CLASSIFICATIONS OF TESTS A classification of tests helps to make clear precisely what the investigator is attempting, and what he may and may not hope to accomplish. Bases of classification include the purpose of the test, the ability measured or type of response called for, the appara- tus or material used, the procedure or technique, and the restrictions which the technique places upon the response. Purpose. The purpose of a vocational test is to provide a record of achievement in a standardized task bearing a close relationship to future performance in an occupation. The test should then attempt to measure either the present occupational ability of the applicant or his capacity for acquiring this occupational ability. A suggested classifi- cation on this basis. follows. 1. Proficiency tests: Measurement of present occupational ability. 2. Prognostic tests: Measurement of potential ability; dis- closure of latent talent. A. Special abilities such as those revealed by an analysis of the occupation. B. General abilities, such as leadership, salesman- ship, planning, intelligence. Ability measured. Since abilities themselves are difficult to classify, it follows that a classification of tests on the basis of abilities cannot be entirely satisfactory. Some com- pilers of tests have grouped them under time-honored psy- chological categories such as the following: 1. Sensory abilities: sight, hearing, pressure, warmth, cold, pain, taste, smell, kinesthesia, equilibrium, organic sensations Limits of sensitivity, lower and upper Thresholds of discrimination Quality Intensity Duration QO PSYCHOLOGICAL TESTS 2. Motor abilities Reflexes Reaction time Speed and accuracy of voluntary movements, such as tapping, tracing, aiming, steadiness, rhythmical move- ments Dexterity in manipulation Special skills Reflexes Strength Endurance Resistance to fatigue 3. Intellectual abilities Perception Quickness and accuracy of identification of objects, including people, words seen or heard, spatial and temporal patterns Susceptibility to illusions Association Association of ideas Rate Range Nature Imagination Vividness Detail Preponderant tendencies Memory Memory span Speed and accuracy of recognition Speed and accuracy of recall Fullness and accuracy of testimony Ability to learn Knowledge General Special and technical Analytical abilities Ability to note similarities and differences Ability to define a problem Reasoning abilities Conception Clearness of ideas oI EMPLOYMENT PSYCHOLOGY Fertility of ideas Judgment Sagacity in noting the relevance of ideas Inference Knowledge and skill in the techniques of test- ing hypotheses Intellectual level, intellectual range, and intellectual speed Conceptual intelligence Mechanical intelligence Social intelligence 4. Attention and will Span of attention Fluctuation of attention Concentration Susceptibility to distractions Suggestibility Motivation Strength of motives and fundamental drives Dominant attitudes Susceptibility to incentives Perseverance of motives Promptness of decision and of action Inhibition 5. Emotional traits Range and strength of interests Intellectual Mechanical Social Nature and degree of emotional expression Intensity Normality Suppressions Temperament Dominant moods and sentiments; complexes; tastes— artistic, social, recreational, vocational 6. Character Persistence of motives Dominant ideals of conduct Integration of behavior tendencies 02 PSYCHOLOGICAL TESTS Strength and relative dominance of instincts, habits and ideals; sublimations Consistency of behavior tendencies Conformity to ethical patterns Independence of judgment 7. Personality Balance and integration of intellectual, emotional, and volitional tendencies Strength of tendencies to seek or to withdraw from social contacts; tendencies to introversion or extro- version; tendencies to be active or passive, domineer- ing or submissive, emotional or thoughtful, and so forth Self-consciousness: shyness, arrogance, and so forth Social effectiveness A behavioristic classification of responses is as follows (2110). Hereditary responses Explicit: grasping Implicit: changes in circulation Habitual responses Explicit: tennis playing Implicit: “thinking” Apparatus or material used. Most tests in current use in industry and in schools are of the paper and pencil sort, since these are the most convenient to administer. But for many purposes it is necessary to utilize a more elaborate equipment. 1. Paper and pencil tests A. Verbal material. Examples: Army Alpha; Thur- stone Clerical. B. Non-verbal material. Examples: Army Beta: pencil maze; Stenquist Mechanical Aptitude Test. 2. Physical apparatus tests. Examples: Form board; Sten- quist Assembly. 3. Personal tests, where the examiner is the essential part of the apparatus. Example: The Moore and Gilli- 03 EMPLOYMENT PSYCHOLOGY land Aggressiveness Test (119) requiring the sub- ject to do problems in mental arithmetic while staring the examiner in the eye. Technique. A convenient basis for classifying tests is the number of persons who may be tested at one time. This classification is commonly limited to two categories: 1. Individual tests 2. Group tests It is pertinent here to point out the relative advantages of these two types of test. A test which can be given to a group of persons at once has the advantage that it reduces the cost, and the time of the examiner. With the right motivation it is not difficult to preserve order, and cheating may be controlled by the exercise of care. A suitable room equipped for the purpose is desirable. The group test has some disadvantages. It limits the variety of reactions which may be studied. If it requires duplicating expensive apparatus the cost may outweigh the saving in time. The individual test, the test given to one person at a time, demands more of the examiner, but permits the use of more elaborate apparatus and the measurement of a greater variety of reactions. In theory some small weight may attach to the distinc- tion between the two types of tests in the way they repro- duce the job conditions; that is, if the job requires the men to work together in groups, then in theory group tests may be the kind to use, and if they work singly, individual tests may be preferable. Such considerations would, however, have weight only if the job analysis had shown that an im- portant social aspect of the job were closely reproduced by the test situation. The individual test not only allows a wider latitude in testing, but is often more practical in application, since can- didates must usually be examined singly. 04 PSYCHOLOGICAL TESTS Gg A test which is scored in terms of the time required to complete a standard task is called a work limit test. A test which takes account of the amount of work done in a given time is called a time limit test. In this form a starting sig- nal and a stopping signal are given, and the performance is scored in terms of the amount accomplished (or the amount accomplished correctly) within the time allowed. In group testing only the second form is recommended, since it is not easy to time the individuals in a group sepa- rately. Some schemes have been devised for the purpose of adapting work limit tests to group procedure by timing the members of a group on a set task, or by directing the examinees to hand in their papers just as soon as they have finished, thus getting an approximate record of time used. These schemes are open to error and depend in part on the honesty of the persons tested. Restrictions on response. A classification of tests ac- cording to restrictions which the technique places on re- sponse is suggested by Thurstone (see below, page 96). TEST ITEMS The items of which the test is constructed should be rep- resentative samples of the type of behavior it is desired to measure. All superfluous items, all items which do not seem to measure just this ability, should be excluded, and those that are retained should furnish, as regards both variety and difficulty, a fair sampling of the ability or in- formation desired. Items which have only a temporary interest and are apt to become out of date should be avoided. A current information test soon becomes a historical test. All test items should be explicit and in the vocabulary of the men in the occupation. In each case there should be no doubt as to the right answer. Items calling for spoken responses have the disadvantage 0s EMPLOYMENT PSYCHOLOGY of difficulty of recording and scoring and the necessary con- stant supervision by the examiner. Where written responses are required, the amount of writing should be reduced to a minimum not only because of the saving in time, but also because of sources of error through variations in speed of writing, illegibility of hand- writing, and difficulty of scoring. Catch questions should be avoided. There are conditions under which catch questions are necessary, but they rarely arise in industry. In wartime it was not uncommon to find conscripts who simulated color-blindness in an effort to avoid military service. A clever test developed in Germany meets this situation (215). It consists of a number of charts each composed of small colored dots, averaging about a quarter of an inch in diam- eter, printed close together. To the person with normal color vision, numerals formed of dots of a single color stand out in each chart. The color-blind person cannot distin- guish these numerals because their gray value is the same as that of the other dots. In several of the charts, however, the numerals have a distinct gray value and can be read even by color-blind persons. The malingerer who is un- aware of this fact is detected when he claims that he cannot see any numerals on these charts. Questions should not follow in such a sequence that one suggests the answer to another. To facilitate scoring, items should be printed so that all answers appear in a column near the edge of the sheet. Verbal tests may require either a spoken or written reply. Thurstone has suggested a classification of verbal test items according to the degree of constraint they impose on the response. A. Free response B. Constrained response 1. Imposed by wording of question ob PSYCHOLOGICAL TESTS 2. Imposed by location of answer on page, and by con- text 3. Imposed by number of available responses provided (a) Multiple choice (6) Two choice The free response or essay type of examination places no limitations on the kind or length of response which may be made. An example is the ordinary school examination ques- tion commencing: “Discuss the development of . . . .” Such a question consumes a great deal of time in answering and is very difficult to score. Even at best it cannot be scored as objectively as some of the other forms. It may be valuable in selecting secretaries who can write good, clear English, or in sampling ability to organize one’s knowledge for logical and effective written presentation. In the oral trade test the response is free but the answer is scored as correct if a certain crucial word is used in the response. Three common types of items calling for constrained re- sponses may be distinguished, varying from the least con- strained to the greatest. Some questions are so worded as to require the answer in certain definite units or to place other limitations on the re- sponse. In the question: “How far is it from New York to Chicago?” no constraint is placed on the response; but if the question is worded: “How many miles is it from New York to Chicago?” constraint is placed on the response by requir- ing it to be in miles. In the free association test the person is given a stimulus word and told to write or speak as quickly as he can the first word it suggests to him. No con- straint is placed on his response. But in the controlled as- sociation test the directions require the person to give a certain sort of reply, such as the opposite of each stimulus word. In the form of controlled association test known as the “completion test,” the stimulus is a sentence with one or more words missing, to be supplied. Variants of this form 07 EMPLOYMENT PSYCHOLOGY are “From New York to Chicago is — —— miles,” and “How far is it from New York to Chicago? — miles.” The response is narrowly limited by the meaning of the context, and the place for recording it is specifically indi- cated. Still greater constraint is placed on the response when instead of allowing the subject to state or write his response, he is asked to indicate which of certain responses is the cor- rect one. If the number of alternatives is more than two, the test is called a multiple choice test. An example is: «From New York to Chicago is 253 540 912 1164 miles. (Underline the correct figure.)” The greatest constraint is placed on the response when the subject is required to choose between two alternatives, as “From New York to Chicago is 1,061 miles. True False (Underline one).” As constraint is placed on responses the objectivity of the examination increases, especially as regards scoring, but the reliability of the separate items decreases. The two- choice response is free from the difficulty of scoring the large variety of answers encountered in free responses. On the other hand, a single correct response, if free, has greater reliability as a measure of information, mainly because the two-choice form allows of guessing. Consequently, as more constraint is placed on responses, the number of items in the examination must be increased to gain reliability. The investigator should strive to obtain objectivity by placing constraint on the individuals responses and yet to retain reliability by allowing some freedom of response or by using a sufficiently large number of items. The happy medium is usually found in questions of types B2 or B3 (a) in the above classification, with not less than 50 items to an examination. Certain forms of test presentation have proved popular. The multiple choice test, with its many variations, is one of these. The usual number of alternative answers is four. 08 PSYCHOLOGICAL TESTS ; They may be placed within a statement as in the example given above, or they may be placed after a question. The incorrect answers should be chosen for their likelihood of confusion with the correct one. The position of the true answer among the alternatives should be determined by chance. In the analogies test, which may take the form of a mul- tiple choice test, each item consists of two sets of terms analogously related to each other and expressed in the form of a proportion. In the earlier forms of this test the fourth term of the proportion is omitted and is to be supplied, as in a completion test. Thus: Grass is to green as sky is to __ at 0". Grass : Green :: Sly ; ——— The same considerations which have led to the wide use of the multiple choice response in other forms of informa- tion and controlled association tests have resulted in making it the preferred form for analogies test items also. In the following example the instructions are to underline the one of the four last terms which bears the same relation or analogy to the third as the second does to the first. The correct answer is indicated: asset—liability; creditor—invoice, banker, debtor, bonus Another form supplies five alternative terms, two of which are correct and must be checked. For convenience in scor- ing, the possible answers may be numbered, the instructions being to write the correct number in the space provided far the answer at the edge of the page, thus: Hammer is to trip-kammer as gimlet is to (1) lathe, (2) drill-press, (3) milling machine, (4) template. . . . .. (2) Free and controlled association tests may be used in a great variety of ways in vocational testing (225). Whipple gives a full account of their use, and interpretations of the functions which they measure (220). They may be used in OG EMPLOYMENT PSYCHOLOGY measuring special information by listing a number of ob- jects and requiring the person examined to name for each the whole of which it is a part, or the use to which it is put, that is, to express in one word the field of its use. Some uses of the free association test are given in Kent and Rosanoff’s monograph (go). Besides its original use in diagnosing types of mental disorder, the free association test has been found valuable, for example, in detecting sales aptitude and differentiating it from mechanical aptitude. The word nam- ing test, in which words are named as rapidly as possible for a specified time, forms a part of the Terman-Binet in- telligence tests, and is also to be found in Whipple’s Manual. For full accounts of the various forms of association tests the reader is referred to the sources of test material. The completion test in a variety of forms is often used and although difficult to score is valuable in measuring special information. In the best forms a blank space is supplied for each missing word. Example: When knocking a cutter out of a collet it is best to use a Sait ee iY the tang hele. An information test may be in the form of pictures of various objects or processes in which important parts are lettered. The subject is instructed to give the names or uses of the lettered parts. All the names may be listed beside the picture and instructions given to place the proper letter be- fore each name. A form of test which has been used to measure ability to deal with concepts of space relations presents a picture or diagram of an apparatus or machine, each part being let- tered, with instructions to indicate what would happen at y if x were moved. (See Figure 4.) Look at Figure 4 and answer as many of the questions below as you can. Answer each question with a single letter. If you don’t know, guess. 11llustration and questions from Stenquist Mechanical Aptitude Tests. Copyright 1921, by World Book Company, Yonkers-on-Hudson, New York. 100 PSYCHOLOGICAL TESTS —W >F — J = 50 Figure 4 Space for Answers ¥. Where does the air COME MNP. u.venrsrrnerasnnene eer. i 2. Where does the air go out?....... de uy bo BR ia ily Se 4 3. When the crank K is turned, which revolves the fastest, £,0 W, or V2....... ip Lian wg 4. Which revolves the slowest?............ VYIITTTD DYN 5. If the crank K is turned in the direction shown by the arrow K, will the fan turn in the direction shown by arrow marked B, or arrow marked X? Write B or 6. If the pinion marked W were the same size as the one marked E, would the fan turn faster, slower, or about the same? Write F if faster, S if slower, and N if about the same... .. ss ID, «vive aero 7 If crank K is made longer, will it turn harder, easier, or about the same? Write H if harder, E if easier, and Sifisame. ove... oe tien iis ve tore cas 101 2 EMPLOYMENT PSYCHOLOGY Pictures or descriptive phrases representing steps in a continuous technical process or successive events in a dra- matic or anecdotal series may be presented in disarranged and apparently haphazard order with instructions to place them in proper sequence. A variation requires the subject himself to supply the last step in the process or series. Directions tests are not concerned so much with measuring informational content as with determining the ability to carry out more or less complicated instructions. There is no limit to the possibilities of such tests. They are good measures of alertness. Examples are to be found in the army report (228) and in Woodworth and Wells (225). The familiar Woodworth and Wells Hard Directions test commences: With your pencil make a dot over any one of these letters F G H IJ], and a comma after the longest of these three words: boy mother girl. Then, if Christmas comes in March, make a cross right here. .....but if not, pass along to the next question, and tell where the sun rises............ If you believe that Edison discovered America, cross out what you just wrote, but if it was some one else, put in a number to complete this sentence: “A horse has. .....feet.” Such droll content never fails to provoke merriment, as well as maximum effort in the persons being examined. It should, however, not be used unless the testing program is firmly intrenched, because business executives do not like to be placed in the position of advocating anything which gives the impression of being ludicrous. Psychological test- ing has made distinctly slower headway in industry than it would have done if some of the test materials had been chosen with more care to avoid any unnecessary appearance of the frivolous or silly. It would be beyond the scope of this chapter to attempt a complete summary of the possibilities in verbal testing. Additional methods of test construction are illustrated by 102 PSYCHOLOGICAL TESTS the familiar number series completion test, synonym- antonym, cancelation, X-O, vocabulary, pairing proverbs, naming what is wrong in a picture, memory span, imagery, memory, word-building, disarranged sentences, report or testimony tests, invention, and so forth. The following references cover pretty completely the varieties of verbal test construction: I, 9, 22, 23, 28, 35; 38, 44, 48, 54, 58, 61, 64, 66, 72, 90, 98, 99, 101, 104, 100, 117, 124, 129, I41, 143, 174, 180, 185, 202, 205, 216, 220, 222, 225, 228, 231. Reference 129 deserves special attention. The following chapter describes typical performance tests and summarizes the technique of test construction. 103 VIII PSYCHOLOGICAL TESTS (Concluded) Performance tests. Technique of test construction. Order of items. Length of test. Time limits. Directions and fore-exercise. Directions for scoring the test. Omnibus tests. Alternative forms. Arrangement in series. Performance tests require something other than a language response. They are usually administered as individual tests, although there are many group performance tests. Their best usefulness is to measure manual skills, mechanical aptitudes, and personality and character traits. They are indispensable in testing the intelligence of illiterate or foreign applicants. A performance test may consist of merely one step or standard task, the score being in terms of the time required to complete this task; or it may consist of a number of units, the score being either in terms of time required, number of units completed within the stated time, or quality of product, measured by comparison with typical samples arranged in a scale of excellence. If apparatus is used, it should be of sturdy construction, and easily replaceable. It is advisable to have a duplicate set for emergencies, and in any event the specifications for the apparatus should be so explicit that any investigator can duplicate it with entire certainty that he is reproducing the standard conditions. To this end the use of rare and unusual materials is deprecated. The apparatus should if possible have automatic devices for recording performance. Laboratory manuals for experiments in psychology are a 104 PSYCHOLOGICAL TESTS 3 fruitful source of information on the technique of perfor- mance tests. The job apparatus suggests materials. Some well-known performance tests are steadiness, tap- ping, tracing, card-sorting, picture completion, form board, maze, puzzle box, manipulation, assembly or construction of a mechanical object of which the parts are given (Sten- quist test, Kelley contruction test), and the newer tests of personality in which a social situation is set up by making the examiner an integral factor in the apparatus (119). An example of a test of the latter sort is to be found on page 118. Some illustrative performance tests are shown in F igures 5, 6, 7, and 8. Explanation of Compound Slide Rest Test (Figure 5). This apparatus consists of a small traveling platform which is made to move in any desired horizontal direction by means of two screws placed at right angles to each other. The direction of movement of the platform is governed by the relative speed with which the screws are rotated. A pencil is placed loosely in a clamp so that its point rests on a piece of paper mounted on the platform. The subject is required to move the platform in such a way as to cause the pencil to trace the outline of a diagram on the piece of paper. The diagram may consist of parallel or con- centric lines forming a track. The person tested is instructed to keep within the track and proceed as fast as he can. Each digres- sion of the line outside the track is recorded as an error. Tagg (181) compared performance in this test with trade ranking, using as subjects young men in training for various trades at a technical school. He reports the following correlations between test performance and trade ability. Draftsmanship ..........cvsex 142 TOarning ... 0. «coi viie deviate oid 200i PRUNE Louvain ich ve al Patternmaking .............. %4 Machine-operating ........... .3- [Boolmaking ....., . o. u. lhl JE Dr. J. L. Prak, psychologist at the Philips Lamp Works, Eind- hoven, Holland, using this test with three groups of boys at a trade school, obtained correlations of 46, .32, and .53 with trade ability as judged by teachers. IO¢L EMPLOYMENT PSYCHOLOGY This test has also been used with success in Germany in choos- ing shop apprentices. Patten (132) used a similar apparatus with students in the College of Engineering of the University of Wisconsin. Instead of pencil and printed diagram he used a metal stylus and a hard- rubber plate in which six small brass pegs were imbedded to form a hexagon, flush with the surface of the plate. As the stylus traveling over the hard-rubber plate came in contact with a brass spot, a buzzer sounded. The time to make the circuit of the six brass spots correlated —.423 with the quality of lathe work turned out in the Shop Practice Course. Explanation of O’Connor Finger Dexterity Test (Figure 6). The equipment for this test consists of 300 brass pins and a flat metal plate (mounted on an oak board) in which 100 holes are drilled part way through. The subject, using the right hand only, takes the pins 3 at a time from a tray in which they are piled loosely, and places 3 in each drilled hole as rapidly as possible. The test is scored in terms of the time required to complete the task. Individuals differ widely in this test, the fastest finishing in less than 6 minutes, while the slowest require from 12 to 15 minutes. Hines and O’Connor (70) have reported results obtained with the use of this test at the West Lynn Works of the General Elec- tric Company. Seventy-seven girls engaged in fine meter or instrument work were tested. Of these, 60 proved satisfactory, 6 failed and left, and 11 left before their proficiency could be judged. Of those scoring 7.50 minutes or less, 91% proved satis- factory, and none were failures. Of those scoring 7.51 minutes or more, 45% proved satisfactory, and 27% were failures. Explanation of the Wechsler Chauffeur Test (Figures 7 and 8). The subject is seated in a dummy car which resembles an ordi- nary cab. He is instructed to look straight in front of him at a gray board 10 feet from where he is seated, upon which intermit- tent lights of different colors are flashed at irregular intervals. When seated in the car he is told to take the position for ordinary driving, that is, his left foot to the left of the clutch, his right foot on the accelerator, and his hands on the steering wheel. He is told to imagine that he is driving along at ordinary speed, and that in no case is he to press down upon the gas which is already being fed at the proper rate. However, upon the flash of certain lights, which serve as signals, he must react in certain ways by an appropriate movement of the hands or feet, or both. For instance—at the flashing of a yellow light (meaning to slow up) 106 Figure 5: Compound slide rest. Figure 6: O'Connor finger dexterity test. O07 Figure 7: Dummy car and flash board for testing drivers’ carefulness and speed of reaction (214). Figure 8: Apparatus for automatically producing stimuli and recording reactions; used in conjunction with the dummy car and flash board in testing carefulness and speed of reaction. 108 PSYCHOLOGICAL TESTS he must come down upon the clutch and foot brake. After each movement or response he is told always to come back to his original position, but in doing so he must not come down hard upon the accelerator. If he does this a mistake is registered. The apparatus is so devised that the signal lights are flashed automatically and remain lighted for a definite short interval of time. In order to be credited with a correct response, the subject must react with the appropriate movements within that given interval of time. If he makes incorrect movements or reacts too slowly, this response is not registered. On the other hand, if he is not careful and in his excitement strikes the accelerator, his mistake is registered automatically on an electric counter. The number of correct responses gives an index of his alertness and speed of reaction; the number of errors, a measure of his care- lessness. Throughout the test the subject’s responses and reac- tions are automatically recorded on electric counters which total each type of response separately. All that is necessary at the end of a test is to read off the figures from the several electric counters. The duration of this test is r 5 minutes (214, pp. 25-26). When tried on the drivers for a cab company in Pittsburgh, this test gave the following results. These figures refer to the €Iror score, a measure of carelessness (214). Those who passed the test (made fewer than 5 errors) aver- aged 1.3 accidents per man. Those who failed in the test (made 5 or more errors) aver- aged 3.0 accidents per man, Of those making no errors in the test, 53.0% had accidents. Of those making from 1 to 3 errors in the test, 81.2% had accidents. Of those making 4 or more errors in the test, 87.5% had accidents. Of those making no errors in the test, 54.6% were rated high- est in carefulness by superintendents. Of those making from 1 to 3 errors in the test, 31.25% were rated highest in carefulness. Of those making 4 or more errors in the test, none were rated highest in carefulness, Figures 7 and 8 are reproduced with the kind permission of Popular Science Monthly. Dodge’s test for the selection of gun-pointers in the navy I0Q EMPLOYMENT PSYCHOLOGY illustrates several principles which it is well to bear in mind in devising any performance test (46). The instrument consists of a battery of four skeleton guns, with training or pointing gear, a mechanism for giving the targets a series of harmonic wave motions of great variety and complexity, a recording device, that shows each movement of the target and the corresponding movements of the gun as the pointer or trainer tries to follow it, and a firing device, that not only indicates the accuracy of the pointing but also the effect of the effort to fire on the pointing coordinations. Points to be noted regarding this test are: 1. The task is closely analogous with the job. >. The instrument is self-recording, the performance being subject to later inspection and analysis. 3. The apparatus is useful for training. With it the man can train himself and does not require the constant atten- dance of an officer or experienced operator during the prac- tice periods. It also releases expensive and limited equip- ment for active service. ORDER OF ITEMS When the items of a test are of practically the same diffi- culty, their order may be determined by chance, but the in- vestigator should assure himself by preliminary trial that the items really are approximately equal in difficulty. The preliminary order may be based on the judgment of the investigator. The relative difficulty of the items may then be determined by finding the percentage of a group that solves each one correctly, or by finding for each item the average time required to do it. If some items are harder and others easier, they ought to be arranged in order of difficulty. When the easiest items are placed at the beginning of a test, the subject quickly becomes adapted to the test requirements and a better dis- tribution of scores is obtained. A number of hard problems at the beginning of the test discourages some persons to TII0O PSYCHOLOGICAL TESTS the point where they refuse to continue, whereas those who are able to complete these difficult problems find the sailing smoother with the problems which follow. The resulting differences in score exaggerate the real differences in ability. If the items are of equal difficulty, speed is measured. If they are in order of increasing difficulty, then not only speed but level of ability is measured. LENGTH OF TEST The test should provide a sufficient sampling of the ability to yield a reliable measure. The law of diminishing returns applies to the length of a test. The first few items are the most valuable. Additional items increase the reliability of the test, but in diminishing proportion. It is better, how- ever, to err on the side of having too many items than too few. The validity of each item may eventually be deter- mined and the test reduced to include only those items which show the greatest vocational predictive value. The test may also sometimes be shortened after correlat- ing scores made in a part of it with scores made in the whole test. Any part of the test which correlates more than .95 with the whole test may be used in place of it. TIME LIMITS A time limit is not essential to a test, since it may be scored on accuracy alone. If accuracy is the variable which the test aims to measure, the time limit may be set so liberally that all but a very few (about 5% or 10%) are able to complete the task. But there is no point in allowing a person to fumble over a task indefinitely when it is obvious that he will not be able to complete it satisfactorily. A time limit which is set for convenience aims to meet this situation, and not to measure speed. On the other hand, if it is necessary to measure speed, the IX EMPLOYMENT PSYCHOLOGY time limit is set so low that not more than 5% or 10% of the group are able to complete the task. Speed is then measured by the number of units completed in the given time. Most persons who take psychological tests are under the impression that they ought to finish the test within the time limit, and only a great amount of explanation will convince them that this is not the case, and that the important thing is their rank relative to the others who take the test. Some examiners have adopted the plan of stating that the test has been made so long that no one can possibly complete it in the time allowed, setting the time limit accordingly. The time limit may be determined by preliminary trial with a few subjects, or it may be determined in a group examination by calling time when 5% or 10% (or go% or 05%), depending on conditions) seem to have finished. This time limit, of course, should be rigidly adhered to thereafter. DIRECTIONS AND FORE-EXERCISE Since the directions are intended to make sure that the person to be tested knows exactly what is expected of him, they must not be too brief. They cannot be too clear and explicit. It is even more important to make the directions absolutely clear in group testing than in individual testing, for time cannot be taken out if questions as to procedure arise after the test has begun. The wording should be chosen in accordance with the educational level of the persons to be examined, and rarely does an investigator strike precisely this level without repeated trials. He unwittingly errs by using terms outside the accustomed vocabulary of his sub- jects, or he chooses illustrations so obvious or childish that they amuse or offend the subject more than they inform him. It is a skilful maker of tests who can strike the narrow zone of clarity while avoiding both the abstruse and the silly. To make sure that the directions are understood, sample tasks and solutions should be given. The best tests provide, TI2 PSYCHOLOGICAL TESTS Hn in connection with the printed directions, problems which give the subject a chance to try for himself and see whether or not he clearly understands what is required. These fore- exercises make for reliability by putting all the contestants on the same footing. In the introductory statement, it is sometimes advisable to use the word “Form” or “Worksample” instead of “Test,” as being less likely to make the subjects apprehensive. The directions may either be given orally or presented on printed sheets with the test, or both. They should not be on the same side of the sheet as the printed test, as that allows the rapid readers to solve some of the problems before the starting signal is given. They should preferably be on the back of the sheet, and at the starting signal the subjects turn the sheets over and begin to work. Instructions should be given to cross out mistakes instead of erasing them. Important statements in the directions should be in capi- tals or bold-face letters. Where speed is not essential, the starting signal may be merely, “Go ahead,” or “Start,” but if speed is desired, a suggestion to that effect is given by saying, “Ready?” (pause) “Go!” The stopping signal may be “Stop” or “Pen- cils up.” Some examiners have made use of a bell as a start- ing and stopping signal. The directions should be given a preliminary tryout with a few subjects to see if they are perfectly explicit and fool- proof, but even with this precaution it will be necessary in most cases to make modifications in the light of further experience. If the test performance is not obviously related to the job, it is most desirable that the directions should furnish some motivation to the subject by pointing out the resemblance of function measured. Instructions for administering trade tests are given by Chapman (28). Ix J Form $333—June, 1025 SAMPLE QUESTIONS The general charecter of the exsmisstion is iadisstad by the following material. Study it carefully, sod suswer each question GENERAL TESTS FOLLOWING INSTRUCTIONS i Write the number of the bral answer on the line st the right. Talla las sont of your ability to tulle msirustians. All directions must be followsd exaotly as shown. sample teat. » 1. The busioess of mail-order Girma bas been greatly increased by the introduction of ~ (epevisl Below a% the lait io 8 Yat of post offess, called a SORTING SCHEME. After gach of thess offices delivery (2)parcel post (3)posiel saviogs (d)airplave mail (Siock boxes. . + + +: w= Sem esamales Afiee "By iE S Jottey °A Tha relare wn the "ALIS i. Feat ET vom aint i heme SEL pe orem sh al sot . delivered the office so that the (1) can bo on BL Joh wi um 3 for. % Tn ets Fiaan Lo a li Tl ok botany (@lessers will not be damaged Boni Tria. ais mal Tor Bowers Towle by way of Felon, mai fo Bovis, sl would be “{Slpublic may not be incoavl FEE Ae x . — SORTING SCHEME ar Sail ent by way of— 3. Mail which ls sent from an addressse’s oid address to his pew sddress is said to be (Dreturnad Woes as isis on 3 ere Crlormarded (received Ghdsiaimsd. + + +n + eens == Bove ale le > i Daly....oooleieniiiiiiinins Camden ....8.....ccouunen 4. A fundomental point is one thet is (DSsal (Ddrastie (emphasised (emsectial Pope fomdes Borns rr . a Pome LoL III tat. “To ilodau must Bodoisg Dost aerly. (1)Vet yt de. do tharoedly: i cet aes thet oni a - at pe ae Ee ™ BRL Sle you wt” Wb ur dees ors Peni ne 4 laowa. (SWEIBuus wheldom, . o civ» vip eon ve AE 0 nS Voth irate a - Statement (2) means most nearly the sme ss “To do, cae must be doing." so write 2 on the line. Vaiss ual nniiecns Meeiteaaaesasseaariens 5. The saying, “Many hands make light work,” mesos most pearly (1)Most people prefer easy obe (2) When several work together the task is casier. (3)bugh light work can be dono EXACTLY A3 GIVEN. Make your numbers and letters ae te us ives iris harder whos wackine JOU MUST FOLLOW DIRECTIONS > . TE A EE ER sens = rt Ga EY. Tired oa. LAE, v : SU on Ea itr ~Wooda” In (hqBORTING SCHEME. Your KY wil ney rad 5 Woods 4. 1a eseh of the two sentences below, tho word printed a italics has been misspelled. 18 3 spelled ac- Flag de lettars aiter Allen ad Dover in the SORTING ‘aad write them after those Bazes pording to its sound. Write the correct spelling of this word op the Liss ef the right. a the KEV. Sor 1a the SORTING SCHEME ann i er put num) 5 7. Tha plan was sonksbunncd by tho commits. - ++ oes mmm IESE (oowing offer. wri the number of the train oa which you would sad wail for that office. eae 3 i i aie ei Ch RT Re hich should be written after Viols, lock for Viola in the SORTING SCHEME. 8. Hia noflepe of the situation was remarkable. r—— pic en the RnB." Bs to KEY D Union 8, and means that mail for Viols is routed throush . Which one of these five may ba spplicd to both books and magarizas, but Bot to postman? noe op Tia 6 This refers to, KEY E Allen C. aad mesos that mail for Desbam is OT a Ee Toners. (iuibotied. (IPUBD. =» = ooo or m= Ar Dsban iy habits 8 Te a EE ema wn Tra Co Writs “6 Sher Acswer "5" ia the only ono which ean bo applied to both “books” aad “magazines. but not ta “'post- Deabain ia the tak bow. Now write the train pumbers sites the others. tan. 0 write 5" ob the lize. A eo Turser v.ooo — Maltor ... — 10. Which one of these ve may be spied to both pot affcs sad stra, but not to srivle desing? Denbem. +0 — Daly corse =r Ee a He : a he; You now recowe Balletla No. 1: IN BOUTING ing questions. tm words In capital lsiters go together in some way. CRANGES TE I Ba Him Mimbo to shun bith ofthe lst ve words toca with te third Never change the lear balers the pans ia the KEY. When 8 letter or number 1 changed. it is Fd ie eabital lettore Ic he aa cas way that the second word io capital letter soos with the fist. Aaya le fete of pamber sis to BAK 15 sad KEY if the shioss or (a boib) hain 2 43 ods oC 11. BACK isto MAIL ss PURSE fa to (Dmooey (suitesso (bag (Downed (Dluzory. —0 Weslo dy ou oo DLE is to BUSY as ABSENCE is to (slo (Dwilisgsess (S)pressace (sadness To make the for We ¢ tbo “A™ after “Wools™ in the SORTING BCHEME aad IPRRLRT AN 0p en, ie Pr OUR REALE Meenas a > ‘Dover so that your SORTING SCHEME will read 15. Over what body dose the Vise President sresidel (DBenst (2) Hous of Repressotatives the “An ples to CL Maks the SR aD es (@1sterior ()Bupreme Court (H)Cabioet. + + » » + + +» + + + ew er § ting the shove changes, write the number of tho BID 0d which you would send mail for cach of the following offices: ! Answer the question from the quotation which follows ft. D = {4. What one word In tbs Quotation describes the person who is not isterested In publie welfare? mmm Bi — ute ur mm As lz slimes: every community there nse eartaip sun snd women who seo Lyowp as publiespisited. Eine) — gress BOSE Othors may be 80d act only as their privats interests seom to require.” Next you rossivo Bulletin No. 2. I distribute 700 letters in 2 bours. in how meny hours would they distribute SR 16. If 4 men can ‘mea hours would they 1.750 (Make cheages in both SORTING SCHEME and KEY. if the names aro in both) a SL RE SR A RO Change KEY C to read: C Camden 3 Changs KEY, to rod: E Aen? Change KEY A to read: A Train 5 Allen Ly way of Bulow are the correet answers to the “Genera! Testa” samples. Check yoor soswers with thes and Feitoa by way of B se bow many error vou have mde, Ba oars Yon Vileriand Dow (hs QUaions ar 40 59 ABIWOR #5 To make tho change for KEY. C, crom out the § after Camden ia ths KEY aad whta “8.” Lesause that you will not make errors in the test. Ba orator Tas been chased fiom 20° Lo 2" hers KEY A sent ‘olion. 3 sad write | Train Ey ac that wail for offices ei der Cumtin Aner LR 0 Jonker seat sheesh Felian t is rou! iret on . Bobo 30 Rebomminninin gRoiond wise the mumber of the tain on which you would send eas for i 8 No. Buiiiicuinnsrcsssnsssnannascasee: F Bowers Pelton .... =o Daly .... ry No. JO. .ooiiiriiniitriniinneaionsneisns 8 a -— No.8. vociniscirnsassasesanaeas 1 Nell co eres nicaressn Dover Lo. em Denham... von Alled Lil No.6 tivirciiinciasnennnsasanns NS JE corvssnssnisii rss sein senes ie 3 Bul No.3: RU OFFICES ESTABLISHED CHANGES IN ROUTING No. 18. causrnrrnnssnsareansse inrases Obes (Add 10 SORTING SCHEME) (Makechanges io both SORTING SCHEME sad KEY if the names are in both) Booker by wey of A Add to KEY: J Tam 4 4 SORTING Wise by way of D Change KEY D to read: D Tisin 8 I crs tr Ei : Ly 15 the SORTING 2 ; i SE amas i i nus . - " : eT ae So BL Bocas tiie bth Sov cmcmess: 51300 Pucker to, the SORTING SCHEME, write “Basher A” om the Erst dotted Hine at the end of Tandvond 0 0 Bor be a tlways be writion slits Lontrecy, . To 8dd “3 Train 4" to the KEY, write "J Train 4" on the fst dotted live at the end of the KEY. Smt 3 is order. You will voueive: no ereditif you skip cities To make the chung fou KEY “KE.” eresout "Allen 7 tn the KEY and write flougrs Then find es ne SORT CT SCHEME for 10 minutes. to get it thoroughly in mind before beginning to write, “ieties after “Bowers” Rn the SORTING SCHEME, sad mite aot a SESE aad write 1,” then, m the KEY, cross out “8” after Union and write “ #." Mecke the other changes ordered. ~- SORTING SCHEME Write the number of the train on which you would send mail for: - 1 Becker ... — Unioh ... —— Desham ..: eo Allen .. —m Malter... ad 2 Pina 3 BE, au. vi 0 hooker Daly pee WBE oss Foon six me: CHAN: 1 rn NVI nes mi Carter Bunset Grande Appleton Sumter Nors.—1f you have answered “8” in place of “9" for Viola, it is because you have failed to follow the em last direction in Bullotin No.3. If you bad changed Union to # in both KEY and SORTING SCHLME, Four Anawer would be = 9.7 1 you have mede all changes corredtiy. your SORTING SCHEME snd KEY should read: 1 SORTING BCHEME KEY. Harbur Eastlake Randsll Leadwood Camden {Bail sent by wey of Tho correct answers to 2 Roluge 4 Poston 6 Lowell | | 8 Fox 10 Roowell TRE. W- ERRRIN Ha 5 ys fri am BoWermassueBereervrvirrnnnee B vsen one . ; Camden.s.. Curernenssisneens © Csindes Ng = + = © Youmay! © Ce ° i Dalyan Fine D cs. Thm — 2 a N . "en 88 you wish. Youtmey =r. % == timc sha +1 *=N a8 you can in the time allowed. DRESS iE re = ns shsemens EB Mier 7 Arrthe 8 0 8 City Boz No City Box No City Box NoJ City Bez NoJ City Bos Noy City Bon Now Boe BB rere i eh puter | oH poi | Reives et Lo a Fira ri A Biri’ & pe, £2 ECE Ue Lesdwood £7 | Lowell | Boston .. | Rayburn . | Milbrook — | Appleton wo MsltéfeseBirecrsusnnnsnss 7 4 rire Fox Carter —| Painter Lakeview Grsade Forest -— 5 8 8 Edison | Desver | Roowell — | Moron — | Hands _ | Chester — Tumer.osoH ooiiianes hi de SLi 3 Porter Saaser Sumter | Bestake Wheeler Xam T= he 5 5 7 7 5 _ EE stwnimpngrranian era Aided Buide Bw Bio goes Ren lowe weeds A Gin ER Peister es "| Lowell Boston =] Mom Edison’ Tw Ugion TE 5 98 8 8 Yilbook | Drege | Coir | Bumie | Ree | Kebum geen REL ce Asean Esa ianssaseiss 23.23 Beaton Lak, Randall Painter Fox Leadwood mm Boho ARAUA nn da sis sas Figure 9: Sample questions prepared by O'Rourke (124) so that all applicants for the Postal Service have equal opportunity to practice in advance of the examination. - & y= Fo EMPLOYMENT PSYCHOLOGY DIRECTIONS FOR SCORING THE TEST In deciding upon a scoring method, objectivity and con- venience are both basic considerations. The score should be simple, precise, and capable of being obtained quickly. Above all, it must not be subject to variation because of the personal equation of the scorer. A key of correct answers may be prepared in a way to facilitate accuracy and speed in scoring. If a test form has been printed so that all answers are in a single vertical column, the key consists of a strip of cardboard for each page, with the correct answers spaced so as to be instantly compared with the test responses. If, however, the test is such that the answers do not all appear in a single vertical column, it is better to make a stencil. Some stencils are of cardboard the size of the page, with windows cut wherever answers are to be. The correct answer is above each open- ing. Such an opaque stencil has the disadvantage that it may hide an answer written in the wrong place. A trans- parent stencil of celluloid or tracing cloth, with answers indicated in India ink, is preferable. There should be a convenient place on the test form to record the score, or a card should be at hand on which each person’s score may be recorded. Below are some of the ways in which a test performance may be scored. Time required to complete the task. Number of units attempted, regardless of accuracy, within the given time limit. Number of units completed correctly. Number of errors, or number of units completed incorrect- ly. Number of units completed correctly minus the number completed incorrectly. This last is the customary method of scoring tests where only two possible answers can be given to each question, 116 PSYCHOLOGICAL TESTS ; as it is supposed to compensate for the effect of guessing. If a person goes through the test and indicates his answers at random, or by guessing, about half of the answers will be correct. If the number wrong is subtracted from the number right, his score will be zero. Those solutions which represent mere guesses cancel each other. In theory this method of scoring eliminates the factor of guesswork. It is true that by guessing, a score of zero on this basis will be more probable than any other score, but the majority of scores obtained by guesswork will not be zero, just as in a normal distribution curve although the modal score is the most probable score, most persons do not make this score. The chief recourse is to increase the number of items. The greater the number of items in a two-choice test, the more restricted the range of scores that can be obtained purely by guessing as compared with scores representing knowledge. Several experimental studies of this method of scoring have been made. Ruch and Stoddard (15 5) and Paterson and Langlie (131) both discovered that the R-W method of scoring was less reliable than R alone. Wood (223) found that the validity of R-W was greater than R alone in four out of five examinations when the number of items in the examination was above fifty. Thurstone gives an exact method of arriving at a proper weighting of right and wrong answers. (See page 187.) Another method of scoring is the difference between true values and guesses or estimates which the subject is re- quired to make. The number of times a standardized prompting must be given in order to make the individual carry on the task to its successful completion has been used. Various ratios may be treated as scores, such as the ratio of an estimate to the true value, the ratio of one score to another, or the ratio of scores in one part of a test to scores in another part. XX : EMPLOYMENT PSYCHOLOGY The score may be the difference between the performance of the subject and some previously established group norm or consensus of expert opinion. The score may be the difference between test perfor- mances of an individual under varying conditions. If the differences between subjects in test performance are discrete, the investigator can assign numerical values to the types of response, in accordance with his judgment of their excellence, or he can build a scale in accordance with experimental findings based on the Cattell-Fullerton principle that “equally often noticed differences are equal,” the principle which underlies the construction of many edu- cational scales such as the Thorndike scale of merit in handwriting. It is not always necessary, however, to assign quantitative values to qualitative differences of response. If the subject is required to place a number of variants in their true order, correlation with true order may be used as the score. Some tests have been scored by rating scales; that is, the examiner observes the type of response and then by refer- ence to a table obtains the score which is assigned to that type of response. This method lacks objectivity, but no one has determined just how much error exists in it. For certain types of responses it is the only feasible scoring method. An example of the use of a rating scale in the scoring of a test is to be found in the Manual of Directions for the Dow- ney Individual Will-Temperament test. The directions for Test 12 (called Resistance to Opposition) are as follows: “Close your eyes and write your name again with this pencil. Be sure to keep your eyes closed.” After the writing of the name is well started, place a small obstruction (such as a small fountain-pen pasteboard box) in front of the pencil-point, exerting enough pressure so that to con- tinue writing will require considerable effort. Continue the pres- sure until the name is completed. If the subject stops movement or opens his eyes, say, “Go on.” 18 PSYCHOLOGICAL TESTS Repeat the urging to continue working, if necessary. If the subject dodges or jumps the obstruction, replace it in front of the pencil. Record should be made of whether or not urging was required before the subject made an effort to continue writing. Also the degree of pressure should be recorded, and observations on such points as speed and the energy used in dodging or in deliberate and persistent evasion. Second trial. Repeat the test. The object of the Resistance to Opposition test is to determine the strenuousness with which the examinee will maintain an activ- ity he has initiated. The man with fighting qualities scores high in this test; the unaggressive person evades the issue or gives up. The scores which may range from o to 10 are assigned by com- paring the reaction of the subject with the following descriptive scale: Strong pressure against obstacle. Writing maintained at initial level; firm, strong stroke, usually enlarged charac- EIS. NO Uraing. «.. is cs eaer iret a 10 Very strong counter-pressure on level, but with some sacrifice of form; letters blurred or telescoped; trembling or undue speed, or other evidence of agitation. No urging...... 9 Very rapid and energetic dodging with or without mainte- nance of form; often, increase in size of letters. Or strong pressure but not on level. No urging................ 8 Very deliberate but gentle counter-pressure. Or deliberate dodging with mild counter-pressure and little loss of form. Or holding examiner’s hand back with the left hand, or protecting one’s own hand with left hand. No urging. 7 Evasive reaction: reversal of movement; shift of position; jumping of obstacle. No urging.................... 6 Very mild counter-pressure; loss of form. No urging.... § Strong pressure after urging and readjustment with mainte- DANCE OF SOT. Lo ae = 4 Moderate pressure after urging with maintenance of form. Or deliberate dodging after urging. ................. 3 Moderate counter-pressure after urging with some attempt 10 DIESEIVE HOI ules vas mss oi err ee a Feeble pressure after urging with loss of form.......... 1 IIQ EMPLOYMENT PSYCHOLOGY Absolute passivity in spite of urging. Typical remarks: “I can’t.” “How can 1 when you stop meP . Jinn va 0 For other methods of scoring tests of personality, see May and Hartshorne (199). OMNIBUS TESTS If accurate timing is not essential, and if the difficulty of administering the tests must be reduced, a number of tests may be combined into an omnibus test. The items of a number of different tests are included to form the body of the omnibus test, and the directions for all of them are assembled at the beginning of the test form. The subject is given a definite time in which to master the directions for the different kinds of tasks. He is then allowed to work on the test until time is called. This form of test saves time and labor in administering, and has been widely used in intelligence testing. The customary procedure in scor- ing an omnibus test is to compute only a total score on the test as a whole. But if the investigator makes use of such an omnibus test he should get the separate scores for the vari- ous types of items and study their value as if they were separate tests. Omnibus tests require of the person examined one ability over and above the ones measured by the individual tests of which they are composed: that of quick readjustment of mental set or shift of attention from one type of perfor- mance to another. The various types of items may be arranged so as to recur at regular intervals or in apparently haphazard order, but without increasing in difficulty. The test is then called a cycle omnibus test. If the items increase in difficulty with each recurrence of a particular type of task, the test is called a spiral omnibus test. 1From Downey Individual Will-Temperament Test. Copyright 1925, by World Book Company, Yonkers-on-Hudson, New York. 20 PSYCHOLOGICAL TESTS ALTERNATIVE FORMS Alternative forms of a test are ordinarily indispensable in order to reduce the possibility of coaching or cheating. These forms should be equal in difficulty and should corre- late highly (over .95) with each other. The best procedure for devising alternative forms of a test is to divide a large list of acceptable items into several piles purely by chance. Then by preliminary trial their actual relative difficulty can be determined with precision. When a battery of tests is used, the forms may, for con- venience, be fastened together, or printed as a booklet. They should be arranged so that only one test appears at a time, the easiest tests at the beginning, and others following in order of difficulty. A very easy and interesting test may be included with no intention of scoring it but of using it merely as a shock absorber, to put the subject at ease when he begins. Tests which do not require timing may be placed at the end, then the booklet may be handed in when com- pleted. Instructions should specify that no one is to turn back to a preceding test. The construction of psychological tests for the measure- ment of abilities is a complicated matter calling for a high degree of ingenuity and sound judgment, as well as a fa- miliarity with the principles outlined in this chapter. No amount of foresight can anticipate all the complications which may upset a smooth-running test performance. The investigator will find it necessary to try out his tests on a few subjects before putting them in final form, in order to correct unforeseen obscurities or sources of misunderstand- ing in the directions or in the body of the test. Before treating of the problems arising in the administer- ing of tests of abilities, chapters will be devoted to an instru- ment for the systematized estimation (rather than measure- ment) of abilities—the rating scale, and to an instrument for gathering factual data—the questionnaire. 121 RATING SCALES Methods of estimating abilities: Order of merit; grouping; the Scott form of man-to-man comparison scale; the scale of alternatives or two steps; per cent scales; multiple-step scales; linear scales; descriptive scales; graphic rating scales. THE rating scale is a method of recording estimates of abilities in a quantitative and comparable way. It is not, strictly speaking, a tool of exact measurement like a test. The records represent subjective impressions and do not have the objectivity and reliability of test scores and other more accurate measures of abilities. Nevertheless, they are an improvement over unsystematic judgments. In addition to their particular use in vocational selection, rating scales have proved valuable in determining progress in training, in indicating specific needs for self-improvement, and in solving questions of promotion or salary readjust- ment. In an investigation in vocational selection the rating scale serves the important purpose of gaging abilities which can- not be measured by more objective methods. The investiga- tor may recognize in advance that a certain ability which he considers essential to success at the job cannot possibly be measured by objective tests at the present stage of prog- ress in psychology. In that event, instead of ignoring the ability entirely, he may choose to construct a rating scale for measuring it. Ratings of employees on such abilities may be made by fellow workers and executives and, in order to determine their validity, may be compared with standings in the criterion of vocational success. Once they have been evaluated in this way, ratings by interviewer, examiner, IX 122 RATING SCALES 5 previous employers, or even by the applicant himself may be made an integral part of the employment process. The rating scale is seen to have its definite place in the investigation. Together with the more objective psycho- logical test and the perhaps less reliable questionnaire, it serves as a means of obtaining systematic information about the employee. Some of its inherent shortcomings may be diminished by an application of the practices about to be described. METHODS OF ESTIMATING ABILITIES Below are descriptions of some of the forms of rating scales and other devices which have been used as aids in recording estimates of abilities (55). In the first three of these forms each individual is compared with other mem- bers of the group. Order of merit. Considering each ability in turn, the judge ranks the men in order of merit, heading his list with the member of the group having the greatest amount of the ability in question and ending with the member having the least amount. Each man’s rank in the list is his rating. If the group is ranked by a number of judges, each man’s average rank in the ability is calculated. The names are then ranked in order of size of this average, which gives each man’s final rating in the ability. If the list of one of the judges is in- complete because he does not know some of the men well enough to pass judgment on them with reference to a certain ability, the investigator will have to obtain average ratings by resorting to one of a number of methods that have been developed to meet this situation (145,192). This form of man-to-man comparison is a brain-racking process, but there is a simple way of making the task less disagreeable. Each man’s name is written on a card. The judge sorts these cards into three groups, representing men X23 EMPLOYMENT PSYCHOLOGY who are high, average, and low in the ability in question. Then he takes each group in turn and ranks the men in it. By assembling his groups of cards in proper order he com- pletes his ranking. The order-of-merit method makes distinctions in ability which are finer than necessary or even possible if the group is large. It gives the rater no framework on which to hang his judgments. Moreover, unless statistical corrections are made, it assumes that a unit difference in rank always indi- cates a constant difference in abilities, whereas this differ- ence is greater at the extremes of ability than in the middle range. If 50 men are ranked in any ability, the difference in ability between the first and second man is greater than the difference between the twenty-fourth and twenty-fifth man. In spite of these disadvantages the order-of-merit method is one of the best known, since it gets at the heart of the matter by disregarding all other persons or impres- sions and simply requiring comparisons between the mem- bers of the group under consideration. Grouping. The judge places the subjects in three groups, consisting of those who excel in the ability, those who are undistinguished in the ability, and those who fall below in the ability. The judge may or may not be instructed to make the groups equal in size. The number of groups may be increased at the discretion of the investigator. Symonds contends that seven is the optimum number of groups, although he has not proved his point experimentally (179). Ratings of several judges may easily be combined if a nu- merical value is assigned to each of the groups. The Scott form of man-to-man comparison scale. This is the rating method originally developed at Carnegie In- stitute of Technology for the selection of salesmen, and later adapted for use in rating officers in the United States Army during the World War (137). The judge makes a master scale for each ability by the following process: He selects the highest and the lowest man in the ability in the group 124A RATING SCALES 125 (or among his acquaintance), a man of average ability, and two other men, one midway between the highest and the average man and one midway between the lowest and the average man. These five men serve as standards by which to judge all the other members of the group with reference to the ability in question. Arbitrary numerical values have to be assigned by the investigator to the steps in the scale represented by these men. For example, the amount of the ability possessed by the highest man may be set at 25, the average man 15, and the lowest man 5, and the other men 20 and 10. The master scale for an illustrative ability then looks like this: CONVINCINGNESS Consider the tact, clearness, and force with which each sales- man presents your interests to customers. Highest MIE vs ries rset ne ra ea 2S High MIE, hiv not caress avs sins sie s3D) Middle ERR CEL RIA hg Low MIE. hl da aay Lowest HTTP ee The judge prepares master scales in this way for each of the abilities in question. Needless to say, the men chosen as representative will vary with the ability. For example, the judge may consider the leader in one ability to be of only average standing in another ability. When used for interviewing purposes the numerical values should be weighted in accordance with the importance of each ability. The master scale for an ability very important for the job under consideration may range from 10 to 50, and for a less important ability from 2 to 10. In rating the group the judge considers one person at a time and records a numerical rating on each ability after comparing him with the master scale. In the example given above, the rating may be any number from 5 to 2s. This method has been found cumbersome in use. It re- quires a good deal of time and careful thought on the part EMPLOYMENT PSYCHOLOGY of the rater in preparing his master scales. Raters differ considerably among themselves in judging men by this method. Some of them tend to ignore the key men entirely in making their ratings and fall back upon their general conceptions of excellence in the ability in question. For an excellent critical analysis of this form of rating scale, see Rugg (156). The scale of alternatives or two steps. The judgment is given in the form of a Yes or No answer to the question: “Does the man possess this ability?” The difficulty with this simple method is that most abilities are not distributed in this all-or-none way. Abilities more often are found to be distributed among people in general in accordance with the normal probability curve. In other words, most people have neither a very small amount nor a very large amount of an ability. The judge finds himself unable to make satisfac- tory or accurate decisions regarding these people who do not deviate greatly from the average in this ability. Per cent scales. The judge rates each man in terms of per cents much as he would mark an examination paper, on the assumption: that 100% is the maximum amount of the ability that any man can possess. This implies a finer dis- crimination than is possible or desirable. Moreover, two raters can rarely be found who agree on how much 100% of the ability is, or 80%, or 50%. They really use different measuring sticks. Per cent scales sometimes require the judge to estimate the per cent of the general population which in his opinion this man would excel in the ability. Multiple-step scales. The judge indicates his estimate of the amount of the ability each man possesses by checking one of a series of symbols. In the following scales, a plus sign indicates a high degree of the ability in question and a minus sign a deficiency in the ability. The meaning of the other symbols is obvious. 126 RATING SCALES 3 Place a circle around the symbol which indicates this man’s capacity for tactful behavior. +! + 4? -? — =I Where only five steps in such a scale are needed it may take this form: +! + ? - =! In another similar form, ¥ represents yes and N repre- sents no in answer to the question: “Does the man possess this ability?” Does the man possess the ability to conduct himself tactfully? Encircle the symbol which represents your answer. y! Pv 2 NY NI An alternative form is ¥ y ? n N Linear scales. A straight line is drawn to represent the range of ability, one end representing the least amount and the other end the greatest amount. A man is rated by mak- ing a check along this line at the place which corresponds to his standing in the ability. Descriptive scales. A number of phrases are assembled, describing different grades of the ability from one extreme to the other. Here is an example of such a list for rating one aspect of social behavior: 1. Extremely breezy and informal 2. Cordial and congenial 3. Meets one half-way 4. Slightly reserved 5. Constrained and formal The judge assigns to each man the number corresponding to the phrase which describes him best. This type of scale is an attempt to make the steps more definite and concrete than in the usual characterizations of a man’s ability as 1. Very superior 2. Superior 3. Average 127 4. Inferior 5. Very inferior . ° . oqo A judge may find that in a scale of a definite ability sev- . . eral of the descriptive phrases apply to one man. For ex- €¢ 2 : 3 ample, the phrase “meets one half-way” may justly describe FOR RECORDING OPINION ON PROSPECTIVE LIFE INSURANCE AGENT O Prospective AGEN eos srr rseressissamsismsrsremremsssrrsimstssseirrisress ras DAE Of INTEIVIEW errs cerere sores Ihave had.............._interviews with the applicant, the total length of which has been.....................hours. In addition, I have investigated him as fOllOWS fr irre temic aienicin woie o serie Ean [91 Lice Law Indicate your opinion on each quality by making a cross (X) on the line, just where you think it ought to be. For example, if in Suan 1 you think the prospective agent 1s a little lower than indicated by the statement “ appearance satisfactory,” but not quite low enough to be recorded * gives somewhat unfavorable impres- sion,” then put the cross on the line somewhere between these two points How does his appearance and manner impress you, especially his health, Creates Good Appearance Gives somewlial Makes a physical build, neatness, fine im- appear satisfactory unfavorable poor etc. pression ance umpression appearance To what extent does the applicant participate se——————————————— ——— a remem, in the conversation Tends to take Initiates Fairly re- Hardly more during interviews? Sheds Sram Bk. sh life insurance? Hot mformed Sones ra well hs thorough _ +» Do you believe that hie will work bard at Mayloafon Willtendto Will be Will work Will be con- life insurance selling? the job take things satisfactory systematically sistent hard fairly easy worker worker Does he show initiative? Has he ever sterisd gad sarried throng Strikes out Shows some Likes to Waits to be ny 1% ons for himself originality be guided directed © What is the attitude of applicant's wife or Esthusiastic I‘avorable Indifferent Not entire Antagonist immediate family? ly “sold” ‘What are applicant's P——————————————————————————————————————————— financial circumstances? In debt more Shigittly Only slight In moderate Well-fixed = then two months in debt assets circumstances (inancially 3 How do you rate reports Ld from others, such as i recommendations or letters Fe ert PE eevee Toyo te FE To from references? Unfavorable Fomewhat Satisfactory 1 I estimate that the applicant Would Write... my —rmeer recess 2MOURt of business in his first year REMARKS: 0 All things considered 1 2 recommend the candidate for a position with this company. BCDC cnr avrrsrsmmiomesrrmsr brs asenmm mtb Buss rear bre opera rtm (SIENALUTE) cover ram ssrerns som ame astro resin madam Form 7R.1 1005-25 All Rights Keacrved Life nsurance Seis Research faresn Figure 10: Graphic rating scale for insurance salesmen. 128 RATING SCALES Before making any rating, read very carefully the definition of the quality in question. Indicate your rating on each quality by placing a check mark (v') on the line opposite the man's name just where you think it should be. For instance, on Quality I, if you think the person you are rating is not an Expert but ranks higher than Competent, put the check some- where between these two points. As far as possible, rate each man on the basis of what he has actually done during the past year. Notice that most of the headings ask how successful the man has been in doing certain things. Each rating should be made on this basis. In cases where this seems unfair, put a circle around the check mark, and if desired write down a word or two of explanation. But all ratings should be on what the man has actually shown in his work, not on what the rater thinks he will show. nor on what he did two years ago. It is better to omit a rating than to guess. These analyses should aid you to determine more accurately the character and value of those in your supervisory force. Their purpose is to cause you to consider each man from all angles and thus tend to lessen the possibility of losing sight of a man’s real value oa account of minor defects. 11, Consider ability to secure results by argansizing and planning the work of bis department. | Enective Effective Only ol | even under under sorma) moderately Weective difficult circumstances effective I tircums tances - Figure 11: Form for rating several subordinates on a single trait. 120 EMPLOYMENT PSYCHOLOGY ~ ANALYSIS SHEET Nd Name — Depertmeat aren DIS ger 1. Consider knowledge of and ex- [Exum - Tem nel perience in particular branch of NU ao worl; technical skill or special. ed traiaing. 1. Consider ability to secure results fos a i by organizing and planning the Sor Juma Swder tay pastels effective work of his department. §IX. Consider ability to devise mew Ingenious and actives piggreseies | Ee and botter methods or readiness constructive; frequently improves readily adopts im slow to fo sdent improved methods, and DUET NC mmm weve med sw success im putting these mew i. jo; now methods methods into effective opera- z ge epgibp a EERSTE ar part of the whole organization. well Does he willingly and effec- tively cooperate with other de- partments, both staff and pro- duction? V. Consider ouccess in developing Ge Theor men Nee | talent and arowsing ambition | in developing satisfactorily develop wen of those in his department. nen VB. Consider snccess in winning the owes Tete mien tae mee en] es EsTan least operation an maintain dis- } O cipline. ‘ Figure 12: Sheet for summarizing a man’s ratings by several supervisors. a man about 80% of the time and ‘slightly reserved” about 20% of the time, in accordance with his moods. In a scale developed at the University of Iowa, the rater, instead of indicating the phrase which best describes the subject, dis- tributes 10 points among the phrases. In rating the man in the example just cited, the rater would place an 8 before 130 RATING SCALES : “meets one half-way” and a 2 before “slightly reserved” (29). Graphic rating scale. What is now commonly known as the Graphic Rating Scale is a combination of the last two forms (55). The straight line represents the range of the ability, and the descriptive phrases, placed beneath it, de- note certain points in this range. This gives the rater free- dom to make as fine gradations in judgment as he cares to, instead of limiting him to five steps, and at the same time does not deprive him of these phrases as aids in making his judgments definite and concrete. The ratings can later be expressed as numerical values by the use of a calibrated scoring stencil. Figure 10 is an example of a graphic rating scale for in- surance salesmen. Ordinarily a graphic rating scale is printed in such a way that the judge must rate each man on all abilities before passing to the next man. The Dennison Manufacturing Company makes use of a form in which the judge considers one ability at a time, rating all the men on that ability be- fore proceeding to the next one. The judge is thus almost forced to make a man-to-man comparison. This plan com- bines the characteristics of the graphic scale and the order- of-merit method. A portion of that scale is given in Figures IT and 12. The following chapter is devoted to a comparison of the relative merits of different rating methods. It also contains suggestions regarding the construction of graphic rating scales, the use of the scales, and the treatment of the data they yield. IY x I 3 2 RATING SCALES (Concluded) Statistical and non-statistical bases for the comparison of rating methods. How to make a graphic rating scale. Obtaining the ratings. Statistical treatment of .ratings. CompaRIsONs of rating methods have been of two sorts, statistical and non-statistical (55). There are approximately seven statistical methods of comparison. One statistical method involves comparison of ratings and test scores. Rugg used this criterion (among others) in evaluating the Army Rating Scale. He compared ratings on intelligence with intelligence test scores, and found no re- lationship between them (156). These findings, however, do not argue solely against the validity of the scale, but point quite as definitely to the well-recognized limitations of the army mental alertness tests considered as measures of gen- eral intelligence in a broader sense. In the following quotation Hayes and Paterson mention two other methods of comparing rating scales. “The graphic rating method was found to be highly reliable, as shown by the close relationship between ratings on the same men by the same judge for different months, and by a close relationship between ratings on the same men by different judges” (67). Rugg also uses the latter criterion in show- ing the weakness of the Army Rating Scale. Certain precautions are necessary when such criteria are employed. Should a judge’s ratings vary from month to month or remain constant? Under certain conditions, it is to be expected that estimates will change from month to RATING SCALES BES month, and a lack of such change indicates a weakness in the scale. If a judge rates a person the same on successive occasions it may be an indication that he has learned noth- ing new about the subject’s personality in the interval; or that he did find reason to vary his estimate, but wished to avoid detection of his initial misjudgment; or, again, it may mean that the scale afforded him no adequate means of expressing an altered judgment. Agreement between raters, as shown by correlation of their ratings as well as by similarity of means and disper- sion of ratings assigned to the same group of workers, is probably a better criterion of the value of a rating method than agreement between the ratings made by a single judge at different times. If the former agreement exists, it is an indication that the scale calls attention to universally noted characteristics and makes them the basis for the rating, pro- vided the judges have had equal opportunity to judge the subject, and no judges are included toward whom the sub- ject would display an exceptional attitude. The data from which Hayes and Paterson draw their conclusions are rat- ings by foremen on their workmen. No matter how closely foremen agree in rating their subordinates, a different story might be told if the subordinates were rated by their wives and their fellow-workers. Agreement among raters is to be expected only if they bear the same social and industrial relationship to the subjects. Such agreement may be con- sidered a good indication of the relative worth of a rating method. The practical difficulty most often encountered is in finding two or more judges who know the men well enough to rate them accurately. A fourth way of determining the relative merit of a rating method has to do with the normality of the distribution of ratings. This method is based on the assumption that the distribution of accurate measures of any ability in a large sampling of the general population takes the form of the normal probability curve. The fact that ratings on an ¥3z : EMPLOYMENT PSYCHOLOGY ability have the same form of distribution as precise mea- surements of that ability in the same group of people, is insufficient evidence for concluding that ratings and mea- surements are correlated. With a large number of cases a spotty distribution indi- cates that certain portions of the scale are being neglected by raters, or that the steps on the scale are not of equal value. This matter of equalizing the steps on the rating scale involves considerable labor, and is usually omitted ex- cept for very refined work (67). If the scores on the rating scales are transmuted into ranks, nothing is lost by inequali- ties of steps on the rating scale. Spread of distribution is a fifth basis for comparison of rating methods. This is an important factor, since sufficient discrimination between abilities is necessary in order to compute correlation coefficients and to distinguish between one man’s ability and another’s. The greater the spread, however, the greater the gross error of any single rating. Absence of %alo effect is a sixth criterion. Halo effect is the tendency for the judge to be influenced in his ratings on a specific ability by a general attitude toward the person he is rating. If the judge likes the person, he tends to rate him high in all favorable abilities; if he dislikes him, he tends to rate him low in all favorable abilities. Such poor discrimi- nation on the part of judges results in high correlations be- tween ratings on unrelated abilities. Thorndike reports correlations of 4.58 between intelligence and leadership, + .51 between intelligence and physique, and --.64 between intelligence and character, when one judge rated 137 avia- tion cadets on the Army Scale (189). He also reports an average correlation of --.67 between general ability for officer work and so highly specialized an ability as flying ability, when the same men were rated by eight judges. The average rater will not rid himself of this bias and exer- cise his analytical powers unless the scale itself aids him to do so; consequently the absence of halo may be con- 134 RATING SCALES 175 sidered a point in favor of a rating scale. With several judges rating the same men, the scale which shows the highest correlation between obviously unrelated abilities may be considered the weakest. A seventh criterion, mentioned by Plant (140), is as fol- lows: Present a person with a list of his acquaintances, and with a set of ratings on one of the men in the list. Ask him to indicate which of the men is the one to whom the ratings apply. That scale is best which yields the most easily iden- tified ratings. Non-statistical points to be kept in mind when choosing a rating scale are simplicity, brevity, convenience, clearness of directions, time required for completing the rating, agree- ableness of the rating task, universality of the scale, and ease of scoring. The graphic rating method is at present the most popu- lar. By the use of this method judges agree closely among themselves in rating the same men, and judgments are con- sistent when repeated after a long time interval. It is simple and easy to grasp, easily filled out and scored, frees the rater from direct quantitative terms, and makes the degree of the ability concrete. Finally, it has been found to be more interesting to use, as well as convenient and reliable. HOW TO MAKE A GRAPHIC RATING SCALE The tendency in the evolution of graphic rating scales has been toward the adoption of the practices recommended be- low. The superiority of these practices has not in all cases been demonstrated experimentally, and much research is still needed in order to develop the most reliable as well as the most convenient forms of scales. It will help the rater if each scale is prefaced by a de- scription of the ability in question. It is important that this description be in terms of be- havior rather than of mental qualities. The more con- TRE EMPLOYMENT PSYCHOLOGY 1 A ) 10 ! “il or _ Vety goodwmatured. Agreeable Rather gjum and Grouchy and unpleasant Very ill-naturea ‘Has, winning manner unresponsive and uncivil, ~~ Figure 13: Distribution of self-judgments of 100 college students in answer to the question “Are you good-natured or ill-natured?” (55). The base line of the figure corresponds with the line of the graphic rating scale. At first glance it would appear-that the men had overrated themselves, but the reason for the skewed distribution is more likely to be found in the fact that the central phrase does not represent the average of the ability. The two phrases to the right are so extreme that their use is avoided altogether. cretely the ability is expressed the greater is the expectation that the various raters will be judging the same thing. Be sure that the ability is not a composite of several abilities that vary independently. Decide definitely upon the extremes of the ability which will probably occur among the persons to be rated. The end phrases should not be so extreme in meaning that their use is completely avoided by raters. An example of a scale which errs in this regard is shown in Figure 13. Place in the center of the scale the phrase describing the neutral or average or typical degree of the ability. Figure 13 shows also what happens when this is not done. Half of this scale is useless. No fine discrimination in ability is made. Figure 14 shows a better distribution of ratings, due to a better selection and placement of phrases. Usually not more than five nor less than three descriptive phrases are used to mark the steps on the scale. 136 RATING SCALES FL en —1_ SE Te stor TI Se ai Figure 14: Distribution of self-judgments of roo college students in answer to the question “What are your tendencies with regard to unburdening your inmost feelings to others?” (55). This is a fairly normal distribution of ratings, and implies that the phrases were well chosen and spaced. If there are five phrases, make the intermediate ones closer in meaning to the central one than to the extremes but maintain an equal amount of linear space between phrases. This has the effect of spreading the distribution. The same end may be accomplished by making the intervals on the scale larger between the central phrase and the intermediate ones than between the intermediate ones and the extremes. The phrases used should have the same meaning to all raters. Slang is occasionally effective, but there must be no doubt about its meaning. Special terms and phrases in use in the occupation may be freely substituted for more general terms. Avoid such terms as average, very, extremely, excellent, good, fair, or poor. It is better to use adjectives which in themselves express varying degrees of the ability. Thus, in place of extremely neat one may say fastidious, or in place of very careless in dress one may say slovenly. Make the descriptive phrases short, concise, and apt. Have the phrases set in small type with plenty of white space between them. Have the two extreme phrases set flush with the ends of the line. Allow no breaks or divisions in the rating line. Where the line is divided vertically into five or ten parts, accuracy of rating is sacrificed for convenience in scoring. 137 :- EMPLOYMENT PSYCHOLOGY The line should not be much more than five inches in length, otherwise it cannot be easily grasped as a whole. The favorable extremes of a series of scales should be placed alternately to the right and to the left. Or, better yet, the scales may be arranged in an apparently haphazard way so far as the position of the favorable extreme is con- cerned, but in half the instances it should be at the left end of the line. This arrangement breaks a motor tendency to check at one side of the page. It helps to reduce the halo effect, resulting in more discriminating judgments. OBTAINING THE RATINGS Regardless of the type of rating scale which is used, ex- perience dictates certain precautions. Ratings on each person should be obtained from at least three competent judges who are thoroughly acquainted with his abilities. This is not always possible; but the cautious investigator will frankly recognize that anything short of this minimum gives him a measure with only very meagre reliability. To get the most representative measure of each ability in each person the ratings of the judges should be averaged, or if two out of three of them agree, the judgment of the ma- jority should be accepted. The raters should be carefully trained in the use of the scale. The investigator should go over it with them in de- tail to make sure that they understand its operation and the pitfalls they are to avoid. Most raters are at first unaccustomed to think of men analytically. They tend to think in terms of general im- pressions, prejudices, or indefinite descriptive phrases. Some information will have to be given them about the statistics of distribution of abilities and the quantitative relationship of differences in abilities. The raters should be warned to avoid being influenced in i3& RATING SCALES N their judgments on any one ability by a general g¢. «© ..i- ©. tude they may have formed toward the person wk = they —~ : are rating (189). This tendency is lessened Ae SS consider all the persons to be rated with regard t raefipst | os ability, then the second ability, and so on. Eig Another influence to be guarded against is closeness of friendship with the person rated. Shen showed that judges rate their friends higher than others (169). According to Knight, supervisors tend to rate old em- ployees high as compared with new employees (96). He gives three reasons for this: (1) The supervisor will not ad- mit that being under his direction has brought no improve- ment in the employee. (2) The supervisor unconsciously identifies himself with older employees who are perhaps more like him than new employees. (3) The supervisor has be- come used to the older employees and overlooks their weak- nesses. Watson (211) lists also the following characteristics of ratings: Poor employees are better observed and more reli- ably rated than are good ones; “general-all-round value” is often more reliably rated than is a more specific trait; people who are good judges of themselves tend to be good judges of other people; and ratings of which the judge is very sure have very much higher reliability than do his ordinary judg- ments. The scale should be placed in the hands of the raters and should be discussed with them several weeks before the rat- ings are called for, to allow them time to observe the sub- jects with reference to the abilities to be rated. Needless to say, the behavior of the subjects will be more typical and characteristic if they do not know during this time that they are being rated. The following are the most important points which should be brought to the attention of the rater, regardless of the type of scale in use: 1. If you have any question about the operation of the 140 EMPLOYMENT PSYCHOLOGY scale, be sure to have it answered before you make your ratings. 2. Be certain that you understand what is meant by each listed ability. 3. Do not begin to rate until you have observed every employee from the point of view of the abilities on the scale. 4. Do not begin to rate the employees until you have considered all of them with regard to the abilities on the scale. 5. Consider one ability at a time and rate all the persons on that ability before proceeding to the next ability. 6. Try to free your rating on any one ability from the influence of the other abilities of the person, or of any general impression or attitude in regard to him. 7. Remember that extremes of ability are rare. Measure- ments of abilities have generally shown that most people are grouped about the average and that fewer persons have the higher or lower degrees of ability. Do not, therefore, con- sider all the persons to be either very high or very low in the ability. 8. When judging a person, call to mind concrete instances of the type of behavior in question. 9. Behavior on the job or when making business contacts is more significant for present purposes than behavior in purely social gatherings. ro. Try to be uninfluenced by the time you have worked with the person or the closeness of your friendship with him. TREATMENT OF RATINGS If possible, ratings should be obtained a second time after an interval of several weeks and averaged with the original set of ratings. Some raters consistently judge higher than others, so that in computing averages the figures for these judges are over- weighted. To correct for this error, the following statistical treatment of ratings is recommended. All ratings should be first transmuted into terms of position in the group rated by the particular judge. In other words, a distribution table is made for all ratings on a certain ability by a certain judge. RATING SCALES : An arbitrary value of 5 is given to all ratings in the upper 10%, 4 to the next 20%, 3 to the next 40%, 2 to the next 20%, and 1 to the lowest 10%. In this way each judgment is expressed by comparison with the total, and the tendency of any judge to underrate or overrate the group as a whole is counteracted. In obtaining average or total ratings these values are substituted for the original ratings. Other and more refined methods are available, such as expressing each rating as a deviation from the average rating given by the judge, with the standard deviation of that judge’s ratings as a unit. These statistical refinements should be avoided when the groups are small, and should be used only when the same group of men is rated by several judges. When considering numerical ratings it is easy to lose sight of the fact that they represent only relative standings unless the zero point of the scale has been accurately deter- mined. That A receives a rating of 25 and B a rating of 50 does not indicate that B is twice as able as A unless the scale has been so constructed that a score of o represents just not any of the ability, and the units of the scale are equal to each other. In an experiment in selection it is not necessary to make an accurate calculation of this zero point, since relative position is our only objective. When deter- mining the validity of the scale, a score is interpreted in terms of the probable success of a person making the score. It may be found that the above ratings indicated that B’s chances of success are twice as great as A’s. This matter will be discussed in Chapters XV and XVI. To evaluate the analysis of abilities incorporated into a rating scale, the ratings on these abilities may be compared statistically with the criterion of success, in the same man- ner as test scores. The statistical procedure involved is de- scribed in Chapters XIII and XIV. If the ratings on an ability for which a test cannot be constructed correlate closely with the criterion of success, the rating scale may be used as a part of the selection I47 EMPLOYMENT PSYCHOLOGY process. It supplements the tests and other measures of ability. This is a common procedure in the selection of candidates for training as salesmen. If the subjects are old employees, ratings are usually obtained from foremen and supervisors. In the employment office, however, ratings must be made by the interviewer or examiner, by the applicant’s references, or even by the ap- plicant himself. Even though a series of ratings by foremen may prove to be valid, the investigator is left in doubt as to the validity of ratings on these same abilities when made by the interviewer on the basis of a brief talk with the appli- cant. It is essential, therefore, to validate ratings by inter- viewer and applicant, as well as by supervisors or fellow workmen. The experimental rating scale may cover a good many items. A scale installed for permanent use in the employ- ment office should be kept within practical length. It should include only the reliable and valid items. Ratings should not be included on abilities or characteris- tics which can be objectively measured by tests or personnel records. Do not ask a foreman to rate his men on atten- dance or tardiness when the facts are matters of record. The rating scale which the investigator develops is seen to serve as a substitute gage of abilities for which no ade- quate objective tests are available. Besides its uses in em- ployment, it will help in deciding questions of transfer and promotion, in discovering need for special training, in mea- suring progress in this training, and in passing upon recom- mendations at periods of salary readjustment. This measuring implement, the rating scale, is generally recognized to be at best a crude tool. Its reliability is low. Its scope is limited. Nevertheless, it is distinctly better than no instrument at all, and this in itself is a justification for the wide use it has found in industry. Tt now remains to consider the third form of examination —the questionnaire. I42 QUESTIONNAIRES: THE PERSONAL HISTORY RECORD AND THE INTEREST ANALYSIS Definition of a questionnaire. Uses of a questionnaire in an investigation of this sort. Characteristics of a good questionnaire. Obtaining the data. IN THE preceding chapters we have considered in some detail two types of aids to the scientific selection of person- nel: one which may be used for measuring abilities, the psychological test; and another which may be used to record systematic estimates of abilities, the rating scale. It remains to consider a third type of device, the questionnaire, which has also an important place in a study in scientific vocational selection. There are basic functional distinctions between the ques- tionnaire and the other two types of examination. A test is a sampling of performance. It answers the ques- tion: What can the worker do under certain circumstances, as, for example, what problems can he solve, what informa- tion can he impart? A rating scale is a systematization of estimates. It ans- wers the question: What do others think of the worker, or what does he think of himself, or, more precisely, what do they say they think of him? A questionnaire is a systematization of inquiry concern- ing facts as they exist or have existed. It answers the ques- tion: What facts does the worker give about himself, or what facts do others give about him? Typical kinds of questionnaires for this purpose include such forms as the application blank and the interest report blank. The end result of the use of a test is a measure of per- XI 143 141 EMPLOYMENT PSYCHOLOGY formance; of a rating scale, an expression of opinion; and of a questionnaire, a statement of fact. In an investigation of the sort we are describing, the main purpose of the questionnaire is to gather information for validation regarding the individual’s personal and family history, his health and habits, and those elements of his social environment, interests, tastes, motives, and person- ality, for the measurement of which tests or rating scales are not adaptable. Much of this material will be available to the investigator on the regular application blanks of the company, but a thorough study usually demands that infor- mation be obtained in greater detail, and with a broader selection of questions. The questionnaires used in research will differ only in length from the ones later used in actual selection of em- ployees. They must be made to include all possibly sig- nificant items, or a sufficiently large number of items. It is too much to expect that more than a minor fraction of these items will show a significant correlation with the criterion of success. The number of items should, therefore, be large enough to allow for this shrinkage and still leave an abun- dance of valid items to include in the employment question- naires developed as a result of this investigation. The same point holds with reference to the number of sources from which information is sought. The research questionnaires may be devised for filling out by the appli- cant or employee himself, by the interviewer, by the em- ployee’s references, by fellow-workers or supervisors, by the medical examiner, and by an investigator who obtains the data from company records or other sources. When the material has been validated and the question- naires condensed to include only significant questions, the information can be obtained for employment purposes from more limited sources: From the applicant himself at the interview, from the applicant’s references, and from the Lh QUESTIONNAIRES 145 medical examiner. (Rating scales have a companion use here.) CHARACTERISTICS OF A GOOD QUESTIONNAIRE An ideal questionnaire gets the special information the investigator wants, and nothing else. It varies in form not only according to the kind of information sought, but also with the source from which the information is to be obtained. Its questions are worded concisely and without ambiguity. It is constructed in a form which tends to motivate the person in whose hands it is placed to give complete and accurate answers. It is convenient to fill out and to score. The questionnaire should be preceded by printed instruc- tions and a statement to the effect that the replies will be treated as confidential. It is advisable also to print instructions to complete the questionnaire rapidly. This not only saves time but over- comes the tendency to study answers and give the best one. The questionnaire should be concise. There are many ways of condensing questions to make them less burdensome and repugnant, as well as more convenient to answer. This not only reassures the subject, but saves printing costs. The following questionnaire for the analysis of interests is a good example of economy of space. The question is not repeated for each item. What was your attitude toward the following subjects when in school? Draw a circle around L if you liked the subject, and around D if vou disliked it. Mathematics LD Music LD Manual training LD Chemistry L 2 Physiology I © Ancient Shop work Iv Mechanical drawing I. 0 languages LD Modern languages L ©) English composition L Spelling LD Penmanship L (+ History L > Civics LD Economics L i{* Commercialsubjects LT Public speaking LD Sociology LD Physics LC Writing should be reduced to a minimum, as in the ex- ample just cited. This not only conserves the time of the worker, but avoids the necessity later for deciphering poor : EMPLOYMENT PSYCHOLOGY . . . handwriting. Where a question can be answered by Yes . . or No, the letters ¥ and NV may be printed after it and the . . . . answer indicated by encircling one of these two letters. If . * . . . only a definite number of replies is possible, these replies . . . may be listed with parentheses after each one, and instruc- rr ee . OCCUPATIONAL INTERESTS (MEN) O ETT — | |,_———— Orgamzation (School COIEZE, OF COMPANY ) momen eee meee This information will be kept confidential. Make your judgments without consulting any one: After each of the occupations there are five symbols Draw a circle around one of these symbols after each occupation as follows Draw a circle around L! if you would like very much doing that kind of work. Draw a circle around L if you would like doing that kind of work Draw a circle around ? if you have no decided feelings toward that kind of work, or if you know nothing about it. Draw a circle around D if you would dislike doing that kind of work. Draw a circle around D! if you would dislike very much doing that kind of work Pay no attention to salary or social differences between the occupations, or to possible family objections; consider only, your own miterest and satisfaction in doing each of the kinds of wark listed. Assume that you have the ability necessary for each of the occupations. Be sure to draw a circle around one of the symbols after each of the occupations. Do not skip any. Actor LL PD DY Mechanical engineer Li'L 7? D'DY Architect LLL? .D DY Missionary LL, Dy DY Arnyy _ificer LVL 2D IH Musician LiL? DD! Artis LY LL 20D) Newspaper reporter LiL. 2 D DY Astronomer EY OL 2D DY Novelist LiL 42a D oD; Athletic director EVOL RD) Office clerk LISL 22D DI Auctioneer LY UL 25D DY Office manager Ly L 2D 0 Auto salesman LtL 2D. D' Orchestra conductor LAGE (3 Ds Auto racer LUGE AD Pharmacist LANL VRS D Sy Auto repairman LAL DL Photographer LAWL SR eD Aviator LUT AP api Physician LOL D Bank teller LYST rea), Poet LSL 2 CD Baseball player fick 2 py af Politician SL ER aD i Bookkeeper tL PED at Printer SP Dye Carpenter NLP AD Private secretary LY LL 2D Cashier LOL D Railway conductor LISLI? 4D Certified public accountant L! L ? D Rancher LYL S2RD Chef LY LD, Real estate salesman Li, PD Chemist LAL ED Retail merchant LAL 2D Civil service employee Li 'L2"D Wf Retail salesman LL PED Clergyman 11 La DR Scientific research worker 1! L ? D Consul LAL 20D Secret service man L1=L 2 1D Dentist AE, SR DY Ship officer LL. Ray Draftsman LY Shop foreman LY LED Editor bY LR DY Social worker LIL 2D Explorer SVL PD Specialty salesman FEL RD) at Factory manager VLD Statistician EAL BE? CID NE Factory worker 2b AL BOTY Steeplejack LIL PD , Farmer VL. uD Stock broker LiL 2D LA Fisherman AL ED) Street-car coniductor ELL POD IEE Foreign correspondent CARLY ID Street-car motorman LY LZoD 1 Forest ranger | LU MES Surgeon LY L 20D ! Hotelkeeper or manager LAS} DL Teacher, LiL 2? Dit Interpreter LY aL ee py rt ‘Toolmaker LULL BaD ue EAE Ra Db Traveling salesman rl LD ht bor arbitrater LUE A/3 RD Typist LISLE 2 oD DY Landscape gardener LiiL, 2 D 5% Watchmaker LY L= 2D SD} 1 2 . OD Toye 5 5 5 2 Di (Add any other occupations here) Lighthouse tender 1! kL yt mri meinem BY pS TD DH Locomotive engineer I! ! Ve E LL ory =m erm ims LIL 7 DDI Magazine writer LY L 2D "DY Sree Ll. 1 2 DD) Copyright, 1933, by C. H. Stoelting Company. Chicago, linoly ! Carre ars EE —— TS ——— ©. aT Figure 15: Form for analysis of work interests. 46 QUESTIONNAIRES EXPERIENCE RECORD D ( Confidential ) Do ET ar Uh Se. Susdering » Shuuge of cud er sure you are taking a step which will lead to your success. If any question does not apply to you wnte “No” or “None” et it ER ON co premrneStA evr pse eemme BW TTV sasesban mop szsares nen ana +192. Last First Middle : 8 Date of Birth. . cenannrorsnssaansites anurans winenns; Se WHEE DOM vas) Sviava as) ansvendossd oa 6. Lineage or birth (American, English, Scotch, Irish, French, Hebrew, etc) ............ rire Nl same adine ? Height. «uvurvee fbn isinians im Weight «. Ib % Na af dependents. Children ..Adules..... 9 Single O Married 0 ‘Widowed 0 Divorced 00 or Separated (O. ia Schooling: : Name of Schaal tek fast Year Grammar School ) High or. Preparatory School College or Technical School! Special Study = Correspondence Courses : 1 Business College or Trade School "Night School 11. Give a full account of how your time has been spent, whether employed or mot, since leaving school. List separately all different positions you have held. For example, if you have been a clerk and a salesman for the same com pany, list each one separately. TTY | Nature of your work | Place Whete eiployed of |Company by whom employed ~1 Figure 16a: A model application blank, page 1. tions given to place a check in the parentheses following the correct answer. For example: Marital status: Single ( ) , married ( \/ ), divorced ( ), separated ( ), widower ( ). After a list of answers like this, a blank space may be left I47 Ye EMPLOYMENT PSYCHOLOGY £2. Give your salary and reasons for leaving your last three positions. JOS, rem Monthly Salary Reasons for Leavin, ‘Last position ik —— et imate] ta ———————————————— Preceding position 2D —————t sets I ——————————————— Preceding position 13. How much time (years and months) have you spent in inside of retail selling? . o.oneerioerinnnenceniananens ++ How much time in Life Insurance selihgZ. .. ov. couunrinuiiasitiinionananeiiiineonessaosssretmuasiansiss . . How much time in other outside SellinZ?. . ...vupeperearunsnrseereesurmnsseceiiionasssosssasssnesasdanios 14. | How long have you been thinking of making the selling of Life Insurance your life work?......ooormecnarnnnnns re What 16 YOR 70 CORSIHET ILE. so vu avs dss sinss us vusiahagnsimesnsssnresinrivnsusiainnsnssosnssrovsennsiunresy , How many times have you talked with a representative of ¢his campany about it?........cooieeiiiierninnrs £5. What is the attitude of your immediate famaby?. . o.oo .uqeenonnrar conyaunecriionniiebiaiisiigeiiipgrniny 16. Occupations of relatives (if not living, state what ceeupation was) 1 P Fathereceneeninninnaninns naive inne BOO cts . In what city or town do you wish to represent this COMPanYY. .....uoviarniiinens ines corenrenneininees 18. Check the forms of investment you now have: Real Bstate. (3, Mortgages (0, Savings Bank 03, Bonds (J, Preferred Stock (J, Common Stock (J, Life Insurance, amount carried $. ... , Accident ot {Health Insurance $........ 19. Are you indebted to any one?. ............. . AMOURE. LL. Lieiiiiee eeeiees ~ 20. Name the social clubs, fraternities, and business organizations to which you belong (Church may be included, if How many times have you been elected to office in those 0rgamizations?........ooueeueirirnren innrnienssees 21. Estimate how many times during the past year you have taken part in each of the following: TE. saan shins Bolo ans aslai sss Handball. viens +o vwsd SWIMMINB cons av st nivsaaied Hunting or Rifle Practice. ...........; Driving an Automobile .. ... ...: Riding a Motoreycle..........8 Boating. ...............; Horseback Riding...... .... . 3 Other Athletic Sports. ......c.ovurernosnsn 22. Estimate how many times during the past year you have ateended the theatre?.......... Attended social affairs . Of JOUT OWR SEX... ess uisstennsnananesinsnssnsnrsnrynanavsasssanorsasnseensna-usniassscsrsasyree 23. Have you any physical inrMIEY2. . «ue uermeuenrniauurioanninnninnsuasaantanaaanaicaa ona deng anges tee . How much time have you lost through illness within past two years?..... .... ...Natureofillness.............. 24. Give the names and addresses of three business men who know vou well and to whom this Company may refer. a err coergsei as en scons et doit Name Address 1. oe ———— AA + et ee ee ee i er err me— Te TEE ee OY Signature of Applicant. Jee. + Please look over your blank and-make sure that you have given some answer to cvery question: Figure 16b: A model application blank, page 2. for very unusual answers. To list all these unusual types is a waste of space. When preferences are investigated, sym- bols may stand for different attitudes. These symbols may be listed, ready for checking, after each item toward which the subject is requested to express his attitude. The voca- 18 QUESTIONNAIRES tional interest analysis given in Figure 15 (57, 58) illus- trates this point. The data of the questionnaire are easier to tabulate if the questionnaire is printed so that answers are in a column near the right-hand margin of the page. It is impossible to insure against falsification of answers, but the probability of obtaining the truth is increased by arranging the questions so that the most profitable direction of falsification is difficult to guess. The vocational interest questionnaire given above illustrates this point. This ques- tionnaire ordinarily reveals whether the applicant’s voca- tional interests are scientific, literary, social, solitary, out- door, indoor, professional, or commercial. If the vocations were grouped in this way instead of being in alphabetical or apparently haphazard order, the applicant would quickly suspect the purpose of the blank and would probably cen- sor his answers. The purpose of the questionnaire is best achieved by its present arrangement which prevents this purpose from being obvious. Any single questionnaire should be brief. If a great amount of information is wanted, the questions should be arranged in a series of questionnaires each covering a specific type of information. The worker may be given one questionnaire at a time in order not to overwhelm him by the mass of information required. OBTAINING THE DATA If the subjects of the investigation are applicants the ex- perimental questionnaires may be filled out by the applicant himself or by the interviewer for him. The former arrange- ment saves time, but unless the interviewer is accessible the applicant will not be able to have matters explained which puzzle him. Except in dealing with the most intelligent employees it is safest for the interviewer to check over the questionnaire with the applicant after it has been filled out. 140 EMPLOYMENT PSYCHOLOGY to make certain that all questions were understood and noth- ing omitted. Experimental questionnaires and rating scales may be sent to the references given by the applicant. Sending out questionnaires by mail, although a troublesome process for both investigator and recipient, is the only method avail- able for obtaining some sorts of information. The literary A. B.C. LIFE INSURANCE COMPANY WASHINGTON, D. C.; lsrchil, 19— John Doe end Company, 260 Fifth Avenue, New York, N. Y. Attention of Mr. T. M. Pitt. Gentlemen: Mr. Wallace H. Smith of 582 Newark Ave., Jergey City, is applying to us for a selling position and refers us to you. To assist him in getting properly placed will you kindly answer the questions on the attached sheet? An addressed en- velope is enclosed for vour reply. You reelize that it will be to Mr. Smith's disadvantage to be appointed to this position if he is not fitted to succeed in it. In order that he may have every consideration we trust you will give the questions your promet ond thougntful attention. > “Y ifr ee 4 Poe ! Richard Roe, / Superintendent of Agencies Figure 17: Letter te references. 150 Encl. QUESTIONNAIRES Please return to Your answer to each question may be indicated by a cross (X) placed in the appropriate brackets. I. Mr....: EE trae ee asuseseayrsx States that he was BO VOUT CIMPIOY G8... vss iaivivisii ns aa xt as 5 ses ws vinnie wuld Er a te tesiaiy sn a DOB EOE oss uiinnslss 1OB ils Does this correspond to your record? Yes ( ) No ¢) 2. He states that he 1eft DECAUSE. .. ov .vreeerrrennres os, lL II— Is this an adequate statement? Yes ( ) No £:). 3. He stated that he received (in salary or commissions). ...........ousenn.... Is this correct? Yes ( ) No ( ). If not correct, how much did he receive?. . . . > o's 4. Were you personally acquainted with him and his work? Yes $Y Nol.) 5. Was he alwavs honest in financial dealings? Yes ') No ). Very Very Good Good Average Poor Poor 6. How do you rate hisindustryd. Ll sali) £1) {) €:) £2) 7. How do you rate his personal habits and condtct?. eu Y $Y iY {. ) ) 8. How do you rate his fitness for a selling position with our organization?..........( ) ) £5) $i) :') 9. Would you be willing to reemploy him? Ves £."') Nol). Signature........ ~ FTTETIR We shall appreciate any additional information that may assist us to judge the applicant’s fitness for the position. The reverse page is left blank for your statement of such information. Figure 18: Report form to accompany letter to references. ISI oo EMPLOYMENT PSYCHOLOGY ingenuity of the more successful direct-mail copy-writers is needed to insure a maximum proportion of replies. Ques- tionnaires to be sent by mail should be exceptionally brief and should be accompanied by a tactful letter explaining the nature of the inquiry and the service which can be done the applicant by giving correct and conscientious answers. The same letter may enclose a rating scale. An example of such a letter and rating scale is given in Figures 17 and 18. It pays to follow up an inquiry. Toops (203a¢) made a careful study of questionnaires sent out to graduates of a trade school, to determine the success of the instruction. By the use of five follow-up letters in addition to the one originally sent with the questionnaire, he succeeded in ob- taining completed questionnaires from 93% of the graduates whom he was able to reach by mail. The cost per reply from the first letter was $2.77; the final cost per reply of all returns was 68 cents. Among the most important points he makes are the following: There was no great difference between the data from those who replied at once and from those who required a large number of follow-up letters to arouse them to reply; in other words, those who reply readily are not a biased sampling. The shorter the ques- tionnaire the better are the chances of getting replies. The cost per return will be decreased by continuing to send follow-up letters until practically all the questionnaires are returned. When the curve of cumulative returns from a mail canvass becomes almost horizontal, it is time for an- other follow-up letter. The week-end holiday is the time when recipients are most likely to fill out questionnaires, and mail should reach them toward the end of the week. The letters which Toops used, printed in full in his article, will suggest ways of wording such letters in order to insure suf- ficient replies. The telephone is better than the mail in checking the worker’s statements and in obtaining information from his references. Data may also be obtained by company in- 552 QUESTIONNAIRES , vestigators who go to the sources for their facts. Such in- formation is apt to be reliable but expensive. It is indis- pensible, however, in hiring some types of employees, such as Pullman porters, where integrity and dependability are most essential. If the subjects of the investigation are employees, these same sources of information may be used. When the questionnaire is distributed to subordinates by an executive who requires it to be filled out, he must explain its purpose in a way which will insure a willingness to co- operate helpfully. The best way is for the investigator himself, wherever possible, to present the questionnaire in person, insuring an understanding of the instructions and a willingness to give with the utmost frankness the informa- tion desired. Questionnaires regarding employees may be devised for completion by supervisors or fellow-workmen. Information about employees may also be obtained from company records and tabulated on specially constructed questionnaires. Much questionnaire information is best secured, not through asking the worker or supervisor to fill out a blank, but through the expensive but more reliable method of the personal interview. People who neglect or refuse to fill in a printed form or answer a letter are often glad to give the desired facts to some one who comes to them in person. Charters (30) has developed the techniques involved in this procedure. They include a careful preliminary training of the interviewers, as well as a standardization of the ques- tions to be asked. It must not be forgotten that a scientific method of selec- tion bases itself on information whose predictive value has been statistically determined. This information about the applicant is obtained by tests, rating scales, and question- naires at the time of employment. The personal interview also has as its main purpose to I53 EMPLOYMENT PSYCHOLOGY gather significant information which can be used in pre- dicting the worker’s probable success or failure in a job. (Other important functions of the interview are discussed in Chapter XIX.) Rating scales and questionnaires serve to focus the interviewer’s attention on essential data and to provide the means of recording these data. Ideally, no facts or figures derived from the interview should be used in deter- mining whether or not the applicant should be accepted until after the importance of these facts has been well estab- lished. These remarks apply as well to facts obtained from the applicant’s references. When questionnaires covering all types of information desired by the investigator have been prepared and data obtained from the group of workers being studied, a statis- tical analysis of the answers to each question must be made in order to determine which of the facts covered are signifi- cant enough in this case to be useful in the selection of applicants. But before plunging into the mathematical procedures required in validating the questionnaires and other examinations, we shall devote a chapter to questions connected with their administration. I54 yn TEST ADMINISTRATION Preliminary details in test administration. The examiner's attitude. The testing room. Procedure in testing. NEw tests must themselves be tested. To evaluate the examinations he has devised, the investigator uses them to obtain scores made by workers of different known levels of vocational accomplishment. He then examines the results to determine which instruments most successfully differen- tiate one level of accomplishment from another. This chap- ter treats of the measuring of these selected workers as a step toward the validation of the examinations. The administration of rating scales and questionnaires has already been discussed in part in the chapters which deal with their development. Tests require far more care in their administration. The conditions need not be as rigidly standardized as those of a laboratory experiment, but tact and courtesy are more essential than in the labora- tory. The investigator should ordinarily leave to the executive most immediately concerned with the success of the in- vestigation the matter of arranging the time of testing and insuring the presence of the employees. If he is testing for ability in some minor occupation, he should arrange with the foreman or supervisor the details of time of testing, and this executive will notify the men when to appear. The investigator would do well to think how he himself would feel in the man’s place. He would be wondering what this was all about and how it was going to benefit him. Will a low performance in the test count against him with his supervisor? Will a high score be used to make him La ISS I EMPLOYMENT PSYCHOLOGY speed up? Such questionings must be put to rest by a brief but clear explanation of the reason for the testing. Some investigators have adopted the additional precaution of an- nouncing that each man’s score will remain confidential — told by the examiner to no one but the man himself unless he asks to have it made known to the foreman or others. Such a policy must, of course, not be announced unless the investigator and the management are able and willing to live up to it scrupulously. Whatever the details of the ar- rangements, they must be such as to deserve and to secure the confidence and hearty cooperation of the workers being examined. The examiner should be assured, but not abrupt, pre- sumptuous, or tactless. A single error of strict straightfor- wardness or of tact may cost an examiner the cooperation of a whole group of workers. A young examiner discovered that an employee’s slowness in learning a new task was due to color-blindness which the supervisor had not suspected. This discovery was a kindness to all concerned; but both the worker and his fellow-employees took umbrage at the examiner’s manner, the zest with which he had all uncon- sciously humiliated the learner, or in other words, “showed him up.” Needless irritations of this sort may upset morale in a production department, as well as interfere with the smooth progress of the investigation. They are readily avoided by a wise examiner who takes the worker’s point of view, and governs himself accordingly. The purpose of the investigation should be told to the men, usually by the investigator. Sometimes the executive or foreman in immediate charge of the workers is more than skeptical—he is antagonistic to the whole notion of objec- tive tests. The investigator, under these circumstances, has to use skill in presentation, for otherwise he may cause antagonism among the men toward the tests. If the execu- tive in charge is favorably disposed, it is well for him to explain briefly to his men the purpose of the investigation. g6 TEST ADMINISTRATION ) No matter whether the employees are being paid an hourly or weekly rate or are on a piece-work or commission basis, the investigator must make certain that the proper arrangements for compensation are made, so that no one will lose in earnings while away from his work to be exam- ined. No employee should feel under compulsion to take the tests, and if no amount of persuasion is sufficient to convince a recalcitrant worker, he should be excused. When the others have been tested, he may change his mind. In the West Lynn Works of the General Electric Company, John- son O'Connor examined in 1924 more than 1,000 employees who came individually and asked for the chance to try themselves on the “job-samples” which he was undertaking to standardize. He made the examination a privilege rather than a chore. The value of the privilege became enhanced in the minds of the employees, as he was able, from time to time, to discover indications of special talent and make recommendations for transfer, promotion, or special training. When men holding responsible positions are to be tested, greater difficulties are sometimes encountered. Executives consider their time too valuable to be spent in experiment- ing, and they can be more independent in their attitude toward what they regard as side-issues. These men may be reached most readily for group testing at conventions or other times of assembly. A high executive who has himself taken or is about to take the tests should lead off and show the way. TEST CONDITIONS Elaborate conditions may be set down to which the test- ing room should conform, but in most industrial and com- mercial organizations the investigator will have to be satis- fied with what he can get. Link (101) describes an inge- nious portable laboratory which he was able to set up near where the men were working. This arrangement, used in 157 1 EMPLOYMENT PSYCHOLOGY German as well as American investigations, has disadvan- tages, but it is necessary if tests are given to shop operatives who for one reason or another cannot be taken far from their work. Below are some of the conditions which a room for indi- vidual testing should satisfy: Privacy should be assured. No one other than the ex- aminer and the subject should occupy the room at the time of testing. Freedom from interruption should be provided for. Rooms used for test purposes should not contain important material which must be referred to frequently. If the telephone proves annoying, the examiner may muffle it. The room should be relatively free from noise. It should be well lighted. It should not be too far from the place of work. It should be cleared of objects likely to distract attention. As far as possible the room should reproduce the condi- tions under which the man works, since unusual conditions are apt to provoke unusual responses. Suitable writing or working space should be provided for both examiner and subject. If the test calls for writing, the pencil, provided by the examiner, should be sharpened at both ends, in order to pre- vent the subject from wasting time during the test by eras- ing, or by attempting to sharpen a broken point. Use of a cumulative stop-watch which has a sliding lever near the stem is recommended, as it is noiseless in operation and allows the investigator to take time out for unforeseen interruptions or difficulties. The usual type of stop-watch springs back to zero after being stopped, and cannot be used for recording cumulative times. Decimal stop-watches and split-second stop-watches have their advantages for certain kinds of measurement. A small screen on the table is sometimes convenient so that the examiner may keep his tests out of sight, or record “£8 TEST ADMINISTRATION = scores without distracting the subject. Formidable-looking recording devices and other distracting apparatus should be in an adjacent room. A set of shelves, files, or cabinets for test forms and appa- ratus should be provided. For group testing, any room of suitable size may be taken over temporarily. Persons who are near-sighted or slightly deaf should be assigned seats near the examiner. Writing space should be provided so that the subjects will not be crowded; and care should be taken to minimize cheating. Desks, tables, or tablet armchairs are needed. PROCEDURE The following suggestions refer specifically to procedure in individual testing. The examiner must first put the subject at ease. If his introductory talk is entirely standardized, it is likely to be artificial and stilted; therefore, it should be flexible and adaptable to any attitude on the part of those examined. Old employees in particular will require skilful persuasion. In- formality and courtesy are essential, but should not be car- ried to familiarity. When testing either one person or a small group the examiner does well to introduce himself. If testing one person he should learn the man’s name and use it when speaking to him. In his introductory talk the examiner should give a frank and brief account of his purpose in giving the tests, unless this has already been taken care of in a general announce- ment to all the subjects. He should convey by his bearing and speech his confidence that every one will wish to cooper- ate by taking the tests. His introductory talk should not be so long as to put the subject on edge or make him nervous. As soon as the subject takes his eyes off the examiner’s and looks around for the tests, it is time to begin the examina- tion. No one should be tested who is in physical discomfort or ISU EMPLOYMENT PSYCHOLOGY suffering from any temporary disability. The examiner will have to differentiate between mere excuses and valid reasons for not taking the tests at the time. The examiner should know thoroughly the directions for the various tests so that he can give them quickly and accu- rately. Above all, the examiner should preserve his composure during the tests. He should do nothing to distract or irritate the subject. A rather easy test or a few simple questions asked in a casual manner may be used as ‘‘shock-absorbers.” A special test may be devised for this purpose or the easiest test may be given first. If the recommendations in Chapter VIII have been followed, each test is preceded by a standard fore-exercise which serves the double purpose of shock- absorber and of means to make certain that each subject understands just what he is to do. The stop-watch should be kept out of sight, and no one should see the scores as they are recorded. If a person shows the effects of hard work on the tests, the program may be interrupted at the conclusion of any test by a few minutes of conversation. The examiner may motivate the subject by commenting favorably on his test performance, but this cannot be done generally without arousing suspicion. If the subject is doing poorly, nothing should be said. He should not be prompted. If the subject shows disappointment at not being able to complete the test during the time allowed, the examiner should reassure him by explaining either that the test is so made that even the most rapid workers are not expected to finish it, or that relative standing is the important factor. While the subject is engaged in completing a test, the ex- aminer may put in the time to advantage by scoring the pre- ceding test, or by making ratings on types of behavior during the test which may later prove to be significant indicators of vocational success. Just before the subject leaves, a warning should be given 160 TEST ADMINISTRATION not to tell others about the tests, because when their turn comes to be tested they will then come already prepared and will be able to score higher than he did. The examiner will, however, be under no illusions as to the effectiveness of these injunctions. He must make provision through alternative forms or otherwise, for control of this complicating factor of fore-knowledge. In general these remarks hold good for group testing also. The examiner will be on the alert to prevent cheating, but should not exercise such conspicuous supervision as to cause nervousness. He should make the usual announcement, warning all against allowing their neighbors to see their papers, lest they lower their own relative standing. No talking is permitted during a test, but there may be a brief period of relaxation between tests. Not more than an hour at a time should be given to test- ing, unless the subjects cannot be brought together again. Workers often request the examiner to give them their scores or to tell them their weak points. A policy in this regard should be adopted and adhered to. It is safest to tell the score to no one but the person examined, and then only if data enough are at hand to enable the examiner to state the score in terms of relative rank, such as “in the top quar- ter” or “in the lower half.” If it is impossible or inadvisable to give out any test scores, one answer to the request is that the whole investigation is as yet in the experimental stage, and the individual’s score is not significant until standards of comparison have been secured. The ground for such an answer should have been prepared in the introductory talk. When the data are gathered, not by examining employees already on the job but by giving the examination to new employees at time of hiring, many of the inconveniences and difficulties here listed are not encountered. On this acount some industries have preferred to use such a procedure in spite of the fact that several years may be required for the accumulation of a sufficient number of cases and suffi- ciently reliable criteria of actual success on the job. 161 EMPLOYMENT PSYCHOLOGY The Scovill Manufacturing Company of Waterbury, Con- necticut, adopted such a policy in standardizing and evaluat- ing its general mental alertness test for factory operatives, many of whom cannot read or write English. The test is given to all new employees at time of hiring. In January, 1926, this investigation had been under way for more than two years, and sufficient data had been accumulated to yield significant critical scores in scarcely a fifth of the depart- ments being studied. In only three of these were the test standards as yet being actually used in employment of new workers, namely, apprentice toolmakers, apprentice electri- cians, and operatives in one department. The results of the actual use of the test in these groups were sufficiently clear- cut, and the management already felt well repaid for the in- vestment made in developing and evaluating the tests. Not only were those employees who were selected on the new basis proving to be superior; the firm was getting a larger number and a higher type of applicants for the jobs where it was becoming known that selection rested on the appli- cant’s performance rather than on the personal estimate or favor of a foreman. Six months later, the investigation had progressed to the stage where sufficient data had been se- cured in more than 60 occupations, and standards for selec- tion and placement in these occupations had been deter- mined for use of the employment department. Such are the objectives which a farsighted examiner keeps before him. We may now assume that the investigator has gathered all his data. He has chosen a group of men who represent vari- ous degrees of success in the vocation in question. He has obtained a variety of measurements and estimates of their abilities by specially constructed tests, questionnaires, and rating scales. The following chapter deals with the next step, the comparison of these measurements with the criter- ion of vocational success for the purpose of determining which measuring instruments differentiate sufficiently be- tween degrees of vocational accomplishment to justify their use in employment. 162 XIII VALIDATION OF THE MEASURING IN STRUMENTS Tabulation. Meaning of reliability. Measures of reliability. Correction for attenuation. Meaning of validity. Measures of validity. Group compari- sons. Differentiation of groups by test scores. Differentiation of groups by questionnaire items. THE psychological measurements and records of the sub- jects which the investigator has gathered by means of his tests, questionnaires, and rating scales mean nothing in themselves. The investigator must determine their validity, that is, the relationship between these measurements and the criterion of vocational accomplishment. Once he has found the extent of this relationship, he can show how far such measurements can be of service in selecting men for the vocation. This and the following chapter deal with the statistical methods of comparing measurements of abilities with the criterion of success. They treat of the ways of determining quantitatively how much relationship there is between these variables. Discovery and measurement of relationship between dif- ferent sets of facts is basic in all science. In the biological sciences it is commonly called the establishment of correla- tion. Ordinary experience and common sense reveal many relationships of dependence between one set of facts and another; but it is only when these facts have been reliably measured and a statistical index of relationship between the two series of measures computed, that knowledge of these relationships can be called scientific. Without such scien- tific knowledge of the interdependence of predictive mea- surement of ability and of actual competence on the job, any recommendations regarding the introduction or the abandon- 162 EMPLOYMENT PSYCHOLOGY ment of specific employment tests or other indexes of prob- able success cannot command the respect of the thoughtful business executive who has learned to appreciate science not only in engineering and finance but also in the realm of human adjustment. The procedure of determining the re- lationship between measurements of ability and the criterion of vocational success requires an understanding of what is meant by coefficients of correlation between two variables, by reliability of the measures on which these correlations are computed, and by validity of the test measurement as judged by its correlation with the criterion. The reader must turn to treatises on statistical method mentioned below for a com- prehensive discussion of theory. Here he will find conve- niently summarized the successive statistical steps to be taken in ascertaining the significant relationships between test score and criterion. The methods of determining the predictive value of definite scores will be reserved for later chapters. There are conditions, to be determined for each investiga- tion, under which the methods and devices for test valida- tion may be omitted entirely and the investigator may proceed at once to the practical methods described in Chap- ters XV and XVI for determining the predictive value of each score. If, for example, distribution curves of test scores have been drawn for vocational successes and for vocational failures, critical scores or critical sections (pre- ferred ranges) may be set off even though the significance of the difference between the two groups in test performance has not been determined. It will be assumed in this discussion of statistical devices that the reader is familiar with one or more of the standard text-books on statistics, such as Rugg (157), Thurstone (196), Otis (125), Yule (233), West (219), Brown and Thomson (21), Jones (83), Thorndike (191), Kelley (86), and Davenport (42); and that he knows the meaning and significance of the most common statistical concepts. The main purpose of these chapters is to outline all the statistical 164 VALIDATION OF MEASURING INSTRUMENTS 16 5 steps that may be required in treatment of data, reminding the investigator of essential procedures while referring him to the authorities for more detailed directions, as well as for complete expositions of the statistical theory underlying the formulas recommended. Tabulation. Several points are to be kept in mind in plan- ning the tabulation of the data which have been gathered. The tabulation should put the figures in shape for rapid and convenient statistical treatment and for ease in locating any individual’s record. Rugg describes various ways in which this may be done (157, ch. iii). If there are as many as 20 measures on 1,000 subjects, it will be advisable to resort to a mechanical tabulating device such as the Powers or the Hollerith machine. In tabulating, the investigator should not fail to make use of additional scores which the test may yield, since each new set of scores derived from a different aspect of a test is the equivalent of another test. An illustration of this is found in a study by Manson (106), in which it was discovered that in discriminating between two groups of students the score in an intelligence test was less important than the number of items skipped, a measure which had hitherto been neglected. If the investigator is working with applicants rather than with old employees, it will be necessary to lay the tabulated data aside until reliable figures on the vocational success of the men are at hand. It is sometimes advisable to place in the hands of an executive a duplicate set of the scores. They should be in a sealed envelope so that they will not affect his judgments before the time arrives for a comparison of these data with the records of accomplishment. Reliability. Before proceeding to the presentation of methods of determining the validity of the measuring instru- ments, it is necessary to give some attention to an important step, the determination of the reliability of the measures. The validity of a test is its relationship with the criterion, that is, the proportion of commonality of abilities measured EMPLOYMENT PSYCHOLOGY by criterion and test. A mathematical expression of the validity of a test answers the question, “To what degree does the test measure abilities which are required for success in the occupation?” The reliability of a test is the accuracy of the measure- ments it yields. It answers the questions “How well does this test predict itself?”, or “How accurately does it mea- sure the abilities it is supposed to measure?” This accuracy can be determined either (1) by repeating the test and corre- lating the two series of measurements, or (2) by correlating records obtained with two similar forms of the test (or the odd items with the even, or the first half with the second half). Either coefficient of correlation is called the coefi- cient of reliability. The first is the upper limit and the sec- ond the lower limit of the true reliability coefficient (86, p. 203). Formulas for use in computing these correlations are found on pages 174 ff. Factors which make for a low reliability coefficient of a test are: 1. Variability in the ability tested; that is, susceptibility to embarrassment, fatigue, and other temporary influences, and increase of the ability through training. 2. Errors in measurement due to faults in the test or in its administration. 3. Restriction in the range of ability measured. If a test has high reliability, the inference may be drawn that the abilities it measures are not subject to fluctuations or errors of measurement. Low reliabilities, however, must be due to one or more of the above factors. The first factor is diminished when only a few minutes are allowed between the first and second forms, or when the odd items are correlated with the even items of one form. It is important, however, to know what fluctuations actually exist in the ability in question, for if the ability is subject to ups and downs as shown in low self-correlation of the test, then it is difficult to measure that ability for purposes of vocational 166 VALIDATION OF MEASURING INSTRUMENTS 167 selection. It will then be necessary in determining the validity of the test to repeat the test and use an average of the two scores. The remedy for errors due to the second factor consists in making the necessary improvements in the test and the pro- cedure used in administering it. The reliability coefficient of a test is raised by increasing the range of ability measured. The relationship between range of test scores and coefficient of reliability is given in the following formula (86, p. 222): ¢ VI=R (1) re ED e— 2 Vi—r In this formula ¢ is the standard deviation of the more re- stricted group and 7 the reliability coefficient obtained with this group, = the standard deviation of the other group and R the reliability coefficient obtained with this group. If a test has an observed reliability coefficient of . 50 when used with a group having a standard deviation of 1, it should yield a reliability coefficient of .87 with a group which has twice as large a standard deviation. The reliability of a test may also be raised by increasing its length. The relationship between the length of a test and its reliability is given by Spearman’s prediction formula, often erroneously called Brown’s formula (86, p. 205): Nrxx, (2) Vipx, = 1+ (—1Dran, Here r.., is the reliability coefficient of a test x, of given length, = the number by which the length of the test is multiplied, and r.,., the reliability of the lengthened test xy, The assumption is made that all of the # parts have the same reliability. Doubling the length of a test having a reliability of .50 increases its reliability to .67. The effect which a change in the length of a test will have upon the correlation of the test with the criterion is shown in the formula at the top of the following page (86, p. 200): EMPLOYMENT PSYCHOLOGY (3) Tey = —_— V = Test ery mn In this formula y represents the criterion. The effect of doubling the length of a test having a relia- bility of .50 and correlating .40 with the criterion is to in- crease its validity to .46. If the test is increased 20 times in length, the validity is raised to .76. The last two formulas may be applied to the use of an average or total score in # tests similar to x, provided these tests are equally weighted. They apply also to » repetitions of the same test, provided the time interval between mea- surements is long enough for memory of the preceding re- sponse to disappear. It is important to know the reliability not only of test scores, ratings, and other measurements, but also of criteria. High correlations between measures and criteria need not be expected unless their reliabilities are high. Foremen’s ratings usually have low reliability in this sense, and the investigator can scarcely expect consistently high correla- tions of test score with such a criterion. If the reliabilities of both test and criterion have been computed, the correlation between one set of test scores and criterion records may be corrected for attenuation due to errors of measurement. If 7.., is the reliability of the test and 7,,, the reliability of the criterion, then the correlation between test and criterion (rx) cannot be greater than A/7rzs + Ty, Drovided similar groups are measured in a sim- ilar way. The size of 7., must be judged by comparison with \/7zm1 + 73, and not with 1.00 as a maximum. The formula for correction is then (4) 7x (corrected for attenuation) = Te The coefficient obtained by the use of this formula is the correlation which may be expected between perfect mea- sures of x and y (125, p. 228). The value of 7 obtained by 168 VALIDATION OF MEASURING INSTRUMEN TS 169 this formula should not be used in place of the uncorrected value in predicting one variable from another. Better correction for attenuation is obtained by correlat- ing both sets of test scores and criterion records. The formula then becomes a Mri Still more refined correction is possible if each set of test scores Is correlated with each of the sets of criterion records (233, p. 213): (£) Toy w= Vr Tog, Tra - Pro ® Try V Tez, * Ty, Validity. Each test or other measuring instrument must be evaluated by making a statistical comparison between the scores of the subjects and their vocational accomplishment. The concomitance of these two variables can be determined accurately and stated mathematically. From this observed relationship may then be inferred the predictive value of the tests under similar future conditions. In the practical situa- tion the test score is always the known variable, and the vocational success the predicted variable. The reason for the relationship between the two variables is a matter of interesting speculation. Unless special circum- stances or weaknesses in the experiment allow extraneous forces to account for the relationship, it may be assumed with reasonable assurance that the relation between the two variables is caused by some common ability or group of abil- ities involved in both performances. Statistical devices for determining the relationship be- tween test and criterion are of two sorts. One of these de- vices, used when the criterion or both the criterion and the test scores are discrete variables, tells how much the differ- ent classes of vocational success, represented by their central tendencies, differ in test performance. Such a device gives an expression of the significance of the difference found be- ) EMPLOYMENT PSYCHOLOGY tween the groups. The second type of device gives a single numerical index of the concomitance of the two variables— a coefficient of correlation. It is usually employed when both test scores and criterion are continuous variables, although some correlation formulas may be used when either or both are discrete variables. GROUP COMPARISONS If the criterion of success is a discrete variable, enabling the investigator to divide the subjects into two or three groups—the successes, the undistinguished, and the failures —the method of group comparisons is indicated. This meth- od discloses how well the measuring instrument under inves- tigation differentiates these groups. For a measuring instru- ment to have prognostic value, the successes and the failures must be well differentiated. Differentiation of groups by test scores may be studied by drawing for each of the groups a distribution curve of scores in the test. The distribution curves for the success group and the failure group should be drawn on the same chart and with the same reference points. The differentiating capacity of the test may be observed by noting the over- lapping of the two distributions. To obtain an accurate index of this differentiation, we must first compute the means and standard deviations of the distributions of test scores for the vocational successes and the vocational failures. The standard deviations of the means are obtained by the formula: (=\ Hi; Td ND i= [EE or N N where # is the deviation of each measure from the mean, d any deviation from the mean and f the frequency of that deviation, 5, the standard deviation of the distribution, and 170 NT VALIDATION OF MEASURING INSTRUMENTS 171 N the number of cases. The standard deviation or standard error of the difference in means is obtained by the following formula: (8) €="V Om, tox Here om and go, are the standard deviations of the two means. These formulas are not reliable where is less than 2 5. The differentiating value of the test is determined by com- paring the difference in means of the successes and failures with the standard error of this difference. If the difference is twice as great as its error, the chances are 43 in 44 that the difference is real and in the direction indicated; if it is three times as great, the chances are 740 in 741. This state- ment of probability is derived from the normal surface of error. If, for instance, the difference in means is three times as great as its standard error, then on one side of the normal surface of error at a distance from the mean equal to three times the standard deviation, the difference will become zero and beyond that point the values will be reversed; that is, the mean which was formerly the greater will now be the lesser. Since in the normal surface of error 1/ 741 of the measures occur beyond 3 at one side of the curve, we conclude that the chances are 1 in 741 that the difference is not real and not in the direction indicated, or 740 in 741 that it is real and in the direction indicated. A good example of the use of this method is to be found in Fernald, Hayes, and Dawley (51) and Dewey, Child, and Ruml (44) (see Table 8). For practical purposes the investigator may consider a test to be significant if the difference in means of the suc- cesses and failures is two or more times as great as its stand- ard error. A different formula is used when a group is compared with a sub-group; for example, when good, fair, and poor workers combined are compared as a whole with one of these three sub-groups. The formula for the standard deviation of the difference I EMPLOYMENT PSYGHOLOGY between the mean of a sub-group and the mean of the total of which it is a part is given by Pearson (Biom., Vol. V, p. 182) as 22 gt 2n n(M—m)? 4 NS Hol Sy where = is the standard deviation of the total and NV the number of cases in the total, ¢ the standard deviation of the sub-group and # the number of cases in the sub-group, M the mean of the total and » the mean of the sub-group (51, p- 114). Differentiation of groups by questionnaire items may be computed in a similar way. A comparison is made of the proportions of each of the two groups (vocational successes and failures) giving certain answers to certain questions. The formula for the standard error of the difference in pro- portions is as follows: (10) 6; = p7 ot 2 where p, is the proportion of one group indicating the an- swer, g, the proportion of the same group not indicating the answer, p. the proportion of the second group indicating the particular answer, gq, the proportion of the second group not indicating the answer, and #z, and 7. the numbers of indi- viduals in each of the two groups. A difference in propor- tions may be considered significant if it is two or more times as great as its standard error. A brief example of the use of this formula in measuring the significance of group differ- ences in answering a questionnaire item, is given below. The method is more fully explained in reference 57. EXAMPLE: Question: Can you do good work while people are looking on? Total Number Proportion Proportion Number Ans. Yes Ans. Yes Not Ans. Yes Successful salesmen 40 (nn) 32 .80 (p) .20(q,) Unsuccessful salesmen 46 (n,) 23 .50 (p) .50(q.) 92 VALIDATION OF MEASURING INSTRUMENTS 173 2g: _ .0049 2s _ .0054 n, n, €: = .0040+.0054 =.0094 €:=.10 Difference in proportions =.80—.50=.30 Since the difference in proportions is 3 times as great as its standard error, it is significant. The formula for comparing the total group with a sub- group by this method is given by Yule (233, p. 271): 2: _ Po Le bh = rtm mm These symbols have the same meaning as in the previous formula, the subscript o referring to the total, r to the sub- group, and 2 to the total minus the sub-group zr. This formula finds use where more than two groups are com- pared. An example is given below. EXAMPLE: Question: Can you do good work while people are looking on? Total Number Proportion Proportion Number Ans. Yes Ans. Yes Not Ans. Yes Successful salesmen 25(n,) 18 72(p1) Average salesmen 50 (ze) 25 Unsuccessful salesmen 40‘ 17 Total 115 60 .52(p,) .48(q,) :s 52-48 9 _ € = “Tis 25 .0078 €nx = .09 Difference in proportions =.72 —.52 =.20 Since the difference in porportions is more than twice as great as its standard error, it is significant. The validity of individual items in any test may be deter- mined in the same manner as the validity of questionnaire items. X1v VALIDATION OF THE MEASURING INSTRUMENTS (Concluded) Correlation: Product-moment formula; method of rank differences; four- fold table method; Pearson biserial 7; correlation ratio. Coefficient of mean square contingency. Coefficient of association. Probable errors of mea- sures of correlation. Interpretation of the size of the correlation coefficient. THE coefficient of correlation (r) is a convenient numer- ical index of the degree of interdependence of two variables. It ranges from 1.00 to —1.00. The closer it approaches unity, the greater the evidence that one variable varies as the other (directly, if the sign is positive, and inversely, if the sign is negative). The familiar product-moment formula of Karl Pearson is the basic one for computing the correlation coefficient: Zxy (12) 7= ST Here x and y represent measurements of an individual in the x series and y series under comparison, each measurement being expressed as a deviation from the mean of its series. N is the number of persons measured, and o, and oc, are the standard deviations of the two series of measurements. A clear and detailed description of the method of comput- ing » may be found in Rugg (157) or Thurstone (196). Forms for use in calculating the coefficient of correlation have been devised by Thurstone (195), Otis (125), Toops (201), Holzinger (74), and others. These forms present the process in outline and have blank spaces for inserting the fig- ures for the particular problem; they serve, therefore, as indispensable time and labor savers and as checks on the in- clusion of all the necessary steps. The first three listed are LR 174 VALIDATION OF MEASURING INSTRUMENTS 175 the most popular forms. The Thurstone form is obtainable from C. H. Stoelting Company, Chicago, Illinois; the Otis, from the World Book Company, Yonkers, New York; and the Toops, from H. A. Toops, Ohio State University, Colum- bus, Ohio. These authors use modified formulas for ease of calculation. The Thurstone Correlation Data Sheet is re- produced in Figures 19 and 20. Explanation of the Thurstone Correlation Data Sheet (Figures 19 and 20). 1. The scatter diagram is plotted, always placing low scores in variable x at the left of the diagram and high scores at the right, and low scores in variable y at the bottom and high scores at the top. 2. Estimate the class interval in which the mean of the x's oc- curs and shade in the appropriate column. Do the same for the ¥’s, shading in the appropriate row. Correction for these esti- mates will be made later. 3. Record in row L the frequencies of the measures in the col umns. 4. Record in row M the deviations from the assumed mean in units of class intervals. 5. Perform the algebraic processes called for in rows N and P. 6. Complete columns 8, 9, 10, and r1. These processes are analogous to those in rows L, M, N ,and P. 7. The figures in columns 12 and 13 are obtained by summing the values of each measure in the row multiplied by its deviation from the assumed mean (as shown in row C ). For example, in row D, oX1=0, 1 X3=3, 2 X2=4, 3 X2=6, and 4 X I == 4, the total being 17, as shown in column 1 3. The fig- ures in row C are minus if they are to the left of the assumed mean. Consequently, two operations are necessary for each row, one for the measures to the right of the mean and one for the measures to the left of the mean. In row E we have —1 Xr ==—TI, recorded in column 12, and § X 1 = 5, recorded in col- umn 13. 8. Add algebraically the values in columns 12 and 1 3 for each row and record in columns 14 and 1 5. For example, —1 and -+ 5 =} 4, which is recorded in column 14. 9. Multiply the values in columns 14 and 15 by the correspond- Ing values in column g. Remember that the values in column 9 I EMPLOYMENT PSYCHOLOGY which are below the assumed mean are prefaced by a negative sign. Record in columns 16 and 17. 10. ZX,,s is the sum of the numbers to the right of the assumed mean in row NV. 11. ZX, is the sum of the numbers to the left of the assumed mean in row NV. 12. Add the above two figures algebraically and record after Xx. 13. Add the numbers in row P on both sides of the assumed mean and record this after 2x2. 14. Perform the operations for the y variable analogous to those in 10, 11, 12, and 13, above, and record in the appropriate places on the form. 15. Values of Zxy,,, and Zxy,., are obtained by adding col- umns 17 and 16 respectively. Zxy is the algebraic sum of these two values. 16. N is the number of cases and is obtained from either col- umn 8 or row L. 17. The remaining operations are self-explanatory. Hull (78) and Dodd (45) have constructed calculating machines to aid in computing correlation coefficients. With practice it becomes possible to estimate 7 roughly from the scatter diagram, which is a graphic presentation of the measurements of all persons measured in each of the two variables whose relationship is being examined. In a scatter diagram the test scores are usually plotted as abscis- sae and the criterion as ordinates. The product-moment formula may take the form: ly (13) v= Siz. ye With the use of this form it is not necessary to make a scat- ter diagram (157, p. 274). Correlation may also be computed without scatter dia- gram if gross measures or raw scores are used instead of deviations from the mean (86, p. 164): >XY-(CX)(ZY)/N (19) r= SGN VV (2V)N =X:—(2X)?/N- VI? —(2Y)*/N IT ry pe as . Woden. | | fd Lal 4 ! A Fly fy fy? Zx | 7x |Zxy 220s ZYpos ZX pas x | 2Xme= mer Uhegr - - + l= += + | 2X Er tice, EEA Aged s a 2 : ad |= 4% at : 2 x2 2 Ze L g=2X dk = ; j= hos } on wi 5 oo eo - fe = i _ i i Zx ’ I L et al. {1 o Co 1 = | - — voy et + + of Ee } — . X-varable sn _ KELATION DATA SHEET X ' Y. variable LL. Thurstore 7 X | ! | Swbye—ts . Be, So tation Fx = ! HIE ETTS 3 _ Pittsburgh, Fa wl a RE ET a Samia mrZiamninlomini brand 4 Ld i I Computer Late N Farm 137.3. Figure 19 re i J 4 J DO AD] [DIT IESE AE AT Bes i resent — pai eC lial aE LT . _ ' 61 lsz|63 6445166 ]57 poo 7ol7/ 72 73)7e 2 |, lo Ze 2+ - 0s 7 VZheer 790 | ZAehuee 2789] oe fe VAT ; post = re (8 es loys rae , 2 / 5 | 4 i « ‘2 | / ol 02 | 3 AAG = oh - ik ZX 78 . $3 Ixy 278 PAE TTT oC Tn TT Ixsvq20l | Zy e 7877 [N: 5p | | | Byrn als Naf E21 a 1 Ly 750 Y= N = | | deme go | Homoy gf cos - r z z F 2 7,2 2 lel ED EEA Ver [10 |e 900 17 tm] 1707 ¢ 18; WE a . _ _4zo/ rel G 7377 — OOS else clr fall, lz] Tel Tah 750 ; | HE 5 5 60-.0/ = EE] : 9.835- 00:43 im Tem Te bred AR EE MEME pT SE CTE EI Te [mls] Joo] [m=] Je ty, Er xls [ols 7 [ao] Toul Jj re Lhd a | Fade glee [| Jem %% | ld | Efe a] Jo] Ja Te (ernen lif - a era 7] 2 36% 3.04 ls] gi: wg 2 [salma mle] Tur| |e a 1a] ss] 9o [ssl] [sa] Jo0 rs Vos | welel T1771 711 BE EET ozo . wl [mlnlmlalel | 1s Foy lig (Blalabler = Te Fe 7) 1221127 > [rR Hh oo 3 228 44, mm ~— Hylse i wo | yma] 70s - 12) 4 al lea we Commer. - os I | 2 . i 2 lo: * iy : 3 = | wl, pr rl en re ir fh th ih! : 3 : J : ” i als = Femetielia i i eaiiyg a . - E | Mi lS oo r ! : + wy dim fy a 3 J ! To lo Tri ps Tr £ fi 75 4 | X-variable Newht in snehia C$ EELATION DATA SHEET ik i : } I, n rT Ll. > For “iv Tse lle lel le ole] rere Jilie prods mn, 7 Co 4 ba [ssa veloc] |roglizufa 72 [4s 24 | Subyects Fadorde <2 YI. Sornagie Testis of Jechrology [5 [debbie] Tabloid | Caner He fie Tome fen — = wp oh en Figure 20 VALIDATION OF MEASURING INSTRUMENTS 177 An alternative formula is ZXY-NM.M, (15) T NEIX—NIL ZV NI where Af. is the mean of the X’s and M, the mean of the Y’s. The X’s and Y’s are gross measurements of each indi- vidual in the two variables being compared. There are several graphic methods of computing the cor- relation coefficient. One of these is Galton’s graphic method, described by Rugg (157, p. 245), which requires the com- putation of the median and the quartiles, or the means and the standard deviations. A scatter diagram is drawn in which the interquartile distance, or the sigma distance, is the same for both variables. Correlation is measured by the closeness with which the line connecting the intersec- tion points of the quartiles or standard deviations (and necessarily passing through the median or mean of the scatter diagram) is approximated by either regression line (the line of the means of the arrays). Another graphic method is given by Yule (233, p. 203): The means of rows and columns are plotted on a diagram, and lines fitted to the points by eye, say by shifting about a stretched black thread until it seems to run as near as may be to all the points. If b,, b, be the slopes of these two lines to the vertical and the horizontal respectively, (16) r=vbb, Other short statistical methods of estimating correlation are given by Yule (233, p. 204) and Brown and Thomson (21, p. 129). Sturges (177) gives a formula for finding » for a whole group if 7 for sections of the group is known. The product-moment formula comes into use when the number of cases is about 30 or more and when the measures are in their original form and not expressed as rank order. It takes into account the absolute value and position of every measure in the series. There are several other methods of measuring correlation I. EMPLOYMENT PSYCHOLOGY which are used when the number of cases is small, when the divisions are alternative rather than continuous, or when the measures are in terms of ranks. The method of rank difference takes into account only the positions of the measures in the series, and should be used when the measures are expressed as ranks. There are two formulas: Spearman’s method of rank differences: (17) ptm N(N2—1) where D represents any difference in the rank of a person in the two series of measurements, and IV represents the num- ber of paired measurements; and his “footrule for correla- tion”: 6Zg (18) R=1 TR where g represents only differences in ranks where there is a gain in the second series over the first. Both p and R may be converted into 7 by use of tables provided in most text- books on statistics, based on formulas derived by Pearson (see Tables 5 and 6). The conversion of p into r has doubt- ful value since the difference will usually be much less than the probable error of the correlation coefficient. For ex- ample, a p value of .48 has an r value of .50, but the prob- able error of a correlation coefficient of .50 with 30 cases is .09. Since the error of the coefficient is so much greater than the correction to obtain 7, it would have been just as well to leave it as p. The Scott Company (164) published con- venient tables to shorten the labor of calculating p. These tables are reproduced in the Appendix as Tables 3 and 4. Instructions for their use are given below. INSTRUCTIONS FOR CALCULATING RANK DIFFERENCE COEFFICI- ENTS (ForMuULA 17) WITH THE USE OF TABLES 3, 4, AND 5. (SEE REFERENCE 164). 1. Arrange each of the two series of measurements to be corre- lated in rank order. Every pair of measurements is thus repre- 78 VALIDATION OF MEASURING INSTRUMENTS 179 sented by a pair of ranks—the relative positions assigned those measurements in the two series. 2. Obtain the difference between the two ranks assigned each case. 3. Square each of these differences. The squares are read from Table 3. Note that the decimals may be omitted from the squares without appreciable error. 4. Add these squares of the differences. This sum (TD?) is to be found in the body of Table 4. 5. Refer in Table 4 to the proper column for the number of cases used (NV). Thus, if 21 cases were ranked, use the second column on page 253. Run down this column until you find the entry nearest the obtained ZD2. 6. Read the entry in this same line at either side of the page (first or last column). This entry is p. The desired coefficient of correlation (7) may be obtained from Table s. Rank Rank Differ- Example: 21 cases are arranged in rank Sones’ || Sores Sod order for two variables; the sum of the I 11 squares of the differences in the correspond- » 26 ing ranks is found to be 531 (five-tenths ‘ A is disregarded). : fr Refer to the second column on page 253, k run down to the entry which most nearly equals 531, that is, 524, and read beside it the value of p, .66. The corresponding 2 value of 7 (.68) is found in Table 5. I 3 Note: (a) If the obtained =D? is nearly L I i midway between two entries, use the p fall- t. 1 ing between the p's corresponding to the ‘ 5: two entries. (If the =D? in the example 2s were 540, p would be .65.) I 21Y, l (b) For ZD? greater than those corre- I ‘ sponding to p=0 (i. e. in case of negative [ I 9 correlation) subtract the obtained =D? 1 10 5a from the entry in that column correspond- [ 21 ing to p= —1.00. Find the p correspond- 2 1:5 5 ing to this number obtained by subtraction. z . Si With a negative sign prefixed, this is the desired ». ELC = 3531.5 The fourfold table method (tetrachoric correlation) as- sumes that both variables are continuous but takes account only of position above or below the measure of central ten- dency in each series. The fourfold table corresponds to the scatter diagram divided into four compartments by the EMPLOYMENT PSYCHOLOGY means or medians of the two distributions, but recognition is given only to the compartments into which the measures fall and not to their absolute value. The fourfold table method is simple but is at best only a rough preliminary indi- cation of the amount of correlation. Here again we have two formulas. The first is Pearson’s cos 7 method, (19) = aN T Vad ++/bc where 7 equals 180°, and a, b, ¢, and d are the numbers of cases in each of the four quadrants of the fourfold table. a is the number of cases above the central tendency (average or median) in both series. d is the number of cases below the central tendency in both series. b is the number of cases above in the first series and below in the second series. ¢ is the number of cases below in the first series and above in the second series. The second formula for tetrachoric correlation is Shep- pard’s method of unlike signs: (20) 7z=cos (Ur) U is the proportion of paired measurements having unlike signs. A pair of measurements has unlike signs if one of the pair is above the central tendency in its series and the other is below the central tendency in its series. This formula is to be used only when the division lines correspond very closely to the medians of the two series. An example of the use of each of these two formulas is given in Figure 21. Table 7 simplifies the computation of 7. These two formulas are to be avoided when the correla- tion is low and the number of cases is small. They cannot be used when the measure of central tendency in one of the series falls within a large class interval, making it impossible to divide the group into two approximately equal parts. For 180 VALIDATION OF MEASURING INSTRUMENTS 181 Below mean Above mean in Test L in Test L Above ++ mean in job pro- + | ficiency rating a pp Below mean in job pro- — ficiency rating \/bc r =C0S—— ——1 Vad +/be Vad+Vbc V21X24+V13X12 r =.43 (from Table 7) r=cos (Un) bpp Us Tr ys = 30 7=.43 (from Table 7) Figure 21: Illustrating the computation of 7 by the fourfold table method. diagnostic purposes the method of critical scores is prefer- able. In case the two series in a fourfold table are discrete vari- ables and dichotomous, the formula to be used is foo ad —be "hy = ——————— CT —— (a+b) (c+d)(a+c)(b+a) he 2 n 13 21 24 12 21) EMPLOYMENT PSYCHOLOGY where a, b, ¢, and d are the numbers of measures in each of the four quadrants. This formula is given by Kelley (86, p. 259) in other symbols as the product-moment correlation between two point distributions. Formula 21 may be used when there are two answers to a test or question, and when the criterion of vocational accom- plishment divides the subjects into only two groups: suc- cesses and failures. Pearson biserial r (86, p. 245) may be obtained when one variate is both quantitative and continuous, while the other, though actually quantitative, admits of only two subdi- visions, or, more technically, is dichotomous. Biserial 7 comes into use when the test scores are continuous, while the criterion is in terms of success and failure, or those leaving the firm and those remaining, and so forth. When for the purpose of making group comparisons two distribution curves are drawn representing scores of each of these dichot- omies in a certain test, biserial » may be computed as an additional index of the validity of the test. The formula for biserial 7 is as follows: (22) ye (M,— My) Pq a2 If the continuous series is represented by x and the dichoto- mous by 3, then in the above formula 3, is the mean of the x scores made by those in the first y category and p the pro- portion of cases in this category, J, the mean of the x scores made by those in the second y category and gq the proportion of cases in this category, ¢ the standard deviation of the total x distribution, and z the height of the ordinate at the point of truncation of the normal distribution, cutting off p proportion of cases. Some values of z are given in Table 1. An illustration of the computation of biserial 7 is given in Figure 22, Ruml (159) shows an application of this formula to the process of setting critical scores and to the determination of a degree of vocational competence which will correlate most 182 VALIDATION OF MEASURING INSTRUMENTS 183 2g % of : Ez I oS L 58 BS ar. je 80) i >" 82 | 1-2 | 2 ¢ o 35 i a =1 31 4,20} C2 2r 38 | 156.46 | a 227.04 Mx=5.27 2 fd? 227.0 «37 | 30 | a | o N 57 1.84 M. JM, z=.3958 (from Table I, opposite p = .45) 6.27 | +03 | ss (M,—M)p 2 03 a bh =" | (2.24).2475 _ 55 | 45 1.84 - .3958 7 Figure 22: Illustrating the computation of biserial r. . i : EMPLOYMENT PSYCHOLOGY highly with the tests. It is not necessary for the tests to predict exactly what a man’s standing in the measure of vocational accomplishment will be; it is only necessary to know whether the tests predict accurately whether he will be a success or a failure. If the measure of vocational suc- cess is continuous, it remains to find where the division line shall be placed for success and failure in order to obtain the highest correlation with the tests. By calculating biserial 7 with the numbers in the alternative categories varying (that is, by calling the successes the upper 10%, 20%, or 30% in the criterion of success), the division point may be found which yields the highest correlation with the test, and the criterion of success may then be fixed at that point. On the other hand, the alternative categories may be set with regard to test scores, the criterion being continuous, and the opti- mum critical score obtained by finding the dividing line the use of which yields the highest correlation with the criterion. The results will be in terms of “per cents” and will have to be transmuted into scores. This method of determining critical scores has little advantage over the method of in- spection. Ruml (160, p. 59) gives elsewhere the formula for the rank-tangential coefficient (¢) which is equivalent to biserial 7 applied to ranked data. The correlation ratio (7) is a truer measure than 7 of the relationship between two variables if this relationship is curvilinear. The formula is em) a (23) pm Ve where 7. is the number of measures in a column, ¥. the arith- metic mean of the column, ¥ the arithmetic mean of all the ’s in the table, o, the standard deviation of all the ¥’s in the table, and NV the total number of measures. The correla- tion ratio is greater than the correlation coefficient where the regression is non-rectilinear, but in rectilinear regression the i84 VALIDATION OF MEASURING INSTRUMENTS 185 two are approximately equal. The correlation ratio always has a positive sign. For discussions of 7 see Rugg or Brown and Thomson. Blanks for convenience of computation are given by Crathorne (40) and Holzinger (74). Figure 23 illustrates the computation of the correlation ratio. The coefficient of mean square contingency (C) while not a measure of correlation, is a somewhat similar measure. It is used to determine the relationship between non-measured, alternative, or attributive variables. Some test responses, answers to questions, and measures of vocational suc- cess are susceptible to this treatment, but both series must be non-quantitative. The data are plotted very much as in the fourfold table, although there may be more than four compartments, depending on the number of attributes of each variable. C measures relationship “in terms of the difference between the numbers of measures actually found in the various compartments of the correlation table (or ‘contingency’ table more generally) and the numbers that might be expected there by pure chance” (157, p. 300). Yule’s formula is where S represents the summation of the values of the fol- lowing fraction for each compartment of the contingency table: (n?) rw N N represents the total number of measures, »,, the number of measures in the compartment of the contingency table, n, the number of measures in the row, and », the number of measures in the column in which this compartment is located. Yule’s coefficient of association (Q) occasionally finds use. The formula is 186 J. EMPLOYMENT PSYCHOLOGY a BS Scores in Aptitude 2 = _ Test G (x) o 3 8 gk RC ff) R&R fo 4 lr a1) 3 09 ° 2| 1.86] an k | & { 14 44| 5 > 2 = | 32 -70 Nx | | J Vx { 3 | 3.67 Ge=3) - of + 771 Fe—3)? | Ap es na(yx—7y)2 1 11.0, I of 3.54 {19.47 N=30 mean of 0 0 yRmGerd=ng Ee JE Oy= \ N= 30 = 1.044 Slna(y:—)?1 =19.47 [S[n=(32—3)*] ~ gm ys Figure 23: Illustrating the computation of the correlation ratio 77). VALIDATION OF MEASURING INSTRUMENTS 187 ad —bc (25) Om where a, b, c¢, and d are the numbers of cases in each of the four quadrants of the fourfold table (233, p. 38). If a test is scored in terms of accuracy, in effect two series of measurements are made. One is the number of right answers, R, and the other the number of wrong answers, WW. Thurstone (199) has derived formulas for weighting right and wrong answers in such a way as to give the highest pos- sible correlation with the criterion, I. The number of wrong answers is multiplied by a constant, C, which is usually neg- ative, and then added to the number of right answers. Thus the score for a test is R + CW. The following formula gives the best value for the multiplying constant, C: (26) C= or(71R * "Rw —"1w) ow(’tw * rw —71r) If this constant is used in scoring the test, the best correla- tion possible between test scores and criterion will be ob- tained. The formula for this correlation is (27) Biro =f 02 £1 Ss: ph. Im Taw — rw The probable error of a correlation coefficient is an index of its unreliability and should be computed as an aid to the interpretation of the significance of the relationship which has been discovered. Thus a coefficient of .50 is a better estimate of the true relationship if its P.E. is .0o2 than if its PE. is .13. The formula for the probable error of 7 computed by any of the variations of the Pearson product-moment formula, or by the rank difference method, is 1—r? (28) PE, =0.6745 = This formula assumes a normal correlation surface, and should therefore be avoided when the number of cases is small (233, pp. 321, 352). It also assumes that the coeffi 3 EMPLOYMENT PSYCHOLOGY cient to which it is applied is the true correlation of the two variables in question. P.E., may be obtained quickly from tables given by Pearson (135), Toops and Miner (204), and Rugg (157), or by graphic methods given by Pearson (135) and Toops (202). The probable error of a correlation ratio is (233, p. 352): I—n? 2 PEy=06745—— The probable error of biserial 7 is (86, p. 249) P.E.,=0.6745———— (30) 0.4748 % The formula for the probable error of the coefficient of mean square contingency is given by Kelley (86, p. 269). The formula given there applies to Pearson’s formula for the coefficient of mean square contingency, which is more com- plex than Yule’s. The general formula for the probable error of » obtained by the fourfold table method is given by Kelley (86, p. 258). The formula to be used when the division lines are at the medians (method of unlike signs) is (86, p. 257): Kelley gives the formulas for the probable error of # cor- rected for attenuation (86, pp. 209-210). We shall quote here only the formula for the probable error of corrected for attenuation by the use of formula 4. Let the corrected value of »,- be represented by 7. 7 I (32) PE, =osrstit I 72x. x, I + (Fo i TA ras Frits 1) LL Ton a Mt +(e 1 lope Ty, 0} 188" VALIDATION OF MEASURING INSTRUMENTS 189 Opinions differ on interpreting the significance of the size of the correlation coefficient. Some authorities contend for a correlation of .60, while others claim that even if the cor- relation is as low as .23 it indicates an important relation- ship between the two variables. Rugg (157, p. 256) de- scribes the significance of different amounts of correlation in the following terms: less than .I5 or .20.....negligible or indifferent .I5 or .20 to .35 or .40.....present but low .35 or .40 to .50 or .60.....markedly present above .60 or .70.....high One cannot generalize about the size of the correlation coefficient in this way. The coefficient must be interpreted in each instance in terms of the total situation, the nature of the group, the thing measured, the reliability of each vari- able, and the size of the probable error of the coefficient. The following are some of the factors which account for low correlation between variables: Actual absence of relation; Restriction in the range of either variable; Failure to keep a third variable constant; Inaccuracy of the original measures; Loaded sampling. In practical use an analysis of the distribution of cases in the scatter diagram may reveal far more significant in- formation than the correlation coefficient. If there is any correlation at all, the scatter diagram should be analyzed in order to set off critical scores and critical sections. We have given consideration to the various methods whereby the extent of the general relationships between measurements of abilities and the criteria of vocational accomplishment are established. In the next chapter we shall deal with the methods of determining the probability of vocational success implied by definite test scores. AY PREDICTION OF VOCATIONAL SUCCESS Prediction based on group comparison. Critical scores and critical sections. THE two foregoing chapters dealt with the determination of the relationship between measurements and the criterion of vocational accomplishment. When significant relation- ships have been discovered either by inspection or by the statistical methods there described, the relative value of par- ticular scores in forecasting vocational success must be de- termined before practical use can be made of these findings. Two general types of procedure are available, one based on the study of group differences, the other on the establish- ment of correlation between measurements. PREDICTION BASED ON GROUP COMPARISON The groups to be compared are made up of the successful and the unsuccessful employees; or those who remain with the company and those who leave; those who learn quickly and those who take a longer time to acquire the necessary vocational skill; those who earn a promotion and those who do not; those who have few accidents and those who have many; those who sell more than their quota and those who sell less, and so forth. When two or more groups represent- ing different levels of vocational accomplishment are to be compared in terms of test scores, practical procedure de- mands a way of determining the score or scores which best differentiate the group or groups. A test score above which or below which the members of one group are represented in conspicuously greater propor- tion than the members of the other group, is called a critical Xn IQ0 PREDICTION OF VOCATIONAL SUCCESS 191 score. The range marked off by the critical score is called a preferred range or a critical section. For instance, an intelligence test score of C + on Army Alpha is a critical score for success in a business career, because the proportion of men who become successes in busi- ness is small among men who score lower than this. For inspectors in one great factory a score of 70 in Army Alpha was found to be a critical score. Of 337 men who in two successive years were trained to this work, 224 were satisfactory. Of the 281 men who scored 70 or better, 75% were satisfactory, while of the 56 men who scored below 70, only 23% proved satisfactory. In other words, the pre- ferred range for men to be trained as inspectors in that plant is 70 or above. Often the preferred range lies between two critical scores, because workers whose abilities are above a certain maxi- mum score are soon dissatisfied, and leave before they have made good. It is quite true of salesmen for certain products and of operatives on certain machine processes that ‘the brighter they are the quicker they leave.” For clerical em- ployees doing purely routine work there is one preferred range, while another range is preferred for jobs where some independent judgment or initiative is of value. The best procedure in establishing critical scores is to plot scores made by members of the two groups—successes and failures—on the same chart, indicating successes by one kind of symbol and failures by another kind. If there is an intermediate group, a third symbol may be used. The in- vestigator may then by inspection easily find the sections of the distribution in which one group is represented in rela- tively greater proportion than the other groups. These critical sections are then set off by critical scores. They should include a large proportion of all the cases. Critical scores may also be determined by the use of the formula for Pearson biserial 7 (formula 22). The significance of critical sections may also be deter- mined by considering the alternative categories assumed in x Xx x Xo en 00 0.0 O 00 0 00000 0 x ere SR XO XX OO 0 0 0 _ 0 oo 0 00000 OX OXXN¥ _ 50 55 60 65 70 75 go 25-40 95 100 105 10 ns 120 125 i130 -y Score . O° 0 =Those who have remained with the company 0 X =Those who have left the company Figure 24: Mental alertness scores of workers who have remained with the firm and workers who have left. Stability at this work apparently calls for a moderate degree of mental alertness. To eliminate those who are too bright or too dull critical scores should be set at v8 and 121. Data from Kenagy and Yoakum (89, p. 255.) 193 ered alternative answers to a question, and comparing the difference in proportions of the two groups found in the crit- 1C¢ Ll sections D] means OI 1ormuia 10 2 2%: hart ©) anda Moore i 15). S ee I 1gurcs 2 4 and 25 Critical scores may also be set with regard to a combina- . > yy TY . 5 ® ® Wo og ® ® ® eo ) a i ® & ee % 0 0) @) oO ] ( n ) | OO j | a NT) y= Wy —r——1 Fes 21 041 2 3 45% 78. 979°1112% Figure 25: Personal history scores of salesmen at different levels of success, Kenagy and Yoakum (89, p. 201). EMPLOYMENT PSYCHOLOGY tion of two tests. If a scatter diagram of scores in two tests is plotted, each individual being represented by a symbol which indicates the class of vocational accomplishment to which he belongs, sections of this scatter diagram may be set off within which one grade of success is represented in rela- tively greater proportion than the others. When these sec- tions are mapped out, the critical scores may be stated in terms of the two tests. For example, the statement may take this form: “It is a favorable indication if a person scores between % and / in Test X and at the same time scores between mm and # in Test VV.” This method should not be pursued unless there is ground for the belief that both tests have differentiating value. Bills (9) has used this technique to good effect. See Figure 26. Several methods are open for determining the prognostic value of answers to questions. All have as their purpose the determination of a scoring method which will give weight to only the significant questions in a questionnaire (or items in a test). If it has been found that a larger proportion of vocational successes than of vocational failures gives a certain answer to a question, it may be assumed that such an answer is a favorable indication of success. We may select in any ques- tionnaire all the questions which are answered in a certain way by widely differing proportions of the two groups (suc- cesses and failures), and score the questionnaire on these items alone. Thus the individual receives a score of +1 for each answer which is given by a larger proportion of suc- cesses than failures; and —1 for each answer which is given by a larger proportion of failures than successes. His score for the questionnaire is then the algebraic sum of the scores assigned to his significant answers. A refinement of this procedure is to determine by formula 10 the significance of the differences in proportions, and to include only those items where the difference in proportions is at least twice its standard error. To be still more exact, each answer may be assigned a weight varying with the size 104 PREDICTION OF VOCATIONAL SUCCESS 195 121-130 i 111—120 © 101-110 | o O oo 91—100 | Oo 0 81—90 O 0 @0]| 71—80 M Od 51-70 | @ O | © © So eo) 51—60 | @ 00 O 41—50 | 31-40 ! ® CC 21-30 ® © © 11—20 do Q 0—10 g ig gl "pled Bu hgill BRL RES d fdudiig gRidiugh id id Scores in Test IX, Part 10 Figure 26: Scores of successful and unsuccessful comptometer operators in two tests, illustrating the use of critical scores in a combination of two tests. Critical scores are indicated in the chart by intersecting lines. It is a favorable indication if an applicant scores above 30 in Test VI and also above 59 in Test IX, Part 10. Data from Bills (9, Dp. 282). of the ratio of the difference in proportion to its standard error; but this procedure is seldom worth the additional labor (57). (See also pages 201-202.) Once the questionnaire has been scored, critical scores and O ™ EMPLOYMENT PSYCHOLOGY critical sections may be set off in the same way that critical scores in a test are determined. The items in a test may be evaluated individually as if they were items in a questionnaire and the score for the test obtained in the way indicated. These total scores may then be examined in order to establish critical scores and sections. When the test is reprinted for use in the employment process, the items which have no validity may be omitted. To mini- mize effects of coaching, alternative test forms of equal valid- ity may be constructed from the significant items, 106 Y PREDICTION OF VOCATIONAL SUCCESS (Concluded) Prediction based on correlation. Regression line and regression equation. Standard error of estimate. Percentile curves. ALTHOUGH the correlation coefficient, if high, gives assur- ance that the test scores are related to success in the voca- tion, it gives no indication of the probable degree of voca- tional success of an individual who obtains a certain test score. To make a prediction of the unknown variable from the known variable, the criterion from the test score, use is made of the regression line or the regression equation. For every correlation table there are two regression lines and two regression equations. The one to be used depends on which is the known variable. The investigator must always be certain that he is using the right regression, namely, the criterion on the test score. Knowledge of the coefficient of correlation between test score and criterion is not a necessary prerequisite to the use of the regression equation, but it must be known in order to judge the value of the regression equation. The regression lines may be obtained in several ways. If a scatter diagram is drawn, the mean of each column of figures may be estimated and represented by a dot, and then by adjusting a tight string over the diagram the straight line may be found which best fits the dots. If this line is drawn on the diagram, it may be used in obtaining the most prob- able value of the unknown variable, the ordinate (usually denoted by V), corresponding to a given value of the known variable plotted as an abscissa (X). (This is known as the LVI 107 EMPLOYMENT PSYCHOLOGY regression of ¥ on X. The regression of X on V, by which X is predicted from ¥Y, may be obtained by estimating the means of the rows of figures.) The regression line as thus determined may be repre- sented by the regression equation: (33) Y=a4+b-X The multiplying constant b is the slope of the regression line, the tangent of the angle formed by the regression line with the horizontal. The additive constant a is necessary if the point of origin of the regression line does not coincide with the zero points of both abscissa and ordinate scales. Its amount may be determined by inspection. Reversing this process, the values of a and b may be ob- tained by the method of least squares (197, p. 9), and the regression line may then be drawn from these determined values. The formulas are ZX .2YXV-zX.3Y (34) gr nyo x Ew >X:ZY-N-Z2XY (35) ES If the mean of the X’s is represented by a vertical line through the scatter diagram and the mean of the ¥’s by a horizontal line, the point of intersection of these twos lines should also be the point of intersection of the two regression lines. This point may be made the origin of the regression lines by expressing all measures in terms of deviations from the means of their distributions. Since the point of origin then has zero value for both distributions, no additive con- stant is necessary. X and V, when they are given as devia- tions from their means, are represented by the symbols x and y. The regression formula then becomes (36) y=b-x The multiplying constant b in the above case is equal to 7 2, Hence, if » and o,, 0. are known, the slope of the 108 PREDICTION OF VOCATIONAL SUCCESS 199 regression line may be determined exactly and not by esti- mation, and the regression equation takes the form: (37) y=roPx This formula gives the most probable value of y corre- sponding to a given value of x, both in terms of deviations from their means. This formula may also be expressed in terms of the orig- inal measures: (38) Y =r (X~—m,) +m, in which 7, is the mean of the ¥’s and 7. the mean of the X's. The criterion (¥) may be predicted from test score (X) by either the regression line or the regression equation. If X is to be predicted from ¥, the letters should be trans- posed wherever they occur in the above discussion. For good discussions of the regression line and regression equation, see Thurstone (196) and Rugg (157). The regression equation gives for each value of the known variable only the most probable value of the predicted vari- able. The error of this predicted value cannot be deter- mined from the regression equation alone. The standard error of estimate of a second variable, predicted from the first, may be found by the formula given by Kelley (86, p. 173): (39) o21=0\V 1—1? This formula applies if the measures are either in their orig- inal form, or expressed as deviations from their respective means. If the measures are expressed as deviations from their respective means with the standard deviation as the unit of measurement (that is, as x/¢,), then the formula for the standard error of estimate is e=V1—7r? 40) EMPLOYMENT PSYCHOLOGY See Kelley (86, p. 173) and Ruml (158). The second term in formula 40 is called the coefficient of alienation (k) by Kelley. The relationship between 7 and € is the same as that of the sine of an angle and the cosine. It is brought out in Figure 27. Compare, for example, the standard error of estimate of a correlation of .30 with one of .10. Note that it is nearly as large. Then compare the standard error of estimate of a correlation of .75 with one of .95. Note that the latter is only about half as large as the former. 10 = == — - 80 +4 5 | 460 3 3 Pi tr 3 2 Oo 20 4 .00 Loo ea = — .00 20 40 60 .80 1.2) Coefficient of Correlation Figure 27: Relationship between the coefficient of correlation (r) and the coefficient of alienation or standard error of estimate (€). The standard error of estimate gives an idea of the relia- bility of the value of the unknown variable which is pre- dicted from a given value of the known variable. The prob- ability is that in 68% of the cases the actual value of the unknown variable will be within the limits of e. 200 . = & BM PREDICTION OF VOCATIONAL SUCCESS 201 A second method of treating measurements which have been evaluated by the method of correlation calls for the construction of percentile curves or tables. This process has been adequately explained by Thurstone (194) and Otis (125). The percentile graph or table gives for each test score the percentage of individuals who obtain that score or less. Instead of predicting success in terms of the cri- terion, this method predicts it in terms of the percentage of applicants whom this individual equals or excels in the test. Tables of deciles, quintiles, or quartiles may be cal- culated, when it is unnecessary to have such a fine division as hundredths. The percentile curve of test scores means nothing in the absence of a high correlation with the cri- terion; but if the correlation between test and criterion is known to be high, it may be inferred that the applicant’s percentile rank in the test gives also his approximate rank in future vocational accomplishment. The value of predictions made by this method, as well as by the method based on the regression equation, depends on the amount of correlation between test and criterion. Formula 21 may be employed in determining the validity of questionnaire or test items whose answers are qualitative and dichotomous. When this has been done, it is necessary to devise a scoring method for the test or questionnaire which shall give proper weight to the valid items. We have already discussed on pages 194-195 the methods of weighting answers to questionnaire items when group differences have been calculated. When formula 21 has been employed, the weight to be assigned to answers should be in accordance with values of 4 in the following formula (39): ThE bil Thre In this formula ¢ is the standard deviation of the frequencies in the categories of the x distribution, or distribution of answers to the items. The value of ¢ is given in the formula at the top of the following page (86, p. 89): EMPLOYMENT PSYCHOLOGY (42) o=V Npq N is the number of cases, p the proportion of these cases giving one answer to the item, and ¢ the proportion giving the other answer. We have described the practical procedure to be followed when the success of the applicant is to be predicted from a measurement. The measurement may be a test score, a questionnaire item, an interview rating, or any one of the facts about an applicant’s personal history which can be quantified and correlated with a criterion of successful ac- complishment. In actual practice, probability of success is, of course, usually predicted from a combination of such items. The more information the better, provided its sig- nificance is known and not merely guessed at. When pre- dicting success from a number of such measurements it is necessary to combine them in such a way that the resulting total score will have maximum predictive value. This pro- cedure forms the subject of the next chapter. 202 1 XVII PREDICTION BY COMBINED SCORES Total scores based on group comparisons. Weighting critical sections. Total scores based on the establishment of correlation. Multiple correlation co- efficient and multiple regression equation. Other methods of combining scores. WHEN the predictive values of scores in the various sepa- rate tests or other measuring instruments have been de- termined, the next step is to find the predictive value of com- bined scores in all of these measuring instruments. Since the valid tests will not all forecast vocational accomplish- ment with equal accuracy, they should be properly weighted in obtaining a combined score. This combined score should yield a more accurate prediction than the score in any one particular test. Such a total score eliminates opinion based only on general impressions of the combined significance of test scores, thus retaining the principle of objectivity. Here as in the preceding chapters the methods are treated under two headings: those to be used where group com- parisons have been made, and those to be used where cor- relations have been calculated. TOTAL SCORES BASED ON GROUP COMPARISONS Tests which have been evaluated by methods of group comparison and for which critical sections have been de- termined should be combined with reference to critical sec- tions and not to raw scores. To add raw scores would in- troduce an unintentional weighting unless correction were made for differences in values of means and standard devi- ations (see pages 206 ff). It would introduce in this case an additional error, inasmuch as critical sections are fixed with- 20% EMPLOYMENT PSYCHOLOGY out regard to differences in scores within the sections. The same unit value must be given to all scores falling within a critical section. The weights must refer to the sections and not to the raw scores. Critical sections may be weighted in accordance with (1) significance of the group differences, (2) predictive value, (3) reliability, or (4) a combination of these indicators of value. If critical sections for the tests have been found, all of which have high significance, there is no point in giving them different weights. The individual is allowed =1 for each critical section in which his scores fall. If, for example, a critical section has been found in which there are many more successes than failures, an applicant who scores in that section is given -} 1 for that performance. Again, if a critical section has been found in that same test or another test in which the failures are represented in much greater proportion than the successes, then any applicant who scores within that section is given —1. If he does not score within a critical section in a test, he receives zero for that test. An individual’s score is the algebraic sum of the points so determined. If the critical sections vary considerably in their signifi- cance, they may be weighted in accordance with their rela- tive significance. If the percentages of the different groups who score in a critical section have been determined (per- centage of all successes and percentage of all failures), and the difference in these percentages divided by the error of the difference, a weight may be assigned to the section in proportion to this quotient. The weight, in other words, should be proportional to the statistical significance of the differences between the percentages of the two groups scor- ing within the several critical sections. This method re- quires then the following figures: 1. Proportion of all successes who score within tbe critical section: 204 PREDICTION BY COMBINED SCORES 2. Proportion of all failures who score within the critical sec- tion; 3. Difference in proportions; 4. Standard error of the difference in proportions, using for- mula 10 and considering any score within a critical sec- tion as the equivalent of a certain answer to a question; 5. Difference in proportions divided by its standard error. If a person scores within a critical section, he is credited with the amount of this fifth value in place of his raw score. His total score is the sum of such values which he earns. Weights may be assigned to critical sections in accordance with the probability that an individual will achieve suc- cess if he scores within the critical section. Thus if 90% of the individuals in the critical section are successes, the probability of success of any person who scores within that section is 9o/100, provided the subjects of the investigation are representative. Similarly, if 20% of the individuals in a critical section are successes, the probability of success of any individual who scores within that section is 20/100. The weights assigned to the critical sections may be in ac- cordance with this ratio or percentage. Note that critical scores are not determined on this basis, but on the basis of difference in percentages of each total group (successes and failures) found within the critical section. If 20% of the individuals in a critical section were successes, and at the same time 20% of all individuals tested were successes, no significance would attach to the critical section. This would not, strictly speaking, be a critical section. But if 20% of the successes and 70% of the fail- ures scored within the critical section, it would be very significant. Weights may be assigned to the critical sections in ac- cordance with the reliability of the test or critical section; that is, in accordance with the length of the test, its con- tinued significance with different groups, or the relative number of cases in the critical section. The latter is a prac- tical index of the reliability of the critical section. A criti- 208 EMPLOYMENT PSYCHOLOGY cal section which includes a small fraction of the total num- ber of cases is less reliable than one which includes 50% of the cases. The use of any one of these factors in weighting critical sections implies that the other factors are constant. It is very improbable that this will be the case. If the investi- gator weights critical sections in accordance with but one factor, he is likely to find himself ignoring other equally im- portant factors. Perhaps the best policy for the investigator to pursue is to use his judgment in weighting the various critical sections, keeping in mind their relative value in terms of validity and reliability. When a scoring method for a questionnaire has been devised by methods previously outlined, critical sections may be determined and weighted for combination with other questionnaires or tests. TOTAL SCORES BASED ON THE ESTABLISHMENT OF CORRELATION Some methods of combining scores make no attempt to determine the precise relative significance of the scores to be combined. In the first five methods here described, no attention is paid to differences in closeness of correlation, but the assumption is made that no test receives considera- tion unless it has shown a significant correlation with the criterion. The last four methods of obtaining the total score, on the other hand, take into consideration the rela- tive validity of the tests. The simplest expedient is that of adding raw scores made in the various tests. This method may err in giving some tests undue weight over others. If the range of scores in one test is from 1 to 5, and in another from 50 to 100, ad- dition of raw scores gives the second test many times the weight of the first. This may or may not be justifiable, de- pending on conditions. Certainly raw scores should never be added without first scrutinizing the relative range of 200 PREDICTION BY COMBINED SCORES 2 scores. Another point not to be overlooked is the relative dispersion of scores in the two tests. The test with the greater standard deviation receives more weight when raw scores are added even though the range of scores in the two tests is the same. The method of adding raw scores should then ordinarily be avoided. Error due to unintentional weighting of the first sort men- tioned in the preceding paragraph may be eliminated by weighting to equalize the medians or means in the various tests. If the median score in one test is five times as great as the median score in another test, then scores in the second test should be multiplied by five. This method, however, does not take into account the relative dispersion of scores in the two tests. To do this the scores in each test should be expressed in terms of the standard deviation of scores in that test. This step reduces to the same unit of measure- ment the scores in the different tests. To express any per- son’s score in a test in terms of the standard deviation, the deviation of his score from the mean of the distribution is divided by the standard deviation of the distribution, and a plus or minus sign affixed, depending on whether his score is above or below the mean. (If the test has a negative cor- relation with the criterion, then a plus sign is affixed before a score which is below the mean, and vice versa.) The probable error of the distribution should be used as the unit measure of dispersion, if the median instead of the mean has been calculated. The individual’s total score is the algebraic sum of his scores in each of the separate tests, as thus computed. Adding percentile standings in the various tests equalizes the weights. So, too, when decile or quintile standings are added. A total score so obtained must be considered as a measure of relative standing only. The above methods assign equal weight to the various tests. Once the matter of unintentional weighting has been guarded against in this way, the proper weight to be given to each test may be determined. This weight will depend both 20% EMPLOYMENT PSYCHOLOGY on the reliability of the measure and on the validity with which it predicts the criterion. Tests may be weighted roughly in accordance with their reliability. The reliability of a test can best be calculated by giving it a second time and correlating the two series of scores. The reliability coefficient, however expressed, may then be used in giving relative weight to the test. Measures of reliability have been discussed in Chapter XIII. A rough system of weighting, when each test has been correlated once with the criterion of success, is to assign weights to the various test scores in accordance with the size of the correlation of the particular test with the cri- terion. Test scores must previously have been equalized by weighting to overcome the weakness pointed out in the first method in this section. Scores in a test may be weighted in accordance with the regression equation of the criterion on that test. If raw scores are used, their values for the test should be multiplied by the fraction Tule and then added to the additive con- stant. In this fraction y refers to the criterion and x to the test. A person’s total score will be the summation of his scores in the separate tests after each has been weighted in this manner. If measures are in terms of deviations from the mean, there is no additive constant. Another method is to weight scores in a test in inverse proportion to the standard error of estimate of the criterion predicted from scores in that test (see pages 199-200). The most accurate statistical device for combining scores in several tests is the multiple regression equation. The multiple correlation coefficient, represented by Ri(s...n), is the correlation between the criterion, 1, and the best weighted combination of the various tests, 23. . .x. The multiple correlation coefficient may be obtained by the formula given by Yule (233, formula 15, p. 248). The formula for three variables is as follows: a Ri (a3) =V/ 1—(1—7%)(1—1%,) 208 43) PREDICTION BY COMBINED SCORES The partial correlation coeflicient of the first order, iss, IS obtained by the following formula (233, p. 238). It ex- presses the correlation between variables r and 3 if variable 2 is held constant. (44) 713.2 Gamers TE TS ee Vi—r 1-7, By substitution, any partial coefficient of the first order may be obtained. The formula for the multiple correlation coefficient for # variables takes the following form: (45) Ritos..0) = V I—(1—7%) (1—1%,) (1-125) <r KI 0.25: + r~1y) The necessary partial correlation coefficients for # variables may be obtained by the following formula (2 33, p- 238): Fionn T12.34: 00h) = Pim. (n=l) Yon.34...(n—1) (46) a V1 or ¢ TT * V1 Se 120.30 nD This formula expresses the correlation between variables z and 2 with all the other measured variables held constant. When the required zero order correlation coefficients (the simple correlation of two variables) have been computed, partial coefficients of the first order may be obtained by formula 44. Coefficients of each succeeding order may be obtained by a process of building up step by step, using formula 46, until all partial coefficients necessary for the computation of R by formula 45 are available. The formulas assume that all relationships are linear. For this reason they must be used with the greatest caution. For example, the relation between test score and length of ser- vice is rarely linear. The sign of R must be inferred from the data. R is never less than the correlation of the criterion with any of the constituent tests, and is usually greater than the highest correlation between the criterion and any one of the tests. The multiple correlation coefficient does not tell how 200 EMPLOYMENT PSYCHOLOGY much weight to give scores in each test when combining scores in the several tests in the series. When making prac- tical applications the proper weighting of tests is found by solving the multiple regression equation. The multiplying constant for each test is the partial regression coefficient of the criterion on that test, when each of the remaining tests is held constant. The equation for three variables when raw scores are used is as follows: (47) Xi=l0: X04 050 Xs +C In this formula the &’s and the C are constants so chosen that when the scores in tests 2 and 3 are multiplied by the &’s and added to C, a final measure, X,, is obtained which will give the highest possible correlation with the criterion (84, p. 14). X, is not to be confused with the criterion as actually measured. The formula for the multiplying con- stants for three variables are (233, p. 237): I Ll bio —0b13 + bs (48) bros I —bss + ba bis = ryt , 13 =p yehe, g2 a3 b... may be obtained by substitution in the above formula. For » variables the formula for X, is (49) Xi =br.au..nX2 +0134. n Xt» +b1n2s...0yXn+C The formula for the multiplying constants for # variables is (233, p- 237): Wi b19.34...cn—1y — D1m.sa...cn—1y 2 Ona.380 + (n—1) (50) EE I —ban.3s...n-1) * bugsa...n—1) These may be computed from &’s of lower orders by the above formulas. The formula for the additive constant, C, for three vari- ables is (51) C=M;—b123M>—b132M3 M, is the mean of the measurements of the criterion, and M, and MM. the means of tests 2 and 3. The formula for the 210 PREDICTION BY COMBINED SCORES constant, C, for # variables is as follows (84, p. 14): (52) C=M—bpss.. ,My—b304... 4 M3— ey. —bin.23eecn—1y Mp If the measures are expressed as deviations from the means of their distributions, the formulas are the same, except that small x’s are used and the additive constant, C, is omitted. The formula for the probable error of a multiple correla- tion coefficient is the same as for a zero order coefficient: 1—R%03...n (53) PE rn. =06745— 52 The formula for the standard error of estimate for the criterion predicted from the scores in tests, weighted in accordance with the multiple regression equation, is, for three variables: (54) oL=0V1—1% Vi-r%, and for » variables: (55) 01.23.00 =I VI—7 VI—123, V I—740 + + VI — 7212.93. + .(n—1) For further discussions of the multiple correlation coeffi- cient and the multiple regression equation, see Kelley (834), Yule (233), Rosenow (151), Symonds (178), Otis (123), Huffaker (77), and Garfiel (60a). (See Figures 28 and 29.) For combining scores in two tests, Thurstone’s method of scoring tests by weighting right and wrong answers may be used. R and W are taken to represent scores in the two tests which are to be combined so as to obtain the highest correlation with the criterion (see page 187). Chapman (26) describes a procedure which may be fol- lowed for “the investigation of the effect of adding a single test to a battery already weighted as though complete in itself.” OTHER METHODS The profile is not, strictly speaking, a method of obtain- ing a total score, but presents graphically an individuals 211 EMPLOYMENT PSYCHOLOGY scores in several tests in relation to each other. A curve is plotted showing his relative standing in the various tests. His good and poor test performances become evident at a glance. Knowing the relative validity of the various tests, Y = . ra rosa wf: 12 BET vi1—rip . VI1—r2m ” Fas—V43 * V3 Bae V I—172% . VI —r2 RE ry Tuu=713 2 71s B20 VI—r%s- Vi-74 = , i Tia.2— 713.2 * 743.2 HBT 1—73s* V1I—7ua i Riese) = VI —(1—1%) (1 =r.) (1 — 1214.03) = Figure 28: Outline for computing the multiple correlation coefficient for four variables. the investigator may make a prediction of the worker’s suc- cess. This method involves subjective judgment in predic- tion. It sometimes leads to judgments on the individual’s weak points based on groundless assumptions as to the abili- ties measured by the tests. It is more useful for vocational 212 PREDICTION BY COMBINED SCORES g rie = o1= 2” = r= oy = z T4= g3 = . Ye = o4= rar= Yaa = (These are equivalent to 7a, 731, etc.) oc o bo =rt—= bsg =rp— = bay =ry—= a2 age g4 o o biz= rn = bu= ru— = b= ro— = a3 a4 a2 i . gn a3 vg 04 b nd bio—bi3 + bos sat b == biz—bia * bao 12.3 I 0 . bog 13.2 I —bg . bas Dis nve b1a— bys bs = Uraip =e bia—bys + bao 14.3 I ~Dss : bas 14.2 I gs 2 Da boo babe bu _ be Dubus bn _ 42.3 I — bsg J bos 43.2 I — bys . bso yo bu=bn ba _ yo bu=bn ba _ 24.3 A 34.2 I= ba b B=: bio.3— bras . bas.3 - Ra I—bos3 * baz.s bisugs= biz.o—Dbras + base _ 15.8 = r—— TT I—bsus * base a bis.o—biz.2 + bass - WE 1 —baz2 + base C=M,—bss My—bi3.04 M3—brs.03 Ms= X1=bpau Xo +bi.2uXs+bunXi+C= Figure 29: Outline for computing the multiple regression equation for four variables. 2173 ] EMPLOYMENT PSYCHOLOGY guidance and placement, in gaging abilities with reference to a number of occupations. For vocational selection, however, a total score has much more predictive value, is easier to obtain, and is less subject to individual errors. If the inves- tigator merely wishes to know the number of tests in which the person obtains a high score, he will find that this is expressed better numerically. The profile method has the advantage of graphic presentation, but a graphic method of presenting a total score may also be easily devised. Another way of predicting from a battery of tests is to give the most valid and reliable test first. If the applicant’s score in ‘this test clearly predicts success, the applicant is hired. If it clearly predicts failure, the applicant is re- jected. If it leaves the matter in doubt, the series of tests is continued until success or failure is unequivocally predicted. But if the applicant is accepted, the series should be com- pleted in order that there may be no serious gaps in test rec- ords when the check-up of predictions is made later on. This method has little to recommend it over a total score. The same statistical methods of evaluation and prediction may be applied to the total score as to scores in any single test. The total score should, of course, have more predictive value than the score in any one test. This completes our outline of statistical procedures useful in determining the reliability and the validity of the tests and in securing from them their maximum predictive value. We shall now consider what economic savings these predic- tions imply, 214 XVIII ECONOMIC VALUE OF THE EXAMINATIONS Necessity for comparison of old and new methods of selection. Relative predictive accuracy. Relative economic value. WILL it pay to use the scientific selection method? The relative cost of installing and using improved methods of selecting employees and the economic savings to the man- agement which follow from their adoption are matters which the investigator cannot ignore. He may have established the validity of the new methods, but to justify their installa- tion he must interpret them to the management in economic terms. He will be asked, first, whether the new methods give more dependable predictions than the old, and second, whether the net savings are greater in view of relative cost. To answer the first of these questions, the investigator must compare the validity of the new method of selecting employees with the one previously in use. The same stand- ards of evaluation must be applied to both. It is possible that the old method may have greater accuracy than the new. There are grounds for believing that a scientific method of selection has greater validity than an unscientific one, but it is unscientific to make an a priori assumption to this effect. The second question requires for its answer a comparison of methods from the economic standpoint. The investigator should estimate, as well as available cost data permit, the relative expense of administering the old and the new pro- cedures and the savings which the use of the more valid method makes possible. In making his report to the management, the investigator’s case for scientific selection must be presented from these two 21K EMPLOYMENT PSYCHOLOGY angles. He must show, first, that the selective method he has devised has greater predictive accuracy than the meth- ods hitherto used in forecasting the vocational success of applicants. He must show, second, that when consideration is given both to cost of installation and operation of the selection methods and to increase in efficiency of operating departments resulting from selection of workers with higher vocational accomplishment, use of the new method will re- sult in a net economic saving. RELATIVE PREDICTIVE ACCURACY The methods of selection most commonly used in industry are of two sorts. They depend either on general impressions gained from application blank and personal interview or on the use of one of the many pseudoscientific methods of char- acter analysis. In either case it devolves upon the investi- gator to make an evaluation of the accuracy of prediction by these methods, employing the same rigorous procedure as in the evaluation of his own examinations, and using the same criterion of vocational accomplishment. Only under these conditions can a real comparison be made. If the interview is emphasized in the method of selection in use, the judgments of independent interviewers may be compared. In most instances the wide disagreement among interviewers will provide an opportunity for demonstrating the greater objectivity of tests. Wembridge (218) compared the validity of an intelligence test modeled upon Army Alpha and Beta with interviewers’ judgments of probable success. His subjects were several hundred machine operators in a clothing plant. The method of payment was that of piece rates with bonuses for a certain standard of quantity and quality. This standard was deter- mined by time studies and was applicable to all workers. The criterion of success was the percentage of this standard attained by the worker. The criterion and ratings by two 216 ECONOMIC VALUE OF EXAMINATIONS - Percentage Hired Efficiency Percentage Hired hy Old Examinations of Employees by New Examinations 0 High [ we Above Average mT | 349%, Z. Below Average [1 7% & Low 07, 0 wm 0 0 SR DRE SE NY Percentage Percentage. Figure 30: Comparison of old and new types of civil service examinations for the selection of postal employees (124). interviewers gave a zero correlation. The criterion and the intelligence test gave a correlation of .51. While making an investigation in the interest of the postal service, O'Rourke (124) compared the validity of a new type of examination with the type long in use, in the selec- tion of mail sorters. His comparisons are shown in Figure 30. His complete report is recommended to the reader for its excellent analysis of improvements which scientific meth- ods of selection make possible. In this connection it may be suggested that no recom- mendation be made looking toward the exclusive use of tests. The interview is valuable for the elimination of individuals who are unsuited for the vocation because of abilities not ordinarily measurable by tests. Interviewers, however, should be provided with rating scales as aids in judging applicants. These scales should be as objective as possible and the predictive value of the material they cover should have been evaluated the same as test scores. The interview and the tests should supplement each other. The investigator will find it advisable to present in graphic form the results of his comparative evaluation (157, 20). 217 EMPLOYMENT PSYCHOLOGY Use should be made of simple graphs which bring out for- cibly the most important facts. Distribution curves are effective when vocational successes and failures are denoted by distinctive symbols. Other graphs may be constructed to show the percentages of successes selected and failures elim- inated by the various methods of hiring. The investigator by showing the value of mass results may minimize the im- pression made on the layman by the individual who stands out as an exception to the rule and who by contrast to the others assumes exaggerated importance. RELATIVE ECONOMIC VALUE If greater accuracy of prediction is possible by the use of the examinations than by the customary methods of selec- tion, the investigator must compute or estimate as accurately as he can how much, in the course of a year, this saving will be. Perhaps the predicted economies through improved selection will not equal the cost of installing and administer- ing the examinations. Sometimes, on the other hand, these predicted economies are so large as to seem incredible at first. In any event, the facts are essential. The research worker should be impatient of mere hopes or vague opinions regarding the relative cost and money value of selection methods old and new. Relative cost of administrating the old and the new meth- ods is unimportant compared with the savings on the pay- roll effected through improved selection. Nevertheless, it is usually advisable to estimate this cost by obtaining actual figures on such savings as these: Advertising. This is commonly advertising in the “Help Wanted” columns of newspapers and magazines, and for comparative purposes should be stated in terms of the cost per successful employee obtained. Correspondence. The cost of correspondence with appli- cants is another item of expense. This consists chiefly of the 218 ECONOMIC VALUE OF EXAMINATIONS 219 fraction of the salary of the executive who has to give time to this correspondence. Traveling expenses. This item may be either the travel- ing expenses of the applicant to the place of employment or of the interviewer or scout to the place of residence of the applicant. Salaries of interviewers and of other workers in the em- ployment department. Some consideration should also be given to the cost of the time devoted by the foreman and other executives to interviewing men. Office overhead. This includes rent, telephone, light, heat, janitor service, and other overhead for office space and equipment in comparison with what would be demanded after adoption of the new measurement methods. The major economies, the savings in the plant or office, remain to be estimated. Even though it be found that the new methods cost more to administer than the old, their net economic value may be very large. These savings will be found by estimating the relative cost of the one or more items which have served as criteria for evaluating the exam- inations. The firm may already have computed the cost of its labor turnover in the departments affected by the investi- gation. If so, these figures will be useful in estimating the dollar value of the new selection methods. In any event, the following sources of economies should be scrutinized. Cost of training. This includes salary of instructors and supervisors, cost of materials used in training, office over- head, and chiefly campensation to the employee for unpro- ductive time during training. On jobs where there is heavy investment in machines or where the cost of power to oper- ate the machines is high, the appropriate charge against cost of training new workers may be several times the wages. Cost of spoilage. A high turnover means that a large per- centage of employees are new and inefficient. This fre- EMPLOYMENT PSYCHOLOGY quently results in spoilage, breakage, or loss of materials, and poor quality of output leading to dissatisfied customers or damage claims; with inexperienced salesmen there is fre- quently loss of sales and alienation of former customers. Wages of inefficient employees. This item covers only the sums paid to employees hired under the less rigorous selec- tion methods, who would be rejected if measurement meth- ods were used and who would, in any event, eventually be discharged or leave of their own accord. Differences in production. If the new method enables the firm to select a larger proportion of the more efficient and able workers, it is logical to expect an increase in average production per man. This can sometimes be estimated fairly well. Savings inferred from decreased labor turnover. Some in- vestigators have aimed at a short-cut to a convincing esti- mate of the savings effected by the adoption of improved employment procedures. Their plan is to secure a general estimate of the cost of labor turnover, and then to make their computation simply by comparison of the turnover before and after the change in employment methods. The utmost caution is needed when such a comparison is made, because the rate of turnover is bound to be influenced by many factors in addition to the effectiveness of the selection process. Changes in general business conditions and in the local labor market must be reckoned with, as well as changes in supervision and the efficiency of management within the plant. In view of these complications which enter into any rigorous analysis of labor turnover, it is not advisable to rely merely on estimates of savings based on a favorable change in the turnover rate. Instead, the more detailed estimates mentioned above are essential. Values too intangible to be measured. One real but less tangible advantage of improved selection methods is an in- crease in prestige through the fact that the company is 220 ECONOMIC VALUE OF EXAMINATIONS 22: judged by the type of men who represent it and the quality of manufactured products which it places on the market. Still less tangible is the effect of turnover on morale or esprit de corps. A large turnover unsettles a department and is a stimulus for men to leave who ordinarily would remain with the company. Adoption of more efficient methods of selection reduces the cost of some or all of the items mentioned above. It also enables the management to use special incentives with those men who are most likely to succeed. The investigator should secure reliable data on as many of these items as is feasible, and make conservative estimates where precise data are not to be had. In his report the im- portance of these items and the possibility of economies may be presented graphically as well as statistically. This report must distinguish clearly between economies which are cer- tain and those which are only probable or of doubtful amount. Estimates must avoid both ambiguity and exagger- ation if they are to command respect. For his self-satisfaction the investigator may rest assured that better selection methods benefit not only the employer but also the worker, by keeping him out of occupations for which he is unfit and in which he will be discontented. In a broad way the results of his researches will be of value to society, as well as to industry. The considerations here brought forward regarding the comparative worth of the old and the new type of procedure in vocational selection will probably lead to a decision in favor of the introduction of the new methods, provided the investigator has shown resourcefulness as well as analytical talent; has been able to secure a dependable criterion of suc- cess on the job; has made a wise choice of both subjects and tests; and has followed rigorous standards of procedure in measurement and in statistical evaluation of data. We may then, in a final chapter, turn to the practical problems of installation and maintenance of these improved methods of vocational selection. XIX THE EXAMINATIONS AT WORK Installation: the examiner; the forms; the room; the procedure; the examiner as interviewer. Maintenance: modifications of procedure and standards. No research in vocational selection is complete which stops short at this point, ignoring responsibility for the in- stallation and maintenance of the new procedures. Having proved to the satisfaction of himself and the man- agement that the scientific method has disclosed certain ways of measuring abilities which are valid in selecting workers for the job in question and which promise an eco- nomic saving to the firm, it remains for the investigator to see that these methods are properly installed. He should also make sure that they are continued in operation either in their original form or with only such changes as are clearly demanded by changing employment conditions. He should see that changes are made only after application of the same rigorous scientific procedure which characterized the orig- inal study. Maintenance, as well as installation, is an indis- pensable stage of research in vocational selection. INSTALLATION The examiner. To administer application blanks, tests, and other examinations after they have once been developed is less of an undertaking than to carry out the original re- search. The investigator, to be free for other similar studies, will prefer to select and develop assistants to do this routine examining. An examiner should be a bright, accurate, tact- ful person, preferably with a background of experience in the plant and in the interviewing room, as well as a familiar- a 5 THE EXAMINATIONS AT WORK 223 ity with the technical details of his own testing job. He should be personally acceptable, courteous, and friendly, for applicants cannot always do themselves justice in a test given by some one who does not command their confidence. He should be well grounded in scientific method and the theory of tests. His immediate training should include ample drill in the exact procedures of examining, scoring, and interpreting results, so that he will not be tempted to upset the carefully established norms by “improving” or otherwise changing standard practice. An outline of the steps by which the examinations were developed will give him respect for the research method, and will provide the background needed for any additional statistical work re- quired by the investigator for the upkeep of the tests and their further evaluation. The forms. The examinations should be used in selection exactly as evaluated. Forms will have to be devised for making their administration operate smoothly. Directions for giving the tests, scoring the tests, preparing scoring sten- cils, and using the forms, should be provided in permanent form. If test forms are to be printed, they may be bound as a booklet for convenience of administration. It is advisable to have all of the applicant’s test records kept on one sheet. This should provide space for recording not only the test scores, but also the significance of these scores as indications of future vocational performance. A good practice is to print the test record form on a folder of letter size, in which all original test sheets may be filed. A record sheet used in the West Lynn Works of the General Electric Company is given in Figure 31. The room. The provisions for testing will depend on the available office space. In general, the room should conform to the requirements set forth in Chapter XII. For indi- vidual testing, the office of the examiner may serve. For group testing, any large room may be used which is unoccu- pied during certain hours of the day. Committee rooms will EMPLOYMENT PSYCHOLOGY _ __... OFFICE WORKSAMPLES _ To undertake a clerical occupation with reasonable assurance of success, examinee should obtain an A grade in each square listed under the particular type of work desired. sping | Acon tag | J PEE = * =| 0 = | . '2 3 Eo : . Alias Ate Pee mes | «+ DOODOOO OOO OOOO0. Abilities Essential: 1 em OOOO YC) Wed Sone oon on Be, CVO COOICY Orin Pinger Dexteritv— i yy a Ter ae ete cme rer fer oe Muscular Coordination —— im Td nia I | et ee mera mea Fa Tweezer Skill. i em eee i To | a Mechanical Analysis — —_———— Nenana as TTR 1]. 0. —_—— VisualMemory_— ~~~ 5 ———— ho ER oC Auditory Memory — 10: —————— bad ee Le is Neatness- login bls Es ere fm tee meee T 13 aulsty (CanBOE, yet be measured WEN... STC) Orderliness.. i th au be hy ei Tinie labile canton yet be messured FRI hl io eo Ability to Follow Instructions i Wis ability cannot yet Le meaau: a i PatiENCE ciate ere CRD I ars Jw fe frm _ Group Contact — > i a _ Acquired Knowledge Desirable: wo a 4 oe = CTRAINIMAT rere WG eo i ee Punctuation... oy Cepeda pl L ny Vocabulary. ee Rapid and Accurate Filing 4; i Fy Ee Rapid and Accurate Machine Cale. — g5y' — ar a Pa re Rapid and Accurate Typing— 5; : ee Ir Rapid Shorthand and Accur Trans. 3) =r Een EE Figure 31: Test record sheet in use in the West Lynn Works of the General Electric Company. The grade in each worksample or test is recorded in the column headed “Grade.” By reference to tables each grade may be transmuted into a letter grade of A, B, C, or D. These letter grades are recorded in the squares alongside the grade. When considering an applicant for any of the clerical positions listed, one follows the instruction at the top of the sheet. Reproduced by permission of the General Electric Company. S24 THE EXAMINATIONS AT WORK 2) be found convenient for testing small groups. But where all applicants are given a mental alertness test as part of the routine procedure of examination, a small group-examina- tion room should be set aside primarily for this purpose. The procedure. No one has ever proposed abolishing the personal interview and relying wholly on objective tests in employment procedure. Personal history record, interview and tests should supplement each other. The usual practice is for the interviewer to see the applicant first. If he dis- covers no outstanding characteristics of the candidate ob- viously rendering him unsuited for the job, he explains the nature of the work, the working conditions and the rate and method of pay, and indicates something of the company policies. Then if the applicant is interested, the interviewer sees that the application blank is filled out and turns the applicant over to the examiner. After such a preliminary sifting, the examinations should be the deciding factor. The interviewer conserves the time of both applicant and examiner by promptly eliminating any who are obviously unqualified. At the same time he makes the first contact of the applicant with the firm a personal and helpful one. The interviewer may also take over the administration of the questionnaires and may rate the appli- cant. He should be entrusted with the task of finally seeing the applicant and informing him of his selection or rejec- tion. The examiner administers the tests and may also rate the applicant. Physical and medical examinations, and in- terview with the foreman or department head under whom the new employee will work, are also usually considered in- dispensable steps before the employee is finally placed on the pay-roll. Examiner as interviewer. Sometimes the examiner is en- trusted with the responsibility of conducting the employ- ment interview, as well as of administering the tests. In this event he should never overlook the fact that the inter- viewer has three functions: to get information, to give in- 2s 2 EMPLOYMENT PSYCHOLOGY formation, and to make a friend. To ascertain whether the applicant has the qualifications necessary for success in the job for which he is applying is a sufficiently obvious func- tion; but no employment interviewer is successful who ignores the fact that he has at least two other highly impor- tant tasks. One of the duties of the interviewer is to inform the appli- cant about the character of the job and the concern in which he is seeking employment. He must make plain what kind of position is open, what the conditions of employment are, and the duties and privileges connected with the work. He must not only inform the applicant as to what his duties will be; he should acquaint him with the disadvantages as well as the advantages of the job and tell him what future it holds out to him. A new worker who is hired with the wrong impression as to conditions of employment, rate of advance- ment, irksomeness of the duties, and so on, soon leaves or becomes a disgruntled employee. In either event a mistake in interviewing has been made. The interview is the applicant’s first formal contact with the company and as such constitutes his first step in training. In it he takes his initial lesson in company policy and gets his first intimations of the personal attitude expected of him. The interviewer teaches as well as learns. Another function of the interview involves making the applicant want to work for the company in the capacity for which he is best qualified. The interviewer must sell his concern, and not consider himself a purchaser only. He must make a friend. A rejected candidate who goes away feeling that this would be a fine place to work, tells his associates. He increases the likelihood that there will be good material applying in the future. An applicant who leaves the employment office with an unfavorable impression becomes a liability, whereas a skilfully conducted interview would have helped to make him an asset to the concern, whether subsequently employed or not. The successful employment interviewer is thus not merely 226 THE EXAMINATIONS AT WORK 2 an inquisitor but also an instructor and a salesman. There is an artistry of interviewing which involves more than skill in extracting information. A sound psychology carefully avoids the use of inquisito- rial practices in the interview or of test procedures bung- lingly handled by novices in a way to defeat the creation in the applicants of the right attitudes and motives. The leaving interview, likewise, is not complete unless it performs the same three functions—getting information, giving information, and making a friend. The employment manager in the modern personnel de- partment usually designates his most skilled interviewer to talk with leaving employees. He sees both those who are discharged and those who leave voluntarily. The foreman sends to the employment office, along with the time slip, a form on which he states the cause of separation. But often he does not know the real cause. It is the duty of the leav- ing interviewer to get underneath the surface and to find out if possible the essential circumstances which led to the final break. This is not an easy task. The employee, having decided to quit, wants to get away with as little delay as possible. So he is apt to give the most plausible reason for going, rather than to speak frankly about matters which might tend to prolong the interview. He says he is leaving for a better job or is going back to school, when the truth is that he does not know what he is going to do, but sees little chance for advancement where he is, or feels that the in- spector who has been passing on the quality of his work has not been treating him fairly. It takes a shrewd, sympa- thetic, sensible questioner to get at the truth. Such an inter- viewer is often able to uncover conditions which are remedi- able and to save the employee to the company. Sometimes he is able to make the employee see that he is acting rashly or unwisely; that the job to which he is going, while paying nominally a better rate, has disadvantages which more than outweigh the difference in pay. If the leaving interviews are 52% _ EMPLOYMENT PSYCHOLOGY skilfully handled, even those employees who separate them- selves from the company are more likely to want to return later on. They go as friends rather than as enemies. Most employment offices keep statistics on causes of leav- ing. Data from the records of casual interviews are worse than useless. They are misleading. But statistics gathered from conscientious and skilful interviews may throw val- uable light on conditions in the plant, on the personality of foremen and supervisors, and on employee attitudes which modify the validity of test predictions. MAINTENANCE It is a responsibility of the investigator to assure himself that so long as the tests are used they are administered strictly in the manner outlined in his report and that no modifications are made which are not justified scientifically. He may even have to assure himself that the tests are being used at all, as there is a natural tendency to revert to former methods of making off-hand judgments, or for examiners to estimate the applicant’s score without completely carrying out the test procedure. Modifications of procedure and standards. The investi- gator must also, from time to time, check the test results against the criterion of success to find whether the tests are still efficient or whether changing circumstances call for modifications in procedure or in critical scores. Conditions of the labor market or of employment within the plant some- times change the predictive accuracy of tests, so that fre- quent checks should be made on their efficiency. Thus, if a supervisor uses special incentives on those who have passed the tests, on the assumption that these new employees will be so successful that it is especially worth while trying to develop them and hold them, a different sort of person may be led to apply for the position. Moreover, some of those who failed to make good under previous conditions might have succeeded if this new type of supervision had been 298 THE EXAMINATIONS AT WORK 2+; applied; therefore, some who fail technically in the tests may prove to be successful when hired under improved job conditions. Maximum value can be attached to test scores only so long as the conditions under which the applicants are employed with the use of tests reproduce in general the conditions under which they were employed when the tests were evaluated. As the type of applicant changes, the test standards may have to vary. Standards will also shift in accordance with changes in the demand for workers and the available supply. Variations in incentives due to changing economic pressure may invalidate some test norms. If men pass the tests and at a later date leave the company or are discharged, the reasons for leaving should be exam- ined critically to see whether the test predictions were really at fault. The investigator must beware lest knowledge of the tests become general on the part of new applicants. He will be prepared, if need be, to provide alternative forms to guard against coaching. He should be on the lookout for important changes in the job itself. He should keep a continual watch on the tests, and after making due allowances for their margin of reliability, check up the circumstances under which they make faulty pre- diction, and only then apply the proper remedies and changes. After the tests have been in operation for a considerable time, the investigator should find out what they have ac- complished in the way of increasing average production per man, decreasing breakage and loss, reducing labor turnover in the department, increasing the average level of applicants, and so forth, in comparison with other departments or plants where the improved methods of selection have not been introduced. These are secondary criteria of the worth of the tests. In obtaining them the statisticians and cost accountants will be able to help. Examples of the presen- zG EMPLOYMENT PSYCHOLOGY 5 6.00 5.72, 1922 1925 wa Fev., wear nr. 1923 1924 1925 fe Jl - Figure 32: Proportion of accidents Figure 33: Proportion of accidents per mile driven for first four months per mile driven for the first four of 1923, 1924, and 1025. months of the vear. Data from the Yellow Cab Company, Chicago Illinois (170). Psycho- logical tests of applicants for positions as driver were installed in January, 1925. The improved method of selection brought about a substantial decrease in accidents per mile driven. The exact figures are not given on the scale. Relative position only is indicated. tation of such findings are given in Figures 32, 33, and 34. Vigilance is the price of exact measurement in vocational selection. It is not enough that a carefully planned investi- gation is carried out by a trained man and that valid and economical measuring instruments for selection are produced and put to use. To assure to both workers and manage- ment the benefits of such scientific procedures, continued oversight by a competent industrial psychologist is essential. This volume has outlined the scientific methods which investigators, in touch with the practical demands of modern business and industry, have at their disposal for increasing the reliability of prediction of an applicant’s future com- petence and satisfaction in his work. Few will question that substantial improvements in the process of vocational adjustment which these scientific pro- cedures make possible are of both economic and social im- 220 THE EXAMINATIONS AT WORK portance. In factory, office, and store, among clerks, skilled tradesmen, salesmen, laborers, and minor supervisors, and in the executive ranks of large-scale enterprises, is there need of better bases of information about available men and women for use in allocating occupational opportunities. The personal problems of government also, in the civil services of municipality, state, and nation, call for the best that science can contribute. Public schools and institutions of higher education have analogous tasks in the selection and 100 80 | 60 40 2: or— _ ei —— 1914 19:0 1916 1917 1918 191. Sul Year Hired with i Figure 34: Reduction of turnover among salesmen for a life insurance com- pany by improvement in methods of hiring. The chart is read as follows: Of the salesmen hired in 1914, 40% remained with the firm in 1915, 18% in 1916, 10% in 1917, 6% in 1918, 5.5% in 1919, and 5% were still with the firm in 1920. The next descending curve gives the same information for those salesmen hired in 1915. The improved methods of selection were installed in 1916. The subsequent years brought a steady increase in the proportion of newly hired employees remaining a year or longer, as shown by the rise in the heavy lines. 231 ~ EMPLOYMENT PSYCHOLOGY development of their teaching and supervisory personnel. Even more obvious are the responsibilities of these insti- tutions in the selection and classification of their students according to aptitude and ability; in helping them most wisely to plan the successive steps of their education; and in placing them, on leaving school, in touch with those voca- tional opportunities in which there is the greatest likelihood of individual satisfaction and success. Each of these areas of educational and occupational adjustment offers its chal- lenge to men of scientific temper with an aptitude for per- sonnel research. Society will value whatever genuine contributions these investigators make to employment procedures, vocational guidance, and related aspects of personnel practice, whether in education or in industry. Conclusions wrought from the facts of occupational analysis, individual measurement and vocational experience, with such statistical tools and experi- mental techniques as have here been assembled, will add materially to the sum of sound principles and practical pro- cedures which together form the substance of employment psychology. The ingenuity, skill, and scientific rigor with which this pioneering is done will determine its real value in helping people find their places in what will be, for them and for their employers, the most satisfactory and reward- ing opportunities for work. 222 REFERENCES REFERENCES Tais list of references lays no claim to completeness. A compre- hensive bibliography on the topics covered in this manual would be in itself a monograph of some extent. References are given only to source-books of material and to works which illustrate the methods discussed in this manual. 1. Arrrort, F. H.,, AND Arrrort, G. W. “Personality Traits: Their Classification and Measurement”; Journal of Abnormal Psychol- ogy and Social Psychology, 1921, Xvi, 6-40. 2. ALLrorT, G. W. “Personality and Character”; Psychological Bulletin, 1921, XViil, 441-455. 3. AMAR, J. Le Moteur Humain (2nd Edition) ; Paris, 1923. 4. BaapE, LipmanN, AND STERN. “Fragment eines Psychographischen Schemas”; Zeitschrift fiir Angewandte Psychologie, 1910, iii, 10I- 215. 5. Barrarp, P. B. Mental Tests; London, 1920. 6. BaArLow, PETER. Barlow’s Tables of Squares, Cubes, Square Roots, Cube Roots, Reciprocals of All Integer Numbers up to 10,000; London and New York, 1912. 7. BENGE, E. J. Standard Practice in Personnel Work; New York, 1920. 8. Bezanson, A., CeALUFOUR, E., Wnirrts, J. H, Axo WHITE, L. “A Study in Labor Mobility”; Annals, 1922, ciii, 163-234. 9. Burs, M. A. “Methods for the Selection of Comptometer Operators and Stenographers”: Journal of Applied Psychology, 1021, v, 275- 233. 10. Bours, M. A. Standardizing the Selection of Clerical Workers; Ser- vice Bulletin, Bureau of Personnel Research, Carnegie Institute of Technology, Vol. 5, No. 9, May, 1923. 11. BinceAM, W. V. Measuring a Workman’s Skill; National Society for Vocational Education, Bulletin No. 30, 1910. 12. BincaaMm, W. V. “Personality and Vocation”; British Journal o} Psychology, 1926, xvi, 354-362. 13. BineaAM, W. V. “The Three Functions of the Interview in Employ- ment”; Management Review, 1926, Xv, 36-38. 14. Brackrorp, K. M. H.,, AND NeEwcomB, A. The Job, the Man, the Boss; New York, 1914. 15. BrooMriELD, D. Problems in Personnel Management; New York, 1923. 16. BoArbMAN, H. Psychological Tests—A Bibliography; New York, 1918. 17. BREWER, J. M., AND OTHERS. Mental Measurement in Educational and Vocational Guidance; Harvard Bulletins in Education, No. X, No- vember, 1924. 18. BrinGes, J. W. An Outline of Abnormal Psychology; Columbus, 1921. 19. Brrr, A. A. Fundamental Conceptions of Psychoanalysis; New York, 1Q21I. 228% 277 EMPLOYMENT PSYCHOLOGY 20. BrintoN, W. C. Graphic Methods for Presenting Facts; New York, 1914. 21. Brown, Wum., AND THOMSON, G. H. The Essentials of Mental Mea- surement; Cambridge, 1921. 22. Burt, C. Mental and Scholastic Tests; London, 1921. 23. Burtt, H. E. Principles of Employment Psychology; Boston, 1926. 24. Capy, V. M. “The Psychology and Pathology of Personality”; Journal of Delinquency, 1922, vii, 225-248. 25. CaADDOCK, R. E. Principles and Methods of Statistics; Boston, 1925. 26. CrapmaN, J. C. “Cumulative Correlation”; Journal of Experimental Psychology, 1922, v, 263-269. 27. CaaPMaN, J. C. “The Individual Injustice and Guessing in the True False Examination”; Journal of Applied Psychology, 1922, vi, 342-348. 28. CuapMaN, J. C. Trade Tests; New York, 1921. 29. Character Education Institution. Character Education Methods; The Iowa Plan, $20,000 Award, 1922, Chevy Chase, Washington, D. C. 30. CrarTERs, W. W. “The Collecting of Unrecorded Specifics”; Journal of Educational Research, 1922, Vv, 280-204. 31. CaarTERs, W. W. “A Technique for Trait Analysis”; Journal of Edu- cational Research, 1924, X, 95-100. 32. Cuarters, W. W., AND WHITLEY, I. B. Methods of Measuring Per- sonnel Effort; American Management Association, New York, 1924. 33. CuartErs, W. W., axp WHITLEY, I. B. Secretarial Duties and Traits; Baltimore, 1924. 34. CimBar, W. Taschenbuch zur Untersuchung Nerviser und Psychischer Krankheiten (2nd Edition) ; Berlin, 1913. 35. Cony, S. Commercial Tests and How to Use Them; Yonkers, 1910. 36. Cook, H. E., AND MaNsoN, G. E. “Abilities Necessary in Effective Re- tail Selling and a Method of Evaluating Them”; Journal of Per- sonnel Research, 1926, v, 74-82. 37. Coss, J. J., AND OUTHWAITE, L. Personnel Management—T opical Out- line and Bibliography; Washington, 1919. 38. Courts, S. A. “The Construction of Measuring Instruments in the Field of Education”; Scientific Monthly, 1925, xxi, 260-290. 39. Cowpery, K. M. “Measurement of Professional Attitudes: Differences between Lawyers, Physicians and Engineers”; Journal of Person- nel Research, 1926, v, 131-141. 40. CrATHORNE, A. R. “Calculation of the Correlation Ratio”; Journal of American Statistical Association, 1922, xviii, 394-396. 41. CreLLE, A. L. Rechentafeln; Berlin, 1891. 42. DAVENPORT, C. B. Statistical Methods (3rd Edition) ; New York, 191%. 43. Davenport, C. B. The Trait Book (2nd Edition); Cold Spring Har- bor, L. I., 1919. 44. Dewey, E., CHILD, E., AND RUML, B. Methods and Results of Test- ing School Children; New York, 1920. 45. Dopp, S. C. “A Correlation Machine”; Industrial Psychology, 1926, i, 46-58. 46. DopGe, R. (Quoted in Yerkes, R. M. “Report of the Psychology 26 REFERENCES Committee of the National Research Council” 3 Psychological Re- view, 1919, Xxvi, 83-149.) 47. DorErTY, M., MaAcLATCHY, J., AND BuckincEAM, B. R. Bibliography of Educational and Psychological Tests and Measurements; De- partment of the Interior, Bureau of Education, Bulletin No. 55, 1923. 48. Downey, J. E. The Will-Temperament and Its Testing; Yonkers, 1922, 49. Essential Personnel Records. Annual Convention Series, No. 27%, American Management Association, New York, 1925. 50. Federal Board for Vocational Education. Bibliography on Vocational Guidance, Bulletin 66; Washington, 1925. 51. FErvaLD, M. R., Haves, M. H. S.,, axp Dawrey, A. A Study of Women Delinquents in New York State; New York, 1920. 52. Foster, W. S. Experiments in Psychology; New York, 1923. 53. Foster, Wm. T. Administration of the College Curriculum; Boston, 1011. 54. Franz, S. I. Handbook of Mental Examination Methods; New York, 1010. 55. FREyD, M. “The Graphic Rating Scale”; Journal of Educational Psychology, 1923, xiv, 83-102. 56. FREyD, M. “Measurement in Vocational Selection—An Outline of Re- search Procedure”; Journal of Personnel Research, 1923, ii, 215- 249, 268-284, 377-385. 57. FrRevyp, M. “A Method for the Study of Vocational Interests”; Journal of Applied Psychology, 1922, Vi, 243-254. 58. FReyp, M. “The Personalities of the Socially and the Mechanically Inclined”; Psychological Monographs, No. 151, 1924. 58a. Freyp, M. “Selection of Promotion Salesmen”: Journal of Personnel Research, 1926, v, 142-156. 59. FRevp, M. “The Statistical Viewpoint in Vocational Selection”; Jour- nal of Applied Psychology, 1923, ix, 349-336. 60. GALLUP, G. H. “Traits of Successful Retail Salespeople; A Preliminary Study”; Journal of Personnel Research, 1926, iv, 474-482. 60a. GARFIEL, E. “Tests of Motor Ability”; Archives of Psychology, No. 62, 1923. 61. Giese, F. Handbuch Psychotechnischer Eignungspriifungen; Halle, 1925. 62. GILBRETH, F. J. Motion Study; New York, 1921. 63. Goring, C. The English Convict; London, 1910. 64. Grirrirrs, Cras. H. Fundamentals of Vocational Psychology; New York, 1924. 65. HaceerTY, M. E. “Recent Developments in Measuring Human Ca- pacities”; Journal of Educational Research, 1921, iii, 241-253. 66. HartsHORNE, H., AND May, M. A. “Testing the Knowledge of Right and Wrong”; Religious Education, February, 1926. 67. Haves, M. H. S., axp Paterson, D. G. “Experimental Development of the Graphic Rating Method”; Psychological Bulletin, 1921, xviii, 98-00. 237 Zz EMPLOYMENT PSYCHOLOGY 68. Heary, WM. The Individual Delinquent; Boston, 1915. 69. HEpNER, H. W. “Better Judgments of Men”; Industrial Psychology, 1926, i, 19-24. yo. HINES, M., AND O’CoNNOR, J. “A Measure of Finger Dexterity”; Journal of Personnel Research, 1926, iv, 379-382. vi. HorriNewortH, H. L. Judging Human Character; New York, 1922. #2. Horrineworte, H. L. Vocational Psychology, New York, 191%. »3. HoLLingworTH, H. L., AND POFFENBERGER, A. T. Applied Psychology; New York, 1923. v4. HOLZINGER, K. J. “A Combination Form for Calculating the Correla- tion Coefficient and Ratios”; Journal of American Statistical As- sociation, 1923, xviii, 623-627. #5. HorLziNGer, K. J. Statistical Tables for Students in Education and Psychology; Chicago, 1925. #6. Huey; K. “Problems Arising and Methods Used in Interviewing and Selecting Employees”; Annals, 1916, Ixv, 208-219. #7. HUFFARER, C. L. “A Contribution to the Technique of Partial Cor- relation”; Journal of Applied Psychology, 1923, vii, 135-142. »8. Huir, C. L. “An Automatic Correlation Calculating Machine”; Journal of American Statistical Association, 1925, XX, 522-531. #9. Hurt, C. L. “The Conversion of Test Series Into Series Which Shall Have Any Assigned Degree of Dispersion”; Journal of Applied Psychology, 1922, vi, 288-300. 80. Hype, WM. D. Self-Measurement; New York, 1908. 81. JomnsoN, O. R. “One Company’s Experience in the Selection of Salesmen”; Administration, 1923, Vv, 646-658. 82. Joint Committee on Standards for Graphic Presentation. Preliminary report published for the purpose of inviting suggestions for the benefit of the committee. Quarterly Publication of the American Statistical Association, 1914-1915, xiv, 790-707. (See also: Report of Committee on Standards for Graphic Presentation, Psychologi- cal Bulletin, 1915, xii, 432-440.) 83. Jones, D. C. First Course in Statistics; London, 1921. 84. Kerrey, T. L. Chart to Facilitate the Calculation of Partial Coeffi- cients of Correlation and Regression Equations; Stanford Univer- sity Publications, School of Education, Special Monograph No. 1, 1021. 8s. Kerrey, T. L. “The Reliability of Test Scores”; Journal of Educa- tional Research, 1921, iii, 370-3%9. 86. Krxriey, T. L. Statistical Method, New York, 1923. 87. Kerry, R. W. Hiring the Worker; New York, 1918. 88. KEmMBLE, W. F. Choosing Employees by Test; New York, 1917. 89. KEnaGy, H. G., axp YoaxuM, C. S. The Selection and Training of Salesmen; New York, 1925. go. Kent, G. H., AND Rosanorr, A. J. “A Study of Association in In- sanity”; American Journal of Insanity, 1910, Ixvii, Nos. 1 and 2. (See also Kent and Rosanoff: “Study of Association in Insanity”; Kings Park State Hospital Bulletin, New York, 1911, iv, 165-302.) o1. KEyNEs, J. M. A Treatise on Probability; London, 1921. 228 REFERENCES z 02. KmvesBURrY, F. A. “Grading the Office Job. II. Making the Analysis”; Administration, 1923, Vv, 303-401. 03. Kingssury, F. A. “Making Rating-Scales Work”; Journal of Person- nel Research, 1923, iv, 1-6. 04. Key, G. H. Guides for History Taking and Clinical Examination; Utica, 1921. 05. Krrson, H. D. Psychology of Vocational Adjustment; Philadelphia, 1025. 06. KnicaT, F. B. “The Effect of the ‘Acquaintance Factor’ Upon Per- sonal Judgments”; Journal of Educational Psychology, 1023, xiv, 120-142. 07. KorvmAUsEr, A. W. “The Psychology of Vocational Selection”; Psychological Bulletin, 1922, xix, 192-229. 98. KorNHAUSER, A. W., AND KinGsBURY, F. A. Psychological Tests in Business; Chicago, 1924. 09. Lar, D. A. The Psychology of Selecting Men; New York, 1925. 100. LaNGFELD, H. S., AND Arrrort, F. H. An Elementary Laboratory Course in Psychology; Boston, 1916. 101. Linx, H. C. Employment Psychology; New York, 1919. 102. LrpmanN, O. “Bibliographie zur Psychologischen Berufsberatung, Berufseignungsforschung und Berufskunde”; Schriften sur Psy- chologie der Berufseignung und des Wirtschaftslebens, 1922, xx, 1-49. 103. LrpmanN, O. Handbuch Psychologischer Hilfsmittel der Psychia- trischen Diagnostik; Leipzig, 1921. 104. McCarr, Wm. A. How to Measure in Education; New York, 1922. 105. Manson, G. E. “Bibliography on Psychological Tests and Other Ob- jective Measures in Industrial Personnel”; Journal of Personnel Research, 1925, iv, 301-328. 106. Manson, G. E. “Personality Differences in Intelligence Test Perform- ance”; Journal of Applied Psychology, 1925, ix, 230-255. 106a. MANSON, G. E. Trait Bibliography; National Research Council, 1926. 107. MarsEALL, WM. C. Graphical Methods; New York, 1921. 108. MARsTON, L. R. “The Emotions of Young Children—An Experimental Study in Introversion and Extroversion”; University of Iowa, Studies in Child Welfare, Vol. 3, No. 3, June 15, 1925. 109. May, M. A,, AND HArTSHORNE, H. “Objective Methods of Measuring Character”; Pedagogical Seminary, 1925, xxxii, 45-67. 110. Measuring and Grading the Supervisory Forces; Annual Convention Series, No. 26, American Management Association, New York, 1925. 111. MENE, F. “Job Analysis for Employment Purposes”; Annals, 1923, CX, 22-31. 112. MEINE, F. Job Specifications; Federal Board for Vocational Educa- tion, Bulletin No. 45, Washington, 1919. 113. MEYER, A. Outlines of Examinations; Bloomingdale Hospital Press, New York, 1918. 114. Mmrer, G. F. “Formulas for Scoring Tests in Which the Maximum Amount of Chance is Determined”; Journal of Educational Psy- chologv, 1925, Xvi, 304-315. 7 20 3 EMPLOYMENT PSYCHOLOGY 115. MINER, J. B. “An Aid to the Analysis of Vocational Interests”; Journal of Educational Research, 1922, v, 311-323. 116. MrrcrELL, D., AND RUGER, G. J. Psychological Tests, Revised and Classified Bibliography; New York, 1918. 117. MonroE, W. S. Measuring the Results of Teaching; Boston, 1918. 118. Moore, B. V. “Personnel Selection of Graduate Engineers”; Psychol- ogical Monographs, No. 138, 1921. 119. Moore, H. T., AND GrrrLiLanNDp, A. R. “The Measurement of Aggres- siveness”; Journal of Applied Psychology, 1921, Vv, 97-118. 120. MoreaN, E. B. Placing and Introducing Employees; Annual Conven- tion Series, No. 18, American Management Association, New York, 1925. 121. Muscio, B. Vocational Guidance—A Review of the Literature; Lon- don, 1921. 122. Myers, C. S. Textbook of Experimental Psychology (2 vols. 2nd Edition) ; London, 1911. 123. NATIONAL SOCIETY FOR THE STUDY OF EpucaTtioN. Twenty-first Vear- book: Intelligence Tests and Their Use; Bloomington, Illinois, 1922. 124. O'ROURKE, L. J. “Saving Dollars and Energy by Personnel Research: An Investigation in the Interest of the Postal Service”; Journal of Personnel Research, 1926, iv, 351-364, 433-450. 125. Otis, A. S. Statistical Method in Educational Measurement; Yonkers, 1925. 126. Parsons, F. Choosing a Vocation; Boston, 1909. 127. PATRIDGE, G. E. An Outline of Individual Study; New York, 1910. 128. PaTERSON, D. G. “Methods of Rating Human Qualities”; Annals, 1923, CX, 81-903. 129. PATERSON, D. G. Preparation and Use of New-Type Examinations; Yonkers, 1925. 130. PATERSON, D. G. “The Scott Company Graphic Rating Scale”; Jour- nal of Personnel Research, 1922, i, 361-376. 131. PATERSON, D. G., AND LANGLIE, T. A. “Empirical Data on the Scor- ing of True-False Tests”; Journal of Applied Psychology, 1925, ix, 339-348. 132. PatTEN, E. F. “An Experiment in Testing Engine Lathe Aptitude’; Journal of Applied Psychology, 1923, vii, 16-29. 133. PAYNE, A. F. The Relation of Vocational and Educational Guidance to Vocational Education; National Vocational Guidance Associa- tion Bulletin, 1923, i, 103-107. 134. PEARSON, K. Grammar of Science; London, 1900. 135. PEARSON, K. Tables for Statisticians and Biometricians; Cambridge, 1914. 136. PERRIN, F. A. C. “Physical Attractiveness and Repulsiveness”; Jour- nal of Experimental Psychology, 1921, iv, 203-217. 137. Personnel System of the United States Army (2 vols.) ; Washington, 1919. 138. Personnel Terminology; Report of Committee on Management Term- inology, Special Paper No. 10, American Management Associa- tion, New York, 1925. 240 REFERENCES z . 139. PINTNER, R., AND PATERSON, D. G. A Scale of Performance Tests; New York, 1917. 140. PLANT, J. S. “Rating Scheme for Conduct”; American Journal of Psychiatry, 1922, i, 547-572. 141. PRESSEY, S. L., AND Pressey, L. C. Introduction to the Use of Stand- ard Tests; Yonkers, 1922. 142. Printing—Hand Composition; United States Army, Educational Man- ual No. 18, Education and Recreation, Special School, Camp Grant, Illinois, 1920. 143. Psychological Tests of Educable Capacity; Report of the Consultative Committee of the Board of Education, London, 1924. 144. Rarer, G. G., AND ALLEN, WM. H. Record Aids in College Manage- ment; Institute for Public Service, New York, 1916. 145. Ream, M. J. “A Statistical Method for Incomplete Order of Merit Ratings”; Journal of Applied Psychology, 1921, v, 261-266. 146. REEDER, W. G. How to Write a Thesis; Bloomington, Illinois, 1923. 147. Renry, P. J. The Selection and Placement of Employees; Federal Board for Vocational Education, Bulletin No. 49, Washington, I9IQ. 148. Rierz, H. L. (Editor.) Handbook of Mathematical Statistics; Bos- ton, 1924. 149. RiGGLEMAN, J. R. Graphic Methods for Presenting Business Statistics; New York, 19026. 150. Roget’s International Thesaurus of English Words and Phrases; New York, 1923. 151. Rosenow, C. “The Analysis of Mental Functions”; Psychological Monographs, No. 106, 1917. 152. Rosexow, C. “Predicting Academic Achievement, a Study in Prob- ability”; Pedagogical Seminary, 1925, xxxii, 628-636. 153. Ross;, Wm. H., AND Rossi, D. I. P. Personnel Administration: A Bibliography; Baltimore, 1925. 154. Ruch, G. M. The Improvement of Written Examinations; New York, 1924. 155. RucH, G. M., AND StopDARD, G. D. “Comparative Reliabilities of Five Types of Objective Examinations”; Journal of Educational Psy- chology, 1925, xvi, 89-103. 156. Rue, H. O. “Is the Rating of Human Character Practicable?”; Journal of Educational Psychology, 1921, xii, 425-438, 485-501; 1922, xiii, 30-42, 81-93. 157. Rucec, H. O. Statistical Methods Applied to Education; Boston, 1917. 158. Rumi, B. “Coefficients of Diagnostic Value”; Journal of Philosophy, 1917, Xiv, 633-637. 159. Rumi, B. “Measurement of the Efficiency of Mental Tests”; Psy- chological Review, 1916, xxiii, 501-507. 160. Rumi, B. “The Reliability of Mental Tests in the Division of an Academic Group”; Psychological Monographs, No. 105, 1917. 161. Sanrorp, E. C. Course in Experimental Psychology; Boston, 1898. 162. SCHNEIDER, H. Selecting Young Men for Particular Jobs; National Association of Corporation Schools, Bulletin No. 7. 72471 3 EMPLOYMENT PSYCHOLOGY 163. SCHULZE, R. Experimental Psychology and Pedagogy; Translated by R. Pintner, London, 1912. 164. Scott Company Laboratory. “Tables to Facilitate the Computation of Coefficients of Correlation by the Rank Difference Method”; Journal of Applied Psychology, 1920, iv, 115-125. 165. Scott, W. D. “A Fourth Method of Checking Results in Vocational Selection”; Journal of Applied Psychology, 191%, i, 61-66. 166. Scott, W. D., AND CLOTHIER, R. C. Personnel Management; Chicago, 1923. 167. SCrRiPTURE, E. W. Thinking, Feeling, Doing; New York, 1803. 168. SHEFFERMAN, N. W. Employment Methods; New York, 1920. 169. SHEN, E. “The Influence of Friendship Upon Personal Ratings”; Journal of Applied Psychology, 1923, ix, 66-68. 170. SNow, A. J. “Tests for Chauffeurs”; Industrial Psychology, 1926, i, 30-45. 171. STERN, E. “Pathopsychographische Untersuchungen”; Archiv fiir Psychiatrie und Nervenkrankheiten, 1920, Ixi, 328-384. 172. STERN, WM. Differentielle Psychologie (3rd Edition); Leipzig, 1921. 173. STERN, WM. Die Intelligenz der Kinder und Jugendlichen und die Methoden ihrer Untersuchung; Leipzig, 1920. 174. STERN, WM., AND WIEGMANN, O. Methodensammlung zur Intelligenz- priifung von Kindern und Jugendlichen (3rd Edition); Leipzig, 1926. ~ 175. SToTT, M. BOOLE. Report on the Present Position of Vocational Guid- ance and Vocational Selection; London, 1924. 176. STRONG, E. K., JR, AND UHRBROCK, R. S. Job Analysis and the Cur- riculum; Baltimore, 1923. 177. STURGES, H. A. “Summary and Group Correlation”; Quarterly Pub- lication of American Statistical Association, 1921, XVii, 778. 178. Symonps, P. M. “Job-Analysis Sheet for Computing Partial and Multiple Coefficients of Correlation and Regression Coefficients”; Teachers College Record, 1925, xxvii, 52-69. 179. Symonps, P. M. “On the Loss of Reliability in Ratings Due to Coarseness of the Scale”; Journal of Experimental Psychology, 1924, Vil, 456-461. 180. Symons, P. M. “The Present Status of Character Measurement”; Journal of Educational Psychology, 1924, Xv, 484-498. 181. Tace, M. ‘Vocational Tests in the Engineering Trade”; Journal of National Institute of Industrial Psychology, 1925, ii, 313-323. 182. TEAD, O., AND GREGG, R. B. Outline of Job Analysis; New York, 1918. 183. TeAD, O., AND METCALF, H. C. Personnel Administration; New York, 1920. 184. TERMAN, I. M. Materials for the Study of Gifted Children; Stanford University. 185. TERMAN, L. M. Measurement of Intelligence; Boston, 1916. 186. TErMaAN, L. M. “The Mental Test as a Psychological Method”; Psychological Review, 1924, Xxxi, 93-117. 187. TErRMAN, L. M., AND Cowpery, K. M. “Stanford’s Program of Uni- versity Personnel Research”; Journal of Personnel Research, 1925, iv, 623-627. 242 REFERENCES a0 188. THOMPSON, J. D. Personnel Research Agencies; Bulletin No. 299, U. S. Bureau of Labor Statistics, Washington, 1921. 189. THORNDIKE, E. L. “A Constant Error in Psychological Rating”; Journal of Applied Psychology, 1920, iv, 25-29. 190. THORNDIKE, E. L. “Measuring Human Intelligence”; Harper's Maga- zine, 1920, cxl, 227-235. 191. THORNDIKE, E. L. Mental and Social Measurements (2nd Edition); New York, 1913. 192. THORNDIKE, E. L. “The Technique of Combining Incomplete Judg- ments of the Relative Positions of # Facts Made by n Judges”; Journal of Philosophy, 1916, xiii, 197-204. 193. THORNDIKE, E. L. Measurement of Intelligence; Garrison, New York, 1926. 104. THURSTONE, L. L. “The Calculation and Interpretation of Percentile Ranks”; Journal of Educational Research, 1922, vi, 225-235. 195. TEURSTONE, L. L. “A Data Sheet for the Pearson Correlation Co- efficient”; Journal of Educational Research, 1922, vi, 49-56. 196. THURSTONE, L. L. The Fundamentals of Statistics; New York, 1925. 197. TEURSTONE, L. L. “The Learning Curve Equation”; Psychological Monographs, No. 114, 1919. 198. THURSTONE, L. L. “The Principles of Vocational Guidance”; British Journal of Psychology, 1924, xiv, 353-361. 199. THURSTONE, L. L. “A Scoring Method for Mental Tests”; Psychologi- cal Bulletin, 1919, Xvi, 235-240. 200. TrrcHENER, E. B. Experimental Psychology (2 vols., 4 parts); New York, 19053. 201. Toors, H. A. “Eliminating the Pit-Falls in Solving Correlation: A Printed Correlation Form”; Journal of Experimental Psychology, 1921, iv, 434-447. 202. Toors, H. A. Trade Tests in Education; New York, 1921. 203. Toors, H. A. “Two Devices for Aiding Calculation”; Journal of Ex- perimental Psychology, 1926, ix, 60-66. 203a. Toors, H. A. “Validating the Questionnaire Method”; Journal of Personnel Research, 1923, ii, 153-1609. 204. Toops, H. A., AND MINER, Z. F. “A Serviceable P.E.r Table”; Journal of Educational Research, 1924, ix, 63-68. 205. TrABUE, M. R., AND STOCKBRIDGE, F. P. Measure Your Mind; Garden City, New York, 1920. 206. TRELEASE, S. F., AND YuLk, E. S. Preparation of Scientific and Tech- 2 nical Papers; Baltimore, 1923. 207. UrrBrOCK, R. S. The Psychological Aspect of Job Analysis; Annual Convention Series, No. 17, American Management Association, New York, 1925. 208. ViteLes, M. S. “The Clinical Viewpoint in Vocational Selection”; Journal of Applied Psychology, 1925, ix, 131-138. 209. VIreLes, M. S. “Job Specifications and Diagnostic Tests of Job Com- petency Designed for the Auditing Division of a Street Railway Company”; Experimental Studies in Psychology and Pedagogy, No. 9, Philadelphia, 1922. 210. ViteLes, M. S. “Standards of Accomplishment: Criteria of Voca- + A.2 5 EMPLOYMENT PSYCHOLOGY " tional Selection”; Journal of Personnel Research, 1926, iv, 483-486. 211. Watson, G. B. “The Measurement of the Less Tangible Qualities”; Vocational Guidance Magazine, 1926, iv, 281-289. 2110. WATSON, J. B. Psychology from the Standpoint of a Behaviorist (2nd Edition) ; Philadelphia, 1924. 212. Warts, F. “The Construction of Tests for the Discovery of Voca- tional Fitness”; Journal of Applied Psychology, 1921, v, 240-252. 213. WEBB, E. “Character and Intelligence”; British Journal of Psychology Monograph Supplements, i, No. 3, 1915. 214. WECHSLER, D. “Tests for Taxicab Drivers”; Journal of Personnel Research, 1926, v, 24-30. 215. WELL, G. Stillings Pseudo-Isochromatische Tafeln zur Priifung des Farbensinnes; Leipzig, 1918. 216. WELLS, F. L. Mental Adjustments; New York, 1914. 217. WeELLs, F. L. “The Systematic Observation of the Personality”; Psychological Review, 1914, XxXi, 205-333. 218. WEMBRIDGE, H. A. “Experiment and Statistics in the Selection of Employees”; Quarterly Publication of American Statistical Asso- ciatien, 1923, xviii, 600-606. 219. West, C. J. Introduction to Mathematical Statistics; Columbus, 1918. 220. WaIPPLE, G. M. Manual of Mental and Physical Tests (2 vols., 3rd Edition) ; Baltimore, 1914. 221. Wiper, H. H. 4 Laboratory Manual of Anthropometry; Philadel- phia, 1920. 222. Woop, B. D. Measurement in Higher Education; New York, 1923. 223. Woop, B. D. “Studies of Achievement Tests”; Journal of Educa- tional Psychology, 1926, xvii, 1-22. 224. WoopworTH, R. S. “Combining the Results of Several Tests”; Psy- chological Review, 1912, XiX, 97-123. 225. WoopworTH, R. S., AND WELLS, F. L. “Association Tests”; Psychologi- cal Monographs, No. 5%, 1911. 226. World Book Company. Bibliography of Tests for Use in Schools. 227. YERKES, R. M,, AND LARUE, D. W. Outline of a Study of the Self; Cambridge, Mass., 1913. 228. YERKES, R. M. (Editor). Psychological Examining in the United States Army; Memoirs of the National Academy of Science, 1921, XV. 229. Yoakum, C. S. “Basic Experiments in Vocational Guidance”; Jour- nal of Personnel Research, 1922, i, 18-34. 230. YoaruM, C. S. Present Limitations of Psychological Tests in Man- agement; Annual Convention Series, No. 21, American Manage- ment Association, New York, 1925. 231. YoaxuM, C. S., AND YERKES, R. M. Army Mental Tests; New York, 1920. 232. YULE, G. U. The Function of Statistical Method in Scientific Investi- gation; Industrial Fatigue Research Board, Report No. 28, Lon- don, 1924. 233. YULE, G. U. Introduction to the Theory of Statistics; London, 1917. 244 REFERENCES JOURNALS 234. Administration; New York. 235. American Journal of Psychiatry; Baltimore. 236. American Journal of Psychology; Worcester, Mass. 237. Annales de I'Institut d’Orientacié Professional; Barcelona. 238. Annals of the American Academy of Political ard Social Science; - Concord, N. H. 239. Archives of Psychology; New York. 240. British Journal of Psychology; Cambridge. 241. Industrial Psychology; Hamilton, N. Y. 242. Industrielle Psychotechnik; Berlin. 243. Journal of the American Statistical Association; Concord, N. H. 244. Journal of Applied Psychology; Baltimore. 245. Journal of Delinquency; Whittier, California. 246. Journal of Educational Psychology; Baltimore. 247. Journal of Educational Research; Bloomington, III. 248. Journal of Experimental Psychology; Princeton. 249. Journal of the National Institute of Industrial Psychology; London. 250. Journal of Personnel Research; Baltimore. 251. Journal of Philosophy; New York. 252. Management Review; New York. 253. L'Orientation Professionnelle; Paris. 254. Pedagogical Seminary; Worcester, Mass. 255. Praktische Psychologie; Leipzig. (Discontinued) 256. Psychological Bulletin; Princeton. 257. Psychological Clinic; Philadelphia. 258. Psychological Monographs; Princeton. 259. Psychotechnische Zeitschrift; Munich. 260. Public Personnel Studies; Washington. 261. Seariieen zur Psychologie der Berufseignung und des Wirtschaftslebens; eipzig. 262. Scientific Monthly; Garrison, N. Y. 263. Teachers College Record; New York. 264. Vocational Guidance Magazine; Cambridge, Mass. 265. Zeitschrift fiir Angewandte Psychologie; Leipzig. 245 TABLES TABLE 1 VALUES oF p X gq AND z CORRESPONDING TO VALUES OF (p + ¢ = 1.00) The z values were obtained from the Kelley-Wood tables, where they are carried out to six decimal places and are given for val- ues of # to the third decimal place. * 7: Hh Xq z .00 or 1.00 .0000 .00C0 .0I 99 .0099 0267 .02 98 .0106 .0484 .03 07 .0201 .0680 04 .06 .0384 .0862 .05 95 .0475 .I031 .06 04 .0564 .1101I .07 03 00651 1343 .08 .02 0736 1487 .09 OI 0819 1624 .1I0 .90 .0900 .I755 WII 89 0979 .1880 12 .88 .1056 .2000 .I3 .87 «T131 2115 14 .86 .1204 .2226 +13 .85 1275 .2332 I6 84 1344 +2433 «17 83 I411 +2531 .I8 .82 .1476 .2624 .19 81 .I539 2714 .20 .80 1600 .2800 .2X «79 16590 .2882 22 78 1716 .2061 .23 17 JX771 .3036 24 76 1824 .3100 +25 «25 1875 .3178 26 J4 .1024 3244 “27 73 J971 .3300 .28 2 .2016 .3366 .29 J .2059 .3423 .30 .70 .2100 3477 «31 .60 .2130 .3528 w32 .68 .2176 .3576 "33 67 .2211 3621 .34 .66 2244 .3664 -35 .65 .2275 3704 .36 .64 .2304 3741 37 63 2331 .3776 .38 62 .2356 .3808 .39 61 2379 .3837 +40 .60 .2400 .3863 41 590 .2410 .3887 .42 .58 .2436 .3909 43 S57 .2451 .3928 44 .56 .2464 3944 45 55 -2475 .3958 46 .54 2484 .3969 47 53 +2401 .3078 48 .52 2496 .3084 49 51 2499 .3988 .50 .50 \2500 .3080 ————————————— eer Oe ———————— *Kelley, T. L., Statistical Method, The Macmillan Company, 1923. Reprinted by per- mission. See pp. 172-173, 182-183. 240 250 EMPLOYMENT PSYCHOLOGY TABLE 2 VALUES OF FUNCTIONS OF 7 r | r2 | 1—r | Vr |vi=|v rE r | r? | 1 —r2 | 7 Tile 1—7r2 .00| .0000| 1.0000 .0000| I.0000 1.0000] .50] .2500| .7500| .707I 071 8660 01] .000I] .9999; .I000| .9950| .9999, .5I| .2601] .7399| .7I4I .7000. .8602 .02: .0004| .9996; .1414| .9899] .0008 .52| .2704} .7296| .721I| .6928 .8542 .03! .0000] .0001| .1732| .9849| .099s| .53| .2809| .7101| .7280| .6856| .8480 .04 ‘0x0 .9084| .2000] .9798 900d] .54| .2016] .7084| .7348| .6782| .8417 .05| .00251 .9975| .2236] .9747] .9987 .55] .3025| .6975| .7416| .6708! .8352 .06| .0036i 0964] 2449 0605] .0082] .56 3136] .6864 7483) .6633| .8285 .07| .0040| .00s51! .2646| .9644| .997s .57| .3249! .675I| .7550| .6557 .8216 .08| .0064| .0036, .2828] .0502| .9968 .58| .3364 .6636| .7616] .6481 .8146 .09| .0081| .9919; .3000| .9539| .9959 .59| .348I 6519] .7681 .6403] .807a .10| .o100l .9900 .3162| .0487| .0950 .60| .3600. .6400| .7746| .6325| .8000 .11| .o121| .9879| .3317| .0434| .0039| .61| .3721 .62790| .7810] .6245] .7924 .12| .o144| .0856| .3464| .0381| .0028| .62| .3844 .6156| .7874| .6164) .7846 .13| .0160| .0831| .3606| .0327| .001s| .63] .3960 .603If .7937| .6083 .7766 .14| .0106| .0804! .3742| .9274] .9902| .64] .4006 .5904| .8000| .6000, .7684 15! .02251 .0775 .3873] .0220| .9887l .6s5l .4225/ .5775] .8062 15016) .7500 16] .0256] .9744| .4000 9165] 0871 66, .4356] .5644| .8124| .5831| .7513 .17| .0280| .9711| .4123| .9r10' .0854ff .67. .4489, .551I 8185| .5745 .7424 .18| .0324| .0676, .4243| .9055 0837 .68 .4624 .5376| .8246 -5657, 7332 .19| .036I| .0639' .4359| .0000 .9818| .60, .4761 .5239| .8307| .5568 .7238 .20| .0400' .9600 .4472| .8944: .9798 70] .4900: .s5100| .8367| .5477' .714I 21 0441] 05501 .4583| ..8888 .9777| .7I, .5041l .4059 .8426| .5385] .7042 .22| .0484! .9516 .4600| .8832 .9755|| .72' .5184, .4816| .8485| .5202 .6940 .23] .0520] .0471 .4796| .8775 .9732| .73 .5329; .4671| .8544| .5106 .6834 .24] .0576i «9424 .4899! .8718 .9708 74 samo as24 .8602 -5099, .6726 2a] coir .0375' .5000| .8660' .0682|| .75! .5625| .4375] .8660| .5000 .6614 «20 0676! .9324| .5099! .8602' .9656| .76] .5776| .4224] 8718 .48990! .6499 27] 0720" .0271| .5106' .8544° .0620| .77| .5020, .4071| .8775| .4796! .6380 .28| .0784 .0216| .5202 .8485 .9600| .78, .6084, .3016| .8832] .4690 .6258 .290| .0841 .0159| .5385 -8426 .0570 79 6241 .3759| .8888 JuB3| oust .30| .0900 AE .8367! .0339 .80| .6400 .3600| .8944| .4472! .6000 31| 0061 .0039| .5568 .8307| .os07|| .81] .6561l .3430| 9000 A530l 5304 .32! .1024 .8976 3657] .8246| .o474| .82| .6724| .3276] .9055| .4243| .5724 +33 1o5g) Toralaoas .8185| .0440| .83] .6889 .311I .9IIO0| .4123. .5578 .34| .1156!1 .8844 .583I Brag) .0404| .84] .7056 .2044| .9165 -4000 5426 .35l .12251 .8775| .5016¢ .8062! .9367| .85| .7225 .2775| .9220] .3873. .5268 .36] .1206] .8704| .6000 .8000! .9330 -86/ 7396] .2604] -9274 .3742| .5103 .37| .1360| .8631| .6083 .7037] .0200| .87' .7569, .2431| .9327| .30006, .4931 .38| .1444| .8556| .6164 .7874 .9250| .88; .7744] .2256] .9381 3464] 4750 .39| .1521| .8479| .6245 £5) .9208 59] 7021 .2079| -9434| -33171 .4560 40] .1600l .8400| .6325 .7746{ .o16s| .0o0l .8100' .1900| .9487| .31621 .4359 .41| 1681 T0008 7681] .or21f .o1r .8281| .1719| .9539| .3000| .4146 .42| .1764| .8236 os 7616] .0075l .902 .8464| .1536] .9592| .2828] .3919 .43| .1840| .8151 .6557| .7550| .0028| .03, .8649| .1351| .9044| .2646) .3676 .44| .1036| .8064 6633) .7483| .8080| .04! R30 10) .9695| .2449| .3412 45] .20251 7975. .6708! .7416 .8930 05] 90251 .0975| .9747| .2236' .3122 .46 2116} .7884 .6782 7348] .8879 -96| .9216| .0784 -9798| .2000{ .2800 .47| .2200 .7791, .6856| .7280| .8827] .07! .0400 .059T| 0849] .1732 .243I 48] .2304 7606) .6928 7211 .8773ll 98 .09604 .0396 .0899| .1414 .I990 .49| .240I .7599; .7000 .7I41 8717 99 .980I .0IQ9Q Doel ooo LI41X .50| .z2500l 7300] 7071] \7071, .8660 1.00| 1.0000; .0000 I.0000 -0000| -0000 TABLES TABLE 3 SQUARES OF DIFFERENCES From o to 80 by Halves (See pages 178-179) Square of Square of Square of Square of Difference Difference Difference Difference Difference Difference Difference Difference 0 0 20 400.00 40 1600.00 60 3600.00 0.5 0.25 20.5 420.25 40.5 1640.25 60.5 3660. 25 I 1.00 21 441.00 41 1681.00 61 3721.00 1.5 2.25 21.5 462.25 41.5 1722.25 61.5 3782.25 2 4.00 22 484.00 42 1764.00 62 3844.00 2.5 6.25 22.5 500.25 42.5 1806.25 62.5 3906.25 3 9.00 23 529.00 43 1849.00 63 3969.00 3.5 12.25 23.3 552.25 43.5 1892.25 63.5 4032.25 3 16.00 24 576.00 44 1036.00 64 4096.00 3.5 20.25 24.5 600.25 44.5 1980.25 64.5 4160.25 3 25.00 25 625.00 45 2025.00 65 4225.00 5.5 30.25 25.5 650.25 45.5 2070.25 65.5 4290.25 . 36.00 26 676.00 46 2116.00 66 4356.00 t.5 42.25 26.5 702.25 46.5 2162.25 66.5 4422.25 2 49.00 27 729.00 47 2209.00 67 4489.00 7:5 56.25 27.5 756.25 47.5 2256.25 67.5 4556.25 8 64.00 28 784.00 48 2304.00 68 4624.00 8.5 72.25 28.5 B12.25 48.5 2352.25 68.5 4602.25 Gu 81.00 29 841.00 49 2401.00 69 4761.00 9.5 00.25 20.5 870.25 49.5 2450.25 69.5 4830.25 10 100.00 30 000.00 50 2500.00 70 4900.00 10.5 110.25 30.5 030.25 50.5 2550.25 70.5 4970.25 Ix 121.00 3I 961.00 51 2601.00 71 5041.00 11.5 132.25 31.5 092.25 51.5 2652.25 21.5 S1X2.25 [2 144.00 32 1024.00 52 2704.00 73 5184.00 [2.5 156.25 32.5 1056.25 52.5 2756.25 12.8 5250.25 3 169.00 33 1089.00 53 2809.00 73 5320.00 13.5 182.25 33.5 1122.25 53.5 2862.25 73.5 5402.25 14 196.00 34 1156.00 54 2016.00 74 5476.00 14.5 210.25 34.5 1190.25 54.5 2070.25 74.5 5550.25 Is 225.00 35 [225.00 55 3025.00 75 5625.00 15.5 240.25 35.5 1260.25 55.5 3080.25 75.5 5700.25 16 256.00 36 1296.00 56 3136.00 76 5776.00 16.5 272.25 36.5 1332.25 56.5 3192.25 76.5 5852.25 x7 280.00 37 1369.00 57 3249.00 77 5020.00 17.5 306.25 37.5 1406.25 57.5 3306.25 77.5 6006.25 8 324.00 38 1444.00 58 3364.00 78 6084.00 18.5 342.25 38.5 1482.25 58.5 3422.25 78.5 6162.25 9 361.00 39 1521.00 50 3481.00 79 6241.00 19.5 380.25 39.5 1560.25 59.5 3540.25 79.5 6320.25 Decimals may be entirely disregarded without appreciably affecting the resulting correla- tion coefficients. The error is never greater than .02, and is usually much less. 251 EMPLOYMENT PSYCHOLOGY TABLE 4 To FACILITATE THE COMPUTATION OF p (See pages 178-179) NuMBER oF CASES RANKED Lig nel Cy 13 14 15 z6 Ty 18 ¥9. [i301 pr I.00 00 00 00 00 00 00 00 00 00 00 I.00 .98 4 6 7 9 II 14 16 19 23 27 .08 .96 9 II 15 18 22 27 33 39 46 53 .96 -94 13 17 22 27 34 41 49 58 68 80 .94 .92 18 23 29 36 45 54 65 78 oI 106 .02 .90 22 29 36 46 56 68 82 97 114 133 .00 .88 26 34 44 55 67 82 908 116 137 160 .88 .86 31 40 51 64 78 05 114 136 160 186 .86 .84 35 46 58 23 00 109 I31 155 182 213 .84 .82 40 51 66 82 10I 122 147 174 205 239 .82 .80 44 57 73 oI I12 136 163 104 228 266 .80 .78 48 63 80 100 123 150 180 213 251 203 .78 .76 53 69 87 109 134 163 106 233 274 319 .76 74 57 74 95 118 146 177 212 252 206 346 .74 12 62 80 102 127 157 190 228 271 319 372 “72 70 66 86 109 137 168 204 245 201 342 399 .70 .68 70 92 116 146 170 218 201 310 365 426 .68 .66 75 97 124 155 190 231 277 320 388 452 .66 .64 79 103 31 164 202 245 294 349 410 479 .64 02 84 109 I38« 173 213 258 310 368 433 505 .62 .60 88 114 146 182 224 272 326 388 456 532 .6o .58 02 120 153 101 235 286 343 407 479 550. .53 .56 07 126 160 200 246 200 359 426 502 585 .56 V54 I0I 132 167 200 258 313 375 446 524 612 .54 .52 106 137 175 218 269 326 392 465 547 638 .52 .50 I10 143 182 228 280 340 408 485 570 665 .50 .48 114 149 189 237 201 33442488504 593 692 .48 .46 119 154 197 246 302 367 441 523 616 7388 | .406 .44 123 160 204 255 314 381 457 543 638 7450.44 =42 128 166 2II 264 325 394 473 562 661 77% .42 .40 132 172 218 273 336 408 490 581 684 708 .40 .38 136 177 226 282 347 422 506 601 707 825 .33 .36 141 183 233 201 358 435 522 620 730 851 .36 -34 145 189 240 300 370 449 539 640 752 878 .34 .32 150 E04 i248 4 VN 300 i 381 462 555 659 775 004 .32 .30 154 200 255 319 392 476 571 678 798 931 .30 .28 158 206 262 328 403 490 588 608 821 958 .28 .26 163 212 269 337 414 503 604 ”17 844 984 .26 .24 167 217 277 346 426 517 620 736 866 101 .24 .22 I72 223 284 355 437 530 636 756 880 1037 .22 .20 176 220 201 364 448 544 653 775 012 1064 .20 .18 180 235 298 373 459 558 669 795 0354 100X818 6 185 240 306 382 470 571 685 814 058% ‘T117 .16 .I4 189 246 313 391 482 585 702 833 080 1144 .14 . 12 194 252 320 400 493 598 718 853 1003 II70 i .I0 198 257 328 410 504 612 734 872 10206 " 1107 .I0 .08 202 263 335 419 515 626 751 891 1040 11224 .08 .06 207 269 342 428 526 639 767 911 1072 1250 .06 “0a TT Rats 340 i437 85381 653 983 J 03081004 | 1277.04 .02 216 280 357 446 549 666 800 050 N1I17 1303 .02 .00 220 286 364 455 560 680 816 069 1140 1330 .00 —1I1.00 440 572 728 010 II20 1360 1632 1038 2280 2660 —I1.00 252 TABLES TABLE 4 (Continued) NUMBER OF CASES RANKED pP 21 22 23 24 25 26 27 28 20 30 pP 1.00 00 00 00 00 00 00 00 00 00 00 I.09 .98 31 35 40 46 52 59 66 73 81 00 .98 .96 62 71 81 92 104 117 131 146 162 180 .96 .94 92 106 121 138 i156 176 107 210 244 270 .94 .02 123 5.142 162 184 208 234 262 202 3235 360 .92 .00 X54. 117 202 230 260 203 328 363 406 450 .00 .88 185 213 243 276 312 351 3903 438 487 539 .88 .86 216 248 283 322 364 410 459 512 568 629 .86 .84 246 283 324 368 416 468 524 585 650 719 .84 .82 277 + 319 364 414 468 527 590 658 731 809 .82 .80 308 354 405 460 520 585 - 653 731 812 800 .80 .78 339 390 445 506 572 644 721 804 803 980 .78 .76 370 425 486 552 624 702 786 877 974 1079 .76 4 400 460 526 508 676 761 852 050 1056 1160 .74 12 431 496 567 644 728 819 ol17 1023 ''3137 1259 12 .70 462 531 607 690 780 878 083 1006 1218 1340 .70 .68 493 567 648 736 832 036 1048 1160 1200 1438 .68 .66 524 602 688 782 884 005 III4 1242 1380 1528 .66 .64 554 638 729 828 936 1053 1179 1315 1462 1618 .64 .62 585 673 769 874 088 1112 1245 1380 1543 1708 .62 .60 616 708 810 020 1040 II70 1310 1462 1624 1708 .60 .58 647 744 850 066 1092 1220 1376 1535 1705 1888 .58 56 678 779 801 1012 1144 1287 1441 1608 1786 19078 .56 .54 708 815 031 1058 11060 1346 1507 1681 1868 2068 .54 .52 739 850 972 1104 1248 1404 1572 1754 1949 2138 .52 .50 770 886 1012 1150 1300 I+3 1638 1827 2030 2248 .50 48 801 921 1052 1196 1352 1I52I X704 1900 2III 2337 .48 .46 832 056 1003 1242 1404 1580 1760 1973 2192 2427 .46 .44 862 002 1133 1288 1456 1638 1835 2046 2274 2517 .44 .42 893 1027 1174 1334 1508 1697 1900 2119 2355 2607 .42 .40 924 1063 1214 1380 1560 I.,5 T1066 2192 243 2697 .40 38 955 1098 1255 1426 1612 1814 2031 2265 25.7 2787 .38 .36 986 1133 1205 1472 1664 1872 2007 2338 2508 2877 .36 .34 1016 1160 1336 1518 1716 1031 2162 2412 2680 2067 .34 .32 1047 1204 1376 1564 1768 1980 2228 2485 2761 3057 .32 .30 1078 1240 1417 1610 1820 2048 2203 2558 28 2 3147 .30 .28 1109 1275 1457 1656 1872 2 06 2359 2631 2023 3236 .28 26 II40 I3II 1498 1702 1024 2 O05 2424 2704 3004 3326 .26 24 I170 1346 1538 1748 1076 2723 2400 2777 3086 3416 .24 .22 1201 1381 1579 1794 2028 2282 2555 2850 3167 3506 22 20 1232 1407 1419 1840 2080 2340 2621 2023 3248 3596 .20 .I8 1263 1452 1660 1886 2132 2399 2686 2006 3320 3686 .I8 .I6 1204 1488 1700 1932 2184 2457 2752 3069 3410 3776 .16 .14 1324 1523 1741 1978 2236 2516 2817 3142 3492 3866 14 .I2 1355 1558 1781 2024 2288 23574 2883 3216 3573 3956 «12 10 1386 1594 1822 2070 2340 2633 2048 3280 3654 4046 .I0 08 1417 1620 1862 2116 2392 2691 3014 3362 3735 4135 .08 06 1448 1665 1903 2162 2444 2750 3079 3435 3816 4225 .06 04 1478 1700 1943 2208 2406 2808 3145 3508 3808 4315 .04 02 IF09 X '6 1084 2254 2548 2%7 3210 3581 3979 4405 .02 -00 I540 17,1 2023 2300 2600 2023 3276 3654 4060 44905 .00 —1.00 3080 3542 4048 4600 5200 5850 6552 7308 8120 8990 —1I1.00 253 EMPLOYMENT PSYCHOLOGY TABLE 4 (Continued) NuMBER OF CASES RANKED Pp 3E 32 33 . 34 ..35. ‘36 37 38 39 40 p 1.00 00 00 00 00 00 00 00 00 00 00 I.00 .08 99 109 120 I3I 143 155 160 183 108 213 .98 .96 198 218 239 262 286 311 337 366 305 426 .96 .94 208 327 350 303 428 466 506 548 503 640 .94 .92 397 436 479 524 571 622 675 731 790 853 .92 -005 A060 1546111508 R655 Nore ilies Saal iors. 0881 IT060 1.00 .88 595 655 718 785 857 032 'Tolz 1007 1186 ‘1270 .88 .86 694 764 838 016 1000 1088 II8r 1279 1383 1492 .86 .84 794 873 057 1047 NI1I42 3243 3350 1462" 1581 1706 .84 .82 803 082 1077 1178 1285 '¥300 1518 1645 1778 1910 .82 .80 992 I00I IIQ7 I300 1428 1554 1687 1828 19076 2132 .80 .78 1001 1200 1316 1I440 I57I 1709 1856 20IX 2174 2345 7S .76" 1100 1300 ' T4300 "1571 1714 1865. 2025 2103 2371 2553 .76 .74 1200 3410 I556 3702 1856 2020 2193 23760 2560 2772 -74 .72 1389 1528 1676 1833 1999 2176 2362 2559 2766 2085 N2 .70 1488 1637 1795 1064 2142 2331 2531 2742 2064 3108 .70 .68 1587 1746 1915 2004 2285 2486 2700 2024 3162 34I1I .68 .66 1686 1855 2035 2225 2428 2642 2868 3107 33590 3624 .66 .64. 1786 1064 2154 2356 2570 2797 3037 3290 3557 3838 .64 .62 1885 2073 2274 2487 2713 2953 3206 3473 3754 4051 .62 .60 1084 2182 2304 2618 2856 3108 3374 3657 3952 42064 .60 .58 2083 2202 2513 2749 2999 3263 3543 3838 4150 4477 .58 .56 2182 2401 2633 2880 3I42 3410 37I2 4021 4347 4690 .56 .54 2282 2510 2753 30II 3284 3574 388 4204 4545 4904 .54 .52 2381 2619 2872 3142 3427 3730 4049 4387 4742 SII7 .52 .50 2480 2728 2092 3273 3570 3885 4218 4570 4940 5330 .50 .48 2579 2837 3112 3403 37I3 4040 4387 4752 5138 5543 .48 .46 2678 2046 3231 3534 3856 4196 4555 4935 5335 5756 .40 .44 2778 3055 335I 3665 3998 435I 4724 S5II8 5533 5970 .44 42 2877 3164 347 3796 4I4I 4507 4803 530I 5730 6183 42 .40 2076 3274 3590 3927 4284 4662 5062 5483 5928 6306 .40 .38 3075 3383 3710 4058 4427 4817 5230 5666 6126 6609 .38 .36 3174 3492 3830 4189 4570 4973 5399 5849 6323 6822 .36 .34 3274 3601 3040 4320 4712 5128 5568 6032 ©6521 7036 .34 .32 3373 3710 40690 445I 4855 5284 5736 6215 6718 7249 .32 .30 3472 3819 4180 4582 4998 5439 5905 6397 6916 7462 .30 .28 357 3928 4308 4712 5I4I 5504 6074 6580 7114 7675 .28 .26 3670 4037 4428 4843 5284 5750 6243 6763 7311 7888 .26 .24 3770 4147 4548 4974 5426 5905 6411 6946 7500 8102 .24 .22 3869 4256 4668 5105 53569 606 6580 7128 7706 8315 .22 .20 3968 4365 4787 5236 5712 6216 6749 731I 7904 8528 .20 .I8 4067 4474 4907 5367 5855 6371 6018 7404 8102 8741 .I8 .16 4166 4583 5027 5498 5008 6527 7086 7677 8200 8954 .1I6 .I4 4266 46092 5146 5629 6140 6682 7255 7860 8497 9I68 14 .I2 4365 480I 5266 5760 6283 6838 7424 8042 8694 938I 12 .I0 4464 4910 5386 5801 6426 6903 7502 8225 8802 0594 .IO .08 4563 5020 5505 6021 6569 7148 7761 8408 09090 9807 .08 .06 4662 5129 5625 6152 6712 7304 7930 8501 9287 10020 .06 .04 4762 5238 5745 6283 6854 7459 8099 8773 0485 10234 .04 .02 4861 5347 5864 6414 6997 7615 8267 8056 0682 10447 .02 .00 4960 5456 5084 6545 7140 47770 8436 0139 9880 10660 .00 —I1.00 0020 I00I2 II968 13000 14280 15540 16872 18278 10760 21320 —I.00 284 TABLES TABLE 4 (Continued) NuMBER OF CASES RANKED p 41 42 43 44 45 46 47 48 49 50 p 1.00 00 00 00 00 00 00 00 00 00 00 I.00 .08 230 247 265 284 304 324 346 368 302 417 .08 96 450" ‘#94 530. 868. ‘Gey 640 1602; 737% 784.833" 06 .94 689 740 795 851 o1I 973 1038 - 1103 1170 1250 .94 .92 018 087 1060 1135 I2I4 1207 1384 1474 1568 1666 .92 .00 TI148 1234 1324 1410 1518 1622 1730 1842 150 2083 .90 .88 1378 1481 1580 1703 1822 1046 2076 22I1 23.2 2499 .88 .86 1607 1728 1854 1987 2125 2270 2421 2579 2744 2016 .86 .84 1837 1975 2110 2270 2420 2594 2767 2048 3130 3332 .84 .82 2066 2221 2384 2554 2732 2919 3II3 3316 3528 3749 .82 .80 2206 2468 2640 2838 3036 3243 3459 3685 3020 4165 .80 .78 2526 2715 2014 3122 3340 3567 3805 4053 4312 4582 .78 .76 2755 2062 3179 3406 3643 3802 4I5I 4422 4704 4908 .76 .74 2085 3200 3443 3680 3047 4216 4497 4790 509 5415 .74 .72 3214 3455 3708 3973 4250 4540 4843 5159 5488 5831 .72 70 3444 3702 3973 4257 4554 4865 5180 5527 5880 6248 .70 .68 3674 3949 4238 4541 4858 5180 5535 5806 6272 6664 .68 .66 3003 4196 4503 4825 5I6I 5513 5881 6264 6664 7081 .66 .64 4133 4443 4768 5108 5465 5837 6227 6633 7050 7497 .64 .62 4362 4690 5033 5392 5768 6162 6572 700I 7448 7914 .62 .60 4502 4936 5208 5676 6072 6486 6018 7370 7840 8330 .60 .58 4822 5183 5562 5960 6376 6810 7264 7738 8232 8747 .58 .56 505I 5430 5827 6244 6679 7135 7610 8107 8624 9163 .56 .54 5281 5677 6002 6527 6983 7459 7956 8475 0016 9580 .54 .52 55I0 5924 6357 6811 7286 7783 8302 8844 0408 9996 .52 .50 5740 6171 6622 7005 7590 8108 8648 0212 0800 10413 .50 .48 5070 ©6417 6887 7370 7804 8432 8994 9580 10192 10829 .48 .46 6190 6664 7152 7663 8197 8756 0340 0949 10584 11246 .46 .44 6420 6011 7417 7946 850I 0080 0686 10317 10976 11662 .44 .42 6658 7158 7682 8230 8804 9405 10032 10686 11368 12079 .42 .40 6888 7405 7946 8514 0108 0720 10378 II1054 I1760 12495 .40 .38 7118 7651 8211 8798 0412 10053 10724 II423 12152 12012 .38 .36 7347 17808 8476 0082 0715 10378 11060 II70I 12544 13328 .36 .34 7577 8145 8741 9365 I00I9 10702 II4I5 I2160 12036 13745 .34 .32 7806 8392 9006 ©0649 10322 11026 II76I 12528 13328 141601 “32 .30 8036 8639 0271 ©0033 10626 11351 12107 12807 13720 14578 .30 .28 8266 8886 0536 10217 10030 II10675 12453 13265 I4II2 14904 .28 .26 8495 0132 080I 1I050I II233 IIQQQ 12790 13634 14504 I54II .26 .24 8725 0379 10065 10784 11537 12323 13145 I4002 14806 15827 .24 .22 8054 0626 10330 I1068 11840 12648 13401 1437 15288 16244 22 20 0184 0873 10505 11352 I2I44 12072 13837 14739 15680 16660 .20 18 0414 10120 10860 11636 12448 13206 14183 15108 10172 17077 .18 .I6 0643 10366 11125 11920 I275I 1362 145290 154760 16464 17493 .16 .I4 ©9873 10613 11300 12203 I3055 139045 14875 15845 168560 17910 14 .I2 10102 10860 11655 12487 13358 14200 15220 16213 17248 18326 «X12 I0 10332 IT707 11020 I277I 13662 14504 15566 16582 17040 18743 .I0 08 10562 11354 12184 13055 13066 14918 15912 16050 18032 19159 .08 06 10701 I"H0I 12449 13339 14260 15242 16258 17310 18424 19576 .06 .04 I1021 11847 12714 13622 14573 15566 16604 17687 18816 19902 .04 ,02 II250 12094 12079 13906 14876 15801 16950 18056 10208 20409 .02 .00 11480 1234I 13244 14190 15180 16215 17206 18424 10600 20825 .00 —I.00 22060 24682 26488 28380 30360 32430 34592 36848 39200 41650 —I.00 255 EMPLOYMENT PSYCHOLOGY TABLE 4 (Continued) NuMBER oF CASES RANKED p 5I 52 53 54 55 56 57 58 59 6o p 1.00 00 00 00 00 00 00 00 00 00 00 I.00 +088 aaa Hi a6a i406 Nii528 i 55 (lh 585 Ni GT) 6508 (684 © yz0 .08 .06 884 037 002 1049 1100 II70 1234 1300 | 13060 . 1440 .06 “gq 801326 i1T106 1 ITAS8 “iiz574 17663 1750" 1851 1051" 2053 | 2I59 .04 .92 1768 1874 1984 2099 2218 2341 2468 2601 2738 2879 .92 «000 27102343 124801 12624 2792 12026 3086" 3251 3422 3500 .90 .88 2652 | 2811 | 2976 3148 3326 35II 3703 3901 4100. 4310 .88 .86 3004 3280 3473 3673 3881 4096 4320 455I 4791 5039 .86 -84 3536 3748 3960 4198 4435 4682 4037 5201 5475 5758 .82 .82 3978 4217 4465 4722 4990 5267 5554 5852 6160 6478 .84 .80 4420 4685 4961 5247 5544 5852 6171 63502 6844 7108 .80 .78 4862 5154 5457 5772 6098 6437 6788 7152 7528 7918 .78 .76 5304 5622 5053 6206 ©0653 7022 7405 7802 «8213 8638 .76 .74 5746 6091 6449 6821 7207 7608 8023 8452 8897 0357 .74 .72 6188 6559 6945 7346 7762 8193 8640 9I03 9582 10077 72 .70 6630 | 7028 ' 7441 7871 8316 8778 0257 9753 102066 10707 -70 .68 7072 7496 7937 8395 8870 0363 0874 10403 10950 11517 .68 .66 7514 7065 8433 8920 9425 9948 10491 11053 11635 12237 .66 .64 7956 8433 8920 9445 9979 10534 IIT08 11703 12319 12956 .64 .62 83908 8902 0426 0969 10534 IIIIQ II725 I2353 I3004 13676 .62 .60 8840 0370 0022 10494 IT1088 II704 I2342 13004 13688 14306 .60 .58 0282 0839 10418 11010 11642 I2289 12060 13654 14372 IS5IIO .58 .56 9724 10307 10914 II543 12197 12874 13577 14304 15057 15836 .56 .54 10166 10776 II4I0 I2008 1I1275I 13460 I4I00 14954 I574I 16553 .54 .52 I0608 11244 11906 12593 I3300 14045 I48II 15604 16426 17275 .52 .50 II050 II7I3 12402 13118 13860 14630 15428 16255 I7II0 I79905 .50 .48 11492 12182 128098 13642 14414 15215 16045 16005 17794 18715 .48 .46 11034 12650 13394 I4AI67 14969 15800 16662 17555 18479 19435 .46 .44 12376 13119 13890 14602 15523 16386 17279 18205 I9I163 20154 .44 .42 12818 13587 14386 15216 16078 16071 17896 18355 10848 20874 .42 .40 13260 14056 14882 15741 16632 17556 18514 10505 20532 21594 .40 .38 13702 14524 15378 16266 17186 18141 IQI3I 20156 21216 22314 .38 .36 14144 14993 15875 16790 17741 18726 10748 20800 2I100I 23034 .36 .34 14586 15461 16371 17315 18295 10312 20305 21456 22585 23753 .34 .32 15028 159030 16867 17840 188350 19897 20982 22100 23270 24473 .32 .30 15470 16398 17363 18365 19404 20482 21599 22756 23954 25103 .30 .28 15912 16867 17850 18889 19058 21007 22216 23406 24638 259013 228 .26 16354 17335 18355 10414 205I3 21652 22833 24057 25323 2060633 . 26 .24 16796 17804 18851 19930 21067 22238 23451 24707 26007 27352 224 .22 17238 18272 19347 20463 21622 22823 24068 25357 206692 28072 "22 .20 17680 18741 10843 20088 22176 23408 24685 26007 27376 28792 .20 .I8 18122 10209 20339 2I5I3 22730 23993 25302 26657 28060 29512 .18 .I6 18564 10678 20835 22037 23285 24578 25910 27308 28745 30232 .16 .I4 19000 20146 21331 22562 23830 25164 26536 27958 20429 3005I .14 .I2 19448 20615 21828 23087 24394 25749 27153 28608 30114 3I67I ST2 .I0 10800 21083 22324 23612 24948 26334 27770 20258 30708 32301 .1I0 .08 20332 21552 22820 24136 25502 26019 28388 20008 31482 33III .08 .06 20774 22020 23316 24661 26057 27504 29005 30558 32167 33831 .06 .04 21216 22489 23812 25186 26611 28090 20622 31200 3285I 34550 .04 .02 21658 22057 24308 25710 27166 28675 30239 31859 33536 35270 .02 .00 22100 23426 24804 26235 27720 20260 30856 32509 34220 35990 .00 —I.00 44200 46852 49608 52470 55440 58520 61712 65018 68440 71980 —I.00 256 TABLES TABLE 4 (Continued) NUMBER OF CASES RANKED Pp 61 62 63 64 65 66 67 68 60 70 p 1.00 00 00 00 00 00 00 00 00 00 00 1.00 .08 756’ 704 833 874 015 9058 1002 7048 (1005 I143 '.08 .06 1513 1588 1667 1747 1830 1016 2005 2006 2190 2286 .06 .94 2269 2383 2500 2621 2746 2874 3007 3144 3284 3420 .04 .92 3026 3177 3333 3494 3601 3832 4009 4192 4379 4572 .92 00 3782 3971 4166 4368 4576 4791 5012 5230 5474 5716 .90 .88 4538 4765 5000 53142 5491 5749 6014 6287 6569 ° 6850 .88 .86 5205 5560 5833 6°15 6406 6707 7016 7335 7664 8002 .86 .84 60s 6354 6666 ©6980 7322 7665 8019 8383 8758 0145 .84 .82 6807 7148 7500 7%62 8237 8623 0021 943I 9853 10288 .82 .80 7564 7942 8333 8736 0152 0581 10023 10479 I0048 II143I .80 .78 8320 8736 0166 0610 I0067 10539 I1020 II527 12043 12574 .78 76 09077 0531 0999 10483 10082 11497 12028 12575 13138 13717 .76 .74 9833 10325 10833 11357 11808 12455 13030 13622 14232 14860 74 .72 10500 IIIIQ II666 12230 12813 I34I3 14032 14670 15327 16003 «12 .70 II346 IIQI3 12400 I3104 I3728 14372 15035 I5718 16422 17147 .70 .68 12102 12708 13332 13978 14643 15330 16037 16766 17517 18200 .68 .66 12850 13502 14166 14851 15558 16288 17030 17844 18612 10433 .66 64 I3615 14206 14000 15725 16474 17246 18042 18862 19706 20576 .64 .62 14372 15000 15832 16598 17389 18204 10044 [9910 20801 21719 .62 .60 15128 15884 16666 17472 18304 19162 20046 20058 21806 22862 .60 .58 15884 16670 17499 18346 10210 20120 21049 22005 2290 24005 .58 .56 16641 17473 18332 19219 20I34 21078 22051 23053 24086 25148 .56 .54 17397 18267 19165 20003 21050 22036 23053 24I0I 25180 26291 .54 .52 18154 IQ06I 19999 20006 21965 22994 24056 25149 26275 27434 52 50 18010 19856 20832 21840 22880 23953 25058 26197 27370 28578 .50 .48 19666 20650 21665 22714 23795 24911 26060 27245 28465 20721 .48 46 20423 21444 224909 23587 24710 25869 27063 28203 20560 30864 .46 .44 21179 22238 23332 24461 25626 26827 28065 20341 30654 32007 .44 .42 21936 23032 24165 25334 206541 27785 29067 30389 31749 33150 .42 .40 22602 23827 24998 26208 27456 28743 30070 31436 32844 34203 .40 .38 23448 24621 25832 27082 28371 20701 31072 32484 33939 35436 .38 .36 24205 25415 26665 27955 20286 30659 32074 33532 35034 36579 .36 .34 24961 26200 27498 28820 30202 31617 33077 34580 36128 37722 .34 .32 25718 27003 28332 29702 3III7 32575 34079 35628 37223 38865 .32 .30 26474 27798 20165 30576 32032 33534 35081 36676 38318 40000 .30 .28 27230 285902 20008 31450 32047 34492 36084 37724 39413 41152 .28 .26 27987 20386 30831 32323 33862 35450 37086 38772 40508 42205 .26 .24 28743 30180 31665 33197 34778 36408 38088 39810 41602 43438 .24 -22 29500 30975 32498 34070 35693 37366 39090 40867 420607 44581 .22 .20 30256 31760 3333 34944 36608 38324 40003 41015 43792 45724 .20 .I8 31012 32563 34164 35818 37523 30282 41005 42063 44887 46867 .18 .16 31760 33357 34998 30601 38438 40240 42007 440II 45082 48010 .1I6 14 32525 34I5I 35831 37565 30354 41108 43100 45059 47076 49153 .I4 -I12 33282 34046 36664 38438 40269 42156 44102 46107 48171 50200 .I2 10 34038 35740 37498 30312 41184 43115 45104 47155 49266 51440 .I0 .08 347904 36534 38331 40186 42000 44073 46107 48202 50361 52583 .08 06 3555I 37328 30164 41059 430I4 45031 47100 49250 51456 53726 .06 04 36307 38123 39007 41033 43930 45080 48III 50208 52550 54860 .04 02 37064 38017 40831 428006 44845 46047 49114 51346 53645 56012 .02 .00 37820 397II 416064 43680 45760 47905 S50IIO6 52304 54740 57I55 .00 —1.00 75640 79422 83328 87360 01520 05810 100232 104788 109480 II43I0 —I.00 257 N EMPLOYMENT PSYCHOLOGY TABLE 4 (Concluded) NuMBER oF CASEs RANKED p 71 72050003 5 a LIS IG 98 79.0 Be. p 1.00 00 00 00 00 00 00 00 00 00 00 I.00 9% liirTo38 yz2448 1206. 1350 1400) ‘1463 "152% 158% 1643 1706 .08 .00 | 2386 2488" 2503' 2701 2812 2026 3043 3163 3286 3413 .06 .94 3578 3732 3880 4052 4218 4380 4565 4745 4930 5II0 .04 .92 4771 4976 5186 5402 5624 5852 6086 6326 6573 6826 .92 .00 5964 6220 6482 6753 7030 7315 7608 7908 8216 8532 .00 .88 7157 7464 7779 8103 8436 8778 0120 0489 0850 10238 .88 .86 8350 8707 0075 0454 9842 1024I I065I IIO7I II502 IIQ45 .86 .84 0542 0051 10372 T0804 11248 11704 12172 12653 13146 1365I .84 .82 10735 III95 II1668 12155 12654 I3I67 130604 14234 14789 15358 .82 .80 11028 124390 12065 13505 I4060 14630 15215 15816 16432 17064 .80 .78 13121 13683 14261 14856 15466 16003 16737 17397 18075 18770 .78 76 14314 14927 15558 16206 16872 17556 18258 18079 19718 20477 70 .74 15506 I6I7I 16854 17557 18278 10019 19780 20561 21362 22183 .74 .72 16699 17415 1815I 18907 19684 20482 2I130I 22142 23005 23890 2 .70 17892 18659 10447 20258 21000 21045 22823 23724 24648 25506 .70 .68 10085 10003 20744 21608 22496 23408 24344 25305 26201 27302 .68 .66 20278 21147 22040 22050 23902 24871 25866 26887 27034 29009 .66 .64 21470 22301 23337 24309 25308 26334 27387 28468 29578 30715 .64 .62 22663 23634 24633 25660 26714 27797 28909 30050 3I22I 32422 62 .60 23856 24878 25030 27010 28120 20260 30430 31632 32864 34128 .60 .58 25049 26122 27226 28361 20526 30723 31952 332I3 34507 35834 .58 .56 26242 27366 28523 20711 30032 32186 33473 34795 30I50 3754I .56 .54 27434 28610 20819 31062 32338 33649 34995 36376 37794 39247 .54 .52 28627 29854 3III6 32412 33744 35112 36510 37958 39437 40054 .52 .50 20820 31098 32412 33763 35150 36575 38038 39540 41080 42660 .50 .48 31013 32342 33708 35113 36556 38038 30560 4II2I 42723 44366 .48 .46 32206 33586 35005 36464 37962 39501 AIO8I 42703 44366 46073 .46 .44 33398 34830 36301 37814 30368 40064 42603 44284 46010 47779 .44 .42 3459 30074 37598 30165 40774 42427 44124 45866 47653 49486 .42 .40 35784 37318 38804 40515 42180 43890 45646 47447 49206 51102 .40 .38 36077 38562 40101 418606 43586 45353 47107 49020 50039 52808 .38 .36 38170 390805 41487 43216 44992 46816 48689 50611 52582 54605 .36 .34 39362 41049 42784 44567 46308 48279 50210 52102 54226 563IT .34 .32 40555 42293 44080 45917 47804 49742 51732 53774 55869 58018 .32 -30 41748 43537 45377 47268 49210 51205 53253 55355 57512 59724 .30 .28 42041 44781 46673 48618 50616 52668 54775 56037 59155 61430 .28 .26 44134 46025 47970 49960 52022 54131 56206 58518 60798 63137 .26 -24 453206 47269' 49266 51319 53428 55594 57818 60100 62442 64843 .24 .22 46519 48513 503563 52670 54834 57057 59339 61682 64085 60550 .22 .20 47712 49757 518590 54020 56240 58520 60861 63263 65728 68256 .20 .I8 48005 5100I 53I56 5537I 57646 50083 62382 64845 67371 60062 .18 .I6 50008 52245 54452 5672I 59052 61446 63004 66426 69014 71660 .I6 .I4 51200 53480 55749 58072 60458 62009 65425 68008 70658 73375 .I4 -I2 52483 54732 57045 59422 61864 64372 66947 69590 7230r 75082 .I2 .I0 53676 55076 58342 60773 63270 65835 68468 7II7I 73944 76788 .I0 .08 54860 57220 50638 62123 64676 67208 69990 72753 75587 78494 .08 .06 56062 58464 60035 63474 66082 68761 7I5II 74334 77230 8020I .06 104 57254 59708 62231 64824 67488 70224 73033 75916 78874 81907 .04 02 58447 60052 63528 66175 68804 71687 74554 77497 80517 83614 .02 .00 59040 62106 64824 67525 70300 73I50 76076 79079 82160 85320 -00 —1.00 119280 124392 129648 135050 140600 146300 152152 158158 164320 170640 —I.00 258 TABLES TABLE 5 VALUES OF # CORRESPONDING TO VALUES OF p (See pages 178-179) 01 01 51 .53 .02 03 .52 .54 03 03 .53 .55 04 .04 .54 .56 05 05 «55 57 .06 06 .56 .58 07 .07 .57 .59 .08 .08 .58 .60 .09 .09 .59 61 .I0 .10 : .60 62 ; XX «12 .61 63 +12 JI3 .62 .64 13 14 63 63 14 .I5 .64 .66 .I5 .16 65 67 I6 J7 .66 .68 17 .I8 67 69 .I8 19 .68 70 .19 .20 .69 JI .20 21 .70 72 “21 22 JT 3 .22 .23 72 74 23 24 73 73 .24 «25 74 .76 25 .26 15 27 26 “27 76 78 27 .28 ] «77 .78 .28 .20 78 79 .29 .30 79 .80 .30 31 .80 81 «31 .32 81 .82 .32 .33 82 .83 .33 34 83 84 34 .35 84 .85 .35 .36 85 .86 .36 37 .86 87 “37 .39 87 .88 .38 .39 .88 .80 .39 41 .80 .90 .40 42 .90 .9I 41 .43 L oI 02 42 44 02 .93 -43 , 45 93 . 94 -44 -46 94 94 45 47 95 95 .46 .48 06 | .06 47 49 97 : 97 .48 .50 08 .08 49 51 99 99 .50 52 1.00 1.00 259 0 EMPLOYMENT PSYCHOLOGY TABLE 6 VALUES OF # CORRESPONDING TO VALUES OF R (See page 178) oe —- ) .00 .00 50 73 .0I 02 SL 74 .02 .04 .52 775 .03 .05 33 76 .04 .07 .54 77 .05 .09 .55 i 78 .06 JIX .56 79 .07 «12 «57 .80 .08 14 .58 81 .09 10 .59 82 .10 .I8 .60 .83 XX .19 ) 01 84 12 21 i 62 .84 “13 .23 ! .03 85 14 .24 .04 .86 15 .26 .65 .87 .16 .28 .66 .88 I7 .20 67 .88 .I18 “31 .68 .89 10 .32 .69 .00 .20 .34 770 .90 21 35 271 OI 22 .37 72 .92 -23 .38 3 73 92 .24 -40 -74 93 .25 41 I 5 93 .26 .43 .76 .04 .27 -44 77 .94 .28 46 78 -95 .20 47 | 79 -95 .30 .49 i .80 .06 3% .50 81 .96 232 .51 .82 97 .33 53 .83 97 .34 .54 84 97 .35 .55 .85 .08 .36 57 .86 .08 .37 .58 .87 .08 .38 .59 .88 .08 .39 Hi .80 99 .40 .62 .00 .00 41 .63 OI 99 42 | .64 .92 -99 43 .05 .03 1.00 44 | .67 .94 1.00 45 .68 .95 1.00 .46 .69 .96 1.00 47 70 .97 1.00 .48 «7X 98 1.00 .49 2 .99 1.00 C0 LE! 1.00 1.00 260 rR k : TABLES TABLE 7 VALUES OF #7 CORRESPONDING TO VALUES OF PERCENTAGE OF UNLIKE SIGNED PAIRS OF MEASURES (U) This table also gives values of 7 corresponding to values of the fraction=—— ~~ ;—in formula 19. Values of the fraction Vad +/bc are located in column U and the corresponding values of r are found in the next column. (See pages 180-182.) rm r r 00 1.00 50 —.00 .0I 1.00 51 —.03 .02 1.00 52 —.06 03 1.00 53 -—.09 04 .09 54 —.13 05 .00 .55 —.16 .06 08 56 —.19 07 08 3 -.22 08 97 5 —.25 00 06 +5 —.28 .1I0 05 .60 -.31I oJ 04 6. -.34 3 .93 62 —.37 J .02 63 —.40 5 .00 64 —.43 15 .80 65 -45 1 .88 .66 —.48 oJ .86 07 -.5% Ji 84 63 -.54 19 83 .60 —.56 .20 87 .70 -—.59 2 70 «7" -.0x 2 78 yan —.64 2 J 7} —.66 24 eT 74 —.68 25 7X Js —.71 24 .68 76 -.73 23 60 77 —.75 .28 .64 78 77 .209 61 79 -=.79 .30 .50 .80 —.81 31 50 81 —.83 32 S54 82 -.84 33 51 33 —.86 34 .48 .84 —.88 .35 .45 85 —.80 20 .43 .86 —.90 3 .40 87 —.02 3 “37 .88 —.03 39 34 80 —.04 .40 31 .00 —.05 4X .28 OI —.96 42 «25 92 -—.07 4: 22 93 —.98 «44 10 04 —.08 vo .I16 95 -.00 4f 13 .06 —.00 4 .09 07 —1.00 4 .06 08 —1.00 4) .03 99 —1.00 en On 1.00 -T 00 261 EMPLOYMENT PSYCHOLOGY TABLE 8 THE PROBABILITY THAT A DIFFERENCE Is REAL (See page 171) h hat the Diff Difference Divided by Standard Nae hot, the Di a Error of Difference tion Indicated Is 1 in oT 2.17 2 2.38 2.62 " 2.90 3 3.24 a 3.65 8 4.13 ’ 4.72 Y 5.43 1.0 6.30 T.I 7.37 1.2 8.69 1.3 10.33 1.4 12.38 1.5 14.97 1.6 18.25 1.7 22.42 1.8 27.86 I.9 34.84 2.0 44.05 2.1 55.87 2.2 71.94 2.3 03.46 2.4 121.95 2.5 161.29 2.6 212.77 2.7 285.71 2.8 384.62 2.9 526.32 3.0 740.74 3.7 1,030.93 3.2 1,455.39 3.3 2,068.68 3.4 2,968.24 3.5 4,299.23 3.6 6,285.36 3.7 9,276.44 3.8 13,831.26 3.0 20,790.02 4.0 31,545.74 4.5 294,117.65 5.0 3,487,066.52 262 INDEX INDEX A C Abilities 64-75, oo ff Cady, V. M. 8s analysis of 65-66 Categories of psychology 90-93 classification of go ff Chance that a difference is real 171, 262 general and specific 68-71 Thapman, J. C. 89, 113, 211 innate and acquired 67-68 Charters, W. W. 66, 72, 153 relative importance of 71-74 “hauffeur test 107-109, 230 salesgirls 73-74 ‘Child, E. 84, 171 shell inspectors 66 ~imbal, W, 60, 85 Accidents 38, 43-46, 230 -lothier, R. C. 14, 89 automobile 43-46, 230 Cody, S. 8s criteria 38 Zoefficient of alienation 200 Administering employment methods, Coefficient of association 183, 187 cost of 218 Coefficient of correlation Administering tests see Correlation see Procedure Color-blindness test 96 Advertising, cost of 218 Combining test scores Age, disturbing variable 51 see Total scores; Weighting Alienation, coefficient of 200 Commissions, as criteria 39 Allport, F. H. 60, 84, 8s Completion tests 97-98, 100 Allport, G. W. 60, 83 ~ompound slide rest 105-106 Analogies tests 99 ~omptometer operator tests 195 Analogy method of test construction 80-81 ‘ook, H. E. 73 Analysis of worker 58-75 corporation schools, standing in 34-35 analysis of abilities 55-06 Correlation 166-169, 174-180, 208-212 general and specific abilities 68-71 biserial » 182-184, 188 innate and acquired abilities 67-68 coefficient of association 185, 187 outline for analysis 60-65 corrected for attenuation 168-169, 188 abilities 64-65 correlation ratio 184-186, 188 education and training 62-53 data sheets and forms 174-176 interests 64 estimating correlation 176-177 personal data 60-61 factors lowering correlation 189 personal history 61 fourfold table 179-182, 188 physical characteristics 63 graphic methods 177 previous experience 61-62 interpretation of amount 189 social characteristics 63-64 mean square contingency 185, 188 purpose 58-50 multiple and partial 208-213 relative importance of abilities 71-75 probable error 187-189, 211 Application blank 60-62, 147-148 product-moment Army tests 69-70, 89, 93, 132, 191, 216 174-177, 181-182, 187-188 Assistant buyer, duties of 21-26 range 174 Association tests 97, 99-100 rank difference 178-179 Attenuation, correction for 168-169, 188 rank-tangential coefficient 184 reliability 165-169, 188 tetrachoric 179-182 weighting right and wrong answers 187 Baade 60 Correspondence, cost of 218 Ballard, P. B. 84 ~oss, J. J. 8s Benge, E. J. 40 ~owdery, K. M. 201 Bezanson, A. 40 Crathorne, A. R. 185 Bills, M. A. 104-195 Criterion of vocational success 30-41 Binet-Simon tests 69 college students 37-38 Biserial » 182-184, 188 criterion applied to a department 47-48 Blackford, K. M. H. 60, 6- definition 30 Bonuses, as criteria z disturbing variables 47 Breakage, as criterion 7 general considerations 42 ff Bridges, J. W. ¢ importance 31-33 Brinton, W. C. 21 postal clerks 35 Brown, Wm. 164, 167, 177, 18 purpose 30 ff Bureau of Personnel Research 8 relative advantages 48 Burt, C. 84 suggested criteria 33-42 Burtt, H. E. 84, So accidents 38 Business information test 82 advancement 40-41 B 26¢ EMPLOYMENT PSYCHOLOGY bonuses i 39 prediction of success from 190-196 length of service = 39-40 questionnaire items 172-173 nhs in societies 41 & test scores 170-173 outpu 35-36 rouping scales 12 ratings Hoh 41-42 Guessing 117, hi responsibility 41 questionnaires 149 salary 38-39 tests standardized examinations 36-38 Gun pointer test Togs Ne standing in corporation schools 34-35 time of training 34 H trade status 41 Halo effect 134- 8 t u : 34-135, 13 use in selecting subjects 49-51 Hartshorne, H. 85, 89, 120 Critical scores and sections Hayes, M. H. S. ¥32-133, 171-172 182-184, 190-196, 203-206 fon Wm. £5 epner, H. W. 85 Hines, M. 108 D H 2, . ollerith machines 6 Davenport, C. B. 60, 85, 164 Hollingworth, H. L. 51, 80-82 ‘5 Dawley, A. 171-172 Holzinger, K. J. 174, 185 Dennison Company rating scale 129-131 Huffaker, C. L. 211 Deseripatye scales 127-131 Hull, C. L. 176 ewey, E. 84, 171 Difference, significance of a I 171-173, 204-205, 262 : le Difficulty analysis 71-173, 204-205 66 Installation of examinations 222-228 Directions test 102 Interests 64, 145-146 Dodd, S. C. 176 analysis of 64 Dodge, R. 109-110 measurement of I45-146 Doherty, M. gs Interview Downey, J. 84, 118-120 18-20, 28-29, 153-154, 217, 225-228 Duties employment procedure 217, 225-227 see Job analysis getting questionnaire information 153-154 Ib analysis 18-20, 28-29 E eaving 227-228 Economic value of examinations 215-221 J Education and training, analysis of 62-63 Job analysis 13-29, 72 Esprit de corps 221 definition 13 Examinations duties 21-28, 72 see Tests; Rating scales; Questionnaires assistant buyer 21-26 Examiner 156, 159-161, 222-223, 225-228 printer 26-28 Eyesight of workers 52 secretary 72 . shell inspector 21 F importance of 13-14 Fernald, M. R. 171-172 Inonip ons 1 ki Finger dexterity test ) 106, 108 procedure in making ry Forms io pdministindion PR sources of information 18-20 Fourfold table method 179-182, 188 § he 3 R 1 Franz, S. I. 84-85 fora Gi 160 Freyd, M. 61, 85, 123, 131-132, TEE oT 4 136-137, 146, 149, 172, 195 K G Beller cong fuction test 5 105 y Le 66-167, 176 Garfiel, E. 211 elley, : 451 S210 ; General common factor 68 ff 132, 788, 199-201, 2105221, 209 J Kemble, W. F. 85 General Electric Company Kenagy, H. G 61, 192-103 57, 108, 157, 223-224 ie ? attitude of workers toward tests 157 ry F.A hoe finger dexterity test 108 Kirby, G.H., 60 84 giving out test scores 57 Knight, F. B. 139 Gi fiscond sheet 22372 2 Kornhauser, A. W. 8s Gilbreth, F. J. 28 Gilliland, A. R. 93-94, 105 - Graphic presentation of results Langfeld, H. S. 84 : 217-218, 230-231 Langlie, T. A. 117 Graphic rating scale 128-133, 135-138 LaRue, D. W. 60, 85 Gregg, R. B. 14 Leaving 39-40, 227-228 Group comparisons 170-173, 190-196 interview at 227-228 group with a sub-group 171-172 reasons for 39-40, 227-228 266 INDEX 267 Length of service, as criterion 39-40 based on group comparisons 190-196 Letter to references 150-151 combined scores 203-214 Linear scales 127 comparison of old and new Link, H. L. 21, 28, 60, 66, 84, 157-158 methods 216-217 Lipmann, O. 60, 84-853 miscellaneous methods 211-214 Literacy, as disturbing variable 53 Preferred range 191-106 Loss due to breakage, as criterion 38 Pressey, L. C. 89 Pressey, S. L. 89 Printer, duties of 26-28 M Probability that a difference is real 171, 262 Maintenance of examinations 228-232 Pr ghable effor 187-189, at Mensots % E. 1%: 3s 165 correlation ratio 188 BY 2 Tru 3s 5, 89, 120 fourfold table 188 McCall, Wm. A. 89 meaning of " . go 187, 189 Mean square contingency 183, 188 mean square contingency 185 Measuring instruments : : multiple correlation 211 see Tests; Rating scales; Questionnaires product-moment 187-188 Mechanical aptitude test 100-101 tank difference 187-188 bi in societies, as criterion 4 x 7 Sortected for attenuation 15% Mendenhall. G » 60 Problem of selection 3 ff M zy Procedure 6-9, 77, 159-162, 225-230 erton, H. W. 63 examining 159-162 Metcalf, H. C. &1 se ’ $ hiring 225-230 Meyer, A. 60, 85 research 6-9, 77 Miner, J. B. 8. £ : ! . purpose o 77 Miner, Z. F. 18 steps 6 ff piischell, D 'S 3 theoretical implications 8-9 4 Te vv 3 9 Production, differences in 220 ore, B. V. 5,193 product-moment correlation Moore, H. T. 93-94, 105 174-177, 187-188 Motion stud; 28 : y ‘ : . Profile method 211-214 Motor defects, as disturbing variable 53 Promotion 40-41 Multiple choice tests 08 Multiple step scales 126-127 Muscio, B. 85 Q Myers, C. S. 84 Quality of output 35-36 Quantity of output 35-36 0 Questionnaires 87, 143-154, 104-106 Objectivity 77-78 characteristics of 145-149 Occupation to be studied, choice of 9-11 definition 87, 143-144 O’Connor, J. 106, 108, 157, 223-224 evaluation of questions 194-196 Dffice overhead, cost of 219 follow-up 152 Jmnibus tests 120 obtaining data 149-154 Irder of merit scales 123-124 uses of 144-145 YRourke, L. J. 35, 80, 114-115, 217 Otis, A. S. 164, 168, 174-175, 201, 211 R Outhwaite, L. 85 Random method of test construction 81-82 Pr Rank difference correlation 178-179, 187-188 Parsons, F. 20, 60 Rank-tangential coefficient 184 Partridge, G. E. 60, 84 Ratings 41-42, 133, 138-142 Paterson, D. G. 84, 89, 117, 132-134 as criteria 41-42 Patten, E. F. 108 obtaining ratings 138-140, 142 Pearson, K. pitfalls in rating 138-140 172, 174, 180, 182, 187-188, 101 reliability of 133 Percentile curves and tables 201, 207 treatment of 140-142 Per cent scales 126 see also Rating scales Perrin, F. A. C. 8s Rating scales 87, 122-138, 143-144 Personal history and data 60-62, 145 ff, 103 comparison of rating methods 132-135 analysis of 60-62 definition 87, 122, 143-144 obtaining data 145 ff halo effect 134-135, 138 scores of salesmen 103 how to make a graphic rating Physical characteristics, analysis of 63 scale 135-138 Pintner, R. 84 methods of rating 123-138 Plant, J. S. 135 Dennison Company scale 129-131 Poffenberger, A. T. 51 descriptive scales 127-131 Postal clerks, tests for 35, 114-115, 217 graphic rating scale 128-133, 135-138 Powers machine 163 grouping 124 Prak, J. L. 105 linear scales 127 Prediction of vocational success 190-217 multiple-step scales 126-127 based on correlation 197-202 order of merit 123-124 EMPLOYMENT PSYCHOLOGY Dep cent Jeales : 126 Standard error scale of alternatives 126 172-173, 100-200, 204-2085, 2II Scott scale 124-126, 132 of a difference in Si hoi uses 122-123, 141-142, 217 tions 172-173, 204-205 Ream, M. J. 123 of estimate 199-200, 2IT References } 233-245 Standardized examinations, as criteria 36-37 Regression equation and line 197-199, 208 Stenquist, J. L. 93, 100-101, 105 Regression equation, multiple 208-211, 213 Steps in research procedure 6 if Reliability 78-79, 133, 166, Stern, E. 60, 85 .. 187-189, 200, 205-206, 208, 21x Stern, Wm. 60, 84. definition 166 Stockbridge, F. P. 82 in weighting scores 205-206, 208 Stoddard, G. D. 117 of correlation 187-189, 211 Stoelting Company 84, 173 of prediction 200 Sturges, H. A. 177 of ratings 133 Subjects of the investigation 49-57, 161-162 of tests 78-79 disturbing variables 5I-53 see also Reliability coefficient : age Sy Reliability coefficient 166-169 literacy 53 correlation for attenuation 168-169 motor defects 53 factors which lessen it 166-167 sensory defects 51-52 relation to length of test 167-168 sex 5I relation to range of scores 167 general considerations 56 Spearman’s prediction formula 167 newly hired versus old Responsibility, as criterion 41 employees 54-56, 161-162 Ringers 55 ringers 55 Roget 60 sampling 49-51 Room used for testing 157-159, 223, 225 Symonds, P, S. 85, 89, 124, 211 Rosanoff, A. J. 100 Rosenow, C. 21I T Rossi, D. I. P. 85 Rossi, Wm. H. 85 Tables 247-262 Ruch, G. M. 117 Tabulation 165 Ruger, G. J. 85 Tagg, M. 105 Rugg, H. O. 126, 132, 164-165, 174, Tead, O. 13-14 176-177, 183, 188-189, 199, 217 Terman, L.M. 60, 84-85, 88, 100 Ruml, B, 84, 171, 182, 184, 200 Tests 69-70, 76-121, 132, 143-144, 155-162, 166, 179-182, 188, 191, 205-206, 208,1216-217, 223-224, 230 S administration of 155-162 Salaries of interviewers 219 alternative forms 121 Salesgirls, abilities of 73-74 analogies tests 99 Sampling method of test construction 80 analogy with ability to be measured 79-82 Sampling subjects 49-5T army tests 69-70, 89, 93, 132, 191, 216 Sanford, E. C. 84 arrangement 1n series 121 Scale of alternatives 126 association tests 97, 99-100 Scatter diagram 176, 197 ff Binet-Simon tests 69 Schulze, R. 84 business information test 82 Scott Company 178-179, 251-258 catch questions 96 Scott scale 124-126, 122 characteristics of a good test 27 ff, 89 Scott, W. D. 14, 55, 89, 124 chauffeur test 107-109, 230 Scovill Manufacturing Co. 39-40, 162 classification of tests go ff Scripture, E. W. : 84 ability measured 00-93 Secretary, duties of 72 apparatus used 93-94 Sensory defects, as disturbing variable 51-52 purpose 90 Sex, as disturbing variable ST restrictions on response 9s ff Shell inspectors 21, 66 technique 94 abilities of 66 color blindness test 96 duties of 2% completion tests 97-98, 100 Shen, E. 139 compound slide rest 105-106 Sheppard, W. F. 180 definition 87, 143-144 Significance of a difference 171, 262 difference between test and experiment 88 Simplicity of tests 79 directions and fore-exercises 112-115 Snow, A. J. 43-46, 230 directions tests 102 Social characteristics, analysis of 63-64 finger dexterity test 106, 108 Spearman, C. 70, 167, 178 group tests 94 Spearman’s prediction formula 167 guessing 117 Spoilage, cost of 219-220 gun pointer test 109-110 Standard deviation 170-172 individual tests 94 of a difference in means 171-172 items of test 9s ff of a distribution 170 order of I10-II1I of a mean 170 performance tests 104-110 see also Standard-error verbal tests 95-103 268 INDEX 20: Kelley construction test 105 decrease among salesmen 231 length of test III savings from decrease 220 mechanical aptitude test 100-101 fresbods of rast Squstruction 80-82 U analogy me 80-81 R random method 81-82 Unbrock, RB. S. ig sampling method 8o v vocational miniature 8o Validit Guid Sujcinle choice tests v3 gon I of 108158 1 69 iki "Mla measures of : 163-189 performance tests 104 ff Vii ee gs Group comparisons; Corrsjgtion postal clerk tests 35, I14-11§, 217 Vo ar tniaty So procedure in testing 159-162 i minialure 0 proficiency tests 90 prognostic tests 90 w record sheet 223-224 v : reliability 78-79, 166, 205-206, 208 Wages inefficient employees 38-39, 220 resemblance to job 82-83 as i Shon 33-39 scoring 116-120 We ay G.B i simplicity : 79 Watson, J 'B. 39 sources of material 83-86 WAL E 3s technique of test construction 89 ff Wechaler o 4 107-100 $951 jug conditions B37 Weighting 116-117, 187, 104-196, 201-214 time limit tests 0s critical sections 203-206 verbal tests 06 ff examination items 194-196, 201-202 will-temperament tests 118-120 ir and wrong answers 116-11 2 187 word-naming tests 100 Weill Sores 2002 5 : T work-limit lests : 95 Wells, F.L. 60, 84-83, 99, 102 etrachoric correlation 179-182, 188 Wembridge, H. A 216-217 Theoretical implications of research West, C. J. 164 procedure 8-9 Whipple, G. M. 84-85, 89, 99-100 Thomson, G. H. 164, 177, 185 Whitley, I. »2 Thorndike, E. L. 118, 123, 134, 139, 164 Wilder, H. H. 8s Thurstone, L. L. 93, 95 ff, 117, 164, Will-temperament tests 118-120 a * 174-176, 187, 198-199, 201, 2II Food Bi : al 249 it ,E.B. 8 h, R. S. » 99, Te & on A 5 8 . 88 Word-naming tests 100 LO0D8; Sa A 14, 59,15, 174-275, 1 Workers to be studied Total scofes Rosas see Subjects of the investigation group comparisons 203-206 World Book Company 84-85, 100, 175 see also Weighting Trabue, M. R. 84 Y Trade status 41 Yellow Gab Company 43-46, 230 Training 34, 62-63, 219 erkes, XK. 0. °, 35, 59 Ezivsis of 62-63 Fokus, & Ss. a nies cost o 219 , Go Ul , ’ time required 34 173, 177, 185, 187-188, 208-211 Traveling expenses 219 Turnover 39-40, 220, 231 , as criterion 39-40 Zimmerman, E. “0 ® 4 9 THE EXAMINATIONS AT WORK 27 an inquisitor but also an instructor and a salesman. There ) is an artistry of interviewing which involves more than skill ’ in extracting information. : A sound psychology carefully avoids the use of inquisito- ; rial practices in the interview or of test procedures bung- lingly handled by novices in a way to defeat the creation in the applicants of the right attitudes and motives. The leaving interview, likewise, is not complete unless it performs the same three functions—getting information, giving information, and making a friend. The employment manager in the modern personnel de- ! partment usually designates his most skilled interviewer to talk with leaving employees. He sees both those who are discharged and those who leave voluntarily. The foreman sends to the employment office, along with the time slip, a form on which he states the cause of separation. But often he does not know the real cause. It is the duty of the leav- ing interviewer to get underneath the surface and to find out if possible the essential circumstances which led to the final break. This is not an easy task. The employee, having decided : to quit, wants to get away with as little delay as possible. : So he is apt to give the most plausible reason for going, rather than to speak frankly about matters which might tend to prolong the interview. He says he is leaving for a better job or is going back to school, when the truth is that ) he does not know what he is going to do, but sees little chance for advancement where he is, or feels that the in- spector who has been passing on the quality of his work has SI not been treating him fairly. It takes a shrewd, sympa- thetic, sensible questioner to get at the truth. Such an inter- viewer is often able to uncover conditions which are remedi- able and to save the employee to the company. Sometimes he is able to make the employee see that he is acting rashly or unwisely; that the job to which he is going, while paying nominally a better rate, has disadvantages which more than outweigh the difference in pay. If the leaving interviews are Z27