A Job Element Approach: The Entry-Level Social Worker Class in State and Local Jurisdictions m United States Civil Service Commission SUNY AT BUFFALO Bureau of Policies and Standards THE LIBRARIES DEPOSITORY COPY _9260 Technical Memorandum 76-10 A JOB ELEMENT APPROACH: THE ENTRY LEVEL SOCIAL WORKER CLASS IN STATE AND LOCAL JURISDICTIONS Kyle Spivey Test Services SectionPersonnel Research and Development Center U. S. Civil Service CommissionWashington, D. C. 20415 A JOB ELEMENT APPROACH: THE ENTRY LEVEL SOCIAL WORKER CLASS IN STATE AND LOCAL JURISDICTIONS ABSTRACT The paper describes procedures and various considerations involved in conducting job element studies for the entry level Social Worker class in State and local jurisdictions. It is ba$ed on extensive study of Ernest S. Primoff's job element methodology and pilot studies for the class in Maryland and Virginia. Specifically, the manual addresses: assembling the job element panel; generating KSAPs; rating KSAPs; determining elements and subelements; placing subelements with their appropriate element; and item writing procedures and considerations. The paper may serve as a basic guide for jurisdictions conducting job element studies for this entry level class. PREFACE The purpose of this paper is twofold: (1) to discuss various considerations in assembling, conducting and coordinating item writing job element panels for the entry level Social Worker class in State and local governments; and (2) to set down specific directions (in script form) for the measurement specialist to follow when conducting job element sessions. The paper has been reviewed by Ernest Primoff and is based on extensive study of his job element methodology. In February and March, 1975, the States of Maryland and Virginia convened job element panels composed of superior Social Worker incumbents and supervisors to generate elements and subelements and to write test items which directly measure subelements. Experiences with the methodology in these States are also discussed in this paper. The first part of this paper will address the first purpose: 1. Assembling the job element panel 2. Generating knowledges, skills, abilities, and personal characteristics (KSAPs) 3. Rating knowledges, skills, abilities, and personal characteristics 4. Determining elements and subelements 5. Placing subelements with their appropriate element 6. Item writing procedures and consideratins Directions for conducting the job element sessions will delineate the procedures involved from beginning to end and comprise the second part of the paper. PART I Considerations Assembling the Job Element Panel There are several considerations which are involved in success fully convening a job element panel. First, it is important to coordinate the job element study with the State or local Agency which employs the class that is being studied. Second, the panel of job knowledge expertsl should reflect various geographic, racial, and functional areas in the State or local jurisdiction. Third, time and location considerations must be addressed. In the State of Maryland, the job analysis and item writing procedures using the job element methodology were coordinated with the Director of Social Services. It was helpful to gain the confidence and interest of a high-level Agency official. The Director was familiar with the various functional areas in which entry level Social Workers (Social Work Assistant I) were involved: for example, Child Welfare, Family Services, Group Homes, and the WIN Program. To insure a thorough job analysis, it is necessary that the areas where Social Work Assistants I are employed be appropriately represented. The Director was also acquainted with the structure of the Department of Social Services in the State. Knowledge of structure and resources assured that the various geographic areas (reflecting urban-rural and socioeconomic differences) and various sized offices were represented by panel members who had experienced these differences. Most importantly, the Director was instrumental in locating superior job incumbents and supervisors in the Department of Social Services. Although the job element methodology and resulting item writing session was explained to the Director in order to justify the panelmembers' time, it was important that the panel members themselves make no special preparation. It was sufficient to write to panel 1The panel is composed of superior job incumbents and supervisors. The entry level Social Worker class involved here requires a Bachelor's degree in any discipline. States often have two entry level Social Worker positions, one that requires a BA in Social Work and one that does not. The position being studied is the latter, and should not be confused with the position requiring a BA in Social Work. members informing them that they had been selected to participate in a job analysis (of the entry level Social Worker position) and item writing session because their agency had identified them as superior workers. A courtesy letter was also sent to each panel member's immediate supervisor stating the general nature of their employee's participation. In Maryland the job element panel was composed of five superior job incumbents and seven supervisors who had extensive knowledge or who had been Social Work Assistants I. 2 Panel members' qualifications, relevant work experience, education, and present position should be collected for documentation purposes. It should be emphasized that this information will be properly protected and will be used only for this purpose. The Maryland panel was convened in Baltimore on Wednesday, February 19, 1975 to generate and rate knowledges, skills, abilities, and personal characteristics (KSAPs) relevant to performing the duties of the entry level Social Worker position (Social Work Assistant I). This session lasted five hours. The following week, Monday, February 24, through Friday, February 28, the panel reconvened to place subelements with their appropriate element(s) and to write items which directly measure subelements. In the interim (Thursday, February 20 and Friday, February 21), the values for elements and subelements were computed from the panel's job element blanks. (A copy of the Job Element Blank is included as Appendix A.) It is suggested that sufficient time be reserved to compute the ratings before item writing sessions begin. In Virginia several panels were convened in various locations to generate and rate KSAPs for the entry level Social Worker position (Social Worker Trainee), rather than convening one panel in one location as was done in Maryland. Having several panels involved presented several methodological problems which still need to be addressed. First, what was to be done with elements and subelements which were generated in one location and were not generated in another lor.ation? Second, job element panels were convened in Norfolk/Chesapeake (southeast) and Farifax County (northeast). The KSAPs generated in these two locations were to be sent to the western part of the State to be rated, in an effort to represent the various geographic areas in the State. This introduced a problem: How could the social workers in the western part of the State responsibly rate KSAPs generated elsewhere when they had not been involved in the brainstorming session where panel discussion clarifies what is meant by a particular KSAP? Third, who would be responsible for writing items? If items were written based on common elements and subelements derived from several 2It is strongly advised that a worker and his/her immediate supervisor are not on the panel together. panels, would the meanings of the elements and subelements be clear to the item writers? Fourth, if the item writing panel was composed of social workers from one geographic location (which was likely since one panel representing various geographic locations was origi nally impossible due to lack of travel funds), this would put an undue pressure, in terms of staff hours on the job, on a particular agency in the State. Also, the items would reflect only the job experiences of one geographic area. Based on these two different experiences in assembling job element panels, it seems much more appropriate to convene one panel in one location to generate KSAPs, to rate KSAPs, and to write items. However, one must be certain that geographic, racial, socioeconomic, and functional area considerations apparent in the job across the State are represented on the one panel. Generating Knowledges, Skills, Abilities, and Personal Characteristics (KSAPs) Once the panel is assembled they are asked to compile an exhaustive list (usually about 200 suggestions) of the KSAPs which are involved in performing the entry level Social Worker position. They are selected for this task because they are well acquainted with the job and because they represent the agency's superior workers. It should be stated that the panel's purpose is not to generate task statements, but to suggest the underlying knowledges, skills, abilities, and personal characteristics which are involved in successfully performing various job tasks or the job as a whole. The session is conducted as a brainstorming session: panel members informally suggest KSAPs. The measurement specialist records all suggestions3 and should not be concerned if the panel's suggesti~ are somewhat redundant. In other words, the measurement specialist should accept suggestions without qualification, since the panel is familiar with the job and is an excellent source of information. If the panel seems to "dry up" at some point during the session, the measurement specialist may probe according to the Directions for Conducting, Part II, of this paper. It is important that the panel really think about the job they are concerned with. Books, papers, and preconceived ideas should be discouraged. If any panel member enlists the aid of such, s/he should be reminded that his/her familiarity and experience with the job is more valuable, and more important, than what a book says. 3The exception is a credentials or license suggestion, i.e., "four years of college". Do not record this, but ask what sort of knowledge, skill, ability,or personal characteristic is manifested by this credential which is important to successful job performance. Once the panel genuinely seems to have exhausted KSAPs, the measurement specialist should go back to the beginning of the list and ask for further clarification and specificity for each of the original suggestions. For example, in Maryland an original suggestion was Understanding of Class and Enviromental Differences. This was further specified as: Ability to Understand and Use for Client's Good His Values, Biases, Beliefs; Knowledge of Attitudes of Working Class Versus Non-Working Class; and Understanding of Life Styles of Ethnic Groups. Rating Knowledges, Skills, Abilities, and Personal Characteristics (KSAPs) Specific instructions and directions for rating the job element blank are included in Part II of this paper. It is necessary that each panel member individually rate the blank for each KSAP (so that a finer assessment is made) after the meanings of each rating (+, ~ 0) have been explained in each of the four columns, "Barely Acceptable," "Superior," "Trouble Likely," and "Practical." Determining Elements and Subelements Computer programs to calculate the values of elements and subelements are available from the U. S. Civil Service Commission for the following equipment: IBM 360, 370 and UNIVAC 1108. The job element blanks may also be hand scored. The value for Total Value is computed to determine elements. KSAPs with a Total Value (TV) of 1.00 or greater are considered elements. Elements are defined as broad, general knowledges, skills, abilitie~or personal characteristics. Since elements have considerable "spread" in terms of range of ability, they are difficult to measure precisely. Hence, items are Written based on subelements. Total Value (TV) is computed by summing all panel members' Item Index (SP+T), summing all Superior (S) ratings, subtracting all Barely Acceptable (B) ratings, and subtracting Practical (P) ratings. The results are then divided by 4N (where N = the number of panel members). t (IT + S -P) Thus: TV= 4N where + = 2 v= 1 0 = 0 and where N= the number of panel members 5 Subelements are those KSAPs with an Item Index CIT) of .50 but a Total Value (TV) not greater than .99. When determining subele ments. the Barely Acceptable column should be also considered. CBA =~). -2N If Barely Acceptable is less than .50, caution should be usedif it is included as a subelement. Caution is suggested because even barelyacceptable workers do not have the KSAP (BA .50 or less), despite the factthat the KSAP is important to the job (IT .50 or greater). Item Index CIT) is computed by adding all panel members' SP + T (where+= 2, / = 1, 0 = 0) and dividing the result by 6N. Hence, IT = !: (SP + T) 6N The Training Value (TR) is a function of those knowledges, skills, abilitie~ and personal characteristics which are important to the job, but which are not generally available in the applicant pop ulation. In order for a worker to successfully perform the job s/he must be traine'd to develop this particular KSAP. Those KSAPs with a Training Value of .75 or above may be considered as a basis for developing a training course or program. The Training Value is determined by the formula TR= -r ( s + T4N+ spl -B> Note that pl is determined from the (P) column. If a panel member rates (P) as +, pl 0 as J, Pl = 1 as O, pl = 2 Based on the computed values of KSAPs a typed list of subelementsand elements should be distributed to panel members when they convenefor the week-long session of item writing. As a group, they should place subelements with each appropriate element. In Maryland, the measurement specialist drew a grid on a flip chart to facilitate subelement placement. Elements were noted by number across the top ofthe chart, and subelements were listed down the side. Each subelement was identified and placed by general panel agreement. Some subelementswere placed under more than one element. This may oc~ur when the meaning of the subelement changes in terms of the element. It is also possiblethat some of the.original elements may be consolidated into one element.This consolidation is justified if there is considerable similarity ofsubelements under elements. For instance, in Maryland three elements,Genuine Concern for the Welfare of Others, Basic Respect for Self andOthers, and Liking for People had nearly the same subelements. This was indicated to the panel, and they decided that the subelements did not have clearly different meanings in terms of the elements. Consequently, one broader element was developed to encompass the meanings of the original three: Genuine Concern and Respect for Self and Others. The subelements from the three elements were nearly the same, and were placed under this new, extended element. If elements are consolidated, it is advisable not to develop a new title, but to take the element with the highest Total Value and consolidate the subelements under it. Item Writing Procedures Items are written to directly measure subelements in terms of their greater element meaning. It is important that items reflect realistic work situations and various realistic ways of dealing with situations, rather than having a single right answer. In Maryland three items were developed by the panel as a group to familiarize them with item writing procedures. The subelement Skill in Listening, under the broader element Ability to Communicate -Written and Oral, was used to exemplify item writing techniques. The panel developed a situation based on their work experience involving a client who had insisted (for two years) that she was interested in working, but had refused two job offers. The client did not have any other responsibilities to explain her refusal. When the situation (item "stem") had been presented, the question was asked: "What is the client really saying?" Based on the situation, the panel generated several possibilities, or alternatives. The alternatives were based on their work experience and professional judgment in dealing with this sort of situation. The alternatives were ranked as "best," "good," "fair," and "poor." Using this method of ranking alternatives, rather than the one right answer format, multiple judgments must be made to answer the item correctly. The panel was also cautioned to introduce situations and require judgments appropriate for an entry level Social Worker. When it seems difficult to measure a subelement, the panel should discuss how they have applied the subelement in their work. An item format can be developed to measure an applicant's ability to apply information to particular situations. For example, in Maryland the panel members discussed how they had applied the subelement Ability to Learn Programs and Community Resources. They agreed that they had learned to distinguish among various programs. Therefore, an item format was developed to see how well the applicant could do this. Four programs were listed and briefly defined: foster care, day care, adoption services, and children's guild. Each program was lettered A, B, C, or D. Four situations requiring the programs were then stated. Applicants were asked to match the situation and its special requirements to one of the programs. Thus, applicants were learning the program and its purpose and subsequently applying their learning to a situation. In Maryland, the twelve panel members were divided into three groups.4 It was suspected that a twelve member panel might be somewhat un manageable during the brainstorming session (but was not in actuality). Indeed, having twelve people writing items considerably increased productivity. The first day ~11 groups were assigned the same element and were asked to write items based directly on subelements and in the context of the element. However, in order to assure item coverage for all subelements, different elements and their subelements were assign ed to the various groups on days 2-5. Item writing groups were urged to observe the following considerations: 1. Does the item measure what it is supposed to measure, i.e., the subelement in the context of the element? 2. Are alternatives distinct in meaning, and do they follow from the situation without making undue assumptions? 3. Are the alternatives appropriately ranked? (Make sure yourank alternatives.) 4. Is enough information included in the stem to appropriately rank the alternatives? 5. Are the judgments required appropriate for an entry level Social Worker? Think back to your experience as an entrylevel Social Worker. 6. Is the language or technical references in the item appropriate for a beginning entry level Social Worker? There are several points the measurement specialist should consider: 1. Several subelements can be measured in one item. However, panel members should be cautioned against measuring too many subelements in one item. It is easier to include fewer subelements, making the situation and alternatives more sub element specific, the item becoming more manageable and unambiguous. It was found that including more than four subelements in an item tended to make the item vague. 4Each group was a mix of supervisors and incumbents from various geographic locations and with various functional area specialities. 2. It is a good idea for the measurement specialist to cursorily review the items produced each day to make sure the panel members understand the item writing procedure and to offer suggested improvements to the panel throughout the session. . However, it cannot be sufficiently stressed that the measurement specialist should not edit items for content, nor should any specific word changes, item shortening, or possible clarification be unilaterally stated. The measurement specialist may suggest changes, but this is not encouraged. 3. The measurement specialist should keep a running tally of subelements which have been measured to assure that all subelements are measured. 4. All items should be reviewed by all panel members according to the criteria by which they were written. Criticism should be put in writing. During the item review session panel members were asked to do this as follows: a. Review item according to the writing criteria (restate these for the panel on a flip chart). b. Please do not comment merely that an item is poor. Rewrite or revise the item so it is better. c. Please respond in writing to criticisms previously noted. d. Can you suggest other possible alternatives for a situation? These alternatives may fall under any of the rank classifications (B, G, F, P). Please note them on the page and rank them, i.e., "other possible alternatives''· B 1________________ p 2________________ e. After the item has been reviewed by panel members it should be returned to the original writers for revision and clarification, according to the criticisms. If the original writers do not agree with the criticisms, they should note strong justification for their disagreement. In Maryland, five and a half days were required to generate 181 KSAPs, rate KSAPs to determine elements and subelements, and write 96 items. The generating and rating session took 5 hours. One half day was spent placing subelements with the elements, 3~ days were spent writing items, and the panel spent one full day reviewing items. Five and a half days was a reasonable time to perform the task. How~ ever, if a supplemental questionnaire is developed an additional two or three days should be allocated to reconvene the panel. Specific guidelines for developing a supplemental questionnaire are not included in this paper because the State of Maryland is currently involved in this effort. However, guidelines will be made available by June, 1975. Weighting, scoring, statistical and validation procedures will be discussed in the future. PART II Directions for Conducting Job Element Panel: Generating Elements and Subelements and Conducting Item Writing Sessions Materials required: Flip chart Typist to type KSAPs during the generating session Sufficient number of job element blanks Duplicating facilities When the panel is assembled, the measurement specialist should say: "Each of you has been selected to participate in this project to develop an exhaustive list of knowledges, skills, abilities,and personal characteristics required to successfully perform the entry level Social Worker position (say exact name of this class in your location). You have been selected because you have been identified by your Agency as a superior worker: in order for you to generate these knowledges, skills, abilities,and personal characteristics, think about superior workers you have known -what were they like? I am here to record your suggestions. This session will be conducted as a brainstorming session: you should make suggestions as they come to mind. Do not be concerned if some of your suggestions are repetitive. If a suggestion is made which you disagree with, you should suggest your opposing idea. I will write down all suggestions, and you will each have the opportunity to rate all the knowledges, skills, abilities, and personal characteristics which are generated. I would also like to say that we are not concerned with the specific tasks that must be done to perform the job. Rather, we are concerned with the underlying knowledges, skills, abilities,and personal characteristics which are needed to perform an important task or to carry out the job as a whole. "You might think of people you have known who you consider superior workers. What knowledges, skills, abilities, and personal characteristics did they have which made them superior? Or, consider barely acceptable workers you have known. Which knowledges, skills, abiliities,and personal characteristics did they lack? In other words, what makes an entry level Social Worker (specify title) superior? What gives a worker trouble if it is missing? ''Also, remember that the job we are concerned with is an entry lev el position. What sort of interests or willingness should a person have to become a superior entry level Social Worker (specify title)? What sorts of knowledges, skills, abilities,or personal character istics should he or she have the Ability to Learn once on the job? (Pause.) "If there are no questions, can someone suggest a knowledge, skill, ability,or personal characteristic important to the entry level Social Worker position (specify title)?" The measurement specialist should write down every panel suggestion, with the exception of specific tasks and credentials (see Part I, page 4). If the panel runs into difficulty generating KSAPs at some point in the session, the measurement specialist may say: "What knowledges, skills, abilities, and personal characteristics does a superior entry level social worker have?" !''What makes a worker weak?" at sort o aracterisdo some When the panel seems to have completed a fairly exhaustive list, usually about 100 KSAPs, the measurement specialist says: "Are there any other knowledges, skills, abilities, or personal characteristics which have been excluded?" (Pause.) If there are, these should be written down. If not, the measurement specialist should distribute the list of KSAPs which have been generated and typed and say: "In that case, we are now interested in further specifying the know ledges, skills, abilities,and personal characteristics." The measurement specialist should read the first KSAP on the list. "How can this KSAP be further defined in terms of the job? What specific knowledges, skills, abilities,and personal characteris tics are important to include under this suggestion? How can this be further clarified or broken down?" The measurement specialist should write down these further sugges tions continuing the original list, numbering consecutively. This further specification should be done for each of the original sug gestions. Once this is completed, the measurement specialist should say: "Now we will have a short break (or lunch). When we get back you will be asked to individually rate all suggestions on a Job Element Blank. I will explain the blank to you when you return." When the panel returns, they should each be given a duplicated list of their suggestions and a sufficient number of "Job Element Blanks" to rate all suggestions. The measurement specialist should say: .,.I would now like you to rate each of the suggestions on the Job Element Blank. Notice that on the left-hand side of the blank there are four categories: Barely Acceptable, Superior, Trouble Likely, and Practical. Each KSAP should be rated in each category with a plus, check, or zero. In order to explain how to rate the blank I am going to cite two examples of knowledges, skills, abilities,or personal characteristics which are not related to the job we are concerned with. For instance, suppose we were rating the Ability to Breathe (write this down on flip chart or blackboard). This ability is rated in each of the four categories with either a plus, check,or zero, according to the meaning of these symbols in each category and taking only the Ability to Breathe into consideration.. Now, would you say all barely acceptable workers have the ability to breathe, some have this ability, or almost none do? (Pause.) I think we would agree that even barely acceptable workers have this ability. Therefore, we should put a plus in this category." "The next category is 'Superior'. This means, does the particular KSAP we are rating, in this case Ability to Breathe, differentiate the superior worker from the barely acceptable one? Does the Ability to Breathe make a person a superior worker? Would it be said so-and-so is a superior worker because s/he breathes well? So, for thi~_co1umn would you say that the Ability to Breathe is very important in distinguishing a superior worker, (plus), is the ability valuable in distinguishing a superior worker, (check), or can it be said that the Ability to Breathe does not distinguish the superior worker. (zero)?" (Pause.) Get the panel to agree that the Ability to Breathe does not differ entiate the superior worker from the barely acceptable worker. The column should be marked '0'. The measurement specialist should say: "The third column, 'Trouble Likely' if not considered, means how much trouble would we have with a worker if we did not include this particular KSAP when we develop· the written test or other selec tion device. For instance, if Ability to Breathe is not considered in our exam, would we have much trouble (plus), some trouble (check) or no trouble (zero) with a worker?" Get the panel to agree that if we did not include or measure an ap plicant's breathing ability, we would not have trouble. Mark the column '0'. 11The fourth column on the blank, 'Practical', means if we demand that applicants have the ability to breathe, can we fill all openinf.s,(plus), some openings, (check), or almost no openings, (zero).' Get the panel to agree that the Ability to Breathe is well represented in the applicant population, and that if we demanded that applicants breathe, we could fill virtually all of our positions. Mark the column '+'. Now explain the rating process a second time with another example, Ability to Saw to a Millionth of an Inch by Hand. "Let's take another example. Say we are trying to rate Ability to Saw to a Millionth of an Inch by Hand for a carpenter job. Would you say that barely acceptable carpenters can saw to a millionth of an inch?" Get them to agree that although some b~rely acceptable carpenters may be able to saw to this precision, it is unlikely that all have the ability (plus). If there is some disagreement over whether some carpenters, (check), or almost no carpenters, (zero), have this ability,remind the panel that this is an example, and that you are interested that they understand the rating process. ·~ould you say a carpenter would be superior if s/he could saw to a millionth of an inch?" Get the panel to agree that this ability would probably indicate that a carpenter would be considered superior if s/he had this abil ity (plus). If there is some disagreement, reemphasize that you are only concerned with whether or not the panel understands what the rating in a particular category means. If a panel member feels that the ability to saw to this precision would be valuable in determin ing or saying that a carpenter was superior, help the member to understand that s/he would put a check in this column. Similarly, if a member felt that this did not differentiate a superior worker, make the person understand that the column should be marked 'O'. Then proceed to the Trouble Likely column: !"-How would vou rate the 'Trouble Likely' column for this ability?" Have the panel make suggestions for ratings and justify their ratings. Similarly, the measurement specialist should say: "How would you rate the 'Practical' column for the Ability to Saw to a Millionth of an Inch?" Have the panel judge the situation and ask how they would rate the ' ability based on their judgment: is it practical to assume that all, some, or almost no carpenters would have the ability and consequently, could we fill all, some, or almost no positions? Once these examples are completed, the measurement specialist should say: "I would now like you to rate each of the KSAPs for the entry level Social Worker position in the same manner. Please make your ratings (plus, check, and zero) legible. If you do not understand what was meant by a particular KSAP, rate it as zero in all categories. You can only rate eighteen KSAPs per page. Make sure that the number of the KSAP you are rating corresponds to the KSAP you are thinking about. Do not leave space between KSAPs, and please rate all KSAPs in all four columns. If the meanings of the four columns on the blank become unclear, you should realize that this is common, and that I will be glad to help you. I am interested in each panel mem ber's individual assessment: do not be concerned if your neighbor's ratings are different from yours. Individual ratings help us to better assess the job. Also, please fill in the information at the top of each blank." The panel should then begin rating each KSAP in a quiet atmosphere. It is not necessary to individually consult with each panel member unless s/he requests help. When raters begin returning their blanks, it is important to scan each rater's blanksto make sure s/he has rated each KSAP in each of the four categories, that the ratings are legible, and that the in formation at the top of the page has been given: job, grade, rater's name and grade, title and location. If the rating blanks seem to be in order, the panel member's blanks should be stapled together and s/he may leave. S/he should be thanked and reminded when the panel will reconvene: time, location, number of days. Reconvening the Panel for Placement of Subelements and Item Writing Session Hand out a duplicated list of elements and a list of subelements. Each subelement should keep the number it had as a KSAP. The meas urement specialist should say: "The results of your ratings have been computed to determine which of the knowledges, skills, abilities,and personal characteristics you generated at the last meeting are most important in successfully per forming the entry level Social Worker position (specify exact title). The list titled "Elements" are those broad KSAPs required to do the job. Subelements are more specific KSAPs. I would like you as a group to place each of the subelements with their most appropriate element." (Turn to flip chart.) Subelement Element 1 2 3 . . . 1 2 3 4 7 12 I "The first subelement (# ) most appropriately specifies which element?"! Note to measurement specialist: the panel may feel a subelement may belong under more than one element. In this case, place the subelement under both (or several) elements as the panel suggests. When the placement of subelements is completed and the list has been completed and typed, the panel should review the placement. The measurement specialist should say: I 11Are there any elements which share several subelements?11 If so, the measurement specialist should find out whether the panel thinks the elements are identical. If both (or several) elements seem to represent the same thing, the separate elements should be consolidated into one element. If there is duplication of subele ments between (among) elements, but the panel does not feel ·that a consolidation should be made, the panel should contrast how the sub element variously changes the meaning of the element, or how the subelement is construed differently in terms of the various elements. After the subelements are placed and any consolidation has taken place, the item writing session should begin. 111 would like to explain how the item writing sessions should pro ceed. Items are written to directly measure subelements, therefore, the items you write should directly illustrate the subelement you are concerned with. The element which the subelement specifies should also be considered. What does the subelement mean in terms of its broader element?11 All items should be written in terms of the following criteria: (Write down the criteria, see Part I, pages 7-8.) 11 For the entry level Social Worker position I would like you to write items according to the following format.l First, describe a situation based on your work experience which directly illustrates the subele ment. (Measurement specialist should choose a subelement as an example.) For example, can you remember and describe a situation where you needed to have this (say the particular KSAP)?11 Wait for panel suggestions, ask them to thoroughly define the situation. Write the situation down on the flip chart. lrf a subelement is stated as an Ability to Learn or has a high training value the panel should also develop item(s) as a group according to the format explained in Part I, page 7. '~iven this situation, we need to generate four alternatives which illustrate how the worker might deal with the situation. Then rank the alternatives as 'best', 'good', 'fair', and 'poor' -that is, which is the best way for a worker to deal with the situation, then which is a good way, a fair way, and a poor way. The alternatives should illustrate or measure the subelement that you are concerned with." Get them to generate alternatives and rank the alternatives as "best", "good", "fair", and "poor". "I should say that more than one subelement may be illustrated in an item. However, you will find that if you attempt to include too many subelements in an item it may become rather vague and unmanageable. We are interested in developing items which are clear and which portray situations which an entry level Social Worker (specify position) would confront on the job. Are there any questions?" If the panel is still unclear about what is to be done, have them choose another subelement and have them develop a situation and possible alternatives in terms of the specific subelement and the broader element. Then assemble the panel into smaller working groups to write items and assign different elements and their subelements to the different groups. Item Review The measurement specialist should say: "I would like you to review all items according to the criteria by which they were written. Please clearly explain your problems with the item. If you feel a situation is unclear, specify this, suggest how the situation might be made clearer. If you feel that an alternative is poor, justify your criticisms and suggest a better alternative. If you feel that the alternatives should be ranked different!~ say why. Also, if you can suggest other plausible alternatives, write them down. Be careful not to make word,or phrase revisions which change the entire meaning of the item. "Be very critical of the items. We have found that constructively criticizing items makes the items better and clearer. Are there any 1uestions?" After the items have been reviewed, they will be returned to the original group who wrote themo The original writers should then make necessary revisions of each item if they believe the criticism is justified. If the original writers do not agree with the criticisms, their disagreement should be noted. When the review session has been completed the panel members should be thanked for participating in the project. Cordial letters should be written to panel members' immediate supervisor, commending the member's participation and thanking the supervisor for allowing the member to participateo * U. S. GOVERNMENT PRINTING OFFICE : 1976-627-915/1105 APPENDIX A JOB ELEMENT BLANK Page No. (col. 1 2 ) Job: Rater Name and Grade: Grade: Title and Location: Rater No Job No . 8 ) I (These columns for use in hand calculation of values) -""-· Barely acceptable To pick out Trouble likely if Practical. Deman-Item Total Training Element workers superior workers not considered ding this element, lndex Value Value No, we can fill P' (B) (S) (T) (P) Columns S X P T (IT) (TV) (+ = 0 SP1 (TR) (Do not +all have + very important + much trouble + all openings J = 1 Punch) J some have j valuable J some trouble j some openings SP + T IT+ S 0 = 2) S + T • 0 almost none have 0 does not 0 safe to ignore 0 almost no -B-p sr1 B -differentiate openings I --- 9-12 13-16 17-20 I i i---------- 21-24 25-28 I 29-32 33-36 - 37-40 41-44 - 45-48 I --· --1---------------------t----------·-. ------·· ··-··-----·4g-s2 lf 53-56 57-6n 61-64 ----t 65-_68 l 69-72 I 73-76 77-80 u.s. Civil Service Commission Note: for all categories except P', + crunts 2, J counts 1, 0 counts 0,Personnel Research and Development Center For category P', +counts O, J counts 1, 0 counts 2. Washington, D.C. No; I