Foreign Relations of the United States, 1977–1980, Volume XXVIII, Organization and Management of Foreign Policy

163. Information Memorandum From the Director General of the Foreign Service and Director of Personnel (Barnes) to the Deputy Under Secretary of State for Management (Read)¹

Washington, June 7, 1978

SUBJECT

• Proposed Modification of the FSO Oral Examination

As you know, there has been considerable discussion in recent years about the appropriateness of the present FSO selection system. It has been contended, especially during the last few years when the FSO intake has been relatively small (less than 200 yearly) that a process which examines from 12,000 to 15,000 persons and selects only a few hundred is not well balanced and should be changed, especially when there are signs that those chosen may be overqualified for the type of work to be initially performed and when women and minorities seem to have less than full opportunities in this process for appointment.

We have looked closely at the process and not surprisingly have concluded that both its major components—the written examination and the oral examination—have areas that need improvement. Both are designed to fit the concept of selecting the generalist/specialist, that is a Foreign Service Officer who has the qualifications required for career entry and advancement in the Foreign Service but also has sufficient background in one or more of the functional areas, so that in the examining process his strengths and interests could be identified with one of these fields. In this way, the overall selection process has been expected to maintain a balanced intake of officers in line with the Department’s needs. On the written examination side, since 1975 the Educational Testing Service (ETS), through a job analysis and with input of Foreign Service Officers in each functional field, has reviewed the knowledge and skill requirements for this generalist/specialist concept and formulated the composite test specifications by which each written examination has been prepared and utilized. This process is being repeated this year to make sure the written examination is as accurate as possible in screening our applicants.

On the oral exam side there has been the formulation each year of a store of job-related questions in each functional area plus those for the Cultural and Americana areas by which three-member examining panels differentiate candidates in terms of a set of defined characteristics. These characteristics, called job elements, are derived through research conducted by ETS and are those judged to be essential to effective performance. The job elements are defined as knowledge, skills, abilities and personal characteristics and the premise is that those candidates who possess the characteristics will be more effective as an FSO than those who do not possess them. Thus each candidate is measured or assessed for these characteristics and a score is given to indicate the degree to which the candidate demonstrates them.

The theoretical concept of this process is professionally sound and well accepted but the key issue, of course, is the extent to which the examination operation can implement fully the concept. For example, the in-basket test² was included in the State process for the first time this year (ICA has used this test since 1974) because certain skills judged as highly important were being measured only indirectly or not at all. Notably, the test has since been judged to be highly useful. Given the time constraints in the form of an April deadline for renewing a contract option with ETS this year, we decided against major changes in the written examination. There is, however, time leeway with regard to the oral and we have pressed the question “Is the present one hour examination interview a valid predictor of performance of Foreign Service Officers?”

The answer given by professional psychologists and in-house personnel generally is “yes-but”. Specific criticisms of the present oral include: that too much of the selection decision rests upon a one hour oral examination; that not enough time is given to the personal interview during this hour; and that the validity of hypothetical questions designed to test interpersonal and operational skills is dependent upon having a common criteria for evaluating the responses and on the questions being administered in a standard format and that if these conditions do not obtain, the results are skewed. We also noted that the oral examination has not been a constant entity from year to year, with the examination precepts having been considerably modified and improved from those of three years ago. Last March we asked ETS to formulate proposals which would improve the oral examination process and ETS has done so in a way that looks promising, effective and realistic.

The proposed oral evaluation process uses what is called an assessment center type format that incorporates a self-inventory exercise, a one-on-one interview, and a leaderless group discussion exercise and a prepared presentation for each candidate. This will subject the candidate to the observations of Deputy Examiners or assessors for a total of about 3½ hours. The process which lasts the whole day is one of overall evaluation based on the current job analysis of Foreign Service work and knowledges, skills, abilities and personal characteristics judged to be important in the performance of that work. The format meshes closely with the in-basket test and would involve evaluation of a candidate’s qualifications based on his performance in the group exercise and individual interview. No selection decision will be made until all data are in, which because the scoring of the in-basket is by ETS, would take a maximum of two weeks. The proposal calls for 3 examiners assessing 6 candidates every other day with 3 or 4 groups being conducted simultaneously in Washington and one group every other day for travelling panels in other cities.

After studying the proposed format we consulted with Civil Service Commission experts as well as officials at other U.S. Government agencies that utilize the assessment center format, specifically the Department of Interior and the Federal Aviation Administration, the latter being the largest user of the assessment center format in the U.S. Government. Uniformly we have been told that their experience with this format has been entirely favorable and successful. Not only does it provide a better measurement of candidates but research indicates no adverse impact on women and minorities. Use of the assessment center techniques has also been sustained in several court cases. At the same time it is clear that none of these organizations has used this format for entry level selection and on the scale we envisage for about 1,500 candidates. Virtually all these organizations use the format for selection of manager or executive level personnel and in this sense we would be plowing virgin ground. In reviewing the various factors and aspects involved, we believe the following are the advantages to be gained:

1. Clearly we will be able to assess candidates better, because more information on the individual will be available and the candidate will be observed in more situations. More of the key characteristics will be measured directly, for example in the interview, the prepared presentation and the group or negotiation exercise.

2. We estimate that the number of candidates to be assessed in a given time period with the same number of examiners will be roughly equivalent or possibly slightly higher than at present.

3. More information will be available to those responsible for the placement function. The personal inventory/checklist will be a particularly useful aid in determining patterns for the various functional fields.

4. In general, there have been no challenges of adverse impact in relation to assessment programs. Women and minorities appear to do as well as others. (This was mentioned by both the Department of Interior and FAA officials.)

5. From past research and experience, this format is probably more acceptable to candidates because they feel they are given more opportunities to show their qualifications.

6. An interview focussed on the in-basket problems will give the candidate and the interviewer the advantage of a “real” setting on which to base their discussion. The stock of situations now presented in the oral panel, which sometimes may become known to other candidates by word of mouth, will not be a problem. (Various experts discount the possibility of “coaching” affecting the validity of this type of assessment format.)

Against this list of advantages, we see two disadvantages, namely that (a) whereas each candidate at present is informed immediately after the oral examining panel finishes its deliberations as to whether the decision is favorable or unfavorable, the candidates must wait for two weeks under the proposed format and (b) related to the foregoing, the administration and processing of applications will be considerably more complex and detailed.

As for resources needed to implement the proposed format, we estimate that a staff of 8 permanent Deputy Examiners (as proposed in the REE reorganization) would be needed plus about 15 TDY examiners for the new format as opposed to 10 permanent Examiners and up to 14 TDY examiners for the present procedure.

However, all quarters have underscored that centralized and thorough training of assessors is critical for successful operation with this format. Personnel staffing will be considerably more difficult, since at present examiners can be drawn on short notice from a large pool of former BEX examiners and from the line bureaus for a few hours. The proposed format will require a pool of perhaps 18 persons who could act as ad hoc examiners to supplement the regular and TDY examiners. In financial resources, a fee of $23,000 would need be paid ETS for instituting this process. This would include designing the test, producing test materials, training the examiners and subsequent monitoring. Rental costs for travelling panel facilities under the proposed format would probably run about $25,000 more than the present program since the proposed format has more exercises and we would need more rooms than we presently do. However, we need to examine the schedule and test procedures more closely to see if we can reduce this; I am also hopeful that by earlier and firmer approaches to GSA we can get U.S. Government facilities at all rather than most of the cities as at present.

In further explanation, there is attached the ETS proposal for this program and a matrix which shows where each desired characteristic is measured in the selection process.³

I note that the International Communication Agency has indicated its willingness to join with State if the proposed format is adopted.

In sum, while a number of aspects in the proposed format are new and while there undoubtedly will be an initial period of adjustment and experimentation, we believe the proposed modification will provide for a more effective, accurate, and defensible selection process.

Through the release of funds previously deferred for the restructuring of the FSO examination, we are able to fully fund this modification.

1. Source: National Archives, RG 59, Records of the Under Secretary for Management (M), 1978–1979, Box 1, Chron June 7–9, 1978. No classification marking.
2. The “in-basket test” was a portion of the oral examination for Foreign Service officers. Candidates were given an “in-basket” that contained an assortment of items typically handled at post. The candidate had to prioritize the order in which the items would be addressed.
3. Attached but not printed.