|
Not only are the references to content validity scattered throughout
the Guidelines, but there are actually two varieties of content validity
discussed in the Guidelines. I shall refer to them as "classic" and
"extended" content validity. In addition the standards that are applied
to them are, in some instances, substantially different. This has, not
surprisingly, led to much confusion. After pointing out (in the Supplementary
Information Section on page 38292) that:
There are three concepts which can be used to validate a selection
procedure. These concepts reflect different approaches to investigating
the job relatedness of selection procedures...
a short straightforward definition of "classic" content validity is given:
In content validity, a selection procedure is justified by showing
that it representatively samples significant parts of the job, such
as a typing test for a typist."
(All verbatim quotations from the Guidelines are indented and shown in bold type.)
The key idea is that the test is constructed by taking a "representative
sample." Thus in classic content validity the content of the test is the
same as the content of the job. This is, of course, the source of the name.
Sameness is about as strong as a relationship can get!
Later in the section, in an "Analysis of comments" subsection the adopting
agencies responded to the comments that were received in response to the
publication of a draft of the guidelines. The genesis of what I am calling
"extended" content validity is explained (on page 38295) as follows:
8. Content validity. The Division of Industrial and Organizational
Psychology of A.P.A. correctly perceived that the provisions of the
draft guidelines concerning content validity, with their emphasis
on observable work behaviors and work products, were "greatly
concerned with minimizing the inferential leap between test and
performance." That division expressed the view that the draft
guidelines neglected situations where a knowledge, skill or ability
is necessary to an outcome, but where the work behavior cannot be
replicated in a test. They recommended that the section be revised.
We believe that the emphasis on observable work behaviors or
observable work products is appropriate and that in order to show
content validity, the gap between the test and performance on the
job should be a small one. We recognize, however that content
validity may be appropriate to support a test which measures a
knowledge, skill or ability which is a necessary prerequisite to
the performance of the job, even though the test might not be
close enough to the work behavior to be considered a work sample,
and the guidelines have been revised appropriately. On the other
hand, tests of mental processes which are not directly observable
and which may be difficult to determine on the basis of observable
work behaviors or work products should not be supported by content
validity.
Because I had chaired the committee that wrote the California Guidelines
on Employee Selection Procedures, I was consulted by several of the people
in the adopting agencies who were responsible for writing the federal
guidelines. I can thus state based on direct knowledge that the concern
about extending the definition of content validity to cover knowledges,
skills and abilities (KSA's) was very great. They felt that they had to
do it, but they were fearful that it would be badly abused. A substantial
amount of the effort required to produce the final draft was devoted to
the attempt to define a line between acceptable and unacceptable uses of
content validity.
By bringing together the UGESP provisions that apply to each type of
content validity I believe that it is easier to understand what the
requirements are in each case.
Classic Content Validity
In the technical standards section (14C) the guidelines make clear
that those choosing a classic content validity strategy (page 38302):
should determine whether it is appropriate to conduct such a study
in the particular employment context. A selection procedure can be
supported by a content validity strategy to the extent that it is
a representative sample of the content of the job.
Thus using the classic approach is appropriate only if a test can be
constructed by taking a representative sample of job content. The
standards for demonstrating classic content validity are given in
14C(4):
To demonstrate the content validity of a selection procedure,
a user should show that the behaviors demonstrated in the selection
procedure are a representative sample of the behavior(s) of the job
in question or that the selection procedure provides a representative
sample of the work product of the job.
An example of a work product would be a properly welded angle joint as
one item in a sample drawn from a welder's job which is then used in a
test for selecting welders. Work behavior is defined in the definitions
section (16Y) as follows (page 38308):
An activity performed to achieve the objectives of the job. Work
behaviors involve observable (physical) and unobservable (mental)
components. A work behavior consists of one or more tasks.
Knowledges, skills, and abilities are not behaviors, although
they may be applied in work behaviors.
The key word is activity. A work behavior is something that the worker
does. Another part of the standards for demonstrating classic content
validity require that all aspects of the test closely resemble the job
(page 38302, column 3):
If a test purports to sample a work behavior or to provide a sample
of a work product, the manner and setting of the selection procedure
and its level and complexity should closely approximate the work
situation. The closer the content and the context of the selection
procedure are to work samples or work behaviors, the stronger is the
basis for showing content validity. As the content of the selection
procedure less resembles a work behavior, or the setting and manner
of the administration of the selection procedure less resembles the
work situation, or the result less resembles a work product, the
less likely the selection procedure is to be content valid, and the
greater the need for other evidence of validity.
The documentation section (15C) defines the elements of the showing that
is required (page 38305). A job analysis (15C(3))must be provided:
The work behaviors, the associated tasks, and, if the behavior
results in a work product, the work products should be completely
described (essential). Measures of criticality and/or importance
of the work behavior(s) and the method of determining these
measures should be provided (essential).
The selection procedure and its content (15C(4)):
should be completely and explicitly described or attached (essential).
The documentation of the relationship between the selection procedure
and the job (15C(5)) requires, among other things, that (page 38305):
The evidence demonstrating that the selection procedure is a
representative work sample [or] a representative sample of the
work behavior(s) ... should be provided (essential). The user
should identify the work behavior(s) which each item or part
of the selection procedure is intended to sample or measure.
The theme that unifies all of the content validity documentation
requirements is that the user is expected to provide the detail
and specificity needed to clearly relate the content of the test
to the content of the job so that the "inferential leap" is very
small.
Extending Content Validity to Knowledges, Skills, and Abilities
The task of extending content validity that faced the adopting agencies
was daunting indeed. The problem was to include KSA's in the situations
where content validation was appropriate and to require criterion-related
validation in the situations where the "inferential leap" from test content
to job performance was "too large." To turn what is in fact a sophisticated
professional judgment into regulations was to some extent a "mission impossible."
It begins with the standard (14C(1)) on appropriateness:
Selection procedures which purport to measure knowledges, skills, or
abilities may in certain circumstances be justified by content validity,
although they may not be representative samples, if the knowledge, skill,
or ability measured by the selection procedure can be operationally
defined as provided in section 14C(4) below, and if that knowledge,
skill, or ability is a necessary prerequisite to successful job
performance.
Two conditions must be satisfied if KSA's are used. The first is that
the KSA can be "operationally defined" using the restrictive standards
in 14C(4) and the second is that the KSA is a necessary prerequisite
to success on the job. Another method used "to put a fence around KSA's"
as it was referred to at the time was to define the three terms in a much
more restricted way than their standard dictionary definitions (Section
16 on page 38307):
M. Knowledge.
A body of information applied directly to the performance of a function.
T. Skill.
A present, observable competence to perform a learned psychomotor act
A. Ability.
A present competence to perform an observable behavior or a behavior which results in an observable product.
O. Observable.
Able to be seen, heard, or otherwise perceived by a person other than the person performing the action.
Notice that skill is restricted to psychomotor acts. A dictionary
definition of psychomotor is: "Of or relating to movement or muscular
activity associated with mental processes." The intent is to clearly
exclude purely mental or social skills which must be validated by
criterion-related strategies. Also note that skills and abilities are
limited to observable events which must be perceived rather than inferred.
Many of the abuses of content validity are attributable to the use of
broad dictionary definitions of KSA's which, if accepted by the adopting
agencies, would allow claims of content validity in almost any situation.
In many of these situations where content validity is inappropriately used,
a criterion-related study would show that the test is, in fact, not
job-related. This, of course, is why criterion-related strategies must
be required for KSA's that involve inferences. The inferences may be
incorrect.
The appropriateness standard (14C(1)) concludes with a paragraph
describing some of the instances where extended content validity is
not appropriate and therefore not acceptable:
A selection procedure based upon inferences about mental
processes cannot be supported solely or primarily on the
basis of content validity. Thus, a content strategy is not
appropriate for demonstrating the validity of selection
procedures which purport to measure traits or constructs,
such as intelligence, aptitude, personality, commonsense,
judgment, leadership, and spatial ability. Content validity
is also not an appropriate strategy when the selection
procedure involves knowledges, skills, or abilities which
an employee will be expected to learn on the job.
In the section on standards for demonstrating content validity
(14C(4)), the same principles are continued in the attempt to
restrict the "inferential leap."
In the case of a selection procedure measuring a knowledge,
skill, or ability, the knowledge, skill, or ability being
measured should be operationally defined.
"Operationally defined" has a clear professional meaning.
"Operationalism" is one of the major approaches to the issues
that are involved in properly defining what is meant by a
"scientific method." A dictionary definition of operationalism
is, "The view that all theoretical terms in science must be
defined only by their procedures or operations." First the
operational definition of knowledge is given:
In the case of a selection procedure measuring a knowledge,
the knowledge being measured should be operationally defined
as that body of learned information which is used in and is
a necessary prerequisite for observable aspects of work
behavior of the job.
Skills and abilities are treated together:
In the case of skills or abilities, the skill or ability
being measured should be operationally defined in terms
of observable aspects of work behavior of the job.
In addition, to be content valid, a selection procedure
measuring a skill or ability should either closely
approximate an observable work behavior, or its product
should closely approximate an observable work product.
In addition to providing operational definitions, there are
two other required showings:
For any selection procedure measuring a knowledge, skill,
or ability the user should show that (a) the selection
procedure measures and is a representative sample of that
knowledge skill, or ability; and (b) that knowledge, skill
or ability is used in and is a necessary prerequisite to
performance of critical or important work behavior(s).
Given the way that KSA's are defined and given the standards
that are applied to them, most of the requirements in the
documentation section (15C) can apply to both classic and
extended content validity. There are, however, three important
specific mentions of KSA's that impose additional requirements
beyond the requirements for classic content validity. The first
involves the job analysis (15C(3)) if KSA's are used:
Where the job analysis also identified the knowledges,
skills, and abilities used in work behavior(s), an
operational definition for each knowledge in terms of
a body of learned information and for each skill and
ability in terms of observable behaviors and outcomes,
and the relationship between each knowledge, skill, or
ability and each work behavior, as well as the method
used to determine this relationship, should be provided
(essential). The work situation should be described,
including the setting in which the work behavior(s)
are performed, and where appropriate, the manner in
which knowledges, skills, or abilities are used, and
the complexity and difficulty of the knowledge, skill,
or ability as used in the work behavior(s).
The second adds to the requirements for information regarding
the selection procedure and its content (15C(4)):
Where the selection procedure purports to measure a knowledge,
skill, or ability, evidence that the selection procedure
measures and is a representative sample of the knowledge,
skill, or ability should be provided (essential).
The third addition involves the relationship between the selection
procedure and the job (15C(5)):
The evidence demonstrating that the selection procedure is
... a representative sample of a knowledge, skill, or ability
as used as a part of a work behavior and necessary for that
behavior should be provided (essential).
|
