Deck 7: How Do We Gather Evidence of Validity Based on Testcriterion Relationships

Question 1

A p value < .01 tells the test user that the likelihood that the relationship being measured was found by chance was ______.

A) less than 5 chances of 100
B) less than 1 chance of 100
C) not significant
D) significant

Answer 1

B

Question 2

Which one of the following statements is FALSE?

A) Evidence for validity of a criterion can be gathered using a similar process to that used when establishing evidence of validity of a test based on content.
B) Criteria must be representative of the events they are supposed to measure.
C) A criterion is valid to the extent that it matches or represents the events in question.
D) Unlike tests, reliability or consistency is unimportant for criteria.

Answer 2

D

Question 3

A subjective criterion is ______.

A) observable and measurable
B) the number of days of absence from work in a year
C) easily calculated leaving little chance of disagreement
D) based on a person's judgment

Answer 3

D

Question 4

When test scores correlate with independent behaviors, attitudes, or events, we say the test's scores have evidence of validity based on the test's ______.

A) relations with an external criteria
B) calculated reliability
C) content
D) face validity

Question 5

What do we call an evaluative standard that researchers use to measure performance, attitude, or motivation?

A) validity measure
B) utility measure
C) reliability measure
D) criterion measure

Question 6

What do we call the measure of performance that we expect to correlate with test scores?

A) the predictor
B) the criterion
C) a standardized test
D) an intercept

Question 7

When it is important to show a relationship between test scores and a future behavior, researchers establish evidence of validity using what method?

A) predictive method
B) concurrent method
C) content method
D) test-retest method

Question 8

Which of the following is TRUE about objective criteria?

A) Their scope is often quite narrow.
B) They are often based on personal experience.
C) They are often based on a person's judgment.
D) They are often expressed as ratings.

Question 9

In organizational settings, researchers often use _____ studies as alternatives to _____ studies, because employers do not want to hire applicants with low test scores.

A) concurrent; predictive
B) predictive; concurrent
C) concurrent; content
D) predictive; content

Question 10

Which method of demonstrating evidence of validity do test developers use when the test scores and criterion scores are collected at approximately the same time?

A) predictive method
B) content method
C) concurrent method
D) convergent method

Question 11

A time interval must elapse between the test administration and the criterion measurement when using which method for establishing evidence of validity?

A) concurrent
B) test-retest
C) predictive
D) content

Question 12

When psychologists use the predictive method to establish evidence of validity for a pre-employment test, they ask the employer to ______.

A) hire all the applicants who take the test
B) hire all the applicants who pass the test
C) use the test on all people employed by the company in a similar job
D) postpone hiring until the criterion is measured

Question 13

Squaring the validity coefficient between a test and a criterion will tell us about the amount of ______.

A) shared variance
B) reliability
C) validity
D) homogeneity

Question 14

Which of the following is observable and measurable, such as the number of accidents on the job, days absent, or disciplinary problems in a month?

A) internal criterion
B) objective criterion
C) subjective criterion
D) validity criterion

Question 15

When test developers correlate test scores with the criterion scores, the resulting number is called the ______.

A) content coefficient
B) criterion coefficient
C) reliability coefficient
D) validity coefficient

Question 16

If the criterion measures more dimensions than those measured by the test, we say there is ______.

A) criterion enhancement
B) criterion contamination
C) criterion failure
D) criterion success

Question 17

Having a restricted range of test scores means that the observed validity coefficient is likely to be ______.

A) higher
B) lower
C) unaffected
D) unpredictable

Question 18

Which one of the following is most likely to be the criterion measure for establishing evidence of validity based on relations with external criteria for a clinical psychological test?

A) academic ability
B) intelligence quotient
C) job performance ratings
D) diagnoses made by two or more clinicians

Question 19

The criterion of success in college is most often students' ______.

A) classroom test scores
B) professors' ratings
C) grade point average
D) community service activities

Question 20

Which of the following statements is FALSE?

A) A psychological test can be reliable, but not valid.
B) Reliability is a characteristic of the test itself; validity depends on how the test is used.
C) When a test is reliable, it is automatically valid as well.
D) It is inappropriate for a publisher to simply state that their test is valid.

Question 21

When we are interested in predicting criterion (Y') from a set of test scores, we can use what statistical procedure?

A) multiple regression
B) linear regression
C) predictive validity
D) correlation

Question 22

In the equation Y' = a + bX, what does a represent?

A) intercept
B) slope
C) individual's score
D) predicted criterion score

Question 23

Researchers must be careful to include participants in their studies who represent the entire possible distribution of performance on both the test and the criterion to avoid ______.

A) practice effects
B) multicultural bias
C) restriction of range
D) criterion contamination

Question 24

Which one of the following was a criterion for the validity study for the Suicide Probability Scale conducted at Father Flanagan's Boys Home in Nebraska?

A) number of suicides
B) number of self-destructive behaviors
C) number of prior attempts of suicides
D) test scores on the Suicide Probability Scale

Question 25

Y' = a + bX is the ______.

A) correlation equation
B) predictive validity equation
C) linear regression equation
D) multiple regression equation

Question 26

A statistical analysis that has more than one set of test scores used for predicting a criterion is called ______.

A) multiple regression
B) linear regression
C) criterion validity
D) predictive validity

Question 27

Watson, Detra, Fox, Ewing, Gearhart, and DeMotts (1996) administered two self-report alcoholism measures to 118 volunteers recruited from a Veterans Administration Medical Center. At approximately the same time, they asked the volunteers to complete the Diagnostic Interview Schedule. The study is an example of ______.

A) multiple regression study
B) concurrent evidence of validity study
C) predictive evidence of validity study
D) restriction-of-range study

Question 28

The coefficient of determination indicates what?

A) how valid the test is
B) how confident we can be about the validity coefficient
C) the amount of test variance that is reliable
D) the amount of variance that the test and the criterion share

Question 29

Which one of the following criteria used in educational settings is subjective?

A) grade point average
B) instructors' letters of recommendation
C) number of dismissals or withdrawals
D) number of courses completed

Question 30

If we wanted to predict job performance using the results of a personality test and an intelligence test, which of the following statistical techniques would be best to use?

A) coefficient of determination
B) linear regression
C) multiple regression
D) correlation

Question 31

In the equation Y' = a + bX, what does X represent?

A) intercept
B) slope
C) individual's score
D) predicted criterion score

Question 32

In a regression equation, the intercept is the place where the ______.

A) x and y axes cross b regression line crosses the y-axis
C) regression line crosses the x-axis
D) slope is zero

Question 33

Which one of the following procedures would be best to use to answer the question, "If a student scores 85 on the academic abilities test, what course grade would we expect the student to receive?"

A) linear regression
B) multiple regression
C) correlation
D) test of significance

Question 34

If the correlation (r) between a test and a criterion is .04, then the coefficient of determination (r²) is ______.

A) .0016
B) .02
C) .16
D) 2

Question 35

In the equation Y' = a + bX, what does Y' represent?

A) intercept
B) slope
C) individual's score
D) predicted criterion score

Question 36

When a calculated correlation is greater than the critical value shown in the table, we can infer that the probability of finding our correlation by chance is less than 5 chances of 100. Therefore, we assume that there is ______.

A) no relationship, and we refer to the correlation coefficient as "significant"
B) a true relationship, and we refer to the correlation coefficient as "not significant"
C) no relationship, and we refer to the correlation coefficient as "not significant"
D) a true relationship, and we refer to the correlation coefficient as "significant"

Question 37

The slope (b) of the regression line is the expected ______.

A) reliability
B) change in a for each unit of b
C) change in X for every one-unit change in y
D) change in Y for every one-unit change in x

Question 38

Studies of the PREParation for Marriage Questionnaire (PREP-M) described in your text suggest that the questionnaire predicts marital satisfaction and stability ______.

A) poorly
B) weakly to moderately
C) moderately to strongly
D) nearly perfectly

Question 39

How much variance do a test and criterion share if the coefficient of determination is .09?

A) 0.09%
B) 3%
C) 9%
D) 81%

Question 40

In the equation Y' = a + bX, what does b represent?

A) intercept
B) slope
C) individual's score
D) predicted criterion score

Question 41

When someone is interested in determining whether adding an additional test to an existing battery of tests makes good sense, we can say they are most likely interested in the ______.

A) incremental validity of the test
B) reliability/precision of the test
C) face validity of the test
D) content validity of the test

Question 42

When interpreting multiple regression results, what do you first look for?

A) amount of variance
B) if R² is statistically significant
C) the size of R²
D) if the b weights are signfiicant

Question 43

Write the linear regression formula. Identify and explain each component of the formula.

Question 44

Jane is putting together a battery of tests to select employees for her company. She is already using a test of cognitive ability and a test of numerical reasoning. She is now considering adding a personality test as well. Which one of the following would be the best statistical procedure to help make a decision about whether to add the personality test?

A) correlation
B) simple linear regression
C) multiple regression
D) Cohen's kappa

Question 45

Which one of the following indicates the expected change in Y for every one-unit change in X_i, when all the other predictors in the equation do not vary or remain constant?

A) b
B) X
C) R
D) df

Question 46

Discuss two distinct methods of evaluating validity coefficients. Explain and give examples of the differences in information that the two methods provide.

Question 47

What is criterion contamination? Give examples of criteria that may be contaminated.

Question 48

Explain the concept and purpose of regression. Give an example of linear regression and an example of multiple regression.

Question 49

Jane is considering using two different tests of mechanical aptitude, both of which have good evidence of validity, to help select her auto mechanics. From prior research, she knows that the two tests are correlated at .93. Which of the following recommendations would be best to make to Jane?

A) give both tests to all applicants
B) give one test or the other, but not both
C) use neither test because testing is not a good way to select mechanics
D) find another test that has a higher correlation than .93 with either of the other two tests

Question 50

______ is a statistic used for interpreting the results of a multiple regression.

A) Coefficient of individual determination
B) Coefficient of multiple determination
C) Coefficient of validity
D) Coefficient of reliability

Question 51

Describe the difference between reliability/precision and evidence of validity based a test's relations with criterion. Give examples.

Question 52

What is restriction of range and what causes it? Explain the consequences of restriction of range. Give an example.

Question 53

Define and give examples of objective and subjective criteria and explain why criteria must be reliable and valid.

Question 54

What does X stand for in a regression equation?

A) the correlation coefficient
B) the number on the y-axis that the linear regression coefficient predicts
C) the intercept on the y-axis
D) an individual's score on the predictor

Question 55

Define validity based on relations with an external criteria and describe two methods for obtaining evidence of it.

Question 56

Which one of the following criteria in an organizational setting is objective?

A) customer satisfaction surveys
B) sales judgments
C) supervisor ratings
D) number of excused absences

Question 57

R² Change is a statistic from a multiple regression which is useful in determining what?

A) correlation coefficient between two tests
B) place where a regression line crosses the y-axis in a regression
C) amount of measurement error present in a test score
D) whether adding an additional test to a test battery is justified

Question 58

Which one of the following symbols is used to indicate a validity coefficient?

A) R
B) r
C) a
D) b

Question 59

What is a criterion and what is its purpose? Give examples of two types of criteria.

Question 60

What are the important psychometric characteristics that a useful criterion must have? Explain your reasoning and give an example for each characteristic.

Question 61

Graduate students at a local college wanted to predict the probability of success for each first-year student. During the summer, they developed a multiple choice test with 50 items based on interviews from first-year students at the end of the previous year. When they asked permission to give the test to this year's students, the dean asked for evidence of test reliability and validity. The graduate students replied the test showed a coefficient alpha of .92 and that the test had evidence of validity based on content. The dean replied that validity evidence based on content validity was not good enough. The dean asked them to submit a plan for gathering evidence of validity based on the tests relations with some external criteria. Provide a plan for a validation study that the graduate students can give to the dean.