Deck 7: Collecting Research Data With Tests and Self-Report Measures

Question 1

The objectivity of a standardized test refers to its

A) degree of relationship to a criterion variable.
B) freedom from tester bias.
C) accuracy of prediction.
D) provision of normative data.

Answer 1

freedom from tester bias.

Question 2

In contrast to the use of performance tests in research, the use of self-report measures generally

A) is directed at measurement of personality characteristics.
B) is meant to demonstrate each respondent's level of a characteristic that is typical for her.
C) is subject to distortion due to response sets.
D) all of the above.

Answer 2

all of the above.

Question 3

The tendency to present oneself in a favorable light when responding to a self-report measure is known as a(n) _________response set.

A) social desirability
B) acquiescence
C) deviance
D) carelessness

Answer 3

social desirability

Question 4

To standardize a test, the developer must establish

A) agreement between researchers and practitioners about test content.
B) the test's reliability.
C) procedures that ensure consistency in how the test is administered and scored.
D) all of the above.

Question 5

In test development, differential item functioning refers to the tendency for

A) some test items to relate more closely with the total test score than other items.
B) individual scorer bias to affect how test items are scored.
C) individuals of equal ability but from different subgroups not to have the same probability of earning a given score.
D) individuals to score differently on each administration of the test.

Question 6

Match each example of evidence about a test below with the type of evidence of test validity that it represents.

-The test was found to include 32 of the 47 topics covered in a textbook.

A) Evidence from relationship to other variables
B) Evidence from consequences of testing
C) Evidence from response processes
D) Evidence from test content

Question 7

Match each example of evidence about a test below with the type of evidence of test validity that it represents.

-The test was found to lead to a higher rate of referral for boys than for girls to a treatment program for attention deficit disorder.

A) Evidence from relationship to other variables
B) Evidence from consequences of testing
C) Evidence from response processes
D) Evidence from test content

Question 8

Match each example of evidence about a test below with the type of evidence of test validity that it represents.

-Students' rank order of performance on the test was found to correspond well to the students' rank order of performance on an established longer test of the same ability.

A) Evidence from relationship to other variables
B) Evidence from consequences of testing
C) Evidence from response processes
D) Evidence from test content

Question 9

Match each example of evidence about a test below with the type of evidence of test validity that it represents.

-The test was found to be a good measure of students' ability to learn independently.

A) Evidence from relationship to other variables
B) Evidence from consequences of testing
C) Evidence from response processes
D) Evidence from test content

Question 10

Test validity is

A) not a unitary concept.
B) an intrinsic property of the test.
C) an intrinsic property of the test scores.
D) associated with inferences that are made from test scores.

Question 11

Recent research on the effects of standards-based instruction has shown that test data collected in school settings generally show

A) high evidence of content-related validity.
B) low evidence of content-related validity.
C) high alignment between curriculum content, teachers' instruction, and assessment by standardized tests.
D) teachers' ability to judge evidence of a test's content-related validity by brief inspection of the items.

Question 12

If a test is used to make decisions that are found to be harmful to certain groups, this represents evidence of the test's

A) lack of validity.
B) misuse.
C) low objectivity.
D) lack of reliability.

Question 13

Match each example of test reliability below with the type of reliability coefficient that is used to calculate it.

-After the test is administered, it is split into two halves and the scores on one half are correlated with those on the other half.

A) Coefficient of equivalence
B) Coefficient alpha
C) Coefficient of internal consistency
D) Coefficient of stability

Question 14

Match each example of test reliability below with the type of reliability coefficient that is used to calculate it.

-Two parallel forms of the test are administered, and scores on one form are correlated with those on the other form.

A) Coefficient of equivalence
B) Coefficient alpha
C) Coefficient of internal consistency
D) Coefficient of stability

Question 15

Match each example of test reliability below with the type of reliability coefficient that is used to calculate it.

-The test is administered and then readministered after a time delay, and scores from the first administration are correlated with those from the second administration.

A) Coefficient of equivalence
B) Coefficient alpha
C) Coefficient of internal consistency
D) Coefficient of stability

Question 16

Match each example of test reliability below with the type of reliability coefficient that is used to calculate it.

-Individual test items are analyzed with the use of standard formulas.

A) Coefficient of equivalence
B) Coefficient alpha
C) Coefficient of internal consistency
D) Coefficient of stability

Question 17

A test is reliable to the extent that it

A) yields scores that are interpreted the same way by different raters.
B) measures a known construct.
C) contains nonrandom measurement error.
D) minimizes measurement error.

Question 18

The various sources of measurement error in a test can be estimated by

A) classical test theory.
B) generalizability theory.
C) the standard error of measurement.
D) coefficient alpha

Question 19

Calculation of the standard error of measurement allows researchers to determine

A) the combined measurement error due to all sources of error that were investigated.
B) the probable range within which individuals' true scores fall.
C) the likelihood that that test was too easy or too difficult for most testees.
D) the reliability of each test item.

Question 20

Item response theory allows researchers to develop tests having all the following advantages over traditional tests except that

A) the test can be customized to students at different ability levels.
B) it is possible to construct many parallel tests of equivalent difficulty.
C) measurement error can be reduced by administering items geared to each individual's ability level.
D) each individual can take a test containing just a few items at an appropriate difficulty level.

Question 21

When the researcher has the choice of using a standardized test or an achievement test constructed by the teachers who are being studied, the researcher

A) should use the teacher-constructed test, because such tests generally are freer of measurement error.
B) should use the teacher-constructed test, because such tests generally have better concurrent validity.
C) should use the standardized test, because such tests generally are better written.
D) can use either test, because both types of test have similar strengths and weaknesses.

Question 22

The interpretation of an individual's test score by comparing it to the scores earned by other individuals is referred to as

A) norm-referenced measurement.
B) individual-referenced measurement.
C) domain-referenced measurement.
D) objectives-referenced measurement.

Question 23

A prespecified standard of performance is most important in

A) norm-referenced measurement.
B) individual-referenced measurement.
C) criterion-referenced measurement.
D) objectives-referenced measurement.

Question 24

In computer-adaptive testing, the test-taker

A) responds only to items that are at a moderate level of difficulty.
B) is allowed as much time as he wishes to respond to each item.
C) responds to items that are matched to his ability level.
D) is allowed to choose the items to which he will respond.

Question 25

The process by which items are placed on a scale in order of relative difficulty in computer-adaptive testing is based on

A) item response theory.
B) generalizability theory.
C) classical test theory.
D) grounded theory.

Question 26

It is usually better to administer an individual test than a group test when

A) it is necessary to ensure standard conditions of administration.
B) a high degree of objectivity in scoring is desired.
C) it is necessary to obtain data on all research participants within a short period of time.
D) the researcher is interested in the process by which the overall score is obtained.

Question 27

Match each of the following types of test with the appropriate definition.

-Intelligence tests

A) Estimate general intellectual level by sampling performance on various tasks.
B) Aim at predicting performance on future tasks.
C) Provide scores indicating individuals' strengths and weaknesses in a given area of the curriculum.
D) Measure knowledge or mastery of a variety of content areas.

Question 28

Match each of the following types of test with the appropriate definition.

-Aptitude tests

A) Estimate general intellectual level by sampling performance on various tasks.
B) Aim at predicting performance on future tasks.
C) Provide scores indicating individuals' strengths and weaknesses in a given area of the curriculum.
D) Measure knowledge or mastery of a variety of content areas.

Question 29

Match each of the following types of test with the appropriate definition.

-Achievement test batteries

A) Estimate general intellectual level by sampling performance on various tasks.
B) Aim at predicting performance on future tasks.
C) Provide scores indicating individuals' strengths and weaknesses in a given area of the curriculum.
D) Measure knowledge or mastery of a variety of content areas.

Question 30

Match each of the following types of test with the appropriate definition.

-Diagnostic tests

A) Estimate general intellectual level by sampling performance on various tasks.
B) Aim at predicting performance on future tasks.
C) Provide scores indicating individuals' strengths and weaknesses in a given area of the curriculum.
D) Measure knowledge or mastery of a variety of content areas.

Question 31

Which of the following is not an appropriate criterion for judging the validity of inferences drawn from a performance assessment?

A) Whether all students had an equal opportunity to acquire the expertise measured by the test
B) The extent to which the test scores correlate with the test-takers' scores on a well-established standardized test
C) The extent to which the test's content represents the content domain covered during instruction
D) The cost of administering the test

Question 32

The hermeneutic approach to test reliability

A) requires test scorers to reach a consensus about the score or rating that each test-taker will receive.
B) requires administration of parallel forms of a test.
C) involves identifying experts with a consistent perspective to score the tests.
D) involves the calculation of a standard error of measurement.

Question 33

To determine an individual's unique structuring of reality, the researcher is advised to administer a

A) variety of attitude scales.
B) personality inventory.
C) combination of attitude scales and a personality inventory.
D) projective test.

Question 34

The Mental Measurement Yearbooks are particularly helpful for

A) determining the latest trends in assessment methodology.
B) identifying procedures that can be used to establish a test's validity and reliability.
C) obtaining a list of tests that are available for measuring a particular variable.
D) obtaining a statistical summary of the test scores of students with different demographic characteristics.

Question 35

The best source of information about tests that are not commercially available is

A) Test Reviews Online.
B) Test Collection.
C) Tests in Microfiche.
D) Standards for Educational and Psychological Testing.

Question 36

Contacting the test developer directly is particularly useful when you wish to obtain

A) tables of norms for the test.
B) recent information about the test that has not yet been published.
C) reliability and validity information about the test.
D) information about how to administer and score the test.

Question 37

In test development, the purpose of item analysis is to determine

A) the difficulty level of each test item.
B) the reliability of each test item.
C) the validity of each test item.
D) all of the above.

Question 38

Put the following steps of test development in the order that they usually occur.
a. Developing a prototype
b. Defining the constructs to be measured
c. Reviewing related tests that have been developed
d. Collecting data on the test's validity and reliability

Question 39

Item analysis is a set of procedures used to

A) develop items for a parallel form of a standardized test.
B) revise the wording of test items to eliminate bias toward certain groups.
C) assess the validity, reliability, and difficulty of each item on a test.
D) all of the above.

Question 40

Which of the following actions by the researcher is least likely to be helpful if protestors question the administration of a certain test in a research study?

A) Defending the test by reviewing with the protestors the merits of each item on the test
B) Demonstrating how the test as a whole is valid and reliable
C) Demonstrating that the test follows the guidelines in the Standards for Educational and Psychological Testing
D) Offering to withdraw individuals from the study if they do not wish to take the test

Question 41

Match each testing procedure below with the outcome for which it is intended.

-Have the teacher tell the students (the research participants) that the test is important and that they should try to do their best.

A) You want to ensure that the research participants depict themselves in a typical, honest manner.
B) You want to obtain the research participants' maximal performance.
C) You want to enhance the research participants' cooperation.

Question 42

Match each testing procedure below with the outcome for which it is intended.

-Remind the research participants that data collected on any particular individual will not be revealed.

A) You want to ensure that the research participants depict themselves in a typical, honest manner.
B) You want to obtain the research participants' maximal performance.
C) You want to enhance the research participants' cooperation.

Question 43

Match each testing procedure below with the outcome for which it is intended.

-Emphasize the official nature of the testing session.

A) You want to ensure that the research participants depict themselves in a typical, honest manner.
B) You want to obtain the research participants' maximal performance.
C) You want to enhance the research participants' cooperation.

Question 44

Name four types of performance tests that are commonly used in educational research.

Question 45

Name five types of personality measures that are commonly used in educational research.

Question 46

Describe three criteria for judging the quality of a test.

Question 47

A researcher uses a theory of academic self-esteem to identify five behavior patterns that are indicative of students' level of academic self-esteem, such as the frequency of positive statements students make about themselves. The researcher next observes 50 students for one month and records the frequency of these behavior patterns for each student. Finally, the researcher develops a paper-and-pencil test of academic self esteem and correlates the students' scores with their level of academic self-esteem based on the observations. What type of evidence of test validity has the researcher established?

Question 48

A researcher has developed a new test designed to measure musical aptitude. She selects a random sample of 200 college freshmen and administers the test to them. One month later she again administers the test to the same students and computes a correlation coefficient between the two sets of scores. What type of reliability does this coefficient indicate?

Question 49

An investigator administers a reading achievement test that has a mean of 220, a standard deviation of 20, and a standard error of measurement of 5. Mary obtains a score of 235. Using the standard error of measurement (sm), what estimate can we make of her true score?

Question 50

What is one advantage of using item response theory to develop an achievement test?

Question 51

Describe four problems to which tests developed within the classical test theory framework are susceptible.

Question 52

Describe three assumptions on which the item-response-theory approach to test construction is based.

Question 53

The superintendent tells a researcher who is conducting a study in her district that he should use achievement tests constructed by the teachers to measure student learning rather than a standardized achievement test. How can the researcher defend the decision to use a standardized test?

Question 54

Name three factors that can distort the scores on a standardized test.

Question 55

Describe three limitations of standardized achievement tests.

Question 56

A student earns a score of 85 on a 100-item achievement test. How would this score be interpreted if the test was criterion-referenced?

Question 57

State two advantages of administering a test in computer format.

Question 58

Describe one situation in which an individually administered test would be preferable to a group- administered test.

Question 59

Name one drawback of administering a standardized test individually as opposed to administering it to an entire group at once.

Question 60

Give one reason why the test ceiling is an important factor to consider in selecting an achievement test to use in a research study.

Question 61

Some school districts require candidates for a teaching position to teach an actual lesson, which is rated for quality. The district also may require candidates to submit a collection of curriculum materials and lesson plans that they have prepared in their previous teaching positions.
a. What is this approach to measurement called?
b. What is the technical name for the collection of completed work?

Question 62

Describe four criteria recommended for judging the validity of performance assessments.

Question 63

Describe at least one advantage and one disadvantage of self-report personality measures.

Question 64

How do the Mental Measurement Yearbooks contribute to the use of tests in research?

Question 65

State one reason why it is important for a researcher to examine an actual copy of an achievement test before deciding to use it in a study.

Question 66

State two questions that can guide a search for relevant information to help you select among the range of tests that appear related to one's sample and research problem.

Question 67

Describe one procedure that customarily is used as part of an item analysis of a test under development.

Question 68

How can the publication Standards for Educational and Psychological Testing help if a test to be administered as part of a research study comes under attack?

Question 69

Describe three actions you could take if you encounter public or professional resistance to the tests that you wish to administer for a research study.

Question 70

A researcher plans to administer a personality measure and wants to ensure that the research sample gives honest responses to its items. What is one procedure that the researcher can use to accomplish this goal?

Question 71

Describe three procedures that might help obtain individuals' maximal effort on a performance test.

Question 72

Describe four procedures that might help obtain individuals' honest responses on a personality measure.

Question 73

Describe a procedure for handling each of these testing problems.
a. A researcher is planning to administer several aptitude tests to groups of college students as part of a research project. What can he do to increase the likelihood that students will give their maximum performance?
b. A researcher wants to administer a measure of sexual attitudes to high school and college students. What can she do to increase the likelihood that students will give frank, honest responses?

Question 74

An investigator has developed a new technique to teach eighth-grade students the characteristics of positive and negative numbers. She wishes to evaluate the technique by using it to teach group A. Group B, which receives conventional instruction on positive and negative numbers, serves as a control. She needs a mathematics achievement test to measure the post-training performance of the two groups. She considers the XYZ Test of Mathematics Achievement. In checking the test manual, she finds that when this test was administered to a sample of eighth-grade students at the end of the academic year, the test scores correlated .72 with their end-of-year grades in mathematics. No other validity evidence was given in the test manual.
a. What type of validity evidence was given in the test manual?
b. What type of validity evidence is most relevant to the proposed study, and why?

Question 75

A researcher is asked by school district personnel to collect data for a period of three years to determine whether their plans to improve third-grade reading skills are successful. The researcher decides to administer the same reading achievement test at the end of each school year for the next three years. List the aspects of their content that she should judge carefully as she examines various achievement tests for possible use in the study.

Question 76

a. Select one standardized test. If your thesis or dissertation will involve the use of standardized tests, select a measure relevant to that study. Otherwise, select a measure that relates to your major professional interests.
b. Obtain a specimen set for your measure. Most departments of psychology, university libraries, or counseling centers have a file of standardized tests. A copy of the test, the test manual, and the scoring key are included for most measures. Your instructor will give you the information you need to obtain a specimen set.
c. Sources of Information
(1) Look up the selected measure in the most recent edition of the Mental Measurements Yearbook available in your institution's library. If your measure is not reviewed, select another measure.
(2) Check the name of the measure you have selected in the most recent three annual volumes of Psychological Abstracts and ERIC. Also check key words related to the topic of the measure. For
example, if your measure were the IPAT Anxiety Scale Questionnaire, you would check the key word anxiety in the subject index. Locate one study in which the measure was used. Read the study,
giving particular attention to data related to the questions below. Prepare a note card on the article and submit it as part of this assignment.
d. Answer the following questions based on your study of the sources mentioned above.
(1) Objectivity
(a) Is the measure subject to administrator bias? Discuss.
(b) Is the measure subject to scoring bias? Discuss.
(2) Normative Data
(a) Are normative data provided? Describe.
(b) Are the samples used in establishing the norms satisfactory? Describe.
(c) How comparable are norm groups to the sample in your study (if this measure is being considered for your thesis or dissertation project)?
(3) Validity
(a) What type of validity evidence is most appropriate for this measure? Why?
(b) What validity evidence is given in your sources? Describe.
(c) Write a brief evaluation of the validity evidence available on this measure.
(4) Reliability
(a) What type of reliability data are available for this measure? Describe.
(b) What are the reliability coefficients?
(c) Write a brief evaluation of the reliability data available for this measure.
e. Submit a typed, double-spaced paper giving the name and a brief description of the measure, including the publisher's name, publication data, administration time, age levels if appropriate, and the types of scores that it yields. Answer the four questions listed in "d" above. Be sure to include your note card with your paper [see (c-2)]. Be prepared to report on your standardized test evaluation in class.

Question 77

Find a test that pertains to a research problem or topic of interest to you. Indicate in writing:
a. how the test possibly is relevant to your research problem.
b. the reference source in which you identified the information.
c. something interesting you learned about the test from the reference source. (Write just two sentences.)