Deck 5: Evaluating Research: Reliability and Validity

Question 1

Lincoln and Guba (1985)argue that instead of validity and reliability,qualitative research should instead be judged based on

A)composite variables.
B)credibility and dependability.
C)statistical significance.
D)a high Cronbach's alpha.

Answer 1

B

Question 2

A perfect Cronbach's alpha score is

A)0.1.
B)1.
C)0.6.
D)0.

Answer 2

B

Question 3

Which of the following is a term that describes the consistency of items on a scale and assesses how well responses to the first item on a scale predict responses to the second item?

A)concurrent validity
B)face validity
C)external validity
D)internal reliability

Answer 3

D

Question 4

There are critiques of the Center for Epidemiologic Studies Depression Scale (CES-D).Some of those critiques surround its development for use with English-speaking people of European descent.Thus,some of the trouble in assessing reliability for depression scales points to which of the following?

A)People from different cultural backgrounds tend to answer questions in similar ways.
B)Depression cannot be measured in a reliable way if surveys are given to people from different cultural backgrounds.
C)No one should use the CES-D because we can't know for sure if the results are accurate.
D)the need to take into account diverse populations when using any measure

Question 5

The NAS poverty line may offer a more accurate picture of poverty in the United States than the FPL,but it

A)is not possible to use it in all 50 states.
B)may not take into account geographic differences.
C)may not be as reliable a measure over time.
D)gives a much more negative picture of poverty in the United States.

Question 6

A team of researchers wants to study whether the type of neighborhood a person lives in impacts his or her health.Because many different observers will be involved in the study,there is a need to make sure these different observers agree with each other when they make evaluations about neighborhood type and about what "health" looks like.If the various observers agree when they look at the data,they would have achieved which of the following?

A)coding
B)intercoder reliability
C)a high Cronbach's alpha
D)high accuracy

Question 7

The discussion of the difference between the federal poverty line (FPL)and the National Academy of Sciences (NAS)poverty line is important because it highlights

A)the complexity of accurately measuring poverty.
B)that poverty is impossible to accurately measure.
C)that it is not possible to understand whether poverty rates have decreased over time.
D)that poverty rates always increase over time.

Question 8

Which of the following terms refers to whether results are true or accurate?

A)fact
B)reliability
C)validity
D)robustness

Question 9

When researchers want to test robustness,or how well an operational protocol is working,they might use which of the following techniques?

A)the test-retest method
B)vetting
C)face validity
D)a normal distribution

Question 10

Ordinary least squares (OLS)regression is a statistical procedure that does what?

A)tests the face validity of the measures
B)estimates how well changes in the independent variables predict changes in the dependent variable
C)makes sure the statistical operations of a study are in line with basic statistical laws and guidelines
D)proves that the right operational definitions were used in the study

Question 11

A sociologist studying American neighborhoods identifies a causal effect of income on the availability of healthy and nutritious foods.Specifically,people with lower incomes tend to live in neighborhoods where there are more fast food restaurants and less access to fresh fruits and vegetables.Given this finding,the researcher can be confident in concluding that the study has

A)face validity.
B)a reliable measure.
C)internal validity.
D)external validity.

Question 12

Which term refers to how detailed and specific a measure is?

A)reliability
B)precision
C)robustness
D)pilot testing

Question 13

The terms validity and reliability would be most applicable to which of the following projects?

A)an interview-based study with 50 respondents on attitudes toward political engagement
B)a focus group including 10 participants on the subject of religious affiliation
C)participant observation carried out in a hospital to identify how patients of different ages are treated by doctors
D)a nationally representative survey of 1,200 Americans about their TV viewing habits

Question 14

Critics of the federal poverty line (FPL)point out that it does not take regional variation in cost of living into account and that what constitutes an adequate diet is highly subjective.Because of these and other issues,the FPL leads to which of the following?

A)It looks like there are more poor people in America than there actually are.
B)Many Americans who are poor do not get counted as poor when they should be.
C)It seems like most Americans are middle-class.
D)Wealthy people might look they are actually middle-class.

Question 15

Suppose a researcher wants to measure difficulties for college students whose parents did not attend college.The researcher writes survey questions about grades,quality of support on campus,and financial resources,but does not ask about whether the student has family support or a good adviser.If the researcher has not tapped into all the dimensions of difficulties for first-generation college students,which of the following would be at risk?

A)content validity
B)face validity
C)predictive validity
D)concurrent validity

Question 16

For which of the following would it make sense to measure using a composite variable?

A)age
B)gender
C)race
D)happiness

Question 17

A researcher creates a scale to measure students' satisfaction with their sociology courses.After identifying six key questions and organizing them into a composite measure,the researcher averages the six items into a variable called "sociology satisfaction." The Cronbach's alpha score was 0.8,which means the

A)measure is unreliable.
B)items on the scale do not predict each other.
C)scale items are all measuring some aspect of satisfaction with sociology courses.
D)researchers should try to adjust the scale so that the Cronbach's alpha is closer to 1.

Question 18

There are approximately equal numbers of men and women in the U.S.population.If a researcher wants to study the impact of gender on whether teenagers decide to apply to college,but only includes women in the study,which of the following would be in jeopardy?

A)external validity
B)criterion-related validity
C)concurrent validity
D)face validity

Question 19

The CBCL,HOME,and CLASS are measures often used in research because they

A)are almost always 100% accurate.
B)are applicable to all populations,and therefore can be used in many different research contexts.
C)have been vetted and therefore researchers can have confidence in using them.
D)only attempt to gather small amounts of information and therefore are less likely to be inaccurate.

Question 20

Which of the following is the correct term for a measure that is consistent,dependable,and predictable?

A)valid
B)reliable
C)credible
D)precise

Question 21

A team of researchers is using surveys to study college students' feelings about the dining hall on campus.In order to ensure reliability,the researchers employ a test-retest method.One potential problem with a test-retest might be that

A)students might decide not to participate in the survey,which would result in the surveys that were collected being considered invalid.
B)students might not understand the test-retest method and therefore the researchers could not complete the surveys.
C)it is very difficult to successfully employ a test-retest method on a college campus.
D)the dining hall could have changed its food in between the two survey points and therefore students' attitudes about the dining hall may have changed.

Question 22

Which of the following is an assumption of OLS regression?

A)The dependent variable will predict changes in the independent variable.
B)The values in a sample follow a bell curve pattern or normal distribution.
C)The coefficient will be -0.50.
D)Cases in the sample are not randomly selected.

Question 23

After conducting a study on happiness,researchers realize that the validity and reliability of the study is not as strong as they had hoped.In essence,they haven't done as good a job of measuring happiness as they had hoped to.This means that

A)future researchers should improve on the reliability and validity of the current study by thinking through how to more accurately measure happiness.
B)the research is useless because it did not perfectly measure happiness.
C)the current project should not be trusted and future researchers should not replicate it;instead,they should start with all new measures.
D)no one should try to measure happiness because it cannot be done accurately.

Question 24

What is the purpose of a pilot test?

A)It is used after a study is conducted to check if the results are accurate.
B)It is used to see how well the measure works and whether it presents any problems that need to be fixed before the study is carried out.
C)It is used to test only controversial questions on a survey.
D)It is used to make sure the researcher is up to the task of carrying out the full project.

Question 25

The test-retest method is an approach in which

A)the same measure is administered to a sample and then re-administered to the same sample later.
B)the same measure is administered to two different samples at different points in time.
C)the same measure is administered to two different samples at the same point in time.
D)two different measures are administered to the same sample at two different points in time.

Question 26

Suppose a researcher wants to measure whether people have experienced racial discrimination in their neighborhood.The researcher might ask questions about whether subjects ever felt they were turned away from renting or buying a home because of their race.The researcher might also ask about whether they have ever been treated unfairly by store clerks or their neighbors.After writing the questions on their survey,the researchers felt they had accurate measures of discrimination because they seemed right.Which kind of validity would they feel they achieved?

A)face validity
B)concurrent validity
C)external reliability
D)predictive validity

Question 27

Which of the following is true of error in social science research?

A)Most social science research involves no error.
B)All social science research involves some level of error.
C)All social science research involves a significant amount of error,but researchers are willing to overlook it.
D)Most social science research has systematic error built into it.

Question 28

Pick a concept for which it would make sense to measure using a composite variable.In no less than three sentences,explain why you would want to use a composite measure for this concept (as opposed to a single measure).

Question 29

The split-half method is when

A)researchers split the research team in two and train different kinds of researchers to accomplish different tasks associated with the study.
B)researchers split the concepts they're studying into two groups and decide which one is more accurate.
C)researchers randomly split the set of items for a measure into two sets to create two separate measures instead of one.
D)an experiment uses a control group.

Question 30

Write a paragraph in which you discuss two difficulties sociologists face in ensuring their research is valid and reliable.Use clear examples from the text to support your answer.

Question 31

Classroom quality can be measured in many different ways.If a researcher is careful to map out each dimension of classroom quality,and observers have a clear plan for how to determine classroom quality when they see it,what have the researchers accomplished?

A)They have increased the reliability of their research method.
B)They made it possible to identify which kinds of classrooms are more likely to be effective.
C)They created a fair assessment tool for determining which classrooms are problematic.
D)They made it possible to determine that a classroom quality scale will have a high alpha.

Question 32

A researcher is getting ready to gather data from a nationally representative sample of 5,000 Americans on their attitudes toward voting in local elections.In order to make sure our measures are reliable,we might gather a random sample of 100 Americans and test the study.What would we call this preliminary study?

A)a test-retest
B)a reliability test
C)a sample population
D)a pilot study

Question 33

In Chapter 5,the authors argue that educational attainment can be measured in many different ways,which makes it difficult to clearly conceptualize and operationalize.Come up with another concept that you think would also be difficult to study,but that might be something interesting for a sociological investigation.In a paragraph,explain the difficulty and how you might go about measuring your topic of interest to make it as reliable as possible.

Question 34

A pilot test involves administering some version of a measurement protocol (a survey,an experiment,an interview)to whom?

A)a small preliminary sample of subjects
B)some subjects who will be part of the eventual study sample
C)some of the researchers' colleagues and friends
D)a group of people who are very different from the eventual sample subjects

Question 35

What is the most effective method for increasing the reliability of a research method?

A)using a dictionary definition to make sure you have carefully defined your concepts
B)doing a very careful job of conceptualization and operationalization
C)working with a team of researchers
D)using OLS regression

Question 36

A common measure of educational attainment includes categories for less than high-school diploma,high-school diploma,some college,college degree,some postgraduate education,and
Postgraduate degree.These categories could be confusing or miss people who went to trade school or who earned any type of nontraditional degree.This measure would have problems with which of the following?

A)It might make people feel uncomfortable about answering the question.
B)It would have problems with precision,which would threaten reliability.
C)It would be hard to test the validity of this question.
D)It would be impossible to use the split-half method to check whether the measure is reliable.

Question 37

The CES-D is considered a reliable composite measure because

A)the alpha score when used in the National Longitudinal Study of Adolescent Health (ADD Health)is 0.87.
B)the items are not used on any other scale-they are unique to the CES-D.
C)people's responses to one item on the CES-D do not seem to predict their responses to other items.
D)the items on the CES-D are relatively new and have not been used in many different studies.

Question 38

Experiments might potentially be critiqued for problems with external validity.Why?

A)Experiments are often poorly designed.
B)It is hard to know if what happens in a lab setting will happen in the real world.
C)Experiments are often poorly biased.
D)Participants who volunteer for experiments are often unreliable.

Question 39

If a researcher asks a participant in a study to envision himself or herself as an unemployed autoworker living in Detroit,Michigan,with a partner and two children,who is struggling to afford food and housing,the researcher is using a

A)survey.
B)precise measure.
C)vignette.
D)normal distribution.