WEEK 3:
RESEARCH DESIGN, MEASUREMENT LEVELS
10 STAGES OF A RESEARCH
DESIGN (like a PhD Dissertation)
1) Problem Formulation. Determine
what you are studying. Explain why it is important. UNC Professor William
Rivenbark’s (an MSU PhD graduate) first article was on casino gambling in other states. He defended
its importance since gambling losses and taxes on the gaming industry was
believed to be a regressive tax. A regressive tax especially hurts lower income
people, because they spend a higher percentage of their income gambling
compared to the higher income. Plus, addiction to gambling is a real social
problem.
2) Literature review. A
researcher then must conduct a thorough review of the literature. You have to
do that yourself for your research paper. You will review the literature on
each of your five hypotheses. A great source of scholarly literature (and
reputable think tanks and government studies) is at jstor (use the
Advanced Search). For example, if my hypothesis is that young people tend to
have more Democratic party identifications compared to old people, I might just
put in “party identification” in the search window. Then, I might type in “partisanship
and age.” Or try “age differences in partisanship.” Finally, try “generational
differences in partisanship.” Even then, I might not find enough articles, so I
go to Google search, type in “age differences in partisanship,” and now find a
number of relevant studies and polls. This will be the most time-consuming part
of your research paper, so it is not due until February 28, which is right after
the first essay exam. You should try to find at least two literature sources
for each of your hypotheses. We will use lab time to help you with this part of
your paper. Generally speaking, some of the best political science journals
are: American Political Science Review, American Journal of Political Science,
Journal of Politics, American Politics Research, and Public Opinion Quarterly. Also,
you will find some literature in the journals of other disciplines, since our
scientific approach to the study of politics does rely on other fields. Students also find reports authored by relevant think tanks and government agencies, including Pew, Gallup. Doing
this kind of research is a very important job-relevant skill, so spend a lot of
time on your literature review. It has helped past students get good internships and jobs, including scholarships at law schools!
3)
Identify the Unit of Analysis. What are you collecting data on? You are getting
information about what kinds of units?
The
four units of analysis are: Individual, county, state, and nation.
a) Individual unit examples are public opinion polls. Polls
collect information about individuals; in the Mississippi Poll, we sometimes
asked 500 individuals as many as 60 different questions. So we had a lot of information
on each of 500 people. So our unit of analysis was the individual.
b) A County unit example is a public policy study examining public
education spending in each of Mississippi's 82 counties, and how that impacts
the educational level, unemployment rates, and average household incomes of each
of the 82 counties. We have these items of information on each of the state’s
82 counties, so our unit of analysis is the county. Another example is in my research
book chapter on the 2020 presidential election in Mississippi. I did not have
any polling data, so I looked at the election results in each county, obtained
Census data on the racial composition of each county, and found a high
relationship between the racial composition of the county and vote for Democrat
Biden. Biden carried every black-majority county, and Trump carried nearly
every white-majority county, suggesting that race (and ideology) remain an
important motivator in the modern South.
c) A State unit example is a public policy study examining education
spending in each of the nation's 50 states. The College Board has gathered data
on the percentage of adults in each state who hold a college degree (and found
that Mississippi is low), and might employ you to gather additional information
on why that might be the case. So you might gather for each state the average
family income, the average state college scholarship amount provided, the
average public university tuition fee, the state high school graduation rate, and
the average ACT score of each state. You are gathering information about every
state, so the state is your unit of analysis.
d) The Nation as the unit of analysis example may be a study
of the level of democracy and the level of press freedom in each nation. This
is very relevant to us, due to the January 6 Trump “insurrection” and
Democratic party claims that our democracy is at risk. There are two websites
that are actually devoted to this issue, and they rank each nation on their
levels of democracy and press freedom. So if you work for such a website, you
may be asked to study why nations vary in their levels of democracy and press
freedom. You might gather information about each nation’s level of wealth (per
capita income), extent of competitive parties (measured by how often different
parties control the top executive position; also measured by difference in
number of seats in the legislature held by the top two parties), average
citizen’s level of possession of an “authoritarian personality” (like the
Milgram experiment, but hard to find a measure of). So you have these items of
information on each of the 150 or so nations of the world. So the unit of
analysis is the nation.
In the first essay exam, I will simply
ask you to give this one-word answer as you identify the unit of analysis of ten different
studies. An example follows. Try to do it yourself, before going to the answers
at the end:
Test question. (10 points) What
is the unit of analysis for each of the following studies? One-word answers for
each of the ten studies is fine-- individual, county, state, nation.
A. A study of the willingness
of Mississippians to vote for a qualified woman as Governor, based on a public
opinion poll of 600 adult Mississippians asking each of them that question,
what their party identification and political ideology is, and what their age
and education is.
B. A study of the effectiveness
of early mail-in voting, based on the following information about each of the
fifty states: change in voter turnout after early mail-in voting was enacted;
which political party controls the governorship; which political party controls
the state legislature; percentage of state population that has a low income;
percentage of state population that is under 25 years of age.
C. A study of the cost
effectiveness of the county unit system of road construction, based on the
following information about each of the state's 82 counties: number of miles of
roads constructed or repaired last year; total expense of road construction
last year; whether the county operates under the county unit system, or whether
it operates under the "beat" system.
D. A study of political
instability in the world, based on the following information about each nation:
the number of years that the current chief executive has been in office;
unemployment rate; change in per capita personal income; how competitive the
political party system is.
E. A study of the quality of
health care in the American states, based on the following information about
each state: average life span; government expenditures on health care per
capita; poverty rate in each state; extent of physical fitness classes in the
public schools.
F. A study of the causes of
President Biden's relatively modest current job approval rate, based on a national
survey of 1500 Americans asking each of them to rate the President's job in
office, what their party identification is, how they would rate the health of
the American economy, how their own personal financial situation has changed
over the past year, and how they would rate Biden’s efforts to solve the
issue that each respondent is most concerned about.
G. A study of world famine,
based on the following information drawn from each nation: percentage of
population that has died of starvation in the past year; presence or absence of
civil war in each nation; per capita income of each nation; extent of harsh
weather conditions over the past year.
H. A study of highway safety in
Mississippi, based on the following information about each county: number of
highways deaths last year adjusted for county population; number of law enforcement
vehicles actively patrolling the highways; average age of the average car in
each county; per capita alcohol consumption in each county.
I. A study of the quality of
American universities, based on the following information from each state: percentage
of college students who graduate within six years; average starting salary of
college graduates; percentage of state population over 25 years of age that has
a college degree; state government expenditure on public higher education
institutions, adjusted for number of students; presence or absence of a merit
system for evaluating the performance of faculty.
J. A study of race relations in
Mississippi based on the following public opinion poll conducted among 600
adult residents, asking each of them: how would you rate the quality of race
relations in Mississippi; what is your race, party identification, and
ideology; have you ever been a victim of police brutality.
The answers are, of course:
A = Individual
B = State
C = County
D = Nation
E = State
F = Individual
G = Nation
H = County
I = State
J = Individual
4) Design the data
collection mode- how you collect the data. Do you use a survey, are you using
legislative roll call data, are you doing an aggregate study (where the unit of
analysis is above the individual), are you doing a content analysis of the
words spoken by a politician?
a) A Survey is an individual opinion survey. It can be of the mass population, or of
a more specialized group, such as government workers.
b) A Roll call analysis deals with congressional or state legislative votes on
public issues, and often includes demographic characteristics of their
districts' constituents.
c) An Aggregate analysis deals with a level of analysis higher than the
individual. It deals with cases that combine numbers of individuals, such as
counties, states, etc. The data are often secondary data analysis, collected by
government agencies, such as the U.S. Census.
d) A Content analysis is a study of the characteristics of messages, such as
how ideologically biased is the mass media, and how many liberal or
conservative themes are voiced by a President or governor. We get to this
subject late in the course.
5) Pre-test your survey (if
you are conducting your own original survey), since this helps you to anticipate
validity problems with your indicators, and suggests variables that you may
have left out. For a statewide public opinion poll of 600 Mississippians who
are asked 60 questions, you might ask a random sample of 25 Starkville
residents the 60 questions, and then ask the interviewers whether the
respondents had difficulty answering any of the questions, and if so why. We
failed to do a pre-test when we included six questions about possible
government reorganization proposals of a former governor (Ray Mabus), and we found
that most people did not support his reform proposals, largely because they
didn’t understand them or the question wordings (for example, he wanted to
combine the governing Community College and IHL Boards; most people didn’t even
know that such public governing boards existed).
6) Data collection-
collect the data, or do a secondary data analysis that uses an existing dataset.
Most surveys collect data through a CATI system. CATI stands for
Computer-Assisted Telephone Interviewing system. Secondary data analysis relies on existing
data sources, such as the University of Michigan National Election Studies
conducted every two years, or the MSU Mississippi Poll previously conducted
every two years. The MSU Poll used the CATI system at MSU’s Social Science
Research Center (SSRC).
7) Data reduction stage.
This stage is obsolete with CATI, since CATI collects the data in a form that
you can immediately analyze. But if you use an in-person or mail surveys, you
may have to manually enter the data into a computer program, such as the SPSS
program that we use in this class.
8) Design the
statistical analysis technique that you plan to use. Use a simple technique first,
such as crosstabs. In your research papers, you will stick with such simple
analytic techniques, since they are easy for potential employers to follow and to
understand.
9) Perform the data analysis,
get the results, show the tables and results, discuss the results. Check out the sample student paper
for some guidance, but this comes later in the class, after you submit your literature
review.
10) Conclusions. Write
up what you found, and answer the “so what” question. Why is your topic
important, and why are your results important. Was your theory upheld or rejected?
What other independent variables would you study in a future study? How could
you better measure your variables? In other words, discuss your ideas for future research.
LEVELS
OF MEASUREMENT
Levels of measurement deals with the
precision of the indicators that you use to measure your concepts. Are they
relatively crude, or are they very sophisticated? Your indicators in your
research papers are the four questions that you choose from the Mississippi Poll
that measure your four concepts. They are your four variables. Our class deals
with three levels of measurement (The fourth is ratio, but that is more complicated,
and I think of it as interval level anyway.). You determine the level of
measurement by looking at the categories of the variables. How can those
categories be compared with each other? These three levels of measurement are:
NOMINAL. It is the lowest level
of measurement, as it involves mere classification. You have no ability to
order the categories from lowest to highest category. An example is religious
denomination. With the answers being different religious groups, such as
Baptist, Methodist, Catholic, Islam, Jewish, Hindu, whatever, you can see that
you cannot order the categories from lowest to highest. You can only classify
poll respondents into these different groups. You can only use a simple method
of analysis, such as crosstabulations. Yet this simple method of analysis is
still informative, as we found that Catholics in Mississippi were not more
pro-life than other religious groups.
ORDINAL is the ability to order
the categories of the variable in terms of a category having more of something
than the next category. But you can't determine how much more of that quality
that the category has, compared to the other category. An example is rating the
job performance of public officials into excellent, good, fair, or poor
categories. Excellent is the highest rating, good is next, fair is next, and
poor is the lowest rating. Another example is agree-disagree categories. If you
are read a statement, such as “Abortion is the taking of a human life, and
should be completely outlawed,” and asked whether you “strongly agree, agree,
disagree, or strongly disagree” with this statement, the level of measurement
is ordinal. You can order the categories from most pro-life (strongly agree) to
most pro-choice (strongly disagree). With ordinal variables, you can use gamma
statistics to measure the strength of the relationship between two variables.
INTERVAL is the ability to order
the categories, and also determine how much of the quality the category has. Interval
variables usually have numbers that have meaning, and that denote how much of
the quality each category has. An example of an interval level variable is family
income. We list the categories in $20,000 levels. Another example is age, which
we measure by asking people in what year they were born. You can use sophisticated
regression techniques with interval data, and we will talk about this technique
later in the course.
The test will have a question on levels of measurement. It will
give you ten variables, and ask you to give one-word answers. Test out your
knowledge by answering the following test questions. Again, try to answer them
yourselves, before going to the answers at the end of the quiz.
Test Question. (10 points) What
is the level of measurement of each of the following indicators? One-word
answers for each of the 10 items is fine-- nominal, ordinal, interval.
A. How many times do you think
that President Donald Trump violated his oath of office by failing to see that
the laws were faithfully executed?
B. Which of the following
people do you most admire? Joe Biden, Donald Trump, Barack Obama, Mike Pence, or
Kamala Harris.
C. How would you rate the competence
of President Joe Biden? Excellent, good, fair, or poor?
D. Where do you think that most
of our state taxes should be spent? On health care, police forces, elementary
and secondary education, higher education, welfare, or prison construction.
E. Do you strongly agree,
agree, disagree, or strongly disagree with the following statement: “America is
a racist nation.”
F. What score did you get on
the ACT exam?
G. Last month, what was your
take home pay, after taxes and benefits were deducted from your paycheck?
H. Where do you get most of
your information about national news from? Newspapers, television, radio, news
magazines, the internet, conversations with other people.
I. How frequently do you vote
in state elections? Always, most of the time, only some of the time, or rarely.
J. How high do you think that
taxes are in Mississippi? Very high, somewhat high, about right, somewhat low,
very low.
The Answers are:
A = Interval. Supporters will say
0. Opponents will give a number ranging from 1 to maybe 1,000
B = Nominal. The response
categories are simply the names of people. Those categories cannot be ordered
from low to high. So this is mere classification. Don’t be confused by the word
“most” in the question; you need to just look at the response categories.
C = Ordinal
D = Nominal
E = Ordinal
F = Interval
G = Interval
H = Nominal. Again, look at the
response categories; don’t be confused by the word “most” in the question
itself.
I = Ordinal
J = Ordinal