[POLS 4150] Significance Tests

Five parts of significance tests

1. Assumptions

2. Hypotheses

3. Test statistic

4. P-value

5. Conclusion/inference

Assumptions

Valid tests about parameters require assuming a few things about the data:

1. Proper definition of the population - If your sampling frame is not representative of the population, your inferences will almost always be wrong/inaccurate.

2. Randomization - Significance tests assume that samples were randomly selected from a population. They essentially fail if this is not true.

3. Distributional assumptions - Some tests assume that variables follow a particular probability distribution (Normal, Binomial,etc.)

4. Sample size - validity of tests increases with larger sample sizes.

Formalizing Hypotheses

\[ \begin{aligned} H_{0}: & \text{ The null hypothesis.}\\ H_{a}:& \text{ The alternative hypothesis.}\\ \end{aligned} \]

• Every significance tests has two hypotheses about the value of the parameter.

• The null hypothesis - a prior belief about a value that the parameter takes.

• The alternative hypothesis - a belief about the value of the parameter if the null hypothesis is found to be untrue/unlikely.

• In general, the null hypothesis represents the status quo or “no effect”.

Do polls suggest that it’s reasonable to believe that Trump will carry Michigan?

\[ \begin{aligned} H_{0}: & \pi_{Trump,MI} = 0.50 \\ H_{a}:& \pi_{Trump,MI} > 0.50 \\ \end{aligned} \] - A presidential candidate gets the electoral votes of a state if they carry over 50% of the votes.

• Samples of registered voters in Michigan can be used to assess whether it is likely that a candidate will carry the state given the polling data.

Are Democrats less Christian than Republicans, on average?

\[ \begin{aligned} H_{0}: & \pi_{CDem} = \pi_{CRep} \\ H_{a}:& \pi_{CDem} < \pi_{CRep} \\ \end{aligned} \] - We can explore whether the population % of Democrats that are Christian are less than the population % of Republicans that are Christian from survey data.

Did shaming voters increase turnout?

\[ \begin{aligned} H_{0}: & Turnout_{T} = Turnout_{C} \\ H_{a}:& Turnout_{T} > Turnout_{C} \\ \end{aligned} \]

• Gerber, Green and Larimer (2008) “Social Pressure and Voter Turnout, Evidence from a Large Field Experiment”

Test statistic

\[ \frac{Obs - Expected}{SE} \]

• The test statistic is what you use to determine whether to reject or not reject the null hypothesis.

• It’s usually calculated as the difference between the observed and expected under the null, divided by the standard error.

P-value

• The p-value is necessary to assess how likely the null hypothesis is given the data.

• Indicates the probability that the test statistic is equal to or more extreme than the value observed as predicted by \(H_{a}\).

• Calculated by presuming that \(H_{0}\) is true.

• Small p-values \(<0.05\) or \(<0.01\) suggest that the observed data are unusual if \(H_{0}\) were true.

Significance test for a mean: are Georgians more conservative than South Carolinians?

• Survey of \(N = 100\) people from GA, \(N = 100\) people from SC.

• Ask: “On a scale from 1 to 5 where 1 is Very Liberal and 5 is Very Conservative, where would you place yourself?”

• Results: \(\bar{x}_{GA} = 3\), \(\bar{x}_{SC} = 4\), \(SE = 1\).

Significance test for a mean: Are older people (65+) “cold” toward muslims and atheists?

Significance test for a mean: Are older people (65+) “cold” toward muslims and atheists?

• Survey of \(N = 1600\) people.

• Results:

Significance test for a mean: Are older people (65+) “cold” toward muslims and atheists?

Steps 1. Define \(H_{0}\) and \(H_{a}\)

2. Choose a test statistic.

3. Find the p-value for the test statistic (p).

4. Compare the p-value to a preset significance value: (\(\alpha = 0.05\) or \(\alpha = 0.01\)).

5. Reject \(H_{0}\)

Putting this all together…

• Assumptions

• Hypotheses

• Test statistic

• P-value

• Conclusion

Significance test for a proportion

\[ z = \frac{\hat{\pi} - \pi_{0}}{\sigma_{0}}; \sigma_{0} = \sqrt{\pi_{0}(1-\pi_{0})/n} \]

• Everything is identical, with the exception of the test statistic and the standard error.

Significance test for a proportion: Do a majority of Republicans think Trump is “even tempered”?

Significance test for a proportion: Do a majority of Republicans think Trump is “even tempered”?

• Let’s assume that N = 1600.

1. Decribe the null and alternative hypothesis.
2. calculate the test statistic.
3. make a decision about the question.

Significance tests in R,

• Let’s make up some feeling thermometer data and do a couple of hypothesis tests using the t.test() function

** Can the population mean be less than 50?“”

religdata<-rnorm(n=100,mean=43,sd = 2)
# Can the population mean be 50 or more? versus less than 50?
t.test(x= religdata,
       mu = 50,
       alternative = "less")

## 
##  One Sample t-test
## 
## data:  religdata
## t = -32.97, df = 99, p-value < 2.2e-16
## alternative hypothesis: true mean is less than 50
## 95 percent confidence interval:
##      -Inf 43.46561
## sample estimates:
## mean of x 
##  43.11908

Significance tests in R

** Can the population mean be equal to 44?**

religdata<-rnorm(n=100,mean=43,sd = 2)
# Can the population mean be 50 or more? versus less than 50?
t.test(x= religdata,
       mu = 44,
       alternative = "two.sided")

## 
##  One Sample t-test
## 
## data:  religdata
## t = -5.6771, df = 99, p-value = 1.37e-07
## alternative hypothesis: true mean is not equal to 44
## 95 percent confidence interval:
##  42.53130 43.29206
## sample estimates:
## mean of x 
##  42.91168

Accepting v. not rejecting the null hypothesis

• It is not correct to say that you “accept” the null hypothesis.

• This is because you are testing whether a parameter value is one of many saying that you “do not reject” emphasizes the fact that you can’t rule out that the null is the correct parameter value.

• But there may be, several other plausible values.