17  Mc Nemar’s Test

McNemar test

The McNemar test is a statistical test used to analyze paired nominal data or matched data collected from two related groups or conditions. It assesses whether there is a significant difference in proportions or frequencies of a categorical outcome between paired observations. The McNemar test is particularly useful when dealing with data that are not independent, such as before-and-after measurements on the same subjects or matched pairs in case-control studies.

17.1 Understanding the McNemar Test:

1. Null and Alternative Hypotheses:

• Null Hypothesis (H0): There is no difference in the proportions or frequencies of the categorical outcome between the two conditions.
  
• Alternative Hypothesis (H1): There is a significant difference in the proportions or frequencies of the categorical outcome between the two conditions.

2. Test Statistic:

• The McNemar test statistic is calculated based on the number of discordant pairs, i.e., pairs where the outcomes differ between the two conditions.
  
• It follows a chi-squared distribution with one degree of freedom under the null hypothesis.

3. Calculation of Test Statistic:

• Let a be the number of discordant pairs where a subject has a positive outcome in the first condition and a negative outcome in the second condition.
  
• Let b be the number of discordant pairs in the opposite direction.
  
• The uncorrected McNemar test statistic is:

\[\chi^2_{M} = \frac{(b - a)^2}{b + a}\]

• The corrected McNemar test statistic (Yates’ continuity correction to adjust for the discrete nature of the data)is:

\[\chi^2_{M, corrected} = \frac{(|b - a| - 1)^2}{b + a}\]

4. Interpretation of Results:

• If the calculated \(\chi^2_M\) value is greater than the critical value from the chi-squared distribution with 1 degree of freedom at the chosen significance level (commonly \(\alpha = 0.05\)), we reject the null hypothesis, indicating a significant difference.
  
• The continuity-corrected value is slightly lower than the uncorrected value, but in most practical cases, both lead to the same conclusion.

17.1.1 McNemar Test: Corrected and Uncorrected

The McNemar test is designed to analyze paired nominal data, particularly when the outcomes are dichotomous (e.g., pass/fail). There are two approaches to calculate the test statistic: uncorrected and continuity-corrected.

Uncorrected McNemar Test

• Calculates the chi-squared statistic directly from the discordant pairs.
• Formula:

\[ \chi^2_{uncorrected} = \frac{(b - a)^2}{b + a} \]

• a = number of pairs where the first condition is positive and the second is negative.
• b = number of pairs where the first condition is negative and the second is positive.
• The uncorrected test shows the theoretical difference but may slightly overestimate significance with small samples.

Continuity-Corrected McNemar Test

• Also known as Yates’ correction, adjusts for small sample discrete data.
• Formula:

\[ \chi^2_{corrected} = \frac{(|b - a| - 1)^2}{b + a} \]

• This correction reduces the test statistic slightly, making it more conservative.
• Recommended for small discordant pairs.

Summary: The uncorrected McNemar test is straightforward, while the continuity-corrected version provides a more conservative estimate. Both are valid, but the corrected test is often preferred for small sample sizes or when using standard software implementations.

Considerations:

• The McNemar test assumes that the data are paired and dichotomous.
  
• It is sensitive to small sample sizes, especially when the number of discordant pairs is small.
  
• Extensions exist for analyzing categorical data with more than two categories.

17.1.2 Applications of McNemar Test

A. Medical Research:

• In clinical trials, the McNemar test is used to assess whether there is a significant difference in treatment outcomes between two treatment groups or before and after treatment within the same group.

B. Education:

• Educational researchers may use the McNemar test to evaluate the effectiveness of teaching methods or interventions by comparing pre-test and post-test scores of students.

C. Epidemiology:

• Epidemiologists use the McNemar test to analyze matched case-control studies or cohort studies where data are collected from the same subjects at different time points.

D. Psychology and Social Sciences:

• Researchers in psychology and social sciences utilize the McNemar test to analyze paired survey responses, preferences, or behaviors before and after exposure to certain stimuli or interventions.

17.2 Example problem: McNemar test.

A study was conducted to evaluate the effectiveness of a new teaching method aimed at improving students’ performance in a mathematics examination. Data were collected from 50 students, each of whom took a pre-test (before the introduction of the new method) and a post-test (after its implementation). In both assessments, students’ outcomes were classified as either “Pass” or “Fail.”

The results are summarized as follows:

• In the pre-test, 25 students passed and 25 students failed.
  
• In the post-test, 35 students passed and 15 students failed.
  
• Of the students who passed the pre-test, 20 also passed the post-test.
  
• Of the students who failed the pre-test, 15 subsequently passed the post-test.

These results can be represented in the following contingency table:

	Passed Post-test (Yes)	Passed Post-test (No)
Passed Pre-test (Yes)	20	5
Passed Pre-test (No)	15	10

The objective of this analysis is to determine whether there is a significant change in the proportion of students who passed before and after the introduction of the new teaching method.

This comparison involves paired categorical data, making it suitable for testing with McNemar’s Test, which assesses whether the proportion of discordant pairs (i.e., students who changed from pass→fail or fail→pass) differs significantly.

17.2.1 Calculation of McNemar Test (Continuity-Corrected)

1. Create a Contingency Table:

	Passed Post-test (Yes)	Passed Post-test (No)
Passed Pre-test (Yes)	20	5
Passed Pre-test (No)	15	10

2. Calculate the McNemar Test Statistic (Continuity-Corrected):

   • \(a\) = Number of discordant pairs where a student passed the pre-test but failed the post-test = 5
   • \(b\) = Number of discordant pairs where a student failed the pre-test but passed the post-test = 15

   \[\chi^2_{M, corrected} = \frac{(|b - a| - 1)^2}{b + a}\] \[\chi^2_{M, corrected} = \frac{(|15 - 5| - 1)^2}{15 + 5} = \frac{(10 - 1)^2}{20} = \frac{81}{20} = 4.05\]

3. Determine the Critical Value:

   • With 1 degree of freedom and \(\alpha = 0.05\), the critical value of the chi-squared distribution is approximately 3.84.

4. Compare the Test Statistic to the Critical Value:

   • Since \(\chi^2_{M, corrected} = 4.05 > 3.84\), we reject the null hypothesis.

Interpretation:

Based on the McNemar test, we conclude that there is a significant difference in the proportions of students passing the exam before and after the teaching method was introduced. In this example, the teaching method appears to have a significant positive effect on students’ performance in the mathematics exam.

McNemar Test in R and Python

McNemar Test in R

# Create a contingency table
cont_table <- matrix(c(20, 5, 15, 10), 
                            nrow = 2, byrow = TRUE)

# Assign row and column names for clarity
rownames(cont_table) <- c("Pre Passed", "Pre Failed")
colnames(cont_table) <- c("Post Passed", "Post Failed")

print(cont_table)

           Post Passed Post Failed
Pre Passed          20           5
Pre Failed          15          10

# Perform McNemar test and print the full result

result <- mcnemar.test(cont_table, correct = TRUE)

print(result)

    McNemar's Chi-squared test with continuity correction

data:  cont_table
McNemar's chi-squared = 4.05, df = 1, p-value = 0.04417

# Interpret the result 
alpha <- 0.05
p <- result$p.value

  
if (p < alpha) {
  cat("(P-value)",  p, "<", alpha, 
      "(significance level / alpha).",  "\n",
      " Reject the null hypothesis: 
      There is a significant difference.\n")
} else {
  cat("(P-value)",  p, ">=", alpha, 
      "(significance level / alpha).", "\n",
      " Do not Reject the null hypothesis: 
      No significant difference.\n")
}

(P-value) 0.04417134 < 0.05 (significance level / alpha). 
  Reject the null hypothesis: 
      There is a significant difference.

Python

McNemar Test in Python

import numpy as np
import pandas as pd
from statsmodels.stats.contingency_tables import mcnemar

#Create a contingency table

table_array = np.array([[20, 5],
                        [15, 10]])

# Convert to a pandas DataFrame for a formatted output
cont_table = pd.DataFrame(table_array,
                index=["Pre Passed", "Pre Failed"],
                columns=["Post Passed", "Post Failed"])

print(cont_table)

            Post Passed  Post Failed
Pre Passed           20            5
Pre Failed           15           10

# Perform the McNemar test and print the full result

result = mcnemar(table_array, exact=False, correction=True)

print(result)

pvalue      0.0441713449084427
statistic   4.05

# Interpret the result with a clear conclusion

alpha = 0.05
p = result.pvalue

if p < alpha:
    print(f"(P-value) {p:.4f} < {alpha} (sig level / alpha).")
    print("Reject the null hypothesis:", "\n", "There is a significant difference.\n")
else:
    print(f"(P-value) {p:.4f} >= {alpha} (significance level / alpha).")
    print("Do not reject the null hypothesis:", "\n", "No significant difference.\n")

(P-value) 0.0442 < 0.05 (sig level / alpha).
Reject the null hypothesis: 
 There is a significant difference.