Hypothesis Testing in Cognitive Research
Rakesh Sengupta
SR University
April 1, 2025
1
Outline
Hypothesis Testing Fundamentals
t-tests
Analysis of Variance (ANOVA)
Mixed-Methods Designs
Conclusion
2
Hypothesis Testing Fundamentals
Null and Alternative Hypotheses: Definitions
Core Concepts
• Null Hypothesis (H◦ ): Statement of no effect/difference (status quo)
• Alternative Hypothesis (H1 ): Statement of effect/difference (research hypothesis)
• Purpose: Framework for statistical decision-making under uncertainty
Hypothesis Testing Logic
• Assume H◦ is true
• Calculate probability of observed data under H◦
• If probability is very low, reject H◦ in favor of H1
Example
Simple Example Testing a new drug:
• H◦ : New drug has no effect
• H1 : New drug has an effect
3
Hypothesis Forms
Different Forms of Hypotheses
H◦ : µ = µ ◦ vs H1 : µ ̸= µ◦ (Two-tailed)
H◦ : µ ≤ µ ◦ vs H1 : µ > µ ◦ (One-tailed, Upper)
H◦ : µ ≥ µ ◦ vs H1 : µ < µ ◦ (One-tailed, Lower)
Choosing the Right Form
• Two-tailed: When interested in any difference (direction doesn’t matter)
• One-tailed: When interested in a specific direction of effect
• Requires a priori justification for one-tailed tests
4
Visual Representation of Hypothesis Testing
α = 0.05 (Two-tailed)
Probability
Rejection Region Rejection Region x
Acceptance Region
Figure 1: Normal Distribution with Rejection Regions
Key Elements
• Sampling distribution under H◦
• Critical values defining rejection regions
• α level determines region size
5
Key Assumptions of Hypothesis Testing
Fundamental Assumptions
• Random Sampling: Data must represent the population
• Independence: Observations must be independent
• Significance Level: α predetermined (typically 0.05)
• Distributional Assumptions: Test statistic follows known distribution under H◦
Consequences of Violations
• Invalid p-values
• Increased Type I/II error rates
• Misleading conclusions
6
Practical Example in Cognitive Research
Example
Memory Training Study Objective: Test if a new memory training program improves recall
performance
• Population: Adults aged 65+ with normal cognition
• Sample: 50 participants randomly assigned to training or control group
• Measurement: Number of words recalled after 1 hour
Hypotheses:
H◦ : µtraining ≤ µcontrol vs H1 : µtraining > µcontrol
Next Steps
• Choose appropriate statistical test
• Calculate test statistic
• Determine p-value
• Make decision to reject or fail to reject H◦
7
Type I and Type II Errors
Error Matrix
H◦ True H◦ False
Reject H◦ Type I Error (α) Correct (Power)
Fail to Reject H◦ Correct Type II Error (β)
Table 1: Error Matrix with Color Coding
Key Definitions
• Type I Error: False positive (Rejecting H◦ when it’s true)
• Type II Error: False negative (Failing to reject H◦ when it’s false)
• Power: Probability of correctly rejecting H◦ (1 - β)
8
Practical Implications of Type I and Type II Errors
Examples in Cognitive Research
• Drug Efficacy Study:
◦ Type I Error: Claiming a drug works when it doesn’t
◦ Type II Error: Missing a truly effective drug
• Memory Experiment:
◦ Type I Error: Detecting a memory improvement that doesn’t exist
◦ Type II Error: Missing a real memory improvement
Error Trade-off
• Lowering α reduces Type I errors but increases Type II errors
• Increasing sample size reduces both Type I and Type II errors
• Power analysis helps determine appropriate sample size
9
Controlling Type I and Type II Errors
Strategies for Error Control
• Significance Level (α):
◦ Typically set to 0.05, but can be adjusted based on context
◦ More critical applications may use lower α (e.g., 0.01)
• Power Analysis:
◦ Calculates required sample size to achieve desired power (typically 0.80)
◦ Factors: Effect size, α, sample size, and variability
• Effect Size:
◦ Practical significance vs. statistical significance
◦ Cohen’s d, eta-squared, etc.
Best Practices
• Pre-register analysis plan to avoid data dredging
• Use appropriate correction methods for multiple comparisons
• Report both p-values and confidence intervals
• Consider Bayesian methods when appropriate
10
t-tests
One-Sample t-test
Formula
Definition and Purpose X̄ − µ
t= √
• Compares a sample mean to a known s/ n
population mean
• Used when population variance is unknown • X̄ : Sample mean
• Tests whether the sample comes from a • µ: Population mean
population with a specific mean • s: Sample standard deviation
• n: Sample size
Example
Reaction Time Study Testing if average reaction time (RT) differs from population mean:
• Sample: X̄ = 450 ms, s = 30, n = 30
• Population: µ = 500 ms
450−500
• Calculation: t = √
30/ 30
= −9.128, Degrees of freedom: df = n − 1 = 29
• Result: Significant difference (p < 0.001) 11
One-Sample t-test: Key Considerations
Assumptions
• Normality: Data should follow a normal distribution
◦ Central Limit Theorem applies for large n (n ≥ 30)
◦ For small n, verify normality with Shapiro-Wilk test or Q-Q plots
• Continuous Data: Measurement scale should be interval or ratio
• Random Sampling: Observations must be randomly selected
When to Use
• Testing if a sample mean differs from a known population mean
• When population variance is unknown
• Common in pre-test/post-test designs with single groups
Limitations
• Sensitive to violations of normality with small samples
• Does not account for contextual factors
12
Independent Samples t-test
Formula
Definition and Purpose
X̄1 − X̄2
• Compares means of two independent t=q 2
s1 s22
groups n1 + n2
• Tests whether the groups come from
populations with different means • X̄1 , X̄2 : Group means
• Used for between-subjects designs • s12 , s22 : Group variances
• n1 , n2 : Group sample sizes
Example
Drug Efficacy Study Comparing RT between control and experimental groups:
• Control: X̄1 = 450, s1 = 30, n1 = 25
• Experimental: X̄2 = 500, s2 = 35, n2 = 25
450−500
• Calculation: t = q
302 352
= −5.09, Deg. of freedom: df ≈ 47 (Welch-Satterthwaite eq.)
25 + 25
• Result: Significant difference (p < 0.001)
13
Degrees of Freedom Calculation (Welch-Satterthwaite)
Formula
2
s12 s22
n1 + n2
df = 2 2 2
s s2
1 2
n1 n2
n1 −1 + n2 −1
14
Degrees of Freedom Calculation (Welch-Satterthwaite)
• Given Data:
◦ Control: n1 = 25, s1 = 30
◦ Experimental: n2 = 25, s2 = 35
• Step 1: Calculate variance terms
s12 302
◦ n1
= 25
= 36
s22 352
◦ =
n2
= 49
25
• Step 2: Sum variance terms
◦ 36 + 49 = 85
• Step 3: Square the sum
◦ 852 = 7225
• Step 4: Calculate denominator terms
(36)2
◦ 24
= 54
(49)2
◦ 24
≈ 100.04
• Step 5: Sum denominator terms
◦ 54 + 100.04 ≈ 154.04
• Step 6: Divide and round
7225
◦ df = 154.04 ≈ 46.9 ≈ 47
15
Degrees of Freedom Calculation (Welch-Satterthwaite)
Why This Method?
• Accounts for unequal variances between groups
• Provides more accurate p-values
• Used when Levene’s test indicates unequal variances
16
Independent Samples t-test: Key Considerations
Assumptions
• Independence: Observations between groups must be independent
• Homogeneity of Variance: Equal variances between groups
◦ Verify with Levene’s test
◦ If violated, use Welch’s t-test (adjusts degrees of freedom)
• Normality: Each group should follow a normal distribution
• Continuous Data: Measurement scale should be interval or ratio
When to Use
• Comparing two independent groups
• Random assignment to groups
• Common in experimental and quasi-experimental designs
Effect Size
X̄1 −X̄2
• Cohen’s d: sp (pooled standard deviation)
• Interpretation: 0.2 (small), 0.5 (medium), 0.8 (large) 17
Dependent Samples t-test
Definition and Purpose
• Compares means of related groups (paired measurements)
• Tests whether the mean difference is significantly different from zero
• Used for within-subjects designs
Example
Formula Memory Training Study Pre-post training RT
measurements:
D̄
t= √ • Mean difference: D̄ = 25 ms
sD / n
• SD of differences: sD = 15
• D̄: Mean difference between paired • Sample size: n = 20
observations 25
• Calculation: t = √
15/ 20
= 7.45
• sD : Standard deviation of differences
• Degrees of freedom: df = n − 1 = 19
• n: Number of pairs
• Result: Significant improvement (p < 0.001)
18
Dependent Samples t-test: Key Considerations
Assumptions
• Related Observations: Data must be paired (e.g., pre/post, matched pairs)
• Normality: Differences should follow a normal distribution
◦ For small n, verify with Shapiro-Wilk test or Q-Q plots
• Continuous Data: Measurement scale should be interval or ratio
When to Use
Advantages
• Repeated measures (pre/post designs)
• Controls for individual differences
• Matched-pair designs
• Generally more powerful than independent
• Longitudinal studies
samples t-test
• Any situation where observations are not
• Requires fewer participants
independent
Effect Size
D̄
• Cohen’s d: sD
• Interpretation: Same as independent samples 19
Analysis of Variance (ANOVA)
One-Way ANOVA
Definition and Purpose
• Tests differences among three or more group means
• Extends the independent samples t-test to multiple groups
• Assesses whether at least one group mean is different from the others
Example
Formula
Drug Efficacy Study Comparing reaction times
MSbetween (RT) across three drug groups:
F =
MSwithin • Group A: X̄1 = 450, n1 = 10
• Group B: X̄2 = 500, n2 = 10
• MSbetween : Mean square between groups • Group C: X̄3 = 550, n3 = 10
(variance explained by the model)
• Calculation: F = 12.5, p < 0.001
• MSwithin : Mean square within groups
(residual variance) • Interpretation: At least one group mean is
significantly different
20
One-Way ANOVA: Key Considerations
Assumptions
• Independence: Observations must be independent between groups
• Normality: Data should follow a normal distribution within each group
• Homogeneity of Variance: Equal variances across groups (verified by Levene’s test)
Post-hoc Tests
• Tukey’s HSD: Compares all possible pairs of means while controlling family-wise error rate
• Bonferroni Correction: Adjusts significance level for multiple comparisons
• When to Use: After finding a significant ANOVA result to determine which specific
groups differ
Effect Size
• η 2 (Eta-squared): Proportion of variance explained by the model
• Interpretation: 0.01 (small), 0.06 (medium), 0.14 (large)
21
Factorial ANOVA
Definition and Purpose
• Tests the effects of two or more independent variables and their interactions
• Allows examination of main effects and interaction effects
• Commonly used in experimental designs with multiple factors
Response
Key Concepts
• Main Effects: The effect of one
Factor B Level 1
independent variable averaged across levels
Factor B Level 2 of the other variable
• Interaction Effects: The combined effect
Factor A of two or more variables
Interaction Effect • Higher-order Interactions: Interactions
involving three or more variables
Figure 2: Factorial Design Interaction
22
Factorial ANOVA: Practical Example
Example
Memory Training Study
• Factor A: Training type (Traditional vs. Advanced)
• Factor B: Training duration (4 weeks vs. 8 weeks)
• Dependent Variable: Memory recall score
Memory Score
Traditional Training
Advanced Training
Training Duration
Interaction Effect
Figure 3: Factorial Design Example
23
Factorial ANOVA: Key Considerations
Assumptions
• Independence: Observations must be independent
• Normality: Data should follow a normal distribution within each group
• Homogeneity of Variance: Equal variances across all groups
• Homogeneity of Covariance: Equal covariance matrices for repeated measures
Effect Size
• η 2 (Eta-squared): Overall effect size
• ω 2 (Omega-squared): Adjusted effect size accounting for sample size
Interpreting Interactions
• Interaction effects can mask main effects
• Significant interactions require simple effects analysis
• Graphical representation helps interpretation
24
Mixed-Methods Designs
Mixed-Methods in Cognitive Research
Definition
Integration of quantitative and qualitative methods
Quantitative Qualitative
Surveys, Experi- Interviews, Observa-
ments tions
Statistical analysis Thematic analysis
Table 2: Methodological Comparison
Example
Memory research:
• Quantitative: RT measurements
• Qualitative: Participant interviews
25
When to Use Mixed-Methods
Advantages
• Triangulation of results
• Richer understanding
• Contextual validation
Challenges
• Integration complexity
• Methodological validity
• Resource-intensive
26
Conclusion
Summary
• Hypothesis testing framework
• t-tests for mean comparisons
• ANOVA for multiple groups
• Factorial designs for interactions
• Mixed-methods for comprehensive insights
Key Takeaway
Statistical test selection depends on:
• Research question
• Data type
• Study design
27
References
Cohen, J. (2013). Statistical Power Analysis for the Behavioral Sciences.
Howell, D. C. (2019). Statistical Methods for Psychology.
Creswell, J. W. and Clark, V. L. P. (2007). Designing and Conducting Mixed Methods
Research.
28
Thank You!
Questions?
29
