lecture-09-experimentation-causal-research

M303 Experimentation and Causal Research

FALL 2026 · LECTURE 09

M303

Experimentation and Causal Research

Separate association from evidence of causality, then design credible experiments.

Where we are

Fundamentals of Data Analysis

Hypothesis testing

Correlation and regression analysis

Secondary data

Primary data

Observational and Qualitative Methods

Surveys

Experiments

Sampling

Sampling schemes, sample size

Problem

Formulation

Research

Design

Data

Collection

Data Analysis &

Recommendation

Preliminary stages of MR

Problem  actionable management problem  research objectives  research questions  hypotheses

Decide on the type of research

Exploratory, descriptive, and causal

Learning Objectives

You should be able to

Understand three necessary conditions to argue causality

Understand what an experiment is

Understand what internal and external validity mean

Understand the threats to internal validity

Learn the basic definitions in experimental design and notation used

Learn the widely used experimental designs

Understand the requirements of true experimental designs

Understand the differences between field and lab experiments

Previous-class review

01

In general, editing and coding are easier in closed-ended questions compared to open-ended ones.

02

Response rates of open-ended questions are lower in general compared to closed-ended questions.

03

Open-ended questions are more open to interviewer bias.

04

Asking more questions tends to decrease the response rates in questionnaires.

05

In a typical questionnaire, the questions related to demographics should be asked at the very beginning.

06

It is recommended to ask engaging questions in the middle.

Previous-class review (continued)

When was the last time you upgraded your computer and tablet?

Double-barreled

Assumes that the respondent has a computer and a tablet and has upgraded both.

Now that you've seen how you can save time, would you buy our service?

Leading question

What is your current age? • 20 to 30 • 31 to 40 • 41 to 50

Does not cover the entire range of ages.

Ten-year increments might be problematic.

What is your opinion of Jimmy John’s sandwiches?

Pretty good · Great · Fantastic · Incredible · The Best Ever

Interpretation of response choices might differ from respondent to respondent.

Only positive response choices are given.

Assumes that the respondent has tried Jimmy John’s before.

Example of a Causal Research Question

Campbell’s soup brand manager asks: “Will replacing our current ad with a new ad lead to an increase in consumer preferences for Campbell’s?”

Current ad

Television used in the original Campbell’s advertising example

“Fresh is Best!”

New ad

“Warms the Kid in You”

Example of a Causal Research Question (Continued)

Campbell’s ad comparison Consumer preference rating Scale shown in the source deck: 1–7

3.7

Fresh is Best

Aug. 1997

6.1

Warms the Kid in You

Nov. 1997

The rating increased after the ad changed. Does this establish that the new ad caused the increase?

What is the difference between correlation and causation?

CorrelationAn association or relationship between two variables

CausationOne variable produces an effect in another variable

Correlation ≠ Causation

Example: “Chocolate Consumption Leads to Nobel Prize”

Chocolate Consumption, Cognitive Function, and Nobel Laureates, New England Journal of Medicine

What are the conditions to argue causality?

Three conditions are necessary to argue that X causes Y:

01

Correlation

Evidence of association between X and Y

02

Temporal precedence

X must occur before Y

03

No third factor driving both X and Y

Control other causal factors

Correlation or Causation?

Before revealing the cartoon: does taking a statistics class settle the difference between correlation and causation?

Cartoon: a student says statistics class changed a mistaken belief that correlation implies causation; the reply notes that the class may have helped

What does “Well, maybe” remind us about claiming that one event caused another?

Correlation or Causation? Do more churches lead more crime?

Correlation

More churches ↔ more crime (r = .85)

2) Temporal precedence

Neither variable clearly precedes the other.

3) No third factor driving both

Large cities have more churches and more crime; smaller cities have fewer churches and less crime.

Correlation or Causation? Does living together lead to divorce?

“Living together leads to divorce”

Don’t live together if you want to stay married. So says a nationwide study of over 2000 couples. The study found that couples who had lived together before getting married were 2.3 times as likely to get divorced as couples who had not lived together.

Correlation or Causation? Does living together lead to divorce?

Correlation

Couples who had lived together before marriage were 2.3 times more likely to divorce.

2) Temporal precedence

Living together → marriage → divorce

3) No third factor driving both

Being less conservative or traditional—and more liberal or independent—could contribute to both outcomes.

What is Spurious Correlation?

Spurious = artificial, fake

Lots of things are correlated without a causal relationship

Especially a problem in big data

Examples of Spurious Correlation

Number of doctors in a region and number of people dying from disease – positively correlated

In highly dense areas, there are more doctors and more people die.

Number of police officers and number of crimes (Glass & Hopkins, 1996) – positively correlated

In highly dense areas, there are more police officers and more crimes.

Teachers’ salaries and the price of liquor (Moore and McCabe, 1993) – positively correlated

Both tend to increase over time.

Tea drinking and lung cancer – negatively correlated

Tea drinkers have a lower risk only because they smoke less.

What is an Experiment?

An experiment is a procedure in which:

One or more factors/independent variables (IVs) are systematically manipulated.

Data on the outcome/dependent variable (DV) are gathered.

Other variables (called extraneous variables) that may influence the effect of the IVs on the DV are controlled.

Example of an Experiment: Shopping Lists

1 1/2 lbs. hamburger

2 loaves Wonder bread

bunch of carrots

1 can Rumford’s Baking Powder

Nescafe instant coffee

can Del Monte peaches

5 lbs. potatoes

1 1/2 lbs. hamburger

2 loaves Wonder bread

bunch of carrots

1 can Rumford’s Baking Powder

1 lb. can Maxwell House coffee (drip ground)

can Del Monte peaches

5 lbs. potatoes

Group 1

Group 2

Do you think this housewife is:

Lazy (Yes/No)

Fail to plan (Yes/No)

Thrifty (Yes/No)

Spendthrift (Yes/No)

Not a good wife (Yes/No)

Good wife (Yes/No)

Example of an Experiment: Shopping Lists (Continued)

Design?

Single factor/IV

2 levels (shopping list: Maxwell Drip Ground vs. Nescafe Instant)

IV(s)?

Type of branded coffee: Maxwell Drip Ground vs. Nescafe Instant

DV?

Personality attributions

Lazy, fails to plan, thrifty, spendthrift, not a good wife, or a good wife

Ex 1: https://www.youtube.com/watch?v=qRIfbkbgsjI

Ex 2: https://www.youtube.com/watch?v=SC3T6vx7Jv4

Open source media ↗Open source media ↗

Experiments in Research

Daniel Kahneman

First Psychologist to win a Nobel Prize in Economics

Points out limitations to rational economic models in explaining real world

Validity in Experiments

Internal Validity

Whether the manipulation of the independent variables actually causes the effect

External Validity

Whether the experiment results can be generalized beyond the experimental setting

Without internal validity, external validity does not even exist.

What are the threats to Internal Validity?

History effect: Something happened in the marketplace.

Events external to the experiment that affect the responses of the people involved in the experiment.

What are the threats to Internal Validity?

Old

Campbell’s Commercial

Sales:

4.2M cans

“Fresh is Best!”

New

Campbell’s Commercial

Sales:

6.1M cans

“Warms the Kid in You”

Aug ‘97

Nov‘97

Progresso Increases Prices

What are the threats to Internal Validity?

History effect: Something happened in the marketplace.

Events external to the experiment that affect the responses of the people involved in the experiment.

Maturation effect: Something changed in the participants.

Changes in the respondents that are a consequence of time, such as aging, getting hungry, or getting tired.

What are the threats to Internal Validity?

Old

Campbell’s Commercial

Sales:

4.2M cans

“Fresh is Best!”

New

Campbell’s Commercial

Sales:

6.1M cans

“Warms the Kid in You”

Aug ‘97

Nov‘97

Customers felt colder

What are the threats to Internal Validity?

History effect: Something happened in the marketplace.

Events external to the experiment that affect the responses of the people involved in the experiment.

Maturation effect: Something changed in the participants.

Changes in the respondents that are a consequence of time, such as aging, getting hungry, or getting tired.

Pre-test effect: Previous measurement took place.

Being measured previously might affect that person’s future behavior.

What are the threats to Internal Validity?

Old

Campbell’s Commercial

Liking:

4/7

“Fresh is Best!”

New

Campbell’s Commercial

Liking:

6.5/7

“Warms the Kid in You”

10:00am

11:00am

9:30am

10:30am

What are the threats to Internal Validity?

History effect: Something happened in the marketplace.

Events external to the experiment that affect the responses of the people involved in the experiment.

Maturation effect: Something changed in the participants.

Changes in the respondents that are a consequence of time, such as aging, getting hungry, or getting tired.

Pre-test effect: Previous measurement took place.

Being measured previously might affect that person’s future behavior.

Instrument variation: Use of different methods.

The method used to collect data changes within the experiment (e.g., questionnaire, interviewer, etc.).

What are the threats to Internal Validity?

Old

Campbell’s Commercial

Sales:

4.2M cans

“Fresh is Best!”

New

Campbell’s Commercial

Sales:

6.1M cans

“Warms the Kid in You”

Aug ‘97

Nov‘97

Measured in the whole US market

Measured in the Test (Northeast) market

What are the threats to Internal Validity?

History effect: Something happened in the marketplace.

Events external to the experiment that affect the responses of the people involved in the experiment.

Maturation effect: Something changed in the participants.

Changes in the respondents that are a consequence of time, such as aging, getting hungry, or getting tired.

Pre-test effect: Previous measurement took place.

Being measured previously might affect that person’s future behavior.

Instrument variation: Use of different methods.

The method used to collect data changes within the experiment (e.g., questionnaire, interviewer, etc.).

Selection bias: Non-random assignment.

If units self-select into the treatment and control groups, this is a serious concern when the reason for selection is related to the outcome of interest.

What are the threats to Internal Validity?

Old

Campbell’s Commercial

Sales:

4.2M cans

“Fresh is Best!”

New

Campbell’s Commercial

Sales:

6.1M cans

“Warms the Kid in You”

Aug ‘97

Nov‘97

Measured in the west

Measured in the East

Aired in the West

Aired in the East

What are the threats to Internal Validity?

History effect: Something happened in the marketplace.

Events external to the experiment that affect the responses of the people involved in the experiment.

Maturation effect: Something changed in the participants.

Changes in the respondents that are a consequence of time, such as aging, getting hungry, or getting tired.

Pre-test effect: Previous measurement took place.

The fact that someone has been measured previously might affect that person’s future behavior.

Instrument variation: Use of different methods.

The method used to collect data changes within the experiment (e.g., questionnaire, interviewer, etc.).

Selection bias: Non-random assignment.

If units self-select into the treatment and control groups, this is a serious concern when the reason for selection is related to the outcome of interest.

Mortality: dropout.

The sample becomes unrepresentative.

Example: Amazon’s price-testing controversy

SEATTLE -- Amazon.com Inc. has been charging customers different prices for the same products in its online video store, in a marketing test that has irked some users. The Internet merchant said it recently began charging customers different prices for the same DVD titles. For example, the company has charged some users $23.97 and others $25.97 for a DVD version of the movie "Men in Black." Patty Smith, an Amazon spokeswoman, said the different prices were part of a test Amazon is conducting "to measure what impacts a customer's decision to purchase or not to purchase" have on the site. Ms. Smith said Amazon customers are selected randomly for the test and that the prices they receive aren't based on any other characteristics.

Basics of Experimental Design - Definitions

Factor / Independent variable: manipulated variable

Level: value of the factor/IV

Outcome/Dependent Variable: outcome that is being measured

Measurement/Observation: recording of/observing the response

Extraneous/Control variables: other variables that might affect the outcome/dependent variable

Treatment/Experimental group: portion of the sample (or population) exposed to the manipulation of the factor/independent variable

Control group: portion of the sample (or population) for which the factor/independent variable remains unchanged

Experimental Design - Notation

O → Observation or measurement

X → Exposure of the experimental units to the treatment

EG → Experimental group

CG → Control group

R → Random Assignment

What are the widely used experimental designs?

Pre-experimental Designs

One group pretest-posttest

True Experimental Designs

Experimental and control groups post-test only

Experimental and control groups pre-test – post-test

What are the problems?

Pre-experimental Designs: One group pretest-posttest

EG O1 X O2

NEW Commercial

Month 2

EG

Sales

Measured

Month 3

O2

Sales

Measured

Month1

O1

(O2 – O1)

What are the requirements of a true experimental design?

Control and experimental groups have to exist!

Random assignment of subjects to treatment is critical!

It ensures that the groups differ only with respect to the independent variable but not the control variables.

True Experimental Designs: Experimental and control groups post-test only

Simplest true experimental design

EG X O1

CG O2

Sales

Measured

Sales

Measured

Week 3

O2

O1

(R)

NEW commercial

to Half Consumers

OLD commercial

to the Other Half

Week 1

X

CG

EG

True Experimental Designs: Experimental and control groups pre-test – post-test

Adding a baseline observation

EG O1 X O3

CG O2 O4

NEW Commercial

to Half Consumers

OLD Commercial

the Other Half

Week 3

X

CG

EG

Sales

Measured

Sales

Measured

Week 5

O4

O3

Sales

Measured

Sales

Measured

Week 1

O2

O1

(R)

(O3 – O1) - (O4 – O2)

Field vs. laboratory experiments

Laboratory Experiment

Field Experiment

Low (Artificial)

High (Few

Control Variables)

High

Low / Short

Subjects Aware of

Participation

High (Natural)

Low (Many

Control Variables)

Low

High / Long

High

Subjects Unaware of

Participation

Field vs. Laboratory

External Validity

Internal Validity

Control

Cost/Time

Exposure to Competition

Subjects’ Awareness

Sum Up

Three necessary conditions to argue X causes Y

Correlation, temporal precedence, and no other factor driving X and Y

Internal and external validity

Internal validity: whether manipulating the independent variable actually causes the effect

External validity: whether the experimental results can be generalized beyond the experimental setting

Threats to internal validity

History, maturation, pre-test effects, and related threats

Basic definitions and notation used in experimental design

IV, DV, experimental group (EG), control group (CG), and related notation

Widely used experimental designs

Pre-experimental designs and true experimental designs

Differences between field and laboratory experiments

An Experiment

In order to go green, SaveOnFoods has announced a new policy:

Bring your own bags, you will get 10¢ discount per bag.

Please think about your own behavior, will you bring your own bags? (Yes/No)

Exercise: Evaluating Experimental Design

A retail hardware chain is considering a self-service system for its 326 stores. Ten stores have been selected for a trial period of 3 months. Ten other stores have been selected in which the traditional shop assistant will operate. Dollar sales for each of the 20 stores will be recorded weekly for the three-month period. Prior to the study, weekly sales will be recorded for a month, a ‘before’ measure. The difference in sales between the experimental and control groups is expected to help the management in its decision.

Exercise Questions

What type of experiment is this? (lab or field?)

What might you say about the internal and external validity (in general)?

What is the independent variable in this study?

What is the dependent variable?

Are there any potential extraneous variables which could influence the results?

What are your suggestions for controlling these extraneous variables?

To sum up

In order to go green, SaveOnFoods has announced a new policy:

If you do not bring your own bags, you need to pay 10¢ per bag.

Please think about your own behavior, will you bring your own bags? (Yes/No)

Next class

COMING UP

Sampling Fundamentals

Next lectureSampling Fundamentals

Following lectureSample Size and Statistical Theory — Part I