text-messages

2019 Cold SMS message testing: Vote Tripling vs Friends & Family vs Social Pressure

Debra Cleaver

Debra Cleaver

— 7 min read

Program conception and fundraising by Debra Cleaver. Randomized controlled trial (RCT) experiment design and analysis by Christopher Mann, Ph.D. Program implementation by a National Civic Organization NCO.

Program goals

  • We wanted to determine the most effective messaging on both the recipient and the recipients' households. These are respectively called the "direct effects" and the "dispersion effects."
  • We were also curious if there would be differences based on gender, past voting history, race and ethnicity, party affiliation, and partisanship. We examined this both at the individual level, and at the household level.

Program implementation

  • We targeted registered voters who were eligible to cast ballots in the November general elections in LA, MS, NJ, PA, VA. We also targeted voters in KY, but vote history was not available in May 2020, which is when the analysis was conducted.
  • Targeted population
    • People of color –OR– Young (18-30)
    • Good quality cell phone from Catalist
    • Households with 2-4 targets (unique address) 
  • Message tested Vote Tripling messaging vs Friends & Family messaging vs Social Pressure messaging.
    • Also tested using "everyone you live with" language for Vote Tripling and Friends and Family vs specifically asking recipients to "remind 3 friends" to vote.

Key takeaways

  • Cold SMS GOTV was very effective in 2019
  • Encouraging dispersion boosts treatment impact.
  • Suggestive evidence favored Friends & Family over VoteTripling, although the differences were not statistically significant. There might be a higher direct treatment effect from Friends & Family. 
  • The "everyone you live with" messaging did not increase dispersion effects, especially with regard to the Vote Tripling group.
  • No evidence of differences in direct or dispersion effects across race and ethnicity, age groups, party affiliation, or partisanship score.
  • Suggested evidence that men might be more susceptible to the Vote Tripling and Friends & Family messaging than women are, although the differences were not statistically significant.

Section 1. Experimental design

  • Set out to test messages against each other, both on the message recipient, and the people in their households.
  • The effect on the message recipient is the “Direct Effect.”  
  • The effect on the recipients' household members is the “Dispersion Effect”

Treatment

Sample

Vote Tripling

112,207 households 

Friends & Family

66,102 households 

Social Pressure

66,106 households 

Control group

66,104 households

Vote Tripling messages

This is based on the best-practices established in the past.

  • Message 1: Election Day in{{city}} is on {{election-date}}. Can we count on you to remind 3 friends to vote?
  • Message 2:
    • Tomorrow {{election-date}} is Election Day. Will you remind {{friends-names}} to vote. OR
    • Tomorrow {{election-date}} is Election Day. Can we count on you to remind 3 friends to vote?

Friends & Family messages

Based on past successful treatments, specifically 2018 NCO cold SMS experiments

  • Message 1: Make sure your friends and family vote! Election Day in {{city}} is on {{election-date}}. Make sure they vote in this important election.
  • Message 2: Tomorrow {{election-date}} is Election Day. If you're registered to vote at {{street}} your polling places is at {{location}}. If you've moved, or were never registered to vote there, reply with LOOKUP.

Social Pressure messages

Based on past successful treatments, specifically 2018 NCO cold SMS experiments.

  • Message 1: Thank you for being a registered voter in {{city}}. While your ballot is secret, whether or not you vote is public record. After the election we will survey you about your experiences at the polls. Election Day is on {{election-date}}
  • Message 2. Tomorrow {{election-date}} is Election Day. If you're registered to vote at {{street}} your polling places is at {{location}}. If you've moved, or were never registered to vote there, reply with LOOKUP.

Section 2. Overall message testing results

Direct turnout in control group: 28.16%.
Dispersion turnout in control group: 28.53%

Section 3. Message variation testing results

Overview

  • We altered the VoteTripling and Friends & Family messages to ”everyone you live with.” Our intent was to increase the measurable dispersion to household members.
  • For the test: 50% of received the standard message and 50% received the "live with" message

Message language

Standard:

  • Friends and Family: Make sure your friends and family vote!  Election Day in {{city}} is on {{election-date}}. Make sure they vote in this important election.
  • Vote Tripling: Election Day in{{city}} is on {{election-date}}. Can we count on you to remind 3 friends to vote?

"Live with" messaging:

  • Friends and Family: Make sure the people you live with vote! Election Day in {{city}} is on {{election-date}}. Make sure they vote in this important election.
  • Vote Tripling: Election Day in{{city}} is on {{election-date}}. Can we count on you to make sure the people you live with vote?

Results

  • Direct effects: no statistically significant difference across message versions on direct effects
  • Dispersion effects: no statistically significant difference across message versions on dispersion effects
  • Combined effects (direct + dispersion)  
    • Soliciting dispersion appears beneficial for Vote Tripling and Friends & Family messaging 
    • Differences between the messages did not reach statistical significance at the 95% confidence interval
    • But the “live with” language seems like a bad idea, especially for VoteTripling messages

Section 4. Differences across the states  

  • Mississippi seems an outlier among states, but the difference was not statistically significant 
  • No statistically significant differences in direct effects
  • No statistically significant differences in dispersion effects

Section 5. Differences across past voting behavior

5.1 Direct effects across past voting behavior 

    • Largest effect among middle propensity voters in terms of direct effects
    • Differences in Vote Tripling message effect were not statistically significant 
    • Differences in Friends&Family and Social Pressure messages were statistically significant 

5.2 Dispersion effects across past voting behavior 

  • Looking closer at dispersion effects by the recipients’ past voting behavior: N0 statistically significant differences in the messages in terms of dispersion effects
  • Looking closer at dispersion effects by the housemates' past voting behavior: The results are not statistically significant, but they do mirror the patterns of direct effect

Section 6: Differences across gender

  • Direct effects: Men may be more treatment responsive to Friends & Family and Social Pressure messaging, but the difference is not statistically significant 
  • Dispersion effects: No statistically significant differences

Section 7. Differences across across race and ethnicity

No evidence of differences in direct or dispersion effects across race and ethnicity 

Section 8. Differences across across age groups

No evidence of differences in direct or dispersion effects across age groups 

Section 9. Differences across party affiliation

  • No evidence of differences in direct or dispersion effects party affiliation

Section 10. Differences across partisanship score

  • No evidence of differences in direct or dispersion effects by partisanship score

SECTION 11. LOOKING CLOSER AT HOUSEHOLDS

In this section we examined impacts by household gender composition, household generational composition, household past voting composition, and household partisanship.

11.1 Differences by household generational composition 

  • Direct effects: No significant differences
  • Dispersion effects:  may be larger in same generation households, but the differences were not statistically significant 

11.2 Differences by household gender composition

  • Direct effects in single gender households may be slightly larger, but the difference is not statistically significant 
  • Dispersion effects: No statistically significant differences in message tests

11.3 Differences by household past voting composition

  • No significant differences in direct effects
  • No significant differences in dispersion effects

11.4 Differences in household partisan composition

  • No significant differences in direct effects
  • No significant differences in dispersion effects

Read more