2017 Alabama special election SMS and direct mail mobilization

Program team: Debra Cleaver (VoteAmerica), Page Gardner (VPC), Christopher Mann (Skidmore College). Experiment design and analysis by Michael Schwam-Baird, Jeff Ferguson, and Miya Woolfalk, all of the Analyst Institute.

Executive summary

In the lead up to the December 2017 US Senate special election in Alabama, two national civic organizations (NCO) collaborated with Analyst Institute to test the effectiveness of GOTV SMS messages and social pressure mail on turnout. The test targeted African American voters with an available cell number and evaluated the mobilizing effect of SMS alone, of mail alone, of combining GOTV SMS and GOTV mail, and of two different types of SMS messages. The informational SMS message gave recipients information about the logistics of voting; the rationalization SMS message prompted recipients to consider and respond with their reasons for voting.

Key findings

• The mail and SMS program as a whole generated approximately 2,050 net votes. Overall, these votes were generated at a reasonable $73 per vote, or 14 votes per $1,000. This finding shows how critical GOTV programs can be in tight election races.
• SMS messages on their own, across both messages, did not significantly affect turnout. Other “cold” SMS GOTV programs have been shown to boost ​turnout by up to 0.2 percentage points (pp).
• However, the combined informational SMS messages likely increased turnout on the margin and was the most cost-effective set of treatments. Combined across the informational SMS only and the SMS + mail conditions, informational SMS likely increased turnout by 0.3pp (p = 0.1) at a cost of just $40 per vote, or 25 votes per $1,000. This positive effect was driven by layering SMS on top of the social pressure mail. By contrast, the combined rationalization text messages did not increase turnout on the margin.
• Finally, the test results indicate that social pressure mail is an effective means of increasing turnout. On the margin, the combined mail conditions increased turnout by 1.1pp (p < 0.01). Social pressure mail generated votes at a cost of $54 per vote, or 18 votes per $1,000.

Key takeaways

• Social pressure mail, either alone or combined with informational SMS messages, is a cost effective way to increase turnout among African-American voters in a special election.
• If groups plan to conduct SMS messaging, informational messages about the process of voting appear to perform better than rationalization messages asking voters to consider and respond with their reasons for voting.

Background

In the lead up to the December 2017 US Senate special election in Alabama, Vote.org and Voter Participation Center (VPC) collaborated with Analyst Institute to test the effectiveness of GOTV SMS messages and social pressure mail on turnout. The test targeted African American voters with an available cell number and evaluated the mobilizing effect of SMS alone, of mail alone, of combining GOTV SMS and GOTV mail, and of two different types of SMS messages. The informational SMS message gave recipients information about the logistics of voting; the rationalization SMS message prompted recipients to consider and respond with their reasons for voting.

Research Questions

This test was designed to answer the following research questions:

• Did the GOTV SMS program increase turnout? How about the GOTV mail program?
• Did a combined SMS and mail program increase turnout over each mode on its own?
• Were GOTV SMS messages that encourage voters to verbalize their reasons for voting (i.e.,voter rationalization messages) more effective than messages that are more informational?

Experimental Design and Implementation

Experimental Universe

The experimental universe comprised 325,403 modeled African American voters in Alabama with unique cell phone numbers. Table 1 shows some of the universe’s characteristics.

Experimental Conditions

The experimental universe was randomized into 5 treatment conditions and a control group:

• Informational SMS: ​Voters in this group received a polling location SMS message and an “informational” SMS message. The informational messages gave recipients information about the logistics of voting (N = 46,471 voters).
• Informational SMS + Mail: ​Voters in this group received a polling location SMS message, an “informational” SMS message, and social pressure GOTV mail (N = 74,997 voters).
• Voter Rationalization SMS: ​Voters in this group received a polling location SMS message and “reasons for voting” SMS messages that prompted recipients to consider and respond with their reasons for voting (N = 46,531 voters).
• Voter Rationalization SMS + Mail: ​Voters in this group received a polling location SMS message, “reasons for voting” SMS messages, and social pressure GOTV mail (N = 75,008 voters).
• Social Pressure Mail Only: ​Voters in this group received social pressure GOTV mail. They were not sent SMS messages (N = 49,879 voters).
• No Contact Control: ​Voters in this group did not receive SMS messages or GOTV mail (N = 32,517 ​voters).

Randomization occurred at the household level, assigning targets with the same registration address to the same experimental condition. To ensure comparability of experimental groups, randomization was stratified by the following household characteristics: household gender composition, average household age, household marital status composition, average household vote history, average household phone confidence score, and household size. See the Technical Appendix for details on covariate balance.

Experimental Implementation

Individuals in the SMS conditions received text messages in two rounds in the final 10 days before the election. In the first round, SMS targets received either an informational text message or a rationalization text message depending on their treatment assignment. The informational message reminded voters about Election Day and that they needed a photo ID to vote. They were then referred to a state website for information about photo ID requirements. The rationalization message asked voters to respond to a survey question about the importance of voting and then, if a response was received, asked voters to text their reasons for voting. In the second round, all SMS recipients (in either condition) received an election reminder with information about their polling location. The full text of each SMS message is available in the Materials Appendix.Individuals in the mail conditions received either one or two pieces of social pressure mail in the final week before the election. A second mail piece was sent to individuals with vote propensity scores between 30-100 excluding people who voted in 4 out of the last 4 general elections. Individuals receiving two pieces of mail were evenly distributed across the mail treatment conditions, making up 48% of mail recipients in each mail condition. See the Materials Appendix for examples of the mail.

Contact Rates in Program

Implementation data from the SMS program allows us to evaluate the number of unique phone numbers texted and the number that responded. Table 2 shows that nearly all of the phone numbers in each SMS condition were able to be texted. Table 3 shows that targets in the rationalization SMS conditions, who were asked to respond to a question, were more likely to respond to the text messages than targets in the informational SMS conditions. We also measured the number of responses using negative words and phrases like “stop”, “go away”, “cancel”, and “unsubscribe” to see if certain conditions inspired more negative reactions. However, the evidence in Table 4 shows that equal percentages of unique phones texted responded negatively to the SMS messages across conditions.

Outcome Measurement

The outcome in this test was voter turnout in the 2017 Alabama special Senate election. This outcome was measured post-election using data from TargetSmart.

Results

Main Results

SMS in combination with mail and mail alone increased turnout by the largest amounts (Figure 1). In particular, mail combined with the informational SMS treatment increased turnout by 1.3pp (p < 0.01) over the control condition and mail alone increased turnout by 0.9pp (p < 0.01). Mail combined with the rationalization SMS treatment matched the effect of mail alone at 0.9 percentage points above control (p < 0.01). SMS considered on its own (when it was not combined with mail) did not increase voting by an amount statistically different from zero. The informational SMS treatment alone increased turnout by 0.1pp (p = 0.7) and the rationalization SMS messages alone decreased turnout by a statistically insignificant 0.2 percentage points (p = 0.5).

We also examined the marginal effect of each SMS message by comparing turnout in the treatment groups that included a given SMS message to the treatment groups with no SMS (Figure 2). For example, the difference in turnout between the treatment groups that received the informational SMS message (the informational message only and informational message + mail groups) and the groups that received no SMS messages (the mail only group and the control group) was 0.3pp (p = 0.1). This represents the combined marginal impact of including the informational SMS message across conditions (assuming that all of the lift that comes from combining SMS and mail is attributable to the SMS). The combined marginal effect of receiving the rationalization SMS message, calculated in the same way, was a statistically insignificant -0.1pp (p = 0.6). Finally, the combined marginal effect of receiving mail was an impressive 1.1pp (p < 0.01, Figure 3).

Net Votes and Votes per $1,000

Overall, the combined SMS and mail program generated approximately 2,050 net votes. These votes were generated at $73 per vote, or 14 votes per $1,000. Table 5 shows the net votes generated by each treatment the cost of those votes. The informational SMS message + mail condition generated votes most efficiently at 18 votes per $1,000.

Combining across conditions yields the overall marginal impact of informational SMS, relational SMS, and mail (Table 6). Combined across the informational SMS only and the informational SMS + mail conditions, the informational SMS message generated additional votes at a cost of $40 per vote, or 25 votes per $1,000. Using the same approach, the rationalization SMS message generated negative votes and a negative cost per vote. Pooling across the mail conditions, social pressure mail generated marginal votes at $54 per vote, or 18 votes per $1,000.

Variation in Treatment Effects

We examined whether treatment effects (pooled by mode) varied by gender, age, marital status, phone confidence score, and vote history. We found some evidence that treatment effects varied by gender (Figure 2). Although the differences in the effect sizes were not statistically significant, they were consistent across modes, providing suggestive evidence that men were more likely to vote than women as a result of the the treatments.

Discussion

Although the effects of individual treatments and treatment combinations varied, it is important to note that overall, the mail and SMS program in Alabama generated approximately 2,050 net votes. This meaningful increase in turnout benefits our democracy and is especially important in an election such as this one, which had a very close result. These votes were generated at a reasonable $73 per vote, or 14 votes per $1,000. These outcomes are impressive given the amount of attention that the Alabama special election received, especially in the final weeks before the election when the treatments were sent. This finding reinforces how critical GOTV programs are in tight election races, as well as how important it is to test GOTV tactics in order to optimize programs.

SMS messages on their own (across both messages) did not significantly affect turnout. This was surprising given the positive effects we have seen in other “cold” SMS GOTV ​programs​. However, the more-standard informational SMS message did have a combined marginal effect on turnout, with a magnitude (0.3pp) consistent with the effects of other cold SMS programs. Informational SMS provides an extra reminder about Election Day and gives straightforward information about what individuals need to do in order to vote, potentially easing the voting process for a significant number of people. Of the three treatments, informational SMS was the most cost-effective on the margin, generating 25 votes per $1,000.

By contrast, rationalization SMS messages did not boost turnout either on its own or when combined with mail. Rationalization messages first asked individuals to respond about their likelihood of voting. If recipients responded, they were then asked to think about their reasons for voting and respond again. Since only 10% of individuals responded in the rationalization message conditions, most recipients were never prompted by the second message to think about their reasons for voting. As a result, if this message is used again, it may be advisable to prompt recipients with an initial rationalization message or send a follow up message regardless of whether recipients respond to the first. It is also possible that in the context of “cold” SMS, individuals felt that questions about their voting intentions were simply too intrusive.

Finally, the test results indicate that social pressure mail is an effective means of increasing turnout across a number of groups. One or two pieces of mail increased turnout by 1.1 (p < 0.01) percentage points.This mail was cost effective, generating marginal votes at a cost of $54 per vote, or 18 votes per $1,000.