absentee

2018 Absentee Recruitment Experiment Using SMS and Email Contact

Debra Cleaver

Oct 15, 2019 — 12 min read

Program conception, design, and fundraising by Debra Cleaver. Program implementation by a national civic organization (NCO). Program evaluation by Pantheon Analytics.

Executive Summary

In October of 2018, NCO ran a randomized control trial to determine whether digital messaging (email and SMS) pushing voters to sign up for absentee ballots could increase overall turnout. The experiment was conducted in eight states that allowed no-excuse absentee voting, with a total audience size (post voter file match) of 212,000 voters.

The overall results of this trial were inconclusive. In the full sample, no statistically significant effects were seen on turnout. Among states without permanent absentee voting, however, a significant half-point increase in absentee voting was observed in the test group. In Florida (the only state in the sample allowing online absentee signup), there was an even larger increase in absentee voting. In non-permanent-absentee states, low-propensity voters saw a significant turnout effect of seven tenths of a percentage point. Among high-propensity voters, the treatment had a small but significant negative effect on turnout (just under half a point), while mid-propensity voters displayed marginally significant method-switching behavior and non-significantly higher turnout.

Overview

Most studies of vote-by-mail jurisdictions, as well as some studies of absentee signup drives, have indicated that when voters (especially low-propensity voters) are induced to sign up for vote-by-mail or absentee balloting, they become more likely to vote. There are some potential explanations for this. Receiving a ballot in the mail makes voting much more convenient, since it gives a broader window of time in which a voter can make their decisions and cast their ballot. It also helps voters avoid pitfalls such as long lines, hostile poll workers, and so on.

Other research, however, has not found this positive turnout effect. Rather, these competing analyses have found that inducing voters to sign up for absentee ballots merely changes their method of voting. Voters who would have voted on Election Day or at an early polling site instead choose to vote by mail with their new absentee voter status. These studies have tended to focus on all types of voters, not just low-propensity voters.

NCO’s target groups of voters (young voters, first-time voters, voters of color, etc) generally fall into the low-propensity voter category. Therefore, in the 2018 election cycle, NCO partnered with Pantheon Analytics to study the effects of nudging the targeted voters to sign up for absentee ballots. We hypothesized that contacting these targets by email and text and providing easy links for absentee signup forms could potentially increase their overall voter turnout.

Research Plan

In 2018, Arizona, California, Florida, Georgia, Idaho, Kansas, North Carolina, and Nebraska, were all are no-excuse absentee states. In those states, NCO had a list of over 260,000 contacts to which it could send both emails and SMS messages in the run-up to the 2018 election. All of these contacts would be receiving NCO’s typical election-year slate of GOTV messaging and other civic communication.

For the experiment, however, a randomized half of these contacts were given additional messaging about signing up to become absentee voters. These messages were delivered in the form of two SMS messages and an email. The other randomized half of these contacts served as a control group, and was not given the additional absentee messaging. The email addresses and mobile numbers for the experimental audience had been obtained by NCO through previous instances when audience members had used NCO’s tools (such as the registration verification tool).

After the election, Pantheon Analytics matched the control and test groups to the updated voter files for each state and produced the analysis presented in this report.

Randomization Procedure

The subscribers in the experimental universe were randomized using a stratified randomization procedure. Stratification is a useful tool for ensuring an even distribution, though it depends on having enough good variables on which to stratify. Since the experiment was using a list that could not be matched to the voter file prior to implementation, the number of variables was somewhat limited , but Pantheon identified two robustly-populated variables on which to stratify. First, subscribers were placed into buckets based on join-date (i.e. the date when the subscriber entered NCO’s system) and geography – in this case, Congressional district. Then, each bucket was randomly divided into two groups: test and control. To ensure that the stratified randomization produced two equivalent groups, balance checks were performed using other variables available on the voter file (in particular, area code and the subscriber’s age was known).

Implementation Timeline

The first text message was delivered to the test audience on October 17, 2018. The email message was delivered in quick succession on October 18th, 2018. Finally, a follow-up text message was delivered on October 24th, 2018. All of these dates were prior to any absentee ballot request deadlines, though in some cases the October 24th text message came close to the deadline. In Arizona and Nebraska, for instance, the deadline was October 26th.

Voter File Match

When the experimental universe was matched to the voter file aer all eight states had published their post-2018 updates. Close to 81% of records in both the test and control groups matched to the voter file.

It is not surprising that some people in NCO’s target universe did not find a match on the voter file. The universe was not specifically selected from the voter file in the first place, but rather from NCO’s contact lists within Mobile Commons. Most of the people in NCO’s Mobile Commons contact list got there by using one of NCO’s portals for registering to vote, verifying their registration status, or signing up for absentee status in previous elections. Therefore, we would expect most – but certainly not all – of the universe to be registered voters. Additionally, there are always a small number of records that do not successfully match to the voter file due to data entry issues, out-of-state moves, and so on.

The analysis below looks only at the test and control group members who matched to the voter file. This audience represents around 212,000 voters (106,000 in each group).

Results: Multi-State Analysis

Analysis of the full matched universe across the eight states shows no significant effect of the intervention. While absentee voting was indeed higher by three tenths of a percentage point in the test group versus the control group, this difference was not statistically significant. Overall turnout was also not significantly higher in the test group, and no other method of voting differed significantly between the two groups.

It’s possible that the treatment (a few emails and text messages) was not powerful enough to prompt a significant change in behavior in the test audience. It is also possible that this audience was not as low-propensity as we had assumed; indeed, turnout for this midterm election was higher than 70%. Finally, it is important to note that some of the states in this sample already had fairly high rates of absentee ballot usage (in particular, Arizona and California). It is more difficult to detect treatment effects when a high proportion of the target population has already taken the desired action.

The high pre-existing absentee rates in Arizona and California aren’t due to chance. Both states maintain permanent absentee lists, and five counties in California had switched to 100% mailed ballot voting in 2018 (with several more to follow suit in 2020). There is a case to be made for excluding these states from the analysis. The chart below shows the results for the six remaining states, which did not maintain permanent absentee lists.

In these six states, absentee voting was elevated by a little over half a percentage point in the test group, a difference that was statistically significant. However, no other differences were significant. The negative results for early and in-person voting, while not significant, may indicate that some of the targeted voters merely switched their method of voting.

We also looked at the experimental results on a state-by-state basis. These subgroup analyses are shown below and in the Appendix.

Results: State-by-State Analysis

In the various states, the differences in turnout and vote method were largely insignificant between the control and test groups, though certain differences were close to marginal significance.

The only significant finding was that, in Florida, there was a statistically significant increase in absentee voting in the test group versus the control group. This difference -- around a percentage point -- had a p-value of .014. Therefore, we can conclude that the email+SMS treatment likely did have a positive effect on absentee signup rates and subsequent absentee voting. Yet, because there was also a negative (though non-significant) decrease in early voting in Florida, the overall turnout rate in Florida was not significantly higher in the test group versus the control.

Florida is one of the few states in the country (and the only state in this study) with an entirely web-based absentee ballot signup procedure. It is possible that NCO was better able to affect absentee signups in Florida because the contact methods – emails and text messages – gave users a more seamless transition to into the signup procedure. NCO may wish to pursue these contact methods further in other online-signup states, which currently include Maryland and Minnesota in addition to Florida. (It is also possible that other states will follow suit in the future and make online absentee signup an option.)

As mentioned previously, Arizona and California are states that already had relatively high absentee or mail rates in the broader population. Thus, perhaps it is not surprising to see no significant effects there. However, the lack of significant effects in Georgia, Idaho, Kansas, and North Carolina may indicate that the treatment was too weak, or the sample sizes too small. Finally, it is important to note that while turnout in Nebraska’s treatment group was more than a point lower than in the control group, this difference was not significant.

The full analyses of each state are below in the Appendix.

Results: Vote Propensity Analysis

One of the theories behind this experiment was that low-propensity voters would benefit more from absentee signup nudges than high-propensity voters. Since NCO’s audience tends to be younger and more diverse, it was a reasonable belief that the target audience would be a relatively low-propensity group. Aer 2018, it is clear that NCO’s audience has a higher vote propensity than anticipated, with over 70% of the matched audience voting in the midterms.

Still, there was heterogeneity in the vote propensity scores of the experimental audience. Therefore, it is prudent to perform a subgroup analysis to look at whether there were varying effects of the experimental treatment for voters with different pre-existing propensities.

The following analysis looks at three groups: those with turnout scores above 50, those with turnout scores below 50, and those who had not been assigned a turnout score. For the latter group, we speculate that average vote propensity would be fairly low; voters without turnout scores tend to be new voters, recent movers, etc.

Overall, there are few differences between the control and test groups. However, the treatment did have a marginally significant positive effect on turnout within the low-propensity group. Turnout effects were also positive in the unscored group, but the difference was not significant. As with the full analysis, however, it is prudent to zero in on the six states without permanent absentee rolls (i.e. removing Arizona and California). The chart below shows this analysis.

Within these six states, turnout was once again marginally higher among low-propensity voters, specifically due to absentee voting. High propensity voters exhibited method-switching behavior, with a marginally significant decrease in early voting (and a not-quite-significant increase in absentee voting). Among unscored voters, turnout using unknown voting methods was significantly higher in the test group. It is possible that the lack of information about their voting method is connected to the reasons they were unscored on the voter file.

Interestingly, turnout was quite high among low-propensity voters, relative to expectations. We would not normally expect turnout above 50% in a group with average scores well below 50. Self-selection may be at work. NCO users may, on paper, look like low-propensity voters, but their initiative in using NCO’s tools reveals them to be more engaged. The analysis below divides the audience into three groups instead of two, for the purpose of singling out the extremities.

With these new delineations, we can see that the positive turnout effect is largely confined to voters at the very lowest end of the turnout spectrum (scores under 20). Oddly, this increase does not appear to be due primarily to absentee voting itself, but rather small increases in turnout rates using all methods. Perhaps the digital contact had more of a general GOTV effect on these voters, rather than spurring a specific interest in absentee voting. Mid-propensity voters, on the other hand, showed a marginally significant increase in absentee voting specifically.

There was a negative effect on turnout, however, among high-propensity voters. It is difficult to explain this negative effect. Perhaps among these very likely voters, a switch to absentee voting -- especially so soon before the election -- creates more confusion than it solves. People with a long track record of voting may have developed their own particular voting rituals around their own preferred voting methods; disrupting these rituals appears to be ill-advised.

Once again, it is worth noting that the group with turnout scores under 20 actually voted at rates closer to 50%. Certainly, some of this may be due to the unprecedented nature of the 2018 election; overall turnout was higher than anyone expected. Still, it is also likely that self-selection explains some of the discrepancy. NCO should keep this in mind when contacting its audience in the future. The type of voter who chooses to interact with NCO’s tools may be a higher-propensity voter than their raw turnout score would suggest.

Spotlight: Florida Vote Propensity Analysis

Since Florida had a large sample size and produced significant results in aggregate, it is worth disaggregating Florida’s audience by turnout score. The analysis below uses the three-tier turnout score delineations, plus the unscored category.

High-propensity voters in Florida showed a lesser (and non-significant) negative turnout effect than in the six-state sample. However, their method-switching behavior was marginally significant. Mid-propensity voters, meanwhile, had a marginally significant increase in absentee voting without a commensurate decrease in other voting methods. Low-propensity voters had a statistically significant positive turnout effect, though not due to absentee voting specifically.

Conclusions

This experiment used a very light treatment (one email and two text messages). It is possible that with a more intensive treatment, greater turnout effects could be observed. Still, there are several useful takeaways from the analysis of this experiment.

First, the analyses of Florida individually and the non-permanent-absentee states collectively show that it is possible to significantly affect the usage of absentee ballots versus other polling methods. Subgroup analysis of Florida reveals that method-switching effects were concentrated within high-propensity voters, while overall turnout effects were positive and significant for low-propensity voters. Indeed, for low-propensity voters across the six-state sample there was a marginally significant turnout effect from the treatment.

Since increasing turnout is a primary goal of NCO, future efforts at using absentee signup nudges should focus more intently on lower-propensity voters. One way to do this, either in another experiment or in a non-experimental outreach program, would be to match the audience to the voter file first, and then target only those voters with lower vote propensity scores (and those lacking scores).

Pantheon recommends further experimentation on this topic. It is quite promising that such a light treatment was able to achieve significant effects with particular audiences. A future experiment could layer on additional emails and text messages, and narrow the audience to the lower-propensity voters most likely to benefit. Additionally, it would be useful to study whether contacting voters far in advance of the election (rather than in the couple of weeks before the absentee deadline) could heighten the turnout effects.