Friday 12th August 2022

Building Tech at Presidential Scale

[ad_1]

Key Takeaways

  • Technology played a central role in the 2020 US Presidential election cycle.
  • The Biden for President 2020 campaign build a robust microservices architecture.
  • Speed of delivery, reliability, and security were critical requirements.
  • CI/CD best practices were employed to deliver changes with unprecedented velocity.
  • Data-movement automation and artificial intelligence were among some of the architectures built by the campaign.

I was the chief technical officer for US President Joe Biden’s campaign during the 2020 election. At Qcon Plus in November 2020, I spoke about some of the elaborate architectures our tech team built and the specific tools we built to solve a variety of problems. This article is a distillation of that talk.

As CTO with Biden for President 2020, I led the technology organization within the campaign. I was responsible for the technology operations as a whole and I had a brilliant team that built the best tech stack in presidential politics.

We covered everything from software engineering to IT operations to cybersecurity and all pieces in between.

I joined the campaign as a distinguished engineer at Target, where I focused on infrastructure and operations with an emphasis on building high-scale, reliable distributed systems. I had previously worked on the Hillary for America 2016 technology platform by way of The Groundwork, which was a campaign adjacent technology company.

Campaign structure

The intricacies of a political campaign’s moving parts influence all of the specific choices when considering tech in that environment.

Figure 1 lays out the different departments within the campaign and gives you a sense of the varying degrees of responsibility across the organization.

Figure 1: The organizational structure of a political campaign

Each of the many teams across the campaign had its own specific focus. Although campaign tech gets a lot of attention in the press, tech is not the most important thing in politics. The goal for everyone is to reach voters, drive the candidate’s message, and connect as many people as possible to the democratic process.

Role of tech on a campaign

Tech on a campaign does quite literally whatever needs to be done. Nearly every vertical of the campaign needs tech in one form or another. In practical terms, my team was responsible for building and managing our cloud footprint, all IT operations, vendor onboarding, and so forth.

Our approach to building technology during the election cycle is simply to build the glue that ties vendors and systems together. This is not the right environment for delving into full-blown product development.

When a vendor or open-source tool we needed didn’t exist or, frankly, when we didn’t have the money to buy something, we would build tools and solutions.
A huge portion of the work that needs to be done by a campaign tech team is simply getting data from point A to point B. Strictly speaking, this involves creating a lot of S3 buckets.

In addition, we ended up developing a wide range of technology over the year and a half of the campaign cycle. We built dozens of websites, some big and some small. We built a mobile app for our field organizing efforts. We built Chrome extensions to help simplify the workloads for dozens of campaign team members. Though unglamorous, we developed Word and Excel macros to drive better efficiency. On a presidential campaign, time is everything, and anything that we could do to simplify a process to save a minute here or there was well worth the investment.

Much of what we did boils down to automating tasks to reduce the load on the team in any way that we could. It’s desirable at times to distill campaign tech down to one specific focus: data, digital, IT, or cybersecurity. In reality, the technology of the campaign is all of those things and more, and is a critical component of everything the campaign does.

What we did

We were a small and scrappy team of highly motivated technologists. We’d take on any request and do our best where we could. Over the course of the campaign, we built and delivered over 100 production services. We built more than 50 Lambda functions that delivered a variety of functionality. We built a best-in-class relational organizing mobile app for the primary cycle, the Team Joe app, which gave the campaign the leverage to connect thousands of eager voters and volunteers directly with the people they knew.

We had more than 10,000 deployments in the short time we were in operation, all with zero downtime and a focus on stability and reliability.

We implemented a bespoke research platform with robust automation that built on cloud-based machine learning, which saved the campaign tens of thousands of hours of human work, and gave us incredible depth of insight. We built a number of services on top of powerful machine-learning infrastructure.

The campaign early on made a commitment to hold ourselves to a higher standard, and to make sure that we would not accept donations from organizations that harm our planet or from individuals who may have ulterior motives. To hold us to that promise, we built an automation framework that vetted our donors in near real time for FEC compliance and campaign commitments.

When we won the primary, we wanted new, fresh branding on the website for the general election so we designed and delivered a brand-new web experience for joebiden.com.

A huge aspect of every campaign is directly reaching voters to let them know of an event in their area or when it’s time to get out to vote. For that, we built an SMS outreach platform that scaled nationwide and saved us millions of dollars in operational expense along the way. Beyond that, we operationalized IT for campaign headquarters, state headquarters, and ultimately facilitated a rapid transition to being a fully remote organization at the onset of quarantine.

Through all of this, we made sure we had world-class cybersecurity. That was a core focus for everything we did.

Ultimately, though, no one job on the campaign is single responsibility, which means everyone needs to help wherever and whenever help is needed, whether that’s calling voters to ask them to go vote, sending text messages, or collecting signatures to get on the ballot. We did it all.

Infrastructure and platform

We relied on a fully cloud-based infrastructure for everything we did. It was paramount that we didn’t spend precious hours reinventing any wheels for our core tech foundations. We used Pantheon for hosting the main website, joebiden.com. For non-website workloads, APIs, and services, we built entirely on top of AWS. In addition, we had a small Kubernetes deployment within AWS that helped us deliver faster simple workloads, mostly for cron jobs.

We still needed to deliver with massive scale and velocity overall. As a small team, it was critical that our build-and-deploy pipeline was reproducible. For continuous integration, we used Travis CI. For continuous delivery, we used Spinnaker.

Once we deployed services and they were up and running in the cloud, they all needed a core set of capabilities like being able to find other services and securely accessing config and secrets. For those, we used HashiCorp’s Consul and Vault. This helped us build fully immutable and differentiated environments between development and production, with very few handcrafted servers along the way — not zero, but very few.

A huge part of the technology footprint was dedicated to the work being done by the analytics team. To ensure they had best-in-class access to tools and services, the analytics data was built on top of AWS Redshift. This afforded a highly scalable environment with granular control over resource utilization.

We used PostgreSQL via RDS as the back-end datastore for all of the services that we built and deployed.

From an operational perspective, we desired a centralized view into the logging activity for every application so that we could quickly troubleshoot and diagnose any problem to achieve the quickest possible recovery. For that, we deployed an ELK Stack and backed it with AWS Elasticsearch. The logs are very important to applications. Metrics were the primary insight into the operational state of our services and…

[ad_2]

Read More:Building Tech at Presidential Scale