big hopes, dashed

What should have been a 21st century digital political coming out party quickly turned into an unmitigated technological meltdown as Twitter’s servers failed under the load of an estimated 600,000 visitors. The plan was for Elon Musk to introduce the Florida governor in a Twitter Space virtual livestream, then have DeSantis take the stage and announce his bid. But it quickly devolved into a string of glitchy audio as Musk and his team wrestled with technology that refused to cooperate.

After about 20 minutes, Twitter pulled the plug on the livestream. When a new livestream was restarted, only a few minutes later, less than half of the original audience stuck around to hear him announce his candidacy.

Musk’s apology – “That was insane, sorry.” – did little to stem the rising tide of headlines that focused more on the collapsed infrastructure than on DeSantis’s message. The hashtag #DeSaster quickly gained momentum, while the @JoeBiden account jokingly tweeted “This link works” alongside a donation link for the President’s own campaign.

David Sacks, the venture capitalist who moderated the event, said the platform simply got weighed under by the crush of attendees.

“I think it crashed because when you multiply a half million people in a room by an account with over 100 million followers, which is Elon’s account, I think that creates just a scalability level that was unprecedented," said Sacks, who was also a product lead at PayPal when Musk owned it.

His comments suggest the company’s infrastructure wasn’t prepared in advance to handle the anticipated traffic. Which, for a web services company that lives and dies on its infrastructure, is a frankly surprising admission.

ongoing impact

 Within hours, Twitter’s chief engineer, Foad Dabiri, tweeted that he had had enough: 

"After almost four incredible years at Twitter, I decided to leave the nest yesterday."

In doing so, Dabiri adds to the mass exodus from a company whose staff has been decimated by 80% since Musk took over late last year.

That exodus lies at the root of the chaos that derailed the DeSantis event – and adds to a long list of outages and anomalies on the platform over that time.

NetBlocks, which monitors internet outages, reported Twitter, which experienced nine widespread outages throughout 2022, suffered at least four similar incidents in February alone.

According to a New York Times report based on interviews with current and former employees, Twitter has shuttered one of its three main data centers and has significantly cut staff and leaders responsible for back-end infrastructure like cloud storage and servers.

Meanwhile, reports of anomalies on the platform continue to pile up. On March 1, two days after the company pink-slipped another 200 staff, a global outage took down user feeds and the For You timeline and search services. Less than a week later, an API issue broke links, images, and a number of third party apps.

Last week, another bug – possibly due to a botched server migration – resulted in tweets that had been previously mass-deleted suddenly reappearing in users’ profiles. The company, which disbanded its media relations team and now responds to media inquiries with an auto-generated poop emoji, did not comment on this latest issue.

death by 1,000 cuts

On an individual basis, each of these outages is hardly cause for alarm. But when viewed within the broader context of a gutted IT competency  coupled with Musk’s highly public campaign to cut costs to the bone, they suggest an organization increasingly unable to keep the lights on. Thinking that Twitter can host high-profile events that attract the equivalent of a mid-sized city’s population is bordering on madness.

The truth is it can’t. And Twitter’s lesson is a stark one for organizations of any size and in any sector who think drastically thinning out back-end spending will help the bottom line.

Unfortunately for them, it never seems to work out as intended. All it does is defer necessary spending while increasing the potential for outages – which ultimately costs the organization, directly and indirectly, far more than any miniscule savings realized by slicing necessary pieces off the infrastructure budget.

It’s the same lesson learned by too many car owners who thought delaying routine maintenance would be an easy way to save some cash. It didn’t work with the old Chevy and it won’t work with Twitter. Or any other business for that matter.

the bottom line

Back in March, we wrote about this very issue. The DeSantis debacle now serves as a stark validation of those fears, and reinforces why this issue was universally important then, and it remains so today.

As staff members begin to plan summer vacations and their kids prepare to finish off the school year, now is the ideal time to shine the light on infrastructure spending and strategy. Because while any cost-cutting-related failures here may not necessarily be as globally witnessed as last week’s Twitter meltdown was, they will undoubtedly be painful to your bottom line.