Dylan Moore is a recent graduate of the University of Southern California who majored in Computer Science & Business Administration. He is obsessed with mobile apps and the idea of making people’s lives better by building great products. He enjoys longboarding, live concerts, and skiing in his free time.
More than 30,000 years after the first cave drawing, it just so happens that humans are still communicating with one another by simply leaving our thoughts on a wall. This is quite a surprise considering how drastic the methods of communication differ between our ancestor’s first few paintings inside a cave and the thousands of everyday posts on friends’ Facebook walls. Communication has evolved from what was once a simple expression of ideas to a complex and technologically involved process. From one medium of communication to the next, these infrastructures are what truly define the way we are able to interact. From the origin of speech to the introduction of the radio to the invention of online social networks, there has been rising emphasis on the flexibility of communication infrastructure. The evolution of communication demands adaptation to new technologies that will serve for quicker exchanges of information. “We’re trying to take in as much data in a day as people used to take in in a lifetime,” states Juan Enriquez, exacerbating the increase in communication . Without the development of flexible infrastructures alongside rapid technological advancements, the grounds we build upon will no longer support the mediums we heavily rely on. The rising popularity of and reliance on such social networking sites as Facebook and Twitter have put additional pressure on developers to create efficient, robust, and flexible infrastructures. To keep up with advancements in technology and produce reliable software, Facebook employs daily “release engineering,” a method for constantly improving its site unobtrusively over the air .
Facebook’s Release Engineering
Testing the software in development is an important step in ensuring the quality and robustness of daily updates. With daily minor releases and therefore less than a workday’s time to test and distribute each new version of Facebook, testing engineers were forced to develop creative solutions. Even the smallest of bugs can affect thousands if not millions of users with the network rapidly approaching a billion people. One simple way testers take advantage of their vast human resources is by setting the current development build as the default Facebook page for its employees . This way, unfinished software is exposed to various testing environments, and employees become testers in their own way, inadvertently checking the newest update for potential bugs. With no quality assurance department, Facebook relies on its testing engineers to build a common testing infrastructure, write automated tests, and update these tests as the codebase matures . Putting the pressure on employees for daily releases forces all software departments to think like release engineers, while in most companies these tasks are disjoint and often uncoordinated . With quick releases becoming more common, a fast testing procedure will turn into an essential asset to a company’s success. Furthermore, technology companies will be challenged to uphold similar principles as users become more accustomed to reliable software that updates more and more frequently. In accordance with this, Facebook’s high quality testing framework allows it to update its site with great confidence and quick speed, a combination that is typically defined as an inverse relationship.
After developing and testing a new build of the Facebook website, the release engineers must distribute it to all users. For seamless distribution, Facebook takes advantage of BitTorrent to push the compiled binary code to its servers . BitTorrent is a rapid file-sharing client that allows the build to quickly propagate over all of Facebook’s servers using peer-to-peer networking. Facebook has optimized its own custom BitTorrent tracker by accessing the relative location of each server and exchanging data over the shortest paths possible . Similar to a wildfire where each blazing tree rapidly catches its neighbors on fire, each server simultaneously passes along slices of code to the closest servers in proximity until they all receive the whole binary file. The entire distribution, when successfully done, takes only 15 minutes , meaning that within 30 minutes of completing a build, hundred of thousands of users already see the minor changes implemented to the website. Major updates are distributed once a week, while minor improvements to the website are made daily. Because each user’s session is not tied to a particular server, “any given page request can be handled by any of the servers in Facebook’s infrastructure” , a “stateless and distributed” architecture that allows for very discrete, undisruptive updates and a consequently flexible infrastructure that can adapt to future change .
Importance of Flexible Infrastructure
Communication is vital for the transferring of information, but the longevity of that information depends upon the channel implemented and the infrastructure it is built upon. While speech is recognized as the most common form of communication between humans, it is probably the least enduring due to a lack of infrastructure. Oral stories, for example, are often modified as they are passed from person to person because no original record is available for reference. This results in a poor transfer of information, as the last receiver has no way to verify the accuracy of what he or she has been told. In contrast, modern telecommunication channels such as radio, television, telephone, and the Internet rely on a systematic arrangement of wires, nodes, receivers, and towers to operate. When all of these pieces work together, information travels reliably and efficiently, but as parts of these networks break or misbehave, data tends to leak, and communication can be lost altogether. Network reliability used to be easily maintained when changes were not as frequently implemented. Now, companies like Facebook have significantly increased the frequency of software updates, meaning their existing infrastructure must constantly change with it. This brings up the importance of a flexible infrastructure when it comes to developing widely used means of online communication such as Facebook, LinkedIn, Twitter, and YouTube. For these companies, having reliable, rapidly evolving infrastructure to support their users’ best interests is what separates them from their competitors and allows them to succeed.
Flexibility in Business
The development of new mediums of communication has been facilitated by the introduction of the Internet, but also capitalized by corporations. According to Avalaunch, “Facebook, Twitter, and YouTube are now considered cornerstones of most social-media strategies in larger companies,” and this includes product marketing, customer communication, and creation of an identifiable fan base . When billboards and TV advertisements fail to reach their target audience, companies turn to Facebook to engage and influence their precise market. The trend towards advertising online using social media has skyrocketed due to Facebook’s extensive reach and growing list of features, which cater to companies trying to reach specific audiences. This way, Facebook has assisted the growth of companies, but more significantly, it has begun to alter various aspects of marketing with every update. As a result, what worked for businesses in 2005, a year after Facebook’s launch, will not work today. Michael Scissons, the president of Syncapse, notes that “winning strategies will look very different in 2013 from previous years…now that many brands have acquired sizable ‘fan’ bases” . In essence, the ever-changing nature of Facebook’s internal infrastructure has forced businesses to follow suit and drastically modify their practices to stay relevant and successful.
Following the Leader
The practice of release engineering puts Facebook at the forefront of developing flexible infrastructures. While implementing an innovative infrastructure was never required, it is evident that their careful and ingenious development process has influenced many to adopt the same. Software companies resistant to embrace the importance of release engineering will eventually suffer, report Hyrum Wright and Dewayne Perry . Furthermore, lack of support to improve infrastructure can hinder an engineer’s ability to properly release new updates . The Next Generation Infrastructures Foundation upholds this belief by advocating “future proof” infrastructures. Since the future is never clear, this organization of “knowledge institutions, market players, and government organizations…more or less demand[s] infrastructures that change their functioning along the way” . In the past, telecommunication has relied on fixed infrastructures such as radio and cell towers, routers, and satellites. With Moore’s Law, the observation that chip technology doubles about every two years, and websites like Facebook, which update daily, a fixed infrastructure is no longer an efficient one. After only a single year without technological improvements, the quality and speed of an unchanging infrastructure will have decreased around 50%. Thus, in managing communication, scaling business strategies, or even developing new technologies, flexibility is key. As noted by the NGIFoundation and Facebook engineers, those who are quicker to adapt are more likely to reduce overhead and thrive. Moreover, companies failing to adopt similar strategies will inevitably be hurt by the future costs of implementing entirely new structures or processes. Facebook enhances its ammunition day by day in preparation for an unpredictable future, but now it’s up to other developing industries to build their stockpile. The only question is: will they be prepared to capitalize on the unpredictable future?
 AdAge, Scissons: “It’s Time to Refresh Your Facebook Marketing Strategy” in AdAge [Online Magazine], AdAge, Oct. 2012, Accessed: Nov. 2012, Available: http://adage.com/article/digitalnext/time-refresh-facebook-marketing-strategy/237651/
 H. Wright, “Release engineering practices and pitfalls” in Proceedings / International Conference on Software Engineering [Journal], IEEE, Jan. 2012, pg. 1283, Accessed Dec. 2012, Available: http://ieeexplore.ieee.org.libproxy.usc.edu/stamp/stamp.jsp?tp=&arnumber=6227099
 H. Zhao, “HipHop for PHP: Move Fast” in Developers Blog [Blog], Facebook, Feb. 2010, Accessed Nov. 2012, Available: https://developers.facebook.com/blog/post/2010/02/02/hiphop-for-php–move-fast/
 J. Enriques, “Will our kids be a different species?” in TED [Online Video], TED: Ideas Worth Spreading, Jun. 2012, Accessed: Oct. 2012, Available: http://www.ted.com/talks/juan_enriquez_will_our_kids_be_a_different_species.html
 J. de Haan, “Flexible Infrastructures for Uncertain Futures” in Futures [Report], Science Direct, Vol 43., Iss. 9, Nov. 2011, Pgs. 921-922, Accessed: Nov. 2012, Available: http://www.sciencedirect.com.libproxy.usc.edu/science/article/pii/S0016328711001352
 M. Siltala, “Social Media Statistics Of The Day” in Avalaunch [Online Infographic], Avalaunch Media, Dec. 2011, Accessed: Nov. 2012, Available: http://avalaunchmedia.com/infographics/social-media-statistics-of-the-day
 R. Paul, “Exclusive: a behind-the-scenes look at Facebook release engineering” in arstechnica [Website], arstechnica, Apr. 2012, Accessed Nov. 2012, Available: http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/
 S. Grimm, “What kind of automated testing does Facebook do?” in Quora [Blog], Quora, Jun. 2010, Accessed: Nov. 2012, Available: http://www.quora.com/Facebook-Engineering/What-kind-of-automated-testing-does-Facebook-do