Distributed Computing, The concept of dividing computing tasks over many independent computers, usually located in different geographical areas and coordinating to act like a single system.
Unlike centralized data centers, in which one cluster handles all processing, distributed computing has no single point of failure, no single point of authority, and no single point of resource use by design, they are federations, not empires.
Distributed computing harnesses multiple interconnected computers to solve complex problems, offering
- Massive scalability,
- Cost-efficiency, and
- High fault tolerance
The definition of ‘distributed computing’ includes a range of distributed systems: a cluster of servers in different cities that replicates a database; peer-to-peer networks, where every participant acts as a client and a server
CDN (Content Delivery Network), that pushes data to the closest node to the end user; and massive parallel scientific computing grids that use all idle computers in the world. The only thing they have in common is that no computer in a distributed system is essential.
A distributed system is one where your computer may cease to work (be unusable) due to the failure of a computer somewhere that you had never heard of at the time it failed. Your job is to make that irrelevant.
Importance
Distribution matters for three reasons: Each reason addresses a limitation of centralized architecture.
- Resiliency,
- Scaling and
- Latency
Resiliency
Resiliency is the most straightforward of these. In the case of a distributed system, if one of the nodes goes down, other nodes will pick up that load. Therefore, there is not one catastrophic failure (a single point of failure) that will collapse the entire server; the system will gracefully degrade, rather than fail altogether.
This is not just a theory; examples such as BitTorrent, blockchain technology and the DNS continue to work even though some or many of their members may suddenly become inactive, due to the fact that these systems were originally designed with regard to the possibility of failure, rather than assuming failure was an exception.
The economic rationale for scaling indicates that centralized systems are vertically scalable (i.e. larger/more costly hardware systems), whereas distributed systems are horizontally scalable (new inexpensive hardware can be added to a distributed system). In its original GFS paper (2003), Google demonstrated that one thousand regular/low-cost servers would likely out perform 10 or 20 very-high-quality servers, so long as the software was written to anticipate and recover from regular/constant server failure.
Latency
Latency refers to the time it takes for data to travel between two different physical locations. Since light travels at a finite speed, a user in lahore who wishes to query a server located in karachi will always experience higher latency than someone who wishes to query a server located in Singapore.
This is because additional latency is introduced from the physical distance between lahore and karachi compared to the much shorter distance between lahore and Singapore. To mitigate this issue, compute resources are distributed across the globe. Edge computing, CDN edge nodes, and regional compute clusters position compute resources closer to the user and reduce round-trip latency from hundreds of milliseconds to single-digit milliseconds.
Technologies that Enable Distributed Computing
The field of distributed computing is built upon more than 20 years of theoretical and physical ingenuity. The CAP theorem, proposed by Eric Brewer in 2000, established a fundamental constraint on all distributed computing. The theorem states that no distributed system can guarantee all three of consistency, availability, and partition tolerance simultaneously. Thus, each distributed system is designed around a particular trade-off among the three properties; the ability of an engineer to understand when and why to sacrifice one of the three properties defines the elegance of the design vs. the fragility of the design.
Google’s MapReduce, published in 2004, provided a solution for processing data using a distributed architecture. With the ability to separate the computation into independent map and reduce operations, it allowed a non-expensive way to process petabytes of data, instead of needing prohibitively expensive centralized hardware.
Consensus algorithms such as Paxos, Raft, and their derivatives are also an important advancement in the field of Distributed Computing, as they each address one of the most difficult problems associated with distributed computing. Specifically, when there are several independent nodes on a network, how does each node reach agreement on a single truth, when it is possible that messages may be lost or delayed and/or that they are corrupted.
Edge Computing and the Next Frontier
The current migration of computation to edge of the network represents the most significant advancement in distributed computing to date. The number of devices connected to the Internet continues to grow rapidly e.g autonomous vehicles, industrial sensors, smart cities and AR glasses. The volume of digital data at the physical edge of the world is now far larger than can or should be transported to centralized data centres for processing.
The primary solution to this challenge is to deploy small, ruggedized computing nodes at cell towers, within factories, within vehicles, and at traffic intersections. It is impossible for an autonomous vehicle to wait for a round-trip of 200 milliseconds to the cloud-based data centre when it has to make a decision to avoid a collision, or for a machine on a manufacturing line to be able to detect that it is about to fail in microseconds instead of seconds. The physical aspects of making machine-speed decisions cannot rely on off-site processing.
The cloud is the first act of computing off of the premises (i.e., moving computing from the premises of a company and onto the cloud). Edge Computing represents the second act of computing by moving computing back to the edge of the network, close to where it has to make decisions.
Democratic Infrastructure
There is an inherent political aspect to distributed computing that does not exist in centralized architectures. Because there is not a single entity that controls the infrastructure, there is also not a single entity that can censor, monitor or shut down the infrastructure unilaterally.
There is a practical side to distributed infrastructure that supports the theory that it allows for journalists to publish censored material, activists to operate under oppressive regimes, and communities to develop their own systems of communication without relying on a centralized authority or private enterprise. The Tor network, the InterPlanetary File System, and projects like Meshtastic mesh network show a consistent tradition of using distribution at the technical level to achieve resilience on the political level.
The same is true of economic resilience. The use of distributed cloud architectures (multi-cloud and hybrid clouds) has become common practice for organizations that cannot depend on a single provider. When AWS’s us-east-1 region is down, an organization with workloads in Azure and GCP are still operational. Distribution is a strategy for continuity, not just a best practice for engineers.
The Hard Problems
However, distributed systems are not without their challenges. The many challenges of distributed systems are well-known among engineers: node time synchronization, debugging failures that are non-deterministic and difficult to reproduce, ensuring that data is consistent when writes are being made simultaneously to multiple locations and managing the operational complexity of having thousands of independent moving parts.
Distributing security across systems creates many different kinds of problems. A centralized system has a clear boundary to protect, but there are no boundaries or edges to protect in a distributed system; all nodes at once are both “inside” the system and “outside” of it. There are many ways that two nodes can communicate with each other, and providing a way to ensure that communications will not be compromised by an outside influence is extremely difficult. Creating a system that can tolerate these conditions, known as Byzantine Fault Tolerance, continues to be the biggest technical challenge for the area of distributed systems.
Not only is the technical side of the calculus creating distributed systems difficult, but also the environmental side of this calculus is difficult, and decentralization leads to lower efficiencies when compared with hyperscale centralized systems. Most smaller nodes do not achieve as high as a PUE ratio as do purpose-built data centers with optimized cooling systems, so the increased efficiency from the resiliency benefits obtained from having distributed systems comes with a greater cost in energy as the level of distribution increases.
Conclusion
In conclusion, the philosophy behind distributed computing facilities is not simply the design of technical architecture; it is also an acknowledgment that systems created from multiple autonomous components that were designed to anticipate and absorb faults may ultimately outperform and survive longer than systems created around an ideal model. This approach is contrary to conventional thinking since the dominant model states that a centralized approach creates the best overall system.
There is an abundance of evidence that supports this approach, for example, the Internet—the largest, most reliable, and most scalable information system ever developed was constructed as a distributed entity. The distributed application environment of the Internet supports this by being the longest-running applications operating on the distributed platform. The distributed paradigm of computing will continue to develop and grow, and will establish itself as the primary operating environment for machines to think, communicate and act because the speed, size, and complexity of our world cannot be contained in a central location.
