Modern API Design with gRPC: Efficient Solutions to Design Modern APIs with gRPC Using Golang for Scalable Distributed Systems (English Edition)

CHAPTER 1

API Evolution over Time

Introduction

In this chapter, we will delve into the fundamental necessity of APIs and trace their evolutionary journey over time. This exploration is crucial to comprehending the rationale behind our shift to alternative frameworks and messaging formats. We will delve into the prominent frameworks that have emerged over different times and also gaze ahead at the prospective horizons. By the chapter’s conclusion, we will have gained a comprehensive grasp of the dynamic evolution that APIs have undergone.

Structure

In this chapter, we will cover the following topics:

Introduction to API

Advent of APIs

Socket-based network communication

Introduction of RPC

SOAP

REST

Costs associated with traditional API frameworks

Serializing and Deserializing cost

Transmitting cost

Setting up a connection cost

Introduction to API

An API, or ‘Application Programming Interface,’ serves as a bridge between software components. I personally perceive the origin of APIs in the pursuit of ‘Code Reusability,’ akin to the motivation behind crafting functions. What sets APIs apart from functions is their inherent ‘Remote-ness,’ where APIs are stationed at network-accessible locations. Servers house these functions, with clients establishing connections to utilize them. This interplay requires a shared understanding between clients and servers, a predefined agreement that sidesteps ambiguity.

Several key reasons drove the development of APIs including,

Modularity and Reusability: APIs allow software developers to create modular and reusable components. Instead of reinventing the wheel, developers can leverage existing APIs to access well-defined functionalities, saving time and effort.

Abstraction and Encapsulation: APIs abstract the underlying complexity of software systems. They provide a simplified and standardized interface that shields developers from the internal implementation details of a component, making it easier to use and maintain.

Interoperability: APIs enable different software systems, often developed by different teams or organizations, to work together regardless of their underlying technologies, platforms, or programming languages.

Scalability and Specialization: APIs allow different teams to work on different parts of a system independently. For instance, a frontend team can work on the user interface while a backend team focuses on data processing, as long as they adhere to the defined API contracts.

Security and Access Control: APIs provide a controlled way for applications to access certain functionalities or data. Access can be managed and restricted based on authentication and authorization mechanisms defined in the API.

Innovation and Ecosystem Growth: APIs encourage innovation by enabling third-party developers to build applications or services that extend the capabilities of a platform. This fosters a rich ecosystem and can contribute to the overall success of a technology or product.

Updates and Maintenance: APIs allow software systems to evolve independently. If an internal implementation changes, as long as the API contract remains consistent, other systems using the API won’t be affected.

Remote Communication: APIs enable remote communication between different parts of a distributed system. This is particularly important in today’s interconnected world, where components might be running on different servers or even in different geographical locations.

Standardization: APIs provide a standardized way of interacting with certain functionalities or services. This standardization ensures consistency and predictability across different applications.

Ease of Development: Developers can build complex applications by leveraging the functionalities provided by APIs without having to create those functionalities from scratch.

In summary, APIs were developed to facilitate seamless communication, integration, and cooperation between different software components, systems, or services. They play a fundamental role in modern software development by enabling interoperability, modularity, and the creation of diverse and interconnected software ecosystems.

Let’s explore more on the code reusability aspect of it. Code reusability is one of the key benefits and motivations for writing APIs. APIs provide a way to encapsulate complex functionalities or services into reusable components that can be easily integrated into different applications. This reusability not only saves development time and effort but also promotes consistency and reduces the risk of errors. Developers can leverage existing APIs to avoid reinventing the wheel and focus on building higher-level functionalities. So, yes, code reusability is a significant aspect of the core need to write APIs.

In the absence of APIs, what would the client’s course of action entail? It’s crucial to grasp the intricacies we would confront if APIs were absent from the equation:

Complex Integration: Integrating different software systems or components would become highly complex and error-prone. Clients would need to develop custom solutions for every interaction, leading to duplicated efforts and increased development time.

Lack of Standardization: Without APIs, there would be no standardized way to interact with external services or functionalities. Clients would need to understand and adapt to the unique protocols and interfaces of each system they want to communicate with.

Inefficient Communication: Communication between systems would be ad-hoc and inefficient. Clients might resort to methods like direct database queries or file exchanges, resulting in slower and less reliable data exchange.

Security Implications: In the absence of APIs, clients may resort to manual data entry or file exchanges, increasing the risk of data breaches due to human error and unstructured data transmission, necessitating enhanced security measures like encryption and access controls.

Limited Reusability: The absence of APIs would hinder code reusability. Developers would need to create functionalities from scratch for every project, leading to a lack of consistency and a higher chance of errors.

Higher Development Costs: Developing custom integrations for each interaction would increase development costs significantly. Clients would need to allocate more resources to handle integration complexities.

Reduced Innovation: Without APIs, third-party developers would struggle to build applications that leverage the capabilities of existing systems. This could stifle innovation and limit the growth of software ecosystems.

Vendor Lock-In: Clients would become heavily dependent on specific vendors or technologies, as switching or migrating to different solutions would require rewriting large portions of code.

Maintenance Challenges: Updates or changes in one system might lead to cascading changes in other systems that interact with it, making maintenance and updates much more challenging.

Scalability Issues: Scaling applications would become more difficult, as custom integrations might not be optimized for performance or adapt well to increased loads.

Limited Interoperability: Different software systems might not be able to communicate effectively, limiting the potential for collaboration and hindering the development of integrated and cohesive software solutions.

In essence, the availability of APIs significantly simplifies and enhances the way software components and systems interact, offering standardization, reusability, efficiency, and the potential for innovation. APIs provide a structured and manageable approach to integration, enabling clients to focus on building higher-level functionalities and creating more robust and flexible software applications.

Advent of APIs

Over time, the evolution of APIs has seen distinct phases of change in both their declaration and definition. In this section, we will delve into a detailed examination of these transformative phases.

Socket-based Network Communication

In earlier days, communicating over a network was a much more low-level and hands-on endeavor. To establish communication with a server, you had to rely on the Sockets library provided by the operating system. This library had no relation to modern concepts like Socket.io or WebSockets; it was a fundamental building block for network communication.

The sockets library offered a relatively simple but raw API with basic functions like send and read which allowed you to transmit and receive individual bits of character data. However, beyond these functions, you were pretty much on your own.

Working with these raw character bits could be quite cumbersome and error-prone. You had to manually handle the encoding and decoding of data, ensuring data integrity and dealing with potential issues like data fragmentation and reassembly. Essentially, you were responsible for managing the entire communication process from start to finish.

Imagine sending a simple text message. You would need to convert your text into bytes, send those bytes over the network using the ‘send’ function, then receive the bytes on the other end and convert them back into a readable message. This manual handling of data at the byte level could lead to mistakes and inefficiencies, not to mention the challenges of dealing with different character encodings.

As a result, even though the chat applications served as valuable learning experiences, they also highlighted the need for higher-level abstractions that could simplify and standardize the process of network communication. This need eventually led to the development of more advanced libraries, protocols, and APIs like gRPC, which provide a more structured and efficient way to communicate over networks without the low-level intricacies of managing individual character bits.

Figure 1.1: Networking in socket-based communication

In those earlier days of network communication, simplicity coexisted with complexity. The reliance on the Sockets library for connecting to servers introduced a world of raw data transmission. While it provided fundamental functions for sending and receiving character bits, it left the burden of data management squarely on the shoulders of developers.

Consider the task of sending a text message. It involved the meticulous conversion of text into bytes, followed by the transmission of those bytes using the ‘send’ function. On the receiving end, these bytes had to be carefully decoded back into a human-readable message. This manual byte-level handling not only introduced room for errors but also created inefficiencies, especially when dealing with various character encodings.

As the demand for network communication grew, so did the need for standardized, simplified, and efficient methods. REST APIs simplified communication by abstracting away the complexities associated with raw data transmission. Developers could now design APIs that offered a more straightforward interface, making it easier to transmit and receive data in a format that aligns with the principles of REST. While REST APIs served as a significant improvement over raw socket-based communication, they still had limitations, especially in scenarios where high-performance and real-time communication were crucial. Developers yearned for even more efficient ways of handling communication over networks, paving the way for the evolution of libraries, protocols, and APIs like gRPC. These yearnings acted as a catalyst for the development of gRPC, a high-performance RPC (Remote Procedure Call) framework developed by Google. Unlike REST, which relies on JSON over HTTP, gRPC uses Protocol Buffers over HTTP/2, providing a more efficient and faster communication protocol.

In this brave new world, developers could now focus on the essence of their communication, freeing themselves from the intricacies of low-level data management. The advent of gRPC and similar technologies marked a significant leap forward in making network communication more accessible, efficient, and reliable than ever before.

Introduction of RPC

The emergence of the Remote Procedure Call (RPC) concept stemmed from the recognition that much of the complexity in networking arises from managing transmitted bits. People began contemplating a scenario where they could transcend network intricacies, seamlessly invoking functions on remote servers in a manner akin to local function calls. This conceptualization paved the way for the advent of RPC.

Figure 1.2: Simplified RPC flow

To enable such a seamless interaction, the initial step involves defining your API in an Interface Definition Language (IDL) file. Subsequently, you can leverage the IDL compiler tool to generate a code stub. This code stub serves as a fundamental bridge, automating the intricate processes of message serialization and deserialization for both the client and server facets of the communication.

With this code stub in place, the complexity of converting data structures into a format that can be transmitted over the network is significantly alleviated. The stub seamlessly handles the conversion of data into a standardized form suitable for transmission and then transforms the received data back into its original structure.

This automation of serialization and deserialization profoundly simplifies the communication process, liberating developers from the meticulous task of manual data conversion. The generated code stub acts as a mediator that ensures data integrity, format consistency, and reliable communication between client and server. It underscores the essence of abstraction in modern network programming, allowing developers to focus on the core logic and functionality of their applications rather than grappling with the nitty-gritty of data encoding and decoding.

SOAP

Before the era of REST came into full swing, there existed SOAP, a protocol initially introduced by Microsoft. SOAP carries forward many RPC characteristics. Notably, SOAP employs WSDL (Web Service Description Language) as its schema file, encapsulated in XML. This XML-based representation ensures language and platform agnosticism, empowering different programming languages and integrated development environments (IDEs) to seamlessly establish communication channels.

Furthermore, SOAP introduces an additional layer of security. It guarantees privacy and integrity within transactions while enabling message-level encryption. This combination results in a robust framework suitable for transactions of enterprise-grade quality.

However, REST’s ascent to prominence was swift, eventually overshadowing SOAP’s influence and asserting its dominance in the technological landscape.

Contrary to common belief, SOAP actually emerged subsequent to the concept of RPC. The distinction lies in the fact that during that period, monolithic architecture didn’t align well with the principles of RPC. Interestingly, the prevalence of RPC has witnessed a resurgence in contemporary times compared to its earlier adoption.

A typical format of a soap request to get book details would look like this:

1.0>

xmlns:SOAP-ENV = http://www.w3.org/2001/12/soap-envelope

SOAP-ENV:encodingStyle = http://www.w3.org/2001/12/soap-encoding>

http://www.example.com/>

HVT076236

The corresponding response would be this:

1.0>

xmlns:soap=http://www.w3.org/2003/05/soap-envelope/

soap:encodingStyle=http://www.w3.org/2003/05/soap-encoding>

Modern API Design

AVA Orange

Understand why SOAP’s replacement was considered!

It was because the data size was too big for few details, including response and requests.

REST

Dissimilar from its counterparts, REST distinguishes itself by not prescribing rigid regulations or a standardized toolkit. Rather, it embodies an architectural style, delineating a collection of architectural principles and understandings. An API that aligns with these REST constraints earns the moniker of being RESTful.

The most noteworthy advantage of REST resides in its simplicity and innate comprehensibility. It harnesses standard HTTP methods and adopts a resource-centric approach to API definition. It’s akin to transplanting the principles of Object-Oriented Programming (OOP) into the realm of APIs. REST’s popularity can be attributed to its accessibility, enabling developers of all backgrounds to craft RESTful APIs with ease.

If we compare SOAP’s request-response messages with REST’s, we get this:

Request to get a book by ISBN:

GET /books/{isbn}

and the response would look like:

{

bookName: Sample Book,

publisherName: Example Publications

}

Interestingly, it’s not uncommon to observe projects predominantly utilizing only the GET and POST HTTP methods. Remarkably, such a streamlined approach does not diminish the potency bestowed by REST’s principles.

However, as I highlighted earlier, this lenient standardization also gives rise to certain challenges:

Code Maintenance and Reliability: As a project scales in complexity, the codebase can become convoluted and prone to errors. The flexibility of REST can inadvertently contribute to this challenge.

Tight Coupling of Teams: The interaction between frontend and backend teams can become intricately intertwined due to the unrestricted nature of REST. Any modifications to the API necessitate collaborative efforts from both sides.

Multiple API Calls: RESTful architectures occasionally involve multiple API calls to retrieve necessary information, leading to increased latency and potential performance bottlenecks.

Over-fetching and Under-fetching: The design of REST APIs sometimes results in fetching more data than needed (over-fetching) or not enough data (under-fetching), impacting efficiency and resource utilization.

Duplicate Endpoints: Developers might find themselves compelled to create duplicate API endpoints to cater to diverse client consumers, such as web browsers and mobile applications.

The duality of REST’s flexibility, enabling rapid development, and its potential to generate complexities and inefficiencies underscores the importance of careful design considerations to strike a balance between ease of use and system efficiency.

The challenges I previously outlined within the context of REST have found their resolutions in the form of GraphQL, a technology that addresses those limitations head-on. However, it’s important to recognize that there are additional underlying intricacies that gRPC, another technology, has effectively tackled. This alignment between gRPC’s solutions and the fervent embrace of microservice architecture has proven to be particularly significant.

GraphQL’s emergence has been a direct response to REST’s issues like over-fetching, under-fetching, and the need for multiple API calls. With GraphQL, the client is empowered to precisely specify the data it requires, facilitating a streamlined and efficient exchange between the client and server. This level of customization effectively eliminates the wastage of resources and improves performance. GraphQL’s schema-driven approach enhances discoverability and empowers frontend and backend teams to work more independently, minimizing tight coupling and facilitating iterative development.

On the other hand, gRPC stepped into the scene aligned with the surging adoption of microservice architecture. gRPC’s proficiency lies in its efficient and performant communication between distributed services. It employs binary serialization and compact payload, resulting in faster data transfer. gRPC’s support for bidirectional streaming enhances real-time interactions. The microservice paradigm, which emphasizes modular and independently deployable components, dovetails well with gRPC’s capabilities, enabling seamless communication within these microservices. gRPC’s auto-generated code and schema-driven approach expedite development and enhance system maintainability.

In essence, GraphQL and gRPC, while addressing different dimensions of the challenges posed by REST and distributed systems, have significantly contributed to the advancement of modern application architectures. Their impact has been particularly pronounced as technology landscapes evolve towards more modular, efficient, and scalable paradigms, like microservices. This convergence of innovative solutions with changing architectural preferences underscores the dynamic nature of technology’s evolution.

In this book, we shall explore the gRPC side of things.

Costs Associated with Traditional API Frameworks

Traditional API frameworks have long been burdened by several inherent costs, each affecting the efficiency and performance of data exchange over networks. Firstly, the expense of transmitting data cannot be overlooked. In the world of traditional APIs, data transfer often incurs high costs, especially when dealing with large payloads. Serialization and deserialization, the processes of encoding and decoding data, come with their own price tag. These operations can consume valuable computational resources and time, leading to delays in response times.

Furthermore, setting up and maintaining network connections adds to the overall cost. Establishing and closing connections can be resource-intensive, and the overhead of creating and managing these connections can hinder scalability and responsiveness. In essence, the traditional approach to APIs has been riddled with these costs, making it imperative for the industry to seek more efficient alternatives.

Figure 1.3: Illustration of various costs associated with request-response cycle

Let’s detail out the costs associated with accessing APIs over the network.

Accessing APIs over a network can entail various costs, both in terms of financial expenses and performance considerations. Here are some costs associated with accessing APIs over a network:

Latency: Network communication introduces latency, the time delay between sending a request and receiving a response. This delay can significantly impact the responsiveness of applications, especially those relying on real-time interactions.

Network Bandwidth: API requests and responses consume network bandwidth. Frequent or large API calls can lead to increased network traffic, affecting overall performance and potentially causing congestion.

Data Transfer Costs: Many cloud providers and API services charge based on the volume of data transferred. APIs with large payloads or high-frequency calls can result in substantial data transfer costs.

Rate Limiting Charges: Certain APIs enforce rate limits on the number of requests within a specified time frame. Exceeding these limits may incur additional charges or necessitate an upgrade to a higher-priced plan.

API Subscription Fees: Numerous APIs, particularly third-party services, come with subscription fees. These fees, based on access levels, usage, and required features, contribute to operational costs.

Infrastructure Costs: For APIs that are hosted on a cloud platform, there are associated infrastructure costs for server provisioning, maintenance, and scaling based on usage demands.

Security and Authentication: Implementing secure communication for APIs often requires encryption, authentication, and authorization mechanisms. These measures can add complexity and overhead to API requests.

Development and Maintenance: Developing and maintaining APIs can involve costs related to coding, testing, documentation, and ongoing updates to ensure reliability and compatibility.

Caching and Content Delivery: To mitigate latency and reduce network traffic, caching mechanisms and content delivery networks (CDNs) might be employed, which can introduce additional costs.

Monitoring and Analytics: Effective monitoring and analytics tools are necessary to track API usage, performance, and potential issues, incurring costs for required tools and resources.

Failover and Redundancy: Ensuring the high availability of APIs often requires redundant infrastructure, failover mechanisms, and disaster recovery strategies, which can involve additional expenses.

Training and Support: To effectively utilize and troubleshoot APIs, training and support might be needed, which can incur costs for education and access to technical assistance.

It’s essential to consider these costs when designing, implementing, and utilizing APIs, as they can impact both the financial aspects and the overall performance of your applications. Balancing cost considerations with the benefits of API usage is crucial for achieving optimal results.

Let's draw our attention particularly on three costs:

Serializing and Deserializing Cost

Serializing and deserializing, also known as serialization and deserialization, refer to the process of converting data structures or objects into a format suitable for transmission or storage and then converting them back into their original form. While these operations are fundamental for data exchange, they come with certain costs that developers should be aware of:

Serialization Costs

Computational Overhead: Serializing data involves converting complex data structures into a format like JSON, XML, or Protocol Buffers. This process requires CPU and memory usage, which can impact the overall performance of the system.

Conversion Time: The serialization process takes time, especially for large or intricate data structures. This can lead to delays when preparing data for transmission.

Increased Payload Size: Serialized data often includes additional metadata and formatting, which can increase the payload size. This can lead to higher network usage and potentially impact response times, particularly in scenarios with limited bandwidth.

Deserialization Costs

Processing Overhead: Deserializing data requires interpreting the serialized format and reconstructing the original data structure. This process also consumes CPU and memory resources.

Latency: Deserialization introduces latency, as the receiving end needs to wait for the entire serialized data to be received and then deserialized before it can be processed.

Parsing Complexity: In more complex data formats, like XML or JSON, parsing and deserialization can become computationally expensive, potentially affecting the system’s overall responsiveness.

Compatibility Overhead: Deserializing data may encounter challenges related to versioning and compatibility, especially in evolving systems where data structures may change over time. Ensuring backward and forward compatibility in deserialization processes can introduce additional complexities. Developers may need to implement strategies such as versioning mechanisms or handling gracefully when encountering different versions of serialized data to prevent errors and maintain system stability.

Mitigating the Costs

Efficient Formats: Choosing efficient serialization formats like Protocol Buffers or