Microservices has emerged as a preferred architectural model for large cloud hosted applications. One of the design patterns in the microservices architecture is an API Gateway, that acts as an abstraction layer to isolate and consolidate backend microservices into use-case specific front end services for client applications. This paper explores the requirements of an API Gateway and evaluates some technology options for implementing it. It also includes an example application that was used to evaluate the technology options to compare results of an informal benchmark test to validate the performance of the technology options so as to arrive at a recommendation.
Large cloud-hosted applications are rapidly moving away from legacy a monolithic architecture to a loosely coupled set of independently deployable microservices. The canonical microservices architecture consists of:
We will use an example of a microservices based Data Processing Application as a reference, so that we can use it to illustrate actual performance of an API Gateway. A simplified architecture of the Data Processing Application is illustrated below:
Let us consider two types of operations that this application handled:
The API Gateway addresses various issues that arise out of the fine-grained services that Microservices applications expose.
The API Gateway thus either acts as a proxy or router to the appropriate backend service, or provides a combined front end service by fanning out to multiple backend services. As a result, the API Gateway needs to be able to handle a large number of simultaneous incoming and outgoing service calls. Therefore, it needs to optimally utilize its resources - CPU and RAM primarily - for each request, so as to be able to handle the maximum number of concurrent service calls. For all service calls involve network I/O requests, each service call will include some I/O wait time that is inversely proportional to the network speed. The following diagram illustrates a single inbound service request that involves executing three sequential outbound service requests.
As seen from this diagram, the execution of each inbound service call involves at least one outbound service call, and each outbound service call involves I/O wait time till the service responds with a result.
We considered two options for the technology platform to implement the API Gateway:
Note: There are also JVM based non-blocking I/O (NIO) platforms such as Netty, Vertx, Spring Reactor, or JBoss Undertow. However these were not considered in this evaluation.
The most important differences between NodeJS and Java are the concurrency and I/O models. Java uses multi-threaded synchronous, blocking I/O while NodeJS uses single threaded asynchronous, non-blocking I/O.
The following diagram illustrates the difference between the two concurrent execution models.
In a multi-threaded environment, multiple requests can be concurrently executed, while in a single-thread environment multiple requests are sequentially executed. Of course, the multi-threaded environment utilizes more resources. It may seem obvious that a multi-threaded environment will be able to handle more requests per second than the single-threaded environment. This is generally true for requests that are compute intensive and utilize the allocated resources extensively. However, in cases such as the API Gateway, where requests involve a lot of I/O wait times, the allocated CPU and memory resources are not utilized during this wait time.
The following diagrams illustrate a scenario where each request involves I/O wait times, in multi-threaded, blocking I/O v/s single threaded non-blocking I/O.
From this diagram it is clear that though the single threaded model takes longer to process multiple requests, the delay is not as much as the previous requests that did not involve I/O wait times. It is therefore very likely that when a large number of concurrent requests are to be handled, the single-threaded, non-blocking I/O model will give better throughput than the multi-threaded, blocking I/O model.
To test the performance of the two platforms, we implemented a sample API gateway that simulated the three backend service calls using timed mock services so that we could control the I/O wait state to a predictable level based on actual expected time taken by each of the backend microservices. We tested a single inbound API for ingestion of batch documents in the two implementations - one using the existing Java EE platform and one using NodeJS. Then we tested the performance of both these applications by running a series of tests for ingestion of the same set of documents. The tests were conducted on the same m4.xlarge EC2 instance on Amazon Web Services with 4 vCPUs and 16 GiB memory. The CPU and RAM utilization was monitored using Amazon CloudWatch. The results were as follows:
JAVA APPLICATION RESULTS
NODE JS APPLICATION RESULTS
PERFORMANCE COMPARISON
The results validated our expectation that the single-threaded, non-blocking I/O model of NodeJS would provide better throughput per server instance than the multi-threaded, blocking I/O model in Java. The final implementation of the Data Processing Engine was deployed using NodeJS and extensive performance tests were carried out on a clustered setup with two DPE nodes.
The following table summarizes the results of the performance tests:
Based on the analysis of the API Gateway requirements and benchmark tests, we recommend using a non-blocking I/O platform such as NodeJS to implement the API Gateway. The tests conducted in this study did not evaluate other non-blocking I/O platforms. Further, subsequent to this study, there have been success stories reported of implementing API Gateways with JVM based platforms such as Netflix’s Hystrix. However, these have not yet been benchmarked by Accion Innovation Center to date.