Increasing Scalability and Database Performance with Microservices

One of the world’s most successful global asset management software company was facing a problem of data flooding creating chaos all along with its technology infrastructure.

A pioneer of Asset Performance Management (APM), the company's business model is to leverage the data from sensors, devices, systems, and smart equipment to minimize unplanned events, incidents, and downtime. Data is the bread and butter for the organization with global clients in more than 80 countries with more than 1,200 licensed sites around the world. The software was used to collect data from assets and devices located in manufacturing plants, to monitor various policies, and to provide alerts and reports to plant management.

Existing software used a legacy data solution using RDBMS to store data received from the assets. As the number of assets increased, the database management system and architecture were unable to scale up to the demands. This scalability issue was beginning to hamper the core value proposition of this company which is to offer fast and accurate analytics of assets to its customers.

To tackle this challenge, the company partnered with Accion to re-architect its solution. This was not an easy problem. Data was a mix of structured and unstructured information and it required an in-depth understanding of the existing architecture with a clear goal-setting. Enhancements demanded to the existing system were significant to be able to handle data volume and velocity.

RE-ENGINEERING DATABASE MANAGEMENT SYSTEM FOR HIGH DATA VOLUME AND VELOCITY

Accion engaged with this company to plan and implement their unique digital journey. The kick-off to the discussions happened with the scheduling of extensive deep-dive sessions wherein Accion engineers highlighted the problems with RDBMS, citing that over the years, Relations Database Management Systems or RDBMS has been the first choice for most organizations to manage their data storage. However, the fast-growing volume of data has made it imperative for companies to build a robust data management system that is able to take the load. In the case of this company, the limited capability to integrate unstructured data from smart devices was choking up the system and needed a major overhaul.

As part of the transformation plan, Accion built a new data processing framework that enabled the transferring of data from multiple data stores (RDBMS, files, etc.) to a single NoSQL database - Cassandra. This new NoSQL store acted as a reporting database that received data on a near real-time basis and served the data through scorecards and reports.

Cassandra is a highly scalable distributed database originally built at Facebook and today enjoys widespread adoption by leading data-driven companies. The key features that have made it so popular and why it was used in this project was the ease of use and its core ability to work wonders with volume and velocity of data. Also as more and more enterprises shift to the cloud, Cassandra is well suited because it can scale from on-premise to the cloud. It performs transparent node balancing wherein data is automatically distributed across nodes. With an increase in demand, new nodes are added and the system transparently distributes the data to the new servers. It is quite robust and works well with Hadoop and HDFS and is accessible via Apache Spark, with support for multiple languages including Java, Python, Ruby, and C++.

The Accion Innovation Centre (AIC) team designed and built a micro-services, polyglot persistence based-architecture for the company. This increased asset reliability and improved the overall performance of their production assets. To further enhance customer experience, a new format for reports of operational data to their internal users was launched so that they can make quick and precise business decisions to serve their customers.

The new solution refactored code for the ingestion engine into a set of micro-services that handled different types of assets. The new architecture also refactored the policy engine into a set of micro-services, allowing new policies or custom-built modules to be added according to customer requirements.

TECHNOLOGY STACK

Frontend: HTML5, CSS, JS, AngularJS, Bootstrap,

Backend: Java, Spring, NodeJS, RabbitMQ, Cassandra, Oracle, Hive, Hadoop,

AWS: ELB, EC2, S3, VPC, SQS

Code: Grunt/Bower

Testing: Karma/Jasmine/Protractor

CI: Jenkins/Chef

If you are looking to build scalable solutions with microservices, Contact us and we will be happy to do a deep dive session with you.