The Need
Our customer, global leader is Logistic industry was looking to develop & deploy microservices that will move data from different systems to a single centralized platform as part of their BI Convergence program.
Impacts Delivered
- As part of BI convergence program, data from all standalone systems were migrated to Azure cloud DB.
- From the Azure DB, the data was brought to a common JSON format before sending to the common convergence platform.
- The solution handled all global data from multiple regions
Our Solution
- Gadgeon team had to ramp up and migrated the data from different systems using microservices and Kafka. Completed the phase -1 development and deployment of the solution as:
- Producer microservices to take data from systems and publish to Kafka.
- Overlapping data is filtered and published to Kafka.
- A consumer microservices is created to pull data from other source data from Kafka topic and pushed to elastic search in the DB.
- Done the initial load of historical data and then continuously take new data based on events.
- Moved the data to Azure and then converted to JSON format, placed it in a staging DB for publishing to Kafka
- The DB system handles all the Global customer database and has many applications/ services deployed on prem. There are three main DBs currently in the system that handles all global data from two regions.
- The DB contained data from multiple sources like SAP, ACCOUNTIS, MAC, BNF, IBS+, etc.
- The producer and consumer microservice were developed and deployed.
- Certain percentage of Data is migrated to the convergence platform as per business requirement and certain percentage to other system dBs'
- Producer microservices to take data from systems and Customer onboarding process system consists of 3 separate systems. It has internal workflow management engine and related services as well as separate DB’s for each system.
- As part of BI convergence data from all the standalone systems are migrated to Azure cloud DB and then brought a common JSON format before sending to the common convergence platform.
- Gadgeon team provided supported data model discussions and all phases of testing by the client.
- Supplied extensive documentation and knowledge transfer to the client team.
Tools & Technologies
- Azure cloud, Microservices on Azure
- JSON format.
- Kafka
Data Migration Process - Block Diagram
