CLIENT
An Applied Artificial Intelligence Solution Provider with the mission to help organisations harvest the power of the AI revolution. Building solutions for a future where machine learning and data science will be the driving factors of businesses in a constantly changing reality.
CHALLENGE
One of Arinti’s main customers, a multinational FMCG company, had the need to implement DevOps on a data analytics environment and to improve its data lake architecture for the Benelux team with the aim to prevent waste in their products production.
The project demanded, as tech leader, a Senior Data Engineer with solid experience in Azure, Databricks & Python, and advanced English and tech consultancy skills. They relied on Arinti to place this tech consultant.
SOLUTION
Pluralit placed Bruno Magri for this project, a talented data engineer expert from Brazil. Along with Arinti, he worked directly with the customer to deliver the best solution.
To implement DevOps in a data analytics environment, Bruno needed to consolidate the different code versions in a final git branch and, for that, he gathered and aligned all correct code from several people in a manual process, hence bridging the gap between the DEV, QA and Production environments.
Leveraging his previous experience, Bruno aligned the development process with the project manager to enable the implementation of all the DevOps pipelines with the correct continuous integration (CI) and continuous deployment (CD) processes across the client environments. Furthermore, to optimise the architecture of the data lake, he suggested and implemented the Medallion Architecture, a series of data layers that guarantees consistency, easy data lineage, and data quality in the data lake house.
RESULTS AND BENEFITS
The implementation of DevOps CI/CD reduced human and data error, increased product reliability and gained efficiency and agility in the delivery of new features.
Thanks to the restructured data lake, they acquired greater flexibility in handling data using a known design pattern and obtained around 45,000 euros of annual cost savings. It also boosted the speed of data delivery due to decreasing the tiers of the database and semantic layers in the cloud.
All these results allowed the end-customer to increase profit and reduce waste, being able to do statistical analysis with all processed data and to foresee the production more efficiently.
The success of this Benelux project turned it into the end-customers “European Data Foundation template", encompassing the best practices to build a data lake for all their European sites.