Background
Our client is largest independent processor of flat-rolled steel and manufacturer of industrial cylinders and pressure vessels in the United States.
Heavily customized BI stack (Informatica mappings, Data Warehouse, RPD Models, and Dashboards) hosted on Oracle Cloud. Cloudera Data lake on AWS to capture detailed operational data from machines, sensors, and shop floor applications to help improve production efficiency and quality.
However, both the Oracle DW and Cloudera Data lake were unstable and unable to integrate with BI tools, hence there was a strong need to modernize the cloud data and analytics platform.
Current Challenges
The client faced several challenges with their existing data infrastructure:
- Stability of the Cloudera data lake – many outages and incidents
- Data redundancy across data warehouse, data lake, and tableau extracts
- Tableau connectivity issue to CDP Impala
- Tableau extracts required to hit query performance targets
- Unable to reliably hit data latency SLA’s
- Complexity of environment requires additional resources
- Lacking sufficient visibility over data pipeline health and data quality
Solution
To address these challenges, we proposed the following solution:
- Recommend and prove out the target state data and analytics architecture
- Evaluate and shortlist technology and tools (Matillion, Talend, DBT, Streamsets, Airflow, Astronomer)
- Pilot two uses on the recommended architecture on Data Warehouse (ETL process) and Data Lake (streaming data) to meet success criteria including acceptable performance
- Help setup Infrastructure of (AWS, Snowflake, Streamsets, environment
- Extend the pilot implementation to migrate DW tables (54 Dim and 60 Fact) and Datalake artifacts (274 base tables and 73 views) for about 800 users
- Setup and automate CI / CD pipeline with Github actions and branches for Streamsets + Snowflake with scheduling and monitoring tools
The Results
- 30% performance improvement by converting Oracle Cloud and Cloudera to AWS-Snowflake stack.
- 20% SLA improvement by modernizing Oracle and Cloudera workloads on AWS-Snowflake.
- 50% cost and time savings when transforming Informatica workflows and Oracle EDW to AWS.
- Reduction in operational expenses, over $350k annual savings from consolidating Data Warehouse and Data Lakes.
Migrating data from on-premises servers to the AWS cloud significantly improved reporting capabilities. The unified and scalable platform facilitated self-service BI, reducing manual reporting efforts, enhancing data availability, and enabling data-driven decision-making.
Related Posts
Unlocking the Power of Cloud Services: Modernization, Migration, and FinOps
Unlocking the Power of Cloud Services: Modernization, Migration, and FinOps Cloud technology has become a cornerstone of modern business operations, offering unparalleled scalability, flexibility, and cost-efficiency. The journey to fully leveraging cloud capabilities involves several…
- Aug 01
Maximizing Business Potential: Unlocking Comprehensive Data Services
Introduction In today’s data-centric landscape, leveraging, modernizing, and migrating data effectively is pivotal for business success. Data serves as the backbone of modern enterprises, fueling strategic decision-making and operational efficiency. Explore comprehensive data services, including…
- Jun 28
Latest Post
“We’re an Al, Data, and Quality Engineering company “
Headquarters
8845 Governors Hill Dr, Suite 201
Cincinnati, OH 45249
Our Branches
Narwal | © 2024 All rights reserved
Comments (8)
Ismael
Apr 22, 2024After looking into a handful of the articles on your web site, I seriously like your technique of blogging.
I saved it to my bookmark site list and will be checking back in the near future.
Please check out my website as well and let me know how you feel.
Ganesh Anabattula
May 17, 2024Thank you for your feedback and for bookmarking our site! We’re glad you enjoy our blogs. We’ll definitely check out your website. If you need any support with AI, Data, or Cloud services, feel free to reach out.
Keep exploring our insights (Blogs, Success Stories, Use Cases, and News) here: https://copy.narwal.ai/insights/
contact us: https://copy.narwal.ai/contact-us/
Reynaldo
May 08, 2024What’s Happening i’m new to this, I stumbled upon this I’ve discovered It absolutely helpful and it
has aided me out loads. I hope to contribute &
aid different customers like its helped me. Good job.
Ganesh Anabattula
May 17, 2024Thank you! We’re glad you found our content helpful. At Narwal, we specialize in AI, Data, Cloud, and Quality Engineering services. If you’re looking to contribute and help others, we’d love to connect with you. Explore our services and see how we can work together to drive innovation and success.
Check our services here: https://copy.narwal.ai/services/
contact us: https://copy.narwal.ai/contact-us/
Concetta
May 28, 2024Greetings from Ohio! I’m bored at work so I decided to browse your
site on my iphone during lunch break. I enjoy the knowledge you provide here and can’t wait to take a look
when I get home. I’m shocked at how quick your blog loaded on my mobile ..
I’m not even using WIFI, just 3G .. Anyways, superb site!
drover sointeru
Jan 09, 2025You actually make it seem so easy with your presentation but I find this matter to be actually something that I think I would never understand. It seems too complex and extremely broad for me. I’m looking forward for your next post, I’ll try to get the hang of it!
zoritoler imol
Jan 22, 2025Hmm it seems like your blog ate my first comment (it was super long) so I guess I’ll just sum it up what I submitted and say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new to the whole thing. Do you have any tips for newbie blog writers? I’d certainly appreciate it.
zoritoler imol
Jan 31, 2025My brother suggested I might like this blog. He was entirely right. This publish actually made my day. You cann’t consider just how a lot time I had spent for this info! Thank you!