Industry Challenges
Processing and computing analytics was a daunting task due to the overwhelming influx of data into business intelligence (BI) tools. Frequent dashboard crashes hindered the ability to deliver up-to-date data analytics and visualizations to customers. Additionally, when multiple team members simultaneously accessed data from databases and data warehouses, it adversely affected data performance and led to access problems. The challenge was to develop an analytics solution that could handle large volumes of data, enable customized business calculations, provide real-time data visualization, and ensure secure data access for multiple team members.
- Volume
With millions of data flowing into BI tools, it is hard to process and compute analytics. - Velocity
High velocity of data generation leads to frequent dashboard crashing, making it difficult to provide recent data analytics and visualization to customers. - Warehouse
There was a need for a robust data warehouse where multiple team members can access data from data warehouse without impacting performance.
Problem Statement
By enabling its customers to automate commerce and marketing, our client is a leader in conversational engagement platforms with over 10 billion messages sent every month. The client wants to track the performance of its bots and unlock meaningful insights from the bot chat sessions.
Our Solution
- The first step was to troubleshoot the major issue of handling large data sizes. Performance auditing was done in existing Tableau reports to record performance information about key events as we interacted with the workbook. This enabled the analysis and troubleshooting of different events that affected performance, such as query execution time, geocoding, extract generation, blending data, etc.
- Next, we used Redshift for Data warehousing and Materialized View creation. Using performance testing, MV’s were created at various levels of granularity, and custom metric calculations were embedded inside the MV itself. This helped to reduce the time taken in Tableau and resulted in lean and light data.
- The optimized MV was published on the Tableau server, making it available in a centralized area for anyone with authorized permissions, resulting in better access and performance.
- New versions of Tableau dashboards were created that were highly optimized and quickly accessible.
- After QA, the optimized dashboard was pushed to the customer-facing environment, enabling customers to access and monitor their WABA numbers and bot performance across various channels in near real time.
- We also implemented RLS, i.e. row level security to restrict user view and provide access to only their data.
- The entire process of continuous improvisation took almost a year to complete.
Technology
Business Impact
- The optimized dashboard performance resulted in faster data processing and analysis, enabling the client’s team and its customers to make informed decisions.
- Near real-time analytics provided to customers enabled them to track bot performance in campaigns and make quick decisions, which could lead to improved customer retention and acquisition of new customers.
- The time taken to create reports was reduced from 3-4 weeks to just one week, increasing efficiency and productivity.
- The solution reduced complexity and enabled seamless data access, allowing team members to focus on analyzing data rather than troubleshooting performance issues. Additionally, the implementation of RLS streamlined user access and reinforced security.
