The Challenge
Creative Assembly’s existing telemetry system was no longer fit for purpose. As data volumes grew, reaching over 150TB and half a billion events per day, the platform struggled to keep up. Reports were taking up to 24 hours to reflect in-game actions, creating delays for QA teams and slowing down decision-making. The data model was also overly complex, having evolved over time with each new game title. With over 70 dimensions - many of which now redundant - maintenance and data analysis was difficult and inefficient. Ultimately, Creative Assembly had outgrown their telemetry system, and needed a solution that would scale with them, as well as improve the state of their data.
The Solution
Working collaboratively, we migrated their telemetry data to a modern architecture using Databricks. This included:
- Streamlined Data Modelling: Reduced 70 dimensions to just 10 by eliminating redundant structures and using business keys directly in fact tables for more efficient data modelling.
- Faster Data Availability: Data now lands in the silver layer within minutes, with aggregated fact tables refreshed every four hours, giving Creative Assembly the flexibility to reduce this time with more compute.
- Agile Delivery: The project was delivered using agile best practices, with clear ceremonies, contingency planning, and proactive communication. This helped them build reliable and consistent development practices.
- Team Enablement: Creative Assembly’s internal data engineer described the experience as “an intense bootcamp” that dramatically accelerated their learning and confidence.
The Results
- Speed: Reporting time dropped from days to minutes, enabling near real-time QA validation and faster iteration cycles.
- Cost Efficiency: Daily processing costs were significantly reduced, with more efficient resource usage.
- Scalability: The solution is now being extended to other titles in the studio’s portfolio, leveraging metadata driven processes to minimise management overhead.
- Simplicity: The new model is cleaner, easier to maintain, and more intuitive for internal teams.
Why it Matters
This project demonstrates how thoughtful data platform modernisation can transform game development workflows. By streamlining telemetry pipelines, simplifying architecture, and empowering internal teams, Creative Assembly now has a telemetry pipeline that supports faster feedback loops, better player experiences, and long-term scalability.
