Why Run an AI-BI Hackathon?
At Advancing Analytics, we believe that the best ideas are born when you give smart people the space to experiment, collaborate, and push boundaries.
We also know that in the consulting world, getting that space can be tricky – when you’ve got client projects ticking over and prospective partners to chase, finding the time to even clear your inbox is hard, let alone dedicating time to experimentation and learning.
That’s why we launched our first-ever internal AI/BI Hackathon—a one-day sprint designed to break down siloes, supercharge delivery skills, and build real, reusable solutions that blend the best of AI and business intelligence.
It was my pleasure to lead this event, and I was astounded at what our teams managed to achieve in a single day.
In this blog, I’ll describe the challenge our contestants faced, the solutions they developed for real-world use cases, and why the Hackathon approach is ideal for fostering collaboration, levelling up your developers, and solving everyday issues in unconventional ways.
.png?width=718&height=480&name=aa%20hackathon%20logo%20(Small).png)
Setting Up for Success: Choosing the Right Dataset
With any data-based hackathon, there is an obvious starting point – finding the right dataset.
If your main focus for the event is unlocking new insights, then you may decide that your participants will get the most out of using real and/or proprietary data. This enables them to consider actual use cases from their everyday work, and it also makes it much easier to implement any developed solutions in real applications.
However, this does come with some disadvantages. Firstly, it may limit your ability to share your participant’s work outside of your organization (or even inside your organization, depending on the sensitivity of the data.) Additionally, you create a lot of reliance on your own data quality, and you may find that participants end up going down a rabbit hole trying to fix the data or understand complex data models and definitions – valuable time that should be spent developing and testing solutions instead.
If you’re more interested in finding new approaches, then a more suitable option may be to use public or synthetic datasets – there are hundreds of thousands of such datasets available online, covering every domain imaginable, and generating synthetic data is pretty straightforward in most AI tools. This immediately removes the complexities around data privacy, meaning you can host and share the data and hackathon outputs with relative ease (although you should always check specific copyright and licensing limitations on public datasets to be safe.)
And, importantly, you can limit the scope – a smaller, more focused dataset can often be better for a hackathon, as it means teams will spend less time trying to figure out what to measure, and can instead focus on how they will measure it and interact with it.
For our AI/BI Hackathon, participants were given access to a public Kaggle dataset spanning 120 years of Olympic history, broken down by year, season, athlete, team and sport, and tracking the number of events completed and medals awarded.
When you’re selecting a dataset, here are some things to look for:
- Long History – most BI solutions these days incorporate some degree of time series analysis, so longitudinal datasets provide great opportunities for exploring these and observing trends over time.
Our Olympics dataset spanned over 100 years of data, so was ideal for supporting this type of analysis. - Wide Range – AI/BI solutions thrive on dimensional data models, so pick something with a couple of interesting dimensions – geography is a great option (who doesn’t love a map visualisation?) and the more ways your developers can slice the data, the more interesting their insights will be.
Not only did our Olympic dataset cover countries across the world, it also introduced some interesting challenges, as athletes have occasionally competed under different teams for different Olympics events – and the names of the teams themselves have sometimes changed due to a range of geopolitical events.
(Fun fact – at the 1964 Tokyo Olympics, Northern Rhodesia achieved independence on the very last day of the games, so their team started the games as Northern Rhodesia and finished the games as Zambia! This is the only known case of a team name changing mid-games.) - Multiple Facts – some of the most interesting applications can be seen when participants have the ability to build compound measures and observe correlations. But be careful not to overload them – a handful of measures should be sufficient, and too many can be overwhelming, particularly with an unfamiliar dataset.
Our Olympics dataset effectively had four facts – the number of events competed in by each athlete, and then whether they achieved a bronze, silver or gold medal. We decided that this would be sufficient for our use case and would enable some interesting dimensional modelling opportunities.
On Your Marks: Planning the Day
Our teams were given a simple mission: explore pre-agreed use cases around AI/BI integration, and deliver compelling, client-ready solutions in under eight hours.
The day was managed remotely: we all got together at 9AM for a briefing on Microsoft Teams, and then participants were provided with a suggested agenda, with time allocated for problem framing, planning, prototyping, building, and developing their final playback. They had to present their work at 4PM the same day, so time was tight!
Each team was provided with a Databricks catalog (containing our sample data and a basic data model), a Power BI workspace, a Genie Room, and a set of use cases—then set loose to plan, build, and test their ideas. Mentoring and support were provided throughout, and at the end of the day, participants and judges voted for the winning team.
There are some key areas to consider when planning your Hackathon:
- Pre-agreed use cases – these can really help to focus your teams’ minds and make sure they are building things that will be useful down the road. We came up with six use cases and teams were able to select their own from the list, which meant that they could pick a project that inspired them and make the most of their special interests.
- Timing – a one-day event is often easier to arrange (and to get approval from project resourcing colleagues!) and our participants found that the time pressure was a really useful motivator, as it encouraged them to plan carefully and to be rigorous about their scope. Longer-term events often struggle with drop-outs and a reduction in focus over time, as your attendee’s other commitments will start to drag them away.
- In-person vs remote – if you have the option, in-person is a strong choice, as it encourages better collaboration and enables ‘over-the-shoulder’ support. However, this doesn’t work for all people or all organisations, and we found that running it remotely worked just as well – and was a lot cheaper! Our only recommendation is to avoid a hybrid approach if possible, as this tends to create barriers to team collaboration – if you have to go remote for some, you may as well offer it for all.
- Mentoring – teams will hit roadblocks as they go, and it’s important to have one or more dedicated resources on hand to support. Ideally this should include someone with technical expertise who can advise on best practice and, hopefully, unblock common issues such as permissions and resource access.
- Resources – if you are providing participants with a development environment and specific datasets, it is imperative that access is set up before the event and fully tested with as many participants as possible. Time will be short on the day and you do not want to waste it on administration!
Team Spirit: Our Teams and Their Projects
Getting the team balance right is critical to maximize the benefits for each participant, so – if possible – our recommendation is to get participants to register in advance, and then allocate them into teams yourself.
Ideally you should aim to build teams which are:
- Cross-functional – a mix of Analysts, Data Engineers, AI Engineers and other less technical resources. This fosters understanding between different departments and enables your attendees to learn from an expert in an unfamiliar field – exactly what we’re looking for in a Hackathon.
- Multi-level – try to avoid teams with entirely junior or senior employees, as they will likely get either confused or bored! Combining levels allows juniors to learn from seniors and ensures that each team has a (relatively) fair chance to build something great.
- Well sized – three is the sweet spot, as this allows for a Developer, an Analyst and a Planner, but larger teams can work too. Plan for a drop-out rate of between 10% and 20%, and try to build teams that could manage losing a member at short notice. You can even keep a ‘bench’ of participants if you are over-subscribed, so that you can swap teams around as needed.
- Not already friends – wherever possible, try to arrange teams who are not already familiar with each other! This encourages the building of new relationships and helps to ensure that teams are on equal footing.
For our event, three cross-functional teams were formed, each tackling a unique use case.
Team 1 (Nikki, Ashley and Peter): Genie and Teams Integration
Team 1 wanted to build a solution that could query different datasets and follow different agent instructions based on the user’s input. They also wanted to integrate this with Microsoft Teams so that users could ask questions about the data models directly and receive a tailored response.
To achieve this, they built multiple Genie Rooms, each with a different dataset (the one we'd provided and two sample datasets from Databricks) and instructions on not just which queries to answer, but on how each Genie to respond - one example used the well-known New York Taxi dataset, and the bot was programmed to respond in the manner of a taxi driver, complete with amusing anecdotes!
They also built a Multi-Agent Supervisor in AgentBricks that would direct user queries to the right resource. The Supervisor agent can be connected to multiple Genie Rooms and given instructions on how to direct user queries - so any questions relating to taxi journeys would go to our Taxi-bot, while questions regarding the Olympics were handled by the Olympics Commentator.
Integration with Teams added another layer of complexity, requiring the creation of an Azure Agent that could channel responses to Teams, and then a number of Azure resources to support querying the Genie endpoint. They were agonizingly close to being fully deployed at 4pm – stumbling over some Azure permissions issues at the last hurdle!
Their solution was so impressive that we are already refactoring it for a real client project – a solid testament to the value of running Hackathons to develop working solutions.
Team 2 (Gareth, Minu and Rebecca): Prompting Model Design
Our second team were aiming to build a solution that could speed up one of the more time-consuming elements of Analytics consulting – translating business requirements and user questions into a working data model. They wanted to use an LLM to generate the required artifacts based on metadata and workshop notes, and they also wanted it to test the results against the human-built model that we provided for them.
Their solution was notebook-based, using Python to extract metadata from the catalog, and requirements from workshop outputs, and then to call an LLM directly, prompting it to generate SQL for fact and dimension views and export them to the workspace.
It would also generate SQL queries that could be used to validate supplied user questions against the LLM model and the human-built model, and then run those queries against both models and compare the results.
The notebooks performed admirably, and were able to generate a working Star Schema that returned comparable results to the one we built ourselves – indicating that this would be a valuable improvement to our consulting processes, and could save a lot of time for us and our clients. We were also impressed at the team's meticulous planning and management of time, and could see the impact this had on the quality of their playback and the completeness of their solution.
Team 3 (Mikey, Minu and Aysha): Prompting Personas
Our third team had a similar aim to Team 2 – reformatting Genie responses based on user persona – but they set out to see what could be achieved without significant code changes, using Genie to retrieve data and then customizing the output based on a persona.
They used the Suggested Prompts as a method for allowing the user to select their own persona for customization of responses, and then they used Instructions to advise the Genie agent how to filter its analysis and format its responses according to the selected persona.
As the day progressed, they refocused their efforts on getting a single persona working, and created a Media Analyst persona for this - designed to narrate athlete's journeys, focusing on breakthrough moments, challenges overcome and comparisons with peers.
They found that getting Genie to respond with the appropriate tone and content was tricky when using only the instructions; responses would oven come back initially in tabular format and further prompting was required to achieve specific output formats. However, their solution was workable in the end and showed real promise – confirming that a low-code solution is absolutely achievable in Databricks.
Everyone's A Winner, Baby: Scoring and Prizes
Building cool stuff, collaborating with your colleagues and getting internal recognition are all well and good, but at the end of the day (pun absolutely intended), your participants will work harder with some additional motivation – namely, prizes!
In our case, we used our internal recognition system to award substantial prizes to our winning team members, and participation rewards for all attendees. We also allocated a specific bonus prize to the individual who participants felt had contributed the most to the day. How you arrange your prizes will depend on your organization and budget, but prizes should be valuable (outside of work if possible!) and should be agreed and communicated well in advance, to encourage greater participation.
But how to decide on the winners? We asked our participants and judges to score team solutions between 1 and 5, based on the following criteria (which, purely coincidentally, all start with the same letter…)
- Innovation: Originality and creativity in solution design – using unusual approaches, new or unfamiliar tools, and clever ways of maximising efficiency
- Insight: Relevance of findings and potential impact – displaying genuinely interesting insights and relevant data enrichment
- Implementation: Completeness and quality of solution – as close to usable as possible, with an understanding of gaps and consideration of how they would implement the solution in full
- Interpretation: Clarity of storytelling and communication in their playback presentations – with compelling content, full team participation and confident handling of questions
- Interaction – Team collaboration, planning and organisation on the day, with logical allocation of work and regular communication within teams
You may choose to add or remove categories, or weight the scoring, depending on whether certain elements are more important to you and your organisation.
We collated responses using Microsoft Forms and then we calculated averages across all categories to determine the final scores. We also got teams to vote for each other (alongside the judges and at a lower weighting), as this helped to ensure that they remained engaged during the final presentations, and also encouraged some friendly competition.
Here’s how our teams did on the day:
- Gold (Team 2): Prompting Model Design – Total Score: 4.2 / 5
- Silver (Team 1): Genie and Teams Integration – Total Score: 4.1 / 5
- Bronze (Team 3): Prompting Personas – Total Score: 3.5 / 5
Reflections
The hackathon was designed to be a celebration of our culture of experimentation and learning, and a chance for colleagues to try something new while developing interesting solutions to real problems. The fact that we have built something that will be used in client work is testament to these aims, and a ringing endorsement of the importance of these events for fostering collaboration and expanding our knowledge.
This shone through in the feedback received from attendees, who gave the event 4.75 out of 5, and particularly enjoyed:
-
Exposure to agentic architecture, engineering and AI concepts
-
Connecting up services they hadn’t used before to build something that could genuinely be useful for clients and colleagues
-
Working actively in a team on multiple parts of a solution at the same time
-
Being given a chunk of time and a starting point
-
Experience of working under time pressure and having to streamline their plans accordingly
They also reported that the event was well timed, presented and coordinated, appreciating the effort put into arranging the materials and use cases. We also received feedback that more build time, larger teams, and more Fabric use cases would have been interesting – we've heard this and will take it on board for the next event.
I’d like to thank Ashley Warren and Mikey Robson for their dedication to the planning and organization of the event, from building the presentation content, to designing the use cases and helping to secure buy-in (and prizes!)
Thanks also to Ust Oldfield, Gavita Regunath, and Terry McCann for giving their time to judge the presentations and provide useful feedback and insights – and to Simon Whiteley and Ben Somerville-Roberts for their background assistance in providing direction and unblocking technical issues.
Lastly, thanks to everyone who came along – your dedication and collaborative spirit made the event a success, and I hope you learned something and had some fun along the way!
I hope this article has demonstrated the value of running Hackathon events, and inspired you to arrange your own.
For more insights into the AI-BI breakthrough, check out Beyond Dashboards: How Conversational AI is Transforming Analytics.
Interested in running your own AI-BI Hackathon? Let's discuss how Advancing Analytics can help you out - we're happy to share our experiences, and we're excited to see what your teams can create!
(Disclaimer: Some images in this article were generated using Microsoft Copilot.)
