A well-built Gold layer should be the first place we look for an answer. Agreed definitions, reliable data and business logic we can reuse. Lovely jubbly, let’s use it!
That foundation might support an external semantic model powering a collection of reports, an AI/BI Dashboard or a Genie Agent answering business questions. It also provides trusted context for Genie One to find its way around the business.
But what happens when the question goes beyond what we’ve already built? …Or someone needs a new report by Thursday!
I know that person. I’ve been that person.
My Journey Started Downstream
I started as a consumer of data, building reporting and answering questions from the business. The existing models didn’t always contain what I needed, and engineering had a busy workload.
My urgent request joined a queue of other urgent requests and seemingly endless migration projects. It was becoming as routine as Finance’s “month-end”. So, I found ways to keep moving: combining datasets, adding transformations and building logic closer to where I used the data. (Data democratisation at its finest, within the Data Team!)
That got the report out. Useful business logic is now buried inside an external semantic model, away from the engineers who might eventually need to support it.
We don’t all work in neatly aligned tribes, with sprints that match the business priorities of the day. A question arrives on Wednesday that nobody mentioned during planning on Monday. Apparently, the business hasn’t read the sprint calendar…and why should it?
Sometimes the dashboards can’t answer it. Neither can the existing Genie experience. Sometimes I don’t even know the final requirement yet. The investigation is how I discover it.
What I needed was somewhere to do that work, with a realistic route for the valuable parts to become something shared.
Gold First. Then Find the Gap.
My starting point now would be simple: what does Gold already give us?
Before creating another model or measure, look for an existing data product. Check its definitions, grain and shared dimensions. If it answers the question, use it.
I don’t want every urgent request to create another interpretation of revenue.
Where something is missing, identify the actual gap. Does the data already exist but need organising differently for a report, dashboard or semantic model? Do we need to investigate a new calculation? Or have we uncovered a missing business definition that several teams could use?
Understanding that gap helps us decide what to build and where it belongs.
Databricks’ own data warehousing guidance describes Gold as the presentation layer, including dimensional models and data marts. It also recognises departmental sandboxes for exploration.
That is the foundation I’m building on. Gold first, with room to investigate and a supported route to improve it.
So, where do analysts get to plant?
In Simon Whiteley’s blog on data curation and Genie Ontology, he describes a walled garden, a maintained park and a more exploratory woodland.
Different levels of curation and authority, with useful knowledge across all three.
I like that because it feels recognisable. Some definitions are carefully maintained. Some serve a particular team. Others are still being worked out.
My question is: where would someone like me have been allowed to plant?
Unity Catalog provides much of the foundation: governed access, ownership, descriptions and lineage. Analysts also need suitable development environments in which to investigate, and a clear destination for the work the business decides to rely on.
Being allowed to look around the garden is useful. Occasionally, I’d like to pick up a trowel, put a Bake Off tent on the lawn and turn some of those ingredients into something worthy of a handshake. 🫱
A Presentation Layer Can Start Thin
Where Gold already meets the consumer’s needs, keep things simple.
Sometimes those shared Gold products need shaping for a particular tool or audience. This is where a downstream presentation, or platinum, layer can help: a set of views, serving tables or semantic definitions built on Gold.
Platinum is the architectural label I’m using here, rather than an additional tier every Databricks implementation needs.
It might begin as a one-to-one semantic model over existing Gold tables, preserving their grain and agreed logic while exposing the relationships, names and measures consumers need.
It can be a logical interface without copying the data between systems. There’s no prize for adding a transformation nobody asked for.
That thin interface can form part of an enterprise data product. Its authority comes from business agreement, accountable ownership, validation and support. Descriptions, governed tags and certification (where supported in Unity Catalog) help communicate that standing.
Why Notebooks
Personally, I’ve found notebooks to be one of the most rewarding ways to build your presentation layer. I can explain the business rules in Markdown, work through the transformations in manageable steps, and keep the reasoning alongside the code. A well-maintained table of contents is also something that makes me ecstatic to see populated!
The specification develops alongside the solution. Engineers can see what I’ve done, why I’ve done it and which assumptions still need challenging. Once reviewed, the logic can populate published tables, with Lakeflow Jobs handling orchestration.
That makes “What are you doing with the Gold data?” a much easier conversation. It also gives Engineering something concrete to assess and prioritise for promotion into Gold.
Simple can still be authoritative.
The dependency matters: the presentation layer consumes Gold; Gold remains independent of its downstream consumers.
If analysts discover a broadly useful definition, the team can review and implement that logic in Gold. Downstream products then consume it from there… you can see how the engineers’ lives are about to get easier in terms of scope. ✅
Publish a Model People Can Depend On
There’s value in publishing a complete, coherent model here: the facts, dimensions and relationships needed by a domain or use case.
A thin layer can still expose a whole model. It simply reuses as much of Gold as needed.
That gives downstream consumers a dependable set of assets, without each report author having to assemble their own collection of tables from across the estate.
If another source is introduced, its identifiers need reconciling with the existing model. That might mean mapping records to shared dimensions or generating surrogate keys for new members. Keys from separate systems aren’t automatically interchangeable, and any remapping must stay consistent across facts and dimensions. Existing keys should come from gold if you’re not changing the grain or introducing additional sources.
For external tools, a coherent model can simplify connection management too. Where the connector supports it, consumers can authenticate against one published source and use parameters to select the appropriate environment, endpoint or catalog.
That’s easier to maintain than a pick ’n’ mix of connections and credentials in every downstream model. Using parameters for them too makes it much easier to switch environments...even more so when you get a release pipeline to handle that automatically.
It also gives engineers a clearer view of what is being consumed. These are the published assets, these are their consumers, and these are the dependencies a change could affect.
Everything in that published model should have a known purpose and consumer. If it stops being used, review it for retirement. We’re publishing a useful product, not opening another cupboard for things we might need one day. My cable drawer follows a different retention policy. Apparently, that old printer cable might still be useful!
Two Routes to the Answer
The same Gold foundations with a Presentation Layer on the top can support two routes:
- External semantic models, and
- Databricks Analytics with Genie Agents. (AI/BI Dashboards, Genie Agents, Genie One)
External Semantic Models
External models often need clear facts and dimensions, predictable relationships and documented business rules. The presentation interface gives them something dependable to consume.
Databricks SQL warehouses can serve published tables and views. Unity Catalog metric views provide shared measures for compatible consumers, while Delta Sharing offers a route for distributing supported assets to other platforms.
A modelling side note: some estates expose Data Vault structures through what they call Gold, although Databricks generally positions Data Vault modelling in Silver. Those structures may need reshaping for BI consumption. A dimensional interface with integer surrogate keys can suit some BI engines better than exposing wide hash keys directly. Keep the mappings traceable and test the actual workload.
The consumer’s needs should influence the interface we publish.
Ossie?
Databricks is contributing to Open Semantic Interchange, now Apache Ossie, which aims to make semantic definitions portable across platforms.
For me, the appeal is simple: agree a definition, govern it properly and reuse it in external semantic models without rebuilding it every time. Some transformations belong upstream; some business meaning can travel through semantic exchange.
It’s an evolving standard, so conversion still needs validation. But it supports the direction I want: more reuse of agreed definitions, fewer opportunities to reinvent revenue.
….watch this space.
Databricks Analytics and Genie Agents
The governed presentation layer can serve AI/BI Dashboards, metric views and Genie Agents, while supported assets and business context contribute to Genie One through Genie Ontology. An analyst’s understanding becomes reusable knowledge, without needing the same explanation every Thursday.
This builds on Databricks’ guidance on operationalising Genie Ontology: strong Gold foundations, deliberate semantics and curated context.
A presentation product can be authoritative for its intended purpose without moving into Gold. Ownership, definitions, scope and approval status make that authority clear. OntoRank uses signals including certification, usage and authorship to help rank context; of course, we should still test Genie’s answers and citations against the approved definitions
A portfolio metric might be right for one team while an enterprise metric governs company-wide reporting. Genie needs to know which applies. Knowledge held in external semantic models needs an explicit integration route to become discoverable too, but if we’re keeping our logic in Databricks, Genie should be able to retrieve these answers without needing an integration. Ossie could help here too!
Gold provides the shared foundations. The governed presentation layer makes them useful to people and agents even further.
The Transformations Are Already Happening
There’s a slightly awkward reality here: downstream BI tools may already be doing transformations that only the analyst knows about. How familiar does that sound? Joins, mappings, exclusions and calculations quietly keeping the reporting running.
That work exists whether the platform team can see it or not.
So, do we leave it there, or give the analyst a supported way to do the same work further upstream, where others can find, review and reuse it?
I’d rather bring it into view.
Start with Gold, give analysts an appropriately permissioned development area in Databricks, and let them build on those foundations. Unity Catalog can govern access to the resulting data assets and capture supported lineage. Its External Lineage feature can also extend that view into downstream tools, with external assets and their relationships registered through the UI or APIs. That helps show what depends on what, although it doesn’t automatically uncover all the transformation logic hiding inside a semantic model, but it’s certainly a start.
Source control makes the transformation logic visible and helps teams stay consistent in their approaches. Databricks can then process and serve the approved outputs.
Here, moving upstream can mean moving out of an external tool and into the presentation layer in Databricks. Shared definitions can move further upstream into Gold once reviewed.
That visibility gives engineering a chance to spot expensive calculations, repeated joins and business rules worth sharing. Where appropriate, we can prepare data once instead of asking every downstream model to repeat the work.
Moving a query doesn’t automatically make it efficient, but it does give us a better chance of improving it. And if I were a betting man, I’d back Databricks SQL, Photon, predictive I/O and well-organised tables to give those downstream transformations a run for their money, with Lakehouse RT another option for suitable low-latency workloads. I’d still check the stopwatch before collecting my winnings.
Give Engineering a Head Start
By the time an investigation is ready to become a supported product, the analyst may have explored the data, found awkward exceptions and built a working example. Ideally, they have agreed expected results with the business, and have had a report or dashboard in front of them too.
That is a substantial head start, and to them, you’re a rockstar to the business.
You’ve done your testing, you’ve got peer(s) to review, so you’re happy with the numbers… but let’s face it, too much of this is going to create a technical backlog, and the Governance Police will be on your doorstep. But keeping engineering in the loop will help to ensure that you’re keeping the gold coverage as high as possible.
Engineers can focus on performance, security, reliability and maintainability. The analyst stays involved to check that the business context and value survive.
The rapid deployment work becomes part of the finished product, rather than something the next person has to redo from scratch. It also ensures that the data people in your organisation are keeping up with the business requirements.
Grow the Analyst Too
This way of working taught me a lot about modelling.
Building something for a real business question forces you to understand grain, facts, dimensions and relationships. You find out why a join doubles the result. Usually, shortly after being pleased that it worked.
A supported development area gives analysts somewhere to learn dimensional modelling with real data and real feedback. Engineering review can turn a working calculation into a conversation about reusable dimensions, historical changes and sensible keys.
That learning benefits the next investigation. Requirements become clearer, prototypes improve, and analysts recognise more of what already exists in Gold.
I’d like us to make room for that progression. It’s difficult to develop modelling skills if your only permitted contribution is filling in a request form.
Govern the Release, Welcome the Contribution
Definitions and deployment configuration can live in source control, with peer review, testing and CI/CD governing what reaches production. Unity Catalog permissions, alongside workspace controls, should restrict direct changes that could bypass that process.
That applies to a thin interface over Gold just as much as a more involved transformation. Once released for supported business use, these are production products. Sitting downstream of Gold doesn’t make them any less deserving of the platform’s release controls and protection.
The IKEA Effect
The IKEA effect describes how people can value something more because they helped build it. I’ve always liked co-building for much the same reason; people have a stake in the result.
The governed presentation layer gives that collaboration a home. Analysts bring the business understanding and working logic. Engineers help make it reliable, secure and maintainable. Together, they release something the business can use.
If I can recognise my contribution in the finished product, I’m more likely to care when it breaks, question when it changes and help keep it useful. That’s the ownership I want to encourage.
Of course, “I built it” doesn’t make it right. Someone still needs to check the legs are on properly.
And “it works” shouldn’t end the conversation either. Use lineage, usage and regular reviews to spot duplicated definitions, unused assets and shared logic ready to move into Gold. Useful shaping can stay in presentation; repeated enterprise rules deserve an upstream conversation.
Build it together, look after it together, and keep improving the foundations it stands on.
Finishing Up
I started this wanting a report by Thursday. What I really needed was a way to deliver it that left something useful behind for everyone else.
That’s what a governed presentation layer offers. Start with Gold, reuse what’s trusted and give analysts a supported place to fill the gaps. Engineers get visible logic, clearer requirements and a working example. The business gets products it can use through external semantic models, dashboards and Genie, with ownership and release controls attached.
Some logic belongs in presentation. Some proves useful enough to move into Gold. Building and reviewing it together helps us make that decision and gives everyone a reason to care about what happens next. There’s your IKEA effect.
So, give analysts somewhere to plant, keep engineers involved and make tending the garden part of the job. We can answer today’s question and leave the platform better prepared for tomorrows.
If this sounds familiar, at Advancing Analytics, we help teams put these approaches into practice, from dimensional modelling and governed data products to release processes and Genie. Get in touch if you’d like help making it work in your estate. Bring us the awkward reporting requirement. We quite like those.
And if you haven’t already, take a look at Simon Whiteley’s piece on data curation and Genie Ontology, which planted the seed for this one.
