Generative AI has quickly become a core pillar of the modern data landscape. Over the last few years, we’ve seen an extraordinary acceleration in how machines can process, understand, and generate unstructured data, particularly free text. What was once experimental is now firmly embedded in day-to-day enterprise workflows, driven by the need for efficiency, automation, and faster insight.
Databricks has leaned into this shift with real momentum. Generative AI on the Databricks platform is no longer a bolt-on capability; it’s increasingly treated as a first-class citizen alongside data engineering, analytics, and machine learning. While text-based use cases have led the way, transforming everything from email drafting to transcript classification and sentiment analysis, the platform’s multimodal capabilities have been maturing just as rapidly.
Multimodal models extend generative AI beyond text, enabling systems to reason across images, documents, and other data types. Databricks has supported multimodal interaction for some time, particularly for vision and text generation. However, one area that tends to receive less attention is multimodal embedding models. These models make it possible to convert images into vector representations that live in the same semantic space as text, allowing users to search and retrieve images using natural language queries.
This capability has become especially relevant with the widespread adoption of retrieval-augmented generation solutions. As organisations build vector databases of documents to support search and question answering, the ability to extend those same patterns to images and mixed media opens up an entirely new set of use cases. In this guide, we’ll demonstrate how to use multimodal embedding models in Databricks and how to integrate them into a practical "embed → retrieve → describe" workflow that fits neatly into a modern RAG or knowledge base solution.
Multimodal embeddings work by mapping different data types, such as text and images, into a single shared vector space. Each modality is processed by its own encoder, but the model is trained so that semantically similar concepts end up close together regardless of their original format. For example, the phrase “a dog wearing sunglasses” and an image showing that scene are positioned near one another in the embedding space.
Once you’ve embedded your content, retrieval becomes a geometry problem rather than a complex modelling task. You compare vectors using a distance metric such as cosine similarity, and the closest matches represent the most relevant results. This is the same foundation most teams already use for text-based RAG, and it translates extremely well to multimodal data when the embeddings share one space.
There are plenty of multimodal embedding options on the market, ranging from open-source models you can host yourself to managed APIs designed for enterprise scale. In this example, we focus on Cohere Embed v4, a multimodal embedding model designed to embed text, images, and mixed media documents into a single vector space without requiring complex preprocessing.
It’s particularly useful in enterprise retrieval scenarios where you want one consistent embedding approach across diverse content types (think: PDFs with images, scanned pages, slides, and mixed-layout documents), and where multilingual search and large-context support are important.
Cohere Embed v4 can be hosted in a few different places depending on your architecture preferences and operational constraints. Embed v4 is available on the Cohere Platform, AWS SageMaker, and Azure AI Foundry. In this guide, we focus on Azure AI Foundry because it integrates neatly into Azure estates and can be connected to Databricks Model Serving using an OpenAI-compatible interface.
If you want the official Cohere release notes and model updates, Cohere maintain a changelog here:
https://docs.cohere.com/changelog/embed-multimodal-v4
Before diving into the steps, it helps to frame the pattern end-to-end:
First, you embed images into vectors and store them somewhere searchable.
Next, you embed a user’s text query into the same vector space.
Then, you retrieve the most similar images using vector similarity search.
Finally, you pass the retrieved image (or top-k images) to a vision-capable model to generate a description, answer a question, or extract structured information.
This keeps retrieval fast and cheap, and reserves the more expensive generative reasoning for only the most relevant items.
High-Level Architecture: Embed → Retrieve → Describe
We begin by setting up the essentials in Databricks. You’ll want a small set of libraries to cover four jobs:
Image handling (to load, resize, and re-encode images)
Vector maths (to compare embeddings using cosine similarity)
Endpoint interaction (to query the embedding model)
Endpoint management (to deploy or update the serving endpoint cleanly)
A practical tip here is to keep your “setup” section tidy and consolidated. It makes the workflow easier to share across teams and reduces the chance of missing dependencies when you operationalise the approach later.
Next, we handle credential management. While it’s possible to configure external endpoints with plain-text secrets, it’s not a pattern you want to take into production. Instead, store your Azure AI Foundry API key in a Databricks secret scope and reference it securely when configuring the serving endpoint.
This gives you three big benefits:
No hard-coded keys in your workspace or shared assets
Cleaner rotation and revocation when credentials change
A clear separation between engineering logic and sensitive credentials
If you’re building anything beyond a personal proof of concept, this is the point where you set yourself up for safe reuse and team adoption.
Before Databricks can call Embed v4, you need a deployed endpoint in Azure AI Foundry.
We start in Azure AI Foundry by creating a model deployment for “embed-v-4-0” and waiting for provisioning to complete. Once deployed, Azure AI Foundry provides two critical pieces of information that you’ll use from Databricks: the Target URI and the Key. These are what Databricks Model Serving will use to authenticate and route embedding requests to the correct external deployment.
The image included in this section shows a completed deployment for “embed-v-4-0”:
With secrets in place, we deploy the serving endpoint. The key detail is that Azure AI Foundry exposes the Cohere model through an OpenAI-compatible interface. In practice, that means configuring Databricks Model Serving to treat the external model as an embeddings endpoint and supplying the Azure-specific connection settings (base URL, API version, and deployment name), plus a secure reference to your API key.
There are two “gotchas” worth calling out because they’re exactly the kinds of details that cause 404s or confusing errors if missed:
First, the API base should be the resource base URL without appending “/models”; including “/models” at the wrong point can lead to a 404.
Second, the openai_api_version value used here is not the same as the one that is often suggested in documentation examples. In practice, the version that consistently returned a valid response in this setup was “2024-06-01”, so that is the version used in this guide.
A robust deployment approach is to write the process so it behaves idempotently: if the endpoint exists, update it; if it doesn’t, create it. This is especially useful when you’re promoting the same pattern across environments (dev/test/prod) or integrating it into automated deployment pipelines.
Once the endpoint is live, we move on to image preparation. Model serving has payload limits, and images can grow substantially after Base64 encoding. To avoid request failures, resize images before embedding.
A sensible approach is:
Scale the longest edge down to a maximum size (for example, 1024 pixels)
Preserve aspect ratio to avoid distortion
Re-encode as JPEG with a reasonable quality setting to reduce bytes
Convert the result to a Base64 data URI (for example, data:image/jpeg;base64,…)
This format allows the image to be passed to the embeddings endpoint in a consistent way, alongside text inputs, which keeps your pipeline simple and uniform.
Now we generate the image embedding by calling the Databricks serving endpoint. Cohere Embed v4 returns a fixed-length numeric vector for each image. That vector becomes your searchable representation of the image.
For a quick demonstration, you can store vectors in memory alongside an image identifier and the encoded image reference. For real-world usage, you’ll typically write embeddings to a Delta table and then sync them into a Databricks Vector Search index. This gives you scalable approximate nearest neighbour search and allows you to query millions of vectors efficiently.
At this stage, you’ve done the “embed” part of the workflow: your image corpus is now vectorised and ready to search.
This is where multimodal embeddings shine. Because text and images are embedded into the same vector space, you can embed a user’s natural-language query and compare it directly to image vectors.
The flow looks like this:
Embed the text query using the same serving endpoint
Compute cosine similarity between the query vector and stored image vectors
Rank results by similarity score and select the top match (or top-k)
In production, the ranking step is where Databricks Vector Search becomes invaluable. Rather than looping over vectors yourself, you query the index and get fast, scalable retrieval. But conceptually, it’s the same operation: find the closest vectors in the shared semantic space.
Retrieval gets you the right image quickly. A vision-capable generative model helps you do something useful with it.
To complete the “embed → retrieve → describe” pattern, take the retrieved image and pass it to a vision model hosted in Databricks, asking it to describe what it sees.
This can be extended into practical enterprise tasks such as:
Generating captions for images in a knowledge base
Answering questions about diagrams, screenshots, or scanned forms
Extracting structured fields from visual documents
Summarising visual evidence for audit or compliance workflows
This is a powerful design because the generative model only sees the most relevant results, which helps control cost and latency while improving response quality.
If you’re taking this beyond a guide and into a real workload, a few practical considerations make a big difference:
Use Delta + Vector Search for persistence and scale
Store both the embedding and a stable reference to the original asset (path, ID, or URI)
Batch your embedding jobs for large corpora rather than embedding one image at a time
Design for payload limits by standardising image resizing early in the pipeline
Retrieve top-k and re-rank where needed (for better recall and quality)
Apply governance through Unity Catalog, especially if images or derived descriptions are sensitive
These steps turn the approach from “cool demo” into something you can safely adopt across teams and domains. If you’d like more detail on operationalising multimodal solutions in production, it’s worth looking at Databricks’ guidance on production architectures for multimodal data integration (the article titled “Multimodal Data Integration: Production Architectures for Healthcare AI”), which covers scalable patterns and design considerations that translate well beyond healthcare.
Multimodal embeddings are a natural next step if you’re already using Databricks for enterprise search or retrieval-augmented generation. They let you extend familiar vector search patterns beyond text into images and mixed media, opening up richer ways for users to find and interact with information.
The key takeaway is simple: once text and images share a vector space, cross-modal search becomes straightforward. Add Databricks Model Serving for clean deployment and Vector Search for scalable retrieval, and you have a strong foundation for multimodal knowledge experiences that feel intuitive to end users and remain maintainable for engineering teams.
Got a question or want to hear how we can build a Databricks platform for your use case? Get in touch today.