loader

Postgres to Databricks Lakebase: find the risks before you migrate

Prefer to watch this in video form? Check out the YouTube version.

Introduction

 Moving an existing PostgreSQL database can sound straightforward. Databricks Lakebase is built on Postgres, is fully managed and sits within the Databricks platform.

But an operational database migration is not just a data-copying exercise. You also need to understand how the application uses the database, which PostgreSQL features it depends on and whether those behaviours will work as expected in the target environment.

Finding those differences during cut-over is too late. That is why we built Lakebase Wizard.

Lakebase Wizard assesses an existing PostgreSQL environment and highlights potential compatibility issues before migration work begins. 

What Lakebase makes possible

 Lakebase brings managed operational Postgres into the Databricks platform. It can support low-latency transactional applications, serve lakehouse data to applications and provide a state store for AI agents. It also supports autoscaling, scale-to-zero and isolated database branches for development and testing. 

 For teams already running PostgreSQL, that raises an obvious question: 

Can we move this workload to Lakebase?

Often, the answer may be yes. But the more useful question is:

What do we need to understand before deciding?

Where PostgreSQL migrations get caught out

 Lakebase is compatible with PostgreSQL, but it is still a managed service with its own supported features and operating model. An existing database may rely on behaviours that need to be adapted. 

Areas to assess include:

  • Extensions:  Lakebase supports a defined set of PostgreSQL extensions. Any extension used by the source database needs to be checked against the supported list for the target Postgres version. 
  • Session state:  Scale-to-zero can close idle connections. Session-level state, including temporary tables, prepared statements, advisory locks, and LISTEN and NOTIFY commands, may therefore be lost.
  • Configuration: Not every database parameter can be changed at instance level in a managed Lakebase environment.
  • Authentication and access: Existing users, credentials and application connection methods need to be mapped to the chosen Lakebase authentication model.
  • Application assumptions: Connection handling, retries, pooling and database-specific logic all need to be tested against the target environment.
  • Operational requirements: Teams should check how monitoring, performance, recovery and support requirements will work after migration.

These issues do not necessarily prevent a migration. They are decisions that should be surfaced before the production move is planned.

What Lakebase Wizard does

Lakebase Wizard profiles the source workload and produces a compatibility assessment.

The purpose is not to label the whole database as simply compatible or incompatible. It is to distinguish between:

  • components that can move without significant change
  • components that require remediation or further testing
  • dependencies that may need a different design
  • issues that could make the current workload unsuitable for migration

This gives teams a more useful basis for a go, partial-go or not-yet decision.

 

Five phases: discover, decide, rehearse, cut over, validate

Our migration approach follows five practical phases.

Discover

Inventory the source database, its extensions, configuration, authentication model and application dependencies. 

Decide

Identify what can move directly, what needs to change and whether any part of the workload should remain outside Lakebase.

Rehearse

Test the proposed changes against an isolated Lakebase environment. Lakebase branches provide a useful way to test, reset and repeat without modifying the production branch.

Cut over

Move the production workload using an agreed migration, rollback and communications plan.

Validate

Check data parity, application behaviour, security and operational performance before the migration is considered complete.

Lakebase Wizard supports the early assessment needed to make these later stages more predictable. 

Make the call early

The value of Lakebase Wizard is not that it makes every PostgreSQL migration effortless. It makes the likely work visible while there is still time to plan for it.

If you are considering moving an existing PostgreSQL workload to Lakebase, start by assessing the real source environment. Identify the gaps, understand the compromises and make a clear go, partial-go or not-yet decision before cut-over is on the calendar.

That is a much better time to find out what might break.

 

Assess your PostgreSQL workload for Lakebase

We can help you identify compatibility risks and define a practical route forward before migration work begins.

Talk to Advancing Analytics

 

 

Simon Whiteley

Author

Simon Whiteley

CTO and co-founder of Advancing Analytics. Simon is a Microsoft Data Platform MVP & Databricks MVP, keynote speaker and data geek passionate about helping companies get the most out of their data.