I have been spending a lot of time in Databricks lately, and one question keeps coming up as I think through migration design: what exactly are we moving?
The obvious inventory includes data, pipelines, jobs, tables, dashboards, and the applications around them. The less visible inventory is access, and that is where I think a migration can become more complicated than it first appears.
Legacy environments accumulate permissions slowly. A database role is created for a project, someone adds access to an S3 bucket during an incident, a service account stays active because a job may still depend on it, or a group gets broader permissions because separating them would have taken more time. Most of those decisions probably made sense when they were made, but years later they have become part of the architecture whether anyone intended that or not.
Before moving a workload, I want to understand who can reach the data today, which identity they use, and which path gives them access. Consider a sensitive record that moves through an operational database, object storage, a warehouse, notebooks, and an application. The database may use roles, S3 has IAM policies, the warehouse has grants, and the application has a service identity. Someone may also have direct access to the underlying files.
When the same data can be reached through several systems, the answer to “who can access this?” stops being a single permission check. Several teams may give different answers, and each of them can be technically correct within the system they manage.
That is one of the reasons Unity Catalog is interesting to me. It gives Databricks a common governance layer for the data and AI assets it governs, which can reduce some of the fragmentation that develops when each part of the stack has its own control model. The part I would be careful not to overstate is what happens simply because that governance layer now exists.
Putting the new environment behind Unity Catalog does not make the old access decisions disappear. If every existing group, entitlement, service identity, and exception is recreated without questioning why it exists, the organization may end up with a more centralized version of the same access model rather than a cleaner one.
I would rather use the migration as a reason to revisit those decisions. If a service account is still active, which workload depends on it? If a group has broad access, does that access still reflect someone's job? If a permission was meant to be temporary, what kept it from being removed?
Storage deserves particular attention because the governed path is not always the only path. Unity Catalog can govern access through Databricks, while a principal with separate cloud-level permission to the underlying storage may still be able to reach the same data another way. A table can therefore have the permissions you intended inside the platform while the files underneath it remain accessible through an older route.
That changes what “finished” means in a migration. Getting the workload running in the new environment proves that the new path works, but it does not tell me whether an unnecessary legacy path is still available. The retirement of old access has to be part of the design rather than something left for cleanup after the migration is declared complete.
The more I work through that problem, the more I think of it as identity debt. We usually talk about technical debt in terms of code and architecture, but access decisions age too. People move between teams, systems change owners, projects end, temporary permissions remain, and service accounts survive longer than the applications that originally needed them. Eventually the permission is still there while the reason behind it becomes difficult to reconstruct.
A migration brings that debt back into view, although dealing with it does not always look like progress. Moving another hundred tables is easy to count. Reviewing groups, tracing application identities, finding owners, identifying sensitive data, and removing permissions that no longer have a purpose takes time without producing the same kind of visible milestone.
I still think that work belongs early in the migration because it affects the target architecture. Rather than beginning with everything available, I would start with one meaningful domain that has real consumers, real service identities, meaningful permissions, and enough sensitivity that the access decisions matter. A pilot that avoids those conditions may prove that the technology works without teaching much about how the organization will actually operate it.
For that domain, I would map the current path before designing the target one: where the data originates, where it is copied, which identities can reach it, what transforms it, what consumes it, and whether there is another route around the control we intend to enforce. That gives us enough context to decide what should be recreated, what should become narrower, what may need to remain federated during the transition, and what should not be migrated at all.
The transition itself also deserves deliberate design because, for some period of time, the legacy environment and the new environment may both be operating. Jobs are being validated, consumers have not all moved, and credentials may still exist on both sides. If retirement is deferred until everything else is finished, it becomes easy to improve governance inside Databricks while leaving old access intact somewhere else.
There is also a broader tradeoff that matters here. Governance alone is not a reason to move everything to Databricks.
If an organization already has predictable workloads, clear ownership, a defensible access model, and an evidence trail it can explain, the cost and operating change of a full replatforming may not be justified. The argument becomes stronger when fragmentation itself has become an engineering problem: access reviews require reconciling IAM policies, warehouse grants, storage permissions, service identities, and old tickets just to understand who can reach a particular dataset.
In that situation, consolidation is solving an operational problem rather than simply introducing a platform with a longer feature list.
That is the distinction I keep coming back to when I think about migration design. I want to be able to ask who can access a piece of data before the migration and ask the same question afterward, with the expectation that the second answer requires less reconstruction because ownership is clearer, unnecessary paths are gone, and the evidence behind an access decision is easier to find.
If moving the workloads leaves us rebuilding the same access story across the same disconnected systems, then we changed where the workloads run without addressing the problem that made the environment difficult to govern in the first place.
For that reason, I would not treat governance as a configuration phase that comes after the migration architecture is finished. It is part of deciding what the new architecture is worth carrying forward, because data and workloads may need to move, but old permissions should not come with them simply because they were already there.