The system that taught me the most was the one nobody could fully explain to me.
It was an identity pipeline in a federal environment. Six source systems fed it. The logic was spread across roughly ten repositories in a mix of languages. Changes replicated into an Oracle staging layer, batch jobs pulled the data into Spark clusters that existed only for the length of a run, and the results landed in a search index that people downstream used to answer identity questions. Orchestration lived in Jenkins. My job, in my first thirty days, was to reverse engineer how it actually worked.
The code was the fast part. Understanding the system took the full thirty days, and the gap between those two things changed how I think about migration.
A pipeline that has run for years is never just its transformation code. It is three layers: the visible logic, the operational behavior, and the downstream expectations. Only the first layer is written down.
Miss the other two and you can reproduce the code perfectly while replacing the system with something different. That is how a migration can look successful while changing the behavior people depended on.
The first layer is the visible logic.
This is what the repositories say: the filters, the joins, the matching rules, the writes. In the pipeline I inherited, this layer looked intimidating at first (ten repositories is a lot of surface area), but code can be read. If migration were only this layer, it would be a translation exercise, and translation is the part modern platforms are genuinely good at.
The second layer is the operational behavior.
This one rarely lives in the code. In my pipeline, it lived in the orchestration. The Jenkins configuration knew things no repository mentioned: which jobs had to finish before others could start, what happened on a retry, what the schedule quietly assumed about when each source would be ready. The compute was ephemeral, so anything the pipeline needed to remember between runs had to live outside the clusters. The real story of a run was scattered across systems that would never show up in a code review.
Even the deployment pattern carried behavior. Releases ran blue-green, and blue-green encodes a quiet assumption: the output can be rebuilt from upstream and swapped as a unit. Rebuild and replay were terms of the contract. Nobody had written the contract down.
The third layer is the downstream expectations.
This is the hardest layer to see, because it lives in other people's systems and habits. Someone expects the data by a certain hour. Someone's month-end process assumes recent records can still change. Someone depends on a field the migration plan has already marked obsolete. I learned this layer the direct way. Some of the most urgent work I did in federal systems started downstream, with a stakeholder asking why a record was wrong, and the answer sitting upstream in behavior nobody had documented. Expectations do not disappear because the new architecture is cleaner. They just lose their protection.
So I no longer describe migration as moving logic. A migration is contract recovery. The legacy system is the only complete record of the agreement between the data, the operations, and the people who depend on both, and most of that agreement was never written anywhere else.
Incremental processing is where this bites first. A legacy job reprocesses a window of recent days because someone, years ago, learned that records arrive late and out of order. The replacement processes only the newest partition because that is cheaper and easier to reason about. On a complete day, both produce the same numbers. Then a source runs late. The old job would have caught the stragglers on its next pass. The new one completes, the dashboard refreshes, and nothing fails, which is exactly the problem. The gap stays invisible until someone compares the output against another system. The row counts matched the whole time. The behavior did not.
This is why validation has to test conditions, not just outputs. Two pipelines can agree on complete data and disagree the moment a source is late, duplicated, or missing. The meaningful question is not whether the outputs match today. It is whether the new system behaves the same way under the conditions that made the old design necessary.
And every behavior you surface hands you a real decision: preserve it, replace it, or remove it on purpose. Preserve everything and you carry old weaknesses into a new platform. Remove what you do not understand and you create failures with no history to explain them. Replace it and you need agreement on what the system should do instead, which means finding the people whose expectations shaped the behavior in the first place. That series of decisions is the migration. The code translation is just the part that fits neatly in a sprint plan.
Before I would call a migrated pipeline done, I want its operational contract on one page: the inputs it expects and when, how it decides a dataset is complete, its tolerance for late and duplicate data, which historical records may still change and for how long, what a safe replay looks like, and the delivery time downstream teams can hold it to.
Writing that page has a cost. It forces disagreements into the open, usually with people who thought the question was settled years ago. That is the point. It is cheaper to find the disagreement before cutover than to find it in a reconciliation meeting after.
These are the questions I carried out of that identity pipeline, and they are the ones I am carrying into Databricks now as I map the platform against systems I have actually run. The tooling changes. The contract problem does not.
The expensive migration failures are rarely the loud ones on cutover day. Some of the harder ones are the quiet failures, where every run goes green while an unwritten agreement goes unhonored, and the first person to notice is standing downstream, holding numbers that no longer match.
A team can reproduce every line of the code and still lose the system, because the system was never only the code. It was the contract, and the contract was hidden in the pipeline.