Over the long life of a business unit, you will eventually reach a point where your favorite system of record is no longer a good fit for your needs. Perhaps you are undergoing a portfolio rationalization effort and have identified a more suitable in-house system to consolidate your workloads. Alternatively, you might be dealing with a system featuring an older architecture or technical debt that needs a fresh start and new implementation to meet your operational and evolving business model goals.
While everyone is likely enthusiastic about transitioning to a better and more modern platform, the moment someone voices the question, “What do we do with all the data in the current system?” things that initially appear straightforward can quickly transform into a problem almost as intricate as the construction of the new or intended system.
Below, I’ll lay out the three basic approaches and the pros and cons of each. Like most thorny design problems, there is no “one-size-fits-all” approach that emerges as a clear winner. You must spend some time thinking about the purpose of the historical data. On one end of the spectrum, you may have long-running contract or account information that requires primary attention in the new system. On the other end, you might only have older transactional information preserved to meet audit and compliance needs, where there is flexibility in response times to inquiries.
Choosing among the different approaches to handling historic data is an art of balancing nonfunctional requirements like access time and ongoing operational expenses against the time and effort of system cutover.
As mentioned earlier, the decision is often influenced by the nature of the data and how frequently it will need to be accessed in the future. In some cases, leaving data in the legacy system until it reaches a final state is feasible, but there are situations where someone will consistently need access to historical data within the context of the new system.
Maintaining the legacy system to handle inquiries on old data is an easy approach from a project perspective but puts a heavy burden on operations.
The lure of avoiding lengthy discussions about mapping, transforming, and validating data migration into a system that models and represents key entities differently can be strong. However, retaining the old system requires allocating resources to its ongoing operation (and possible licensing), along with the necessity for well-documented playbooks and runbooks to aid in understanding the system over time. The old system will also likely need periodic patching for software vulnerabilities and possibly even component upgrades to stay current with vendor support requirements.
Nevertheless, this is a fairly common method when executing wave migration of cohorts, allowing old items to naturally phase out in the legacy system instead of migrating them to the new system.
If you are contemplating this option, consider the support and operational costs of running two systems in parallel throughout the data's lifecycle, and have a plan for addressing reporting needs for KPIs and year over year values (hint: a reporting mart or data warehouse can be an excellent solution for merging old and new system data).
Depending on the nature of your system and data, a data lake or warehouse might be sufficient to meet your historical data needs. If so, consider yourself fortunate!
In some cases, your data lake may take you most of the way there, and you just need to spend some analysis and effort to create customized extracts or reports that can be referenced in the future for historical needs.
This is a good, low-expense approach to addressing your historical data, but it’s a little risky in terms of making sure that you anticipate all your future needs and meet them with the extract. Additionally, complications may arise if you need to secure subsets of your data based on intricate roles and rules.
One last caveat is that if you store data in a data lake or warehouse, you will likely need technicians (rather than businesspeople) involved in retrieving and reporting historical data. This often means longer and more complex retrieval times compared to direct user-level access through an application.
If you need fast and transparent access to all your historical data in the new system, migrating it from the legacy system is necessary. Unfortunately, this is usually a substantial effort that will impact your cutover project timelines and take some detail-oriented attention to get right.
It’s a wonderful solution once it’s done, but the new system will likely have a different “world view” about key entities, their relationships, and possibly even detailed formula calculations that require careful thought and design to get right. Data set alignment at the beginning (potentially an archival date) and end (cutover point) also might take more thought than you would naively think.
And then, once you think you have the migration correct, validation is still a labor-intensive task. The new system's model and formula calculation rules will often make you spend a considerable amount of time and project money validating the migration's success.
If you must migrate a large set of historical data into a new system, take the problem seriously. It will be a significant effort that requires challenging, detail-oriented to get right. Still, it’s a great way to ensure the new system meets your needs, and the presence of data familiar to the users makes the training experience easier to plan and execute.
The approach you choose to manage your historical data needs will be driven by your data access needs, project timeline, budget, and tolerance for ongoing operational costs. It's unfortunate that dealing with historical data is complex, but the root issue lies in the fact that transitioning to a "better" system requires some changes. If the new system's model exactly mirrors the old system, questioning the need to move may be warranted.
While a reporting ecosystem, like a data warehouse or data lake, can alleviate some of the burden of data migration by merging the view between old and new systems, there are instances where you must migrate and transform your data into the new system for the high-quality experience users and customers expect.
A partner with deep expertise in connecting the data journey can assist in choosing the right approach for your business. Contact Concord if you’re looking to gain more confidence in your data infrastructure and devise a roadmap to modernize and optimize your data ecosystem.
Not sure on your next step? We'd love to hear about your business challenges. No pitch. No strings attached.
