Our client was experiencing long run times on their data pipeline jobs due to the volume of data being processed. The pipeline often took over an hour to run and would fail unexpectedly, resulting in downstream reporting impact.
Concord reviewed the existing pipeline and broke it down into parts to identify the troublesome areas. We learned the pipeline slowed down because it was processing historical data every time the pipeline ran.
To reduce pipeline strain, our team developed and deployed a static historical table that the pipeline could reference that only processed current year data.
As a result:
