Vertical squiggly line
Man working on computer.
Data Solutions & Analytics

Reducing Pipeline Strain

Project Overview
Project Overview

Our client was experiencing long run times on their data pipeline jobs due to the volume of data being processed. The pipeline often took over an hour to run and would fail unexpectedly, resulting in downstream reporting impact.

Results area

Concord reviewed the existing pipeline and broke it down into parts to identify the troublesome areas. We learned the pipeline slowed down because it was processing historical data every time the pipeline ran.


To reduce pipeline strain, our team developed and deployed a static historical table that the pipeline could reference that only processed current year data.


As a result:


  • The client saw a 35% improvement in the speed of processing, reducing the time from 110 minutes to 71 minutes. 
  • The data source became reliable again and downstream reporting experienced minimal disruption in it’s daily refresh.
  • Analysts were trained on optimization tactics in pipeline development to ensure pipelines are developed to engineering best practices going forward.

Similar Projects

Learn more
Learn more
Learn more