Projects we have delivered

The challenge

A large estate of Pentaho ETL jobs and reports had grown over the years. It was hard to maintain, hard to scale and tied to a tool the team wanted to retire.

What we did

• Catalogued every job and report, and grouped them by pattern so common logic could be converted once and reused.
• Re-built the transformations in Python and PySpark, with AI-assisted conversion and engineer review.
• Moved scheduling, dependencies and restarts to Apache Airflow.
• Compared old and new outputs job by job before each cut-over.

The outcome

The 600+ ETL jobs and 400+ reports now run as Python / PySpark code under Airflow, without the Pentaho runtime.

The challenge

Business-critical ETL ran as Ab Initio graphs. The licences were costly and the skills were scarce, and the team wanted open, cloud-ready code.

What we did

• Analysed the graphs and mapped their components (reformat, join, rollup, lookups) to PySpark equivalents.
• Converted graphs to Python and PySpark, keeping the business logic identical.
• Re-created graph scheduling and execution as Apache Airflow workflows.
• Validated row counts and values against the original outputs.

The outcome

The Ab Initio graphs run as version-controlled Python and PySpark code, scheduled and executed by Airflow.

The challenge

Reporting and data preparation depended on SAS programs, which kept licensing cost and a niche skill set in the loop.

What we did

• Reviewed each SAS program and documented its inputs, logic and outputs.
• Re-wrote the logic as SQL stored procedures in the target database.
• Reconciled results with the SAS output before switching over.

The outcome

The SAS programs were replaced by SQL stored procedures that the database team can run and maintain.