Loading content

Please standby, while we are retrieving your information

Audit Balance Control and Reconciliation Service

Brings more transparency, control, and monitoring to your data & AI pipelines

ABCR services

Address data pipeline orchestration requirements of cloud data & AI modernization solutions. The orchestration service leverages Airflow to use schedule, monitor, and control the flow of data pipelines. All actions on cloud data and AI platforms are captured in metadata.
AUDIT AND CONTROL & Balance and reconciliation

This framework is an important part of any complex data management pipeline.At a high level, the ABCR framework provides an 'easy to use' mechanism that allows applications to interact with the data management metadata such that they can focus on ABCR, rather than on specific database or configuration file implementations. It provides the ability to store metadata related to various activities that encompass data management and helps to monitor and update various data management tasks, like ingestion, validation, and more. Ability to identify execution failures, followed by notification and execution of failed jobs as needed. It also enables auditing executions at each phase to keep track of changes data has gone through. Reconciliation service helps to compare data that rests in Data Lake or data between source and target (on-premise, other clouds, databases, file servers, etc.).

Idea migration


ABCR stands for Audit, Balance, Control and Reconciliation.

As name suggests this module works around all other modules and helps in auditing, balancing, controlling and reconciliating various activities and jobs performed under IDEA.

Let’s look at different services offered by this function

  • ABCR_A is for Audit
  • ABCR_B_R is for Balance and Reconciliation
  • ABCR_C is for Control

  • ABCR Features

    A brief summary






    The most appropriate definition of 'audit' as applicable to IDEA is 'a systematic review or assessment of something'. From an IDEA perspective, In most cases, this will translate into having a complete picture of various jobs executed under the IDEA umbrella. Under ABCR, audit information can be retrieved from two parts - the job run table and the application log.

    It enables user to add significant event entries to job run table which can be recorded and also allows to review the status of each event. Also additional information about the job success / failure can also be recorded.

    The amount of information logged by each application depends on the application as well as the logging level used by the application.


    The term (Data) Reconciliation is typically used to describe a verification phase during data migration, where the target data is compared with the original source data, in order to ensure that the data has been transferred correctly.

    Reconciliation is a special type of job under ABCR, that can be used to validate data between any two data stores. It enables user to select and compare data from any of the supported data stores.

    Entity based comparison allows the user to select a table from the source and compare them with same set of table(s) in the destination. This method allows the user to compare one of more tables at the same time.

    Query based comparison allows the user to formulate a query on the source and the target and compare the results.


    Job management is related to the situation where jobs are represented on an individual basis..
    It covers two parts

    It is related to the situation where jobs are represented on an individual basis.
    • Helps to manage and execute jobs on individual basis.
    • Supports on-demand job execution and scheduled job execution.

    For execution and monitoring the individual jobs, there is job management. For orchestrating one or more jobs as a group / batch, there is workflow management which supports

    • Database/ Dataset level mapping
    • Table level mapping
    • View level mapping
    • Column level mapping


    A workflow enables user to define a pipeline of seemingly independent tasks. Though many of the features in the modernization / migration project are developed as independent, modular pieces of functionality, to achieve the end goal, this service performs one or more of these tasks in a particular sequence.

    In other words, A workflow allows user to string together multiple tasks and execute them in a way that helps to achieve end goal of data processing.it enables user to define simple and/or complex relationships between various tasks. If the workflow is parameterized, it can be reused for multiple executions, for different operations.

    Design for Industrialization

    Service Benefits

    More Transparency to Data

    The ability to balance and reconcile data helps to trust the data more as it moves through different steps of migration and transformation.

    More Control of Pipelines

    Ability to view Spark DAGs & Tasks and monitor and control the data pipeline give flexibility in Data Operations.

    Low Code No Code Data Pipeline creation

    With LCNC orchestration modules developers can drag and drop components on the canvas and easily build the data flows in the pipeline and generate DAGs with a click of a button.

    Domain Outlook

    With a body of work, sub-body of work, and unit of work users can tag pipelines around a business boundary, functionality, and data domains, which can help in data domain governance and build data mesh in the future.
    Python, Spark, Airflow, Kubernetes, and UI tools like React.js and Redux.
    The balance and reconciliation service relies on Spark to compare data frames. Spark clusters can auto-scale very fast, hence reconciliation services can scale for large data. For movement of a large volume of data, network utilization may shoot up and there is a dependency on network bandwidth between on-premise and cloud.
    Yes, you can. Some reports are already available in the IDEA ABCR module. If you want to build custom reports, you can also access the metamodel where the audit and reconciliation information is saved. Also, Airflow comes with UI to describe the status of data pipelines at the summary and detail level.
    Currently, IDEA uses Airflow. For other data workflow orchestration technologies, IDEA needs to be customized. Please contact IDEA Support team for placing your request.
    Next Steps

    To learn more about IDEA by Capgemini and how we can help make data your competitive edge.
    Visit : www.capgemini.com/ideabycapgemini

    • Mukesh Jain
      IDEA Head

    • Harsh Vardhan
      IDEA Chief Digital Leader & Chief Technology Architect

    • Eric Reich
      Offer Leader and Global Head AI & Data Engineering VP

    • Aurobindo Saha
      IDEA Principal Sales Architect

    • Sameer Kolhatkar
      IDEA GTM Lead

    • Sandip Brahmachari
      IDEA D&S Lead sandip.brahmachary@capgemini.com

    • Anupam Srivastava
      IDEA Engineering Lead

    • Subramanian Srinivasan
      IDEA Shared Services Lead