Please standby, while we are retrieving your information
Audit Balance Control and Reconciliation Service
ABCR services
AUDIT AND CONTROL & Balance and reconciliation
This framework is an important part of any complex data management pipeline.At a high level, the ABCR framework provides an 'easy to use' mechanism that allows applications to interact with the data management metadata such that they can focus on ABCR, rather than on specific database or configuration file implementations. It provides the ability to store metadata related to various activities that encompass data management and helps to monitor and update various data management tasks, like ingestion, validation, and more. Ability to identify execution failures, followed by notification and execution of failed jobs as needed. It also enables auditing executions at each phase to keep track of changes data has gone through. Reconciliation service helps to compare data that rests in Data Lake or data between source and target (on-premise, other clouds, databases, file servers, etc.).
ABCR
ABCR stands for Audit, Balance, Control and Reconciliation.
As name suggests this module works around all other modules and helps in auditing, balancing, controlling and reconciliating various activities and jobs performed under IDEA.
Let’s look at different services offered by this function
ABCR Features
A brief summary
Audit
Reconcilation
Control
Workflow
Audit
The most appropriate definition of 'audit' as applicable to IDEA is 'a systematic review or assessment of something'. From an IDEA perspective, In most cases, this will translate into having a complete picture of various jobs executed under the IDEA umbrella. Under ABCR, audit information can be retrieved from two parts - the job run table and the application log.
It enables user to add significant event entries to job run table which can be recorded and also allows to review the status of each event. Also additional information about the job success / failure can also be recorded.
The amount of information logged by each application depends on the application as well as the logging level used by the application.
Reconciliation
The term (Data) Reconciliation is typically used to describe a verification phase during data migration, where the target data is compared with the original source data, in order to ensure that the data has been transferred correctly.
Reconciliation is a special type of job under ABCR, that can be used to validate data between any two data stores. It enables user to select and compare data from any of the supported data stores.
Entity based comparison allows the user to select a table from the source and compare them with same set of table(s) in the destination. This method allows the user to compare one of more tables at the same time.
Query based comparison allows the user to formulate a query on the source and the target and compare the results.
Control
Job management is related to the situation where jobs are represented on an individual basis..It covers two parts
- Helps to manage and execute jobs on individual basis.
- Supports on-demand job execution and scheduled job execution.
For execution and monitoring the individual jobs, there is job management. For orchestrating one or more jobs as a group / batch, there is workflow management which supports
- Database/ Dataset level mapping
- Table level mapping
- View level mapping
- Column level mapping
Workflow
A workflow enables user to define a pipeline of seemingly independent tasks. Though many of the features in the modernization / migration project are developed as independent, modular pieces of functionality, to achieve the end goal, this service performs one or more of these tasks in a particular sequence.
In other words, A workflow allows user to string together multiple tasks and execute them in a way that helps to achieve end goal of data processing.it enables user to define simple and/or complex relationships between various tasks. If the workflow is parameterized, it can be reused for multiple executions, for different operations.
Design for Industrialization
Service Benefits
More Transparency to Data
The ability to balance and reconcile data helps to trust the data more as it moves through different steps of migration and transformation.More Control of Pipelines
Ability to view Spark DAGs & Tasks and monitor and control the data pipeline give flexibility in Data Operations.Low Code No Code Data Pipeline creation
With LCNC orchestration modules developers can drag and drop components on the canvas and easily build the data flows in the pipeline and generate DAGs with a click of a button.Domain Outlook
With a body of work, sub-body of work, and unit of work users can tag pipelines around a business boundary, functionality, and data domains, which can help in data domain governance and build data mesh in the future.Next Steps
To learn more about IDEA by Capgemini and
how we can help make data your competitive edge.
Visit : www.capgemini.com/ideabycapgemini
