Migration Asset

Enterprise Data Warehouse (EDW) are central repositories of integrated data purpose-built and optimized to support reporting and analytics. EDWs store current and historic data in an integrated repository to enable business intelligence throughout the enterprise. Building these warehouses has traditionally been a multi-year project, with great thought and data management best practices leveraged to build them.

Teradata is an established legacy data warehouse that has been around for decades, and it shows. Most deployments require complex scripting and an army of people to maintain them. A Teradata user faces first-hand challenges of administering and maintaining this platform coupled with unpredictable performance and low concurrency.

The Migration Asset provides the customer with the advanced tooling needed to analyze their existing data estate and create a customized migration plan to a modern, cloud-based platform.

The Migrations are those the user has to move historic data typically to get the project started in the cloud, it can be from data warehouses, and it can be from Adobe or other sources. The user migrating a data warehouse to the cloud to get started.

Migration Asset contains various services such as :

• Teradata DWH to Redshift Migration
• Teradata DWH to Snowflake Migration

Teradata - Redshift

AWS Redshift is a Cloud-based data warehouse service. Redshift is a petabyte scale, powerful and fully managed relational data warehouse service.

Redshift is a Mass Parallel Processing (MPP) Data Warehouse. Redshift handles large amounts of data and workloads for optimal configuration and high performance even for large datasets.

Amazon Redshift architecture:

Fundamental for building a data platform with Redshift.

1. The user has the option to
1.   Change the name
2.   Change data type
3.   Change constraints
4.   Change partition as well as distribution type along with the key
2. In this option all these inputs come from user through an UI and this information are stored in a metadata table
3. The metadata table becomes the driving table along with the mapping table to build the target table based on the DDL that is generated in the target database.

In IDEA, the project will follow the option 2 as the starting point to quickly test the end to end migration. Here the source and target database are as follows

1. Source Database - Teradata
2. Target Database - Redshift

Teradata DWH to Azure Redshift Migration contains various services such as

• Linked Service
• Discovery
• Schema Migration
• Data Migration
• ETL Migration
• Pipeline Studio Service

Linked Services

Linked services are much like connection strings, which define the connection information required to establish connection to external sources.
Users need to perform following steps to complete linked service registration

• Create Linked Service
• View Linked Service
• Edit Linked Service
• Test Connection
• Delete Linked Service

Discovery

Discovery module has been used to query the metadata of database (Source/Target).
Discovery contains various services such as

• Discovery Service
• Complexity Analyzer

Discovery service

Discovery service is used to extract the metadata information from source and target databases.
To perform discovery for given system (DB), User need to perform following steps

• Discovery Job Creation
• View Discovery Job
• Edit Discovery Job
• Execution of Discovery Job
• Monitoring Discovery Job
• Delete Discovery Job

Complexity Analyzer

Complexity Analyzer defines the complexity of ETL/ELT scripts in the aspect of conversion.
Users need to perform following steps for Complexity Analyzer Service

• Complexity Analyzer Job Creation
• View Complexity Job
• Edit Complexity Job
• Execution of Complexity Job
• Monitoring Job Execution
• Delete Complexity Job

Schema Migration service

Schema Migration is used to migrate the table structure from Source to Destination based on metadata mapping.
Users need to perform following steps for Schema Migration Service.

• Schema Migration Job Creation
• View Schema Migration Job
• Edit Schema Migration Job
• Execution of Schema Migration Job
• Monitoring Job Execution
• Delete Schema Migration Job

Data Migration Service

Data Migration Module has been used to migrate table data from Source to Target.
Users need to perform following steps for create migration.

• Data Migration Job Creation
• View Data Migration Job
• Edit Data Migration Job
• Execution of Data Migration Job
• Monitoring Job Execution
• Delete Data Migration Job

ETL Migration

ETL Migration service supports BTEQ conversion into RedshiftSQL and Redshift Procedure based on the mapping document available in IDEA metadata. ETL script might have DDL (temporary table creation) and DML statements and the ETL migration supports both DDL and DML conversion into Redshift compatible statements.

Pipeline Studio Service

Pipeline Studio is the industry-leading pipeline management design software and engineering solution that combines graphical configuration and reporting tools with industry-proven simulation engines.

Teradata - Snowflake

Below is the data architecture diagram that states the data flow and integration required with ESB and command Center to full the data as a service model

After login to IDEA Accelerator, user will be seen list of projects assigned to him/her.

To do migration for given project, User need to select project and click on continue.

Teradata DWH to Azure Snowflake Migration contains various services such as

• Linked Service
• Discovery
• Schema Migration
• Data Migration
• ETL Migration
• Pipeline Studio Service

Linked Service

Link service is used to register the snowflake subscription detail and the azure integration detail. The registered subscription will be referred by Discovery / Schema migration / Data migration service to migrate schema and data from Teradata into snowflake. Linked services are much like connection strings, which define the connection information required to establish connection to external sources.
Users need to perform following steps to complete linked service registration

• Create Linked Service
• View Linked Service
• Edit Linked Service
• Test Connection
• Delete Linked Service

Discovery Service

Discovery service is used to extract the metadata information from source and target databases.
To perform discovery for given system (DB), User need to perform following steps

• Discovery Job Creation
• View Discovery Job
• Edit Discovery Job
• Execution of Discovery Job
• Monitoring Discovery Job
• Delete Discovery Job

Complexity Analyzer

Complexity Analyzer defines the complexity of ETL/ELT scripts in the aspect of conversion.
User need to perform following steps for Complexity Analyzer Service

• Complexity Analyzer Job Creation
• View Complexity Job
• Edit Complexity Job
• Execution of Complexity Job
• Monitoring Job Execution
• Delete Complexity Job

Schema Migration

Schema Migration is used to migrate the table structure from Source to Destination based on metadata mapping.
Users need to perform following steps for Schema Migration Service.

• Schema Migration Job Creation
• View Schema Migration Job
• Edit Schema Migration Job
• Execution of Schema Migration Job
• Monitoring Job Execution
• Delete Schema Migration Job

Data Migration Service

Data Migration Module has been used to migrate table data from Source to Target.
Users need to perform following steps for create migration.

• Data Migration Job Creation
• View Data Migration Job
• Edit Data Migration Job
• Execution of Data Migration Job
• Monitoring Job Execution
• Delete Data Migration Job

ETL Migration Service

ETL Migration Process takes place when you want to move information from one system into a Data Warehouse environment.
Users need to perform following steps for ETL Migration Service.

• ETL Migration Job Creation
• View ETL Migration Job
• Edit ETL Migration Job
• Execution of ETL Migration Job
• Monitoring Job Execution
• Delete ETL Migration

Pipeline Studio Service

Pipeline Studio is the industry-leading pipeline management design software and engineering solution that combines graphical configuration and reporting tools with industry-proven simulation engines.