Oracle Data Integrator

An Introduction for Developers

(Part-1)

By
Nagarjuna Yellanki
 2

Objective
Training Objectives
After completing this Training, you should be able to:

Describe the Oracle Data Integrator (ODI) architecture

and apply ODI Topology Concepts for data integration.

Describe Oracle Data Integrator model concepts

Design ODI interfaces, procedures, and packages to

perform data transformations
 3

Agenda
Topic 1: Introduction
Topic 2: Repositories and Agents
Topic 3: Topology
Topic 4:Project
Topic 5:Model
Topic 6:Interfaces
Topic 7:Procedures
Topic 8:Packages
Topic 9 : Scenarios
Miscellaneous Topics.
 4

Topic 1
Introduction
 5

Agenda : Introduction
What is Oracle Data Integrator?
Why Oracle Data Integrator?
Overview of ODI Architecture
Overview of ODI Components
Overview of ODI Agents
Overview of Oracle Data Integrator Repositories
 6

What is Oracle Data Integrator?

A widely used data integration software product, Oracle Data
Integrator provides a new declarative design approach to defining
data transformation and integration processes, resulting in faster
and simpler development and maintenance.

Based on a unique E-LT architecture, Oracle Data Integrator not

only guarantees the highest level of performance possible for the
execution of data transformation and validation processes but is
also the most cost-effective solution available today.

Oracle Data Integrator provides a unified infrastructure to

streamline data and application integration projects.
 7

Why Oracle Data Integrator?

Use the Database as the ETL engine
ODI supports heterogeneous databases, not just Oracle

Built for SOA environments

Support for Web Services

Supports batch, event-based and real-time integration

Data Integrity Controls create a data firewall

Reduces data prep time by not processing erroneous data

Extensible through Knowledge Modules

Change Data Capture
Slowly Changing Dimensions
Bulk load

Java client application with server elements

 8

Overview of ODI Architecture (11g)

 9

Overview of ODI Components

ODI Studio Components:
Designer Navigator
Operator Navigator
Topology Navigator
Security Navigator
ODI agents
ODI Console
ODI repositories
 10

Overview of ODI Agents

The run-time agent functions as a listener and a scheduler agent.
The agent executes jobs on demand (model reverses, packages,
scenarios, interfaces, and so forth), for example when the job is
manually launched from a user interface or from a command line.
The agent is also to start the execution of scenarios according to a
schedule defined in Oracle Data Integrator.

Agents are not data transformation servers. They do not perform

any data transformation, but instead only orchestrate integration
processes.

Two Types of Agents

Standalone Agent
Java EE agent
 11

Overview of Oracle Data Integrator Repositories

There are two types of repositories in Oracle Data Integrator:

Master Repository -
Master Repository is a data structure containing information on the
topology of the company's IT resources, on security and on version
management of projects and data models. In general, you need only
one master repository.

Work Repository -
Work Repository is a data structure containing information on data
models, projects, and their use. Several work repositories can be
created with several master repositories if necessary.

Both the repositories are stored on a relational database accessible in

client/server mode from the different Oracle Data Integrator modules.
 12

Topic 2
Repositories and Agents
 13

Agenda : Administering ODI

Repositories and Agents
Creating Repository Storage Spaces
Creating and Connecting to the Master
Repository
Creating and Connecting to the Work Repository
Managing ODI Agents
Starting/Stopping Agent
 14

Administering ODI Repositories and Agents

Administering the ODI Repositories

Creating Repository Storage Spaces
Creating and Connecting to the Master Repository
Creating and Connecting to the Work Repository

Managing ODI Agents

What Is an Agent?
Starting an Agent
Stopping an Agent
 15

Creating Repository Storage Spaces

Create a storage space in each database that will be the host for a
repository.

Oracle SQL syntax to create a User/Schema to host the Master

repository.

Syntax : create user <MY_SCHEMA> identified by <MY_PASS>

default tablespace <MY_TBS> temporary tablespace
<MY_TEMP>;

grant connect, resource to <MY_SCHEMA>;

Example : create user master_rep identified by MASTER_REP;

grant connect, resource to master_rep;
 16

Create Master repository

Master Repository

Master Repository is the data structure containing information about

the topology of the companys IT resources, security, and version
management of projects and data models. This repository
is stored on a relational database accessible in client/server mode
from the different ODI modules.

Creating the Master repository consists of creating the tables and the
automatic importing of definitions for the different technologies.

To create the Master repository, launch bin/repcreate.bat or

bin/repcreate.sh (Applicable for 10g)
 17

Connecting to the Master Repository

Oracle Data Integrator Connection:

Login Name: A generic alias (for example: Master Repository)

User: SUPERVISOR (use capitals)
Password: SUNOPSIS (use capitals; SUNOPSIS is the default password of
the SUPERVISOR account )

Database Connection:

User: User ID/login of the owner of tables you created for the Master
repository
Password: The users password
Driver List and Driver Name: The driver required to connect to the DBMS
supporting the Master repository that you have just created.
Url : The complete JDBC Url of the data server hosting the repository
 18

Importing Master Repository

Importing existing Master Repository

The Master Repository Import Wizard option enables you to import the
Master Repository.

Mode of Import
(Insert/Update/Insert _ Update, File, Zip)

Objects Import
The process will import the Data Servers, Schemas, Logical Connects,
Contexts , etc)
 19

Create Work Repository

Work Repository -
Work Repository is the data structure containing information about
data models, projects, and their use. This repository is stored on a
relational database accessible in client/server mode from the
different ODI modules.

The Create Work Repository wizard uses to create the work

Repository.

To create the work repository, login to the topology manager.

 20

Connecting to the Work Repository

Oracle Data Integrator Connection:

Login Name: A generic alias (for example: Master Repository)

User: SUPERVISOR (use capitals)
Password: SUNOPSIS (use capitals; SUNOPSIS is the default password of the SUPERVISOR
account)

Database Connection:

User: User ID/login of the owner of tables you created for the Master repository
Password: The users password
Driver List and Driver Name: The driver required to connect to the DBMS
Supporting the Master repository that you have just created.
Url: The complete JDBC Url of the data server hosting the repository

Work Repository Connection:

Choosing Work Repository

 21

Managing ODI Agents

Agent -
An agent is a run-time component of ODI that
orchestrates the integration process. It is a lightweight
Java program that retrieves code from
the repository at run time.

Agents can do one of the following:

Execute objects on demand
Execute according to predefined schedules
 22

Starting an Agent

On Windows
Agent.bat "-port=20300
launches the Standalone agent on he port 20300.
On UNIX:
./agent.sh "-port=20300" "-name=Agent001"
launches the standalone agent and names it
Agent001.
 23

Stopping An Agent

On Windows:
agentstop "-PORT=20300
Stops the Listener agent on the port 20300.

On UNIX:
./agentstop.sh
stops the Listener agent on the default port.
 24

Topic 3
ODI Topology Concepts
 25

Agenda : Overview of ODI Topology

Overview of ODI Topology
Data Servers and Physical Schemas
Defining the Physical Architecture
Logical Schema & Context
Defining the Logical Architecture -and Mapping
using context
Defining Agents
Planning the Topology
 26

Overview of ODI Topology

Topology
The representation of the information system in ODI. ODI uses the
topology in order to connect to resources in the IS for integration
processes.

We have separation between logical and Physical Topologies

What kind of things are defined in Topology?

Technologies (Oracle,DB2,File,Etc)
Data types for a Given technology
Data Servers : Definition of your servers ,databases , etc.
Schema : Sub-division of data servers
Agents : Oracle Data integrator run-time modules
Contexts: Connects Physical topology with local Connections.
languages and Actions
 27

Data Servers and Physical Schemas

Data Server
Any System capable of storing data and making it available in
the form of tables.

We can use JDBC or JNDI to connect to a data server.

Physical Schema
A technology-dependent sub-division of a data server.
Datastores on a data server are located in a physical schema.

In a Physical schema always has 2 data server schemas.

The Data Schema : Which contains the Datastores.
The Work Schema : which stores temporary objects
 28

Defining the Physical Architecture

Each physical data server must be defined once
in the topology.

User each data server, define a physical schema

for each sub division of the server that will be
used.
 29

Logical Schema & Context

Logical Schema -
A single alias for several different physical schemas that have very
similar data structures.
This provides an Abstract view of the resources.

Context -
Maps logical resources onto their implementation as physical
resources.
In a given context one logical resources is mapped to one unique
physical resource.

In design time we use logical resources which are resolved to the

action physical resources using agent based on the context of
execution.
 30

Defining the Logical Architecture

Define one logical schema for each group of physical
schema containing a similar data structures.

If similar groups of resources appear in different

situations then one context is required for each situation.

A single physical resource may be mapped in several

contexts.

In a given context a logical resource is mapped at most

to one physical resource.
 31

Defining Agents
Physical agent represent components running at run
time.

Physical agents must also be abstracted as logical agents.

Depending on the context, the logical agent will be

mapped onto one of the physical agents.

While executing a job we specify the logical agent name.

 32

Planning the Topology

Identify the physical architecture
All the data servers
All physical schemas
Required physical agents

Identify the contexts.

Define the logical architecture.

Name the logical schemas
Name the logical agents

Map the logical schemas with physical schemas.