Teradata Utilities
Teradata Utilities
Client Tools
2 Application Utilities
• Teradata Database provides several application
utilities for data manipulation
• Each utility exploits the capabilities of Teradata
parallel architecture
• Teradata application utilities are supported on
several hardware platforms including a wide range
of channel-attached mainframes and LAN-attached
clients
• Regardless of host platform, all access between the
host and the Teradata Database relies on Call Level
Interface (CLI)
Client Tools
3 Application Utilities ….
• The Teradata application utilities run on the host
computer, whether it is mainframe or a workstation
• BTEQ is a transparent interface to CLI, to transmit
textual SQL to Teradata server, and deliver response
to the user
• FastLoad, MultiLoad and TPump performs high
volume data transfer from the host to Teradata
• Fast Export performs high volume SELECTs to
export data from Teradata server to the host
• CLI is the vehicle that makes parallel access possible
Client Tools
4 Application Utilities ….
Operating System
Support Environment
Fast
BTEQ Multi Fast
Load TPump
Load Export
CLI
Host
Server
Client Tools
5 Request and Response Flow
Application
Request
CLI
MTDP
MOSI
Response Server
Client Tools
6 Data Flow
Host
Gateway
PE PE
Data Data
Disk Disk Disk Disk
Client Tools
7 Application Development
More
INMODs
Flexibility Application
Utilities
Less
Easy Ease of Use Difficult
Selection of the right vehicle can be crucial and
depends on how difficult it is to implement and
maintain
Use of application utilities
• Offer less complexity
• Takes full advantage of parallel processing
Client Tools
8 Using An INMOD Routine
The application utilities allow input data to be read or
pre-processed by a user-written INMOD routine.
An INMOD routine can perform various functions:
• Generate records to be passed to the utility
• Validate a data record before passing it to the utility
• Read data directly from one or more database systems
• Select specific records for input to the Teradata
Database
• Perform data conversions not supported by utilities
• Add or change data fields in the records
Client Tools
9 Using An INMOD Routine
IMS ORACLE
User–written INMOD
Server
10
Client Tools
BTEQ
FastLoad
Fast Export
MultiLoad
TPump
Client Tools
11 BTEQ
• Batch mode utility for submitting SQL requests to
the Teradata Database
• Runs on every supported platform – laptop to
mainframe
• Flexible and easy to use report writer
• Exports data to a client system from the Teradata
Database
• Reads input data and imports it to the Teradata
Database
• Limited ability to branch forward to a LABEL
Client Tools BTEQ
12 Request Flow
Server
PE
Server
PE Generated answer set
Commands:
OS
RUN
IMPORT
REPEAT
QUIET
EXPORT
INDICDATA/RECORDMODE
Client Tools BTEQ
19 Sequence Control Commands
Controls the sequence in which other BTEQ commands
and SQL statements will be executed within the scripts
and macros.
Commands:
IF THEN
ELSE
ENDIF
GOTO
Client Tools BTEQ
40 Format Control Commands
To specify the way in which BTEQ presents information
on the screen/printer.
Commands:
FOOTING
HEADING
RETLIMIT
FOLDLINE
SUPPRESS
QUIET
Client Tools BTEQ
41 EXPORT
BTEQ by default delivers a response to all SQL queries
that includes a helpful message along with helpful
diagnostic information about the time taken to perform
the query.
.EXPORT REPORT
Sends results to a host file in field mode. Data set
contains column headings and formatted data.
column A column B column C
1 2 3
4 5 6
7 8 9
Transfers data one column at a time with numeric data
converted to character.
Client Tools BTEQ
45 Data Modes ….
Record mode is set by .EXPORT DATA.
.EXPORT DATA
Sends results to a host file in record mode.
.EXPORT INDICDATA
Sends query results that contain indicator variables to a
host file. Allows host programs to deal with NULLs.
NULL Columns
Concepts:
Load into an empty table with no secondary indexes.
Creates two error tables.
Checkpoints can be taken for restarts.
Client Tools FastLoad
53 Restrictions
• If an AMP goes down, FastLoad cannot be restarted
until it is back online
• The Teradata Database will accommodate up to 15
FL/ML/FE runs at one time
• Tables defined with referential integrity or secondary
indexes cannot be loaded with FastLoad
• Duplicate rows cannot be loaded into multi-set table
with FastLoad
Client Tools FastLoad
54 FastLoad Phases – Phase 1
Phase 1
• FastLoad uses one SQL session to define AMP
steps
• The PE sends a block to each AMP
• AMPs hash each record and redistribute them to the
AMP responsible for the hash value
• Records are written to the target table in unsorted
blocks
Client Tools FastLoad
55 Phase 1
BYNET
AMP AMP
Disk Disk
Client tools
Tools FastLoad
56 Phase 1
PE
Data
Host BYNET
AMP AMP
Disk Disk
Client Tools FastLoad
57 FastLoad Phases – Phase 2
Phase 2
• When loading completes, each AMP sorts the
target table, puts the rows into blocks, and
writes the blocks to the disk
• Fall back rows are then generated if required
Client Tools FastLoad
58 Phase 2
PE
BYNET
AMP AMP
Data
Disk
Client Tools FastLoad
59 FastLoad Script
fastload < floadjob > floadjob.out
.logon Tdpid/User,Passwd
.begin loading Customer Name of empty table
errorfiles custerr1, custerr2; Starts Phase 1
define
CustNum ( Integer ),
Name ( Char(20) ) Defines input
file = CustData.dat; record
insert into Customer values SQL Insert
(:CustNum, :Name); statement
.end Loading; Start phase 2; if
.logoff; omitted utility
will pause
Client Tools FastLoad
60 Paused FastLoad Job
SQL & Data
datafile1 PE
BYNET
AMP AMP
Disk Disk
Client Tools FastLoad
61 Paused FastLoad Job ….
PE
datafile1
Data
BYNET
AMP AMP
Disk Disk
Client Tools FastLoad
62 Paused FastLoad Job ….
PE
datafile n
Data
BYNET
AMP AMP
Disk Disk
Client Tools FastLoad
63 Paused FastLoad Job ….
PE
BYNET
AMP AMP
Disk Disk
Client Tools FastLoad
64 Error Tables
Error Table 1
Contains one row for each row which failed to be
loaded due to constraint violations or translation errors.
Column_Name Content
Error_Code The Error code in DBC.ErrorMsgs
Error Field name The column that cause the error
Data Parcel The data record sent by the host
Error Table 2
Captures rows that contain duplicate values for UPIs
Client Tools FastLoad
65 INMOD
To Specify:
• Number of SESSIONS required
• Number of hours Fast Export will try to establish a
connection
• Number of minutes between each logon attempts
• Parameter for specifying the notify user exit option.
Client Tools Fast Export
78 END Export
.END EXPORT
• Delimits a series of commands that define a single
EXPORT action
• Must be the last directive in a Fast Export script
• Causes the utility to send the SELECT to the
Teradata Database
Client Tools Fast Export
79 Sample Fast Export Script
Table A
Table B
update Insert
MultiLoad Table C
Delete
Table D
Host Table E
Server
Client Tools MultiLoad
83 MultiLoad Limitations
• No data retrieval capability
• Arithmetic functions, exponentiation or
aggregates are not processed
• Cannot process tables with Secondary Indexes or
Referential Integrity defined
• Import tasks require use of Primary Index
Client Tools MultiLoad
84 MultiLoad Tasks
MultiLoad allows INSERT, UPDATE, DELETE and
UPSERT operations against up to five target tables per
task.
Two distinct tasks are:
IMPORT task:
These are the tasks which intermix a number of different
SQL/DML statements and apply them to up to five
different tables depending on the APPLY conditions
DELETE task:
These are tasks which execute a single DELETE statement
on a single table.
Client Tools MultiLoad
85 IMPORT TASK
Host inserts
updates
Input
deletes
Data
upserts
MultiLoad
Apply Conditions
Server
T1 T2 T3 T4 T5
Client Tools MultiLoad
86 DELETE TASK
Host
Data value
deletes only
MultiLoad
T1
Client tools MultiLoad
88 Sample MultiLoad Script
.logtable logtable001; define logtable
.logon TDPID/User/Passwd;
.begin mload tables Emploee, Emp_History; begin loading
.layout Employee_Trans; define
.filler Transcode 1 char(3);
.field Empno * smallint;
.field Deptno * smallint;
.field Salary * decimal(8,2);
.dml label payroll;
do insert for missing update rows;
update Employee set Salary = :Salary where
Empno = :Empno;
insert into Employee (Empno, Salary ) values
(:Empno, :Salary );
Client Tools MultiLoad
89 Sample MultiLoad Script ….
.dml label Terminate;
delete from Employee where Empno = :Empno;
insert into Emp_History (Empno, Deptno)
values (:Empno, :Deptno );
.import infile Filename; File name
layout Employee_Trans Insert
apply Payroll where Transcode = ‘PAY’ statement
apply Terminate where Transcode = ‘DEL’;
.end mload End loading
.logoff
Client Tools MultiLoad
90 Script Components
.logtable defines the name of the restart log.
.logon defines username which will own the sessions.
.begin mload tables defines the tables which will
participate in the MultiLoad.
.layout defines the layout of the incoming record(s).
.field defines the name of an input field, its position in the
record, and its datatype.
.dml label defines a set of DML instructions which will
be applied if conditions are met.
Client tools MultiLoad
91 Script Components ….
do insert for missing update rows indicates that UPSERT
processing follows:
do update if present.
do insert if not present.
.import infile references filename of the input file.
layout references previously defined layout.
apply references label to be applied and conditions
under which to do so.
.end mload defines end of MultiLoad script.
.logoff terminate the sessions.
Client Tools MultiLoad
92 IMPORT Phases
IMPORT consists of five separate phases of processing.
Preliminary Phase Basic setup
DML Phase Get DML steps down on AMPs
Acquisition Phase Send the input data to AMPs and
sort it
Application Phase Apply the input data to the
appropriate target tables
End Phase Basic clean up
Client Tools MultiLoad
93 Preliminary Phase
Validate all statements MultiLoad and SQL
Start all sessions #AMPS + 2
Create work tables One per target table
Create error tables Two per target table
Create restart log One per IMPORT run
Apply locks to target tables Prevent DDL
Client Tools MultiLoad
94 DML Transaction Phase
Send prototype DML to the Server
Send DML steps in work tables
Add a USING modifier to the request
Host data to be filled in from input file
Client Tools MultiLoad
95 Acquisition Phase
Get the data from host and apply it to appropriate AMP
worktables.
- Duplicate rows for each successful APPLY
- Make blocks and send to AMPs
- Deblock and resend record to ‘correct’ AMP
Reblock and store in worktable of target table.
- Sort the reblocked records in the work tables
- Sort by hash value and sequence to be applied
Set up transition to the application phase.
- Upgrade locks on target tables to write
- Set table headers for application phase
Client Tools MultiLoad
96 Application Phase
Execute MLOAD for each target table as a single multi-
statement request.
- End of host interaction until end of phase
- AMPs independently apply changes to target
tables
- Executed as a single transaction without
rollback
- Restartable based on last checkpoint
- No Transient Journal needed
Client Tools MultiLoad
97 Cleanup Phase
Execute END MLOAD processing as a series of
transactions performed by the host utility.
- All locks are released
- Table headers are restored across all AMPs
- Dictionary cache of Target tables is spoiled
- Statistics are reported
- Final error code is reported
- Target tables are made available to other users
- Work tables are dropped
- Logtable is dropped
Mload Session logoff.
- Logoff request sent to each AMP with a session
Client Tools MultiLoad
98 DELETE/IMPORT Differences
DELETE tasks operate very similarly to IMPORT tasks
with some differences.
• Deleting based on a Primary Index is not permitted
• DML DELETE statement sent to each AMP with a
match tag parcel
• No Acquisition phase because no variable input
record to apply
• Application phase reads each target block and
deletes qualifying rows
• All other aspects similar to IMPORT task
Client Tools MultiLoad
99 Restarting MultiLoad
DBC restart
• MLOAD reinitiated automatically after DBC
recovery
• Continue from checkpoint without user interaction
Host restart
• Resubmit the script as-is
• MLOAD determines its stopping point and restarts
Client Tools MultiLoad
100 Application Utility Checklist
BTEQ FastLoad Fast Export MultiLoad TPump
DDL Functions All Limited No All
DML Functions All INS SEL Ins/upd/del
Multiple DML Yes No Yes Yes
Multiple tables Yes No Yes Yes
Multiple sessions Yes Yes Yes Yes
Protocol used SQL FastLoad EXPORT MultiLoad
Conditional Exp. Yes No Yes Yes
Arithmetic Calc Yes No Yes Yes
Data conversion Yes Limited Yes Yes
Error files No Yes No Yes
Error limits No Yes No Yes
User routines No Yes Yes Yes
Client Tools
101 TPump
• Allows near real-time updates from transactional
systems into the wearhouse
• Performs INSERT, UPDATE, and DELETE
operations, or a combination, to more than 60 tables
at a time from the same source
• Alternative to MultiLoad for low-volume batch
maintenance of large databases
• Allows target tables to:
- Have secondary indexes and Referential Integrity
constraints
- Be MULISET or SET
- Be populated or empty
Client Tools
102 TPump
• Allows conditional processing
• Supports automatic restarts
• No session limit
• Uses row-hash locks, allowing concurrent updates on
the same table
• User can specify how many updates occur minute by
minute
• No limit to the number of concurrent instances
Client Tools TPump
103 Limitations
• Exponential operators are not allowed
• Aggregate operators are not allowed
• Arithmetic functions are not supported
• There is a limit of four IMPORT commands
within a single TPump “load” task
Client Tools TPump
104 TPump Sample Script
.logtable logtbl002;
.logon TDPID/User,Passwd;
.begin load
errlimit 50
checkpoint 1
sessions 11
tenacity 2
error table tlnterr BTET pack 10;
.layout lay1a;
.filler ATest * byteint;
.field AByteInt * byteint;
.field AInteger * integer;
Client Tools TPump
105 TPump Sample Script ….