DB-13: OpenEdge VLDB

Dan Foreman

OpenEdge VLDB
(Very Large DataBases)
DataBases)
Dan Foreman
BravePoint
[email protected]

DB-13: OpenEdge VLDB

Introduction - Dan Foreman

Progress User since 1984
Author of:
Progress Database Administration Guide
Progress Performance Tuning Guide
V10 Database Administration Jumpstart
Virtual System Tables
Pro Dump & Load
ProMonitor
2
DB-13: OpenEdge VLDB

Audience Survey
Progress Version
Progress V8
Progress V9
OpenEdge R10.0*
OpenEdge R10.1A
OpenEdge R10.1B

3
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Audience Survey
Largest Single Database Size
> 1tb
> 500gb
> 250gb
> 100gb
Everyone else can leave the room

4
DB-13: OpenEdge VLDB

Agenda
Definition of VLDB
Common Characteristics of VLDB
Growth Rates and Capacity Planning
Top Challenges for VLDB Customers
Wish List
Questions
Conclusion
5
DB-13: OpenEdge VLDB

Definition of VLDB
Minimum of 100gb
Single Database (not a set)
For this presentation I didn
didnt
differentiate between allocated space
and High Water Mark
OpenEdge DB only (no Oracle allowed)
If the site was not a BravePoint
customer, they needed to be willing to
share various statistics
6
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Note about Single DB Requirement

Two V9 sites previously had much
larger DBs but split their VLDB into
multiple DBs for performance reasons
Reads per Second beyond a certain
point would not improve for a single
DB regardless of spin or B values
The speed limit is because certain
actions need to be serialized - for
example, adjusting the LRU chain when
reading a block
7
DB-13: OpenEdge VLDB

Units of Measure
kb
mb
gb
tb
pb
eb

=
=
=
=
=
=

Kilobytes
Megabytes (1000kb)
Gigabytes (1000mb)
Terabytes (1000gb)
Petabytes (1000tb)
Exabytes (1000pb)

8
DB-13: OpenEdge VLDB

Progress History DB Size LImits

V8
64gb
256gb

1k DB Block Size
8k DB Block Size

V9
Maximum Areas: 1,000 (some are reserved)
Area Size: 1k Blk Size & 256 RPB = 8gb
Area Size: 8k Blk Size & 1 RPB = 16tb
995 Areas * 16tb = 15,920tb = 16pb
9
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Progress History DB Size Limits

OpenEdge 10
32,000 Areas (starting in OE 10.1B)
Max File Size of 1tb * 1024 Extents = 1pb
1pb * 32000 Areas approx 32eb

10
DB-13: OpenEdge VLDB

Some Progress Limits - Fragments

A record can be split into two or more
fragments
Each fragment has a ROWID address
V9 and V10.0* to V10.1A
2 billion fragments per Area

OpenEdge 10.1B
9,223,372,036,854,775,807 (9 quintillion
for 1k DB block & 256 RPB)
A mere 1 or 2.5 quintillion for 4k/8k DB
block sizes
11
DB-13: OpenEdge VLDB

Database Sizes
Site

Version

HWM

Allocated

Anonymous 9.1E02

1.7 TB

2.0 TB

AHM

9.1E0412

743GB

941GB

Wachovia

9.1E0409

359GB

400GB

Broder

10.1A0205 290GB

339GB

Toll (OZ)

10.0B0520 123GB

183GB

12
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Largest Table
Site

Records

Size

Anonymous 825 million 139gb

Same
Table?
yes

Wachovia

202 million 39gb

yes

AHM

431 million 67gb

no

Broder
Toll (OZ)

81 million
147m

no
no

31gb
18gb

13
DB-13: OpenEdge VLDB

Progress Version
Three were on V9.1E*
One was R10.1A SP02 05
One was R10.0B SP05 20
All sites were 6464-bit Progress except
the Anonymous site due to a
disagreement
disagreement about licensing

14
DB-13: OpenEdge VLDB

Server Demographics
Site

Server

Anon

IBM P590 AIX 5.2

AHM

Fujitsu
M850
Wachovia Sun
V1280
Broder
IBM P590
Toll

OS

RAM

CPU

64gb

32

Solaris 10 64gb

16

Solaris
2.8

32gb

AIX 5.3

32gb

16

Sun V890 Solaris 10 32gb

8
15

DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Server Demographics
Where is Windows?
One site was migrating from Tru64 to
Linux the day this presentation was
due
One of our HP/UX customers
submitted but they left out too many
details to be included in this
presentation
16
DB-13: OpenEdge VLDB

Disk Array Demographics

IBM DS8300; 8tb
IBM DS4800; 8tb; 100 disks
EMC DMX; 1tb; 128 disks
EMC CX700 unknown (under control of
corporate administrators)
HP XP1024 (Hitachi)
Sun StorEdge 9990 (aka HDS 9990)
12 GB Cache
Disks are 146GB/15K FC
Total about 3TB
17
DB-13: OpenEdge VLDB

Disk Array Demographics

Bottom line: No JBOD is being used
in the VLDB world

18
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Concurrent Database Connections

Anon:

472 (355 AppSrv;

AppSrv; 115 W/S)
4203 actual users
AHM:
2999 (.lic
(.lic))
Toll:
1864 (.lic
(.lic))
Broder:
1163 (.lic
Broder:
(.lic))
Wachovia: 389 (.lic
(.lic))

19
DB-13: OpenEdge VLDB

Monitoring Tools
ProMonitor
Fathom Management
Homegrown
ProTop

20
DB-13: OpenEdge VLDB

Daily Growth Rates

Site

Approximate Growth per Day

AHM

600mb

Toll

280mb

Wachovia

250mb

Broder

TBA

21
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Capacity Planning
No site had a formal capacity planning
process

22
DB-13: OpenEdge VLDB

Capacity Planning Tools - CPU

sar
nmon (AIX)
Adrian Performance Monitor (Solaris)
User
Users Scream
Scream

23
DB-13: OpenEdge VLDB

Capacity Planning Tools - RAM

vmstat
nmon (AIX)

24
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Capacity Planning Tools - Disk

iostat
SAN Vendor
Vendors tools

25
DB-13: OpenEdge VLDB

Capacity Planning Tools DB Growth

Area Status (_areastatus
(_areastatus)) Reports
dbanalys + Excel
Fathom Management
ProMonitor

26
DB-13: OpenEdge VLDB

Number of Dedicated DBAs

Lowest:
Highest:

.3 (i.e. 30% of 1 person)

2

27
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

DB-13: OpenEdge VLDB

Dan Foreman

Backup Methods
probkup online
probkup online to disk
Full each Sunday
Incremental for the rest of the week

proquiet + Snap Copy

Shutdown, Snap Copy, Restart
Customer was uncomfortable with hot
backup
28
DB-13: OpenEdge VLDB

Backups - Wachovia
Full B/U once a week (5hrs)
Two Incrementals Daily (3hrs each)
These are a backstop to the occasional
AI glitch

Warm Spare restarted weekly

AI files are shipped hourly

29
DB-13: OpenEdge VLDB

Database Replication
Hot: Fathom Replication (2)
Warm: After Imaging (2)
Cool: Restore from Snap Copy (2)

30
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

10

DB-13: OpenEdge VLDB

Dan Foreman

Replication Issues
Getting Fathom Replication to
integrate smoothly with Veritas
Cluster Server
Long Redo Phase Bug

31
DB-13: OpenEdge VLDB

Maintenance Windows
Anonymous
Every Sunday 0300-0400
Everything else is negotiated

Toll
Weekly DB restart at 0400 on Sunday
morning for 15 minutes
Every 6-8 weeks for 6 hours (only if there
is a valid reason which is normally DB
or Application Software Upgrade)
32
DB-13: OpenEdge VLDB

Maintenance Windows
AHM
Once a week 2200-0200
Quarterly for Dump/Load

Wachovia
Weekdays 1700-2300
Weekends by negotiation

Broder
5 minutes every night
33
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

11

DB-13: OpenEdge VLDB

Dan Foreman

Tuning
The recommendation of 10000 * (# of
CPUs) for setting spin does not scale
well
Generally spin should be under 50000

34
DB-13: OpenEdge VLDB

Anonymous

Tuning

-n 5000
-bibufs 1000 -aibufs 1500
-aistall
-Mn 70 -Mi 3 -Ma 10 -Mpb 50
-L 1500000
-B 160000 (* 8k = 1.28gb)
-t -T /s430/temp/batch
-spin 4000
-tablerangesize 800 -indexrangesize 2000
-semsets 32
-directio
-pinshm
-napmax 500
-Bpmax 40000
35
DB-13: OpenEdge VLDB

Tuning
AHM Startup
-B 2000000 (* 8k blocks = 16gb)
-spin 25000
-bibufs 130 -aibufs 130
-L 100000
-N tcp -S ntunifipc2
-Mn 1000 -Mi 1 -Ma 50 -Mpb 2 -n 3000
-Mxs 70
-aistall bistall
-tablerangesize 525 -indexrangesize 1000
-semsets 128
-ServerType Both
-DBService replserv -pica 8192

36

DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

12

DB-13: OpenEdge VLDB

Dan Foreman

Tuning
Broder
-S liv1
-L 400000
-B 200000
-c 100
-spin 30000
-n 2000
-bibufs 25
-aibufs 25
-directio
-semsets 5
-tablerangesize 700
-Mn 21 -Mi 2 -Mpb 10

-indexrangesize 1600

37
DB-13: OpenEdge VLDB

Wachovia

Tuning

-B 2883584
-bibufs 30 -aibufs 45
-bithold 2048
-aistall
-n
500
-N
TCP
-L
30000
-Ma 6
-Mn 300
-Mpb 50
-spin 160000
-semsets 30
-Bpmax 2048
-Mxs 65536
-indexrangesize 2500
-tablerangesize 1000
-SQLTempDisk 4000000 -SQLTempPgSize 20
38
DB-13: OpenEdge VLDB

Tuning
Toll
-B 600000
-L 1000000
-n 2200
-spin 160000
-aibufs 200 -bibufs 120
-aistall
-bistall
-bithold 8000
-tablerangesize 700
-indexrangesize 700
-Mn 200
-Mi 5 -Ma 10
-semsets 10
-pinshm
-napmax 500
-directio
39
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

13

DB-13: OpenEdge VLDB

Dan Foreman

Storage Areas
Anonymous
560 Storage Areas (Table/Index)
No fixed extents
Large files enabled (of course)
8k DB Block Size
RPB
256 for smaller Areas
64 for large Areas
1 for index Areas
40
DB-13: OpenEdge VLDB

Dump/Load
Broder:
Broder:
AHM:
Wachovia:
Anonymous:

Annual Pro D&L

Quarterly Binary D&L
Are you kidding?
kidding?
D&L by Area or None

41
DB-13: OpenEdge VLDB

Dump/Load - AHM
Typical outage duration is 2424-36 hours
Do not dump the entire database. Find
the worst offenders by table
(fragmentation or scatter, or a
combination), and then full dump and
load the storage areas that contain
the majority of these worst offenders
The intention is to select a subset of
the DB that can be completed within
the outage window
42
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

14

DB-13: OpenEdge VLDB

Dan Foreman

Dump/Load - AHM
Fathom Replication needs to be
resync
resyncd which means a full backup
and restore to a separate machine
With a 743gb DB, that's a 12 hour
backup, and a 1414-16 hour restore

43
DB-13: OpenEdge VLDB

Dump/Load - AHM
While loading should be faster than
dumping, the limit of only one load
per storage area to do an index build
inline (build indexes), or do multiple
loads with a full index rebuild after the
loads complete, makes loading the
critical factor in the outage window
Only an area that can be fully dumped
and loaded in the time frame can be
processed
44
DB-13: OpenEdge VLDB

Top Challenges
24 hours is not enough
enough
Physical Redo Phase in AI roll
forward. It could take seconds,
minutes or several hours.
hours.
Who else has this problem?

Client Performance visibility/tracking

It's impossible to add indexes to
some tables
tables
45
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

15

DB-13: OpenEdge VLDB

Dan Foreman

Top Challenges
Knowing more about what a program
is doing would be HUGE help. The
last X number of DB (access)
statements executed would be nice.
Like the SQL query plan, but hopefully
more comprehensible!
comprehensible!

46
DB-13: OpenEdge VLDB

Wish List
What program is a Client running (#1)
Backup by Area
Anonymous probkup is 14-16 hours

Table Partitioning (AKA Horizontal

Partitions)
Online SQL Permissions Changes
Online dump/load
Change DB parameters on line
47
DB-13: OpenEdge VLDB

Summary
Don
Dont be afraid of OpenEdge VLDBs
especially with OpenEdge 10.1B
VLDB on Windows might be a lonely
place

48
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

16

DB-13: OpenEdge VLDB

Dan Foreman

Questions

49
DB-13: OpenEdge VLDB

Conclusion
Thank you for coming!
Don
Dont forget your evaluations (both
good and bad)

50
DB-13: OpenEdge VLDB

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

17