Exadata Cloud@Customer

Technical Architecture

May 2023
Copyright © 2022, 2023 Oracle and/or its affiliates
Copyright Ⓒ 2022, 2023 Oracle and/or its affiliates.
This software and related documentation are provided under a license agreement containing restrictions on use and
disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or
allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit,
perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or
decompilation of this software, unless required by law for interoperability, is prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If you find
any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of
the U.S. Government, then the following notice is applicable:
U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any
programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial
computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental
regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any
operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be
subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S.
Government.
This software or hardware is developed for general use in a variety of information management applications. It is not
developed or intended for use in any inherently dangerous applications, including applications that may create a risk of
personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all
appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its
affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their
respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used
under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo,
and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered
trademark of The Open Group.
This software or hardware and documentation may provide access to or information about content, products, and
services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all
warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an
applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss,
costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set
forth in an applicable agreement between you and Oracle.
Table of Contents
Table of Contents 2
Exadata Cloud@Customer Architecture Overview 3
Notes 3
Related Resources 4
Exadata Cloud@Customer Network 5
Notes 5
Related Resources 6
Rack Overview 7
Notes 7
Related Resources 8
VM Clusters and Database Servers 9
Notes 9
Related Resources 10
Storage Servers 11
Notes 11
Related Resources 12
Network Interfaces 13
Notes 13
Related Resources 14
Exadata Cloud@Customer Architecture Overview

Your Data Center Oracle Cloud Region

HTTPS
Your OCI Tenancy
Your Exadata and Autonomous Exadata Infrastructure
Database Administrators Service Management
(OCI Console, CLI API Calls
Rest APIs)
Your Client/ Persistent
Oracle Service Tenancy HTTPS
Backup VM Cluster Secure Oracle Cloud
Networks Automation
Operations (Console,
Tunnel
(Outgoing) REST APIs)

Your Client Applications/

Administrators SSH
(Oracle Net, SSH) Temporary
Secure
Control Plane Server (CPS) Operator
Tunnel
Management Bastion
Infrastructure (Outgoing) VCN Admin Server Server

Notes
Oracle Exadata Cloud@Customer runs Oracle Exadata Database Service and Oracle Autonomous
Database Service with Oracle-owned and managed infrastructure located in your data center.
Your databases run on one or more virtual machine (VM) clusters that reside on Exadata
infrastructure in your data center. Exadata Cloud@Customer also includes control plane server
(CPS) infrastructure that connects to an OCI region for cloud automation and administration.
Your client applications and administrators connect to the VM clusters through client and backup
networks that you create. You access your databases through standard Oracle database connection
methods, such as Oracle Net. You access the VM clusters through standard Oracle Linux methods,
such as token-based Secure Shell (SSH).
Your Exadata and Autonomous Database administrators can use the web-based OCI Console,
command-line interface (CLI), and REST APIs to connect to your OCI tenancy over an HTTPS
connection. Your tenancy sends service management API calls to the Oracle service tenancy. The
Oracle service tenancy connects to the Oracle-managed admin virtual cloud network (VCN). An
outgoing, persistent, secure automation tunnel connects the CPS infrastructure in your data center
to the Oracle-managed admin VCN in the OCI region for delivering cloud automation commands to

Exadata Cloud@Customer Technical Architecture 4

the VM clusters.
For Autonomous Database, separate persistent automation and temporary operator tunnels are
created, targeting separate service tenancy administrative VCNs. If you're using both Oracle
Exadata Database Service and Oracle Autonomous Database Service, you'll have two persistent
tunnels and, as required, up to two temporary tunnels. These separate tunnels enforce a strict
separation of duties for developer operations personnel, who only have authorization to take action
on specific VM cluster resources, segregated by service type.
Oracle Cloud Operations can use a console or REST APIs over an HTTPS or SSH connection to
manage the infrastructure. Oracle operator SSH connections travel through bastion and
management servers, the admin VCN, and an outgoing, temporary, secure operator tunnel from the
CPS to the admin VCN. Oracle operator HTTPS connections travel to the infrastructure through the
Oracle service tenancy, admin VCN, and the outgoing, persistent, secure automation tunnel.
Oracle operator SSH connections travel through bastion and management servers; the admin VCN;
and an outgoing, temporary, secure operator tunnel from the CPS to the admin VCN. You can
optionally control authorization of SSH access by Oracle operators by registering infrastructure
resources with a separate Operator Access Control.
Related Resources
Autonomous Database on Dedicated Exadata Infrastructure
Exadata Database Service on Cloud@Customer Overview
Network Requirements
Oracle Operator Access Control
VM Clusters

Exadata Cloud@Customer Technical Architecture 5

Exadata Cloud@Customer Network

Oracle
Cloud Region

Your Data Center

ExaC@C Service
Endpoints

Switch/Router

Exadata Infrastructure
Object
Public Virtual Storage
Private VM Cluster Private
Subnet Subnet Circuit (optional)
(Client) (Backup)
FastConnect
Your Oracle
Edge Edge OCI Telemetry
Your Client Database Switch/Router Customer
Applications/ Premises Monitoring
Administrators Equipment

Control Plane Server

(CPS) Infrastructure Identity
Service
NTP DNS
server server

Logging
Service

Notes
Your databases run on one or more virtual machine (VM) clusters that reside on Exadata
infrastructure in your data center. Exadata Cloud@Customer also includes control plane server
(CPS) infrastructure that connects to an Oracle Cloud Infrastructure (OCI) region for cloud
automation and administration.
Your data center connects to the OCI region to access Exadata Cloud@Customer (ExaC@C) service
endpoints, Object Storage, OCI telemetry monitoring, identity service, and logging service.
You can optionally connect your data center to the OCI region with FastConnect. This option uses a

Exadata Cloud@Customer Technical Architecture 6

public virtual circuit to connect the edge of your network to the Oracle edge in the OCI region. Your
edge connects to customer-premises equipment (CPE) in your data center.
The CPE connects to a switch or router, which directs traffic to and from the CPS, network time
protocol (NTP) server, domain name system (DNS) server. If you're not using FastConnect, your
network should also include a proxy server (not shown in the diagram).
Your client applications and administrators connect to your VM clusters and databases over the
private client and backup subnets that you access through a switch or router in your data center.
You create and manage the client and backup subnets.
Related Resources
Autonomous Database on Dedicated Exadata Infrastructure
Exadata Cloud@Customer Security Controls (PDF)
Exadata Database Service on Cloud@Customer
Network Requirements
VM Clusters

Exadata Cloud@Customer Technical Architecture 7

Rack Overview

Your Data Center Client

Network Backup
Network

Database Database
Server Server
Expandable
up to 8
database Host up to
servers 8 VMs per
database
VM VM server

Network Fabric

Expandable
up to 12
Storage Storage Storage storage
servers
Server Server Server

Management Network

X9M-2 Quarter Rack

Notes
Oracle Exadata Cloud@Customer includes multiple database servers and Exadata storage servers
connected by high-speed, low-latency network fabric. The Exadata rack resides in your data center.
You can choose from two elastic system shapes: elastic base and elastic. Each system shape can be
scaled with additional database and storage servers.
The diagram shows a rack with two database and three storage servers.
On X9M-2 systems, you can host up to 8 VMs per database server.

Exadata Cloud@Customer Technical Architecture 8

With Elastic Compute and Storage Expansion, you can dynamically expand the compute capacity
with up to 8 database servers and the storage capacity with up to 12 total storage servers.
Note: Today, when deploying Autonomous Database VM clusters, you can't elastically expand the
VM clusters after provisioning, so if you need larger clusters, first expand your infrastructure and
then provision the Autonomous Database VM cluster.
The client and backup networks provide access to your virtual machine (VM) clusters on the
database servers. Oracle manages the infrastructure through the management network, which
connects the database and storage server hardware.
Related Resources
Elastic Storage Expansion
Storage Configuration Requirements
System and Shape Configuration Options
VM Clusters

Exadata Cloud@Customer Technical Architecture 9

VM Clusters and Database Servers

Your Data Center

Hypervisor Hypervisor
2 OCPUs 2 OCPUs
16 GB RAM 16 GB RAM
VM Guest (X9M-2) Client Backup VM Guest (X9M-2) Client Backup
Up to 62 OCPUs Up to 62 OCPUs
Up to 1390 GB DRAM Up to 1390 GB DRAM

SSH Public Management Tools SSH Public Management Tools

Key dbaascli, ExaCLI Key dbaascli, ExaCLI

Users Oracle Database Users Oracle Database

root, opc, All features and options root, opc, All features and options
oracle, grid Oracle Grid oracle, grid Oracle Grid
Infrastructure Infrastructure
Network Fabric Ports Network Fabric Ports

Network Fabric

Storage Storage Storage

Server Server Server
Management Network
X9M-2 Quarter Rack

Notes
Each Oracle Exadata Cloud@Customer database server contains one or more virtual machine (VM)
guests running on a hypervisor and you can designate each VM to host either Exadata Database or
Autonomous Database Service instances. This configuration ensures a distinct separation between
the Oracle-managed and customer-managed components for the Exadata Cloud@Customer
platform.
Oracle manages the hypervisors through the management network. Each hypervisor uses minimal
resources: only 2 CPU cores (OCPUs) and 16 GB of RAM.

Exadata Cloud@Customer Technical Architecture 10

The client and backup networks connect to the VM guests through bonded network interfaces to
maximize performance and availability.
If you include Oracle Database software licenses in your Exadata Database Service or Autonomous
Database subscription, each VM guest has a complete Oracle Database installation that includes all
the features of Oracle Database Enterprise Edition plus all the database enterprise management
packs and all the Enterprise Edition options, such as Oracle Database In-Memory and Oracle Real
Application Clusters (RAC), as well as Oracle Grid Infrastructure. Alternatively, you can use Oracle
Database software licenses that you already own.
Note: Autonomous Database Service prevents customer access to the VM guests because Oracle
manages these as part of the service. Administrative actions against the database are limited to
protect the service configuration for availability and security. Given that SSH access to VM guests is
removed, customer administrative actions in Autonomous Database are performed by an ADMIN
user, not by the standard SYSTEM/SYSDBA users, and they're limited to the service-defined APIs,
console, and CLI.
Exadata Database Service allows customer access to the VM guests. VM guests require an SSH
public/private key pair for operating system security. You register a public key in each guest, and
you retain the private key that enables access to the VM operating system. VM guests include
standard user accounts, such as oracle, opc, grid, and root.
As a result of this configuration, Exadata Database Service customers manage the VM guests and all
the software they contain, including the Oracle-provided management tools, including dbaascli for
database lifecycle management and ExaCLI for monitoring and managing your Exadata storage
servers.
Each X9M-2 database server can have up to 62 OCPUs and 1390 GB of DRAM. The diagram shows a
quarter rack with two database and three storage servers.
The database servers connect to the Exadata storage servers through network fabric ports. The
Exadata infrastructure resides in your data center.
Oracle manages the infrastructure and, for Autonomous Database, the VM guests and database,
through the service-specific management network that connects the database and storage server
hardware.
Related Resources
Exadata Cloud Management Interfaces
System and Shape Configuration Options
VM Clusters

Exadata Cloud@Customer Technical Architecture 11

Storage Servers

Your Data Center Client Network Backup Network

Management Network
Database Database
Server Server
Network Fabric

Exadata Storage Server Exadata Storage Server Exadata Storage Server

(X9M-2) (X9M-2) (X9M-2)

Total
Flash Storage (25.6 TB raw) Flash Storage (25.6 TB raw) Flash Storage (25.6 TB raw) 76.8 TB
raw
PMEM (1.5 TB) PMEM (1.5 TB) PMEM (1.5 TB) 4.5 TB
PMEM
DATA - HIGH Redundancy
Usable Space: 66.8 TB - 152.6 TB
ASM RECO - HIGH Redundancy Total
Disk Usable Space: 38.2 TB - 114.5 TB 190.8 TB
Groups usable

SPARSE - Optional - HIGH Redundancy

Usable Space: 0 TB - 38.2 TB

X9M-2 Quarter Rack

Notes
When you configure Oracle Exadata Cloud@Customer, Oracle Automatic Storage Management
(ASM) provisions the storage space inside the Exadata storage servers. By default, ASM creates the
following disk groups:
The DATA disk group stores Oracle Database data files.
The RECO disk group stores the Fast Recovery Area (FRA), which is an area of storage where
Oracle Database can create and manage various files that are related to backup and recovery.
Only on Exadata Database Service, you can optionally create the SPARSE disk group to support

Exadata Cloud@Customer Technical Architecture 12

Exadata snapshots. If you create the SPARSE disk group, ASM allocates less space to the DATA and
RECO disk groups.
Exadata Cloud@Customer platform customers also need to decide whether to allocate storage to
perform local backups to your Exadata storage. If you choose this option, ASM allocates more space
to the RECO disk group and less to the DATA disk group.
To learn more about how ASM allocates space to disk groups, see .
In X9M-2 systems, each Exadata storage server has 25.6 TB of raw flash storage and 1.5 TB of
persistent memory (PMEM). The total for three storage servers is 76.8 TB of raw flash storage and
4.5 TB of PMEM.
Three storage servers have a total of 190.8 TB of usable ASM storage capacity. With this
configuration the ASM disk groups have the following usable space allowances:
DATA: 66.8 - 152.6 TB
RECO: 38.2 - 114.5
(Optional) SPARSE: 0 - 38.2 TB
The usable storage capacity is the storage available for Oracle Database files after taking into
account high-redundancy ASM mirroring (triple mirroring), which provides highly resilient database
storage on all Exadata Cloud@Customer configurations. The usable storage capacity does not factor
in the use of Exadata compression capabilities, which can increase the effective storage capacity.
The database servers connect to the Exadata storage servers through network fabric ports with
active bonding. The Exadata infrastructure resides in your data center. The diagram shows an X9M-
2 quarter rack with two database and three storage servers.
Oracle manages the infrastructure through the management network, which connects the
database and storage server hardware.
The client and backup networks provide access to your virtual machine (VM) clusters on the
database servers. Oracle manages the infrastructure through the management network, which
connects the database and storage server hardware.
Related Resources
Storage Configuration Requirements
System and Shape Configuration Options

Exadata Cloud@Customer Technical Architecture 13

Network Interfaces

Your Data Center

Backup
bond Layer 2
Client VLAN Switch
bond
Exadata
Infrastructure
1
VNIC
2
3
VNIC
NAT VM 4 5

SR-IOV
Database Server

Storage
Network
Storage Server

Management Storage
Network Network Switch

Management
Network Switch

Control Plane
Persistent Server (CPS)
Secure Control Plane
Automation Server Network
Oracle Cloud Region Tunnel
(outgoing)
VCN
Admin Internet
Access Switch

Notes
Your databases run on one or more virtual machine (VM) clusters that reside on Exadata
infrastructure in your data center. Exadata Cloud@Customer also includes two control plane servers
(CPS) that connect to an Oracle Cloud Infrastructure (OCI) region for cloud automation and
administration.
Note: For clarity, the diagram displays only one database server, storage server, and CPS.
The client and backup networks provide access to your VM clusters on database servers through a
layer 2 virtual local area network (VLAN) switch that you manage and bonded virtual network

Exadata Cloud@Customer Technical Architecture 14

interface cards (VNICs).
Your VMs access your dedicated Exadata storage servers through a private, nonrouted interconnect
network with SR-IOV mapped interfaces. Each physical Exadata database server and storage server
has a highly available (HA) (active/standby) connection to a pair of redundant storage networking
switches.
The database and storage servers are interconnected through a layer 2 management network and
switches. There is no direct access from the management network to your client and backup
networks. A subset of Oracle cloud automation functionality accesses your VMs through the
management network by using a VNIC interface and a network address translation (NAT) address.
The CPS connects to the storage servers through the storage network switch, and it connects to the
management network switch through the control plane network. An outgoing, persistent, secure
automation tunnel connects the CPS to the Oracle-managed admin VCN in the OCI region through
an internet access switch that you manage.
The following interfaces connect the various networks to the VM:
1. bondeth0 for the client network
2. bondeth1 for the backup network
3. eth0 for the management network
4. re0 for a storage/Real Application Clusters (RAC) interconnect (isolated through VLAN)
5. re1 for a storage/RAC interconnect (isolated through VLAN)
Related Resources
Autonomous Database on Dedicated Exadata Infrastructure
Exadata Cloud@Customer Security Controls (PDF)
Exadata Database Service on Cloud@Customer
Network Requirements

Exadata Cloud@Customer Technical Architecture 15