Introducing Exadata X9M

The Exadata X9M family, including Exadata Database Machine, Exadata Cloud@Customer, and Zero Data Loss Recovery Appliance, continue to take advantage of our latest cutting edge network technology introduced in Exadata X8M, RDMA over Converged Ethernet (RoCE). Also introduced in Exadata X8M, and continuing in X9M, is the shared storage persistent memory acceleration tier. This combination of powerful technologies in Exadata’s unique architecture has allowed Exadata System Software, the not-so-secret-sauce of Exadata, to once again increase performance for OLTP and Analytics workloads, improve consolidation abilities, enable reduced infrastructure spend, and remove management overheads for the X9M release.

We’ll get into the details in a second, but let’s first look at some workloads running on Exadata X9M:

Firstly, high-performance online transaction processing and modern workloads such as IoT, Financial Trading, and eCommerce. For this, the world’s fastest OLTP Database machine just got even faster (and not by a little!). Using sophisticated software introduced with Exadata X8M and persistent memory, we saw IOs per second reach 16 million in a single rack. Now, Exadata X9M boasts 27.6 million 8k read IOs per second, a 70% increase! Latency remains at a blistering 19 microseconds for Exadata X9M. As with all Exadata performance metrics, these are real-world, end-to-end database workload numbers, not low-level IO tools using small-sized test workloads typically used by storage vendors.

There is something for the analytics workloads too. Scan throughput (the bandwidth allowing bulk amounts of data to be processed in Exadata Storage Servers) in Exadata X9M tops the 1TB (Terabyte) per second mark in a single rack, up from 560 GB (Gigabytes) per second on Exadata X8M. An 87% increase generation over a generation! The fastest analytics database machine – now even faster.

Consolidation on Exadata becomes even more enticing with Exadata X9M. An increase in drive capacity per storage server, and increased CPU cores and memory footprint in the database servers, means you can now run more Oracle database workloads on the Exadata platform. (Did you know that Exadata allows over 4000 pluggable databases per container database? That’s a lot of room for consolidation.) Higher consolidation densities mean achieving more with less. Support more users with less infrastructure spending. Combined with lower management overheads, your savings in CAPEX and OPEX begin to multiply. More databases, in fewer floor tiles.

How are these incredible numbers achieved, and how can we satisfy the most dense workloads, you ask? It’s through Exadata’s unique software and hardware combined architecture, which pushes computing power from the database servers to the smart storage servers. As new technologies like persistent memory are engineered into the caching hierarchy alongside existing NVMe flash cache, Exadata can make the most out of the increases in PCIe and memory bus performance. And, by utilizing local hardware performance of the storage servers, and only sending relevant blocks back to the database server, it means that as storage capacity increases, so too does storage performance. Also, with Exadata, Maximum Availability Architecture is a core tenet, which means redundancy is built-in, not an afterthought.

Let’s have a look at the hardware for Exadata X9M.

Exadata Hardware

Exadata X9M-2 Database Server

Exadata X9M two-socket Database Servers use the latest 3rd Generation Intel® Xeon® Scalable Processors. Intel® Xeon® Platinum 8358 Processor, a thirty-two core processor, delivers an increase in cores of 33% over the Exadata X8M generation. Memory size in Exadata X9M-2 has increased in step with the increased CPU core density, allowing customers to maintain memory to core ratios with options of 512 GB, 1 TB, and 2 TB per server. In addition, the memory bandwidth increases by 64%, greatly accelerating memory bandwidth-sensitive operations. To help you get up and running quicker, these memory options are available at ordering time, removing the need to add the memory at installation time. Local storage on the database server has received a performance overhaul, swapping out the 4x 1.2 TB 10K-RPM hard disk drives with 2x 3.84 TB NVMe SSD drives (expandable to 4x 3.84 TB drives), allowing for faster software updates and backups. Exadata X9M sees the inclusion of PCIe4.0, which provides a significant boost to RDMA over Converged Ethernet (RoCE) networking. Speaking of client networking, additional flexibility has been incorporated into the default configuration; you can now order Oracle Exadata X9M-2 Database Servers with up to three network interface cards.

Exadata X9M-2 High Capacity Storage Server

Exadata X9M-2 High Capacity Storage Servers adopt the latest sixteen-core 3rd Generation Intel® Xeon® Scalable Processors. Memory gets an upgrade in the storage tier to 256GB, up from 192GB in the X8M. Similar to the Database Server, the memory bandwidth increases 64%. The increased memory bandwidth helps the shared persistent memory caching tier, which moves to the next generation, Intel® Optane™ Persistent Memory Series 200. PCIe4.0 in Exadata X9M boosts the Flash card performance, which also moves to the next generation, the F640v3, capacity is maintained at 6.4 TB. The RoCE Network Fabric receives an increase in performance with PCIe4.0, in step with the database server. Finally, the twelve hard disk drives increase in capacity to 18TB, up 28% from X8M.

Exadata X9M-2 Extreme Flash Storage Server

Exadata X9M-2 Extreme Flash Storage Servers keep in step with the High Capacity Storage Server, adopting the latest sixteen-core 3rd Generation Intel® Xeon® Scalable Processors, memory, and persistent memory. As with the X9M High Capacity Storage Server, the inclusion of PCIe4.0 boosts the Flash card performance, which also moves to the next generation, the F640v3. Capacity of the eight flash cards continues at 6.4 TB each. As with the High Capacity Storage Server, RoCE Network Fabric in the Extreme Flash Storage Server receives an increase in performance with PCIe4.0 as well.

Exadata X9M-2 Extended Storage Server

Exadata X9M-2 Extended Storage Servers adopts the latest sixteen-core 3rd Generation Intel® Xeon® Scalable Processor. Memory remains at 96GB. Similar to the Database and other Storage Servers, RoCE Network Fabric receives an increase in performance with PCIe4.0. The twelve hard disk drives increase in capacity to 18TB, up from 14TB in X8M, a 28% increase.

Exadata X9M-2 Storage Expansion Rack

Exadata Storage Expansion Rack adopts the latest Exadata X9M-2 High Capacity and Extreme Flash Storage Servers. Additional High Capacity, Extreme Flash, and Extended Storage Servers can be added to existing X8M or X9M Storage Expansion Racks.

Scalable Licensing of Cores

Exadata X9M allows users to limit the number of active cores in the database servers, reducing required database software licenses. At installation time, a minimum of 14 Database Server cores must be enabled on the X9M-2 servers. Oracle Exadata Deployment Assistant (OEDA) is used during installation to implement this reduction. You are then able to increase the active core count when required over time. This feature, called Capacity On Demand, has been available since Exadata X4-2 generation. The minimum number of active cores required has been constant through Exadata X5, X6, X7, X8, X8M, and now X9M.

Exadata System Software 21.2

As we announced in May this year, the Exadata System Software Release 21.2 brings further innovations to the performance and manageability of your Exadata fleet. Exadata 21.2 builds on all the innovations delivered in Exadata 20.1. New features include:

• Persistent Storage Indexes
• Persistent Columnar Cache
• Smart Scan Metadata Sharing
• ACFS IO Caching

You can read details on these updates in our announcement here and in the Exadata Database Machine System Overview. Some hardware-specific updates that weren’t mentioned, but included in 21.2 include:

Support for Exadata X9M Hardware

The Exadata X9M family requires Exadata System Software release 21.2.0.0.0 at a minimum. Additional software support for new hardware servers and components were added to Exadata 21.2.

Smart Scan Fast Decryption

Smart scans on encrypted data require temporary staging buffers to decrypt data from the IO buffer, followed by disk filtering and projection. Previous to Exadata SW 21.2, this temporary buffer is copied back to the data IO buffer for filtering and projection. Now with Exadata Software 21.2 and Exadata X9M hardware that enables new instruction sets, Smart Scan Fast Decryption implements algorithms to use the temporary buffer directly in filtering and projection. This results in 2.4x faster decryption for smart scans in X9M, and works with all currently supported Oracle Database versions.

Persistent Memory in Exadata X8M

The introduction of Persistent Memory (PMEM) marks the beginning of a revolution in the computing industry. There has always been a separation between system memory where the contents are ephemeral and byte-addressable, and storage where the data is persistent and block oriented. Persistent Memory (such as Intel Optane DC Persistent Memory) blurs the line between storage and memory by being both byte-addressable as well as persistent.

This new class of Non-Volatile Memory is fast enough to operate alongside conventional (volatile) DRAM in a DIMM (Dual In-Line Memory Module) form factor. Integrating into systems in DIMM slots means that Persistent Memory is able to play a vastly different role than conventional block-oriented storage such as Disk or SSD.

This article is the 2nd in a series covering Persistent Memory and how Oracle is using this technology to revolutionize database systems. Exadata is the first system on the market that is specifically designed to take advantage of Persistent Memory and accelerate the performance of Oracle Databases while providing full redundancy and data protection that users require for their mission-critical systems.

If you haven’t read our Persistent Memory Primer article, be sure to learn some of the fundamentals outlined in that article here: https://blogs.oracle.com/database/persistent-memory-primer

Oracle Exadata X8M

There are 3 major changes in Exadata X8M that are driving the latest performance advancements of the platform.  These changes are highlighted in RED below.

The internal fabric has been changed from 40Gbps (Gigabits per second) InfiniBand to 100Gbps RDMA over Converged Ethernet (RoCE) fabric. The RDMA (Remote Direct Memory Access) capabilities of this network fabric are critical for getting the most out of the Persistent Memory.  

Of course, Persistent Memory in Exadata resides in the storage servers, which means it’s fully redundant, and the amount of Persistent Memory scales as the system scales.  

The Linux KVM (Kernel Virtual Machine) inside of Exadata reduces the overhead of virtualization and allows the system to be configured with the largest supported memory configuration of 1.5TB per database server.

Taken in combination, these new features are keeping Exadata far ahead of the competition for database performance and price/performance. For more information on Exadata X8M, please see here: https://www.oracle.com/engineered-systems/exadata/.

Combination of RoCE and PMEM

The latest performance advances of Exadata rely on the combination of RoCE with Persistent Memory.

While each of these technologies provides benefits alone, the combination of them was required to make the next leap in the performance of the Exadata platform.

The performance of Persistent Memory is now measured in microseconds and even nanoseconds, which means other portions of the I/O stack have become much more significant. RDMA across the Converged Ethernet fabric allows Exadata to make the most of the performance of Persistent Memory. The best way to illustrate this is to look at what would happen if Persistent Memory was introduced into Exadata without RoCE.

Persistent Memory without RoCE

• Persistent Memory should deliver internal response times in the range of 6 µsec (microseconds) for processing 8K blocks of data.
• This response time is getting into the range where time spent in the existing I/O path is becoming a significant slice of the overall time.
• While Exadata has, for many years, been delivering faster I/O response times than other storage solutions, this presents an opportunity for Oracle’s development team to make some big performance improvements.
• We would normally expect about 6 microseconds (6,000 nanoseconds) read latency to access Persistent Memory from within the Exadata Storage Server.  
• However, this low latency would be overshadowed by the layers of software, context switches, and network protocol overhead.

In the diagram below, we see an older version of Oracle Database (prior to 19c) running on Exadata X8M. The database makes I/O requests from the Exadata storage as normal, but data is cached in Persistent Memory:

• The Exadata Storage Software will cache hot blocks regardless of the database version being used. Older database versions (prior to 19c) will use the conventional Exadata I/O path.
• However, Oracle Database 19c (and later versions) will use RDMA to access data directly in Persistent Memory rather than using the conventional I/O path.
• This level of integration between the database and storage is only possible due to the tight software/hardware integration of Exadata.  
• This new feature is known as the Exadata X8M Persistent Memory Data Accelerator.

Persistent Memory Data Accelerator

• Oracle Exadata X8M uses Persistent Memory (internally) to achieve dramatically higher I/O rates as well as industry-leading low-latency storage access.
• The Oracle Exadata Storage Software is fully integrated with Persistent Memory and addresses the topics discussed above while allowing Exadata X8M to take full advantage of the performance of Persistent Memory. 
• Exadata X8M is able to achieve less than 19µsec (microsecond) storage latency and 16 million IOPS (Input/Output Operations Per Second) within a single rack.
• Customers do not need to configure, tune or even choose Persistent Memory in Exadata X8M.

Persistent Memory is automatically included in all Exadata systems and there is nothing to configure or administer.

The RoCE network inside Exadata X8M enables RDMA over a Converged Ethernet fabric, accessing data residing in Persistent Memory. This combination of technologies allows Exadata to achieve near-memory speeds with storage that is fully redundant and fully protected from failures.

Persistent Memory Commit Accelerator

• Persistent Memory in Exadata also accelerates commit processing in Oracle Databases.  
• Commit processing in any database represents a performance bottleneck, especially for OLTP systems.
• If commits are slow, the entire database can be slowed down across all users and transactions system-wide. 

Persistent Memory in combination with RDMA is used to accelerate log writes in Exadata X8M, which therefore improves the performance of commits.

Exadata X8M delivers up to 8X faster log writes than the previous generation of Exadata, which was already the fastest on the market.

We can see the dramatic effects of PMEM in action by simply looking at the database performance metrics.

Seeing PMEM in Action on Exadata!

• The combination of RoCE and PMEM results in dramatically faster response times and much higher IOPS (Input/Output Operations Per Second) than other storage solutions.
• In this first release of PMEM on Exadata, the unsurpassed performance is focused on the biggest area of benefit, which is database operations that result in the highest IOPS.
• Single block reads are the highest IOPS event in any Oracle database.
• We see this as “cell single block physical read” in Exadata, which equates to “DB file sequential read” on non-Exadata systems.

The following AWR (Automatic Workload Repository) report screenshot shows this in action:

• The Exadata Storage Software will cache the most frequently read blocks of data into Persistent Memory.
• Those blocks will be accessed by pre-19c databases via the pre-existing Exadata I/O path, whereas 19c databases and later will use Remote Direct Memory Access (RDMA) to access those blocks.
• Persistent Memory is currently used as a write-through cache, so data is always persisted to Flash and Disk.
• Exadata also uses Persistent Memory to accelerate commit processing, which is one of the most performance-sensitive operations of any database, regardless of the workload.  
• The Oracle Database has to externalize commit records into redo logs and ensure those records are persisted to storage before returning control to the application, so speeding up commit processing delivers performance increases to the entire database.

There are 2 primary Oracle Database performance metrics related to commit processing as follows:

• logfile sync
• logfile parallel write

These events can be viewed in AWR reports to see the benefits of RoCE and PMEM in Exadata X8M, which delivers up to 8X faster commit processing than the previous X8 release.

Data Integrity with PMEM in Exadata

It is important to note that Exadata does not suffer from the data integrity issues outlined in MOS Note# 2608116.1. The Exadata Storage Software is designed to address the data integrity challenges that are presented by the behavior of Persistent Memory. Exadata Storage Software uses Persistent Memory in AppDirect mode with devdax (Device Direct Access), and directly manages how data is written into Persistent Memory to ensure data integrity. The Exadata Storage Software also mirrors all writes to data across storage cells to protect against data loss in the event of a failure. Oracle recommends triple mirroring, or what is known as High Redundancy in Exadata to provide the best protection, even ensuring redundancy during maintenance.