intel optane dc persistent memory devices on cisco ucs ......page 6 of 24 capability. power shell...
TRANSCRIPT
© 2020 Cisco and/or its affiliates. All rights reserved. Page 1 of 24
Intel Optane DC Persistent Memory Devices (DCPMM) on Cisco UCS Servers for
Microsoft SQL 2019 Databases
White Paper
Cisco Public
© 2020 Cisco and/or its affiliates. All rights reserved. Page 2 of 24
Contents
Executive Summary 3
Audience & Scope 3
Technology Overview 3
Cisco UCSM and IMC—Intel Optane DC Persistent Memory Devices configuration 7
Operating System—Intel Optane DC Persistent Memory Devices configuration 12
SQL Server Database use cases 19
Conclusion 24
Appendix 24
© 2020 Cisco and/or its affiliates. All rights reserved. Page 3 of 24
Executive Summary
Storage has been the most essential critical resource for many enterprise applications. Over a period, there
have been many features and advancements which enabled exponential growth in compute processing
power leaving the storage performance far behind and thereby creating huge gap between the compute
and storage performance. Additionally, ever-increasing amount of data and the need to access more data
quickly have further increased the gap. To some extent, performance gap between these two areas has
been filled by introducing an intermediate temporary storage using Non-Volatile Memory devices.
However, due to huge volumes of data, the modern applications like large system databases, IoT
applications, BigData analytical engines, need higher memory capacity to deal with larger datasets and
there is also a need to keep larger amounts of data closer to the CPU.
Intel has recently introduced a disruptive Datacenter Persistent memory technology to address the
challenges discussed above. This document provides steps for provisioning Intel Optane DC Persistent
Memory Devices (Intel Optane PMEMs) using Cisco UCS system as well as Operating Systems along with
some of the DCPMM and BIOS recommendations for specific SQL Server database deployments.
This document focuses on the following aspects.
● Configuring Intel Optane DC Persistent Memory Devices using Cisco Unified Manager (UCSM) and
Cisco Integrated Management controller (IMC)
● Steps for provisioning Intel Optane DC Persistent Memory Devices through Windows 2019 and RHEL
8 Operating Systems along with few DCPMM and BIOS configuration recommendations for SQL
Server database workloads.
● Few SQL Server database use cases which can benefit by using Intel Optane DC Persistent Memory
Devices.
Audience & Scope
This document can be referenced by IT professionals, infrastructure administrators and database
specialists who work on planning, designing and implementing Microsoft SQL Server database solutions
using Intel Optane DC Persistent Memory Devices. It is expected that readers have some knowledge about
Intel Optane persistent memory devices, Cisco UCS Manager and Cisco IMC etc.
The document focuses on configuration recommendations using Intel Optane DC Persistent Memory
Devices for SQL Server 2019 databases for specific use cases on BareMetal deployments only. This
document does not cover other management and troubleshooting aspects of Intel Optane DC Persistent
Memory Devices.
This paper does not capture performance details as the performance gains are workload dependent.
Technology Overview
Intel Optane Datacenter Persistent Memory
Intel Optane DC persistent memory is an innovative memory technology that delivers a unique combination
of affordable large capacity and support for data persistence. Intel Optane DC Persistent Modules are
based on Intel 3D XPoint non-volatile memory technology sits in between memory and storage tiers and
will deliver the best of both the worlds through the convergence of memory and storage product
characteristics. This technology introduces architectural changes in the servers where in the DC Persistent
© 2020 Cisco and/or its affiliates. All rights reserved. Page 4 of 24
Memory Modules are DDR4 socket compatible and can co-exists with conventional DDR4 DRAM DIMMs on
the same platform.
These devices can be used as regular non-volatile memory devices as well as persistent storage media
depends on how they are configured. Currently Intel Optane DC Persistent Memory Devices are supported
by Intel Xeon 2nd Generation Scalable processors and are available in three different capacities of 128GiB,
256GiB and 512GiB. These capacities are much larger alternative to DRAM which currently caps at 128
GiB. The following screen shot shows Intel DC Persistent memory Module.
Operating modes of Intel DC Persistent Memory Modules
Intel DC Persistent Memory Modules (PMEM) can be configured as Memory mode, App Direct Mode, or a
combination of both the modes (also referred as Mixed mode/Dual mode).
Memory Mode: In this mode, Intel Optane PMEMs act as nonvolatile memory and since these devices are
available in larger sizes, it enables larger memory capacities available to the Operating Systems. Hence it is
kind of memory-extender while operating as DRAM like speed. The actual DRAM is used as Cache for
DCPMM memory. CPU memory controller uses the DRAM as cache and the Intel Optane DC persistent
memory as addressable main memory. Hence Operating system sees Intel Optane DC Persistent memory
as main memory while the DRAM memory is invisible to the OS.
Since the Intel Optane PMEMs are available in large capacities than the DDR4 based DRAM devices, the
system will have large memory capacities up to 6TiB in two-socket server and 12TiB in 4-socker server.
This huge memory capacity within a single server enables certain memory bound applications, like In-
Memory databases, traditional large working set database and Bigdata applications etc., to deal with larger
datasets. It also enables customers to consolidate many virtual machines in a single server platform in
virtualization environments.
App Direct Mode: In this mode, Intel Optane PMEM devices act as persistent storage media and provides
blazing IO performance to the applications. In App Direct mode, these devices can be configured in the
following ways.
Interleaved sets: When Intel Optane PMEM Devices configured as interleaved sets, all the PMEM devices
within the socket are consolidated and presented as single logical disk to the Operating System enabling
read/write operation striped across all the PMEM devices. For SQL Server databases deployments, it is
recommended to use PMEM in this fashion for better IO performance and reduced maintenance.
Non-Interleaved sets: When Intel Optane PMEMs are configured as Non-Interleaved fashion, each
individual PMEM device is presented to the Operating System as individual logical storage disk. This option
gives more granularity and better control over data placement. For SQL Server databases, this gives more
control on how data and log files can be distributed across the individual PMEM disks.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 5 of 24
In the App Direct mode, Intel Optane PMEM storage can be used in two different ways. Firs, they can be
used as regular block storage in which Intel Optane PMEM storage is exposed as block storage like
traditional SATA/SAS and NVMe block devices. They can be formatted with traditional file systems like
NTFS, ReFS, XFS and EXT4, and can be consumed by any legacy applications. It does not need any
application level changes and hence it provides better adaptability of these devices in the market.
Intel Optane PMEMs can also be used as byte-addressable devices when formatted with Direct Access
(DAX) file system option. Dax is a mechanism that enables Operating Systems to get direct access to the
files stored on persistent memory devices by implementing memory mapping of stored files. Direct access
to the PMEM devices bypasses the tradition kernel IO subsystem resulting in breakthrough IO acceleration.
Some of the persistent memory aware applications, such as Microsoft SQL Server databases, will greatly
benefit from this type of access as the read and write requests to such devices do not go through the
software stack of the file system. Instead such applications can directly access these devices using user
space load/store methods which are typical memory related operations involved in reading and writing
pages from DRAM memory.
Note that DAX mode comes at the cost of allocating per-page metadata required for memory mapping of
the files stored on these PMEM devices. For every 4KiB, 64 Bytes are required for maintaining the memory
mapping data structures. This additional storage can be allocated from either DRAM memory or reserved
portion of persistent memory itself. For instance, the amount of metadata storage requirement for a 3TiB
persistent volumes is 48GiB. On Linux systems, while creating namespaces on the PMEM devices, use --
map=mem option for consuming DRAM memory and –map=dev for consuming PMEM storage itself.
Because Intel Optane PMEMs are available in larger capacities, the metadata storage for memory mapping
can be stored on PMEM volumes itself. If there is enough DRAM memory available in the system, DRAM
memory can be used which will also result in faster access to the metadata.
Mixed Mode: This mode is the combination of both Memory and App Direct modes. It allows us to
configure percentage of DCPMM capacity used in Memory Mode and the remaining in App Direct Mode.
This mode provides flexibility to customers to split the persistent memory capacity to best fit their business
needs.
Intel Optane DC Persistent Memory Devices can be managed and monitored configured using two ways.
1. Infrastructure management tools like Cisco UCS and IMC.
2. Operating Systems utilities like ipmctl (Linux), ndctl and PowerShell (Windows) etc.
Red Hat Enterprise Linux 8
RHEL 8 provides support for a wide range of new and innovative technologies, which includes support for
Intel DC Persistent Memory. Support for PMEM has been available since RHEL 7.3 as block storage and
enhanced to great extent in RHEL 8 and supports DAX option. DAX option is supported by XFS and EXT4
filesystems. An official support statement from Red Hat can be found at:
https://access.redhat.com/articles/4070821. ipmctl and ndctl utilities can be used to manage PMEM
devices.
Windows Server 2019
Intel Optane Persistent Memory is fully supported in Windows Server 2019. Memory and App Direct modes
are supported Window 2019. Persistent Memory devices are discovered by a new media type called
Storage Class Memory (SCM). In the App Direct mode, NTFS and ReFS filesystems supports these devices
when PMEMs configured in traditional block storage devices while only NTFS filesystem supports DAX
© 2020 Cisco and/or its affiliates. All rights reserved. Page 6 of 24
capability. Power shell command-lets can be used to manage the Intel Optane DC Persistent Memory
Devices.
SQL Server 2019
SQL Server 2019 is the latest release from Microsoft which includes support for Persistent memory devices
in both Linux and Windows Operating systems. SQL Server Hybrid Buffer Pool and Enlightenment mode are
the new features that leverages Intel DC Persistent Memory devices for blazing IO performance. Tail of Log
(also referred as Transaction commit acceleration) is another use case which can leverage PMEMs and
supported since SQL Server 2016 SP1. Upcoming sections will provide more details on different use cases
which can benefit from using Intel DC Persistent memory devices.
Cisco UCSM
Cisco UCS Manager (UCSM) provides unified, embedded management for all software and hardware
components in the Cisco UCS. Using Single Connect technology, it manages, controls, and administers
multiple chassis for thousands of virtual machines. Administrators use the software to manage the entire
Cisco Unified Computing System as a single logical entity through an intuitive GUI, CLI, or an XML API.
Cisco UCS Manager resides on a pair of Cisco UCS 6300 and 6400 Series Fabric Interconnects using a
clustered, active-standby configuration for high availability.
Cisco UCS Manager offers unified embedded management interface that integrates server, network, and
storage. Cisco UCS Manager performs auto-discovery to detect inventory, manage, and provision system
components that are added or changed. It offers comprehensive set of XML API for third part integration,
exposes 9000 points of integration and facilitates custom development for automation, orchestration, and
to achieve new levels of system visibility and control.
Service profiles benefit both virtualized and non-virtualized environments and increase the mobility of non-
virtualized servers, such as when moving workloads from server to server or taking a server offline for
service or upgrade. Profiles can also be used in conjunction with virtualization clusters to bring new
resources online easily, complementing existing virtual machine mobility.
For more information about Cisco UCS Manager, go to: https://www.cisco.com/c/en/us/products/servers-
unified-computing/ucs-manager/index.html
Cisco Integrated Management Controller (IMC)
The Cisco® Integrated Management Controller is a baseboard management controller that provides
embedded server management for Cisco UCS™ C-Series Rack Servers and Cisco S-Series Storage
Servers. The Cisco IMC enables system management in the data center and across distributed branch-
office locations. It supports multiple management interfaces, including a Web User Interface (Web UI), a
Command-Line Interface (CLI), and an XML API that is consistent with the one used by Cisco UCS
Manager. IMC also supports industry-standard management protocols, including Redfish v1.01, Simple
Network Management Protocol Version 3 (SNMPv3), and Intelligent Platform Management Interface Version
2.0 (IPMIv2.0).
Cisco IMC and UCS Manager release 4.0(4) introduces support for Intel Optane DC Persistent Memory
Devices on the Cisco UCS M5 servers that are based on the Intel Xeon Second Generation Scalable
Processors. Cisco UCS policies allows us to configure Intel Optane PMEM devices in various modes as
discussed above sections and allows it to be consumed in the Server profiles which will be applied to the
blade or rack server.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 7 of 24
Intel Persistent Memory devices can also be configured using Cisco CIMC when IMC is used to manage
Cisco UCS rack server in stand-alone mode (Not UCSM managed). It supports various ways to configure
Intel Optane DC Persistent Memory Devices like IMC GUI, CLI and XML etc.
Cisco UCS M5 Server for Intel Optane DC Persistent Memory Devices
Various Cisco UCS M5 rack and blade servers which are based on Intel 2nd Gen Scalable Processors
support Intel Optane PMEMs in all the operating modes described in the above sections. The list of servers
that support Intel Optane PMEMs:
● Cisco UCS C240 M5
● Cisco UCS C220 M5
● Cisco UCS B200 M5
● Cisco UCS C480 M5
● Cisco UCS B480 M5
● Cisco UCS S3260 M5
Cisco UCSM and IMC—Intel Optane DC Persistent Memory Devices configuration
This section provides steps for configuring Intel Optane PEMs using Cisco UCSM GUI and Cisco IMC.
Note: Intel Optane PMEMs can also be configured using in various ways Cisco IMC CLI, IMC XML, UCSM
CLI. For complete management and troubleshooting steps for configuring Intel Optane PMEM, refer the
link: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/persistent-
memory/b_Configuring_Managing_DC-Persistent-Memory-Modules.html
Configuring Intel Optane DC Persistent Memory Devices using Cisco UCSM policies
Cisco CIMC and UCS Manager release 4.0(4) introduces support for Intel Optane PMEM on the Cisco UCS
M5 servers that are based on the Intel Xeon 2nd Generation Scalable Processors.
To ensure the best server performance, it is important that to follow memory performance guidelines and
population rules before you install or replace persistent memory modules.
Refer following link for more details on the population guidelines for 2-socker based servers ( C240M5,
C220M5 and B200 M5):
https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/hw/C240M5/install/C240M5/C240M5_
chapter_010.html - concept_v1f_mtr_tgb
Refer follow link for more details on the population guidelines for 4-socker based servers ( C480M5 and
B480M5):
https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/hw/C480M5/install/C480M5/C480M5_
chapter_011.html - concept_v1f_mtr_tgb
Once the server is populated with DDR4 & Intel Optane PMEMs , below steps to be followed to do the
configuration.
UCSM Policy for configuring Intel Optane DC Persistent Memory Devices
To configure Intel Optane PMEM devices using Cisco UCSM, a persistent memory policy needs to be
created. This policy will be consumed by a Service Profile which will be associated to a rack or blade
© 2020 Cisco and/or its affiliates. All rights reserved. Page 8 of 24
server. Below are the steps for creating a UCMS policy for Intel Optane DC Persistent Memory Devices
configuration.
1. Log on to the UCSM and navigate to servers -> Policies -> root -> Persistent Memory Policy. Right
click on the Persistent Memory Policy and select create option. On the new prompted window enter
name of the Policy and Description.
2. Under general tab, Click on Add button to create a Goal.
3. Enter 100 under Memory Mode (%) for completely configuring Intel Optane PMEM as volatile
memory. “Persistent Memory Type” setting is not applicable for Memory Mode. Also creating
Namespace is not applicable for when PMEMs are used in memory mode.
4. Enter 0 under Memory Mode (%) for completely configuring Intel Optane PMEM as Persistent
memory.
5. Select App Direct or App Direct NonInterleaved options depending on the workload requirement.
The following figure shows how to configure Intel Optane DC Persistent Memory Devices as Interleaved App Direct mode using UCSM policy.
6. It is a good practice to create namespaces using vendor-agnostic utilities like ndctl in RHEL and
PowerShell in Windows. Hence skip the Namespace creation at this stage. Creating Namespaces
will be covered in detail in the corresponding sections of Windows 2019 and RHEL 8.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 9 of 24
7. To secure the Intel Optane PMEMs, click on “Create Local Security” and enter Secure Passphrase
and Deployed Secure Passphrase and click Ok.
8. Click Ok to close the Create Policy window.
Using DCPMM Policy in Service Profile
Once UCSM policy for Intel Optane DC Persistent Memory Devices is created, it needs to be used in a
service profile and then apply the service profile to a server. Creation of full server profile is out of the
scope of this document (refer UCSM documentation). To use the DCPMM policy in the service profile,
follow the below steps.
1. Go to Server tab in the UCSM and select the service profile in which you want to include the
persistent memory service profile.
2. In the Work pane, click the Policies tab.
3. In the Policies area, expand Persistent Memory Policy.
4. From the Persistent Memory Policy drop-down list, select the persistent memory policy that you
want to include in this service profile
5. Click Save Changes.
Finally, apply the service profile to a server which has the Intel Optane persistent memory DIMMS installed.
The following screen shot shows applying “DCPMM-AppDirect” persistent memory policy to a sample
service profile named as ‘TEST1’.
The following two screen shot show more details and temperature statistics of Intel Optane DCPMM
devices collected by Cisco UCSM manager.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 10 of 24
Once the service profile is applied to the server successfully, install required Operating System and then
Namespaces can be configured using steps explained in the following sections.
Configuring Intel Optane DC Persistent Memory Devices using Cisco CIMC
This section provides steps to configure Intel Optane DC Persistent Memory Devices devices using Cisco
IMC.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 11 of 24
1. Log on to the Cisco IMC web page of a stand-alone server and click on Compute on the left pane.
On the right-hand side pane, click on persistent Memory tab. This tab will display all the Intel
Optane DC Persistent memory devices under DIMM Details section.
2. Click on “Configure Memory Usage” on the top. This opens up a pop up to configure the Intel
Optane PMEMs.
3. On the Configure Memory Usage window, enter value for Memory Mode text box. Enter ‘100’ to
configure Intel Optane PMEMs in Memory mode. Enter ‘0’ to configure PMEMs in App Direct mode.
Persistent Memory Type Option is enabled only when the Intel Optane PMEMs are configured in App
Direct mode. From the drop down list, select either App Direct (interleaved) or
AppDirectNonInterleaved option based on requirement as shown in the below figure.
4. Click on ‘Create Goal” button to create the goal.
5. Do not create any Namespaces now. As mentioned earlier in the UCMS section, it is recommended
to create Namespace within Operating System.
6. Click on ‘Save’ to save the configuration. IMC will now reboot the server to apply the configuration.
7. Once the server is rebooted, you can view the Region details when Intel Optane PMEMs are
configured in AppDirect Mode as show in the following screenshot.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 12 of 24
Once the server is online, install required Operating System and then Namespaces can be configured using
steps explained in the below sections.
Operating System—Intel Optane DC Persistent Memory Devices configuration
Several Operating Systems and different distributions of Linux include support for both App Direct and
Memory mode. Refer the complete list of OS support for Intel Optane DC Persistent Memory Devices:
https://www.intel.com/content/www/us/en/support/articles/000032860/memory-and-storage/data-
center-persistent-memory.html
This section provides steps to provision Intel Optane PMEMs in RedHat Enterprise Linux 8 and Windows
Server 2019. In RHEL8, ipmctl and ndctl utilities are used to provision Intel Optane PMEMs while in
Windows 2019, ndctl utility or Power shell command-lets are used. Make sure to install ipmctl and ndctl
tools in RHEL 8.
Note that creating Namespaces is only applicable when Intel Optane PMEMs are configured as either “App
Direct” or “App Direct Non-Interleaved” mode. Namespaces are not applicable when DCPMM is
configured in Memory Mode.
Note that the prerequisite before proceeding with the following steps is to create a goal (which ultimately
creates regions automatically) using either UCSM or IMC.
Configuring Intel Optane DC Persistent Memory Devices for RHEL
This section provides steps to create Namespaces using in RedHat Enterprise Linux 8 Server.
In the supported Linux distributions for persistent devices, ipmctl and ndctl are the two tools being used to
completely manage the goals and namespaces. However, since goal is already configured using UCSM or
IMC at the platform level, the following sections covers listing goals using ipmctl and managing
namespaces using ndctl tool.
Open command shell with root login and follow the below steps to create namespaces or persistent
memory disks in RHEL 8 server.
1. Ipmctl tool can be to manage the Intel Optane DC Persistent Memory Devices in the Linux based
Operating Systems. Use ipmctl show -topology command to list both DDR4 DRAM and Intel Optane
PMEM devices as shown below.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 13 of 24
2. The ipmctl show -dimm command displays the persistent memory modules discovered in the
system and verifies that software can communicate with them.
3. To check the capacity provisioned for use in different operating modes, use ipmctl show -
memoryresources command. The MemoryCapacity and AppDirectCapacity values can be used to
determine if the system was configured in Memory mode, App Direct mode, or mixed mode. The
screen shot below shows that the persistent memory modules are currently configured in App Direct
mode.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 14 of 24
Note that ipmctl is powerful utility which can be used to create/delete goals and capable of performing other troubleshooting actions as well. For the complete list of supported commands by ipmctl execute ipmctl show -help.
Since Goal is already configured using Cisco UCSM/IMC at the infrastructure level, it is not advised to create goal using ipmctl tool. Therefore, the next step is to proceed with creating name spaces (partitions) using ndctl utility.
4. ndctl tool gives better control and more options when provisioning Persistent memory devices. one
should focus on the following options of ndctl utility for optimal configuration of Persistent Memory
Devices.
◦ --mode=fsdax: Filesystem-DAX mode is the default mode of a namespace when specifying ndctl
create-namespace with no options. It creates a block device (/dev/pmemX[.Y]) that supports the DAX
capabilities of Linux filesystems (xfs and ext4 to date). This needs PMEM aware applications to be able
to benefit from the Direct access provided by DAX option.
◦ --mode=raw: Raw mode is effectively just a memory disk that does not support DAX. This mode is
compatible with other operating systems, but again, does not support DAX operation. This mode can
be used for legacy applications.
◦ --map=mem: When PMEMs are configured with fsdax mode, the metadata required for maintaining
memory mapping will be stored in DRAM memory. Use this option when you have enough DRAM
memory available in the system.
◦ --map=dev: When PMEMs are configured with fsdax mode, the metadata required for maintaining
memory mapping, will be stored in PMEMs itself.
Note that --map option is applicable only when fsdax mode is used. Other modes do not maintain metadata hence does not need additional storage.
5. The following screen shot shows two namespaces created with fsdax option and stores the
metadata in DRAM memory (--map=mem). This was tested on a two socket Cisco C240 M5 server
populated with 12x 64G DDR4 and 12x 512G Intel Optane PMEMs. Verify that Namespaces are
created with fsdax option in order use the PMEM for direct access byte addressable storage.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 15 of 24
Note that multiple Namespaces (partitions) can created in a region, The size of the namespace is same as the region size minus size of the metadata by default. We can specify size of the namespace by using --size option.
6. Once a name space is created with DAX option, Linux PMEM drivers creates new device in
/dev/pmemX.Y format. X represents Region id and Y represents Namespace number which is
sequentially increasing number.
7. To view the existing Namespaces, execute ndctl list. To list all the Namespaces in the given Region,
execute ndctl list –region <region Id>.
8. Once a Namespace is created on the PMEM device, format the volumes with a filesystem. RHEL8
supports xfs and Ext4 filesystem with fsdax option. The below screen shot shows formatting PMEM
devices with XFS filesystem and mounting with --o dax option and finally setting 2MB as stripe unit
size which is recommended for SQL Server database deployments.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 16 of 24
9. When planning to leverage Intel Optane PMEMs for SQL Server databases on Linux platforms, it is
recommended enable the SQL Server trace flag 3979 as per the recommendation suggested by
Microsoft: https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-configure-pmem?view=sql-
server-ver15
For other SQL on Linux best practices: https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-
performance-best-practices?view=sql-server-linux-2017
Configuring Intel Optane DC Persistent Memory Devices for Windows
The following sections provides steps to create Namespaces using Powershell commands in Windows
Server 2019.
Open Powershell with administrator rights and follow the below steps to create namespaces or persistent
memory disks in Windows server 2019.
1. First list all the available Intel Optane PMEM devices in Windows server by running “Get-
PmemPhysicalDevice” as shown below.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 17 of 24
2. Execute “Get-PmemUnusedRegion” to list the regions created by the goal that you set in the server
platform using UCSM or IMC.
3. Next step is to create a Namespace or persistent memory disk using New-PmemDisk command let.
It accepts the RegionId as input which you can pipe it using Get-PmemUnusedRegion. If you do not
pass the specific RegionId, it creates one Persistent Device for each unused region.
Note that when New-PmemDisk is executed with “-AtomicityType BlockTranlationTable” option, AppDicrect storage device will be used as traditional storage device where access to these devices needs to go thorugh the complete IO stack of filesystem.
4. Once the persistent memory volumes created as shown above, the device manager and Disk
Management UI will list all the available Persistent memory devices as shown below.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 18 of 24
5. Next step is to initialize, create partition and format the persistent memory devices with DAX option.
Microsoft recommends using the largest allocation unit size available for NTFS (2 MB in Windows
Server 2019) when formatting PMEM devices as referenced here: https://docs.microsoft.com/en-
us/sql/database-engine/configure-windows/hybrid-buffer-pool?view=sql-server-ver15 - best-
practices-for-hybrid-buffer-pool
© 2020 Cisco and/or its affiliates. All rights reserved. Page 19 of 24
6. Verify if the device is formatted for DAX by using fsutil.exe utility as shown above.
Now the persistent memory devices have been configured with DAX option and ready for consumption as
storage by SQL Server databases.
SQL Server Database use cases
Different configuration modes of Intel Optane DC Persistent memory allow us to use them in different
databases use cases. The following sections discusses few use cases in which SQL Server databases can
be deployed for getting maximum benefit from Intel Optane DC Persistent memory devices when
configured in App Direct mode.
Persistent Memory Devices for Transaction Log Files and Tail of Log Caching
In Large scale transactional systems, log commit rate is one of the important factors that can affect the
overall transactional throughput. The faster the log performance (hardening log buffers to persistent
storage media), the higher is the transaction throughput. Intel Optane DC Persistent Memory Devices can
be leveraged to improve SQL Server transaction throughput by using them as persistent storage medium.
Persistent Memory Devices as traditional Block storage
When Intel Optane DC Persistent Memory Devices formatted without DAX option, they can be accessed as
regular block devices just like any other SATA/SAS/NVMe devices. Intel Optane PMEM devices, which are
installed on the memory bus closer to the CPUs, provide much lower device access latencies than today’s
fastest NVMe storage devices. Also, this option does not need any application changes. Old legacy
applications can leverage these high performing persistent memory devices for data persistence without
any changes in the application.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 20 of 24
Typically, in a traditional high-volume transitional system, transaction log becomes bottleneck as log
buffers need to be hardened to the persistent media before committing any transaction. When transaction
log files of such databases are migrated from legacy block devices to the persistent memory devices, the
transaction throughput can be improved by multi fold due to lower access latencies of persistent memory
devices.
Moving user database transaction log file to the persistent memory device is a simple step which can be
referred here: https://docs.microsoft.com/en-us/sql/relational-databases/databases/move-user-
databases?view=sql-server-ver15
Tail of Log
Tail of Log Caching feature (also known as Transaction commit acceleration) was first introduced in SQL
Server 2016 SP1 in combination with Windows Server 2016 when NVDIMMs were introduced to the
market.
When Persistent memory devices are formatted using filesystem with DAX option, Operating system allows
byte-addressable direct access to the Intel Optane PMEM devices by eliminating IO kernel software stack.
This benefits PMEM aware applications, such as SQL Server, to directly access these devices at much
lesser latencies using user space load/store commands. This enables SQL Server to directly store log
buffers (typically ranges 512 Bytes to 60KB) on PMEM storage which gives DRAM-like performance. Since
PMEM devices are natively persistent, there is no need to flush log buffers again to the other persistent
media (like SSDs/HDDs). Typically, the log flushing to traditional persistent devices (SSDs/HDDs) is the
costly IO operation and have been the main bottleneck in high transactional systems which is now avoided
to a large extent using PMEM storage, thereby increasing the transaction throughput by multi folds.
For more details on Tail of Log Caching implementation in SQL Server 2016, refer the link:
https://docs.microsoft.com/en-us/archive/blogs/bobsql/how-it-works-it-just-runs-faster-non-volatile-
memory-sql-server-tail-of-log-caching-on-nvdimm.
This use case does not need much PMEM storage as Intel Optane DC Persistent Memory Devices are
available in much bigger sizes (compared to NVDIMMs). Also Tail of Log caching needs only 20MB size of
PMEM storage for adding additional Log file on the PMEM storage. Hence select least size (128G) Intel
Optane DC Persistent Memory Devices and populate them in 2-1-1 fashion as explained here:
https://www.intel.com/content/dam/support/us/en/documents/memory-and-storage/data-center-
persistent-mem/Population-Configuration.pdf
Once the persistent memory volume is configured and formatted as a DAX option, all that remains is to add
a new log file to the database using the same syntax as any other log file, where the file resides on the
DAX volume. The log file on the DAX volume will be sized at 20MB regardless of the size specified with the
ADD FILE command as shown below:
© 2020 Cisco and/or its affiliates. All rights reserved. Page 21 of 24
In order to disable the persistent log buffer feature, remove the log file from the DAX volume using below
T-SQL commands:
ALTER DATABASE < User-DB> SET SINGLE_USER
ALTER DATABASE < User-DB> REMOVE FILE < Name of DAXLOG >
ALTER DATABASE < User-DB> SET MULTI_USER
Any log records being kept in the log buffer will be written to disk, and at that point there is no unique data
in the persistent log buffer, so it can be safely removed.
Note that as Intel Optane DC Persistent Memory Devices are available bigger sizes, complete database can
now be stored on the DAX enabled PMEM storage volumes. The following sections discuss use cases and
features which can benefit from large Intel Optane DC Persistent Memory Device sizes.
Hybrid Buffer Pool (HBP)
Hybrid Buffer Pool is all about extending SQL Server limited DRAM-based buffer pool sizes to much bigger
sizes by memory mapping all the database files onto Persistent Memory devices which are near DRAM
performant and natively persistent too.
Hybrid Buffer Pool is a new feature introduced in SQL Server 2019. It requires Persistent Memory devices
(PMEM) to be formatted with DAX option. It allows database engine to directly access data pages in
database files stored on persistent memory devices. This option is currently supported by NTFS, XFS and
EXT4 files systems.
When databases files are stored on the PMEM devices, SQL Server will automatically detect if data files
reside on an appropriately formatted PMEM device and perform memory mapping in user space. This
mapping happens during startup, when a new database is attached, restored, created, or when the hybrid
buffer pool feature is enabled for a database.
In a traditional system without PMEM, SQL Server caches data pages in the buffer pool. With hybrid buffer
pool, SQL Server skips performing a copy of the page into the DRAM-based portion of the buffer pool, and
instead accesses the page directly on the database file that lives on a PMEM device. Read access to data
© 2020 Cisco and/or its affiliates. All rights reserved. Page 22 of 24
files on PMEM devices for hybrid buffer pool is performed directly by following a pointer to the data pages
on the PMEM device.
Only clean pages can be accessed directly on a PMEM device. When a page is marked as dirty it is copied
to the DRAM buffer pool before eventually being written back to the PMEM device and marked as clean
again. This will occur during regular checkpoint operations. Hybrid buffer pool feature is available for both
Windows and Linux. Read heavy workloads with larger working set which require more memory would
benefit from this feature. Below section lists the steps for configuring Hybrid Buffer Pool.
Configuring Hybrid Buffer Pool
In SQL Server 2019, Hybrid Buffer Pool (HBP) must be configured first at instance level (server-scoped).
Then HBP can be enabled or disabled at individual database level (database-scoped).
Use the following T-SQL command to first enable the Hybrid Buffer Pool at instance level.
ALTER SERVER CONFIGURATION SET MEMORY_OPTIMIZED HYBRID_BUFFER_POOL = ON;
Note that HBP is disabled by default at the instance level. For the setting change to take effect, SQL Server
instance must be restarted. A restart is needed to facilitate allocating enough hash pages, to account for
total PMEM capacity on the server.
The next is to enable HBP for the individual user databases by using the below T-SQL command. It is
assumed that the database files, for which HBP is enabled, are stored on DAX enabled persistent memory
volumes.
ALTER DATABASE <user-DBName> SET MEMORY_OPTIMIZED = ON;
By default, hybrid buffer pool is set to enable at all the user databases as well as system databases. You
can disable HBP for system databases manually.
Note that if the instance-scoped setting for the hybrid buffer pool is set to disabled, the hybrid buffer pool
will not be used by any user database.
HBP can be disabled at the instance level and individual databases using following commands.
When HBP is disabled at the instance level, it is required to restart the SQL instance for the change to take
effect.
ALTER SERVER CONFIGURATION SET MEMORY_OPTIMIZED HYBRID_BUFFER_POOL = OFF;
ALTER DATABASE < user-DBName > SET MEMORY_OPTIMIZED = OFF;
The following screen shot shows T-SQL commands to enable HBP at instance level and database level and
shows some important HBP related messages captured in the SQL Server error log.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 23 of 24
Note: Please ensure to enable trace flag 1810 in order to view the memory mapping information of
database files in the SQL Server errorlog files.
For optimal performance, while creating user databases, make sure the sizes of database files are created
in multiples of 2MB as the volumes are formatted with 2MB allocation unit size in windows as well as RHEL.
In Windows environments, enable Lock Pages in Memory setting for SQL Server service account using
Windows Local Group Policy editor (gpedit.msc). Refer the link for configuring this option:
https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/enable-the-lock-pages-in-
memory-option-windows?view=sql-server-ver15
When you enable HBP for a database, it is recommended to store all the data and transaction log files on
the DAX enabled storage volumes. Storing few data files in DAX enabled storage volumes and few data
files on traditional block storage volumes is not supported.
To view the current HBP configuration, execute the below command to report HBD status for each
database.
SELECT * FROM sys.server_memory_optimized_hybrid_buffer_pool_configuration; go
SELECT name, is_memory_optimized_enabled FROM sys.databases; go
SQL Server 2019 Enlightenment Mode
As discussed earlier, when Intel Optane DC Persistent Memory Devices formatted with DAX option in the
AppDirect mode, offer byte-addressable storage bypassing the Kernel IO stack. SQL Server 2019
leverages this feature (called enlightenment mode) which allows us to store data and log files directly on
the DAX formatted volumes there by enabling faster access at much lower latencies and better
transactional throughput. As of today, SQL Server 2019 enlightenment mode is fully supported on Linux
operating systems only (refer here: https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-
configure-pmem?view=sql-server-ver15). To leverage this feature, create a database on DAX formatted
volumes and it automatically get enabled for enlightenment mode.
© 2020 Cisco and/or its affiliates. All rights reserved. Page 24 of 24
Conclusion
The new Intel Optane DC persistent memory redefines traditional architectures, offering a large and
persistent memory tier at affordable cost. With breakthrough blazing IO performance, higher capacity
persistent memory and combined with Second Gen Intel Xeon Scalable processors, Intel Optane DC
persistent memory accelerates IT transformation to support the demands of the data era, with faster-than-
ever-before analytics, cloud services, and next-generation communication services. PMEM aware
application, like SQL Server databases, will get maximum benefit from the Intel optane DC Persistent
memory. It is recommended to test applications with Intel Optane DCPMM options before production
rollout, to ensure the performance gains as the mileage may vary based on different parameters such as
data volumes, user concurrency etc
Appendix
Linux utilities
Refer the below github links for ipmctl and ndctl utilities.
https://github.com/intel/ipmctl
https://github.com/pmem/ndctl
Recommended BIOS Setting when using Intel Optane DC Persistent Memory
The following list provides recommended BIOS settings when using Intel DC Persistent Memory devices in
App Direct mode.
For SQL Server database deployments on Cisco UCS M5 server with Intel Optane Persistent Memory
Modules, the following tables provides some of the BIOS options are recommended for high performance.
Package C State Limit C0 C1 State
Processor C-States (C, C1E, C3, C6, C7) Set processor C State
Sub NUMA Clustering Disabled
IMC Interleave Auto
XPT, UPI, LLC Prefetch Enabled
Turbo Mode Enabled
Hardware P-States HWPM Native Mode
Power Performance Tuning OS
Energy/Performance Bias Config Balanced Performance
Processor EPP profile Performance
Printed in USA 219376-15 02/20