hpe 3par adaptive flash cache technical white paper

HPE 3PAR Adaptive Flash Cache

Contents Executive summary ................................................................................................................................................................................................................................................................. 2

Definitions ........................................................................................................................................................................................................................................................................................ 2

Limitations ....................................................................................................................................................................................................................................................................................... 2

IO workloads .................................................................................................................................................................................................................................................................................. 2 Using Adaptive Flash Cache to increase read hits and improve random read performance ...............................................................................3

What gets cached in Adaptive Flash Cache?....................................................................................................................................................................................................3

Moving data from a node’s DRAM read cache into Adaptive Flash Cache ......................................................................................................................... 4

Data in Adaptive Flash Cache........................................................................................................................................................................................................................................ 5

Demoting LRU queues .................................................................................................................................................................................................................................................. 5

Creating and configuring Adaptive Flash Cache .......................................................................................................................................................................................... 6 Createflashcache command use ........................................................................................................................................................................................................................... 6

Showflashcache command use .............................................................................................................................................................................................................................. 7

Setflashcache command use .................................................................................................................................................................................................................................... 8

Removeflashcache command use ...................................................................................................................................................................................................................... 11

Flash Cache statistics .......................................................................................................................................................................................................................................................... 12

Adaptive Flash Cache statcache command ............................................................................................................................................................................................. 12 Display CMP (DRAM) and FMP (Flash Cache) statistics on a per node basis with the statcache command ............................. 12

Display CMP (DRAM) and FMP (Flash Cache) statistics on a VV basis with the statcache –v command .................................... 13

Display metadata statistics by adding –metadata to the statcache –v command ............................................................................................... 14

Increasing and decreasing the size of Flash Cache ................................................................................................................................................................................. 16

Balancing Flash Cache after adding SSDs to the system ................................................................................................................................................................... 16

Adaptive Flash Cache simulator ................................................................................................................................................................................................................................ 16 HPE 3PAR Adaptive Flash Cache vs. HPE 3PAR Adaptive Optimization ........................................................................................................................... 17

Technical white paper


Executive summary One of the most difficult workloads for a storage array to handle is a random read workload. A “random read workload” refers to a sequence of read requests where the sequence of future read requests cannot be predicted based on previous read requests. Random read workloads are independent of the size of the IO request, but it is the generally accepted norm that the IOs in a random read workload are small block (<64 KiB). For random read workloads, assuming the active working set size is large enough compared to the amount of cache on the array, data generally needs to be read from the back end storage to satisfy the read request. Although a properly sized array assumes no cache hits for random read workloads it is in fact the case, in most situations, that some limited percentage of the random read workload will be serviced from cache (i.e., a block of data that was read previously will be reread by the server before it is flushed from read cache). By increasing the size of the primary read cache on an array or by adding a large second level of read cache (Level-2 cache), it is possible to increase the probability that a block of random read data to be serviced multiple times out of a much faster cache tier than the spinning media where it resides on the back end of the array. HPE 3PAR Adaptive Flash Cache (AFC) is a built-in array functionality of the HPE 3PAR StoreServ that does this by using capacity on solid-state drives (SSDs) (flash) to act as Level-2 read cache holding random read data that has been removed from DRAM read cache.

An additional benefit of Adaptive Flash Cache is that it may increase the overall IOPS and lower the overall read IO latency an array can deliver by “unloading” a percentage of random read IOPS from spinning media on the back end of the array to much faster flash media. Using SSDs as a Level-2 read cache to hold random read data that has been removed from DRAM cache is a cost-effective way of keeping more random read data on very fast media to improve overall read performance.

HPE 3PAR Adaptive Flash Cache is available free of charge on all HPE 3PAR StoreServ arrays running HPE 3PAR OS 3.2.1 and later. Adaptive Flash Cache operates on small block random read data only (read data whose IO size is < 64 KiB). This white paper is intended to explain the operation of Adaptive Flash Cache technology on HPE 3PAR StoreServ arrays and to propose excellent methods for enabling Adaptive Flash Cache in a customer environment so that applications achieve greater benefit from the use of this technology.

Definitions AFC—Adaptive Flash Cache. HPE 3PAR implementation that uses flash (SSD) storage as a Level-2 read cache on an HPE 3PAR StoreServ array. AFC is available beginning with the 3.2.1 release of the HPE 3PAR StoreServ OS.

CMP—Cache Memory Page. This refers to a 16 KiB page of physical DRAM cache memory that contains data for either a read IO or a write IO.

DRAM—Dynamic random-access memory.

FMP—Flash Memory Page. This refers to a 16 KiB page of physical flash in the Flash Cache.

SSD—Solid-state drive. This “disk drive” uses non-volatile flash memory rather than a rotating platter to store blocks of data.

Limitations HPE 3PAR Adaptive Flash Cache feature is included as part of the HPE 3PAR OS and is supported on all HPE 3PAR StoreServ Storage systems that are supported with HPE 3PAR OS 3.2.1 and above. The minimum amount of Flash Cache that can be specified on an HPE 3PAR StoreServ array regardless of model is 64 GB per node pair. The maximum amount of Flash Cache that can be configured per node pair is limited by StoreServ array model. Please refer to the HPE 3PAR Support Matrix in Single Point of Connectivity Knowledge (SPOCK) for details.

IO workloads There are four basic workloads that a storage array must be able to service:

1. Sequential writes

2. Random writes

3. Sequential reads

4. Random reads

http://www.hpe.com/Storage/Spock/


The first three workloads listed previously (sequential writes, random writes, and sequential reads) all benefit from cache on a storage array controller because the algorithms designed into the array software leverage the array cache to service these workloads improving IO for these workloads and hence reducing the IO latency experienced by a host. For write IOs (both sequential and random), data is mirrored in cache on two separate nodes in the array and then the server is told the IO is complete. This removes the need to flush the data to relatively slow drives before sending IO complete to the server. Caching of write data also allows for the grouping of small independent random writes into larger single IOs or even into full stripe writes hence reducing the total number of write IOs occurring to the back end of the array. In the case of sequential write workloads not only are the writes acknowledged to the server after being mirrored across two nodes but the array software can perform full stripe writes of the data to the hard disk drives (HDDs) on the back end of the array making cache flushes very efficient. In the case of sequential read workloads, the array software will detect the sequential read nature of the IO stream and will begin to prefetch large amounts of data from HDDs on the back end of the array into read cache in anticipation of the server requesting the data in the near future and then being able to service these requests from read cache.

The fourth workload mentioned previously, random reads, generally does not benefit from array cache and hence that is why these IOs are termed “random.” Because the data being read is random in nature, the array algorithms cannot anticipate the data that will be requested and have that data prestaged into read cache before a host requests it. As a result, random reads result in a lot of what are termed “read miss” or “cache miss” events where the requested data is not in read cache and must be retrieved from the back end of the array. So, to satisfy a random read request, data has to come from the HDDs on the back end of the array, and this puts a limit on the number of random read IOPS the array can deliver and also impacts the response time latency of those IOPS.

Using Adaptive Flash Cache to increase read hits and improve random read performance If an array’s read cache is sufficiently large compared to its working set size, it is possible to turn a percentage of what would have been random read “read misses” or “cache misses” into read hits where the data being requested by a server is already in read cache. There are a number of ways to achieve this. One of the easiest is to add additional DRAM to an array simply to increase the size of read cache, but this is an expensive proposition. A better solution is to use SSDs as a Level-2 read cache to hold small block random read data that is being removed from DRAM cache with the expectation that some percentage of that data will be reread. This is a more cost effective way of keeping a larger percentage of read data on very fast media to improve overall random read performance. This is exactly what HPE 3PAR Adaptive Flash Cache does; it utilizes flash storage (SSDs) as a second level (Level-2) cache to hold random read data that is being removed from DRAM cache, keeping it on high performance flash media longer in the event the data is accessed again by a host.

What gets cached in Adaptive Flash Cache? With Adaptive Flash Cache only small block random read data (IO size < 64 KiB) that is resident in a node’s DRAM cache can be considered for caching in the Flash Cache. Data is not cached directly from spinning media into Flash Cache; it only gets placed into Flash Cache after having been resident as read data in a node’s DRAM cache first. Data that is prefetched into DRAM cache as a result of a sequential read-ahead stream (regardless of the IO size) or data that ends up in DRAM cache for IOs >= 64 KiB is not considered for caching into Flash Cache. Although Adaptive Flash Cache is not used as an extension of write cache for write data, it is possible for data that ends up in DRAM memory as a result of a small block random write operations (IO size < 64 KiB) to be destaged to Flash Cache. This occurs after dirty write data is written to disk as during a cache flush operation and the CMP becomes marked as a valid page of read data in the DRAM cache (a CMP containing clean, valid data). At this point, the CMP containing the data is treated by the array caching algorithms as if it were a page of small block random read data and it will ultimately find itself in Flash Cache when it is eventually removed from DRAM cache. Sequential write data will not be destaged to Flash Cache regardless of the IO size, and random write data for IOs >= 64 KiB will not be destaged to Flash Cache even though write data for a sequential write IO stream also becomes read data when flushed to the back end of the array.

Data that is read into DRAM cache from an SSD tier will never be cached in Flash Cache because that data is already on a high-performance flash tier and would not benefit from being in Flash Cache.


Figure 1. HPE 3PAR Adaptive Flash Cache

Moving data from a node’s DRAM read cache into Adaptive Flash Cache Data in a node’s DRAM read cache starts to destage to Flash Cache when the node’s DRAM memory becomes 90 percent full. Read data is copied from a node’s DRAM cache to Flash Cache when the caching algorithms are looking to free up CMPs in a node’s DRAM memory and a read CMP is chosen to be freed up. If the read CMP to be freed meets the criteria described below, it will be destaged (copied) to Flash Cache before the DRAM CMP containing the data is marked as free (available for reuse).

To summarize, the following criteria determine whether a given DRAM CMP will be cached in Flash Cache:

1. The data resides in DRAM cache as a result of a small block random read (i.e., the IO size must be < 64 KiB) or the data must reside in DRAM cache as a result of a small block random write (i.e., the IO size must be < 64 KiB) that became a read CMP as the result of dirty write data being written to the back end HDDs. When a CMP containing write data is flushed to the back end of the array, the CMP becomes a read data CMP.

2. The data must not be the result of a sequential read stream (regardless of the IO size).

3. The data must not be a read CMP that previously was a write CMP as the result of a large block write that was flushed (i.e., the IO size was > 64 KiB).

4. The data must not be a read CMP as the result of a sequential write stream that was flushed (regardless of the IO size).

5. The data must not have come from an SSD tier.


Data in Adaptive Flash Cache When a page of read data moves from DRAM cache to AFC, it is copied to an FMP that resides on one of five least recently used (LRU) queues that are maintained by AFC. These LRU queues are used for tracking how hot (frequently accessed) data is while it resides in AFC. FMPs move from lower priority LRU queues to higher priority queues when they are accessed (read) by a host and they move from higher to lower LRU queues when LRU queues are demoted. If an FMP continues to be accessed once it gets into Flash Cache, it will continually be promoted to a higher priority LRU and can in fact stay in Flash Cache indefinitely.

There are five different LRU queues maintained by AFC:

1. Dormant—This is the lowest priority LRU queue and holds FMPs that contain invalid data or that contain valid data that is a candidate for reuse due to their lack of accesses since entering Flash Cache. These FMPs are used when a new DRAM CMP, which does not already have an entry in Flash Cache, needs to be copied to Flash Cache. A Dormant LRU FMP is immediately promoted to the “Norm” LRU queue when it is used to hold a new page of data entering Flash Cache. Existing valid FMPs in this LRU are promoted to Warm if they are accessed by a server.

2. Cold—FMPs are never promoted from Dormant to Cold. FMPs in this LRU queue are promoted to Warm if they are accessed. FMPs are moved here from Norm when all LRU queues are demoted down.

3. Norm—When a DRAM CMP is initially destaged to Flash Cache, it starts in this LRU queue. FMPs in this LRU queue are promoted to Warm if they are accessed once.

4. Warm—FMPs from this LRU queue are promoted to Hot if they are accessed twice.

5. Hot—FMPs from the Warm queue are promoted to Hot if they are accessed twice and they stay here on subsequent hits.

When a server reads data in Flash Cache, it is copied to DRAM read cache where will once again eventually be removed but it does not have to be copied to an LRU queue when it is. The data already resides in Flash Cache on the LRU queue it was promoted to because of the host read. If data sits in Flash Cache long enough without being accessed, it will eventually be aged out of the Flash Cache due to LRU demotion or be reused for a new DRAM CMP entering Flash Cache.

Demoting LRU queues When a new DRAM CMP enters Flash Cache, it uses an FMP from the Dormant LRU. When all of the FMPs in the Dormant LRU become used (the Dormant LRU contains zero FMPs) all LRU queues are demoted one level with the Hot LRU becoming the Warm LRU, Warm becoming Norm, Norm becoming Cold and Cold becoming Dormant (note that the Hot LRU will be empty at this time). When this occurs data isn’t actually moved between the LRUs, the LRUs simply change places. This process occurs any time the Dormant LRU does not have an FMP available for a DRAM CMP that is moving from DRAM cache to Flash Cache.

The following example uses output from the “statcache” command to demonstrate LRU demotion:

Initially the system looks like the following where the Dormant LRU contains many FMPs (Notice that the “Cold” LRU happens to contain zero FMPs in this example).

----------------- FMP Queue Statistics ------------------

Node Dormant Cold Norm Warm Hot Destage Read Flush WrtBack

0 24056921 0 1107007 1484 412 0 0 0 0

Over time as data is destaged from DRAM to Flash Cache, FMPs are taken from the Dormant LRU and are moved to the Norm LRU to hold this data. As data in the Norm and Warm LRUs is accessed (i.e., read hit) those FMPs are promoted to Warm and Hot respectively. As data in Hot is accessed, it remains in Hot. Over time as new data is destaged from DRAM to Flash Cache the system can eventually look something like this where the Dormant LRU has run out of FMPs (Notice that the “Cold” LRU still contains zero FMPs).




0 0 0 25132171 7000 26653 0 0 0 0

At this point, since the Dormant LRU is empty, all LRUs are demoted one level.



0 0 25132171 7000 26653 0 0 0 0 0

After the demotion, the Dormant LRU is still empty so a second demotion will occur.



0 25132171 7000 26653 0 0 0 0 0 0

At this point, the Dormant LRU has available pages and the process starts over again.

Creating and configuring Adaptive Flash Cache Adaptive Flash Cache is simple to create and configure. There are new CLI commands that are used to create, enable, disable, clear, display the status of, and remove Flash Cache on an HPE 3PAR StoreServ array.

Createflashcache command use Use the “createflashcache” command to specify how much SSD capacity on each node pair should be allocated for Flash Cache use. All node pairs will be configured for the same amount of Flash Cache, so all node pairs must have enough available SSD space to satisfy the amount of Flash Cache specified in the createflashcache command. The quantity of Flash Cache to be enabled can be specified either in gigabytes or in terabytes. If you specify the amount of Flash Cache in gigabytes, it must be in 16 GB increments. When Flash Cache is created on the array, it is created as a special RAID 1 Logical Disk behind each node in the array that is named “fcacheld.x” where “x” is the node ID.

Note Flash Cache must be created in 16 GB increments.

Use the showspace command to determine how much available SSD space your array has for creating Flash Cache. Keep in mind that Flash Cache is created as fully allocated RAID 1 Logical Disks behind all nodes on the array.

cli% showspace -p -devtype SSD -t r1 -ha mag

--Estimated(MB)---

RawFree UsableFree

1896448 948224

The following example will create 128 GB of Flash Cache for each node pair in the array:

cli% createflashcache 128g


Here is an example of the Flash Cache Logical Disks for a two-node system that has been configured for 128 GB of Flash Cache:

cli% showld

id Name RAID -Detailed_State- Own Size MB Used MB Use Lgct LgId WThru MapV

0 admin.usr.0 1 normal 0/1 5120 5120 V 0 --- N Y

1 admin.usr.1 1 normal 1/0 5120 5120 V 0 --- N Y

2 .srdata.usr.0 1 normal 0/1 40960 40960 V 0 --- N Y

3 .srdata.usr.1 1 normal 1/0 40960 40960 V 0 --- N Y

4 log0.0 1 normal 0/- 20480 0 log 0 --- Y N

5 log1.0 1 normal 1/- 20480 0 log 0 --- Y N

6 pdsld0.0 1 normal 1/0 1024 0 P,F 0 --- Y N

7 pdsld0.1 1 normal 1/0 8192 0 P 0 --- Y N

8 pdsld0.2 1 normal 1/0 8192 0 P 0 --- Y N

9 fcacheld.0 1 normal 0/1 65536 0 FLC 0 --- N N

10 fcacheld.1 1 normal 1/0 65536 0 FLC 0 --- N N

Showflashcache command use Use the “showflashcache” command to display how much Flash Cache has been allocated for each node on the array, to display how much of the configured Flash Cache is actually in use (contains cached data), and to display which VVsets and Virtual Volumes (VVs) on the system Flash Cache has been enabled for. Note that all nodes should always show the same amount of Flash Cache.

Display the status of Flash Cache for all nodes on the system:

cli% showflashcache

-(MB)-

Node Mode State Size Used%

0 SSD normal 65536 0


-------------------------------

2 total 131072

Display the status of VVsets with Flash Cache enabled on the system:

cli% showflashcache -vvset

Id VVSetName AFCPolicy

1 devtest enabled

----------------------

1 total


Display the status of VVs with Flash Cache enabled:

cli% showflashcache -vv

VVid VVName AFCPolicy

50 devtest.48 enabled




-------------------------

4 total

Setflashcache command use Once Flash Cache has been created use the “setflashcache” command to enable and disable Flash Cache on the system Adaptive Flash Cache operates in one of two modes, “System Mode” and “VVset Mode.” System Mode, as the name implies, affects the entire system and is entered whenever the sys:all target is specified with either the “enable” or “disable” subcommands to the setflashcache command. When the sys:all target is specified Flash Cache is either enabled or disabled for the entire system (all VVs regardless of whether they are defined in a VVset or not) and any prior Flash Cache settings specified via the “vvset:<VVset name>” target to the setflashcache command are overridden. While in System Mode any Flash Cache setting specified via the “vvset:<VVset name>” target are also not visible via the showflashcache command (all VVs will either be specified as having Flash Cache enabled or disabled. You will not be able to get information on individual VVsets or individual VVs).

Note: Even though showflashcache will not provide information on individual VVs or VVsets while you are in System Mode you can still get information on Flash Cache’s effect on VVs by using the statcache and srstatcache commands.

Even though settings specified via the “vvset:<VVset name>” target are not visible while in System Mode they are still saved and will be applied when System Mode is exited. Also, any changes made to VVsets using the “vvset:<VVset name>” target while in System Mode will also be saved and will become effective when you exit System Mode.

To exit System Mode you specify the “clear” subcommand to the sys:all target. When you exit System Mode any prior Flash Cache settings you specified via the “vvset:<VVset name>” target will take effect and will be visible via the showflashcache command.

To enable or disable Flash Cache for individual VVsets use the “vvset:<VVset name>” target with the enable or disable sub commands. There is no way to control Flash Cache for individual VVs not defined in a VVset. If you want to control Flash Cache (enable or disable it) on a VV by VV basis then the VVs must first be placed into their own individual VVsets and then the “enable” and “disable” subcommands can be applied to those VVsets to control them.

To reiterate, the “sys:all” target when used with the “enable” or “disable” subcommand enters System Mode and these subcommands takes precedence over individual VVs and VVsets. The specified subcommand is applied against all VVs defined on the system whether or not they are defined in a VVset. The “clear” subcommand can only be applied to the “sys:all” target and exits System Mode. Any setting specified via use of the “vvset:<VVset name>” target prior to entering System Mode or specified while in System Mode become effective when System Mode is exited.


Table 1. setflashcache command options

setflashcache enable sys:all Enable Flash Cache for all VVs defined on the system whether or not they are defined in a VVset.

setflashcache disable sys:all Disable Flash Cache for all VVs defined on the system.

setflashcache clear sys:all Disable Flash Cache for all VVs that are not defined in a VVset on the system. Note that the “clear” command will always affect all VVs that are not defined in a VVset and can only be applied to the “sys:all” target.

setflashcache enable vvset:devtest Enable Flash Cache for the VVs defined in the VVset named “devtest.”

setflashcache disable vvset:devtest Disable Flash Cache for the VVs in the VVset “devtest.”

If a VV is a member of more than one VVset Flash Cache for that VV will be enabled if Flash Cache is enabled for any of the VVsets the VV is a member of. If Flash cache is enabled for multiple VVsets a VV belongs to then Flash Cache must be disabled for all the VVsets the VV belongs to in order to disable Flash Cache for that VV. In the following example, the VV named “devtest.49” is a member of VVset “devtest” and VVset “flashcachetest.”

Flash Cache is enabled for VVset “devtest”, which includes VV devtest.49:



1 devtest enabled

----------------------

1 total







-------------------------

4 total

Now let’s enable Flash Cache for VVset “flashcachetest”:

cli% setflashcache enable vvset:flashcachetest

Flash Cache is now enabled for both devtest and flashcachetest:



1 devtest enabled

7 flashcachetest enabled

---------------------------

2 total


Check that Flash Cache is still enabled for VV devtest.49:







-------------------------

4 total

Now let’s disable Flash Cache for VVset devtest:

cli% setflashcache disable vvset:devtest

Now, Flash Cache is only enabled for VVset flashcachetest and is still enabled for VV devtest.49:



7 flashcachetest enabled

---------------------------

1 total




-------------------------

1 total

Note The “clear” option can only be applied to the “sys:all” target:

cli% setflashcache clear vvset:devtest

setflashcache: clear command cannot be used with vvset

SYNTAX

setflashcache {enable|disable} {vvset:<name|pattern>|sys:all} ...

setflashcache {clear} {sys:all}

SUBCOMMAND


enable

Will turn on the flash cache policy for the target object.

disable

Will turn off flash cache policy for the target object.

clear

Will turn off policy and can only be issued against

the sys:all target.

Removeflashcache command use Use the “removeflashcache” CLI command to disable and remove all Flash Cache from the array.

cli% showflashcache

-(MB)-

Node Mode State Size Used%



-------------------------------

2 total 131072

cli% removeflashcache

Are you sure you want to remove the flash cache?

select q=quit y=yes n=no: y

cli% showflashcache

Flash Cache is not present.

Note The removeflashcache command does not affect any Flash Cache rules defined on the system via the setflashcache command.

Please refer to the “HPE 3PAR Command Line Interface Reference HPE 3PAR OS 3.2.1” for details on all of the options and explanation on all the output fields for the Adaptive Flash Cache commands.

http://h20565.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03618134


Flash Cache statistics There are a number of statistics that can be viewed on an HPE 3PAR StoreServ array configured for Adaptive Flash Cache that provide insight into how much Adaptive Flash Cache is being utilized. Statistics on the DRAM cache (CMP) usage on an HPE 3PAR StoreServ array can be viewed via the “statcmp” CLI command (See the “HPE 3PAR Command Line Interface Reference HPE 3PAR OS 3.2.1” for details on the statcmp command).

Adaptive Flash Cache statcache command With the introduction of Adaptive Flash Cache, the “statcache” command has been introduced to show not only the DRAM CMP statistics but to also show the FMPs statistics for data held in Flash Cache. With the statcache command, you can get a side-by-side view of both the CMP and FMP cache usage statistics as well as unique FMP specific statistics. The data from this command can be used to make decisions on whether to add or remove VVs from AFC or to adjust the size of AFC.

These statistics can be used to determine how effective AFC is on a system wide and on a per VV basis. A high Hit % for FMP statistics is indicative of data that has aged out of CMP cache into Flash Cache and then reread from AFC saving IOs from having to be serviced by spinning media on the back end of the array.

Note The Flash Cache statistics include write statistics as well as read statistics. The write statistics displayed are applicable to CMP data only and not FMP data as Adaptive Flash Cache operates on read data only, so all write statistics for FMP will read zero (0).

Display CMP (DRAM) and FMP (Flash Cache) statistics on a per node basis with the statcache command The statcache command by itself displays statistics for both CMP and FMP cache. These statistics can be used to determine how effective AFC is system wide. A non-zero Hit % for FMP statistics is indicative of data that has been cached into Flash Cache and was then reread by a host saving IOs from having to be serviced by slow media on the back end of the array. This indicates that AFC is having the desired effect of caching data that aged out of DRAM cache but that was accessed relatively soon afterward by a host. Please refer to the “HPE 3PAR Command Line Interface Reference HPE 3PAR OS 3.2.1” for details on the statcache command.

cli% statcache

15:12:35 04/22/2014 ------- Current -------- -------- Total ---------

CMP FMP Total CMP FMP Total

Node Type Accesses Hit% Hit% Hit% Accesses Hit% Hit% Hit%

0 Read 0 0 0 0 0 0 0 0

0 Write 0 0 0 0 0 0 0 0

1 Read 0 0 0 0 0 0 0 0

1 Write 0 0 0 0 0 0 0 0

Internal Flashcache Activity

----- Current ------ ------- Total --------

Node Type Accesses IO/s MB/s Accesses IO/s MB/s

0 Read Back 0 0 0 0 0 0

0 Destaged Write 0 0 0 0 0 0


1 Read Back 0 0 0 0 0 0

1 Destaged Write 0 0 0 0 0 0

--------------- FMP Queue Statistics ----------------


0 8388608 0 0 0 0 0 0 0 0

1 8388608 0 0 0 0 0 0 0 0

-------------------- CMP Queue Statistics --------------------

Node Free Clean Write1 WriteN WrtSched Writing DcowPend DcowProc

0 1796802 58859 0 0 0 0 0 0

1 1796521 58200 0 0 0 0 0 0

Display CMP (DRAM) and FMP (Flash Cache) statistics on a VV basis with the statcache –v command The statistics displayed by the “statcache –v” command can be used to determine how effective AFC is on a per VV basis. A non-zero Hit % for FMP statistics is indicative of data that has aged out of DRAM cache into Flash Cache that was then reread from Flash Cache saving IOs from having to be serviced by slow media on the back end of the array. If there are VVs that consistently have a very low or zero (0) FMP Hit %, you should consider disabling AFC for those VVs as they are consuming AFC space but their workloads are such that they do not reread data that has been placed in AFC and are not benefiting from having their data in AFC.

cli% statcache -v

15:14:29 04/22/2014 ------- Current -------- -------- Total ---------


VVid VVname Type Accesses Hit% Hit% Hit% Accesses Hit% Hit% Hit%

0 admin Read 0 0 0 0 0 0 0 0

0 admin Write 0 0 0 0 0 0 0 0

1 .srdata Read 0 0 0 0 0 0 0 0

1 .srdata Write 0 0 0 0 0 0 0 0

5 VV25150_0001 Read 0 0 0 0 0 0 0 0

5 VV25150_0001 Write 0 0 0 0 0 0 0 0

7 VV25202_0001 Read 0 0 0 0 0 0 0 0

7 VV25202_0001 Write 0 0 0 0 0 0 0 0

8 VV25205_0001 Read 0 0 0 0 0 0 0 0

8 VV25205_0001 Write 0 0 0 0 0 0 0 0

9 VV25214_0000 Read 0 0 0 0 0 0 0 0

9 VV25214_0000 Write 0 0 0 0 0 0 0 0


- Internal Flashcache Activity -

- Current -- -- Total ---

VVid VVname Type Accesses Accesses

0 admin Read Back 0 0

0 admin Destaged Write 0 0

1 .srdata Read Back 0 0

1 .srdata Destaged Write 0 0

5 VV25150_0001 Read Back 0 0

5 VV25150_0001 Destaged Write 0 0

7 VV25202_0001 Read Back 0 0


8 VV25205_0001 Read Back 0 0


9 VV25214_0000 Read Back 0 0


Display metadata statistics by adding –metadata to the statcache –v command All snapshot administration data (SA data) associated with any TPVV (or snapshot) on the array is cached in Flash Cache if Flash Cache has been enabled on the array. This occurs regardless of whether or not Flash Cache is enabled for the TPVVs or snapshots. In other words, if Flash Cache has been created on the array, all SA data on the array will be cached in the Flash Cache.

The statcache –v –metadata command adds additional statistics for the snapshot administration data to the VV statistics.

cli% statcache -v -metadata

15:16:01 04/22/2014 ------- Current -------- -------- Total ---------


VVid VVname Type Accesses Hit% Hit% Hit% Accesses Hit% Hit% Hit%

0 admin Read 0 0 0 0 0 0 0 0

0 admin Write 0 0 0 0 0 0 0 0

0 admin MRead 0 0 0 0 0 0 0 0

0 admin MWrite 0 0 0 0 0 0 0 0

1 .srdata Read 0 0 0 0 0 0 0 0

1 .srdata Write 0 0 0 0 0 0 0 0

1 .srdata MRead 0 0 0 0 0 0 0 0

1 .srdata MWrite 0 0 0 0 0 0 0 0


5 VV25150_0001 Read 0 0 0 0 0 0 0 0

5 VV25150_0001 Write 0 0 0 0 0 0 0 0

5 VV25150_0001 MRead 0 0 0 0 0 0 0 0

5 VV25150_0001 MWrite 0 0 0 0 0 0 0 0

7 VV25202_0001 Read 0 0 0 0 0 0 0 0

7 VV25202_0001 Write 0 0 0 0 0 0 0 0

7 VV25202_0001 MRead 0 0 0 0 0 0 0 0

7 VV25202_0001 MWrite 0 0 0 0 0 0 0 0

8 VV25205_0001 Read 0 0 0 0 0 0 0 0

8 VV25205_0001 Write 0 0 0 0 0 0 0 0

8 VV25205_0001 MRead 0 0 0 0 0 0 0 0

8 VV25205_0001 MWrite 0 0 0 0 0 0 0 0

9 VV25214_0000 Read 0 0 0 0 0 0 0 0

9 VV25214_0000 Write 0 0 0 0 0 0 0 0

9 VV25214_0000 MRead 0 0 0 0 0 0 0 0

9 VV25214_0000 MWrite 0 0 0 0 0 0 0 0

- Internal Flashcache Activity -

- Current -- -- Total ---

VVid VVname Type (user+meta) Accesses Accesses

0 admin Read Back 0 0

0 admin Destaged Write 0 0

1 .srdata Read Back 0 0

1 .srdata Destaged Write 0 0

5 VV25150_0001 Read Back 0 0


7 VV25202_0001 Read Back 0 0


8 VV25205_0001 Read Back 0 0


9 VV25214_0000 Read Back 0 0



Please refer to the “HPE 3PAR Command Line Interface Reference HPE 3PAR OS 3.2.1” or the CLI help for an explanation of all the fields displayed by the statcache CLI command.

Increasing and decreasing the size of Flash Cache The configured Flash Cache size on a system cannot be changed dynamically. To increase or decrease the amount of Flash Cache configured on the system, you must first remove Flash Cache with the “removeflashcache” command (This will disable Flash Cache for all VVs) and then recreate Flash Cache with the “createflashcache” command specifying the new desired size. Because Flash cache must first be removed to change its size, it is recommended that this be done during a maintenance period if possible because it will take some time for the Flash Cache to warm up after being recreated and this may have a noticeable effect on the system performance. All existing configuration rules created with the setflashcache command will persist upon Flash Cache removal and recreation.

Balancing Flash Cache after adding SSDs to the system HPE 3PAR Autonomic Rebalance provided via the “tunesys” operation does not apply to Flash Cache LDs. After adding additional SSDs to a system, you can rebalance Flash Cache across the new SSDs by first removing Flash Cache with the “removeflashcache” command (This will disable Flash Cache for all VVs) and then recreating Flash Cache with the “createflashcache” command. Because Flash cache must be removed to rebalance across new SSDs is recommended that this be done during a maintenance period if possible because it will take some time for the Flash Cache to warm up after being recreated, and this warm-up period may have a noticeable effect on the system performance. Rules created with the setflashcache command will persist when Flash Cache is recreated.

Adaptive Flash Cache simulator For HPE 3PAR StoreServ arrays running HPE 3PAR OS 3.2.1 and higher, there is an Adaptive Flash Cache simulator that can be run even if the system does not contain any SSDs. The simulator allows the StoreServ array to track Flash Cache statistics without actually using SSD space. These statistics can then be analyzed using the standard Flash Cache CLI commands to determine how much benefit there would be to adding SSDs to the StoreServ and configuring Flash Cache. It can even be used to determine how much Flash Cache to create and configure.

Use the –sim option to the createflashcache command to enable the Adaptive Flash Cache simulator.

cli% createflashcache –sim <size>

Once the simulator has been started, use the “setflashcache” command to specify the VVs that should be considered (or the entire system) for Flash Cache. After the system has run for some time (hours or days), you can then view the Flash Cache statistics via the “statcache” command (or the srstatcache command if you wish to review historical data) to determine how much benefit can be derived from enabling Flash Cache for the system or for specific VVs or VVsets. The simulator can be easily used to try out different “sizes” of Flash Cache on a system to help you find the right mix.

Remember, the larger the configured Flash Cache the better the probability for cache hits. A balance needs to be struck between how large the Flash Cache is and how many hits it receives.

Once you are finished with the simulator and ready to enable Adaptive Flash Cache in a live manner, all you need to do is run the createflashcache command and specify the quantity of Flash Cache you desire per node pair. All of the configuration rules created via the setflashcache command while running the simulator will be enabled automatically. Note that it will take some time for the Flash cache to warm up once you start it.

Please remember to stop the simulator via the removeflashcache command even if you have decided not to enable Flash Cache, as the simulator does consume some system resources (CPU and memory) while it is running and leaving it in place wastes these system resources. Any policies you created via the setflashcache command will be maintained unless you remove them via the “disable” option to the setflashcache command.

Note An HPE 3PAR System Reporter license is not necessary to display Flash Cache statistics with the srstatcache CLI command.


Sign up for updates

Rate this document

© Copyright 2014–2016 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

4AA5-5397ENW, January 2016, Rev. 2

HPE 3PAR Adaptive Flash Cache vs. HPE 3PAR Adaptive Optimization Even though Adaptive Flash Cache caches small block read data that resides on spinning media onto SSD, it does not compete with nor is it a replacement for Adaptive Optimization (AO). Where AO focuses on moving larger chunks of data (128 MiB regions) between different tiers of storage based on historical data access patterns to help save cost by migrating seldom used data to large, slow nearline (NL) HDDs and more heavily accessed data to a Fast-Class or an SSD tier, Adaptive Flash Cache is focused on improving small block random read performance for data that resides on spinning media.

AFC and AO can and do coexist effectively on a system. AFC caches small block read data (IO < 64 KiB) that come from a spinning media pool (for example, 10k FC HDDs) into DRAM read cache (and which is about to be evicted from DRAM cache) onto SSD space allocated on the node. Even though both AFC and AO move “hot” data to SSD, the SSD space that is allocated to Flash Cache is not the same thing as an SSD tier found in an AO policy. While AO moves data between tiers of storage (up to three tiers) in 128 MiB “regions,” it does so based on how heavily the regions are accessed (region density) over a specified measurement interval and on a scheduled basis. AO also considers both read and write IOs to the regions when making moves. Flash Cache, on the other hand, can cache only read data from a node’s DRAM read cache onto Flash Cache SSD space and does so on a real-time basis. Also, while the unit of data that AO operates on is a “region” (128 MiB), Flash Cache operates on 16 KiB CMPs. Flash Cache and AO can coexist together and are in fact complementary to one another.

The SSDs in an array can be shared between Flash Cache and AO due to the chunklet level virtualization of SSDs that occurs on the system or the SSD space allocated to Flash Cache can be dedicated SSDs. This means you do not have to dedicate specific SSDs as Flash Cache and instead can allocate a certain amount of SSD capacity behind each node to be used as Flash Cache.

It should be noted that read accesses to data that reside in Flash Cache are not included in the region density data collected by System Reporter and hence will not have an effect on AO moving the corresponding region to a higher tier as a result of heavy access. This means it is possible for some applications to have data that, even though it is heavily accessed on a daily basis, does not get migrated to higher tier of storage by AO because the region accesses are not counted as a result of that hot data ending up in AFC. As a result, if this daily hot data resides on NL HDDs, it may suffer a noticeable performance lag until the data makes its way into Flash Cache.

Learn more at hp.com/go/3PAR

http://www.hpe.com/info/getupdated

http://www.facebook.com/sharer.php?u=http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA5-5397ENW

http://twitter.com/home/?status=HPE 3PAR Adaptive Flash Cache+@+http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA5-5397ENW

http://www.linkedin.com/shareArticle?mini=true&ro=true&url=http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA5-5397ENW&title=HPE 3PAR Adaptive Flash Cache+&armin=armin

https://hpresearch.az1.qualtrics.com/SE/?SID=SV_6EuQKOr2Ku1CUzH&Pubnumber=4AA5-5397ENW






















http://twitter.com/home/?status=HPE%203PAR%20Adaptive%20Flash%20Cache+%40+http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA5-5397ENW

http://www.linkedin.com/shareArticle?mini=true&ro=true&url=http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA5-5397ENW&title=HPE%203PAR%20Adaptive%20Flash%20Cache+&armin=armin


http://www.hp.com/go/3par

hpe 3par adaptive flash cache technical white paper

Documents