New in version 20.0.0: (Train)
Starting in the 20.0.0 (Train) release, the virtual persistent memory (vPMEM) feature in Nova allows a deployment using the libvirt compute driver to provide vPMEMs for instances using physical persistent memory (PMEM) that can provide virtual devices.
PMEM must be partitioned into PMEM namespaces for applications to use.
This vPMEM feature only uses PMEM namespaces in devdax
mode as QEMU
vPMEM backends. If you want to dive into related notions, the document
NVDIMM Linux kernel document is recommended.
To enable vPMEMs, follow the steps below.
The following are required to support the vPMEM feature:
Persistent Memory Hardware
One such product is Intel® Optane™ DC Persistent Memory. ipmctl is used to configure it.
Linux Kernel version >= 4.18 with the following modules loaded:
dax_pmem
, nd_pmem
, device_dax
, nd_btt
Note
NVDIMM support is present in the Linux Kernel v4.0 or newer. It is recommended to use Kernel version 4.2 or later since NVDIMM support is enabled by default. We met some bugs in older versions, and we have done all verification works with OpenStack on 4.18 version, so 4.18 version and newer will probably guarantee its functionality.
The vPMEM feature has been verified under the software and hardware listed above.
Create PMEM namespaces as vPMEM backends using the ndctl utility.
For example, to create a 30GiB namespace named ns3
:
$ sudo ndctl create-namespace -s 30G -m devdax -M mem -n ns3
{
"dev":"namespace1.0",
"mode":"devdax",
"map":"mem",
"size":"30.00 GiB (32.21 GB)",
"uuid":"937e9269-512b-4f65-9ac6-b74b61075c11",
"raw_uuid":"17760832-a062-4aef-9d3b-95ea32038066",
"daxregion":{
"id":1,
"size":"30.00 GiB (32.21 GB)",
"align":2097152,
"devices":[
{
"chardev":"dax1.0",
"size":"30.00 GiB (32.21 GB)"
}
]
},
"name":"ns3",
"numa_node":1
}
Then list the available PMEM namespaces on the host:
$ ndctl list -X
[
{
...
"size":6440353792,
...
"name":"ns0",
...
},
{
...
"size":6440353792,
...
"name":"ns1",
...
},
{
...
"size":6440353792,
...
"name":"ns2",
...
},
{
...
"size":32210157568,
...
"name":"ns3",
...
}
]
Specify which PMEM namespaces should be available to instances.
Edit libvirt.pmem_namespaces
:
[libvirt]
# pmem_namespaces=$LABEL:$NSNAME[|$NSNAME][,$LABEL:$NSNAME[|$NSNAME]]
pmem_namespaces = 6GB:ns0|ns1|ns2,LARGE:ns3
Configured PMEM namespaces must have already been created on the host as
described above. The conf syntax allows the admin to associate one or more
namespace $NSNAME
s with an arbitrary $LABEL
that can subsequently
be used in a flavor to request one of those namespaces. It is recommended,
but not required, for namespaces under a single $LABEL
to be the same
size.
Restart the nova-compute
service.
Nova will invoke ndctl to identify the configured PMEM namespaces, and report vPMEM resources to placement.
Specify a comma-separated list of the $LABEL
s from
libvirt.pmem_namespaces
to the flavor’s hw:pmem
property. Note that multiple instances of the same label are permitted:
$ openstack flavor set --property hw:pmem='6GB' my_flavor
$ openstack flavor set --property hw:pmem='6GB,LARGE' my_flavor_large
$ openstack flavor set --property hw:pmem='6GB,6GB' m1.medium
Note
If a NUMA topology is specified, all vPMEM devices will be put on guest NUMA node 0; otherwise nova will generate one NUMA node automatically for the guest.
Based on the above examples, an openstack server create
request with
my_flavor_large
will spawn an instance with two vPMEMs. One, corresponding
to the LARGE
label, will be ns3
; the other, corresponding to the 6G
label, will be arbitrarily chosen from ns0
, ns1
, or ns2
.
Note
Using vPMEM inside a virtual machine requires the following:
dax_pmem
, nd_pmem
, device_dax
, and nd_btt
kernel
modules;Note
When resizing an instance with vPMEMs, the vPMEM data won’t be migrated.
This section describes how to check that:
Note
Inventories and allocations related to vPMEM resource classes are on the root resource provider related to the compute node.
Get the list of resource providers
$ openstack resource provider list
+--------------------------------------+--------+------------+
| uuid | name | generation |
+--------------------------------------+--------+------------+
| 1bc545f9-891f-4930-ab2b-88a56078f4be | host-1 | 47 |
| 7d994aef-680d-43d4-9325-a67c807e648e | host-2 | 67 |
--------------------------------------+---------+------------+
Check the inventory of each resource provider to see resource classes
Each $LABEL
configured in libvirt.pmem_namespaces
is used to generate a resource class named CUSTOM_PMEM_NAMESPACE_$LABEL
.
Nova will report to Placement the number of vPMEM namespaces configured for
each $LABEL
. For example, assuming host-1
was configured as
described above:
$ openstack resource provider inventory list 1bc545f9-891f-4930-ab2b-88a56078f4be
+-----------------------------+------------------+----------+----------+-----------+----------+--------+
| resource_class | allocation_ratio | max_unit | reserved | step_size | min_unit | total |
+-----------------------------+------------------+----------+----------+-----------+----------+--------+
| VCPU | 16.0 | 64 | 0 | 1 | 1 | 64 |
| MEMORY_MB | 1.5 | 190604 | 512 | 1 | 1 | 190604 |
| CUSTOM_PMEM_NAMESPACE_LARGE | 1.0 | 1 | 0 | 1 | 1 | 1 |
| CUSTOM_PMEM_NAMESPACE_6GB | 1.0 | 3 | 0 | 1 | 1 | 3 |
| DISK_GB | 1.0 | 439 | 0 | 1 | 1 | 439 |
+-----------------------------+------------------+----------+----------+-----------+----------+--------+
Here you can see the vPMEM resource classes prefixed with
CUSTOM_PMEM_NAMESPACE_
. The LARGE
label was configured with one
namespace (ns3
), so it has an inventory of 1
. Since the 6GB
label was configured with three namespaces (ns0
, ns1
, and ns2
),
the CUSTOM_PMEM_NAMESPACE_6GB
inventory has a total
and max_unit
of 3
.
Check allocations for each server that is using vPMEMs
$ openstack server list
+--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
| 41d3e139-de5c-40fd-9d82-016b72f2ba1d | server-with-2-vpmems | ACTIVE | private=10.0.0.24 | ubuntu-bionic | my_flavor_large |
| a616a7f6-b285-4adf-a885-dd8426dd9e6a | server-with-1-vpmem | ACTIVE | private=10.0.0.13 | ubuntu-bionic | my_flavor |
+--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
$ openstack resource provider allocation show 41d3e139-de5c-40fd-9d82-016b72f2ba1d
+--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
| resource_provider | generation | resources |
+--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
| 1bc545f9-891f-4930-ab2b-88a56078f4be | 49 | {u'MEMORY_MB': 32768, u'VCPU': 16, u'DISK_GB': 20, u'CUSTOM_PMEM_NAMESPACE_6GB': 1, u'CUSTOM_PMEM_NAMESPACE_LARGE': 1} |
+--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
$ openstack resource provider allocation show a616a7f6-b285-4adf-a885-dd8426dd9e6a
+--------------------------------------+------------+-----------------------------------------------------------------------------------+
| resource_provider | generation | resources |
+--------------------------------------+------------+-----------------------------------------------------------------------------------+
| 1bc545f9-891f-4930-ab2b-88a56078f4be | 49 | {u'MEMORY_MB': 8192, u'VCPU': 8, u'DISK_GB': 20, u'CUSTOM_PMEM_NAMESPACE_6GB': 1} |
+--------------------------------------+------------+-----------------------------------------------------------------------------------+
In this example, two servers were created. server-with-2-vpmems
used
my_flavor_large
asking for one 6GB
vPMEM and one LARGE
vPMEM.
server-with-1-vpmem
used my_flavor
asking for a single 6GB
vPMEM.
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.