The Dell EMC PowerMax drivers, PowerMaxISCSIDriver
and
PowerMaxFCDriver
, support the use of Dell EMC PowerMax and VMAX storage
arrays with the Cinder Block Storage project. They both provide equivalent
functions and differ only in support for their respective host attachment
methods.
The drivers perform volume operations by communicating with the back-end PowerMax storage management software. They use the Requests HTTP library to communicate with a Unisphere for PowerMax instance, using a RESTAPI interface in the backend to perform PowerMax and VMAX storage operations.
Note
While PowerMax
will be used throughout this document, it will be used
to collectively categorize the following supported arrays, PowerMax 2000,
8000, VMAX All Flash 250F, 450F, 850F and 950F and VMAX Hybrid. Please note
there will be extended support of the VMAX Hybrid series until further
notice.
The Dell EMC PowerMax Cinder driver supports the VMAX-3 hybrid series, VMAX All-Flash series and the PowerMax arrays.
The array operating system software, Solutions Enabler 9.1.x series, and Unisphere for PowerMax 9.1.x series are required to run Dell EMC PowerMax Cinder driver.
Download Solutions Enabler and Unisphere from the Dell EMC’s support web site
(login is required). See the Dell EMC Solutions Enabler 9.1.x Installation
and Configuration Guide
and Dell EMC Unisphere for PowerMax Installation
Guide
at the Dell EMC Support site.
Note
While it is not explicitly documented which OS versions should be installed on a particular array, it is recommended to install the latest PowerMax OS as supported by Unisphere for PowerMax, that the PowerMax driver supports for a given OpenStack release.
OpenStack | Unisphere for PowerMax | PowerMax OS |
---|---|---|
Ussuri | 9.1.x | 5978.479 |
Train | 9.1.x | 5978.444 |
Stein | 9.0.x | 5978.221 |
However, a Hybrid array can only run HyperMax OS 5977, and is still supported until further notice. Some functionality will not be available in older versions of the OS. If in any doubt, please contact your customer representative.
The storage system requires a Unisphere for PowerMax (SMC) eLicense.
There are two licenses for the PowerMax 2000 and 8000:
The Dell EMC PowerMax cinder driver requires the Pro software package.
For full functionality including SRDF for the VMAX All Flash, the FX package, or the F package plus the SRDF a la carte add on is required.
There are five Dell EMC Software Suites sold with the VMAX Hybrid arrays:
The Dell EMC PowerMax Cinder driver requires the Advanced Suite and the Local Replication Suite or the Total Productivity Pack (it includes the Advanced Suite and the Local Replication Suite) for the VMAX Hybrid.
Using PowerMax Remote Replication functionality will also require the Remote Replication Suite.
Note
Each are licensed separately. For further details on how to get the relevant license(s), reference eLicensing Support below.
To activate your entitlements and obtain your PowerMax license files, visit the Service Center on Dell EMC Support, as directed on your License Authorization Code (LAC) letter emailed to you.
For help with missing or incorrect entitlements after activation (that is, expected functionality remains unavailable because it is not licensed), contact your EMC account representative or authorized reseller.
For help with any errors applying license files through Solutions Enabler, contact the Dell EMC Customer Support Center.
If you are missing a LAC letter or require further instructions on
activating your licenses through the Online Support site, contact EMC’s
worldwide Licensing team at licensing@emc.com
or call:
North America, Latin America, APJK, Australia, New Zealand: SVC4EMC (800-782-4362) and follow the voice prompts.
EMEA: +353 (0) 21 4879862 and follow the voice prompts.
If you require help or assistance with PowerMax and Cinder please open a Service Request (SR) through standard support channels at Dell EMC Support. When opening a SR please include the following information:
{cinder_install_dir}/cinder/volume/drivers/dell_emc/powermax/fc.py
PowerMax drivers support these operations:
PowerMax drivers also support the following features:
Note
shortHostName
will be altered using the following formula, if its length
exceeds 16 characters. This is because the storage group and masking view
names cannot exceed 64 characters:
if len(shortHostName) > 16:
1. Perform md5 hash on the shortHostName
2. Convert output of 1. to hex
3. Take last 6 characters of shortHostName and append output of 2.
4. If the length of output of 3. exceeds 16 characters, join the
first 8 characters and last 8 characters.
Note
portgroup_name
will be altered using the following formula, if its
length exceeds 12 characters. This is because the storage group and masking
view names cannot exceed 64 characters:
if len(portgroup_name) > 12:
1. Perform md5 hash on the portgroup_name
2. Convert output of 1. to hex
3. Take last 6 characters of portgroup_name and append output of 2.
4. If the length of output of 3. exceeds 12 characters, join the
first 6 characters and last 6 characters.
Masking views are dynamically created by the PowerMax FC and iSCSI drivers
using the following naming conventions. [protocol]
is either I
for
volumes attached over iSCSI or F
for volumes attached over Fibre Channel.
OS-[shortHostName]-[protocol]-[portgroup_name]-MV
For each host that is attached to PowerMax volumes using the drivers, an
initiator group is created or re-used (per attachment type). All initiators
of the appropriate type known for that host are included in the group. At
each new attach volume operation, the PowerMax driver retrieves the initiators
(either WWNNs or IQNs) from OpenStack and adds or updates the contents of the
Initiator Group as required. Names are of the following format. [protocol]
is either I
for volumes attached over iSCSI or F
for volumes attached
over Fibre Channel.
OS-[shortHostName]-[protocol]-IG
Note
Hosts attaching to OpenStack managed PowerMax storage cannot also attach to storage on the same PowerMax that are not managed by OpenStack.
PowerMax array FA ports to be used in a new masking view are retrieved from the port group provided as the extra spec on the volume type, or chosen from the list provided in the Dell EMC configuration file.
As volumes are attached to a host, they are either added to an existing
storage group (if it exists) or a new storage group is created and the volume
is then added. Storage groups contain volumes created from a pool, attached
to a single host, over a single connection type (iSCSI or FC). [protocol]
is either I
for volumes attached over iSCSI or F
for volumes attached
over Fibre Channel. PowerMax Cinder driver utilizes cascaded storage groups -
a parent
storage group which is associated with the masking view, which
contains child
storage groups for each configured
SRP/slo/workload/compression-enabled or disabled/replication-enabled or
disabled combination.
PowerMax, VMAX All Flash and Hybrid
Parent storage group:
OS-[shortHostName]-[protocol]-[portgroup_name]-SG
Child storage groups:
OS-[shortHostName]-[SRP]-[ServiceLevel/Workload]-[portgroup_name]-CD-RE
Note
CD and RE are only set if compression is explicitly disabled or replication explicitly enabled. See the compression 11. All Flash compression support and replication Volume replication support sections below.
Note
For VMAX All Flash with PowerMax OS (5978) or greater, workload if set will be ignored and set to NONE.
Download Solutions Enabler from Dell EMC Support and install it.
You can install Solutions Enabler on a non-OpenStack host. Supported
platforms include different flavors of Windows, Red Hat, and SUSE Linux.
Solutions Enabler can be installed on a physical server, or as a Virtual
Appliance (a VMware ESX server VM). Additionally, starting with HYPERMAX
OS Q3 2015, you can manage VMAX3 arrays using the Embedded Management
(eManagement) container application. See the Dell EMC Solutions Enabler
9.1.x Installation and Configuration Guide
on Dell EMC Support for
more details.
Note
You must discover storage arrays before you can use the PowerMax drivers.
Follow instructions in Dell EMC Solutions Enabler 9.1.x Installation
and Configuration Guide
on Dell EMC Support for more details.
Download Unisphere from Dell EMC Support and install it.
Unisphere can be installed in local, remote, or embedded configurations
- i.e., on the same server running Solutions Enabler; on a server
connected to the Solutions Enabler server; or using the eManagement
container application (containing Solutions Enabler and Unisphere for
PowerMax). See Dell EMC Solutions Enabler 9.1.x Installation and
Configuration Guide
at Dell EMC Support.
Zone Manager is required when there is a fabric between the host and array. This is necessary for larger configurations where pre-zoning would be too complex and open-zoning would raise security concerns.
open-iscsi
package (or distro equivalent) is installed
on all Compute nodes.Note
You can only ping the PowerMax iSCSI target ports when there is a valid masking view. An attach operation creates this masking view.
Note
VMAX driver was rebranded to PowerMax in Stein, so some of the driver
specific tags have also changed. Legacy tags like vmax_srp
,
vmax_array
, vmax_service_level
and vmax_port_group
, as well
as the old driver location, will continue to work until the ‘V’ release.
Configuration option = Default value | Description |
---|---|
initiator_check = False |
(Boolean) Use this value to enable the initiator_check. |
interval = 3 |
(Integer) Use this value to specify length of the interval in seconds. |
load_balance = False |
(Boolean) Enable/disable load balancing for a PowerMax backend. |
load_balance_real_time = False |
(Boolean) Enable/disable real-time performance metrics for Port level load balancing for a PowerMax backend. |
load_data_format = Avg |
(String) Performance data format, not applicable for real-time metrics. Available options are “avg” and “max”. |
load_look_back = 60 |
(Integer) How far in minutes to look back for diagnostic performance metrics in load calculation, minimum of 0 maximum of 1440 (24 hours). |
load_look_back_real_time = 1 |
(Integer) How far in minutes to look back for real-time performance metrics in load calculation, minimum of 1 maximum of 10. |
port_group_load_metric = PercentBusy |
(String) Metric used for port group load calculation. |
port_load_metric = PercentBusy |
(String) Metric used for port load calculation. |
powermax_array = None |
(String) Serial number of the array to connect to. |
powermax_array_tag_list = None |
(List of String) List of user assigned name for storage array. |
powermax_port_group_name_template = portGroupName |
(String) User defined override for port group name. |
powermax_port_groups = None |
(List of String) List of port groups containing frontend ports configured prior for server connection. |
powermax_service_level = None |
(String) Service level to use for provisioning storage. Setting this as an extra spec in pool_name is preferable. |
powermax_short_host_name_template = shortHostName |
(String) User defined override for short host name. |
powermax_snapvx_unlink_limit = 3 |
(Integer) Use this value to specify the maximum number of unlinks for the temporary snapshots before a clone operation. |
powermax_srp = None |
(String) Storage resource pool on array to use for provisioning. |
retries = 200 |
(Integer) Use this value to specify number of retries. |
u4p_failover_autofailback = True |
(Boolean) If the driver should automatically failback to the primary instance of Unisphere when a successful connection is re-established. |
u4p_failover_backoff_factor = 1 |
(Integer) A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). Retries will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. |
u4p_failover_retries = 3 |
(Integer) The maximum number of retries each connection should attempt. Note, this applies only to failed DNS lookups, socket connections and connection timeouts, never to requests where data has made it to the server. |
u4p_failover_target = None |
(Dict of String) Dictionary of Unisphere failover target info. |
u4p_failover_timeout = 20.0 |
(Integer) How long to wait for the server to send data before giving up. |
vmax_workload = None |
(String) Workload, setting this as an extra spec in pool_name is preferable. |
Note
san_api_port
is 8443
by default but can be changed if
necessary. For the purposes of this documentation the default is
assumed so the tag will not appear in any of the cinder.conf
extracts below.
Note
PowerMax PortGroups
must be pre-configured to expose volumes managed
by the array. Port groups can be supplied in cinder.conf
, or
can be specified as an extra spec storagetype:portgroupname
on a
volume type. The latter gives the user more control. When a dynamic
masking view is created by the PowerMax driver, if there is no port group
specified as an extra specification, the port group is chosen randomly
from the PortGroup list, to evenly distribute load across the set of
groups provided.
Note
Service Level can be added to cinder.conf
when the backend is the
default case and there is no associated volume type. This not a recommended
configuration as it is too restrictive. Workload is NONE
for PowerMax
and any All Flash with PowerMax OS (5978) or greater.
PowerMax parameter | cinder.conf parameter | Default | Required |
---|---|---|---|
ServiceLevel |
powermax_service_level |
None |
No |
To configure PowerMax block storage, add the following entries to
/etc/cinder/cinder.conf
:
enabled_backends = CONF_GROUP_ISCSI, CONF_GROUP_FC
[CONF_GROUP_ISCSI]
volume_driver = cinder.volume.drivers.dell_emc.powermax.iscsi.PowerMaxISCSIDriver
volume_backend_name = POWERMAX_ISCSI
powermax_port_groups = [OS-ISCSI-PG]
san_ip = 10.10.10.10
san_login = my_username
san_password = my_password
powermax_array = 000123456789
powermax_srp = SRP_1
[CONF_GROUP_FC]
volume_driver = cinder.volume.drivers.dell_emc.powermax.fc.PowerMaxFCDriver
volume_backend_name = POWERMAX_FC
powermax_port_groups = [OS-FC-PG]
san_ip = 10.10.10.10
san_login = my_username
san_password = my_password
powermax_array = 000123456789
powermax_srp = SRP_1
In this example, two back-end configuration groups are enabled:
CONF_GROUP_ISCSI
and CONF_GROUP_FC
. Each configuration group has a
section describing unique parameters for connections, drivers and the
volume_backend_name
.
Get the CA certificate of the Unisphere server. This pulls the CA cert file
and saves it as .pem
file:
# openssl s_client -showcerts \
-connect my_unisphere_host:8443 \
</dev/null 2>/dev/null \
| openssl x509 -outform PEM > my_unisphere_host.pem
Where my_unisphere_host
is the hostname of the unisphere instance and
my_unisphere_host.pem
is the name of the .pem
file.
Add this path to cinder.conf
under the PowerMax backend stanza and set
SSL verify to True
driver_ssl_cert_verify = True
driver_ssl_cert_path = /path/to/my_unisphere_host.pem
OR
follow the steps 3-6 below if you would like to add the CA cert to
the system certificate bundle instead of specifying the path to cert:
OPTIONAL: Copy the .pem
cert to the system certificate
directory and convert to .crt
:
# cp my_unisphere_host.pem /usr/share/ca-certificates/ca_cert.crt
OPTIONAL: Update CA certificate database with the following command. Ensure you select to enable the cert from step 3 when prompted:
# sudo dpkg-reconfigure ca-certificates
OPTIONAL: Set a system environment variable to tell the Requests library to use the system cert bundle instead of the default Certifi bundle:
# export REQUESTS_CA_BUNDLE = /etc/ssl/certs/ca-certificates.crt
OPTIONAL: Set cert verification to True
under the PowerMax backend
stanza in cinder.conf
:
# driver_ssl_cert_verify = True
Ensure driver_ssl_cert_verify
is set to True
in cinder.conf
backend stanzas if steps 3-6 are followed, otherwise ensure both
driver_ssl_cert_path
and driver_ssl_cert_verify
are set in
cinder.conf
backend stanzas.
Once cinder.conf
has been updated, Openstack CLI commands need to be
issued in order to create and associate OpenStack volume types with the
declared volume_backend_names
.
Additionally, each volume type will need an associated pool_name
- an
extra specification indicating the service level/ workload combination to
be used for that volume type.
Note
The pool_name
is an additional property which has to be set and is of
the format: <ServiceLevel>+<SRP>+<Array ID>
. This can be obtained from
the output of the cinder get-pools--detail
. Workload is NONE for
PowerMax or any All Flash with PowerMax OS (5978) or greater.
There is also the option to assign a port group to a volume type by
setting the storagetype:portgroupname
extra specification.
$ openstack volume type create POWERMAX_ISCSI_SILVER
$ openstack volume type set --property volume_backend_name=ISCSI_backend \
--property pool_name=Silver+SRP_1+000123456789 \
--property storagetype:portgroupname=OS-PG2 \
POWERMAX_ISCSI_SILVER
$ openstack volume type create POWERMAX_FC_DIAMOND
$ openstack volume type set --property volume_backend_name=FC_backend \
--property pool_name=Gold+SRP_1+000123456789 \
--property storagetype:portgroupname=OS-PG1 \
POWERMAX_FC_GOLD
By issuing these commands, the Block Storage volume type
POWERMAX_ISCSI_SILVER
is associated with the ISCSI_backend
, a Silver
Service Level.
The type POWERMAX_FC_DIAMOND
is associated with the FC_backend
, a
Diamond Service Level.
The ServiceLevel
manages the underlying storage to provide expected
performance. Setting the ServiceLevel
to None
means that non-FAST
managed storage groups will be created instead (storage groups not
associated with any service level).
openstack volume type set --property pool_name=None+SRP_1+000123456789
Note
PowerMax and Hybrid support Diamond
, Platinum
, Gold
, Silver
,
Bronze
, Optimized
, and None
service levels. VMAX All Flash
running HyperMax OS (5977) supports Diamond
and None
. Hybrid and All
Flash support DSS_REP
, DSS
, OLTP_REP
, OLTP
, and None
workloads, the latter up until ucode 5977. Please refer to Stein PowerMax
online documentation if you wish to use workload
. There is no support
for workloads in PowerMax OS (5978) or greater. These will be silently
ignored if set for VMAX All-Flash arrays which have been upgraded to
PowerMax OS (5988).
By default, interval
and retries
are 3
seconds and 200
retries
respectively. These determine how long (interval
) and how many times
(retries
) a user is willing to wait for a single Rest call,
3*200=600seconds
. Depending on usage, these may need to be overridden by
the user in cinder.conf
. For example, if performance is a factor, then the
interval
should be decreased to check the job status more frequently, and
if multiple concurrent provisioning requests are issued then retries
should be increased so calls will not timeout prematurely.
In the example below, the driver checks every 3 seconds for the status of the job. It will continue checking for 200 retries before it times out.
Add the following lines to the PowerMax backend in cinder.conf
:
[CONF_GROUP_ISCSI]
volume_driver = cinder.volume.drivers.dell_emc.powermax.iscsi.PowerMaxISCSIDriver
volume_backend_name = POWERMAX_ISCSI
powermax_port_groups = [OS-ISCSI-PG]
san_ip = 10.10.10.10
san_login = my_username
san_password = my_password
powermax_array = 000123456789
powermax_srp = SRP_1
interval = 1
retries = 700
This supports one-way initiator CHAP authentication functionality into the PowerMax backend. With CHAP one-way authentication, the storage array challenges the host during the initial link negotiation process and expects to receive a valid credential and CHAP secret in response. When challenged, the host transmits a CHAP credential and CHAP secret to the storage array. The storage array looks for this credential and CHAP secret which stored in the host initiator’s initiator group (IG) information in the ACLX database. Once a positive authentication occurs, the storage array sends an acceptance message to the host. However, if the storage array fails to find any record of the credential/secret pair, it sends a rejection message, and the link is closed.
The host initiator IQN is required along with the credentials the host initiator will use to log into the storage array with. The same credentials should be used in a multi node system if connecting to the same array.
Enable one-way CHAP authentication for the iSCSI initiator on the storage
array using SYMCLI
. Template and example shown below. For the purpose of
this setup, the credential/secret used would be my_username/my_password
with iSCSI initiator of iqn.1991-05.com.company.lcseb130
# symaccess -sid <SymmID> -iscsi <iscsi> \
{enable chap | disable chap | set chap} \
-cred <Credential> -secret <Secret>
# symaccess -sid 128 \
-iscsi iqn.1991-05.com.company.lcseb130 \
set chap -cred my_username -secret my_password
Set the configuration in the PowerMax backend group in cinder.conf
using
the following parameters and restart cinder.
Configuration options | Value required for CHAP | Required for CHAP |
---|---|---|
use_chap_auth |
True |
Yes |
chap_username |
my_username |
Yes |
chap_password |
my_password |
Yes |
[POWERMAX_ISCSI]
volume_driver = cinder.volume.drivers.dell_emc.powermax.iscsi.PowerMaxISCSIDriver
volume_backend_name = POWERMAX_ISCSI
san_ip = 10.10.10.10
san_login = my_u4v_username
san_password = my_u4v_password
powermax_srp = SRP_1
powermax_array = 000123456789
powermax_port_groups = [OS-ISCSI-PG]
use_chap_auth = True
chap_username = my_username
chap_password = my_password
Using SYMCLI
, enable CHAP authentication for a host initiator as
described above, but do not set use_chap_auth
, chap_username
or
chap_password
in cinder.conf
. Create a bootable volume.
openstack volume create --size 1 \
--image <image_name> \
--type <volume_type> \
test
Boot instance named test_server
using the volume created above:
openstack server create --volume test \
--flavor m1.small \
--nic net-id=private \
test_server
Verify the volume operation succeeds but the boot instance fails as CHAP authentication fails.
Update cinder.conf
with use_chap_auth
set to true and
chap_username
and chap_password
set with the correct credentials.
Rerun openstack server create
Verify that the boot instance operation ran correctly and the volume is accessible.
Verify that both the volume and boot instance operations ran successfully and the user is able to access the volume.
Quality of service (QoS) has traditionally been associated with network bandwidth usage. Network administrators set limitations on certain networks in terms of bandwidth usage for clients. This enables them to provide a tiered level of service based on cost. The Nova/Cinder QoS offer similar functionality based on volume type setting limits on host storage bandwidth per service offering. Each volume type is tied to specific QoS attributes some of which are unique to each storage vendor. In the hypervisor, the QoS limits the following:
QoS enforcement in Cinder is done either at the hyper-visor (front-end), the storage subsystem (back-end), or both. This section focuses on QoS limits that are enforced by either the PowerMax backend and the hyper-visor front end interchangeably or just back end (Vendor Specific). The PowerMax driver offers support for Total bytes/sec limit in throughput and Total IOPS/sec limit of IOPS.
The PowerMax driver supports the following attributes that are front end/back end agnostic
total_iops_sec
- Maximum IOPs (in I/Os per second). Valid values range
from 100 IO/Sec to 100000 IO/sec.total_bytes_sec
- Maximum bandwidth (throughput) in bytes per second.
Valid values range from 1048576 bytes (1MB) to 104857600000 bytes (100,000MB)The PowerMax driver offers the following attribute that is vendor specific to
the PowerMax and dependent on the total_iops_sec
and/or total_bytes_sec
being set.
Dynamic Distribution
- Enables/Disables dynamic distribution of host I/O
limits. Possible values are:Always
- Enables full dynamic distribution mode. When enabled, the
configured host I/O limits will be dynamically distributed across the
configured ports, thereby allowing the limits on each individual port to
adjust to fluctuating demand.OnFailure
- Enables port failure capability. When enabled, the fraction
of configured host I/O limits available to a configured port will adjust
based on the number of ports currently online.Never
- Disables this feature (Default).Prerequisites - PowerMax
Key | Value |
---|---|
total_iops_sec |
500 |
total_bytes_sec |
104857600 (100MB) |
DistributionType |
Always |
Create QoS Specs with the prerequisite values above:
$ openstack volume qos create --consumer back-end \
--property total_iops_sec=500 \
--property total_bytes_sec=104857600 \
--property DistributionType=Always \
my_qos
Associate QoS specs with specified volume type:
$ openstack volume qos associate my_qos my_volume_type
Create volume with the volume type indicated above:
$ openstack volume create --size 1 --type my_volume_type my_volume
Outcome - PowerMax (storage group)
100
500
Always
Outcome - Block Storage (Cinder)
Volume is created against volume type and QoS is enforced with the parameters above.
Prerequisites - PowerMax
2000
2000
Never
Key | Value |
---|---|
total_iops_sec |
500 |
total_bytes_sec |
104857600 (100MB) |
DistributionType |
Always |
Create QoS specifications with the prerequisite values above. The consumer in this use case is both for front-end and back-end:
$ openstack volume qos create --consumer back-end \
--property total_iops_sec=500 \
--property total_bytes_sec=104857600 \
--property DistributionType=Always \
my_qos
Associate QoS specifications with specified volume type:
$ openstack volume qos associate my_qos my_volume_type
Create volume with the volume type indicated above:
$ openstack volume create --size 1 --type my_volume_type my_volume
Attach the volume created in step 3 to an instance
$ openstack server add volume my_volume my_instance
Outcome - PowerMax (storage group)
100
500
Always
Outcome - Block Storage (Cinder)
Volume is created against volume type and QoS is enforced with the parameters above.
Outcome - Hypervisor (Nova)
Libvirt
includes an extra xml
flag within the <disk>
section called
iotune
that is responsible for rate limitation. To confirm that, first get
the OS-EXT-SRV-ATTR:instance_name
value of the server instance,
for example instance-00000003
.
$ openstack server show <serverid>
+-------------------------------------+-----------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | myhost |
| OS-EXT-SRV-ATTR:hypervisor_hostname | myhost |
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-11-02T08:15:42.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | private=fd21:99c2:73f3:0:f816:3eff:febe:30ed, 10.0.0.3 |
| config_drive | |
| created | 2017-11-02T08:15:34Z |
| flavor | m1.tiny (1) |
| hostId | e7b8312581f9fbb8508587d45c0b6fb4dc86102c632ed1f3a6a49d42 |
| id | 0ef0ff4c-dbda-4dc7-b8ed-45d2fc2f31db |
| image | cirros-0.3.5-x86_64-disk (b7c220f5-2408-4296-9e58-fc5a41cb7e9d) |
| key_name | myhostname |
| name | myhosthame |
| progress | 0 |
| project_id | bae4b97a0d8b42c28a5add483981e5db |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2017-11-02T08:15:42Z |
| user_id | 7bccf456740546799a7e20457f13c38b |
| volumes_attached | |
+-------------------------------------+-----------------------------------------------------------------+
We then run the following command using the OS-EXT-SRV-ATTR:instance_name
retrieved above.
$ virsh dumpxml instance-00000003 | grep -1 "total_bytes_sec\|total_iops_sec"
The output of the command contains the XML below. It is found between the
<disk>
start and end tag.
<iotune>
<total_bytes_sec>104857600</total_bytes_sec>
<total_iops_sec>500</total_iops_sec>
</iotune>
Prerequisites - PowerMax
100
500
Always
Key | Value |
---|---|
total_iops_sec |
500 |
total_bytes_sec |
104857600 (100MB) |
DistributionType |
OnFailure |
Create QoS specifications with the prerequisite values above:
$ openstack volume qos create --consumer back-end \
--property total_iops_sec=500 \
--property total_bytes_sec=104857600 \
--property DistributionType=Always \
my_qos
Associate QoS specifications with specified volume type:
$ openstack volume qos associate my_qos my_volume
Create volume with the volume type indicated above:
$ openstack volume create --size 1 --type my_volume_type my_volume
Outcome - PowerMax (storage group)
100
500
OnFailure
Outcome - Block Storage (Cinder)
Volume is created against volume type and QOS is enforced with the parameters above.
Prerequisites - PowerMax
No Limit
No Limit
N/A
Key | Value |
---|---|
DistributionType |
Always |
Create QoS specifications with the prerequisite values above:
$ openstack volume qos create --consumer back-end \
--property DistributionType=Always \
my_qos
Associate QoS specifications with specified volume type:
$ openstack volume qos associate my_qos my_volume_type
Create volume with the volume type indicated above:
$ openstack volume create --size 1 --type my_volume_type my_volume
Outcome - PowerMax (storage group)
No Limit
No Limit
N/A
Outcome - Block Storage (Cinder)
Volume is created against volume type and there is no QoS change.
open-iscsi
on all nodes on your systemOn Ubuntu:
# apt-get install multipath-tools #multipath modules
# apt-get install sysfsutils sg3-utils #file system utilities
# apt-get install scsitools #SCSI tools
On openSUSE and SUSE Linux Enterprise Server:
# zipper install multipath-tools #multipath modules
# zipper install sysfsutils sg3-utils #file system utilities
# zipper install scsitools #SCSI tools
On Red Hat Enterprise Linux and CentOS:
# yum install iscsi-initiator-utils #ensure iSCSI is installed
# yum install device-mapper-multipath #multipath modules
# yum install sysfsutils sg3-utils #file system utilities
The multi-path configuration file may be edited for better management and
performance. Log in as a privileged user and make the following changes to
/etc/multipath.conf
on the Compute (Nova) node(s).
devices {
# Device attributed for EMC PowerMax
device {
vendor "EMC"
product "SYMMETRIX"
path_grouping_policy multibus
getuid_callout "/lib/udev/scsi_id --page=pre-spc3-83 --whitelisted --device=/dev/%n"
path_selector "round-robin 0"
path_checker tur
features "0"
hardware_handler "0"
prio const
rr_weight uniform
no_path_retry 6
rr_min_io 1000
rr_min_io_rq 1
}
}
You may need to reboot the host after installing the MPIO tools or restart iSCSI and multi-path services.
On Ubuntu:
# service open-iscsi restart
# service multipath-tools restart
On openSUSE, SUSE Linux Enterprise Server, Red Hat Enterprise Linux, and CentOS:
# systemctl restart open-iscsi
# systemctl restart multipath-tools
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1G 0 disk
..360000970000196701868533030303235 (dm-6) 252:6 0 1G 0 mpath
sdb 8:16 0 1G 0 disk
..360000970000196701868533030303235 (dm-6) 252:6 0 1G 0 mpath
vda 253:0 0 1T 0 disk
On Compute (Nova) node, add the following flag in the [libvirt]
section of
nova.conf
and nova-cpu.conf
:
volume_use_multipath = True
On Cinder controller node, multi-path for image transfer can be enabled in
cinder.conf
for each backend section or in [backend_defaults]
section
as a common configuration for all backends.
use_multipath_for_image_xfer = True
Restart nova-compute
and cinder-volume
services after the change.
Create a 3GB PowerMax volume.
Create an instance from image out of native LVM storage or from PowerMax storage, for example, from a bootable volume
Attach the 3GB volume to the new instance:
# multipath -ll
mpath102 (360000970000196700531533030383039) dm-3 EMC,SYMMETRIX
size=3G features='1 queue_if_no_path' hwhandler='0' wp=rw
'-+- policy='round-robin 0' prio=1 status=active
33:0:0:1 sdb 8:16 active ready running
'- 34:0:0:1 sdc 8:32 active ready running
Use the lsblk
command to see the multi-path device:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE
sdb 8:0 0 3G 0 disk
..360000970000196700531533030383039 (dm-6) 252:6 0 3G 0 mpath
sdc 8:16 0 3G 0 disk
..360000970000196700531533030383039 (dm-6) 252:6 0 3G 0 mpath
vda
On an All Flash array, the creation of any storage group has a compressed
attribute by default. Setting compression on a storage group does not mean
that all the devices will be immediately compressed. It means that for all
incoming writes compression will be considered. Setting compression off
on
a storage group does not mean that all the devices will be uncompressed.
It means all the writes to compressed tracks will make these tracks
uncompressed.
Note
This feature is only applicable for All Flash arrays, 250F, 450F, 850F and 950F and PowerMax 2000 and 8000. It was first introduced Solutions Enabler 8.3.0.11 or later and is enabled by default when associated with a Service Level. This means volumes added to any newly created storage groups will be compressed.
POWERMAX_COMPRESSION_DISABLED
.volume_backend_name
.storagetype:disablecompression = True
.OS-<srp>-<servicelevel>-<workload>-CD-SG
, and
compression is disabled on that storage group.OS-<shorthostname>-<srp>-<servicelevel/workload>-<portgroup>-CD
, and
compression is disabled on that storage group.OS-<srp>-<servicelevel>-<workload>-CD-SG
,
and compression is disabled on that storage group.OS-<srp>-<servicelevel>-<workload>-CD-SG
storage group,
it should also be deleted.POWERMAX_COMPRESSION_ENABLED
.volume_backend_name
as before.storagetype:disablecompression = False
or DO NOT set this extra spec.POWERMAX_COMPRESSION_DISABLED
to
POWERMAX_COMPRESSION_ENABLED
.OS-<srp>-<servicelevel>-<workload>-SG
, and compression is enabled on
that storage group.Note
If extra spec storagetype:disablecompression
is set on a Hybrid, it is
ignored because compression is not an available feature on a VMAX3 Hybrid.
Please refer to the official OpenStack over-subscription documentation for further information on using over-subscription with PowerMax.
Non-live migration (sometimes referred to simply as ‘migration’). The instance is shut down for a period of time to be moved to another hyper-visor. In this case, the instance recognizes that it was rebooted.
Live migration (or ‘true live migration’). Almost no instance downtime. Useful when the instances must be kept running during the migration. The different types of live migration are:
The PowerMax driver supports shared volume-backed live migration.
In PowerMax, A volume cannot belong to two or more FAST storage groups at the same time. To get around this limitation we leverage both cascaded storage groups and a temporary non-FAST storage group.
A volume can remain ‘live’ if moved between masking views that have the same initiator group and port groups which preserves the host path.
During live migration, the following steps are performed by the PowerMax driver on the volume:
Please refer to the official OpenStack documentation on configuring migrations and live migration usage for more information.
Note
OpenStack Oslo uses an open standard for messaging middleware known as
AMQP
. This messaging middleware (the RPC messaging system) enables the
OpenStack services that run on multiple servers to talk to each other.
By default, the RPC messaging client is set to timeout after 60 seconds,
meaning if any operation you perform takes longer than 60 seconds to
complete the operation will timeout and fail with the ERROR message
Messaging Timeout: Timed out waiting for a reply to message ID
[message_id]
If this occurs, increase the rpc_response_timeout
flag value in
cinder.conf
and nova.conf
on all Cinder and Nova nodes and restart
the services.
What to change this value to will depend entirely on your own environment, you might only need to increase it slightly, or if your environment is under heavy network load it could need a bit more time than normal. Fine tuning is required here, change the value and run intensive operations to determine if your timeout value matches your environment requirements.
At a minimum please set rpc_response_timeout
to 240
, but this will
need to be raised if high concurrency is a factor. This should be
sufficient for all Cinder backup commands also.
NOVA-INST-DIR/instances/
(for example, /opt/stack/data/nova/instances
)
has to be mounted by shared storage. Ensure that NOVA-INST-DIR
(set with
state_path
in the nova.conf
file) is the same on all hosts.
Configure your DNS or /etc/hosts
and ensure it is consistent across all
hosts. Make sure that the three hosts can perform name resolution with each
other. As a test, use the ping command to ping each host from one another.
$ ping HostA
$ ping HostB
$ ping HostC
Export NOVA-INST-DIR/instances
from HostA
, and ensure it is readable
and writable by the Compute user on HostB
and HostC
. Please refer to
the relevant OS documentation for further details, for example
Ubuntu NFS Documentation
On all compute nodes, enable the execute/search
bit on your shared
directory to allow qemu
to be able to use the images within the
directories. On all hosts, run the following command:
$ chmod o+x NOVA-INST-DIR/instances
Note
If migrating from compute to controller, make sure to run step two above on the controller node to export the instance directory.
For our use case shown below, we have three hosts with host names HostA
,
HostB
and HostC
. HostA
is the controller node while HostB
and
HostC
are the compute nodes. The following were also used in live
migration.
m1.small
using
2048 RAM, 20GB of Disk and 1 VCPU.Create a bootable volume.
$ openstack volume create --size 2 \
--image cirros-0.3.5-x86_64-disk \
--volume_lm_1
Launch an instance using the volume created above on HostB
.
$ openstack server create --volume volume_lm_1 \
--flavor m1.small \
--nic net-id=private \
--security-group default \
--availability-zone nova:HostB \
server_lm_1
Confirm on HostB
has the instance created by running:
$ openstack server show server_lm_1 | grep "hypervisor_hostname\|instance_name"
| OS-EXT-SRV-ATTR:hypervisor_hostname | HostB
| OS-EXT-SRV-ATTR:instance_name | instance-00000006
Confirm, through virsh
using the instance_name returned in step 3
(instance-00000006
), on HostB
that the instance is created using:
$ virsh list --all
Id Name State
--------------------------------
1 instance-00000006 Running
Migrate the instance from HostB
to HostA
with:
$ openstack server migrate --live HostA \
server_lm_1
Run the command on step 3 above when the instance is back in available status. The hypervisor should be on Host A.
Run the command on Step 4 on Host A to confirm that the instance is
created through virsh
.
PowerMax cinder driver supports the ability to attach a volume to multiple hosts/servers simultaneously. Please see the official OpenStack multi-attach documentation for configuration information.
In PowerMax, a volume cannot belong to two or more FAST storage groups at the same time. This can cause issues when we are attaching a volume to multiple instances on different hosts. To get around this limitation, we leverage both cascaded storage groups and non-FAST storage groups (i.e. a storage group with no service level, workload, or SRP specified).
Note
If no service level is assigned to the volume type, no extra work on the backend is required – the volume is attached to and detached from each host as normal.
Volume Multi-attach-Vol-1
(with a multi-attach capable volume type, and
associated with a Diamond Service Level) is attached to Instance
Multi-attach-Instance-A
on HostA. We then issue the command to attach
Multi-attach-Vol-1
to Multi-attach-Instance-B
on HostB:
HostA
masking view, the volume is moved from the FAST managed
storage group to the non-FAST managed storage group within the parent
storage group.HostB
– i.e., it is added to a FAST
managed storage group within the parent storage group of the HostB
masking view. The volume now belongs to two masking views, and is exposed to
both HostA
and HostB
.We then decide to detach the volume from Multi-attach-Instance-B
on
HostB
:
HostB
– i.e., it is removed from
the FAST managed storage group within the parent storage group of the
HostB
masking view – this includes cleanup of the associated elements
if required. The volume now belongs to one masking view, and is no longer
exposed to HostB
.HostA
masking view, the volume is returned to the FAST managed
storage group from the non-FAST managed storage group within the parent
storage group. The non-FAST managed storage group is cleaned up,
if required.Encryption is supported through the use of OpenStack Barbican. Only front-end encryption is supported, back-end encryption is handled at the hardware level with Data at Rest Encryption (D@RE).
For further information on OpenStack Barbican including setup and configuration please refer to the following official Barbican documentation.
Volume metadata is returned to the user in both the Cinder Volume logs and with volumes and snapshots created in Cinder via the UI or CLI.
If debug is enabled in the default section of cinder.conf
, PowerMax Cinder
driver will log additional volume information in the Cinder volume log,
on each successful operation. The facilitates bridging the gap between
OpenStack and the Array by tracing and describing the volume from a VMAX/
PowerMax view point.
+------------------------------------+---------------------------------------------------------+
| Key | Value |
+------------------------------------+---------------------------------------------------------+
| service_level | Gold |
| is_compression_disabled | no |
| powermax_cinder_driver_version | 3.2.0 |
| identifier_name | OS-819470ab-a6d4-49cc-b4db-6f85e82822b7 |
| openstack_release | 13.0.0.0b3.dev3 |
| volume_id | 819470ab-a6d4-49cc-b4db-6f85e82822b7 |
| storage_model | PowerMax_8000 |
| successful_operation | delete |
| default_sg_name | OS-DEFAULT_SRP-Gold-NONE-SG |
| device_id | 01C03 |
| unisphere_for_powermax_version | V9.0.0.9 |
| workload | NONE |
| openstack_version | 13.0.0 |
| volume_updated_time | 2018-08-03 03:13:53 |
| platform | Linux-4.4.0-127-generic-x86_64-with-Ubuntu-16.04-xenial |
| python_version | 2.7.12 |
| volume_size | 20 |
| srp | DEFAULT_SRP |
| openstack_name | 90_Test_Vol56 |
| storage_firmware_version | 5978.143.144 |
| serial_number | 000123456789 |
+------------------------------------+---------------------------------------------------------+
By default metadata will be set on all volume and snapshot objects created in Cinder. This information represents the state of the object on the backend PowerMax and will change when volume attributes are changed by performing actions on them such as re-type or attaching to an instance.
demo@openstack-controller:~$ cinder show powermax-volume
+--------------------------------+------------------------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------------------------+
| metadata | ArrayID : 000123456789 |
| | ArrayModel : PowerMax_8000 |
| | CompressionDisabled : False |
| | Configuration : TDEV |
| | DeviceID : 0012F |
| | DeviceLabel : OS-d87edb98-60fd-49dd-bb0f-cc388cf6f3f4 |
| | Emulation : FBA |
| | ReplicationEnabled : False |
| | ServiceLevel : Diamond |
| | Workload : None |
| name | powermax-volume |
+--------------------------------+------------------------------------------------------------+
This feature facilitates high availability of Unisphere for PowerMax servers,
allowing for one or more backup unisphere instances in the event of a loss in
connection to the primary Unisphere instance. The PowerMax driver will
cycle through the list of failover instances, trying each until a successful
connection is made. The ordering is first in, first out (FIFO), so the first
u4p_failover_target
specified in cinder.conf
will be the first
selected, the second u4p_failover_target
in cinder.conf
will be the
second selected, and so on until all failover targets are exhausted.
Note
Replication must be configured in advance for remote failover to work
successfully. Human intervention will also be required to failover from R1
array to R2 array in Cinder using cinder failover-host
command
(see Volume replication support for replication setup details).
Note
The remote target array must be registered as local to the remote instance of Unisphere
The following configuration changes need to be made in cinder.conf
in order
to support the failover to secondary Unisphere. Cinder services will need to
be restarted for changes to take effect.
u4p_failover_timeout = 30
u4p_failover_retries = 3
u4p_failover_backoff_factor = 1
u4p_failover_autofailback = True
u4p_failover_target = san_ip:10.10.10.12,
san_api_port: 8443,
san_login:my_username,
san_password:my_password,
driver_ssl_cert_verify: False,
u4p_failover_target = san_ip:10.10.10.13,
san_api_port: 8443
san_login:my_username,
san_password:my_password,
driver_ssl_cert_verify: True,
driver_ssl_cert_path: /path/to/my_unisphere_host.pem
Note
u4p_failover_target
key value pairs will need to be on the same
line (separated by commas) in cinder.conf
. They are displayed on
separated lines above for readability.
Note
To add more than one Unisphere failover target create additional
u4p_failover_target
details for the Unisphere instance. These will be
cycled through in a first-in, first-out (FIFO) basis, the first failover
target in cinder.conf
will be the first backup instance of Unisphere
used by the PowerMax driver.
The PowerMax driver can now leverage the enhanced volume delete feature-set made available in the PowerMax 5978 Foxtail uCode release. These enhancements allow volume deallocation & deletion to be combined into a single call. Previously, volume deallocation & deletion were split into separate tasks; now a single REST call is dispatched and a response code on the projected outcome of their request is issued rapidly allowing other task execution to proceed without the delay. No additional configuration is necessary, the system will automatically determine when to use either the rapid or legacy compliant volume deletion sequence based on the connected PowerMax array’s metadata.
uCode Level | Supported In-Use Volume Extend Operations | |||
---|---|---|---|---|
R1 uCode Level | R2 uCode Level | Sync | Async | Metro |
5978.444 | 5978.444 | Y | Y | Y |
5978.444 | 5978.221 | Y | Y | N |
5978.221 | 5978.221 | Y | Y | N |
allow_extend
is only applicable on Hybrid arrays or All Flash arrays
with HyperMax OS. If included elsewhere, it is ignored.Unisphere for PowerMax 9.1 supports tagging of storage groups and arrays, so the user can give their own ‘tag’ for ease of searching and/or grouping.
storagetype:storagegrouptags
.powermax_array_tag_list
. It expects a list of one or more comma
separated values, for example
powermax_array_tag_list=[value1,value2, value3]
Set a storage group tag on a volume type:
$ openstack volume type set --property storagetype:storagegrouptags=myStorageGroupTag1,myStorageGroupTag2
Set an array tag on the PowerMax backend:
[POWERMAX_ISCSI]
volume_driver = cinder.volume.drivers.dell_emc.powermax.iscsi.PowerMaxISCSIDriver
volume_backend_name = POWERMAX_ISCSI
san_ip = 10.10.10.10
san_login = my_u4v_username
san_password = my_u4v_password
powermax_srp = SRP_1
powermax_array = 000123456789
powermax_port_groups = [OS-ISCSI-PG]
powermax_array_tag_list = [openstack1, openstack2]
This functionality allows the user to customize the short host name and port group name that are contained in the PowerMax driver storage groups and masking views names. For current functionality please refer to PowerMax naming conventions for more details.
As the storage group name and masking view name are limited to 64 characters the short host name needs to be truncated to 16 characters or less and port group needs to be truncated to 12 characters or less. This functionality offers a little bit more flexibility to determine how these truncated components should look.
cinder.conf
will have 2 new configuration options,
short_host_name_template
and port_group_name_template
.powermax_short_host_name_template
and
powermax_port_group_name_template
do not adhere to the rules, then
the operation will fail early and gracefully with a clear description as
to the problem.userdef
is used, the onus is on the user to make sure it will be
unique among all short host names (controller and compute nodes) and
unique among port groups.powermax_short_host_name_template | Description | Rule |
---|---|---|
shortHostName | This is the default option | Existing functionality, if over 16 characters then see PowerMax naming conventions, otherwise short host name |
shortHostName[:x])uuid[:x] e.g. shortHostName[:6]uuid[:9] | First x characters of the short host name and x uuid characters created from md5 hash of short host name | Must be less than 16 characters |
shortHostName[:x]userdef e.g. shortHostName[:6]-testHost | First x characters of the short host name and a user defined x char name. NB - the responsibility is on the user for uniqueness | Must be less than 16 characters |
shortHostName[-x:]uuid[:x] e.g. shortHostName[-6:]uuid[:9] | Last x characters of the short host name and x uuid characters created from md5 hash of short host name | Must be less than 16 characters |
shortHostName[-x:]userdef e.g. shortHostName[-6:]-testHost | Last x characters of the short host name and a user defined x char name. NB - the responsibility is on the user for uniqueness | Must be less than 16 characters |
powermax_port_group_name_template | Description | Rule |
---|---|---|
portGroupName | This is the default option | Existing functionality, if over 12 characters then see PowerMax naming conventions, otherwise port group name |
portGroupName[:x])uuid[:x] e.g. portGroupName[:6]uuid[:5] | First x characters of the port group name and x uuid characters created from md5 hash of port group name | Must be less than 12 characters |
portGroupName[:x]userdef e.g. portGroupName[:6]-test | First x characters of the port group name and a user defined x char name. NB - the responsibility is on the user for uniqueness | Must be less than 12 characters |
portGroupName[-x:]uuid[:x] e.g. portGroupName[-6:]uuid[:5] | Last x characters of the port group name and x uuid characters created from md5 hash of port group name | Must be less than 12 characters |
portGroupName[-x:]userdef e.g. portGroupName[-6:]-test | Last x characters of the port group name and a user defined x char name. NB - the responsibility is on the user for uniqueness | Must be less than 12 characters |
Configure an SRDF group between the chosen source and target
arrays for the PowerMax Cinder driver to use. The source array must
correspond with the powermax_array
entry in cinder.conf
.
Select both the director and the ports for the SRDF emulation to use on
both sides. Bear in mind that network topology is important when choosing
director endpoints. Supported modes are Synchronous
, Asynchronous
,
and Metro
.
Note
If the source and target arrays are not managed by the same Unisphere
server (that is, the target array is remotely connected to server -
for example, if you are using embedded management), in the event of a
full disaster scenario (i.e. the primary array is completely lost and
all connectivity to it is gone), the Unisphere server would no longer
be able to contact the target array. In this scenario, the volumes would
be automatically failed over to the target array, but administrator
intervention would be required to either; configure the target (remote)
array as local to the current Unisphere server (if it is a stand-alone
server), or enter the details of a second Unisphere server to the
cinder.conf
, which is locally connected to the target array (for
example, the embedded management Unisphere server of the target array),
and restart the Cinder volume service.
Note
If you are setting up an SRDF/Metro configuration, it is recommended that you configure a Witness or vWitness for bias management. Please see the SRDF Metro Overview & Best Practices guide for more information.
Note
The PowerMax Cinder drivers do not support Cascaded SRDF.
Enable replication in /etc/cinder/cinder.conf
.
To enable the replication functionality in PowerMax Cinder driver, it is
necessary to create a replication volume-type. The corresponding
back-end stanza in cinder.conf
for this volume-type must then
include a replication_device
parameter. This parameter defines a
single replication target array and takes the form of a list of key
value pairs.
enabled_backends = POWERMAX_FC_REPLICATION
[POWERMAX_FC_REPLICATION]
volume_driver = cinder.volume.drivers.dell_emc.powermax.fc.PowerMaxFCDriver
san_ip = 10.10.10.10
san_login = my_u4v_username
san_password = my_u4v_password
powermax_srp = SRP_1
powermax_array = 000123456789
powermax_port_groups = [OS-FC-PG]
volume_backend_name = POWERMAX_FC_REPLICATION
replication_device = target_device_id:000197811111,
remote_port_group:os-failover-pg,
remote_pool:SRP_1,
rdf_group_label: 28_11_07,
mode:Metro,
metro_use_bias:False,
sync_interval:3,
sync_retries:200
Note
replication_device
key value pairs will need to be on the same
line (separated by commas) in cinder.conf
. They are displayed here on
separate lines above for improved readability.
target_device_id
The unique PowerMax array serial number of the
target array. For full failover functionality, the source and target
PowerMax arrays must be discovered and managed by the same U4V server.
remote_port_group
The name of a PowerMax port group that has been
pre-configured to expose volumes managed by this backend in the event
of a failover. Make sure that this port group contains either all FC or
all iSCSI port groups (for a given back end), as appropriate for the
configured driver (iSCSI or FC).
remote_pool
The unique pool name for the given target array.
rdf_group_label
The name of a PowerMax SRDF group that has been
pre-configured between the source and target arrays.
mode
The SRDF replication mode. Options are Synchronous
,
Asynchronous
, and Metro
. This defaults to Synchronous
if not
set.
metro_use_bias
Flag to indicate if ‘bias’ protection should be
used instead of Witness. This defaults to False.
sync_interval
How long in seconds to wait between intervals for SRDF
sync checks during Cinder PowerMax SRDF operations. Default is 3 seconds.
sync_retries
How many times to retry RDF sync checks during Cinder
PowerMax SRDF operations. Default is 200 retries.
allow_extend
Only applicable to Hybrid arrays or All Flash arrays
running HyperMax OS (5977). It is a flag for allowing the extension of
replicated volumes. To extend a volume in an SRDF relationship, this
relationship must first be broken, the R1 device extended, and a new
device pair established. If not explicitly set, this flag defaults to
False
.
Note
As the SRDF link must be severed, due caution should be exercised when performing this operation. If absolutely necessary, only one source and target pair should be extended at a time (only only applicable to Hybrid arrays or All Flash arrays with HyperMax OS).
Create a replication-enabled
volume type. Once the
replication_device
parameter has been entered in the PowerMax
backend entry in the cinder.conf
, a corresponding volume type
needs to be created replication_enabled
property set. See
above 6. Create volume types for details.
# openstack volume type set --property replication_enabled="<is> True" \
POWERMAX_FC_REPLICATION
Note
Service Level and Workload: An attempt will be made to create a storage group on the target array with the same service level and workload combination as the primary. However, if this combination is unavailable on the target (for example, in a situation where the source array is a Hybrid, the target array is an All Flash, and an All Flash incompatible service level like Bronze is configured), no service level will be applied.
Setting multiple replication devices in cinder.conf allows the use of all the
supported replication modes simultaneously. Up to three replication devices
can be set, one for each of the replication modes available. An additional
volume type extra spec
(storagetype:replication_device_backend_id
) is
then used to determine which replication device should be utilized when
attempting to perform an operation on a volume which is replication enabled.
All details, guidelines and recommendations set out in the
Configure a single replication target section also apply in a multiple
replication device scenario.
cinder.conf
.target_device_id
, remote_port_group
and
remote_pool
should be identical across replication devices.backend_id
and rdf_group_label
values must be unique across
all replication devices.replication_device
to cinder.conf:cinder.conf
for editingbackend_id
key
with a value of backend_id_legacy_rep
. If this key is already
defined, it’s value must be updated to backend_id_legacy_rep
.backend_id
key set. The
value of these must not
be backend_id_legacy_rep
.Example existing backend stanza pre-multiple replication:
enabled_backends = POWERMAX_FC_REPLICATION
[POWERMAX_FC_REPLICATION]
volume_driver = cinder.volume.drivers.dell_emc.powermax.fc.PowerMaxFCDriver
san_ip = 10.10.10.10
san_login = my_u4v_username
san_password = my_u4v_password
powermax_srp = SRP_1
powermax_array = 000123456789
powermax_port_groups = [OS-FC-PG]
volume_backend_name = POWERMAX_FC_REPLICATION
replication_device = backend_id:id,
target_device_id:000197811111,
remote_port_group:os-failover-pg,
remote_pool:SRP_1,
rdf_group_label: 28_11_07,
mode:Metro,
metro_use_bias:False,
sync_interval:3,
sync_retries:200
Example updated backend stanza:
enabled_backends = POWERMAX_FC_REPLICATION
[POWERMAX_FC_REPLICATION]
volume_driver = cinder.volume.drivers.dell_emc.powermax.fc.PowerMaxFCDriver
san_ip = 10.10.10.10
san_login = my_u4v_username
san_password = my_u4v_password
powermax_srp = SRP_1
powermax_array = 000123456789
powermax_port_groups = [OS-FC-PG]
volume_backend_name = POWERMAX_FC_REPLICATION
replication_device = backend_id:backend_id_legacy_rep
target_device_id:000197811111,
remote_port_group:os-failover-pg,
remote_pool:SRP_1,
rdf_group_label: 28_11_07,
mode:Metro,
metro_use_bias:False,
sync_interval:3,
sync_retries:200
replication_device = backend_id:sync-rep-id
target_device_id:000197811111,
remote_port_group:os-failover-pg,
remote_pool:SRP_1,
rdf_group_label: 29_12_08,
mode:Synchronous,
sync_interval:3,
sync_retries:200
replication_device = backend_id:async-rep-id
target_device_id:000197811111,
remote_port_group:os-failover-pg,
remote_pool:SRP_1,
rdf_group_label: 30_13_09,
mode:Asynchronous,
sync_interval:3,
sync_retries:200
Note
For environments without existing replication devices. The
backend_id
values can be set to any value for all replication devices.
The backend_id_legacy_rep
value is only needed when updating a legacy
system with an existing replication device to use multiple replication
devices.
The additional replication devices defined in cinder.conf
will be detected
after restarting the cinder volume service.
To specify which replication_device
a volume type should use an additional
property named storagetype:replication_device_backend_id
must be added to
the extra specs of the volume type. The id value assigned to the
storagetype:replication_device_backend_id
key in the volume type must
match the backend_id
assigned to the replication_device
in
cinder.conf
.
# openstack volume type set \
--property storagetype:replication_device_backend_id="<id>" \
<VOLUME_TYPE>
Note
Specifying which replication device to use is done in addition to the basic replication setup for a volume type seen in Configure a single replication target
Note
In a legacy system where volume types are present that were replication
enabled before adding multiple replication devices, the
storagetype:replication_device_backend_id
should be omitted from any
volume type that does/will use the legacy replication_device
i.e.
when storagetype:replication_device_backend_id
is omitted the
replication_device with a backend_id
of backend_id_legacy_rep
will be used.
Most features are supported, except for the following:
image_volume_cache_enabled = True
), but one of two actions must be taken
when creating the cached volume:allow_extend
option in the
replication_device parameter
is set to True
. This is only
applicable to Hybrid arrays or All Flash array with HyperMax OS.Note
Failover and failback operations are not applicable in Metro configurations.
In the event of a disaster, or where there is required downtime, upgrade of the primary array for example, the administrator can issue the failover host command to failover to the configured target:
# cinder failover-host cinder_host@POWERMAX_FC_REPLICATION
Note
In cases where multiple replication devices are enabled, a backend_id must
be specified during initial failover. This can be achieved by appending
--backend_id <backend_id>
to the failover command above. The backend_id
specified must match one of the backend_ids specified in cinder.conf's
replication_device's
.
After issuing cinder failover-host
Cinder will set the R2 array as the
target array for Cinder, however, to get existing instances to use this new
array and paths to volumes it is necessary to first shelve Nova instances and
then unshelve them, this will effectively restart the Nova instance and
re-establish data paths between Nova instances and the volumes on the R2 array.
# nova shelve <server>
# nova unshelve [--availability-zone <availability_zone>] <server>
When a host is in failover mode performing normal volume or snapshot
provisioning will not be possible, failover host mode simply provides access
to replicated volumes to minimise environment down-time. The primary objective
whilst in failover mode should be to get the R1 array back online. When the
primary array becomes available again, you can initiate a fail-back using the
same failover command and specifying --backend_id default
:
# cinder failover-host cinder_host@POWERMAX_FC_REPLICATION --backend_id default
After issuing the failover command to revert to the default backend host it is necessary to re-issue the Nova shelve and unshelve commands to restore the data paths between Nova instances and their corresponding back end volumes. Once reverted to the default backend volume and snapshot provisioning operations can continue as normal.
Asynchronous and metro volumes in an RDF session, i.e. belonging to an SRDF
group, must be managed together for RDF operations (although there is a
consistency exempt
option for creating and deleting pairs in an Async
group). To facilitate this management, we create an internal RDF management
storage group on the backend. This RDF management storage group will use the
following naming convention:
OS-[rdf_group_label]-[replication_mode]-rdf-sg
It is crucial for correct management that the volumes in this storage group
directly correspond to the volumes in the RDF group. For this reason, it is
imperative that the RDF group specified in the cinder.conf
is for the
exclusive use by this Cinder backend. If there are any issues with the state
of your RDF enabled volumes prior to performing additional operations in Cinder
you will be notified in the Cinder volume logs.
SRDF/Metro is a high availability solution. It works by masking both sides of the RDF relationship to the host, and presenting all paths to the host, appearing that they all point to the one device. In order to do this, there needs to be multi-path software running to manage writing to the multiple paths.
Note
The metro issue around formatting volumes when they are added to existing metro RDF groups has been fixed in Unisphere for PowerMax 9.1, however, it has only been addressed on arrays with PowerMax OS and will not be available on arrays running a HyperMax OS.
Volume retype with storage assisted migration is supported now for PowerMax arrays. Cinder requires that for storage assisted migration, a volume cannot be retyped across backends. For using storage assisted volume retype, follow these steps:
Note
The Ussuri release of OpenStack supports retyping in-use volumes to and from replication enabled volume types with limited exception of volumes with Metro replication enabled. To retype to a volume-type that is Metro enabled the volume must first be detached then retyped. The reason for this is so the paths from the instance to the Metro R1 & R2 volumes must be initialised, this is not possible on the R2 device whilst a volume is attached.
Note
When multiple replication devices are configured. If retyping from one replication mode to another the R1 device ID is preserved and a new R2 side device is created. As a result, the device ID on the R2 array may be different after the retype operation has completed.
Note
Retyping an in-use volume to a metro enabled volume type is not currently
supported via storage-assisted migration. This retype can still be
performed using host-assisted migration by setting the migration-policy
to on-demand
.
cinder retype --migration-policy on-demand <volume> <volume-type>
For migrating a volume from one Service Level or Workload combination to another, use volume retype with the migration-policy to on-demand. The target volume type should have the same volume_backend_name configured and should have the desired pool_name to which you are trying to retype to (please refer to 6. Create volume types for details).
$ cinder retype --migration-policy on-demand <volume> <volume-type>
Generic volume group operations are performed through the CLI using API version 3.1x of the Cinder API. Generic volume groups are multi-purpose groups which can be used for various features. The PowerMax driver supports consistent group snapshots and replication groups. Consistent group snapshots allows the user to take group snapshots which are consistent based on the group specs. Replication groups allow for tenant facing APIs to enable and disable replication, and to failover and failback, a group of volumes. Generic volume groups have replaced the deprecated consistency groups.
To create a consistent group snapshot, set a group-spec, having the key
consistent_group_snapshot_enabled
set to <is> True
on the group.
cinder --os-volume-api-version 3.11 group-type-key GROUP_TYPE set consistent_group_snapshot_enabled="<is> True"
Similarly the same key should be set on any volume type which is specified while creating the group.
# openstack volume type set --property replication_enabled="<is> True" /
POWERMAX_REPLICATION
If this key is not set on the group-spec or volume type, then the generic volume group will be created/managed by Cinder (not the PowerMax driver).
Note
The consistent group snapshot should not be confused with the PowerMax consistency group which is an SRDF construct.
As with Consistent group snapshot consistent_group_snapshot_enabled
should
be set to true on the group and the volume type for replication groups.
Only Synchronous replication is supported for use with Replication Groups.
When a volume is created into a replication group, replication is on by
default. The disable_replication
api suspends I/O traffic on the devices,
but does NOT remove replication for the group. The enable_replication
api
resumes I/O traffic on the RDF links. The failover_group
api allows a group
to be failed over and back without failing over the entire host. See below for
usage.
Note
A generic volume group can be both consistent group snapshot enabled and consistent group replication enabled.
Storage groups are created on the PowerMax as a result of creation of generic volume groups. These storage groups follow a different naming convention and are of the following format depending upon whether the groups have a name.
TruncatedGroupName_GroupUUID or GroupUUID
Please refer to the official OpenStack block-storage groups documentation for the most up to date group operations
Generic volume group operations no longer require the user to specify the
Cinder CLI version, however, performing generic volume group replication
operations still require this setting. When running generic volume group
commands set the value --os-volume-api-version
to 3.38
. These
commands are not listed in the latest Cinder CLI documentation so will
remain here until added to the latest Cinder CLI version or deprecated
from Cinder.
cinder --os-volume-api-version 3.38 group-enable-replication GROUP
cinder --os-volume-api-version 3.38 group-disable-replication GROUP
cinder --os-volume-api-version 3.38 group-failover-replication GROUP
cinder --os-volume-api-version 3.38 group-failover-replication GROUP /
--secondary-backend-id default
Managing volumes in OpenStack is the process whereby a volume which exists on the storage device is imported into OpenStack to be made available for use in the OpenStack environment. For a volume to be valid for managing into OpenStack, the following prerequisites must be met:
For a volume to exist in a Cinder managed pool, it must reside in the same Storage Resource Pool (SRP) as the backend which is configured for use in OpenStack. Specifying the pool correctly can be entered manually as it follows the same format:
Pool format: <service_level>+<srp>+<array_id>
Pool example: Diamond+SRP_1+111111111111
Key | Value |
---|---|
service_level |
The service level of the volume to be managed |
srp |
The Storage Resource Pool configured for use by the backend |
array_id |
The PowerMax serial number (12 digit numerical) |
With your pool name defined you can now manage the volume into OpenStack, this
is possible with the CLI command cinder manage
. The bootable
parameter
is optional in the command, if the volume to be managed into OpenStack is not
bootable leave this parameter out. OpenStack will also determine the size of
the value when it is managed so there is no need to specify the volume size.
Command format:
$ cinder manage --name <new_volume_name> --volume-type <powermax_vol_type> \
--availability-zone <av_zone> <--bootable> <host> <identifier>
Command Example:
$ cinder manage --name powermax_managed_volume --volume-type POWERMAX_ISCSI_DIAMOND \
--availability-zone nova demo@POWERMAX_ISCSI_DIAMOND#Diamond+SRP_1+111111111111 031D8
After the above command has been run, the volume will be available for use in the same way as any other OpenStack PowerMax volume.
Note
An unmanaged volume with a prefix of OS-
in its identifier name cannot
be managed into OpenStack, as this is a reserved keyword for managed
volumes. If the identifier name has this prefix, an exception will be thrown
by the PowerMax driver on a manage operation.
Whilst it is not possible to manage volumes into OpenStack that are part of a SRDF relationship, it is possible to manage a volume into OpenStack and enable replication at the same time. This is done by having a replication enabled PowerMax volume type (for more information see section Volume Replication) during the manage volume process you specify the replication volume type as the chosen volume type. Once managed, replication will be enabled for that volume.
Note
It is not possible to manage into OpenStack SnapVX linked target volumes, only volumes which are a SnapVX source are permitted. We do not want a scenario where a snapshot source can exist outside of OpenStack management.
Unmanaging a volume is not the same as deleting a volume. When a volume is deleted from OpenStack, it is also deleted from the PowerMax at the same time. Unmanaging a volume is the process whereby a volume is removed from OpenStack but it remains for further use on the PowerMax. The volume can also be managed back into OpenStack at a later date using the process discussed in the previous section. Unmanaging volume is carried out using the Cinder unmanage CLI command:
Command format:
$ cinder unmanage <volume_name/volume_id>
Command example:
$ cinder unmanage powermax_test_vol
Once unmanaged from OpenStack, the volume can still be retrieved using its
device ID or OpenStack volume ID. Within Unisphere you will also notice that
the OS-
prefix has been removed, this is another visual indication that
the volume is no longer managed by OpenStack.
Users can manage PowerMax SnapVX snapshots into OpenStack if the source volume already exists in Cinder. Similarly, users will be able to unmanage OpenStack snapshots to remove them from Cinder but keep them on the storage backend.
Set-up, restrictions and requirements:
It is possible to manage PowerMax SnapVX snapshots into OpenStack, where the source volume from which the snapshot is taken already exists in, and is managed by OpenStack Cinder. The source volume may have been created in OpenStack Cinder, or it may have been managed in to OpenStack Cinder also. With the support of managing SnapVX snapshots included in OpenStack Queens, the restriction around managing SnapVX source volumes has been removed.
Note
It is not possible to manage into OpenStack SnapVX linked target volumes, only volumes which are a SnapVX source are permitted. We do not want a scenario where a snapshot source can exist outside of OpenStack management.
Requirements/restrictions:
OS-
.Command structure:
$ cinder snapshot-manage --id-type source-name
[--name <name>]
[--description <description>]
[--metadata [<key=value> [<key=value> ...]]]
<volume name/id> <identifier>
Positional arguments:
<volume name/id>
Source OpenStack volume name<identifier>
Name of existing snapshot on PowerMax backendOptional arguments:
--name <name>
Snapshot name (Default=``None``)--description <description>
Snapshot description (Default=``None``)--metadata [<key=value> [<key=value> ...]]
Metadata key=value
pairs
(Default=``None``)Example:
$ cinder snapshot-manage --name SnapshotManaged \
--description "Managed Queens Feb18" \
powermax-vol-1 PowerMaxSnapshot
Where:
SnapshotManaged
.Managed Queens Feb18
.powermax-vol-1
.PowerMaxSnapshot
.Outcome:
After the process of managing the Snapshot has completed, the SnapVX snapshot
on the PowerMax backend will be prefixed by the letters OS-
, leaving the
snapshot in this example named OS-PowerMaxSnapshot
. The associated snapshot
managed by Cinder will be present for use under the name SnapshotManaged
.
Unmanaging a snapshot in Cinder is the process whereby the snapshot is removed from and no longer managed by Cinder, but it still exists on the storage backend. Unmanaging a SnapVX snapshot in OpenStack Cinder follows this behaviour, whereby after unmanaging a PowerMax SnapVX snapshot from Cinder, the snapshot is removed from OpenStack but is still present for use on the PowerMax backend.
Requirements/Restrictions:
Command Structure:
Identify the SnapVX snapshot you want to unmanage from OpenStack Cinder, note the snapshot name or ID as specified by Cinder. Using the Cinder CLI use the following command structure to unmanage the SnapVX snapshot from Cinder:
$ cinder snapshot-unmanage <snapshot>
Positional arguments:
<snapshot>
Cinder snapshot name or ID.Example:
$ cinder snapshot-unmanage SnapshotManaged
Where:
After the process of unmanaging the SnapVX snapshot in Cinder, the snapshot on
the PowerMax backend will have the OS-
prefix removed to indicate it is no
longer OpenStack managed. In the example above, the snapshot after unmanaging
from OpenStack will be named PowerMaxSnapshot
on the storage backend.
Volumes that can be managed by and imported into Openstack.
List manageable volume is filtered by:
Ready
Normal
FBA
TDEV
private
encapsulated
reserved
OS-
.Snapshots that can be managed by and imported into Openstack
List manageable snapshots is filtered by:
FBA
.TDEV
.private
.OS-
or temp-
.Note
There is some delay in the syncing of the Unisphere for PowerMax database
when the state/properties of a volume is modified using symcli
. To
prevent this it is preferable to modify state/properties of volumes within
Unisphere.
PowerMax Cinder driver support Cinder backup functionality. For further information on setup, configuration and usage please see the official OpenStack volume backup documentation and related volume backup CLI guide.
Seamless upgrades from an SMI-S based driver to REST API based driver, following the setup instructions above, are supported with a few exceptions:
live migration
functionality will not work on already
attached/in-use legacy volumes without first migrating the volumes to
the new REST masking view structure. If you are upgrading from Newton
or Ocata to Pike or greater please contact Dell EMC Support and we
will guide you through the process.Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.