Skip to content

DL380 Raid Notes

Problem: Where are my disks???

When we installed the os on our new (to us) prolient DL380, Only a single disk was visible in spite of there having been 6 disks installed. This is because the DL380s disk controller was set up in raid mode an did not expose disks until they were configured as "logical" disks.

This is unlike the Dell PowerEdge we have which detects and presents the drives in a hot swappable fashion while still allowing some disks to participate in raid arrays.

Since we use hardware raid mirroring on the boot disks, Adding, removing or replacing disks requires configuring the raid controller.

Using HP utilities to configure the controller without downing the server

HP provides utilities and officially supports bionic and hosts a repo for it. It includes a server that can be accessed graphically as well as a command line interface. [#fn1 (1)] For the purpose of maintaining disks we only need ssacli and perhaps ssaducli.

Install the hp supported utilities.

root@kb2018:~# echo "deb http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current non-free" >> /etc/apt/sources.list.d/hp.list
root@kb2018:~# root@kb2018:/etc/apt# wget http://downloads.linux.hpe.com/SDR/repo/mcp/GPG-KEY-mcp
--2018-11-12 09:00:29--  http://downloads.linux.hpe.com/SDR/repo/mcp/GPG-KEY-mcp
Resolving downloads.linux.hpe.com (downloads.linux.hpe.com)... 15.249.152.85
Connecting to downloads.linux.hpe.com (downloads.linux.hpe.com)|15.249.152.85|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 994
Saving to: ‘GPG-KEY-mcp’

GPG-KEY-mcp                                  100%[=============================================================================================>]     994  --.-KB/s    in 0s

2018-11-12 09:00:30 (90.5 MB/s) - ‘GPG-KEY-mcp’ saved [994/994]

root@kb2018:/etc/apt# apt-key add GPG-KEY-mcp
OK
root@kb2018:/etc/apt# apt-get update
Ign:1 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]                      
Hit:3 http://us.archive.ubuntu.com/ubuntu bionic InRelease                                           
Get:4 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release [6,051 B]        
Get:5 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release.gpg [490 B]                                
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                                                               
Hit:7 http://archive.ubuntu.com/ubuntu bionic InRelease                                                              
Ign:5 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release.gpg                                 
Get:8 http://archive.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]             
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                      
Reading package lists... Done      
W: GPG error: http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY C208ADDE26C2B797
E: The repository 'http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@kb2018:/etc/apt# key=C208ADDE26C2B797
root@kb2018:/etc/apt# gpg --keyserver keyserver.ubuntu.com --recv-keys $key
gpg: key C208ADDE26C2B797: public key "Hewlett Packard Enterprise Company RSA-2048-25 <signhp@hpe.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@kb2018:/etc/apt# gpg --armor --export $key |apt-key add -
OK
root@kb2018:/etc/apt# apt-get update
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                                                             
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                                                                      
Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease                                                                                                      
Ign:5 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current InRelease                                                                                        
Get:6 http://archive.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]                                
Get:7 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release [6,051 B]                       
Get:8 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release.gpg [490 B]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                       
Get:10 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current/non-free amd64 Packages [1,971 B]
Fetched 352 kB in 1s (288 kB/s)                                        
Reading package lists... Done
root@kb2018:/etc/apt# apt-get install ssacli ssaducli
...

Once the issues with his signature were resolved (above) I was able to instal the ssacli. [#fn2 (2)]

Seeing the drives

Use the ssacli to show the unassigned drives after inserting fresh disks.

root@kb2018:/etc/apt# ssacli 
Smart Storage Administrator CLI 3.30.13.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.

=> set target controller slot=0

   "controller slot=0"

=> pd all show

Smart Array P410i in Slot 0 (Embedded)

   Array A

      physicaldrive 2C:1:1 (port 2C:box 1:bay 1, SAS HDD, 146 GB, OK)
      physicaldrive 2C:1:2 (port 2C:box 1:bay 2, SAS HDD, 146 GB, OK)

   Array B

      physicaldrive 2C:1:3 (port 2C:box 1:bay 3, SATA SSD, 240 GB, OK)

   Array C

      physicaldrive 2C:1:4 (port 2C:box 1:bay 4, SATA SSD, 240 GB, OK)

   Array D

      physicaldrive 3C:1:5 (port 3C:box 1:bay 5, SAS HDD, 300 GB, OK)

   Array E

      physicaldrive 3C:1:6 (port 3C:box 1:bay 6, SAS HDD, 300 GB, OK)

   Unassigned

      physicaldrive 3C:1:7 (port 3C:box 1:bay 7, SAS HDD, 146 GB, OK)
      physicaldrive 3C:1:8 (port 3C:box 1:bay 8, SAS HDD, 146 GB, OK)

letting the OS see the drives

Once we know what drives are available we can create logical drives which will be presented to the os (assuming the same set target command above)

=> set target controller slot=0
...
=> create type=ld drives=3C:1:7 size=max raid=0  
=> create type=ld drives=3C:1:8 size=max raid=0
quit

Removing drive

Before removing drives you should make sure that they are unmounted or detached (zfs). After removing a drive you should delete the logical disk that it is associated with.

=> set target controller slot=0
...
=> Array G delete

Increasing write performance

once we get a ups we should be able to use the controllers write cache safely.

=> controller slot=0 modify drivewritecache=enable

Warning: Without the proper safety precautions, use of write cache on physical 
         drives could cause data loss in the event of power failure.  To ensure
         data is properly protected, use redundant power supplies and
         Uninterruptible Power Supplies. Also, if you have multiple storage
         enclosures, all data should be mirrored across them. Use of this
         feature is not recommended unless these precautions are followed.
         Continue? (y/n) n

=>

See also: [wiki:DL380RaidBios my notes on configuring the disks the hard way]

footnotes

[=#fn1 1]) This was discovered after digging around for the perccli raid utilities provided by dell (officially supported only on commercial RPM based systems but installable using alien)

[=#fn2 2]) The biggest pain in the ass other than the weirdness with the public signature was that HP fucking rebranded the hpssacli to ssacli. Most of the good web info and hp docs still reference the old utility name (nothing else changed).

references

  • http://h10032.www1.hp.com/ctg/Manual/c02289065.pdf (2010)
  • https://amk1.wordpress.com/2013/11/22/zfs-with-hp-smart-array-p410i/
  • https://content.etilize.com/User-Manual/1033728289.pdf
  • http://www.sysadminshare.com/2012/05/hpacucli-commands-referrence.html
  • https://wiki.debian.org/LinuxRaidForAdmins
  • https://www.golinuxhub.com/2017/05/hot-swapping-broken-hdd-with-software.html
  • https://kallesplayground.wordpress.com/useful-stuff/hp-smart-array-cli-commands-under-esxi/
  • http://downloads.linux.hpe.com/SDR/project/mcp/
  • https://wiki.debian.org/HP/ProLiant#HP_Repository
  • https://binaryimpulse.com/2013/09/hp-array-configuration-utility-command-cheat-sheet/
  • https://bibszone.wordpress.com/2016/02/11/hp-smart-array-cli-commands/
  • https://h50146.www5.hpe.com/products/software/oe/linux/mainstream/support/doc/general/mgmt/ssa_cli/files/v240_130/hpssacli-2.40-13.0_help.txt
  • https://unixlab.weebly.com/raid-array.html
  • https://hardforum.com/threads/hp-dl380p-gen8-p420i-controller-hbamode.1852528/

addendum (output from ssacli show detailed config)

=>ctrl all show config detail

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: 5001438013631A40
   Cache Serial Number: PBCDH0CRH0V0L0
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 6.64-0
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Parallel Surface Scan Supported: No
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 25% Read / 75% Write
   Drive Write Cache: Disabled
   Total Cache Size: 0.5
   Total Cache Memory Available: 0.4
   No-Battery Write Cache: Disabled
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True
   Number of Ports: 2 Internal only
   Encryption: Not Set
   Driver Name: hpsa
   Driver Version: 3.4.20
   Driver Supports SSD Smart Path: True
   PCI Address (Domain:Bus:Device.Function): 0000:05:00.0
   Port Max Phy Rate Limiting Supported: False
   Host Serial Number: USE135N52V
   Sanitize Erase Supported: False
   Primary Boot Volume: None
   Secondary Boot Volume: None



   HP SAS Expander Card at Port 2C, Box 1, OK

      Power Supply Status: Not Redundant
      Vendor ID: HP
      Serial Number: RF15BP2689
      Firmware Version: 2.10
      Drive Bays: 24
      Port: 2C
      Box: 1
      Location: Internal

   Expander 250 
      Device Number: 250
      Firmware Version: 2.10
      WWID: 5001438014526C66
      Box: 1
      Vendor ID: HP

   HP SAS Expander Card SEP 248 
      Device Number: 248
      Firmware Version: 2.10
      Hardware Revision: Rev C
      WWID: 5001438014526C65
      Box: 2
      Vendor ID: HP
      Model: HP SAS EXP Card

   Physical Drives
      physicaldrive 2C:1:4 (port 2C:box 1:bay 4, SATA SSD, 240 GB, OK)
      physicaldrive 2C:1:3 (port 2C:box 1:bay 3, SATA SSD, 240 GB, OK)
      physicaldrive 2C:1:2 (port 2C:box 1:bay 2, SAS HDD, 146 GB, OK)
      physicaldrive 2C:1:1 (port 2C:box 1:bay 1, SAS HDD, 146 GB, OK)
      physicaldrive 3C:1:6 (port 3C:box 1:bay 6, SAS HDD, 300 GB, OK)
      physicaldrive 3C:1:5 (port 3C:box 1:bay 5, SAS HDD, 300 GB, OK)



   HP SAS Expander Card at Port 4C, Box 2, OK

      Power Supply Status: Not Redundant
      Vendor ID: HP
      Serial Number: RF15BP2689
      Firmware Version: 2.10
      Drive Bays: 24
      Port: 4C
      Box: 2
      Location: Internal

   Expander 250 
      Device Number: 250
      Firmware Version: 2.10
      WWID: 5001438014526C66
      Box: 1
      Vendor ID: HP

   HP SAS Expander Card SEP 248 
      Device Number: 248
      Firmware Version: 2.10
      Hardware Revision: Rev C
      WWID: 5001438014526C65
      Box: 2
      Vendor ID: HP
      Model: HP SAS EXP Card

   Physical Drives
      None attached


   Port Name: 1I
         Port ID: 0
         Port Connection Number: 0
         SAS Address: 5001438013631A40
         Port Location: Internal

   Port Name: 2I
         Port ID: 1
         Port Connection Number: 1
         SAS Address: 5001438013631A44
         Port Location: Internal

   Array: A
      Interface Type: SAS
      Unused Space: 6 MB (0.00%)
      Used Space: 273.40 GB (100.00%)
      Status: OK
      Array Type: Data 
      Smart Path: disable


      Logical Drive: 1
         Size: 136.70 GB
         Fault Tolerance: 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 35132
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Unrecoverable Media Errors: None
         Caching:  Enabled
         Unique Identifier: 600508B1001CAA24339C082CBF1B0912
         Disk Name: /dev/sda 
         Mount Points: / 80.0 GB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: A0E0B9A75001438013631A40256F
         Mirror Group 1:
            physicaldrive 2C:1:2 (port 2C:box 1:bay 2, SAS HDD, 146 GB, OK)
         Mirror Group 2:
            physicaldrive 2C:1:1 (port 2C:box 1:bay 1, SAS HDD, 146 GB, OK)
         Drive Type: Data
         LD Acceleration Method: Controller Cache


      physicaldrive 2C:1:1
         Port: 2C
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Rotational Speed: 15000
         Firmware Revision: HPDD
         Serial Number: PLWGTWSE
         WWID: 5000CCA00B53489D
         Model: HP      EH0146FARWD
         Current Temperature (C): 35
         Maximum Temperature (C): 42
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False
         Shingled Magnetic Recording Support: None

      physicaldrive 2C:1:2
         Port: 2C
         Box: 1
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Rotational Speed: 15000
         Firmware Revision: HPDD
         Serial Number: PLWP0XNE
         WWID: 5000CCA00B5E9B11
         Model: HP      EH0146FARWD
         Current Temperature (C): 34
         Maximum Temperature (C): 47
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False
         Shingled Magnetic Recording Support: None



   Array: B
      Interface Type: Solid State SATA
      Unused Space: 2 MB (0.00%)
      Used Space: 223.54 GB (100.00%)
      Status: OK
      Array Type: Data 
      Smart Path: disable


      Logical Drive: 2
         Size: 223.54 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 57450
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CC841DD71B0E330404FF4
         Disk Name: /dev/sdb 
         Mount Points: None
         Logical Drive Label: ABABB8965001438013631A40D1E0
         Drive Type: Data
         LD Acceleration Method: Controller Cache


      physicaldrive 2C:1:3
         Port: 2C
         Box: 1
         Bay: 3
         Status: OK
         Drive Type: Data Drive
         Interface Type: Solid State SATA
         Size: 240 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Firmware Revision: Q0410A
         Serial Number: AB20180827A0101371
         WWID: 5001438014526C41
         Model: ATA     TEAML5Lite3D240G
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         SSD Smart Trip Wearout: Not Supported
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps
         Sanitize Erase Supported: False
         Shingled Magnetic Recording Support: None



   Array: C
      Interface Type: Solid State SATA
      Unused Space: 2 MB (0.00%)
      Used Space: 223.54 GB (100.00%)
      Status: OK
      Array Type: Data 
      Smart Path: disable


      Logical Drive: 3
         Size: 223.54 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 57450
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CD1056D9358D036DE54EB
         Disk Name: /dev/sdc 
         Mount Points: None
         Logical Drive Label: ABAB89005001438013631A4045F6
         Drive Type: Data
         LD Acceleration Method: Controller Cache


      physicaldrive 2C:1:4
         Port: 2C
         Box: 1
         Bay: 4
         Status: OK
         Drive Type: Data Drive
         Interface Type: Solid State SATA
         Size: 240 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Firmware Revision: Q0410A
         Serial Number: AB20180827A0100293
         WWID: 5001438014526C40
         Model: ATA     TEAML5Lite3D240G
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         SSD Smart Trip Wearout: Not Supported
         PHY Count: 1
         PHY Transfer Rate: 3.0Gbps
         Sanitize Erase Supported: False
         Shingled Magnetic Recording Support: None



   Array: D
      Interface Type: SAS
      Unused Space: 0 MB (0.00%)
      Used Space: 279.37 GB (100.00%)
      Status: OK
      Array Type: Data 
      Smart Path: disable


      Logical Drive: 4
         Size: 279.37 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C868C26439B55D426224F
         Disk Name: /dev/sdd 
         Mount Points: None
         Logical Drive Label: ABAB99875001438013631A40A72E
         Drive Type: Data
         LD Acceleration Method: Controller Cache


      physicaldrive 3C:1:5
         Port: 3C
         Box: 1
         Bay: 5
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Rotational Speed: 10000
         Firmware Revision: HPD6 (FW update is recommended to minimum version: HPD7)
         Serial Number: PQJ0EM4B
         WWID: 5000CCA025718881
         Model: HP      EG0300FBDBR
         Current Temperature (C): 31
         Maximum Temperature (C): 44
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False
         Shingled Magnetic Recording Support: None



   Array: E
      Interface Type: SAS
      Unused Space: 0 MB (0.00%)
      Used Space: 279.37 GB (100.00%)
      Status: OK
      Array Type: Data 
      Smart Path: disable


      Logical Drive: 5
         Size: 279.37 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C380646CF15536E61E692
         Disk Name: /dev/sde 
         Mount Points: None
         Logical Drive Label: ABABE9D05001438013631A4088C8
         Drive Type: Data
         LD Acceleration Method: Controller Cache


      physicaldrive 3C:1:6
         Port: 3C
         Box: 1
         Bay: 6
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Rotational Speed: 10000
         Firmware Revision: HPD6 (FW update is recommended to minimum version: HPD7)
         Serial Number: PMVJ07DB
         WWID: 5000CCA0211D1B55
         Model: HP      EG0300FBDBR
         Current Temperature (C): 31
         Maximum Temperature (C): 57
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False
         Shingled Magnetic Recording Support: None


   Expander 250 
      Device Number: 250
      Firmware Version: 2.10
      WWID: 5001438014526C66
      Box: 1
      Vendor ID: HP

   Expander 250 
      Device Number: 250
      Firmware Version: 2.10
      WWID: 5001438014526C66
      Box: 1
      Vendor ID: HP

   HP SAS Expander Card SEP 248 
      Device Number: 248
      Firmware Version: 2.10
      Hardware Revision: Rev C
      WWID: 5001438014526C65
      Box: 2
      Vendor ID: HP
      Model: HP SAS EXP Card

   HP SAS Expander Card SEP 248 
      Device Number: 248
      Firmware Version: 2.10
      Hardware Revision: Rev C
      WWID: 5001438014526C65
      Box: 2
      Vendor ID: HP
      Model: HP SAS EXP Card

   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 249 
      Device Number: 249
      Firmware Version: RevC
      WWID: 5001438013631A4F
      Vendor ID: PMCSIERA
      Model: SRC 8x6G