Storage on smcleod.net

Flash Storage and SSD Failure Rate Update (March 2018)

Tue, 20 Mar 2018 00:00:00 +0000

It was almost 3 years ago that my open source storage project went into production. In that time it’s been running 24/7 serving as highly available solid state storage for hundreds of VMs and several virtualisation clusters across our two main sites.

I’m happy to report that the clusters have been operating very successfully since their conception.

Since moving away from proprietary ‘black box’ vendor SANs, we haven’t had a single SAN issue, storage outage.

Talk - Clustered, Distributed File and Volume Storage with GlusterFS

Tue, 14 Nov 2017 00:00:00 +0000

Using GlusterFS to provide volume storage to Kubernetes as a replacement for our existing file and static content hosting.

This talk was given at Infracoders on Tuesday 14th November 2017.

Click below to view slides (PDF version):

Direct download link

GlusterFS

Mon, 25 Sep 2017 00:00:00 +0000

We’re in the process of shifting from using our custom ‘glue’ for orchestrating Docker deployments to Kubernetes, When we first deployed Docker to replace LXC and our legacy Puppet-heavy application configuration and deployment systems there really wasn’t any existing tool to manage this, thus we rolled our own, mainly a few Ruby scripts combined with a Puppet / Hiera / Mcollective driven workflow.

The main objective is to replace our legacy NFS file servers used to host uploads / attachments and static files for our web applications, while NFS(v4) performance is adequate, it is a clear single point of failure and of course, there are the age old stale mount problems should network interruptions occur.

Update Delayed Serial STONITH Design

Mon, 04 Jul 2016 00:00:00 +0000

note: This is a follow up post from 2015-07-21-rcd-stonith

A Linux Cluster Base STONITH provider for use with modern Pacemaker clusters

This has since been accepted and merged into Fedora’s code base and as such will make it’s way to RHEL.

Source Code: Github
Diptrace CAD Design: Github
I have open sourced the CAD circuit design and made this available within this repo under CAD Design and Schematics
Related RedHat Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1240868

`v1` vs `v2/v3` versions of the `rcd_serial` STONITH system

The v2/v3 cables include the following improvements:

Benchmarking IO with FIO

Fri, 29 Apr 2016 00:00:00 +0000

This is a quick tldr there are many other situations and options you could consider

FIO man page
IOP/s = Input or Output operations per second
Throughput = How many MB/s can you read/write continuously

Variables worth tuning based on your situation

--iodepth

The iodepth is very dependant on your hardware.

Rotational drives without much cache and high latency (i.e. desktop SATA drives) will not benefit from a large iodepth, Values between 16 to 64 could be sensible.

Fix XenServer SR with corrupt or invalid metadata

Mon, 18 Jan 2016 00:00:00 +0000

If a disk / VDI is orphaned or only partially deleted you’ll notice that under the SR it’s not assigned to any VM.

This can cause issues that look like metadata corruption resulting in the inability to migrate VMs or edit storage.

For example:

[root@xenserver-host ~]# xe vdi-destroy uuid=6c2cd848-ac0e-441c-9cd6-9865fca7fe8b
Error code: SR_BACKEND_FAILURE_181
Error parameters: , Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters:
  /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/MGT, 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b.
  Error: Failed to write file with params [3, 0, 512, 512]. Error: 5],

Removing stale VDIs

To fix this, you need to remove those VDIs from the SR after first deleting the logical volume:

iSCSI SCSI-ID / Serial Persistence

Mon, 14 Dec 2015 00:00:00 +0000

“Having a SCSI ID is a f*cking idiotic thing to do.”

- Linus Torvalds

…and after the amount of time I’ve wasted getting XenServer to play nicely with LIO iSCSI failover I tend to agree.

The Problem

One oddity of Xen / XenServer’s storage subsystem is that it identifies iSCSI storage repositories via a calculated SCSI ID rather than the iSCSI Serial - which would be the sane thing to do.

How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync

Mon, 09 Nov 2015 00:00:00 +0000

Slides

Failover Demo

SAN Intro

Wed, 07 Oct 2015 00:00:00 +0000

SSD Storage - Two Months In Production

Sun, 13 Sep 2015 00:00:00 +0000

Over the last two months I’ve been running selected IO intensive servers off the the SSD storage cluster, these hosts include (among others) our:

Primary Puppetmaster
Gitlab server
Redmine app and database servers
Nagios servers
Several Docker database host servers

Reliability

We haven’t had any software or hardware failures since commissioning the storage units.

During this time we have had 3 disk failures on our HP StoreVirtual SANs that have required us to call the supporting vendor and replace failed disks.

iSCSI Benchmarking

Fri, 24 Jul 2015 00:00:00 +0000

The following are benchmarks from our testings of our iSCSI SSD storage.

67,300 read IOP/s on a VM on iSCSI

(Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)
Per VM and scales to 1,000,000 IOP/s total

root@dev-samm:/mnt/pmt1 128 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=read
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
bs: 1 (f=1): [R] [55.6% done] [262.1M/0K /s] [67.3K/0  iops] [eta 00m:04s]

38,500 random 4k write IOP/s on a VM on iSCSI

(Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)
Per VM and scales to 700,000 IOP/s total

root@dev-samm:/mnt/pmt1 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
bs: 1 (f=1): [w] [26.3% done] [0K/150.2M /s] [0 /38.5K iops] [eta 00m:14s]

Raw device latency on storage units

Intel DC3600 1.2T PCIe NVMe

root@s1-san6:/proc  # ioping /dev/nvme0n1p1
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=1 time=104 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=2 time=83 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=3 time=51 us
4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=4 time=71 us

SanDisk SDSSDXPS960G SATA

root@pm-san5:/proc  # ioping /dev/sdc
4.0 KiB from /dev/sdc (device 894.3 GiB): request=1 time=4.2 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=2 time=4.1 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=3 time=4.1 ms
4.0 KiB from /dev/sdc (device 894.3 GiB): request=4 time=4.1 ms

Micron_M600_MTFDDAK1T0MBF SATA

root@pm-san5:/proc  # ioping /dev/sdf
4.0 KiB from /dev/sdf (device 953.9 GiB): request=1 time=157 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=2 time=190 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=3 time=65 us
4.0 KiB from /dev/sdf (device 953.9 GiB): request=4 time=181 us
```shell
## Latency on the a VM

- (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM)

```shell
root@dev-samm:/mnt 127 # ioping pmt1/
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=1 time=0.6 ms
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=2 time=0.7 ms
4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=3 time=0.7 ms

--- pmt1/ (ext4 /dev/xvdb1) ioping statistics ---
3 requests completed in 2159.1 ms, 1508 iops, 5.9 mb/s
min/avg/max/mdev = 0.6/0.7/0.7/0.1 ms
root@dev-samm:/mnt  # ioping pmt2/
4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=1 time=0.6 ms
4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=2 time=0.8 ms

--- pmt2/ (ext4 /dev/xvdc1) ioping statistics ---
2 requests completed in 1658.4 ms, 1470 iops, 5.7 mb/s
min/avg/max/mdev = 0.6/0.7/0.8/0.1 ms
root@dev-samm:/mnt  # ioping pmt3/
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=1 time=0.6 ms
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=2 time=0.9 ms
4096 bytes from pmt3/ (ext4 /dev/xvde1): request=3 time=0.9 ms

Delayed Serial STONITH

Tue, 21 Jul 2015 00:00:00 +0000

A modified version of John Sutton’s rcd_serial cable coupled with our Supermicro reset switch hijacker:

This works with the rcd_serial fence agent plugin.

Reasons rcd_serial makes for a very good STONITH mechanism:

It has no dependency on power state.
It has no dependency on network state.
It has no dependency on node operational state.
It has no dependency on external hardware.
It costs less that $5 + time to build.
It is incredibly simple and reliable.

Essentially the most common STONITH agent type in use is probably those that control UPS / PDUs, while this sounds like a good idea in theory there are a number of issues with relying on a UPS / PDU:

Video - Cluster Failover Performance Demo

Sun, 12 Jul 2015 00:00:00 +0000

CentOS 7 and HA

Tue, 07 Jul 2015 00:00:00 +0000

First some background…

One of the many lessons I’ve learnt from my Linux HA / Storage clustering project is that the Debian HA ecosystem is essentially broken, We reached the point where packages were too old, too buggy or in Debian 8’s case - outright missing.

In the past I was very disappointed with RHEL/CentOS 5 / 6 and (until now) have been quite satisfied with Debian as a stable server distribution with historicity more modern packages and kernels.

SSD Storage Cluster - Update and Diagram

Wed, 17 Jun 2015 00:00:00 +0000

Due to several recent events beyond my control I’m a bit behind on the project - hence the lack of updates which I apologise for.

The goods news is that I’m back working to finish off the clusters and I’m happy to report that all is going to plan.

Here is the final digram of the two-node cluster design:

Plain text version available here

This was generated from the LCMC tool (beware - it’s java!).

Video - Storage Cluster Failover Demo

Thu, 14 May 2015 00:00:00 +0000

A brief demonstration of the failover and recovery process on the storage clusters I’ve been building.

Talk - High Performance Software Defined Storage

Wed, 15 Apr 2015 00:00:00 +0000

A high level talk from Infracoders Melbourne on 12/04/2015.

There’s also a low quality recording available here:

Building a high performance SSD SAN - Part 1

Mon, 16 Feb 2015 00:00:00 +0000

Over the coming month I will be architecting, building and testing a modular, high performance SSD-only storage solution.

I’ll be documenting my progress / findings along the way and open sourcing all the information as a public guide.

With recent price drops and durability improvements in solid state storage now is better time than any to ditch those old magnets.

Modular server manufacturers such as SuperMicro have spent large on R&D thanks to the ever growing requirements from cloud vendors that utilise their hardware.

Direct-Attach SSD Storage - Performance & Comparisons

Sun, 15 Feb 2015 00:00:00 +0000

Further to my earlier post on XenServer storage performance with regards to directly attaching storage from the host, I have been analysing the performance of various SSD storage options.

I have attached a HP DS2220sb storage blade to an existing server blade and compared performance with 4 and 6 SSD RAID-10 to our existing iSCSI SANs.

While the P420i RAID controller in the DS2220sb is clearly saturated and unable to provide throughput much over 1,100MB/s - the IOP/s available to PostgreSQL are still a very considerably performance improvement over our P4530 SAN - in fact, 6 SSD’s result in a 39.9x performance increase!

XenServer, SSDs & VM Storage Performance