Benchmarking IO with FIO

This is a quick tldr there are many other situations and options you could consider FIO man page IOP/s = Input or Output operations per second Throughput = How many MB/s can you read/write continuously Variables worth tuning based on your situation --iodepth The iodepth is very dependant on your hardware. Rotational drives without much cache and high latency (i.e. desktop SATA drives) will not benefit from a large iodepth, Values between 16 to 64 could be sensible. ...

April 29, 2016 · 2 min · 393 words · Sam McLeod

Mirroring a Gitlab project to Github

Let’s pretend you have a project on Gitlab called ask-izzy and you want to mirror it up to Gitlab which is located at https://github.com/ask-izzy/ask-izzy Assuming you’re running Gitlab as the default user of git and that your repositories are stored in /mnt/repositories you can following something similar to the following instructions: Grant write access to Github Get your Gitlab install’s pubkey from the git user cat /home/git/.ssh/id_rsa.pub On Github add this pubkey as deploy key on the repo, make sure you tick the option to allow write access. ...

February 4, 2016 · 1 min · 145 words · Sam McLeod

AskIzzy

Today we launched a mobile website for homeless people … and it was launched by one of Australia’s many recent Prime Ministers Today alone we served up over 87,000 requests As many of you know, I work with Infoxchange as the operations lead. When I first heard the idea of a website or app for people that have found or are worried about finding themselves homeless in Australia I really didn’t think it made sense - until I saw the stats showing how many homeless people in Australia have regular access to a smart phone and data either via a cellular provider or free WiFi. ...

January 29, 2016 · 1 min · 206 words · Sam McLeod

Fix XenServer SR with corrupt or invalid metadata

If a disk / VDI is orphaned or only partially deleted you’ll notice that under the SR it’s not assigned to any VM. This can cause issues that look like metadata corruption resulting in the inability to migrate VMs or edit storage. For example: [root@xenserver-host ~]# xe vdi-destroy uuid=6c2cd848-ac0e-441c-9cd6-9865fca7fe8b Error code: SR_BACKEND_FAILURE_181 Error parameters: , Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters: /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/MGT, 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b. Error: Failed to write file with params [3, 0, 512, 512]. Error: 5], Removing stale VDIs To fix this, you need to remove those VDIs from the SR after first deleting the logical volume: ...

January 18, 2016 · 2 min · 296 words · Sam McLeod

iSCSI SCSI-ID / Serial Persistence

“Having a SCSI ID is a f*cking idiotic thing to do.” - Linus Torvalds …and after the amount of time I’ve wasted getting XenServer to play nicely with LIO iSCSI failover I tend to agree. The Problem One oddity of Xen / XenServer’s storage subsystem is that it identifies iSCSI storage repositories via a calculated SCSI ID rather than the iSCSI Serial - which would be the sane thing to do. ...

December 14, 2015 · 3 min · 626 words · Sam McLeod

How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync

Slides Failover Demo

November 9, 2015 · 1 min · 3 words · Sam McLeod

SAN Intro

October 7, 2015 · 0 min · 0 words · Sam McLeod

Replacing Junos Pulse with OpenConnect

In an attempt to avoid using the Juniper Pulse (Now Pulse Secure) VPN client we tried OpenConnect but found that DNS did not work correctly when connected to the VPN. This bug has now been resolved recently but has not made it’s way into a new build, in fact there have been no releases for 6 months. Luckily the OpenConnect was not too difficult to build from source. Build OpenConnect on OSX Remove old openconnect and install deps brew remove openconnect brew install libxml2 lzlib openssl libtool libevent Build openconnect wget git.infradead.org/users/dwmw2/openconnect.git/snapshot/0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz tar zxvf 0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz cd openconnect-0f1ec30 LIBTOOLIZE=glibtoolize ./autogen.sh PATH=/usr/local/opt/gettext/bin:$PATH ./configure make make install To connect sudo openconnect --juniper -u myusername www.myserver.com If you’re comfortable with allowing admin users to run openconnect without entering a sudo password, add the following using sudo visudo: ...

September 22, 2015 · 1 min · 137 words · Sam McLeod

SSD Storage - Two Months In Production

Over the last two months I’ve been running selected IO intensive servers off the the SSD storage cluster, these hosts include (among others) our: Primary Puppetmaster Gitlab server Redmine app and database servers Nagios servers Several Docker database host servers Reliability We haven’t had any software or hardware failures since commissioning the storage units. During this time we have had 3 disk failures on our HP StoreVirtual SANs that have required us to call the supporting vendor and replace failed disks. ...

September 13, 2015 · 2 min · 376 words · Sam McLeod

OS X Software Update Channels For Betas

Set update channel to receive developer beta update sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11seed-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz Set update channel to receive public beta update sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz List available updates sudo softwareupdate --list Set update channel to receive default, stable updates sudo softwareupdate --clear-catalog Show current settings defaults read /Library/Preferences/com.apple.SoftwareUpdate.plist Write setting manually defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz

September 1, 2015 · 1 min · 55 words · Sam McLeod