Generating Diagrams with with AI / LLMs

The two tools I use with AI / LLMs for generating diagrams are Mermaid and Excalidraw. Mermaid (or MermaidJS) is a popular diagramming library and format supported by many tools and is often rendered inside markdown (e.g. in a readme.md). Excalidraw is an excellent, free and open source diagramming and visualisation tool. I also often make use of a third party Obsidian plugin for Excalidraw. Excalidraw It has a ‘generate diagram with AI’ feature which if you’re using the excalidraw.com online editor offers a few free generations each day (I think this uses a low-end OpenAI model). If you’re running Excalidraw locally or by using the brilliant Obsidian plugin - you can provide any OpenAI compatible API endpoint and model for AI generations. Behind the scenes Excalidraw AI generates and then renders MermaidJS. ...

October 8, 2024 · Sam McLeod

Ingest: Streamlining Content Preparation for LLMs

Ingest is a tool I’ve written to make my life easier when preparing content for LLMs. It parses directories of plain text files, such as source code, documentation etc… into a single markdown file suitable for ingestion by AI/LLMs. Ingest can also estimate vRAM requirements for a given model, quantisation and context length: Features Traverse directory structures and generate a tree view Include/exclude files based on glob patterns Estimate vRAM requirements and check model compatibility using another package I’ve created called quantest Parse output directly to LLMs such as Ollama or any OpenAI compatible API for processing Generate and include git diffs and logs Count approximate tokens for LLM compatibility Customisable output templates Copy output to clipboard (when available) Export to file or print to console Optional JSON output Ingest Intro (“Podcast” Episode): ...

July 29, 2024 · Sam McLeod

LLM Parameter Playground

Here’s a fun little tool I’ve been hacking on to explore the effects of different inference parameters on LLMs. You can find the code and instructions for running it locally on GitHub. It started as a fork of rooben-me’s tone-changer-open, which itself was a “fork” of Figma’s tone generator, I’ve made quite a few changes to make it more focused on local LLMs and advanced parameter exploration.

July 20, 2024 · Sam McLeod

Code, Chaos, and Copilots (AI/LLM Talk July 2024)

Code, Chaos, and Copilots is a talk I gave in July 2024 as an intro to how I use AI/LLMs to augment my capabilities every day. What I use AI/LLMs for Prompting tips Codegen workflow Picking the right models Model formats Context windows Quantisation Model servers Inference parameters Clients & tools Getting started cheat-sheets Download Slide Deck Disclaimer I’m not a ML Engineer or data scientist, As such, the information presented here is based on my understanding of the subject and may not be 100% accurate or complete. ...

July 18, 2024 · Sam McLeod

SuperPrompter - Supercharge your text prompts for AI/LLM image generation

SuperPrompter is a Python-based application that utilises the SuperPrompt-v1 model to generate optimised text prompts for AI/LLM image generation (for use with Stable Diffusion etc…) from user prompts. See Brian Fitzgerald’s Blog for a detailed explanation of the SuperPrompt-v1 model and its capabilities / limitations. Features Utilises the SuperPrompt-v1 model for text generation. A basic (aka ugly) graphical user interface built with tkinter. Customisable generation parameters (max new tokens, repetition penalty, temperature, top p, top k, seed). Optional logging of input parameters and generated outputs. Bundling options to include or exclude pre-downloaded model files. Unloads the models when the application is idle to free up memory. Prebuilt Binaries Check releases page to see if there are any prebuilt binaries available for your platform. ...

March 22, 2024 · 2 min · 422 words · Sam McLeod

Llamalink - Ollama to LM Studio LLM Model Linker

Two of my most commonly used LLM tools are Ollama and LM Studio. Unfortunately they store their models in different locations and filenames. Manually copying or linking files was a pain, so I wrote a simple command-line tool to automate the process. This is why I created Llamalink. Ollama is a cross-platform model server that allows you to run LLMs and manage their models in a similar way to Docker containers and images, while LM Studio is a macOS app that provides a user-friendly interface for running LLMs. ...

March 21, 2024 · 3 min · 427 words · Sam McLeod

Open source, locally hosted AI powered Siri replacement

Offline AI / LLM Assistant More info on this soon but the basic idea was to use Willow, Home Assistant and local LLM models to create a locally hosted, offline, AI powered Siri replacement and interface it with ESP32 S3 Box 3 devices. ...

November 20, 2023 · 1 min · 192 words · Sam McLeod

MBA Washing

“MBA Washing” refers to the phenomenon where individuals, often with a strong theoretical or academic background but limited recent practical experience, adopt and reinterpret industry-specific terminology and cultural movements. – This reinterpretation is typically influenced by their academic learning, recent literature, and biases towards larger enterprise perspectives. As a result, the original intent and practical effectiveness of these concepts, such as DevOps or Platform Engineering for example, may be diluted or misrepresented. ...

November 13, 2023 · 3 min · 560 words · Sam McLeod

SDXL LoRA Training

A talk I gave to some peers on creating your own SDXL LoRA models from my tinkering around over the last few weeks. ...

October 30, 2023 · 1 min · 37 words · Sam McLeod

Introduction to AI and Large Language Models (LLMs)

This is a high level intro to LLMs that I’m writing for a few friends that are new to the concept. It is far from complete, definitely contains some errors and is a work in progress. This is a work in progress and a living document. Language models, or LLMs, are a type of artificial intelligence that can generate text based on a given prompt. They work by learning patterns in large amounts of text data and using those patterns to generate new text. LLMs can be used for a variety of tasks, such as generating chatbots, answering questions, and creating art. ...

January 26, 2023 · 13 min · 2741 words · Sam McLeod

Making Work Visible - Avoid DMs

We create more value by having conversations in public instead of behind closed doors. ...

October 20, 2022 · 6 min · 1071 words · Sam McLeod

Camels Dressed As Unicorns

Stop trying to hire with titles like ‘DevOps Engineer’ or ‘Cloud Engineer’ “DevOps is … not a job title” “It’s more of a cultural practice, like innovation, and it makes just as little sense hiring “innovation engineers” and expecting your organisation to be innovative, without also creating the culture to foster innovation.” (Joel Shea when responding to this posted on LinkedIn) In most cases it’s clear the organisation doesn’t truly know what they want or need and likely don’t understand the nuances of the aspects of engineering. ...

August 8, 2019 · 2 min · 328 words · Sam McLeod

Looking For New Opportunities

As stated in my announcement post - after 7~ years, I resigned from Infoxchange several months ago. I planned on taking a holiday, relaxing, decompressing, dipping my toes in some off-the-shelf tech I haven’t spent a lot of time with and performing some personal growth through reading a number of books that have been sitting on my ’to read’ shelf. This is exactly what I’ve done. I’m now open to new opportunities I’d rather find a workplace that I can add great value too and be happy to work with than rush into a new role quickly, obviously if something great turns up right away - I’ll pursue it. ...

August 6, 2019 · 4 min · 763 words · Sam McLeod

Leaving Infoxchange

After 7~ years, I have resigned from Infoxchange. In 2012 I was hired to work with Infoxchange to “shake things up” (in the Systems Operations team) and stabilise the hosting environment. As Team Lead of Systems Operations I was tasked with melding the team and then for us to work to provide a stable, robust, modern and scalable platform for Infoxchange developed application hosting and product delivery and in a short time, with a small budget - we did just that. ...

June 29, 2019 · 3 min · 470 words · Sam McLeod

Goodbye XenSever - Hello XCP-ng

In 2018 I set out to replace our XenSever 7.2 based virtualisation after Citrix essentially screwed over free / open source users. This project was to directly replace XenServer 7.2 with something supported and manageable for our traditional virtualisation needs. High Level Selection Considerations I evaluated a number of options, with the primary candidates below. Key criteria (at a high level) I was evaluating: Ease of moving from our existing XenServer 7.2 based hypervisor clusters. Security (architecture, hardening, monitoring, logging). Cost (both licensing if any and self-support / management costs). VM Performance (Storage IOPs and throughput, Network latency and throughput, Processing latency, steal from over-provisioned workloads). Management UI/UX and performance (for BAU activities). Management / Cluster SPOFs, fail-over and redundancy. Installation and upgrade process. Update and security patching frequency. Networking design and complexity. Community (size, engagement, acceptance of suggestions / MRs). Reliable live VM migrations. Ease of management for a small team (Part of a low TCO). Risk of vendor and technology lock-in. Risk of survival (will it still be well maintained over the next 1-3 years). Stability and reliability above practically all else. XenServer 7.6 (w/ Paid License) For Potentially priority support from Citrix for issues. Easy upgrade from 7.2. Fresh installs and upgrades are simple, painless and easy to pxeboot, licensing can be a pain after install however. Against Slow moving development. Incredibly expensive, it would cost us something like $140,800 AUD per year (shelf price) for our 32 hosts (each with 2 sockets). Features heavily restricted by licensing model. Mostly older-fashioned enterprise users. No web management interface (although you can use Xen Orchestra from the folks behind XCP-ng). Diminishing community since XenServer licensing changes in 7.3. Poor storage performance compared to KVM based solutions. Many outdated packages, while kind of based on CentOS 7, there are a great deal of packages from older releases or completely custom rebuilt. Uses old technologies like EXT3 and doesn’t support SSD/Flash TRIM/DISCARD functions. SELinux not enforcing or supported. OpenvSwitch still uses a lot of Dom0 CPU (same as XenServer). Dom0 often ends up limiting VM storage operations (tapdisk maxing out Dom0 CPU). Applying updates can be painful or at least slow, requiring binary ISO files to be downloaded from Citrix. Uses Jira for bug tracking (I just can’t stand the thing, it’s painfully laggy, give my GitLab or Github over it any day). XCP-ng XCP-ng is a relatively recent fork from XenServer after it was open sourced, tracking upstream but clearly prioritising modernisation and community. ...

February 5, 2019 · 8 min · 1550 words · Sam McLeod

Run youtube-dl (or similar) in the background

I wanted an alias or function to use youtube-dl in the background. Looking around the web lots of people seemed to want this and most of them were banging their heads against a wall due to: A) bash quoting B) backgrounding dying when their terminal closed C) passing the argument (url in my case) to the function Here’s a simple function I whipped up that seems to ‘just works’™: function yt { nohup youtube-dl "$1" --no-progress 2>&1 > youtube-dl-"$(date +%Y%m%d-%H%M%S)".log & } And if you don’t want logs, simply send the output to /dev/null: ...

July 30, 2018 · 2 min · 231 words · Sam McLeod

Disabling scroll-wheel zoom in Firefox

This feature annoys me endlessly, I end up zoomed in and out of websites all over the internet. … But the fix is easy and there’s no addons required. Navigate to about:config (in Firefox’s URL bar) Change the value of the following two properties to 0: mousewheel.with_control.action mousewheel.with_meta.action If you use Firefox sync and want these settings to sync between your machines, also add the following properties: Create two new properties both of type boolean and set them to true: ...

July 16, 2018 · 1 min · 83 words · Sam McLeod

Flash Storage and SSD Failure Rate Update (March 2018)

It was almost 3 years ago that my open source storage project went into production. In that time it’s been running 24/7 serving as highly available solid state storage for hundreds of VMs and several virtualisation clusters across our two main sites. I’m happy to report that the clusters have been operating very successfully since their conception. Since moving away from proprietary ‘black box’ vendor SANs, we haven’t had a single SAN issue, storage outage. ...

March 20, 2018 · 2 min · 326 words · Sam McLeod

HP 4951C Protocol Analyser

My good friend Joel Shea received a most unlikely gift this Christmas - A vintage HP 4951 Protocol Analyser. According to the HP Computer Museum: Original Price: $3595 The 4951B was replaced by the 4951C and 4952A in 1986. Both new models handled Async, BSC, SDLC, HDLC, X.25 and SNA protocols. The 4951C also handled DDCMP, while the 4952A did not. The 4952A handled X.21 while the 4951C did not. Both new analysers used a floppy dive (618 KB) for removable media. ...

December 27, 2017 · 1 min · 91 words · Sam McLeod

Talk - Clustered, Distributed File and Volume Storage with GlusterFS

Using GlusterFS to provide volume storage to Kubernetes as a replacement for our existing file and static content hosting. This talk was given at Infracoders on Tuesday 14th November 2017. NOTE: Below link to slides currently broken - will fix soon! (03/08/2019) Click below to view slides (PDF version): Direct download link

November 14, 2017 · 1 min · 52 words · Sam McLeod

Applying syntax in Sublime based on the first file line

In vim, you can add a comment at the top of files to set the syntax, e.g.: # vim: syntax=ruby In SublimeText there are many ways to detect syntax, one interesting approach I’ve recently found useful is to match on the top line in the file. For example, with Puppet there is a file called Puppetfile, it has no extension but it’s really Ruby syntax, so it’s useful to add linting incase you miss something simple like a , and break deployments. ...

October 25, 2017 · 1 min · 162 words · Sam McLeod

Broadcom, Or How I Learned To Start Worrying And Drop The Packet

Earlier this week we started the process to upgrade one of our hypervisor compute clusters when we encountered a rather painful bug with HP’s Broadcom NIC chipsets. We were part way through a routine rolling pool upgrade of our hypervisor (XenServer) cluster when we observed unexpected and intermittent loss of connectivity between several VMs, then entire XenServer hosts. The problems appeared to impact hosts that hadn’t yet upgraded to XenServer 7.2. We now attribute this to a symptom of extreme packet loss between the hosts in the pool and thanks to buggy firmware from Broadcom and HP. ...

October 13, 2017 · 6 min · 1240 words · Sam McLeod

GlusterFS

We’re in the process of shifting from using our custom ‘glue’ for orchestrating Docker deployments to Kubernetes, When we first deployed Docker to replace LXC and our legacy Puppet-heavy application configuration and deployment systems there really wasn’t any existing tool to manage this, thus we rolled our own, mainly a few Ruby scripts combined with a Puppet / Hiera / Mcollective driven workflow. The main objective is to replace our legacy NFS file servers used to host uploads / attachments and static files for our web applications, while NFS(v4) performance is adequate, it is a clear single point of failure and of course, there are the age old stale mount problems should network interruptions occur. ...

September 25, 2017 · 6 min · 1106 words · Sam McLeod

Return Of The RSS

Of all the tools for reading news and subscribing to software releases, I still find RSS the most useful. I use Feedly to manage my rss subscriptions and keep all my devices in sync, but instead of using the Feedly’s own client, I use an app called Reeder as the client / reader itself. Link: My Feedly RSS Feed Feedly RSS feed subscription management Features: Keyword alerts. Browser plugins to subscribe to (current) url. Notation and highlighting support (a bit like Evernote). Search and filtering across large numbers of feeds / content. IFTTT, Zapier, Buffer and Hootsuite integration. Built in save / share functionality (that I only use when I’m on the website). Backup feeds to Dropbox. Very fast, regardless of the fact that I’m in Australia - which often impacts the performance of apps / sites that tend to be hosted on AWS in the US as the latency is so high. Article de-duplication is currently being developed I believe, so I’m looking forward to that! Easy manual import, export and backup (no vendor lock-in is important to me). Public sharing of your Feedly feeds (we’re getting very meta here!). Reeder A (really) beautiful and fast iOS / macOS client ...

September 22, 2017 · 2 min · 283 words · Sam McLeod

The State of Android in 2016 & The OnePlus 3 Phone

I wanted to try Android for a couple of weeks, I like staying on top of technology, gadgets and making sure I never become a blind ‘zealot’ for any platform or brand. The OnePlus 3 I did a lot of research and decided to try the “Oneplus 3” as it was good bang-for-buck, ran the latest software had plenty of grunt with the latest 8 core, high clock speed Qualcomm processor coupled with 6GB of DDR4 - the specs really are very impressive, especially for a $400USD phone. ...

July 11, 2016 · 4 min · 721 words · Sam McLeod

Update Delayed Serial STONITH Design

note: This is a follow up post from 2015-07-21-rcd-stonith A Linux Cluster Base STONITH provider for use with modern Pacemaker clusters This has since been accepted and merged into Fedora’s code base and as such will make it’s way to RHEL. Source Code: Github Diptrace CAD Design: Github I have open sourced the CAD circuit design and made this available within this repo under CAD Design and Schematics Related RedHat Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1240868 v1 vs v2/v3 versions of the rcd_serial STONITH system The v2/v3 cables include the following improvements: ...

July 4, 2016 · 2 min · 217 words · Sam McLeod

Monitoring SystemD Units With Nagios

Ever forgotten to add a critical service to monitoring? Want to know if a service or process fails without explicitly monitoring every service on a host? …Then why not use SystemD’s existing knowledge of all the enabled services? Thanks to ‘Kbyte’ who made a simple Nagios plugin to do just this! Requirements Python3 (For RHEL/CentOS 7 yum install python34) python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin PyNagSystemD

May 23, 2016 · 1 min · 68 words · Sam McLeod

Online Conversion from SQL_ASCII to UTF8 in PostgreSQL

Scripts and source available here: sql_ascii_to_utf8 The Goal To be able to take a Postgres Database which is in SQL_ASCII encoding, and import it into a UTF8 encoded database. Requirements: Python3 (For RHEL/CentOS 7 yum install python34) python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin PyNagSystemD The Problem PostreSQL will generate errors like this if it encounters any non-UTF8 byte-sequences during a database restore: # pg_dump -Fc test_badchar | pg_restore -d test_badchar_utf8 pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 2839; 0 26852 TABLE DATA table101 postgres pg_restore: [archiver (db)] COPY failed for table "table101": ERROR: invalid byte sequence for encoding "UTF8": 0x91 CONTEXT: COPY table101, line 1 WARNING: errors ignored on restore: 1 And the corresponding data will be omitted from the database (in this case, the whole table, even the rows which did not have a problem): ...

May 23, 2016 · 5 min · 902 words · Sam McLeod

Speeding Up rsync

The most common way to use rsync is probably as such: rsync -avr user@<source>:<source_dir> <dest_dir> Resulting in 30-35MB/s depending on file sizes This can be improved by using a more efficient, less secure encryption algorithm, disabling compression and telling the SSH client to disable some unneeded features that slow things down. With the settings below I have achieved 100MB/s (at work between VMs) and over 300MB/s at home between SSD drives. ...

May 3, 2016 · 2 min · 224 words · Sam McLeod

Benchmarking IO with FIO

This is a quick tldr there are many other situations and options you could consider FIO man page IOP/s = Input or Output operations per second Throughput = How many MB/s can you read/write continuously Variables worth tuning based on your situation --iodepth The iodepth is very dependant on your hardware. Rotational drives without much cache and high latency (i.e. desktop SATA drives) will not benefit from a large iodepth, Values between 16 to 64 could be sensible. ...

April 29, 2016 · 2 min · 393 words · Sam McLeod

Mirroring a Gitlab project to Github

Let’s pretend you have a project on Gitlab called ask-izzy and you want to mirror it up to Gitlab which is located at https://github.com/ask-izzy/ask-izzy Assuming you’re running Gitlab as the default user of git and that your repositories are stored in /mnt/repositories you can following something similar to the following instructions: Grant write access to Github Get your Gitlab install’s pubkey from the git user cat /home/git/.ssh/id_rsa.pub On Github add this pubkey as deploy key on the repo, make sure you tick the option to allow write access. ...

February 4, 2016 · 1 min · 145 words · Sam McLeod

AskIzzy

Today we launched a mobile website for homeless people … and it was launched by one of Australia’s many recent Prime Ministers Today alone we served up over 87,000 requests As many of you know, I work with Infoxchange as the operations lead. When I first heard the idea of a website or app for people that have found or are worried about finding themselves homeless in Australia I really didn’t think it made sense - until I saw the stats showing how many homeless people in Australia have regular access to a smart phone and data either via a cellular provider or free WiFi. ...

January 29, 2016 · 1 min · 206 words · Sam McLeod

Fix XenServer SR with corrupt or invalid metadata

If a disk / VDI is orphaned or only partially deleted you’ll notice that under the SR it’s not assigned to any VM. This can cause issues that look like metadata corruption resulting in the inability to migrate VMs or edit storage. For example: [root@xenserver-host ~]# xe vdi-destroy uuid=6c2cd848-ac0e-441c-9cd6-9865fca7fe8b Error code: SR_BACKEND_FAILURE_181 Error parameters: , Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters: /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/MGT, 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b. Error: Failed to write file with params [3, 0, 512, 512]. Error: 5], Removing stale VDIs To fix this, you need to remove those VDIs from the SR after first deleting the logical volume: ...

January 18, 2016 · 2 min · 296 words · Sam McLeod

iSCSI SCSI-ID / Serial Persistence

“Having a SCSI ID is a f*cking idiotic thing to do.” - Linus Torvalds …and after the amount of time I’ve wasted getting XenServer to play nicely with LIO iSCSI failover I tend to agree. The Problem One oddity of Xen / XenServer’s storage subsystem is that it identifies iSCSI storage repositories via a calculated SCSI ID rather than the iSCSI Serial - which would be the sane thing to do. ...

December 14, 2015 · 3 min · 626 words · Sam McLeod

How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync

Slides Failover Demo

November 9, 2015 · 1 min · 3 words · Sam McLeod

SAN Intro

October 7, 2015 · 0 min · 0 words · Sam McLeod

Replacing Junos Pulse with OpenConnect

In an attempt to avoid using the Juniper Pulse (Now Pulse Secure) VPN client we tried OpenConnect but found that DNS did not work correctly when connected to the VPN. This bug has now been resolved recently but has not made it’s way into a new build, in fact there have been no releases for 6 months. Luckily the OpenConnect was not too difficult to build from source. Build OpenConnect on OSX Remove old openconnect and install deps brew remove openconnect brew install libxml2 lzlib openssl libtool libevent Build openconnect wget git.infradead.org/users/dwmw2/openconnect.git/snapshot/0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz tar zxvf 0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz cd openconnect-0f1ec30 LIBTOOLIZE=glibtoolize ./autogen.sh PATH=/usr/local/opt/gettext/bin:$PATH ./configure make make install To connect sudo openconnect --juniper -u myusername www.myserver.com If you’re comfortable with allowing admin users to run openconnect without entering a sudo password, add the following using sudo visudo: ...

September 22, 2015 · 1 min · 137 words · Sam McLeod

SSD Storage - Two Months In Production

Over the last two months I’ve been running selected IO intensive servers off the the SSD storage cluster, these hosts include (among others) our: Primary Puppetmaster Gitlab server Redmine app and database servers Nagios servers Several Docker database host servers Reliability We haven’t had any software or hardware failures since commissioning the storage units. During this time we have had 3 disk failures on our HP StoreVirtual SANs that have required us to call the supporting vendor and replace failed disks. ...

September 13, 2015 · 2 min · 376 words · Sam McLeod

OS X Software Update Channels For Betas

Set update channel to receive developer beta update sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11seed-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz Set update channel to receive public beta update sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz List available updates sudo softwareupdate --list Set update channel to receive default, stable updates sudo softwareupdate --clear-catalog Show current settings defaults read /Library/Preferences/com.apple.SoftwareUpdate.plist Write setting manually defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz

September 1, 2015 · 1 min · 55 words · Sam McLeod

iSCSI Benchmarking

The following are benchmarks from our testings of our iSCSI SSD storage. 67,300 read IOP/s on a VM on iSCSI (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM) Per VM and scales to 1,000,000 IOP/s total root@dev-samm:/mnt/pmt1 128 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=read test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128 2.0.8 Starting 1 process bs: 1 (f=1): [R] [55.6% done] [262.1M/0K /s] [67.3K/0 iops] [eta 00m:04s] 38,500 random 4k write IOP/s on a VM on iSCSI (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM) Per VM and scales to 700,000 IOP/s total root@dev-samm:/mnt/pmt1 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=randwrite test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128 2.0.8 Starting 1 process bs: 1 (f=1): [w] [26.3% done] [0K/150.2M /s] [0 /38.5K iops] [eta 00m:14s] Raw device latency on storage units Intel DC3600 1.2T PCIe NVMe root@s1-san6:/proc # ioping /dev/nvme0n1p1 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=1 time=104 us 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=2 time=83 us 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=3 time=51 us 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=4 time=71 us SanDisk SDSSDXPS960G SATA root@pm-san5:/proc # ioping /dev/sdc 4.0 KiB from /dev/sdc (device 894.3 GiB): request=1 time=4.2 ms 4.0 KiB from /dev/sdc (device 894.3 GiB): request=2 time=4.1 ms 4.0 KiB from /dev/sdc (device 894.3 GiB): request=3 time=4.1 ms 4.0 KiB from /dev/sdc (device 894.3 GiB): request=4 time=4.1 ms Micron_M600_MTFDDAK1T0MBF SATA root@pm-san5:/proc # ioping /dev/sdf 4.0 KiB from /dev/sdf (device 953.9 GiB): request=1 time=157 us 4.0 KiB from /dev/sdf (device 953.9 GiB): request=2 time=190 us 4.0 KiB from /dev/sdf (device 953.9 GiB): request=3 time=65 us 4.0 KiB from /dev/sdf (device 953.9 GiB): request=4 time=181 us ```shell ## Latency on the a VM - (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM) ```shell root@dev-samm:/mnt 127 # ioping pmt1/ 4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=1 time=0.6 ms 4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=2 time=0.7 ms 4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=3 time=0.7 ms --- pmt1/ (ext4 /dev/xvdb1) ioping statistics --- 3 requests completed in 2159.1 ms, 1508 iops, 5.9 mb/s min/avg/max/mdev = 0.6/0.7/0.7/0.1 ms root@dev-samm:/mnt # ioping pmt2/ 4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=1 time=0.6 ms 4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=2 time=0.8 ms --- pmt2/ (ext4 /dev/xvdc1) ioping statistics --- 2 requests completed in 1658.4 ms, 1470 iops, 5.7 mb/s min/avg/max/mdev = 0.6/0.7/0.8/0.1 ms root@dev-samm:/mnt # ioping pmt3/ 4096 bytes from pmt3/ (ext4 /dev/xvde1): request=1 time=0.6 ms 4096 bytes from pmt3/ (ext4 /dev/xvde1): request=2 time=0.9 ms 4096 bytes from pmt3/ (ext4 /dev/xvde1): request=3 time=0.9 ms ...

July 24, 2015 · 3 min · 456 words · Sam McLeod

Delayed Serial STONITH

A modified version of John Sutton’s rcd_serial cable coupled with our Supermicro reset switch hijacker: This works with the rcd_serial fence agent plugin. Reasons rcd_serial makes for a very good STONITH mechanism: It has no dependency on power state. It has no dependency on network state. It has no dependency on node operational state. It has no dependency on external hardware. It costs less that $5 + time to build. It is incredibly simple and reliable. Essentially the most common STONITH agent type in use is probably those that control UPS / PDUs, while this sounds like a good idea in theory there are a number of issues with relying on a UPS / PDU: ...

July 21, 2015 · 3 min · 450 words · Sam McLeod

Video - Cluster Failover Performance Demo

July 12, 2015 · 0 min · 0 words · Sam McLeod

CentOS 7 and HA

First some background… One of the many lessons I’ve learnt from my Linux HA / Storage clustering project is that the Debian HA ecosystem is essentially broken, We reached the point where packages were too old, too buggy or in Debian 8’s case - outright missing. In the past I was very disappointed with RHEL/CentOS 5 / 6 and (until now) have been quite satisfied with Debian as a stable server distribution with historicity more modern packages and kernels. ...

July 7, 2015 · 3 min · 558 words · Sam McLeod

SSD Storage Cluster - Update and Diagram

Due to several recent events beyond my control I’m a bit behind on the project - hence the lack of updates which I apologise for. The goods news is that I’m back working to finish off the clusters and I’m happy to report that all is going to plan. Here is the final digram of the two-node cluster design: Plain text version available here This was generated from the LCMC tool (beware - it’s java!). ...

June 17, 2015 · 1 min · 79 words · Sam McLeod

Video - Storage Cluster Failover Demo

A brief demonstration of the failover and recovery process on the storage clusters I’ve been building.

May 14, 2015 · 1 min · 16 words · Sam McLeod

Talk - High Performance Software Defined Storage

A high level talk from Infracoders Melbourne on 12/04/2015. There’s also a low quality recording available here: Related posts: Building a high performance SSD SAN - Part 1

April 15, 2015 · 1 min · 28 words · Sam McLeod

Continuous integration for the Linux Kernel - Built within Docker

Linux Kernel CI for Debian Github: sammcj/kernel-ci Those of us using technologies such as Docker and BTRFS or simply trying to gain a performance edge on the competition have a lot to gain from the features and performance of recent Kernel updates (especially from 3.18 onwards). ‘Enterprise’ Linux distributions such as RHEL & variants are concerningly out of date when comes to the Kernel. Many people seem to have forgotten what Linux is… Linux IS the Kernel. ...

March 15, 2015 · 2 min · 310 words · Sam McLeod

Xen Orchestra Docker Image

Docker config to setup XO which is a web interface to visualize and administrate your XenServer (or XAPI enabled) hosts Github: sammcj/docker-xen-orchestra Running the app Updates are pushed to the Docker Hub’s automated build service: https://registry.hub.docker.com/u/sammcj/docker-xen-orchestra From Docker Hub docker pull sammcj/docker-xen-orchestra docker run -d -p 8000:80 sammcj/docker-xen-orchestra Building git clone https://github.com/sammcj/docker-xen-orchestra.git cd docker-xen-orchestra # Edit whatever config you want to change docker build -t xen-orchestra . See https://xen-orchestra.com for information on Xen Orchestra ...

February 26, 2015 · 1 min · 74 words · Sam McLeod

Delete Government-Linked Certificate Authorities in OSX

Inspired by http://zitseng.com/archives/7489 Source (Github) WARNINGS Do not run unless you understand what this is doing The CA system is broken by design - This is not a fix for that This is merely a band-aid for those interested or concerned about these root CAs Usage chmod +x delete_gov_roots.sh ./delete_gov_roots.sh You’ll be prompted for your password as root access is required to delete system-wide root certs. See Also http://convergence.io https://addons.mozilla.org/en-US/firefox/addon/certificate-patrol/ https://github.com/kirei/catt https://www.eff.org/observatory https://bugzilla.mozilla.org/show_bug.cgi?id=478418 http://support.apple.com/en-us/HT202858 https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning

February 23, 2015 · 1 min · 75 words · Sam McLeod

Building a high performance SSD SAN - Part 1

Over the coming month I will be architecting, building and testing a modular, high performance SSD-only storage solution. I’ll be documenting my progress / findings along the way and open sourcing all the information as a public guide. With recent price drops and durability improvements in solid state storage now is better time than any to ditch those old magnets. Modular server manufacturers such as SuperMicro have spent large on R&D thanks to the ever growing requirements from cloud vendors that utilise their hardware. ...

February 16, 2015 · 8 min · 1590 words · Sam McLeod

Direct-Attach SSD Storage - Performance & Comparisons

Further to my earlier post on XenServer storage performance with regards to directly attaching storage from the host, I have been analysing the performance of various SSD storage options. I have attached a HP DS2220sb storage blade to an existing server blade and compared performance with 4 and 6 SSD RAID-10 to our existing iSCSI SANs. While the P420i RAID controller in the DS2220sb is clearly saturated and unable to provide throughput much over 1,100MB/s - the IOP/s available to PostgreSQL are still a very considerably performance improvement over our P4530 SAN - in fact, 6 SSD’s result in a 39.9x performance increase! ...

February 15, 2015 · 1 min · 110 words · Sam McLeod

Search - A Journey of Delivery on a Budget

“Search - A Journey of Delivery on a Budget” Presented a Melbourne Search - July 2014 - “Search - A Journey of Delivery on a Budget” (Click for slides)

February 15, 2015 · 1 min · 29 words · Sam McLeod

Talk - 24 Months

The way we work at Infoxchange has changed greatly. A retrospective journey into transforming Infoxchange’s technology and culture over the past 24 months - presented a Melbourne DevOps - December 2014

February 15, 2015 · 1 min · 31 words · Sam McLeod

The Best Of - 2014 Edition

At the end of every year I note down a summary of the best applications, hardware & websites I’ve enjoyed & depended on throughout the year (and often for some time before). This post has long since been superseded. You can find the latest version of this post here. Software / General Use Fastmail - https://www.fastmail.com Evernote - https://evernote.com Reeder - http://reederapp.com Keynote - https://www.apple.com/au/mac/keynote Lastpass - https://lastpass.com Plex - https://plex.tv Calibre - http://calibre-ebook.com Software / Geek Use Sublime Text - http://www.sublimetext.com/3 Homebrew - http://brew.sh DropSync - http://mudflatsoftware.com Beets - http://beets.radbox.org Textual - http://www.codeux.com/textual XLD - http://tmkk.undo.jp/xld/index_e.html Code Academy - http://www.codecademy.com Exercism.io - http://exercism.io Sickbeard + Headphones + Couchpotato + Sabnzbd - http://www.totalhtpc.com/ultimate-usenet-guide.html Software / Mobile Tweetbot - http://tapbots.com/software/tweetbot Pushover - http://pushover.net Lastpass - https://lastpass.com Keynote - https://www.apple.com/au/ios/keynote Reeder - http://reederapp.com/ios Evernote- https://evernote.com Plex - https://plex.tv Backblaze - https://www.backblaze.com WTF Podcast - http://www.wtfpod.com/app MiniHack - https://itunes.apple.com/au/app/minihack-for-hacker-news/id631108846?mt=8 Uber - https://itunes.apple.com/au/app/uber/id368677368?mt=8 Goodreads - https://itunes.apple.com/au/app/goodreads-book-recommendations/id355833469?mt=8 Notify4M - https://itunes.apple.com/au/app/notify4m/id499161979?mt=8 Bandcamp - https://itunes.apple.com/au/app/bandcamp/id706408639?mt=8 Hype Machine - https://itunes.apple.com/au/app/hype-machine/id414315986?mt=8 Nuzzel (Only got onto this today) - https://itunes.apple.com/au/app/nuzzel-news-from-your-friends/id692285770?mt=8 Alien Blue - http://www.reddit.com/r/alienblue Software / SysAdmin or DevOps Specific Gitlab - http://gitlab.org Gitlab-CI - https://about.gitlab.com/gitlab-ci Dash - http://kapeli.com/dash PostgreSQL (Makes my list every year & just keeps getting better) - http://www.postgresql.org PGBadger - http://dalibo.github.io/pgbadger Docker - https://www.docker.com Consul - https://consul.io Puppet (I couldn’t do my job as well without it) - http://puppetlabs.com iTerm - http://iterm2.com Nginx - http://nginx.org Htop - http://hisham.hm/htop Bonnie++ - http://linux.die.net/man/8/bonnie++ Openfire - https://www.igniterealtime.org/projects/openfire Hiera-Eyaml - https://github.com/TomPoulton/hiera-eyaml Rubinius - http://rubini.us Puma - http://puma.io XenServer - http://xenserver.org ElasticSearch - www.elasticsearch.org Logstash - http://logstash.net FPM - https://github.com/jordansissel/fpm PFsense - https://www.pfsense.org Debian Jessie (Not quite released but the next great version of the best linux Distro IMO) - https://www.debian.org/releases/jessie Check_MK Multisite - https://mathias-kettner.de/checkmk_multisite.html PWSafe - https://itunes.apple.com/au/app/pwsafe-password-safe-compatible/id520993579?mt=12 Supervisord - http://supervisord.org Websites Last.fm - http://www.last.fm/user/sammcj2000 Feedly - https://feedly.com HackerNews - https://news.ycombinator.com/news Lucidchart - http://lucidchart.com/ MondoTunes (Might be a little biased here!) - http://mondotunes.org Hardware Das Ultimate 4- http://www.daskeyboard.com Logitech Performance MX - http://www.logitech.com/en-au/product/performance-mouse-mx 15’’ Macbook Pro Retina - http://www.apple.com/au/macbook-pro iPhone 6+ (because its bigger than bigger, or something) - https://www.apple.com/iphone-6 CuBox - http://www.solid-run.com/product/cubox-i4pro Parani SD1000 Bluetooth Serial Adapter - http://www.senaindustrial.com/products/industrial_bluetooth/sd1000.php SanDisk Extreme Pro 480GB SSD - http://www.newegg.com/Product/Product.aspx?Item=N82E16820171999 Other / Non-tech All Day Socks - http://alldaysocks.com Sennheiser Amperior On-Ear Headphones - http://www.head-fi.org/products/sennheiser-amperior-on-ear-headphones O2 + ODAC - http://www.jdslabs.com/products/48/o2-odac-combo Benchmark DAC1 HDR - http://benchmarkmedia.com/products/benchmark-dac1-hdr-digital-to-analog-converter Bellroy Wallets - http://bellroy.com Ink Shoes - http://www.inkshoes.it Kindle Paperwhite - http://www.amazon.com.au/gp/feature.html?docId=3077740006 Nerf Jolt - http://nerf.wikia.com/wiki/Jolt_EX-1 Books The Phoenix Project (Reread this year for the 3rd time) - http://itrevolution.com/books/phoenix-project-DevOps-book Surely You’re Joking Mr Feynman! - https://www.goodreads.com/book/show/5544.Surely_You_re_Joking_Mr_Feynman_ The Dark Tower Series - https://www.goodreads.com/book/show/43615.The_Gunslinger Snow Crash - https://www.goodreads.com/book/show/830.Snow_Crash

February 15, 2015 · 2 min · 426 words · Sam McLeod

XenServer, SSDs & VM Storage Performance

Intro At Infoxchange we use XenServer as our Virtualisation of choice. There are many reasons for this including: Open Source. Offers greater performance than VMware. Affordability (it’s free unless you purchase support). Proven backend Xen is very reliable. Reliable cross-host migrations of VMs. The XenCentre client, (although having to run in a Windows VM) is quick and simple to use. Upgrades and patches have proven to be more reliable than VMware. OpenStack while interesting, is not yet reliable or streamlined enough for our small team of 4 to implement and manage. XenServer Storage & Filesystems Unfortunately the downside to XenServer is that it’s underlying OS is quite old. The latest version (6.5) about to be released is still based on Centos 5 and still lacks any form of EXT4 and BTRFS support, direct disk access is not available… without some tweaking and has no real support for TRIM unless you have direct disk access and are happy with EXT3. ...

February 15, 2015 · 5 min · 970 words · Sam McLeod