The personal blog of Sam McLeod. I write about AI, DevOps, Platform Engineering, and other geeky topics.

LLM Parameter Playground

Here’s a fun little tool I’ve been hacking on to explore the effects of different inference parameters on LLMs. You can find the code and instructions for running it locally on GitHub. It started as a fork of rooben-me’s tone-changer-open, which itself was a “fork” of Figma’s tone generator, I’ve made quite a few changes to make it more focused on local LLMs and advanced parameter exploration.

July 20, 2024 · Sam McLeod

Code, Chaos, and Copilots (AI/LLM Talk July 2024)

Code, Chaos, and Copilots is a talk I gave in July 2024 as an intro to how I use AI/LLMs to augment my capabilities every day. What I use AI/LLMs for Prompting tips Codegen workflow Picking the right models Model formats Context windows Quantisation Model servers Inference parameters Clients & tools Getting started cheat-sheets Download Slide Deck Disclaimer I’m not a ML Engineer or data scientist, As such, the information presented here is based on my understanding of the subject and may not be 100% accurate or complete....

July 18, 2024 · Sam McLeod

Understanding AI/LLM Quantisation Through Interactive Visualisations

AI models (“LLMs” in this case) have inherently large sizes and computational requirements that often pose challenges for deployment and use. ...

July 17, 2024 · Sam McLeod

Rating AI Tools

I spend a lot of time working with AI tools and often get asked for recommendations. The following is a list of some of the more notable tools I’ve tried and rated them based on my experience. Just because something has a low rating doesn’t mean it’s bad - it just means it didn’t work well for me and I wouldn’t personally recommend it. I plan on keeping this updated as I try new tools (we’ll see how that goes)....

July 3, 2024 · 1 min · 101 words · Sam McLeod

Gollama: Ollama Model Manager

Gollama on Github Gollama is a client for Ollama for managing models. It provides a TUI for listing, filtering, sorting, selecting, inspecting (coming soon!) and deleting models and can link Ollama models to LM-Studio. The project started off as a rewrite of my llamalink project, but I decided to expand it to include more features and make it more user-friendly. ...

June 1, 2024 · 3 min · 607 words · Sam McLeod

Confuddlement: Download Confluence Spaces as Markdown, Summarise with Ollama

Confuddlement on Github I was tired of manually downloading Confluence pages and converting them to Markdown, so I wrote a small command-line tool designed to simplify this process. Confuddlement is a Go-based tool that uses the Confluence REST API to fetch page content and convert it to Markdown files. It can fetch pages from multiple spaces, skip pages that have already been fetched, and summarise the content of fetched pages using the Ollama API. $ go run ./main.go Confuddlement 0.3.0 Spaces: [COOLTEAM, MANAGEMENT] Fetching content from space COOLTEAM COOLTEAM (Totally Cool Team Homepage) Retrospectives Decision log Development Onboarding Saved page COOLTEAM - Feature List to ./confluence_dump/COOLTEAM - Feature Skipping page 7. Support, less than 300 characters MANAGEMENT (Department of Overhead and Bureaucracy) Painful Change Management Illogical Diagrams Saved page ./confluence_dump/Painful Change Saved page Illogical Diagrams to ./confluence_dump/Ilogical Done! $ go run ./main.go summarise Select a file to summarise: 0: + COOLTEAM - Feature List 1: + Painful Change Management 2: + Illogical Diagrams Enter the number of the file to summarise: 1 Summarising Painful Change Management... "Change management in the enterprise is painful and slow. It involves many forms and approvals." go run main.go -q 'who is the CEO?' -s 'management' -r 2 Querying the LLM with the prompt 'who is the CEO?'... "The CEO of the company is Peewee Herman." Usage Running the Program Copy .env.template to .env and update the environment variables. Run the program using the command go run main.go or build the program using the command go build and run the resulting executable. The program will fetch Confluence pages and save them as Markdown files in the specified directory. Querying the documents with AI You can summarise the content of a fetched page using the Ollama API by running the program with the summarise argument: go run main.go summarise To perform a custom query, you can use the query argument: -q: The query to to provide to the LLM. -s: The search term to match documents against. -r: The number of lines before and after the search term to include in the context to the LLM. go run main.go -q 'who is the CEO?' -s 'management' -r 2 Querying the LLM with the prompt 'who is the CEO?'... "The CEO of the company is Peewee Herman." ...

May 23, 2024 · 3 min · 605 words · Sam McLeod

NVApi - Nvidia GPU Monitoring API

NVApi is a small application I’ve written for monitoring and presenting utilisation metrics from Nvidia GPUs. This can be used to monitor GPU memory, temperature, power usage, and utilisation of GPUs in a system and can easily be integrated into tools such as HomeAssistant or Prometheus. The package uses the Nvidia Management Library (NVML) and provides a simple API for monitoring Nvidia GPUs along with a basic GUI client. NVApi on Github ...

May 18, 2024 · 3 min · 458 words · Sam McLeod

SuperPrompter - Supercharge your text prompts for AI/LLM image generation

SuperPrompter is a Python-based application that utilises the SuperPrompt-v1 model to generate optimised text prompts for AI/LLM image generation (for use with Stable Diffusion etc…) from user prompts. See Brian Fitzgerald’s Blog for a detailed explanation of the SuperPrompt-v1 model and its capabilities / limitations. Features Utilises the SuperPrompt-v1 model for text generation. A basic (aka ugly) graphical user interface built with tkinter. Customisable generation parameters (max new tokens, repetition penalty, temperature, top p, top k, seed)....

March 22, 2024 · 2 min · 422 words · Sam McLeod

Llamalink - Ollama to LM Studio LLM Model Linker

Two of my most commonly used LLM tools are Ollama and LM Studio. Unfortunately they store their models in different locations and filenames. Manually copying or linking files was a pain, so I wrote a simple command-line tool to automate the process. This is why I created Llamalink. Ollama is a cross-platform model server that allows you to run LLMs and manage their models in a similar way to Docker containers and images, while LM Studio is a macOS app that provides a user-friendly interface for running LLMs....

March 21, 2024 · 3 min · 427 words · Sam McLeod

Introduction to AI and Large Language Models (LLMs)

This is a high level intro to LLMs that I’m writing for a few friends that are new to the concept. It is far from complete, definitely contains some errors and is a work in progress. This is a work in progress and a living document. Language models, or LLMs, are a type of artificial intelligence that can generate text based on a given prompt. They work by learning patterns in large amounts of text data and using those patterns to generate new text....

November 26, 2023 · 13 min · 2741 words · Sam McLeod

Open source, locally hosted AI powered Siri replacement

Offline AI / LLM Assistant More info on this soon but the basic idea was to use Willow, Home Assistant and local LLM models to create a locally hosted, offline, AI powered Siri replacement and interface it with ESP32 S3 Box 3 devices. ...

November 20, 2023 · 1 min · 192 words · Sam McLeod

Fun with Makefiles - Dynamic Menu Generation


November 14, 2023 · 2 min · 257 words · Sam McLeod

MBA Washing

“MBA Washing” refers to the phenomenon where individuals, often with a strong theoretical or academic background but limited recent practical experience, adopt and reinterpret industry-specific terminology and cultural movements. – This reinterpretation is typically influenced by their academic learning, recent literature, and biases towards larger enterprise perspectives. As a result, the original intent and practical effectiveness of these concepts, such as DevOps or Platform Engineering for example, may be diluted or misrepresented. ...

November 13, 2023 · 3 min · 560 words · Sam McLeod

SDXL LoRA Training

A talk I gave to some peers on creating your own SDXL LoRA models from my tinkering around over the last few weeks. ...

October 30, 2023 · 1 min · 37 words · Sam McLeod

Fixing AMD CPU Scaling on Fedora

Recently, after replacing my home server I noticed that the CPU (Ryzen 7600) was only scaling between 3000MHz and 3800MHz, which is the base and the first level boost clock of the CPU. I was expecting it to scale down to as low as 400Mhz when idle, and up to 5.17Ghz on boost. ...

July 9, 2023 · 4 min · 710 words · Sam McLeod

Defaulting git clone to shallow (depth=1)

Before adding this to my shell config, I would manually add –depth=1 to all my git clones. ...

May 29, 2023 · 1 min · 176 words · Sam McLeod

How likely would you be to block a company from asking you to rate everything they do?

I’m tired of getting emails from companies asking me to rate their products and services It feels like you can’t even buy a coffee these days without being asked if you would recommend the coffee shop to a friend or colleague. ...

May 26, 2023 · 2 min · 422 words · Sam McLeod

Escaping the Time Trap: Why Estimating Effort, Not Time, Leads to Greater Success

Many organisations - while branding themselves as “Agile” - continue to make the mistake of estimating project velocity based on time than effort. ...

April 28, 2023 · 8 min · 1681 words · Sam McLeod

Open Budgeting - Subscriptions & Licences


January 5, 2023 · 3 min · 471 words · Sam McLeod

Energy, Sustainability and Deployment Frequency

Deployment / Delivery Frequency I often end up needing to advocate for more frequent delivery/deployments with clients. There’s the usual benefits commonly discussed such as improved feedback, reduced risk, well understood processes, maintainable dependencies etc… however what’s often missed entirely is how it relates to the health and sustainability of the team. ...

December 31, 2022 · 2 min · 373 words · Sam McLeod

Zsh Configuration and Plugins - Part Two


November 18, 2022 · 12 min · 2435 words · Sam McLeod

Github Not-So-Reusable Actions

Github Actions is a “BYOBE” (Bring Your Own Bloody Everything) offering that provides basic CI with surprisingly convoluted configuration to Github. The product as a whole is an exercise in frustration, one of the worst parts is the lack of reusability and the complexity required to achieve it. Github’s concept of reusable workflows on Github Actions is clearly a cobbled together afterthought. ...

November 6, 2022 · 3 min · 615 words · Sam McLeod

YAML Anchors and Aliases

The YAML Spec defined a simple way to reuse values in YAML documents by means of Anchors and Aliases. ...

November 6, 2022 · 2 min · 305 words · Sam McLeod

Zsh Configuration and Plugins - Part One

As of 2022, I’ve been using zsh as my primary shell for 14 years. Over that time I’ve experimented with and accumulated a lot of configuration, scripts, hacks, plugins and themes. I’ve settled on a configuration that I’m mostly happy with, over this series I’m going to share my current setup. ...

November 6, 2022 · 4 min · 730 words · Sam McLeod

Encourage Your Peers To Contribute To Open Source

Contributing to Open Source is important to the quality and maintainability of the software and engineering communities we rely on every day - so why is it that so many developers/engineers never participate? ...

October 30, 2022 · 3 min · 535 words · Sam McLeod

Making Work Visible - Avoid DMs

We create more value by having conversations in public instead of behind closed doors. ...

October 20, 2022 · 6 min · 1071 words · Sam McLeod

The Best Of - 2022 Edition

Near the end of each year I note down a summary of the best apps I’ve enjoyed using throughout the year, here’s 2022. ...

October 19, 2022 · 2 min · 340 words · Sam McLeod

It's 2022 and we're (still) not deploying enough

We’re (still) not deploying enough It’s 2022 and not deploying frequently enough is still one of the most common causes of software failure. ...

August 23, 2022 · 6 min · 1155 words · Sam McLeod

DALL·E 2 - Selected works

Over the past month I’ve been playing with OpenAI’s DALL·E 2, below are some of the interesting images I’ve generated. “DALL·E 2 is a new AI system that can create realistic images and art from a description in natural language.” “A renaissance oil painting of two developers arguing over which javascript framework is the worst” “A Ralph Steadman painting of 7 cats sitting on a fence, and one raven” “A poster of Bruce Lee posing in front of a wall painted with traditional Japanese dragon artwork” “An Art Nouveau painting of Batman outside a church” “Old man yells at cloud” Variations on “A romantic oil painting of a chicken wearing beats by dre and smoking a cigarette” “A Romantic oil painting of a chicken wearing ripped jeans, aviator sunglasses, smoking a cigarette leaning up against a wall” “A romantic painting of snoop dog and a cool chicken wearing ripped jeans and aviators standing in a recording booth rapping into a microphone” “A dutch masters style oil painting of the rapper snoop dogg rapping into a microphone along side a cool looking animal wearing ripped jeans” “An oil painting of the rapper snoop dogg as a marvel super hero whose super power is smoking” “A romantic style oil painting of the rapper snoop dogg as a marvel super hero whose super power is smoking” “A detailed psychedelic painting of a person listening to the band TOOL for the first time and having their third eye opened” “A white Romantic Sculpture of the dude from the film the big lebowski” “A dutch masters oil painting of a robot cat” “A red Tesla model 3 crashed on the side of the road due to it’s poorly designed suspension” “A kiwi walking over hot coals” “A Ralph Steadman style ink painting of cats sitting on a fence smoking” “Two chickens playing chess while smoking” “Cats get dressed up to go to the theatre” “A grumpy cat drinking black coffee and smoking a cigarette” “A female artist with brown and grey hair hangs white deer antlers in an art gallery” “A black and white cat and a boarder collie dog in the style of Grayson Perry” “A lonely cat by a window in a Ralph Steadman style” “A post-apocalyptic city at night ruled by cats” “A border collie dog and black and white cat in the style of Gustav Klimt” “DevOps hipsters deploy to the cloud” “A psychedelic monkey smoking” “Cats smoking cigars and playing poker” “An elite hacker breaks into the computer mainframe, his eyes are red, his beard is long, he is in a dark room at his desk” “Cats captaining space ships with lasers” “Two lamas duelling with lightsabers at a disco” “A dark, detailed oil painting of a post-apocalyptic city a night, two terminator cyborgs are patrolling the street” “A photo of a beautiful British field in the summer, there are butterflys, birds and a small pond....

July 8, 2022 · 4 min · 751 words · Sam McLeod

Beginner Harmonica Practice Exercises

Beginner Harmonica Practice Exercises Way back in around 2008 I wrote up a doc with a list of beginner harp exercises and practice riffs, at the time it was uploaded to a few forums and a shared Google doc - both have since disappeared. To this day I still get regular emails asking for a copy of it so I thought I’d copy it to a post here. Note: I haven’t updated these since around 2011....

June 13, 2022 · 13 min · 2599 words · Sam McLeod

Firefox Addons for 2022

Firefox Addons - 2022 Edition My list of must-have Firefox addons - 2022 edition Updated: 2022-11-07 Privacy and Security Firstly - you should have Firefox’s Enhanced Tracking Protection enabled. Don’t Track Me Google LocalCDN UTM Tracking Token Stripper Note: You can accomplish some of what this does by setting up the removeparam uBlock origin rules I’ve listed below. Multi-Account Containers Useful for setting sites such as Amazon, eBay, Twitter, LinkedIn, Banking etc… each to always open in their own isolated container....

May 17, 2022 · 3 min · 442 words · Sam McLeod

Goodbye Evernote, Hello Bear

Until recently I used Evernote for my notes, web clippings and document drafts, I was a subscriber going on 14 years and had over 3000 notes. However in recent years Evernote has gone downhill to the point it could be considered hostile to it’s users. The native macOS app was replaced with an Electron Javascript webframe and like almost all Electron apps I’ve tried it was plagued by poor performance, memory leaks and aggressive idle CPU utilisation and above all else it now lacks the low latency feeling the native application had....

July 22, 2021 · 2 min · 277 words · Sam McLeod

Rancilio Silvia Upgrade

Weekend upgrades to my Ranchilio Silvia v4 espresso machine. Added a digital PID kit for improved temperature control Replaced the boiler element Thermal insulation Basic “smart” power control with Siri integration Upgrade the standard Ranchilio basket to a VST 18g Ridged Rebuild Photos Note the damage to the boiler elements terminals, this was shorting and causing the houses RCD to trip Wiring the PID controller Replacing the boiler thermostat with one connected to the PID controller...

July 11, 2021 · 1 min · 83 words · Sam McLeod

DevOps Team Charter

A template team charter for software and platform engineering teams. Research1 shows there is great value in people embracing a shared vision and ideas that are bigger than themselves. When it comes to a team charter, it’s important to remember that it’s not a static document. It’s a living document that should be updated as the team evolves and grows. Keep in mind that One size doesn’t fit all. Consider your (teams) mission, cultural background and values....

November 18, 2020 · 5 min · 1046 words · Sam McLeod

Ferrari F12 Berlinetta

Last weekend I had a chance encounter with a Ferrari F12 Berlinetta and the owner kindly let me jump in and take a few photos. Specs 6.3L v12 544 KW (740 Horsepower) 690 Nm at 6000RPM 1525 kg 0-100 in 3.1s 0-200 in 8.5s Maximum speed: ‘over 340 km/h’ Was ‘The fastest road car Ferrari ever built’ Photos

November 8, 2020 · 1 min · 58 words · Sam McLeod

The Best Of - 2020 Edition

Near the end of every year I note down a summary of the best apps, hardware & podcasts I’ve enjoyed throughout the year (and often for some time before). This post has been superseded. You can find the latest version of this post here. Software and Services Plex Things Pixelmator Pro Reeder NextMeeting Bumpr Fastmail 1Password Calibre Backblaze iTerm Gitlab PFsense Stay Onyx Evernote Handbrake Amphetamine Firefox Software - Mobile Plex Things Reeder Overcast Evernote MiniHack Alien Blue Singal Prompt Browser Addons uBlock Origin Dencentraleyes Hardware LG C9 OLED TV Das Ultimate 4 Logitech MX Master 3 16’’ Macbook Pro iPhone 12 Pro AppleTV 4K Audioquest DragonFly Red USB DAC Ultrasone Signature Pro Headphones Kindle Paperwhite All Day Socks Bellroy Wallets Podcasts The Skeptics Guide To The Universion (SGU) You Are Not So Smart Risky....

October 10, 2020 · 1 min · 213 words · Sam McLeod

Rest in Peace Jimothy

You magnificent girl, you brought me 14 years of love and happiness Photos

October 8, 2020 · 1 min · 13 words · Sam McLeod

Camels Dressed As Unicorns

Stop trying to hire with titles like ‘DevOps Engineer’ or ‘Cloud Engineer’ “DevOps is … not a job title” “It’s more of a cultural practice, like innovation, and it makes just as little sense hiring “innovation engineers” and expecting your organisation to be innovative, without also creating the culture to foster innovation.” (Joel Shea when responding to this posted on LinkedIn) In most cases it’s clear the organisation doesn’t truly know what they want or need and likely don’t understand the nuances of the aspects of engineering....

August 8, 2019 · 2 min · 328 words · Sam McLeod

Looking For New Opportunities

As stated in my announcement post - after 7~ years, I resigned from Infoxchange several months ago. I planned on taking a holiday, relaxing, decompressing, dipping my toes in some off-the-shelf tech I haven’t spent a lot of time with and performing some personal growth through reading a number of books that have been sitting on my ’to read’ shelf. This is exactly what I’ve done. I’m now open to new opportunities I’d rather find a workplace that I can add great value too and be happy to work with than rush into a new role quickly, obviously if something great turns up right away - I’ll pursue it....

August 6, 2019 · 4 min · 763 words · Sam McLeod

Leaving Infoxchange

After 7~ years, I have resigned from Infoxchange. In 2012 I was hired to work with Infoxchange to “shake things up” (in the Systems Operations team) and stabilise the hosting environment. As Team Lead of Systems Operations I was tasked with melding the team and then for us to work to provide a stable, robust, modern and scalable platform for Infoxchange developed application hosting and product delivery and in a short time, with a small budget - we did just that....

June 29, 2019 · 3 min · 470 words · Sam McLeod

Goodbye XenSever - Hello XCP-ng

In 2018 I set out to replace our XenSever 7.2 based virtualisation after Citrix essentially screwed over free / open source users. This project was to directly replace XenServer 7.2 with something supported and manageable for our traditional virtualisation needs. High Level Selection Considerations I evaluated a number of options, with the primary candidates below. Key criteria (at a high level) I was evaluating: Ease of moving from our existing XenServer 7....

February 5, 2019 · 8 min · 1550 words · Sam McLeod

New Zealand Otago Trip in a 2019 Mustang GT

In late November of 2018 I was lucky enough to get my hands on the upcoming 2019 Sport spec Mustang GT v8 to be its first driver. I’m not usually an American Muscle enthusiast, but the new 2019 GT really is quite different from its slightly older counterparts, with a new specially developed 10 speed gearbox (yes, 10 gears - because more is better right?), tuned up suspension and so seemed like it would make for a pretty fun trip....

December 20, 2018 · 1 min · 207 words · Sam McLeod

Run youtube-dl (or similar) in the background

I wanted an alias or function to use youtube-dl in the background. Looking around the web lots of people seemed to want this and most of them were banging their heads against a wall due to: A) bash quoting B) backgrounding dying when their terminal closed C) passing the argument (url in my case) to the function Here’s a simple function I whipped up that seems to ‘just works’™: function yt { nohup youtube-dl "$1" --no-progress 2>&1 > youtube-dl-"$(date +%Y%m%d-%H%M%S)"....

July 30, 2018 · 2 min · 231 words · Sam McLeod

Disabling scroll-wheel zoom in Firefox

This feature annoys me endlessly, I end up zoomed in and out of websites all over the internet. … But the fix is easy and there’s no addons required. Navigate to about:config (in Firefox’s URL bar) Change the value of the following two properties to 0: mousewheel.with_control.action mousewheel.with_meta.action If you use Firefox sync and want these settings to sync between your machines, also add the following properties: Create two new properties both of type boolean and set them to true:...

July 16, 2018 · 1 min · 83 words · Sam McLeod

Highlands Festival of Speed 2018

Last weekend I headed to New Zealand and marshalled at the Highlands Festival of Speed. The Highlands Festival of Speed is a weekend packed with some of the best modern classic racing you will see in New Zealand. Featuring the Pre 65 Racing, Mainland Muscle, Formula Libre, South Island Porsche, OSCA, Highlands Modern Classics/Nostalgic Classics and the Highlands Sprint Series. This is a weekend of true blue Kiwi racing starring some of New Zealand’s most well known racing legends....

April 11, 2018 · 1 min · 140 words · Sam McLeod

Flash Storage and SSD Failure Rate Update (March 2018)

It was almost 3 years ago that my open source storage project went into production. In that time it’s been running 24/7 serving as highly available solid state storage for hundreds of VMs and several virtualisation clusters across our two main sites. I’m happy to report that the clusters have been operating very successfully since their conception. Since moving away from proprietary ‘black box’ vendor SANs, we haven’t had a single SAN issue, storage outage....

March 20, 2018 · 2 min · 326 words · Sam McLeod

Theory of constraints

“A chain is no stronger than its weakest link” Any improvement made anywhere besides the bottleneck is an illusion. Any improvement made after the bottleneck is useless because it will always remain starved waiting for work from the bottleneck. Any improvement made before the bottleneck merely results in more ‘work’ piling up at the bottleneck. Identify the system’s constraint(s) (that which prevents the organisation from obtaining more of the goal in a unit of time) Decide how to exploit the system’s constraint(s) (how to get the most out of the constraint) Subordinate everything else to the above decision (align the whole system or organization to support the decision made above) Elevate the system’s constraint(s) (make other major changes needed to increase the constraint’s capacity) Warning!...

January 17, 2018 · 1 min · 159 words · Sam McLeod

Top album picks for 2017

Here are my top album pics for 2017 (in no particular order) Note: This is by no means an exhaustive list, it’s just the top albums that really stood out to me and in all fairness, I’ve thought of several others since so there may be a follow up post (or two). Father John Misty - Pure Comedy As a bonus, a short film was released with the album: From Wikipedia:...

December 28, 2017 · 4 min · 641 words · Sam McLeod

HP 4951C Protocol Analyser

My good friend Joel Shea received a most unlikely gift this Christmas - A vintage HP 4951 Protocol Analyser. According to the HP Computer Museum: Original Price: $3595 The 4951B was replaced by the 4951C and 4952A in 1986. Both new models handled Async, BSC, SDLC, HDLC, X.25 and SNA protocols. The 4951C also handled DDCMP, while the 4952A did not. The 4952A handled X.21 while the 4951C did not. Both new analysers used a floppy dive (618 KB) for removable media....

December 27, 2017 · 1 min · 91 words · Sam McLeod

Talk - Clustered, Distributed File and Volume Storage with GlusterFS

Using GlusterFS to provide volume storage to Kubernetes as a replacement for our existing file and static content hosting. This talk was given at Infracoders on Tuesday 14th November 2017. NOTE: Below link to slides currently broken - will fix soon! (03/08/2019) Click below to view slides (PDF version): Direct download link

November 14, 2017 · 1 min · 52 words · Sam McLeod

Applying syntax in Sublime based on the first file line

In vim, you can add a comment at the top of files to set the syntax, e.g.: # vim: syntax=ruby In SublimeText there are many ways to detect syntax, one interesting approach I’ve recently found useful is to match on the top line in the file. For example, with Puppet there is a file called Puppetfile, it has no extension but it’s really Ruby syntax, so it’s useful to add linting incase you miss something simple like a , and break deployments....

October 25, 2017 · 1 min · 162 words · Sam McLeod

Broadcom, Or How I Learned To Start Worrying And Drop The Packet

Earlier this week we started the process to upgrade one of our hypervisor compute clusters when we encountered a rather painful bug with HP’s Broadcom NIC chipsets. We were part way through a routine rolling pool upgrade of our hypervisor (XenServer) cluster when we observed unexpected and intermittent loss of connectivity between several VMs, then entire XenServer hosts. The problems appeared to impact hosts that hadn’t yet upgraded to XenServer 7....

October 13, 2017 · 6 min · 1240 words · Sam McLeod


We’re in the process of shifting from using our custom ‘glue’ for orchestrating Docker deployments to Kubernetes, When we first deployed Docker to replace LXC and our legacy Puppet-heavy application configuration and deployment systems there really wasn’t any existing tool to manage this, thus we rolled our own, mainly a few Ruby scripts combined with a Puppet / Hiera / Mcollective driven workflow. The main objective is to replace our legacy NFS file servers used to host uploads / attachments and static files for our web applications, while NFS(v4) performance is adequate, it is a clear single point of failure and of course, there are the age old stale mount problems should network interruptions occur....

September 25, 2017 · 6 min · 1106 words · Sam McLeod

Return Of The RSS

Of all the tools for reading news and subscribing to software releases, I still find RSS the most useful. I use Feedly to manage my rss subscriptions and keep all my devices in sync, but instead of using the Feedly’s own client, I use an app called Reeder as the client / reader itself. Link: My Feedly RSS Feed Feedly RSS feed subscription management Features: Keyword alerts. Browser plugins to subscribe to (current) url....

September 22, 2017 · 2 min · 283 words · Sam McLeod

MH-Z19 CO2 sensor reader, logger and visualiser

MH-Z19 CO2 sensor reader, logger and visualiser Reads data from UART(serial)-connected MH-Z19 (or MH-Z14) sensor using python 3. If you dare to install nodejs you can visualise the logged data (using html and plotly.js library). Repository: sammcj/CO2-Logger Usage Note this post is from 2016, in 2021 I replaced my custom Co2 loggers with an Aranet4. While very expensive, is and excellent off-the-shelf solution, with many features. Connection Sensor can be queried using 3....

December 21, 2016 · 3 min · 552 words · Sam McLeod

Castle Hill Winter Trip 2016 Photos

August 6, 2016 · 0 min · 0 words · Sam McLeod

The State of Android in 2016 & The OnePlus 3 Phone

I wanted to try Android for a couple of weeks, I like staying on top of technology, gadgets and making sure I never become a blind ‘zealot’ for any platform or brand. The OnePlus 3 I did a lot of research and decided to try the “Oneplus 3” as it was good bang-for-buck, ran the latest software had plenty of grunt with the latest 8 core, high clock speed Qualcomm processor coupled with 6GB of DDR4 - the specs really are very impressive, especially for a $400USD phone....

July 11, 2016 · 4 min · 721 words · Sam McLeod

Update Delayed Serial STONITH Design

note: This is a follow up post from 2015-07-21-rcd-stonith A Linux Cluster Base STONITH provider for use with modern Pacemaker clusters This has since been accepted and merged into Fedora’s code base and as such will make it’s way to RHEL. Source Code: Github Diptrace CAD Design: Github I have open sourced the CAD circuit design and made this available within this repo under CAD Design and Schematics Related RedHat Bug: https://bugzilla....

July 4, 2016 · 2 min · 217 words · Sam McLeod

Monitoring SystemD Units With Nagios

Ever forgotten to add a critical service to monitoring? Want to know if a service or process fails without explicitly monitoring every service on a host? …Then why not use SystemD’s existing knowledge of all the enabled services? Thanks to ‘Kbyte’ who made a simple Nagios plugin to do just this! Requirements Python3 (For RHEL/CentOS 7 yum install python34) python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin PyNagSystemD

May 23, 2016 · 1 min · 68 words · Sam McLeod

Online Conversion from SQL_ASCII to UTF8 in PostgreSQL

Scripts and source available here: sql_ascii_to_utf8 The Goal To be able to take a Postgres Database which is in SQL_ASCII encoding, and import it into a UTF8 encoded database. Requirements: Python3 (For RHEL/CentOS 7 yum install python34) python-nagiosplugin My pre-built RPMs or pip3 install nagiosplugin PyNagSystemD The Problem PostreSQL will generate errors like this if it encounters any non-UTF8 byte-sequences during a database restore: # pg_dump -Fc test_badchar | pg_restore -d test_badchar_utf8 pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 2839; 0 26852 TABLE DATA table101 postgres pg_restore: [archiver (db)] COPY failed for table "table101": ERROR: invalid byte sequence for encoding "UTF8": 0x91 CONTEXT: COPY table101, line 1 WARNING: errors ignored on restore: 1 And the corresponding data will be omitted from the database (in this case, the whole table, even the rows which did not have a problem):...

May 23, 2016 · 5 min · 902 words · Sam McLeod

Speeding Up rsync

The most common way to use rsync is probably as such: rsync -avr user@<source>:<source_dir> <dest_dir> Resulting in 30-35MB/s depending on file sizes This can be improved by using a more efficient, less secure encryption algorithm, disabling compression and telling the SSH client to disable some unneeded features that slow things down. With the settings below I have achieved 100MB/s (at work between VMs) and over 300MB/s at home between SSD drives....

May 3, 2016 · 2 min · 224 words · Sam McLeod

Benchmarking IO with FIO

This is a quick tldr there are many other situations and options you could consider FIO man page IOP/s = Input or Output operations per second Throughput = How many MB/s can you read/write continuously Variables worth tuning based on your situation --iodepth The iodepth is very dependant on your hardware. Rotational drives without much cache and high latency (i.e. desktop SATA drives) will not benefit from a large iodepth, Values between 16 to 64 could be sensible....

April 29, 2016 · 2 min · 393 words · Sam McLeod

Mirroring a Gitlab project to Github

Let’s pretend you have a project on Gitlab called ask-izzy and you want to mirror it up to Gitlab which is located at Assuming you’re running Gitlab as the default user of git and that your repositories are stored in /mnt/repositories you can following something similar to the following instructions: Grant write access to Github Get your Gitlab install’s pubkey from the git user cat /home/git/.ssh/ On Github add this pubkey as deploy key on the repo, make sure you tick the option to allow write access....

February 4, 2016 · 1 min · 145 words · Sam McLeod


Today we launched a mobile website for homeless people … and it was launched by one of Australia’s many recent Prime Ministers Today alone we served up over 87,000 requests As many of you know, I work with Infoxchange as the operations lead. When I first heard the idea of a website or app for people that have found or are worried about finding themselves homeless in Australia I really didn’t think it made sense - until I saw the stats showing how many homeless people in Australia have regular access to a smart phone and data either via a cellular provider or free WiFi....

January 29, 2016 · 1 min · 206 words · Sam McLeod

Fix XenServer SR with corrupt or invalid metadata

If a disk / VDI is orphaned or only partially deleted you’ll notice that under the SR it’s not assigned to any VM. This can cause issues that look like metadata corruption resulting in the inability to migrate VMs or edit storage. For example: [root@xenserver-host ~]# xe vdi-destroy uuid=6c2cd848-ac0e-441c-9cd6-9865fca7fe8b Error code: SR_BACKEND_FAILURE_181 Error parameters: , Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters: /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/MGT, 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b. Error: Failed to write file with params [3, 0, 512, 512]....

January 18, 2016 · 2 min · 296 words · Sam McLeod

iSCSI SCSI-ID / Serial Persistence

“Having a SCSI ID is a f*cking idiotic thing to do.” - Linus Torvalds …and after the amount of time I’ve wasted getting XenServer to play nicely with LIO iSCSI failover I tend to agree. The Problem One oddity of Xen / XenServer’s storage subsystem is that it identifies iSCSI storage repositories via a calculated SCSI ID rather than the iSCSI Serial - which would be the sane thing to do....

December 14, 2015 · 3 min · 626 words · Sam McLeod

How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync

Slides Failover Demo

November 9, 2015 · 1 min · 3 words · Sam McLeod

SAN Intro

October 7, 2015 · 0 min · 0 words · Sam McLeod

Replacing Junos Pulse with OpenConnect

In an attempt to avoid using the Juniper Pulse (Now Pulse Secure) VPN client we tried OpenConnect but found that DNS did not work correctly when connected to the VPN. This bug has now been resolved recently but has not made it’s way into a new build, in fact there have been no releases for 6 months. Luckily the OpenConnect was not too difficult to build from source. Build OpenConnect on OSX Remove old openconnect and install deps brew remove openconnect brew install libxml2 lzlib openssl libtool libevent Build openconnect wget git....

September 22, 2015 · 1 min · 137 words · Sam McLeod

SSD Storage - Two Months In Production

Over the last two months I’ve been running selected IO intensive servers off the the SSD storage cluster, these hosts include (among others) our: Primary Puppetmaster Gitlab server Redmine app and database servers Nagios servers Several Docker database host servers Reliability We haven’t had any software or hardware failures since commissioning the storage units. During this time we have had 3 disk failures on our HP StoreVirtual SANs that have required us to call the supporting vendor and replace failed disks....

September 13, 2015 · 2 min · 376 words · Sam McLeod

OS X Software Update Channels For Betas

Set update channel to receive developer beta update sudo softwareupdate --set-catalog Set update channel to receive public beta update sudo softwareupdate --set-catalog List available updates sudo softwareupdate --list Set update channel to receive default, stable updates sudo softwareupdate --clear-catalog Show current settings defaults read /Library/Preferences/ Write setting manually defaults write /Library/Preferences/ CatalogURL

September 1, 2015 · 1 min · 55 words · Sam McLeod

iSCSI Benchmarking

The following are benchmarks from our testings of our iSCSI SSD storage. 67,300 read IOP/s on a VM on iSCSI (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM) Per VM and scales to 1,000,000 IOP/s total root@dev-samm:/mnt/pmt1 128 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=read test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128 2.0.8 Starting 1 process bs: 1 (f=1): [R] [55....

July 24, 2015 · 3 min · 456 words · Sam McLeod

Delayed Serial STONITH

A modified version of John Sutton’s rcd_serial cable coupled with our Supermicro reset switch hijacker: This works with the rcd_serial fence agent plugin. Reasons rcd_serial makes for a very good STONITH mechanism: It has no dependency on power state. It has no dependency on network state. It has no dependency on node operational state. It has no dependency on external hardware. It costs less that $5 + time to build. It is incredibly simple and reliable....

July 21, 2015 · 3 min · 450 words · Sam McLeod

Video - Cluster Failover Performance Demo

July 12, 2015 · 0 min · 0 words · Sam McLeod

CentOS 7 and HA

First some background… One of the many lessons I’ve learnt from my Linux HA / Storage clustering project is that the Debian HA ecosystem is essentially broken, We reached the point where packages were too old, too buggy or in Debian 8’s case - outright missing. In the past I was very disappointed with RHEL/CentOS 5 / 6 and (until now) have been quite satisfied with Debian as a stable server distribution with historicity more modern packages and kernels....

July 7, 2015 · 3 min · 558 words · Sam McLeod

SSD Storage Cluster - Update and Diagram

Due to several recent events beyond my control I’m a bit behind on the project - hence the lack of updates which I apologise for. The goods news is that I’m back working to finish off the clusters and I’m happy to report that all is going to plan. Here is the final digram of the two-node cluster design: Plain text version available here This was generated from the LCMC tool (beware - it’s java!...

June 17, 2015 · 1 min · 79 words · Sam McLeod

Video - Storage Cluster Failover Demo

A brief demonstration of the failover and recovery process on the storage clusters I’ve been building.

May 14, 2015 · 1 min · 16 words · Sam McLeod

Talk - High Performance Software Defined Storage

A high level talk from Infracoders Melbourne on 12/04/2015. There’s also a low quality recording available here: Related posts: Building a high performance SSD SAN - Part 1

April 15, 2015 · 1 min · 28 words · Sam McLeod

Continuous integration for the Linux Kernel - Built within Docker

Linux Kernel CI for Debian Github: sammcj/kernel-ci Those of us using technologies such as Docker and BTRFS or simply trying to gain a performance edge on the competition have a lot to gain from the features and performance of recent Kernel updates (especially from 3.18 onwards). ‘Enterprise’ Linux distributions such as RHEL & variants are concerningly out of date when comes to the Kernel. Many people seem to have forgotten what Linux is… Linux IS the Kernel....

March 15, 2015 · 2 min · 310 words · Sam McLeod

Xen Orchestra Docker Image

Docker config to setup XO which is a web interface to visualize and administrate your XenServer (or XAPI enabled) hosts Github: sammcj/docker-xen-orchestra Running the app Updates are pushed to the Docker Hub’s automated build service: From Docker Hub docker pull sammcj/docker-xen-orchestra docker run -d -p 8000:80 sammcj/docker-xen-orchestra Building git clone cd docker-xen-orchestra # Edit whatever config you want to change docker build -t xen-orchestra . See for information on Xen Orchestra

February 26, 2015 · 1 min · 74 words · Sam McLeod

Delete Government-Linked Certificate Authorities in OSX

Inspired by Source (Github) WARNINGS Do not run unless you understand what this is doing The CA system is broken by design - This is not a fix for that This is merely a band-aid for those interested or concerned about these root CAs Usage chmod +x ./ You’ll be prompted for your password as root access is required to delete system-wide root certs. See Also https://github....

February 23, 2015 · 1 min · 75 words · Sam McLeod

Building a high performance SSD SAN - Part 1

Over the coming month I will be architecting, building and testing a modular, high performance SSD-only storage solution. I’ll be documenting my progress / findings along the way and open sourcing all the information as a public guide. With recent price drops and durability improvements in solid state storage now is better time than any to ditch those old magnets. Modular server manufacturers such as SuperMicro have spent large on R&D thanks to the ever growing requirements from cloud vendors that utilise their hardware....

February 16, 2015 · 8 min · 1590 words · Sam McLeod

Direct-Attach SSD Storage - Performance & Comparisons

Further to my earlier post on XenServer storage performance with regards to directly attaching storage from the host, I have been analysing the performance of various SSD storage options. I have attached a HP DS2220sb storage blade to an existing server blade and compared performance with 4 and 6 SSD RAID-10 to our existing iSCSI SANs. While the P420i RAID controller in the DS2220sb is clearly saturated and unable to provide throughput much over 1,100MB/s - the IOP/s available to PostgreSQL are still a very considerably performance improvement over our P4530 SAN - in fact, 6 SSD’s result in a 39....

February 15, 2015 · 1 min · 110 words · Sam McLeod

Search - A Journey of Delivery on a Budget

“Search - A Journey of Delivery on a Budget” Presented a Melbourne Search - July 2014 - “Search - A Journey of Delivery on a Budget” (Click for slides)

February 15, 2015 · 1 min · 29 words · Sam McLeod

Talk - 24 Months

The way we work at Infoxchange has changed greatly. A retrospective journey into transforming Infoxchange’s technology and culture over the past 24 months - presented a Melbourne DevOps - December 2014

February 15, 2015 · 1 min · 31 words · Sam McLeod

The Best Of - 2014 Edition

At the end of every year I note down a summary of the best applications, hardware & websites I’ve enjoyed & depended on throughout the year (and often for some time before). This post has long since been superseded. You can find the latest version of this post here. Software / General Use Fastmail - Evernote - Reeder - Keynote - Lastpass - Plex - https://plex....

February 15, 2015 · 2 min · 426 words · Sam McLeod

XenServer, SSDs & VM Storage Performance

Intro At Infoxchange we use XenServer as our Virtualisation of choice. There are many reasons for this including: Open Source. Offers greater performance than VMware. Affordability (it’s free unless you purchase support). Proven backend Xen is very reliable. Reliable cross-host migrations of VMs. The XenCentre client, (although having to run in a Windows VM) is quick and simple to use. Upgrades and patches have proven to be more reliable than VMware....

February 15, 2015 · 5 min · 970 words · Sam McLeod

BMW E36 328i Manual Conversion Programming

Reprogram BMW after doing Automatic to Manual Conversion Based on my E36 328is experience, written in 2010. Assumptions This guide assumes the following: You have programmed a BMW ECU/DME before using DIS or similar. You are taking all the necessary safety precautions. (Fully charged battery etc…) You have DIS working (We used EasyDIS 1.0, Base 44) You have Ediabas (INPA, NCS Expert, IFH Serve) installed and working. You’re able to read between the lines of our crappy document, some steps may differ slightly and our wording may be somewhat imprecise....

June 10, 2010 · 6 min · 1081 words · Sam McLeod