# posts - [OpenSpeccer - OpenSpec Dashboard](https://smcleod.net/2026/10/openspeccer-openspec-dashboard/index.md): OpenSpeccer - OpenSpec Dashboard --- title: "OpenSpeccer - OpenSpec Dashboard" date: 2026-10-10T23:00:00+10:00 tags: ["AI", "LLM", "agentic coding", "agents", "skills", "claude code", "openspec", "spec driven development"] author: "Sam McLeod" draft: false aliases: - /2026/10/openspeccer-a-read-only-openspec-dashboard/ description: "A lightweight dashboard for OpenSpec projects, shipped as an Agent Skill." searchHidden: false cover: image: "dashboard.png" alt: "OpenSpeccer dashboard showing spec counts, a kanban of changes and specs in flight" hidden: true --- [OpenSpeccer](https://github.com/sammcj/agentic-coding/tree/main/Skills/openspeccer) is a read-only web dashboard for any repository using [OpenSpec](https://github.com/Fission-AI/OpenSpec), showing what's in flight, how far along each change is, and what every spec currently says. It ships as an Agent Skill, runs on Python's standard library and needs no install or build step. {{< wide-image src="dashboard.png" alt="OpenSpeccer dashboard" >}} ## Why With OpenSpec the state of a project lives in a pile of markdown files spread across `openspec/changes/`. When I've had several agents working in separate git worktrees, each holding its own copy of a change, working out what's actually happening means opening a lot of files in a lot of directories. I wanted one page I could glance at, that updates as the agents work, and that never writes anything. ## Getting started 1. Copy [`Skills/openspeccer`](https://github.com/sammcj/agentic-coding/tree/main/Skills/openspeccer) into your agent's skills directory (`~/.claude/skills/` for Claude Code). 2. Ask your agent to open the OpenSpec dashboard, or run it yourself: ```shell ~/.claude/skills/openspeccer/scripts/serve.py ~/git/my-project ``` With no path it searches upward from the current directory for `openspec/`. It serves on port 4380 (or the next free port) and opens your browser. Running it again for the same repo reuses the server that's already up. The OpenSpec CLI is optional. Without it, everything except the Schemas and Commands pages still works. ## A tour ### Dashboard The dashboard is a kanban of active changes by stage: proposed, planned, in progress, ready to archive, and recently archived. Across the top are spec and requirement counts, task progress across active changes, average lifecycle, and how many active changes have gone untouched for 30 days. Below the board, _Specs in flight_ lists every spec an active change touches, and which changes touch it. When worktrees hold different copies of the same change, its card shows a "versions" badge. ### Changes Each change opens with one tab per artifact: tasks, specs, design and proposal. Sort the tabs by modified time, schema order or name. The tasks tab shows progress per section with the checklist rendered as markdown. {{< wide-image src="changes.png" alt="A change's tasks tab with per-section progress" >}} The specs tab renders each delta spec as requirement and scenario cards under their ADDED or MODIFIED heading, with a link to the main spec it changes. {{< wide-image src="change-specs.png" alt="A change's specs tab showing added requirements as cards" >}} The proposal and design tabs render as markdown with an outline down the side. {{< wide-image src="change-proposal.png" alt="A change's proposal tab" >}} {{< wide-image src="change-design.png" alt="A change's design tab with its outline" >}} The change list shows active and archived changes together, with active ones first. ### Specs Specs show as a filterable tree, with a dot on any spec an active change is modifying. Each spec is broken into requirement cards with their scenarios, BDD keywords highlighted and an outline down the side. {{< wide-image src="specs.png" alt="A spec broken into requirement and scenario cards" >}} The _History_ tab lists every change with a delta for the spec, newest first. _Show delta_ expands that change's delta in place. {{< wide-image src="spec-history.png" alt="A spec's history of changes with deltas" >}} _Diff_ compares the spec between any two git revisions, or a revision and the working copy. {{< wide-image src="spec-diff.png" alt="A spec diffed between a past revision and the working copy" >}} ### Timeline The timeline has a burndown up top: - Open changes now, plus changes opened and archived per week over the last four weeks - A projected date for clearing the open changes, or "Not burning down" when they're opening as fast as they're archiving - A step chart of open changes over time, and a bar chart of changes opened (up) and archived (down) per week Below that, each change's lifecycle is a bar from creation to archive, coloured by stage, and can be grouped by spec. {{< wide-image src="timeline.png" alt="Timeline with burndown stats, open changes chart and change lifecycles" >}} ### Schemas Schemas are drawn as a workflow graph from their `requires` declarations, so you can see which artifacts depend on which before implementation starts. Edges a longer path already implies are left out to keep the graph readable. {{< wide-image src="schema.png" alt="The spec-driven schema as a workflow graph" >}} ### Agents The Agents page lists each coding agent in the project and which OpenSpec skills or commands it has installed. {{< wide-image src="agents.png" alt="Agents with OpenSpec skills and commands installed" >}} ### Commands A reference for every command and option the installed OpenSpec CLI has, read from its own `--help` output so it matches your version. Click a command to copy it. There are links to the official docs at the top. {{< wide-image src="commands.png" alt="Commands reference built from the installed OpenSpec CLI" >}} ### Export Specs, changes and the timeline each have an Export menu: save as Markdown, copy as Markdown, or save the page as a standalone HTML file. A change exports every artifact verbatim, a quick way to hand a change to someone who doesn't have the repo. ## Without a browser Agents can use it too. When I ask "what's in flight?", the agent runs it with `--summary` and answers from the output, without starting a server: ```text cotyper ($HOME/git/sammcj/cotyper) Specs 24 (187 requirements) | Active 2 | Archived 6 | Active tasks 87/90 (96%) In progress: - mvp-a [spec-driven] 71/74 tasks, updated 2026-10-08 @ worktree-agent-a74082cbc1d24bbbb Ready to archive: - suggestion-quality [spec-driven] 16/16 tasks, updated 2026-10-08 Recently archived: - 2026-10-08-spike-placement - 2026-10-08-mvp-c - 2026-10-08-mvp-b ``` `--change ` prints one change's artifacts and tasks as JSON, and `--json` prints the lot. ## Design notes - **Read-only.** Creating, editing and archiving stay with OpenSpec and your agent. - **Live.** It polls the `openspec/` directories and pushes updates to the page over server-sent events, so there's nothing to refresh while agents work. - **Worktree aware.** Active changes from every git worktree are aggregated by default, with jj workspaces available behind `--jj`. - **No install or build.** The server is Python standard library. The frontend is Preact, htm and marked, vendored, with no build step. - **Reads from disk.** Specs and changes are parsed straight from disk. The CLI is only asked about the things it owns: which schemas exist and what its commands are. The skill is in my [agentic-coding](https://github.com/sammcj/agentic-coding) repo along with my other skills. - [Writing and Reviewing Agent Skills - Common Pitfalls](https://smcleod.net/2026/07/writing-and-reviewing-agent-skills-common-pitfalls/index.md): Writing and Reviewing Agent Skills - Common Pitfalls --- title: 'Writing and Reviewing Agent Skills - Common Pitfalls' date: 2026-07-04T01:00:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ['AI', 'LLM', 'agentic coding', 'agents', 'skills', 'claude', 'claude code'] author: 'Sam McLeod' draft: false description: 'The top pitfalls I keep seeing when reviewing Agent Skills, and what to do instead' searchHidden: false --- I write and review a _lot_ of [Agent](https://code.claude.com/docs/en/skills) [Skills](https://agentskills.io), and find myself frequently pushing back in reviews, as many authors assume they're writing "_just another markdown prompt_" without considering that they're actually working with one component of an agentic system. The **TLDR** of my feedback is usually along the lines of: > 1. The descriptions _only_ job is to tell the agent when the skill should be triggered. > 2. Keep the description as terse as possible while still triggering. > 3. The SKILL.md should be lean with detail pushed to reference files. > 4. Put repeatable work in scripts rather than relying on model inference. --- ## Skill Creator Primer I recommend teams use or create their own version of my [skill-creator-primer](https://github.com/sammcj/agentic-coding/blob/main/Skills/skill-creator-primer/SKILL.md) skill. The primer works alongside the official Anthropic skill-creator skill and aims to catch common pitfalls and align with best practices. > [!TIP] Proactively use the primer > While the primer should activate passively when working with skills, to get the most out of it ask your agent to use it to review your work: > > _"First please activate your skill creator primer skill._ > _Then, I want you to look at my coworkers PR: _ > _and conduct a critical review."_ ## The top five pitfalls ### 1. Descriptions written as summaries The descriptions only job is telling the agent when to load the skill. Summarising the contents invites the agent to act on the summary and skip the skill. - Roughly 30-55 words, imperative, naming the specific triggers. - The description should only contain information that is specific to knowing when to trigger the skill. - Some agents under-trigger, so be slightly assertive while keeping it tight. - Unsure it fires? Write a handful of trigger evals: realistic prompts labelled should/shouldn't trigger. > [!EXAMPLE] Example: > description: 'You MUST use this skill whenever the user mentions dashboards, monitoring or metrics.' > [!TIP] If your skill doesn't need to be triggered by an agent > Set `disable-model-invocation: true` in the frontmatter so its description isn't loaded into every session ### 2. No progressive disclosure The main `SKILL.md` should inline only what every use of the skill needs, regardless of the situation. Practising progressive disclosure keeps skills lean, reusable and composable. Other bundled information such as handling for specific scenarios goes in `references/.md` behind a pointer that says when to read it, e.g: > [!EXAMPLE] Example: > "Step 3: If working on frontend codebase you must first read 'references/frontend.md'" ### 3. Making the model do a script's job If a step is often repeated, deterministic or requires a high degree of accuracy, bundle a small script. - A script is faster, should provide the same result every run and is testable. - Favour using Python and its stdlib where possible. - If you _need_ to add dependencies use [PEP-723](https://peps.python.org/pep-0723/) inline metadata. - Instruct the agent to run with `uv run scripts/my-script.py `. ### 4. Too much content, too many skills Skills are for agents, not humans. Terse wins. The deletion test: cut a line and ask whether the agent's behaviour would change. If not, it stays cut. The same can be true for having too many skills (without good reason to do so) - especially if the descriptions aren't tight, focused or well differentiated. Favour one skill with progressive disclosure (references for each scenario) over lots of smaller but related skills. ### 5. Stepped workflows with no structure For workflows that must be followed within a skill: - Instruct the agent to track each step. - Number the steps. - Trigger a review after all steps are complete. > [!EXAMPLE] Example: > Create and track a task / TODO for each step: > 1. Do x > 2. Do y > 3. Do z > > Once all steps are completed conduct a critical review of the work and make any necessary corrections. Agents drift and finish early on long procedures; a visible checklist stops that. The same is true if your skill instructs the agent to task sub-agents with their own workflows. ## General tips - Don't wrap text in markdown - manual line breaks mean more lines for the agent to read in. - For reference files over ~100 lines, add a simple TOC near the top (but not in the main `SKILL.md`). - If your skill is very complex, add an `AGENTS.md` / `CLAUDE.md` in its root to help you and others maintain it. - A single entry point for interacting with your software project (`make lint`, `make test`, `make build`, `make run`, etc.) makes it easy for agents and skills to do the right thing everywhere. ## A note on Agent Skills vs Custom Agents - Custom Agents: - Can be thought of as a "persona", think as if you were asked to put yourself in someone else's shoes. - May include workflows / ways of working but keep it light and instead have them use skills. - Good for adversarial tasks as they get their own context and world view. - Operate within their own context (usually). - For knowledge, detailed workflows, helper tools etc., favour Skills over Custom Agents. - Skills are powerful as they're composable and often portable. - If you need a custom agent and want it to be read-only, set `permissionsMode: plan` in its frontmatter. - There's no need to create a custom agent for normal development tasks - that's what the main / general agent is for. ## Review signals Quick tells that a skill wasn't reviewed carefully: - Descriptions that have information not specific to knowing when to trigger the skill. - Descriptions that are more than 1-3 tight sentences - if it looks like a paragraph, it probably hasn't been thought through. - Overly verbose, long form prose in the body of the skill or resources. - Getting agents to do the work of scripts, have them write and drive the scripts instead. - _"When to use"_ in the body - if the agent is reading that, it's too late. It's also a sign the author may not have understood what they were creating. ## Security tips Treat third party skills as you would any other software or library obtained from the internet - don't blindly install them without first reviewing their content. - Skills can include both prompts (`SKILL.md`, references) and scripts, review the markdown for malicious or risky instructions that could mislead the agent. - If the skill contains scripts, consider running dependency and SAST scanning tools (e.g. SonarQube) across the content. - When writing scripts within skills, aim to use the standard library where possible. - Apply the principle of least privilege in the frontmatter configuration - does the skill really need web search or write tools? - Where possible, run your agentic coding tool in a sandbox. - [Measuring Model Quantisation Quality with KL Divergence](https://smcleod.net/2026/04/measuring-model-quantisation-quality-with-kl-divergence/index.md): Measuring Model Quantisation Quality with KL Divergence --- title: 'Measuring Model Quantisation Quality with KL Divergence' date: 2026-04-28T01:00:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ['AI', 'LLM', 'mlx', 'quantisation', 'qwen'] author: 'Sam McLeod' draft: false description: 'Using KL divergence to rank MLX quantisations of Qwen 3.6 (27B dense and 35B-A3B MoE)' searchHidden: false cover: image: "qwen3.6-quality.png" alt: "Quality comparison: KL divergence vs Qwen3.6-27B (bf16)" --- KL divergence against a known-good reference answers **"how much did this quant change the model's behaviour?"** rather than "how good is this model overall?". ## What KLD measures KL divergence measures how much one probability distribution disagrees with another. At each token position, both the reference and the quantised model emit a distribution over the full vocabulary (~248k tokens for Qwen-class models). The reference might say _"~80% likely `the`, ~5% likely `a`, ..."_; the quant says something slightly different. KLD compares the two per position and averages. Read it as "information lost when you use the quant in place of the reference". Zero means identical predictions; bigger means more divergent. Why this beats perplexity for ranking quants: perplexity mixes the model's intrinsic uncertainty with quant damage. KLD against a fixed reference isolates the damage. Rough scale (in nats, on Qwen-class instruct models): | Mean KLD | Reads as | | ---------------- | ---------------------------------------------------------------------------------------------------------------- | | `< 1e-4` | Distributions essentially identical | | `1e-3` to `5e-3` | Very close; typical of well-made 6-bit | | `1e-2` to `5e-2` | Measurably larger drift; typical 4-bit territory. Whether it's perceptible under sampling isn't well-established | | `> 1e-1` | Substantial divergence; sampled outputs likely to differ obviously | KLD is a divergence (asymmetric, no triangle inequality), not a perceptual measure. Treat the bands above as ranking aids, not quality verdicts. Two caveats: - **Prefill only.** KLD is measured at prompt positions. It doesn't track how sampled outputs drift over a long generation, where per-token divergences compound. Lower KLD tracks subjective quality but doesn't guarantee it. - **Direction matters.** I compute `KL(reference ‖ quant)`, weighting divergence by the _reference's_ probabilities. Disagreements where the reference is confident count more than ones in the long tail. That's the right direction for "how well does the quant imitate the reference". ## Evaluation modes mlx-kld has two evaluation modes: - **Short mode**: ~50 short prompts. Quick (1-2 minutes for six quants), but the small token budget makes results noisy and swing-prone on unlucky positions. - **Long mode**: streams WikiText-2's raw test split in 32 chunks of `n_ctx=2048` tokens (65,536 per comparison), scoring only the second half of each chunk (matching `llama.cpp`'s perplexity convention). Slower (5-10 minutes for six quants), but produces stable rankings and catches quants that degrade past their training context boundary. Results below are long-mode only. Short-mode rankings broadly agreed but with wider noise on close pairs. The two means aren't directly comparable; the scale table above is approximate for long-mode. ## Qwen 3.6 27B results Reference: `Qwen/Qwen3.6-27B` at bf16 (55.56 GB), measured against six MLX quantisations: RTN uniform (4/6/8-bit), Unsloth Dynamic mixed-precision 4-bit, and oQ data-driven mixed-precision (4-bit and 6-bit). 65,536 tokens per comparison from WikiText-2. Hardware: Apple M5 Max, 128 GB. ![Quality comparison: KL divergence vs Qwen3.6-27B (bf16). Lower-left is better.](qwen3.6-quality.png) ### Headline numbers KL divergence vs the bf16 reference, ordered by mean KLD (lower = closer to bf16): {{% wide-table %}} | Model | Size | Eff. bpw | Mean KLD | Mean KLD / bpw | Median KLD | P95 KLD | P99 KLD | Max KLD | | --------------------------------- | -------- | -------- | ------------- | -------------- | ------------- | ------------- | ------------- | -------------- | | `mlx-community/Qwen3.6-27B-8bit` | 29.50 GB | 9.69 | 0.01388 | 0.00143 | 0.00063 | 0.00454 | 0.02963 | 21.20596 | | `mlx-community/Qwen3.6-27B-6bit` | 22.78 GB | 7.48 | **0.02869** ★ | **0.00383** ★ | 0.00201 | 0.01562 | **0.10848** ★ | 26.31875 | | `Jundot/Qwen3.6-27B-oQ6` | 23.30 GB | 7.65 | 0.02956 | 0.00386 | **0.00194** ★ | **0.01511** ★ | 0.11315 | **25.44921** ★ | | `unsloth/Qwen3.6-27B-UD-MLX-4bit` | 26.19 GB | 8.60 | 0.05922 | 0.00688 | 0.00733 | 0.08340 | 0.53137 | 26.38841 | | `Jundot/Qwen3.6-27B-oQ4` | 16.69 GB | 5.48 | 0.11004 | 0.02008 | 0.02009 | 0.16473 | 0.95891 | 27.10723 | | `mlx-community/Qwen3.6-27B-4bit` | 16.05 GB | 5.27 | 0.11324 | 0.02147 | 0.02180 | 0.18127 | 1.13275 | 26.91660 | {{% /wide-table %}} ★ = best on that metric, excluding 8-bit (which trivially leads on bit width). Mean KLD / bpw is a damage-per-bit ratio (lower is better); most useful within a bit class, since KLD doesn't scale linearly with bit reduction across classes. ### Per-tensor configs The interesting differences between mixed-precision quants are in _which_ tensors got the extra bits: {{% wide-table %}} | Model | Quant family | Base bits | Overrides | Tensors elevated | | --------------------------------- | ----------------------- | --------- | ---------------------------------- | ----------------------------------------------------------------------------------------------- | | `mlx-community/Qwen3.6-27B-8bit` | RTN affine (uniform) | 8/gs64 | none | (uniform 8-bit) | | `mlx-community/Qwen3.6-27B-6bit` | RTN affine (uniform) | 6/gs64 | none | (uniform 6-bit) | | `mlx-community/Qwen3.6-27B-4bit` | RTN affine (uniform) | 4/gs64 | none | (uniform 4-bit) | | `unsloth/Qwen3.6-27B-UD-MLX-4bit` | Unsloth Dynamic (mixed) | 4/gs64 | 258 (4-bit×189, 5-bit×3, 8-bit×66) | embed_tokens×1, lm_head×1, self_attn×64, mlp×192. 8-bit boosts on every ~4th attention layer | | `Jundot/Qwen3.6-27B-oQ4` | oQ (data-driven mixed) | 4/gs64 | 110 (5-bit×108, 6-bit×2) | linear_attn(SSM)×73, mlp×32, self_attn×5. **Entire +1 bpw budget spent on SSM and MLP tensors** | | `Jundot/Qwen3.6-27B-oQ6` | oQ (data-driven mixed) | 6/gs64 | 30 (8-bit×30) | embed_tokens×1, self_attn×10, linear_attn(SSM)×13, mlp×6 | {{% /wide-table %}} ### What stands out **8-bit tracks bf16 most closely.** Mean KLD ~0.014, ~2x lower than 6-bit and ~8x lower than 4-bit. Short mode had 8-bit at ~1.6e-3, which is the metric being too generous on a small token budget: short prompts only score early-context positions where 8-bit's predictions are mostly trivial. In practice 8-bit is a drop-in for bf16; the only real cost is the bit width itself. **At 6 bpw, RTN and oQ6 are effectively tied.** RTN edges oQ6 on mean (0.02869 vs 0.02956) and P99; oQ6 wins on median, P95 and Max. Short mode had oQ6 winning by ~14% on mean; long mode collapses that to a ~3% gap the other way. Call it a draw and prefer the simpler recipe. Both sit in the band where any quality drop should be imperceptible. **At 4-bit, oQ4 beats RTN-4bit by ~3%** (0.110 vs 0.113). The SSM/MLP protection helps, but nowhere near the ~16% short mode showed. Both sit in the "noticeable degradation" band; choose by file size. **`unsloth/Qwen3.6-27B-UD-MLX-4bit` is the bad deal here.** Effective 8.6 bpw (more memory than RTN-6bit) and mean KLD 0.059, ~2x worse than 6-bit. The "every 4th attention layer to 8-bit" heuristic doesn't transfer to Qwen 3.6; you pay 6-bit memory for distinctly worse than 6-bit quality. Unsloth describes their MLX algorithm as _"still evolving"_, and their GGUF quants are strong, so this'll likely improve. Skip it for now. **UD and oQ allocate bits very differently.** UD spreads bits across repeating layer positions (attention every 4th layer, all MLP at base, plus embedding). oQ4 spends its budget on 73 SSM tensors flagged fragile by data-driven sensitivity, plus 32 MLP tensors, leaving `lm_head` at 4-bit. oQ6 spreads a smaller 30-tensor budget across embed, attention, SSM and MLP. ## Qwen 3.6 35B-A3B results The MoE variant. `Qwen3.6-35B-A3B` activates ~3B of its 35B parameters per token, so prefill is much faster than the dense 27B. Reference is `mlx-community/Qwen3.6-35B-A3B-8bit` (37.72 GB), not bf16. The 27B result showed 8-bit vs bf16 sits well below the 6-bit numbers, so this ranks lower-bit quants fine without loading 70+ GB of bf16 MoE weights. Absolute KLD here understates divergence from bf16 by the 8-bit-vs-bf16 gap. Five MLX quantisations, 65,536 tokens per comparison. ![Quality comparison: KL divergence vs Qwen3.6-35B-A3B-8bit. Lower-left is better.](qwen3.6-35b-a3b-quality.png) ### Headline numbers {{% wide-table %}} | Model | Size | Eff. bpw | Mean KLD | Mean KLD / bpw | Median KLD | P95 KLD | P99 KLD | Max KLD | | --------------------------------------------------- | -------- | -------- | ------------- | -------------- | ------------- | ------------- | ------------- | ------------- | | `Jundot/Qwen3.6-35B-A3B-oQ6` | 29.41 GB | 6.89 | **0.01039** ★ | **0.00151** ★ | **0.00396** ★ | **0.03398** ★ | **0.10955** ★ | **3.09347** ★ | | `mlx-community/Qwen3.6-35B-A3B-6bit` | 29.06 GB | 6.81 | 0.01189 | 0.00174 | 0.00503 | 0.03732 | 0.11440 | 6.95977 | | `mlx-community/Qwen3.6-35B-A3B-4bit-DWQ` | 20.66 GB | 4.84 | 0.02663 | 0.00550 | 0.01082 | 0.08504 | 0.28286 | 3.89949 | | `Jundot/Qwen3.6-35B-A3B-oQ4` | 21.10 GB | 4.95 | 0.04024 | 0.00813 | 0.01621 | 0.13556 | 0.42157 | 6.84052 | | `mlx-community/Qwen3.6-35B-A3B-4bit` (RTN baseline) | 20.40 GB | 4.78 | 0.07418 | 0.01551 | 0.03924 | 0.23813 | 0.62739 | 11.49803 | {{% /wide-table %}} ### Quant architecture The MoE has tensor categories the dense model doesn't: `router` (the gating network picking which experts each token hits) and `shared_expert` (an expert that runs on every token). Per-tensor budgets vary widely: {{% wide-table %}} | Model | Quant family | Base bits | Overrides | Tensors elevated | | --------------------------------------------------- | ------------------ | --------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------- | | `Jundot/Qwen3.6-35B-A3B-oQ6` | oQ | 6/gs64 | 228 (6-bit×30, 8-bit×198) | embed_tokens×1, linear_attn(SSM)×50, **shared_expert×160**, self_attn×16, lm_head×1 | | `mlx-community/Qwen3.6-35B-A3B-6bit` | RTN affine (mixed) | 6/gs64 | 80 (8-bit×80) | **router×40, shared_expert×40** | | `mlx-community/Qwen3.6-35B-A3B-4bit-DWQ` | DWQ | 8/gs64 | 512 (4-bit×240, 8-bit×272) | embed_tokens×1, linear_attn(SSM)×150, router×40, mlp×120, shared_expert×160, self_attn×40, lm_head×1 | | `Jundot/Qwen3.6-35B-A3B-oQ4` | oQ | 4/gs64 | 306 (5-bit×113, 6-bit×14, 8-bit×179) | embed_tokens×1, linear_attn(SSM)×125, shared_expert×160, self_attn×19, lm_head×1 | | `mlx-community/Qwen3.6-35B-A3B-4bit` (RTN baseline) | RTN affine (mixed) | 4/gs64 | 80 (8-bit×80) | router×40, shared_expert×40 | {{% /wide-table %}} DWQ's config has an 8-bit base with 4-bit downcasts on most tensors, the inverse of the usual "4-bit base + 8-bit elevations" pattern. Effective bpw still lands at 4.84, so the "4-bit" name matches the on-disk reality. Unsloth Dynamic on the dense 27B doesn't match its on-disk size (8.60 bpw despite the "4-bit" name). ### What stands out **MoE 6-bit is closer to 8-bit than dense 27B's 6-bit was to bf16.** Mean KLD ~0.011 here vs the 8-bit reference, against dense 27B's ~0.029 vs bf16. Different references, but the 6-to-8 step buys less on the MoE: both 6-bit recipes already lift routers and shared experts to 8-bit, so the remaining damage is in tensors that fire less often. **The plain RTN 4-bit is the worst, but no longer unusable.** `mlx-community/Qwen3.6-35B-A3B-4bit` (RTN with only router and shared_expert at 8-bit) sits at mean KLD 0.074, ~3x worse than DWQ-4bit. P99 is 0.63: bad, but not the 1.97 short mode showed. Coherent but noticeably degraded. If you need 4-bit, DWQ is the pick. **At 4-bit, DWQ beats oQ4 by ~34% on mean** (0.027 vs 0.040). DWQ protects more: 512 overrides covering embed, lm_head, all 150 SSM tensors, all 40 routers, every shared_expert, plus 40 attention and 120 MLP. oQ4 protects 125 SSM tensors and all 160 shared_experts but leaves the routers at 4-bit. On MoE, the router is sensitive enough that the bits oQ4 saves there cost more than they save. **At 6-bit, oQ6 wins by ~13% on mean** (0.0104 vs 0.0119) and on every other percentile. oQ6 lifts all 160 shared_experts and 50 SSM tensors to 8-bit; RTN-6bit lifts 40 routers + 40 shared_experts. Protecting the SSM tensors on top is worth ~13%. **Quant rankings don't generalise between dense and MoE.** Dense 27B: oQ4 beats RTN-4bit by ~3%. MoE: oQ4 beats the RTN baseline by ~46% but loses to DWQ by ~34%, because DWQ protects the routers and oQ4 doesn't. Pick a quant for the actual model, not "the same family at the same bit width". **MoE prefill is much faster than the dense 27B's despite being larger on disk,** because only ~3B parameters fire per token. ## What I'd actually run Anything in the 5.8-7 bpw band from a well-made quant should be indistinguishable from bf16 in practice. Choose on memory and speed, not quality. Two things drive quality at a given bit width: the _recipe_ (which tensors got more bits) and the implementation (group size, calibration data). For Qwen 3.6 27B (dense) on a 64-128 GB Mac: - **`mlx-community/Qwen3.6-27B-6bit`** (22.78 GB) is my default. Practical sweet spot, well within the band where any quality drop should be imperceptible. - **`Jundot/Qwen3.6-27B-oQ6`** (23.30 GB) if your workload is sensitive to large excursions (lowest Max and median KLD). Otherwise equivalent to RTN-6bit. - **`mlx-community/Qwen3.6-27B-8bit`** (29.50 GB) if you have the memory and want the closest match. Doesn't buy anything you'd notice over a good 6-bit. - **`mlx-community/Qwen3.6-27B-4bit`** or **`Jundot/Qwen3.6-27B-oQ4`** (16-17 GB) if memory is tight. Within 3% of each other on mean KLD; pick whichever's already cached. This is where the recipe starts to matter and divergence is more likely to surface in outputs. - **Skip `unsloth/Qwen3.6-27B-UD-MLX-4bit`.** Effective 8.6 bpw (more memory than RTN-6bit) and ~2x worse mean KLD. For Qwen 3.6 35B-A3B (MoE): - **`Jundot/Qwen3.6-35B-A3B-oQ6`** (29.41 GB) is the quality winner. ~13% better mean KLD than RTN-6bit and best on every percentile, with similar speed. - **`mlx-community/Qwen3.6-35B-A3B-6bit`** (29.06 GB) if you prefer the simpler recipe; quality gap to oQ6 is real on the metric but you're unlikely to feel it. - **`mlx-community/Qwen3.6-35B-A3B-4bit-DWQ`** (20.66 GB) is the right 4-bit choice. ~34% better mean KLD than oQ4 at similar speed. - **Skip `Jundot/Qwen3.6-35B-A3B-oQ4`**. DWQ dominates it on every metric at the same effective bit width. - **Don't use plain `mlx-community/Qwen3.6-35B-A3B-4bit`.** Mean KLD 3x worse than DWQ; divergence starts to surface in outputs at this level. DWQ is the only 4-bit worth running on this model. --- ## Measuring KLD with [mlx-kld](https://github.com/sammcj/mlx-kld) [mlx-kld](https://github.com/sammcj/mlx-kld) is what I've been using for this on MLX. It's my fork/rewrite of `TipKnuckle/mlx-kld`. ### How the tool works Two stages, with the expensive bit cached: 1. Load the reference, run a forward pass on each prompt, store the log-softmax outputs as a sparse top-K cache (~500-1000x smaller than dense), unload. 2. For each comparison model: load, run the same forward passes using the cached reference token IDs verbatim, compute per-token KL, unload before the next. Only one model is in memory at a time. Forward passes use `--chunk-tokens 2048` with `mlx-lm`'s shared KV cache; bit-equivalent to an un-chunked pass. Reference token IDs are authoritative: the comparison model is fed exactly the tokens the reference saw, never re-tokenised. Sparse top-K stores only the K most-likely log-probs per position, plus a _tail-mass scalar_ (the total probability of every other token combined). K=256 default keeps 256 explicit log-probs out of ~248k vocab entries. Lumping the tail into one bin introduces a small rank-preserving underestimate (~4% at K=256 default; see error table below). Compression on real Qwen output is ~420x at K=256 (210 MB dense, 0.5 MB sparse for ~300 tokens). #### Running it ```bash # Stage 1: cache the reference once. --short --long runs both modes and stores # both in the cache; you can also run them separately and merge later. Pointing # --compare at the reference doubles as a self-comparison sanity check (KL must be 0). mlx-kld \ --reference Qwen/Qwen3.6-27B \ --compare Qwen/Qwen3.6-27B \ --short --long \ --top-k-cache 256 \ --save-reference qwen27b_bf16_ref # Stage 2: rank as many quants as you like against the cached reference. mlx-kld \ --load-reference qwen27b_bf16_ref \ --short --long \ --compare \ mlx-community/Qwen3.6-27B-4bit \ mlx-community/Qwen3.6-27B-6bit \ mlx-community/Qwen3.6-27B-8bit \ unsloth/Qwen3.6-27B-UD-MLX-4bit \ Jundot/Qwen3.6-27B-oQ4 \ Jundot/Qwen3.6-27B-oQ6 \ --top-k 20 --chunk-tokens 2048 --json-summary-only \ --chart results/qwen3.6-quality.png \ --output results/qwen3.6 ``` Each run reports KLD stats (mean, median, std, P95, P99, max), model metadata (size, effective bpw, quant family, group size, per-tensor override count and bit distribution), and weights versioning (SHA-256 of the safetensors index plus mtime of the newest weight file, so you can tell whether two runs hit the same bytes-on-disk). ## Sanity checks and approximation error - **Self-comparison must be zero.** Pointing `--compare` at the reference produces KL = 0. If not, the tokenisation or cache load path is broken. Mine was, briefly: cached references were re-tokenising on the comparison side, silently invalidating the comparison when chat-template settings differed across invocations. Use the cached reference token IDs verbatim. - **Sparse top-K is rank-preserving but biased low.** I built dense and K=64/128/256/512/1024 caches over a 10-prompt subset, measured against the same 4-bit comparison model: | Cache | Cache size | Mean KLD on 4-bit | Rel. error vs dense | | ------------------ | ---------- | ----------------- | ------------------- | | dense (full vocab) | 210.0 MB | 0.018870 | (reference) | | K=64 | 0.1 MB | 0.017619 | -6.63% | | K=128 | 0.2 MB | 0.017871 | -5.30% | | **K=256** | **0.5 MB** | **0.018107** | **-4.05%** | | K=512 | 0.9 MB | 0.018286 | -3.10% | | K=1024 | 1.8 MB | 0.018435 | -2.31% | Every K produces a small *negative* bias: lumping the tail into one bin under-counts disagreement from low-mass tail tokens. ~4% at the K=256 default (range -2.3% at K=1024 to -6.6% at K=64), rank-preserving across all six comparison models. K=512 buys ~1% more accuracy at 2x the storage. ## Caveats What this measurement doesn't tell you: - **Generation drift.** Prefill KLD is per-position next-token disagreement. It doesn't track how sampled outputs diverge over a long generation, where per-token errors compound. A generation mode sampling N tokens at fixed prefill positions and measuring exact-match length plus trajectory KLD would close the gap; on the list. - **Confidence intervals.** Means above are point estimates. 65k tokens makes the close pairs much more stable than short mode but still point estimates. A paired bootstrap on per-position differences would give 95% CIs. - **No category breakdown.** WikiText-2 is mostly Wikipedia prose, so long mode says less about code, JSON, maths or non-English than short mode. A quant that holds up on prose but falls apart on code would look fine here. Tagging chunks by category, or running long mode against more diverse corpora, would fix this. --- Tool, prompts, and reproduction commands in [the repo](https://github.com/sammcj/mlx-kld). If you're building a quant of something interesting and want to know whether the bit allocation paid off, this is what I'd reach for. - [New Apple Silicon M4 & M5 HiDPI Limitation on 4K External Displays](https://smcleod.net/2026/03/new-apple-silicon-m4-m5-hidpi-limitation-on-4k-external-displays/index.md): New Apple Silicon M4 & M5 HiDPI Limitation on 4K External Displays --- title: 'New Apple Silicon M4 & M5 HiDPI Limitation on 4K External Displays' slug: 'apple-silicon-4k-limitations' date: 2026-03-29T01:00:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ['apple', '4k', 'monitor', 'software', 'hardware', 'bugs'] author: 'Sam McLeod' draft: false description: 'A regression in external display support on Apple Silicon M4/M5 generation chips prevents full HiDPI modes on 4K monitors, limiting users to either blurry non-HiDPI or reduced workspace HiDPI.' searchHidden: false --- Starting with the M4 and including the new M5 generations of Apple Silicon, macOS no longer offers or allows full-resolution HiDPI 4k modes for external displays. The maximum HiDPI mode available on a 3840x2160 panel is now just 3360x1890 - _M2/M3 machines did not have this limitation_. ![System Settings > Display on a M5 Max](macOS.jpeg) With this regression Apple is leaving users to choose between: - Full screen real estate at 4k (3840x2160) with blurry text due to HiDPI being disabled. or - Reduced screen real estate at 3.3k (3360x1890) with sharp text (HiDPI) but significantly less usable working space, and macOS's UI looking ridiculously oversized. ## A regression in the display controller architecture This is likely not a hardware limitation. The M5 Max supports 8K at 60Hz per Apple's own specs, and its Display Coprocessor (DCP) reports identical capabilities to the M2 Max for the same display. The M4/M5 generation changed how framebuffer budgets are allocated across display pipes, and the budget for the path that standard 4K displays use dropped from 7680 pixels wide (enough for 3840x2160 HiDPI) to 6720 (only enough for 3360x1890 HiDPI). Disassembly of the DCP firmware shows the 6720 value is a hardcoded constant - the details are below. ## Environment and Test Setup | Property | M5 Max (affected) | M2 Max (working) | | ----------------- | ------------------------------------------- | ------------------------------------------- | | Chip | Apple M5 Max | Apple M2 Max | | Model ID | Mac17,6 | Mac14,6 | | GPU Cores | 40 | 38 | | macOS | 26.4 (25E246) | 26.4 (25E246) | | Display | LG HDR 4K 32UN880 | LG HDR 4K 32UN880 | | Native Resolution | 3840x2160 | 3840x2160 | | Connection | USB-C/Thunderbolt, HBR3 (8.1 Gbps), 4 lanes | USB-C/Thunderbolt, HBR3 (8.1 Gbps), 4 lanes | | Max HiDPI Mode | **3360x1890** | **3840x2160** | Both machines report identical DCP parameters for the LG display: ``` MaxActivePixelRate = 497,664,000 MaxTotalPixelRate = 537,600,000 MaxW = 3840 MaxH = 2160 MaxBpc = 10 ``` | | | | ---------------------- | -------------------- | | ![M5](m5-low-dpi.jpeg) | ![M2](m2-hidpi.jpeg) | ## Diagnosis and Troubleshooting ### Display Override Plist (scale-resolutions) **What**: Wrote a display override plist to `/Library/Displays/Contents/Resources/Overrides/DisplayVendorID-1e6d/DisplayProductID-7750` containing scale-resolutions entries for 7680x4320 HiDPI. **Result**: No effect on M5 Max. The identical plist produces 3840x2160 HiDPI on M2 Max. WindowServer on M5 Max refuses to enumerate the mode regardless of plist content. The override plist that works on M2 Max: ```xml default-resolution AA8AAIhwAAAAPA== DisplayProductID 30544 DisplayVendorID 7789 scale-resolutions AAAeAAAAEOAAAAAJACAAAA== AAAeAAAAEOA= AAAPAAAACHA= ``` ### EDID Patching (Software Override) **What**: Wrote a patched EDID into the override plist's `IODisplayEDID` key with: - Preferred timing doubled to 4095x4095 (12-bit EDID field maximum) - Pixel clock set to maximum (655.35 MHz) - Range limits boosted to 2550 MHz max pixel clock, 255 kHz max H-freq, 255 Hz max V-freq **Result**: No effect with these values. However, waydabber (BetterDisplay developer) [has confirmed](https://github.com/waydabber/BetterDisplay/discussions/4215) that software EDID overrides did work for him on M4 in the past - he got an 8K framebuffer on a 4K TV by adding a valid 8K timing and defining it as the native resolution. The catch: a 4K panel can't actually accept an 8K signal, so this confirms the mechanism (scaled modes derive from the system's idea of native resolution) without providing a practical fix. ### EDID Hardware Flash - VIC 199 (8K) and DisplayID Extension **What**: Created a patched EDID with VIC 199 (7680x4320@60Hz) added to the CEA Video Data Block, keeping the preferred detailed timing at 3840x2160. Successfully flashed to the LG monitor's EEPROM via BetterDisplay. **Result**: The DCP read VIC 199 from the hardware EDID and updated its reported capabilities: `MaxW` changed from 3840 to 7680, `MaxH` from 2160 to 4320, and `MaxActivePixelRate` to 1,990,656,000. The DCP also allocated 2 pipes (`PipeIDs=(0,2)`, `MaxPipes=2`) as it would for a real 8K display. However, the sub-pipe 0 framebuffer budget (`MaxSrcRectWidthForPipe`) remained at 6720, and no 3840x2160 HiDPI mode appeared. A further attempt added a DisplayID Type I Detailed Timing for 7680x4320@30Hz marked as preferred and native. This **did** generate a 3840x2160 scale=2.0 mode in the CG mode list. However, when selected, macOS attempted to output 7680x4320 on the wire (since the EDID declared it as a supported output mode), which the LG could not display. A DisplayID Display Parameters block (declaring 7680x4320 as native pixel format without creating an output timing) did not generate any new modes. ### EDID Flash to Monitor EEPROM (Range Limits Only) **What**: Created a patched EDID binary with boosted range limits only (keeping preferred timing at native 3840x2160 to avoid breaking display output), attempted to flash to the LG monitor's EEPROM via BetterDisplay's "Upload EDID" feature. **Result**: The range-limits-only flash did not change any DCP parameters. The DCP derives `MaxActivePixelRate` from the preferred timing's pixel clock, not from the range limits. A subsequent flash with VIC 199 added to the Video Data Block was successful (see "EDID Hardware Flash" section above). ### IOKit Registry Override (DisplayHints) **What**: Attempted to modify the DCP's `DisplayHints` dictionary and `ConnectionMapping` array directly in the IOKit registry using `IORegistryEntrySetCFProperty`, targeting higher MaxW, MaxH, and MaxActivePixelRate values. **Result**: The DCP driver explicitly rejects userspace property writes with `kIOReturnUnsupported` (kern_return=-536870201). These properties are owned by the kernel-level `AppleDisplayCrossbar` driver and cannot be modified from userspace. ### Display Re-probe **What**: Used `IOServiceRequestProbe` to trigger the DCP to re-read display information after writing override plists. **Result**: No effect on mode enumeration. The DCP re-reads from the physical display, not from software overrides. ### WindowServer Cache Clearing **What**: Deleted `~/Library/Preferences/ByHost/com.apple.windowserver.displays.*.plist` and attempted to restart WindowServer. Also performed a full reboot. **Result**: `killall WindowServer` on macOS 26 does not actually restart WindowServer (no display flicker, no session interruption). Full reboot with the override plist in place still did not produce the 3840x2160 HiDPI mode. The cache was not the issue. ### Reducing Connected Display Count **What**: Disconnected the third display (U13ZA) to test whether the DCP's bandwidth budget across display pipes was the constraint. **Result**: No effect. With only 2 displays (LG + built-in), the mode list remained identical. The limitation is not related to the number of connected displays. ### HDMI vs USB-C/Thunderbolt **What**: Considered switching from USB-C/DisplayPort to HDMI. **Result**: Not attempted; HDMI 2.0 has less bandwidth (14.4 Gbps vs 25.92 Gbps on DP 1.4 HBR3), so would be the same or worse. ### SkyLight Private API (SLConfigureDisplayWithDisplayMode) **What**: Used `SLConfigureDisplayWithDisplayMode` from the private SkyLight framework to attempt to directly apply a 3840x2160 HiDPI mode (7680x4320 pixel backing, scale=2.0) to the LG display. The mode was sourced from both the CG mode list and from other displays. **Result**: Returns error code 1000 when the mode is not in the display's own mode list. The SkyLight display configuration API validates modes against the same DCP-derived mode list as WindowServer. There is no private API path to bypass the mode list validation. ### Faking Apple Display Identity **What**: Tested whether the DCP firmware treats Apple-branded displays differently. Added `IOGFlags = 4` (`kIOGFlagAppleLook`) to the LG's display override plist, then created a patched EDID with Apple's vendor ID ("APP" = 0x0610) and Pro Display XDR product code (0xAE21), uploaded via BetterDisplay. **Result**: The display appeared as "Apple Pro Display X" in System Settings, but `ExternalAppleLook` remained `No` on all `IOMobileFramebufferShim` instances and `MaxVideoSrcDownscalingWidth` stayed at 6720. Apple displays are identified by a hardware-level handshake over Thunderbolt/DisplayPort AUX, not by EDID vendor ID. Apple's own displays have native resolutions where HiDPI backing stores fit within the 6720 budget anyway (Pro Display XDR: 6016x3384, Studio Display: 5120x2880), so `ExternalAppleLook` may not even affect scaler budgets. ### Comprehensive IOKit Write Attempts Every available IOKit API and service was tested for writing `MaxVideoSrcDownscalingWidth` and `IOMFBMaxSrcPixels`: | Service | Method | Result | | ------------------------- | -------------------------------------- | ----------------------------------------------- | | IOMobileFramebufferShim | IORegistryEntrySetCFProperty | kIOReturnUnsupported (no setProperties handler) | | IOMobileFramebufferShim | IOConnectSetCFProperties (user client) | kIOReturnNotReady | | IOMobileFramebufferShim | IOConnectCallMethod selectors 0-30 | All kIOReturnUnsupported (stub user client) | | AppleARMIODevice (parent) | IORegistryEntrySetCFProperty | kIOReturnNotReady | | AppleDCPLinkService | IORegistryEntrySetCFProperty | kIOReturnNotReady | | DCPAVServiceProxy | IORegistryEntrySetCFProperty | kIOReturnNotReady | | AppleDisplayCrossbar | IORegistryEntrySetCFProperty | kIOReturnNotReady | | IOAVController | IORegistryEntrySetCFProperty | **Accepted** but does not propagate to DCP | `IOAVController` is the only service that accepted property writes, but it just stores them in its own registry entry without forwarding anything to the DCP pipeline. ## Where the limit is applied The DCP reports identical capability parameters on both machines - `MaxActivePixelRate`, `MaxW`, `MaxH`, `MaxTotalPixelRate` all match. These come from the display's EDID, so that's expected. The difference shows up in WindowServer's mode list. On M2 Max, `CGSGetNumberOfDisplayModes` includes 3840x2160 at scale=2.0. On M5 Max, with the same DCP parameters and the same override plists, that mode doesn't exist. ### IOMFBMaxSrcPixels - the per-sub-pipe framebuffer budget The `IOMFBMaxSrcPixels` property on the `IOMobileFramebufferShim` IOKit service exposes framebuffer size budgets. The M2 Max and M5 Max use different structures here. **M2 Max** uses a flat per-controller budget: ``` # Each external display controller (all identical): "IOMFBMaxSrcPixels" = { "MaxSrcRectWidth" = 7680, "MaxSrcRectTotal" = 33177600, ... } ``` Every external display controller gets a flat `MaxSrcRectWidth` of **7680** and `MaxSrcRectTotal` of **33,177,600** (7680 x 4320). With a 7680-pixel budget, 3840x2160 HiDPI (7680x4320 backing store) fits. **M5 Max** restructured to per-sub-pipe budgets within each controller: ``` # Each external display controller (all identical): "IOMFBMaxSrcPixels" = { "MaxSrcRectHeightForPipe" = (4608, 4608, 4608, 4608), "MaxSrcRectWidthForPipe" = (6720, 7680, 7680, 7680), "MaxSrcBufferHeight" = 16384, "MaxSrcBufferWidth" = 16384, "IOMFBCompressionSupport" = 1 } ``` The 4 values in `MaxSrcRectWidthForPipe` are **sub-pipes within each display controller**, not separate display outputs. A single-stream 4K display only uses sub-pipe 0. Sub-pipes 1-3 are for multi-pipe configurations (8K displays use 2 sub-pipes simultaneously, which is why an 8K EDID causes the DCP to assign `PipeIDs=(0,2)` with `MaxPipes=2`). | Sub-pipe | MaxSrcRectWidth | What it's for | | -------- | --------------- | ---------------------------------------------------- | | 0 | **6720** | Single-stream output (used by all standard displays) | | 1 | 7680 | Multi-pipe mode only (e.g. 8K displays) | | 2 | 7680 | Multi-pipe mode only | | 3 | 7680 | Multi-pipe mode only | Sub-pipe 0's budget of 6720 pixels lines up exactly with the observed cap: 3360x1890 HiDPI needs a 6720x3780 backing store. For 3840x2160 HiDPI, the backing store would need to be 7680 pixels wide. Sub-pipes 1-3 have this budget, but they're only used in multi-pipe mode for displays that output above 4K. | Property | M2 Max | M5 Max | | -------------------------- | ------------------------ | --------------------------------------------------- | | Budget structure | Flat per controller | Per sub-pipe array | | External controller budget | `MaxSrcRectWidth` = 7680 | `MaxSrcRectWidthForPipe` = (6720, 7680, 7680, 7680) | | Single-stream max width | **7680** | **6720** | | `MaxSrcRectTotal` | 33,177,600 (7680x4320) | Not present | | LG pipe assignment | `PipeIDs=(1)` | `PipeIDs=(0)` | | 3840x2160 HiDPI | Yes | No | This property is set by the kernel-level `IOMobileFramebufferShim` driver and can't be modified from userspace. ### MaxVideoSrcDownscalingWidth There's also a `MaxVideoSrcDownscalingWidth` property on each `IOMobileFramebufferShim` that controls the maximum width the video scaler will accept for HiDPI downscaling: | Controller | MaxSrcRectWidthForPipe[0] | MaxVideoSrcDownscalingWidth | Ratio | | -------------------- | ------------------------- | --------------------------- | ----- | | Internal display | 5120 | **10744** | 2.1x | | External controllers | 6720 | **6720** | 1.0x | The internal display's scaler runs at 2.1x its sub-pipe width. The external controllers get zero headroom - scaler budget equals sub-pipe budget. If external controllers got even a fraction of the internal display's ratio, 3840x2160 HiDPI (7680 backing store) would fit. ### The budget is fixed at boot Testing confirmed that both `MaxSrcRectWidthForPipe` and `MaxVideoSrcDownscalingWidth` are set when the driver loads and do not change at runtime, regardless of what you do: | Test | Sub-pipe 0 MaxSrcRectWidth | MaxVideoSrcDownscalingWidth | Changed? | | -------------------------------------------------- | -------------------------- | --------------------------- | ------------------------------------- | | LG on USB-C port 1 | 6720 | 6720 | No | | LG on USB-C port 2 | 6720 | 6720 | No | | LG on USB-C port 3 | 6720 | 6720 | No | | LG on HDMI | 6720 | 6720 | No | | U13ZA disconnected (2 displays total) | 6720 | 6720 | No | | Clamshell mode (LG only, lid closed) | 6720 | 6720 | No | | EDID with VIC 199 (8K) flashed to monitor | 6720 | 6720 | No | | EDID with 4095x4095 preferred DTD | 6720 | 6720 | No | | EDID with DisplayID 7680x4320 timing | 6720 | 6720 | No | | Software override plist with 8K default-resolution | 6720 | 6720 | No | | BetterDisplay native resolution set to 7680x4320 | 6720 | 6720 | No | | 8K-only EDID (VIC 199 sole mode) | 6720 | 6720 | No (got 2 pipes but budget unchanged) | Ghidra disassembly later confirmed these values are hardcoded constants in the DCP firmware binary, not derived from EDID or any runtime input. ### What's going on BetterDisplay developer waydabber ([discussion #4215](https://github.com/waydabber/BetterDisplay/discussions/4215)) initially described this as dynamic resource allocation: > "Generally 3840x2160 HiDPI is not available with any M4 generation Mac on non-8K displays due to the new dynamic nature of how the system allocates resources. There might be exceptions maybe - when the system concludes that no other displays could be attached and there are resources left still for a higher resolution framebuffer. But normally the system allocates as low framebuffer size as possible, anticipating further displays to be connected and saving room for those." The Ghidra disassembly tells a simpler story: the sub-pipe 0 budget is a hardcoded constant (6720) in the DCP firmware, not dynamically calculated. It doesn't change based on connected displays, EDID data, or any runtime condition. Even in clamshell mode with a single external display, the budget stays at 6720. Putting it together: - The DCP reports identical capabilities on M2 Max and M5 Max (same `MaxW`, `MaxH`, `MaxActivePixelRate`) - The M2 Max uses a flat per-controller framebuffer budget (`MaxSrcRectWidth=7680`), giving every external display enough backing store width for 3840x2160 HiDPI - The M5 Max restructured to per-sub-pipe budgets (`MaxSrcRectWidthForPipe=(6720, 7680, 7680, 7680)`), where the single-stream sub-pipe (the only one a 4K display can use) is capped at 6720 - This caps the backing store width and therefore caps HiDPI at 3360x1890 on M5 Max - Disconnecting other displays, switching ports, or closing the laptop lid doesn't change the sub-pipe budgets - Adding VIC 199 (8K) to the EDID changes DCP-reported MaxW/MaxH but doesn't affect the sub-pipe budget - Adding a DisplayID 7680x4320 timing creates a 3840x2160 scale=2.0 mode, but macOS tries to output 8K on the wire (treating it as a real output mode rather than a scaling mode), which the 4K panel can't display The full stack from display to application: ``` Physical Display (EDID/DPCD) | DCP Co-processor Firmware (t605xdcp.im4p) [Hardcoded constant: 0x1A40 (6720) for external sub-pipe 0] [Signed, runs on dedicated ARM core, not encrypted] | IOMobileFramebufferShim (kernel kext) [Receives values from DCP via RTKit mailbox] [No setProperties handler - read-only from userspace] | AppleDisplayCrossbar (kernel kext) [Mode generation uses MaxVideoSrcDownscalingWidth as cap] | WindowServer / SkyLight [CGSGetNumberOfDisplayModes enumerates only valid modes] | CoreGraphics / Applications ``` On M2/M3, the system generated HiDPI modes up to 2.0x the native resolution (3840x2160 native = 7680x4320 backing store). On M4/M5, sub-pipe 0's budget of 6720 caps this at ~1.75x. The pipe budget is a hardcoded constant; the runtime scaler gate (`MaxVideoSrcDownscalingWidth`) is derived from hardware config that likely traces back to the same value. The architectural change from M2's flat 7680 budget to M5's sub-pipe model, with a smaller constant for sub-pipe 0, is what broke 4K HiDPI. --- ## What could fix this Apple would need to update the DCP firmware (`t605xdcp.im4p` and equivalents for other M4/M5 variants): 1. Increase the hardcoded constant from 0x1A40 to 0x1E00 and update the hardware config register accordingly 2. Allow multi-pipe for HiDPI backing stores even when the output resolution fits in a single pipe 3. Make the allocation dynamic based on actually connected displays 4. Expose the RuntimeProperty or boot-arg mechanism on production hardware I've filed Apple Feedback FB22365722. waydabber's take on Apple's position: > "Apple is certainly aware of the issue but since so far it affected a minority, the solution was to add a disclaimer to the product page about the availability of scaled resolutions, which shows some level of deliberation and a conclusion that they don't want to change things just yet." A 5K or 8K panel may not hit the exact same limit since its EDID native resolution is high enough that 1.75x scaling still provides a usable backing store. ## Appendix: Diagnostic Commands and Output Commands to reproduce this on any Mac. All except #3 work without special permissions. Command #6 is the most useful single diagnostic for this issue. ### Diagnostic commands ```bash # 1. DCP rate limits and native caps per display ioreg -l -w0 | grep -o '"MaxActivePixelRate"=[0-9]*\|"MaxW"=[0-9]*\|"MaxH"=[0-9]*' \ | paste - - - | sort -u # 2. System profiler display summary system_profiler SPDisplaysDataType # 3. All HiDPI modes for a display (requires Screen Recording permission) # Use BetterDisplay, SwitchResX, or any tool that calls # CGSGetNumberOfDisplayModes / CGSGetDisplayModeDescriptionOfLength. # Example output format shown below. # 4. Display connection details and DisplayHints ioreg -l -w0 | grep -B5 -A2 'MaxActivePixelRate' | grep -v EventLog # 5. ConnectionMapping (per-pipe allocation) ioreg -l -w0 | grep "ConnectionMapping" # 6. Per-pipe framebuffer budgets (the key constraint on M4/M5) ioreg -l -w0 | grep "IOMFBMaxSrcPixels" # 7. Video scaler downscaling width cap per controller ioreg -l -w0 | grep "MaxVideoSrcDownscalingWidth" ``` ### M2 Max (Mac14,6) - Working Display: LG HDR 4K 32UN880 (3840x2160) via USB-C/Thunderbolt, macOS 26.4. > Note: Commands 2, 4, 5 were captured without the LG connected. The mode list (command 3) was captured with the LG connected in a separate session. **Command 1 - DCP rate limits:** ``` "MaxW"=3840 "MaxActivePixelRate"=497664000 "MaxH"=2160 "MaxW"=3840 "MaxActivePixelRate"=552950718 "MaxH"=2400 ``` **Command 2 - System profiler (LG not connected at capture time):** ``` Graphics/Displays: Apple M2 Max: Chipset Model: Apple M2 Max Type: GPU Bus: Built-In Total Number of Cores: 38 Vendor: Apple (0x106b) Metal Support: Metal 4 Displays: Color LCD: Display Type: Built-in Liquid Retina XDR Display Resolution: 3456 x 2234 Retina Main Display: Yes Mirror: Off Online: Yes Automatically Adjust Brightness: No Connection Type: Internal ``` **Command 3 - HiDPI modes (LG connected, top 5):** ``` mode: {resolution=3840x2160, scale = 2.0, freq = 60, bits/pixel = 16} mode: {resolution=3840x2160, scale = 2.0, freq = 30, bits/pixel = 16} mode: {resolution=3360x1890, scale = 2.0, freq = 60, bits/pixel = 16} mode: {resolution=3360x1890, scale = 2.0, freq = 30, bits/pixel = 16} mode: {resolution=3200x1800, scale = 2.0, freq = 60, bits/pixel = 16} ``` Note: **3840x2160 at scale = 2.0 is present** as the highest available HiDPI mode. **Command 4 - Display connection details (LG not connected at capture time):** ``` DisplayHints = {} ``` When LG is connected, reports identical values to M5 Max: ``` "DisplayHints" = { "MaxBpc"=10, "EDID UUID"="1E6D5077-0000-0000-0520-0104B5462878", "MaxTotalPixelRate"=537600000, "MaxW"=3840, "MaxActivePixelRate"=497664000, "Tiled"=No, "ProductName"="LG HDR 4K", "MaxH"=2160 } ``` **Command 5 - ConnectionMapping (LG not connected at capture time):** ``` "ConnectionMapping" = () ``` ### M5 Max (Mac17,6) - Affected Display: LG HDR 4K 32UN880 (3840x2160) via USB-C/Thunderbolt, macOS 26.4. **Command 1 - DCP rate limits:** ``` "MaxW"=3840 "MaxActivePixelRate"=497664000 "MaxH"=2160 "MaxW"=3840 "MaxActivePixelRate"=552950718 "MaxH"=2400 ``` Identical to M2 Max. **Command 2 - System profiler:** ``` Graphics/Displays: Apple M5 Max: Chipset Model: Apple M5 Max Type: GPU Bus: Built-In Total Number of Cores: 40 Vendor: Apple (0x106b) Metal Support: Metal 4 Displays: LG HDR 4K: Resolution: 6720 x 3780 UI Looks like: 3360 x 1890 @ 60.00Hz Main Display: Yes Mirror: Off Online: Yes Rotation: Supported Color LCD: Display Type: Built-in Liquid Retina XDR Display Resolution: 3456 x 2234 Retina Mirror: Off Online: Yes Automatically Adjust Brightness: Yes Connection Type: Internal U13ZA: Resolution: 3840 x 2400 (WQUXGA) UI Looks like: 1920 x 1200 @ 60.00Hz Mirror: Off Online: Yes Rotation: Supported ``` Note: The LG's backing store is 6720x3780 (3360x1890 HiDPI). This is 1.75x the native resolution, not the 2.0x (7680x4320) needed for 3840x2160 HiDPI. **Command 3 - HiDPI modes (top 5):** ``` mode: {resolution=3360x1890, scale = 2.0, freq = 60, bits/pixel = 16} mode: {resolution=3360x1890, scale = 2.0, freq = 30, bits/pixel = 16} mode: {resolution=3200x1800, scale = 2.0, freq = 60, bits/pixel = 16} mode: {resolution=3200x1800, scale = 2.0, freq = 30, bits/pixel = 16} mode: {resolution=3008x1692, scale = 2.0, freq = 60, bits/pixel = 16} ``` Note: **3840x2160 at scale = 2.0 is absent.** Maximum HiDPI is 3360x1890. **Command 4 - Display connection details:** ``` "DisplayHints" = { "MaxBpc"=10, "EDID UUID"="1E6D5077-0000-0000-0520-0104B5462878", "MaxTotalPixelRate"=537600000, "MaxW"=3840, "MaxActivePixelRate"=497664000, "Tiled"=No, "ProductName"="LG HDR 4K", "MaxH"=2160 } ``` Identical to M2 Max. **Command 5 - ConnectionMapping:** ``` "ConnectionMapping" = ( { "MaxW"=3840, "MaxTotalPixelRate"=537600000, "MaxActivePixelRate"=497664000, "ProductName"="LG HDR 4K", "Address"="0.2.0", "PipeIDs"=(0), "MaxPipes"=1, "MaxH"=2160 }, { "MaxW"=3840, "MaxTotalPixelRate"=594000000, "MaxActivePixelRate"=552950718, "ProductName"="U13ZA", "Address"="0.0.1", "PipeIDs"=(1), "MaxPipes"=1, "MaxH"=2400 } ) ``` **Command 6 - IOMFBMaxSrcPixels (per-pipe framebuffer budgets):** ``` "IOMFBMaxSrcPixels" = { "MaxSrcRectHeightForPipe" = (4608, 4608, 4608, 4608), "MaxSrcRectWidthForPipe" = (6720, 7680, 7680, 7680), "MaxSrcBufferHeight" = 16384, "MaxSrcBufferWidth" = 16384, "IOMFBCompressionSupport" = 1 } ``` Sub-pipe 0 (the single-stream path all standard displays use) has a `MaxSrcRectWidthForPipe` of 6720, which is exactly the backing store width for 3360x1890 HiDPI. Sub-pipes 1-3 have 7680 but are only used for multi-pipe 8K configurations. ### Key observation DCP-reported capabilities are identical between M2 Max and M5 Max for the same display. The difference is in the display controller architecture: the M2 Max uses a flat per-controller budget (`MaxSrcRectWidth=7680`), while the M5 Max uses per-sub-pipe budgets (`MaxSrcRectWidthForPipe=(6720, 7680, 7680, 7680)`). The single-stream sub-pipe (the only one a 4K display uses) gets a 6720-wide budget on M5 Max, capping HiDPI at 3360x1890 before WindowServer ever builds its mode list. ## Appendix: DCP Firmware Analysis The DCP co-processor firmware is stored as an Image4 payload (im4p) on the Preboot volume at `/System/Volumes/Preboot//restore/Firmware/dcp/`. Each SoC variant has its own firmware file (e.g. `t605xdcp.im4p` for M5 Max). The M5 Max firmware is LZFSE-compressed (4.1MB compressed, 16.4MB decompressed) and not encrypted - extractable with [img4tool](https://github.com/tihmstar/img4tool): ```bash img4tool -e t605xdcp.im4p -o t605xdcp.bin ``` All the IOKit property names involved in the HiDPI limit (`MaxVideoSrcDownscalingWidth`, `MaxSrcRectWidthForPipe`, `IOMFBMaxSrcPixels`, `ExternalAppleLook`, etc.) come from this firmware, not from any kext. The kexts are fully stripped and searching for 0x1A40 (6720) in them returned nothing. The runtime enforcement function is `IOMFB::UPPipe::verify_downscaling(SwapRequest *)`, which checks `MaxVideoSrcDownscalingWidth` (not the `MaxSrcRectWidthForPipe` array) against the requested source width. The pipe width array informs mode enumeration; the swap validation checks a separate field. ### RuntimeProperty boot-args The firmware has a runtime property system (`IOMobileFramebuffer::parse_RTP_boot_args()`) that suggests properties can be overridden at boot: | RuntimeProperty | Purpose | | -------------------------------------------------- | -------------------------------------------------- | | `iomfb_RuntimeProperty_ExternalAppleLook` | Toggle Apple display identity on external displays | | `iomfb_RuntimeProperty_allocateDefaultFramebuffer` | Control default framebuffer allocation | | `iomfb_enable_bw_check` | Enable/disable bandwidth checking | | `iomfb_disable_rt_bw` | Disable real-time bandwidth management | | `iomfb_dual_pipe_policy` | Controls dual-pipe assignment policy | With SIP reduced and Startup Security set to Reduced Security, these were tested via `sudo nvram boot-args=`. iBoot passed them through to the device tree, but the DCP firmware did not act on any of them. The DCP likely receives its configuration through a separate channel, or `parse_RTP_boot_args()` is a development-only code path disabled on production hardware. ### Disassembly Decompiling the DCP firmware in Ghidra, the function that builds the `IOMFBMaxSrcPixels` dictionary (`FUN_002eb62c` in `UPPipeDCP_H17GSCD.cpp`) checks whether the display is internal or external, then assigns pipe widths from hardcoded constants: | Constant | Hex | Decimal | Used for | | -------------- | ------ | -------- | ------------------------------------ | | `DAT_00001a40` | 0x1A40 | **6720** | External display sub-pipe 0 width | | `0x1e00` | 0x1E00 | **7680** | External display sub-pipes 1-3 width | | `DAT_00001400` | 0x1400 | **5120** | Internal display sub-pipe 0 width | | `DAT_00001200` | 0x1200 | **4608** | All sub-pipe heights | `MaxVideoSrcDownscalingWidth` (the runtime gate) is not a simple hardcoded literal. It's initialised from a hardware config register or firmware config table and may be recalculated during display configuration. But on external controllers it ends up at 6720, same as `MaxSrcRectWidthForPipe[0]`, suggesting both derive from the same source. The limitation operates at two levels: `MaxSrcRectWidthForPipe[0]` (hardcoded 0x1A40) controls which modes get enumerated, and `MaxVideoSrcDownscalingWidth` (derived, but also 6720) is the runtime check that blocks swap requests exceeding the budget. Why we think this is a firmware policy choice rather than a hardware constraint: - Sub-pipes 1-3 of the same external controllers support 7680 (hardcoded 0x1E00) - The internal display's `MaxVideoSrcDownscalingWidth` is 10744, suggesting the scaler silicon handles values well above 6720 - 8K displays work via multi-pipe, where each pipe handles 3840 width (within budget) - 4K at 2x HiDPI (7680 backing store) falls in a gap: too wide for single-pipe at 6720, but the DCP won't assign multi-pipe because the 3840x2160 output resolution doesn't need it ## Virtual Display Mirror Workaround (Temporary) I wrote a small menu bar app ([force-hidpi](https://github.com/sammcj/force-hidpi)) that works around the pipe 0 budget by creating a virtual display at 7680x4320 via CoreGraphics' `CGVirtualDisplay` API (a semi-public API also used by DisplayLink), then hardware-mirroring the physical 4K panel from it. ![](settings.jpeg) ![](details.jpeg) The hardware mirror path in the DCP bypasses `verify_downscaling` entirely - mirrored displays receive pre-composited frames from the compositor rather than direct backing store swaps through the pipe budget check. `system_profiler` confirms "Hardware Mirror: Yes" with the virtual display as the source. This gets 3840x2160 HiDPI working on the physical panel. Text is measurably sharper than 1.0x scale (confirmed by more content visible at the same font size), and macOS's UI elements render at proper 2x density. The virtual display uses PQ (ST 2084) EOTF for 16-bit/64bpp compositing, with a PQ-to-SDR gamma correction table applied so the output looks correct on SDR panels. There are trade-offs. The app must stay running (quitting it tears down the virtual display and reverts to 1.0x). An extra "display" appears in System Settings. Text rendering is not quite identical to native HiDPI on an M2 Max driving the same panel - whether that's the extra compositing pass or something else in the mirror path, I haven't been able to pin down. It uses CoreGraphics' `CGVirtualDisplay` API which is semi-public (used by DisplayLink and other hardware vendors) but could still break with future macOS updates. The point is - it's a hack, not a fix. The real solution is Apple updating the DCP firmware constant from 0x1A40 to 0x1E00. _Note: If you want an all-rounder display configuration tool for macOS consider supporting [BetterDisplay](https://betterdisplay.pro)._ ## References - [BetterDisplay Discussion #4215](https://github.com/waydabber/BetterDisplay/discussions/4215) - waydabber's description of the M4-generation limitation - [MacRumors discussion](https://forums.macrumors.com/threads/5k2k-at-120hz-with-mac-mini-m4.2441289/page-29) - [Apple Discussions](https://discussions.apple.com/thread/256265624?sortBy=oldest_first&page=1) - [Apple M5 Max specifications](https://www.apple.com/macbook-pro/specs/) - claims 8K (7680x4320) at 60Hz over Thunderbolt - [img4tool](https://github.com/tihmstar/img4tool) - used to extract DCP firmware (`t605xdcp.im4p`) - [Ghidra](https://ghidra-sre.org/) - used to disassemble the ARM64 DCP firmware binary, confirming the 6720 constant - Apple Feedback FB22365722 - [The advice I find myself repeating every time someone asks how to get started with Claude Code](https://smcleod.net/2026/03/the-advice-i-find-myself-repeating-every-time-someone-asks-how-to-get-started-with-claude-code/index.md): The advice I find myself repeating every time someone asks how to get started with Claude Code --- title: 'The advice I find myself repeating every time someone asks how to get started with Claude Code' date: 2026-03-05T01:00:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ['AI', 'claude', 'agentic coding', 'claude code', 'tips', 'productivity'] author: 'Sam McLeod' draft: false description: 'I spend a lot of my time helping people who are getting started with Claude Code. These are the key things I find myself repeating.' searchHidden: false --- - **Configuration** - [Sandboxing](#enable-sandboxing) - constrain file access and network connections - [Permissions](#configure-agent-permissions-allow-ask-deny) - pre-approve safe operations, hard-block dangerous ones - [Agent rules](#agent-rules) - concise, scoped CLAUDE.md files that shape agent behaviour - [Recommended plugins & tools](#recommended-plugins-and-tools) - 3rd party tools that improve safety, context usage and code navigation - **Extend knowledge and capabilities** - [Skills](#lean-into-skills) - dynamic knowledge acquisition with progressive disclosure - [Language servers](#enable-language-servers-lsp) - give the agent go-to-definition, find-references, and type checking - [MCP tools](#take-a-minimalist-approach-to-mcp-tools) - external tool servers, used sparingly - **Workflow** - [Plan before acting](#use-planning-mode) - read-only exploration and task definition - [Embrace starting fresh sessions](#embrace-starting-fresh-sessions) - keep context clean - [Template out common commands](#create-commands-or-skills-for-frequent-prompts) - reusable prompts for common tasks ![](/2025/04/my-plan-document-act-review-flow-for-agentic-software-development/setup-plan-act-iterate.svg) --- ## Configuration ### Agent Rules Add tight, layered rules - Claude Code loads `CLAUDE.md` files into context on every single message. Every line you add is tokens the model has to process and remember. Treat it like a config file, not a wiki. Use two layers: - **Global** (`~/.claude/CLAUDE.md`) - things that apply to every project: language preferences, security rules, things to never do - **Project** (`/CLAUDE.md` or `/.claude/CLAUDE.md`) - repo-specific context: build commands, architecture decisions, testing patterns ```text ~/.claude/CLAUDE.md # always loaded, every project /CLAUDE.md # loaded for that project only /.claude/CLAUDE.md # alternative location (same effect) ``` Think about the signal-to-noise ratio. If a rule isn't relevant to the current task, it's just noise misleading the prediction engine. Be clear, concise and specific. Don't blindly import someone else's entire rule set (including mine). Project-level CLAUDE.md files should be checked into the repo so your whole team benefits. Remember: Rules are suggestions, not hard controls. - [Claude Memory / Rules](https://code.claude.com/docs/en/memory) - [Claude.md vs Rules vs Skills](https://code.claude.com/docs/en/features-overview#claude-md-vs-rules-vs-skills) --- ### Enable sandboxing [Sandboxing](https://code.claude.com/docs/en/sandboxing) constrains what Claude Code can read, write and connect to. It's the single fastest way to limit the blast radius of a runaway agent. In `~/.claude/settings.json`: ```json { "sandbox": { "enabled": true, "autoAllowBashIfSandboxed": true, "network": { "allowedDomains": [ "*.github.com", "*.githubusercontent.com", "*.npmjs.com", "*.pypi.org", "*.crates.io", "*.golang.org", "*.docker.io", "*.smcleod.net" ] }, "excludedCommands": ["docker", "ssh", "uv run"] } } ``` The network allowlist stops the agent from making unexpected outbound connections. `autoAllowBashIfSandboxed` means you don't get prompted for every shell command when the sandbox is already constraining what it can do. `excludedCommands` lets you exempt specific commands that need network access the sandbox would otherwise block. Note: The sandbox is not a silver bullet. Claude can actually choose not to use the Sandbox if it encounters problems with it - and somewhat [idiotically there's no way to disable this - _yet_](https://github.com/anthropics/claude-code/issues/10089). ### Configure agent permissions (allow, ask, deny) [Permissions](https://code.claude.com/docs/en/permissions) let you pre-approve safe operations (so you're not interrupted constantly) and hard-block dangerous ones (so a confused agent can't do real damage). The syntax is `Tool(pattern *)`. ```json { "permissions": { "allow": [ "Bash(git diff *)", "Bash(git status *)", "Bash(go *)", "Bash(cargo *)", "Edit(./**/*.go)", "Edit(./**/*.ts*)", "Edit(./**/*.py)", "WebFetch", "WebSearch" ], "deny": [ "Bash(sudo *)", "Bash(git push *)", "Bash(rm -rf ~*)", "Read(~/.ssh/**)", "Read(**/secrets/**)" ], "ask": [ "Bash(git add *)", "Bash(git commit *)", "Bash(rm -rf *)", "Bash(ssh *)", "Read(**/.env)" ] } } ``` The deny list is your hard safety net. `git push` and `sudo` are the obvious ones. The ask list catches things that are usually fine but you want to eyeball first, like commits and destructive deletes. Everything in allow runs without prompting. --- ## Lean into skills [Skills](https://agentskills.io) are an effective way of extending your agent's knowledge. Think of them like a library where the agent can see all the book spines and pull out the relevant one whenever it encounters a situation where that knowledge would be useful. They're simple markdown files that Claude loads into context on demand, rather than permanently like rules. Skills live in `~/.claude/skills/` (global) and `.claude/skills/` (project-level). Each skill has a `SKILL.md` with a name and concise description in its front matter. The agent sees all skill names and descriptions automatically and self-selects the relevant ones. ```text ~/.claude/skills/ ├── code-review/ ├── go-testing/ ├── shell-scripting/ ├── systematic-debugging/ ├── extract-wisdom/ └── ... ``` As well as the official skills documentation, you can see my [skill creator skill](https://github.com/sammcj/agentic-coding/blob/main/Skills/skill-creator/SKILL.md) (meta, I know). Note: Never blindly install skills you find online, always review the content and any attached scripts to make sure they're not malicious and are aligned with your goals. - Anthropic publishes some [official skills](https://github.com/anthropics/skills) for common tasks like creating PPTX and DOCX files. - I publish most of my skills publicly in my [agentic-coding](https://github.com/sammcj/agentic-coding/tree/main/Skills) repo. --- ## Agent Tools ### Enable Language Servers (LSP) Claude Code has a built-in LSP tool that gives the agent access to go-to-definition, find-references, hover types, and diagnostics - the same code intelligence IDEs use. With LSP enabled, the agent can navigate code structurally rather than relying on text search - jumping to definitions, finding all call sites before refactoring, and catching type errors as it works. It's enabled by installing [code intelligence plugins](https://code.claude.com/docs/en/discover-plugins#code-intelligence) from the official marketplace: ```bash # Browse available code intelligence plugins (try searching for 'lsp') /plugin # Or install directly, e.g. /plugin install typescript-lsp@claude-plugins-official ``` Some LSPs may also need the language servers installed for the languages you work with: ```bash # Python pip install python-lsp-server # or: uvx install python-lsp-server # Go (included with the Go toolchain) go install golang.org/x/tools/gopls@latest # Rust rustup component add rust-analyzer # TypeScript / JavaScript pnpm install -g typescript typescript-language-server ``` ### Take a minimalist approach to MCP tools Every MCP server you enable adds tool descriptions to the context. If you have 10 servers each exposing 5-20 tools, that's a lot of tokens spent in every interaction just telling the model what's available. - If you have easy to use CLI tools (e.g. `gh` for Github) - let the agent use those instead of adding permanent MCP servers for them. - Don't load every MCP server for every project. Use project-level `.claude/settings.json` to scope servers to where they're actually needed. - For 99% of tasks [I only have one MCP server enabled](https://github.com/sammcj/mcp-devtools). ### Recommended plugins and tools - [cc-safety-net](https://ccsafetynet.com/) (hooks that try to catch some destructive commands). - [context-mode](https://github.com/mksglu/context-mode) which significantly reduces context usage. - [Graphify](https://github.com/Graphify-Labs/graphify) can be useful for creating graphs of code and LLM wikis providing them with semantic search and context. --- ## Workflow ### Use planning mode Planning mode (`shift+tab` to toggle) keeps the agent in read-only exploration mode. It can use sub-agents, read files, search code, use tools to research - but it can't make changes until you approve its plan. This is valuable for anything beyond a simple, targeted fix. The plan itself is an artefact you can inspect and refine with Claude, when you're ready Claude will offer to start a fresh session to act upon the plan. **[TLDR]({{< ref "posts/2025-04-28-agentic-coding-dev-flow" >}}): plan first, act second, iterate.** ### Embrace starting fresh sessions LLMs are stateless. Every message you send includes the full conversation history. As a session gets longer, the model has more to process; it slows down and output quality degrades. Embrace starting fresh sessions, operating from a plan allows you to do this without losing where you're up to. ### Create commands (or skills) for frequent prompts Create commands or skills for frequently used prompts, providing you quick access via `/command-name`. Here's my most-used one, `/self-review` that I run after Claude completes multiple tasks: ```markdown --- description: Conduct a critical self-review of your changes & fix any errors --- Please conduct a critical self-review of your changes, check for any errors or changes you missed, correct any errors you may find. ``` --- Most of the rules, skills, hooks and commands I use are in my [agentic-coding](https://github.com/sammcj/agentic-coding) repo. Take what's useful, ignore the rest. - [Patching NVIDIA's driver and vLLM to enable P2P on consumer GPUs](https://smcleod.net/2026/02/patching-nvidias-driver-and-vllm-to-enable-p2p-on-consumer-gpus/index.md): Patching NVIDIA's driver and vLLM to enable P2P on consumer GPUs --- title: "Patching NVIDIA's driver and vLLM to enable P2P on consumer GPUs" date: 2026-02-25T23:59:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: [ 'AI', 'LLM', 'tech', 'vllm', 'llama.cpp', 'gguf', 'awq', 'cuda', 'nvidia', 'GPU', 'performance', 'optimisation', ] author: 'Sam McLeod' draft: false summary: 'NVIDIA artificially restricts peer-to-peer (P2P) GPU communication to their enterprise cards. Turns out this is a software limitation, not a hardware one. I patched my drivers to remove it, hacked vLLM to take advantage of it, and got a 15-50% throughput improvement running Qwen 3.5 35b on dual RTX 3090s.' searchHidden: false # cover: # image: "todo.jpg" # alt: "todo" # relative: true # Set to true for page bundle images --- NVIDIA artificially restricts peer-to-peer (P2P) GPU communication to their enterprise cards. Turns out this is a software limitation, not a hardware one. I patched my drivers to remove it, hacked vLLM to take advantage of it, and got a 10-30% throughput improvement running Qwen 3.5 35b on dual RTX 3090s. ## Why P2P Matters When you run a model across multiple GPUs with tensor parallelism, the GPUs need to constantly exchange data. Without P2P, all inter-GPU communication goes through the CPU and system memory - a round trip that adds latency and chews through PCIe bandwidth inefficiently. With P2P enabled, GPUs can read and write directly to each other's memory over DMA, cutting out the middleman entirely. Despite NVIDIA charging a premium for their consumer 90 series cards (RTX 3090, 4090, 5090) they block this feature in the driver. The hardware supports it fine in my experience - it's purely a product segmentation decision to push people towards their professional lineup where P2P is enabled. ## The Patch A community fork of NVIDIA's open GPU kernel modules exists that aids with removing this restriction: [aikitoria/open-gpu-kernel-modules](https://github.com/aikitoria/open-gpu-kernel-modules). The patch modifies the kernel driver to force enable BAR1 P2P mode on GPUs that aren't supposed to have it. Transfers then happen by writing directly to the other GPU's physical addresses over DMA. ### Prerequisites IOMMU must be configured in passthrough mode. Without this, DMA transfers between GPUs will fail. Note that passthrough mode reduces IOMMU's isolation guarantees - if you're running untrusted software or devices in otherwise highly secure operating environments, understand the trade-off before doing this. Edit `/etc/default/grub` and add `amd_iommu=on iommu=pt` (or `intel_iommu=on iommu=pt` for Intel) to `GRUB_CMDLINE_LINUX`: ```bash GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/root amd_pstate=guided nvidia-drm.modeset=1 amd_iommu=on iommu=pt" ``` Then regenerate grub and reboot: ```bash # Fedora: sudo grub2-mkconfig -o /boot/grub2/grub.cfg sudo dracut --regenerate-all --force # Debian-based distros: # sudo update-grub ``` ### Installing the Patched Drivers 1. Find the latest supported patched driver version by looking at the latest branch in [aikitoria/open-gpu-kernel-modules](https://github.com/aikitoria/open-gpu-kernel-modules) (`590.48.01` at the time of writing this) 2. Download the matching [NVIDIA driver](https://www.nvidia.com/en-us/drivers/details/258750/) version 3. When installing the driver select the open source driver (so we can modify it with the patch) 4. Clone the repo and run `./install.sh`, when it completes it will show the nvidia-smi output 5. Reboot ### Verify P2P is Working ``` nvidia-smi topo -p2p r GPU0 GPU1 GPU0 X OK GPU1 OK X Legend: X = Self OK = Status Ok CNS = Chipset not supported GNS = GPU not supported TNS = Topology not supported NS = Not supported U = Unknown ``` If you see `OK` between your GPUs instead of `NS`, P2P is active. --- ## Patching vLLM My daily LLM inference server is llama.cpp, which is far more user friendly and flexible than vLLM, however vLLM's performance is usually superior as such for especially heavy or performance sensitive workloads I sometimes switch to spinning up vLLM. vLLM has a very efficient implementation of tensor parallelism especially for MoE models, so for workloads that need those features it's often the best choice if you can wear the pain points. vLLM comes with many downsides: - It's more difficult to setup - Every model requires its own set of configuration parameters - Its topology constraints can be sub-optimal for setups with an uneven number of GPUs - It has limited support for quantisation formats such as GGUF, it does support AWQ quants but they're not as common, especially at anything other than 4-bit and 8-bit (dynamic) precision - Its vRAM -> RAM offloading functionality is experimental and not as robust or flexible as llama.cpp's, so it's only suitable if the models will fit entirely in GPU memory - vLLM's OpenAI compatible API server requires setting static configuration at launch. This means that the model must be set to load when the server starts (no hot-swapping / dynamic loading) and does not free up vRAM when models are idle. _Note: This could be worked around by coupling with MostlyGeek's fantastic [llama-swap](https://github.com/mostlygeek/llama-swap) project but that's beyond the scope of this post._ To top it all off - vLLM has a device capability check that rejects P2P on consumer GPUs - even if they're compatible (useful, no?). Even with the driver patch applied and P2P operational, vLLM will refuse to use it. It turns out the check lives in `vllm/platforms/cuda.py` and queries `pynvml` for P2P support - which still reports the card as unsupported because the user-space library vLLM uses has hard coded logic and as such - doesn't know about the support that's been added to our driver. I bypassed this with a one-liner in my vLLM Dockerfile: ```dockerfile RUN sed -i 's/handles = \[pynvml.nvmlDeviceGetHandleByIndex(i) for i in physical_device_ids\]/return True/g' \ /usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py ``` This is a dirty hack - it short-circuits the P2P check to always return `True`. It works, but you'd want to be more surgical about it if you're doing this for anything beyond a personal inference server. --- ## Tuning vLLM's Fused MoE Kernel ### What is fused_moe? MoE models (like Qwen 3.5) route each token through a subset of "expert" sub-networks. At inference time, this means running a batch of matrix multiplications - one per active expert - every forward pass. The naive approach runs these as separate GEMM operations, but vLLM fuses them into a single Triton kernel called `fused_moe`. This kernel combines the routed weight application, the matrix multiplications, and the activation function into one GPU dispatch, avoiding the overhead of launching many small kernels separately. The performance of this kernel depends heavily on how its work is tiled across the GPU's streaming multiprocessors. Parameters like block sizes (`BLOCK_SIZE_M`, `BLOCK_SIZE_N`, `BLOCK_SIZE_K`), the number of warps, pipeline stages, and group sizes all affect how efficiently the kernel uses the GPU's compute units, shared memory, and L2 cache. ### Why tune it? vLLM ships with pre-tuned configurations for datacenter GPUs (A100, H100, B200, etc). Consumer cards like the RTX 3090 are not included. Without a tuned config, vLLM falls back to heuristic defaults and logs a warning: ``` WARNING ... Using default MoE config. Performance might be sub-optimal! ``` The defaults are conservative and leave performance on the table. The RTX 3090's Ampere architecture has different SM counts, shared memory sizes, and cache hierarchies compared to the datacenter cards these defaults target. ### Generating a tuned config vLLM includes a benchmarking script that searches over the Triton kernel parameter space and finds the fastest configuration for your specific hardware, however it takes a very long time to run (many hours) as its search space is huge (~1900 configurations per batch size). I came across [MissionSquad/vllm-moe-configs](https://github.com/MissionSquad/vllm-moe-configs) repo, which provides an incredibly useful patch that makes this tuner workable on single-GPU consumer setups by narrowing the search space from ~1900 configurations down to a manageable number while retaining most of the performance benefits. The output is a JSON file named by expert count, intermediate size, and device - for example `E=48,N=768,device_name=NVIDIA_GeForce_RTX_3090.json`. Each entry maps a batch size to the optimal block dimensions, warp count, and pipeline stages: ```json { "triton_version": "3.5.1", "1": { "BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 128, "GROUP_SIZE_M": 32, "num_warps": 8, "num_stages": 3 }, "16": { "BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 128, "GROUP_SIZE_M": 16, "num_warps": 8, "num_stages": 3 }, ... } ``` The full config covers batch sizes from 1 through 4096. I've published mine as a [gist](https://gist.github.com/sammcj/e45c2ad08a4191f0fbbaa1b842a5a778). ### Using the tuned config Drop the JSON file into vLLM's config directory or point vLLM at it with the `VLLM_TUNED_CONFIG_FOLDER` environment variable: ```bash VLLM_TUNED_CONFIG_FOLDER=/path/to/configs vllm serve ... ``` On startup, vLLM should no longer print the "Using default MoE config" warning. The tuned parameters ensure the Triton kernel is compiled with block sizes and pipeline depths that match what the RTX 3090 can actually execute efficiently, rather than relying on conservative guesses. ## Results Running Qwen 3.5 35b-a3b (a MoE model with ~3 billion active parameters per token) with 64k context on dual RTX 3090s: | Setting | Value | Without P2P | | ---------------------- | --------------------- | ----------- | | vLLM version | | | | llama.cpp version | | | | Model | Qwen 3.5 35b-a3b | | | Context length | 64k | | | vLLM Quantisation | 4-bit (Dynamic) AWQ | | | llama.cpp Quantisation | Unsloth UD-Q4_K_XL | | | Inference engine | vLLM | | | GPUs | 2x RTX 3090 | | | CPU | Ryzen 9 9900X 12-core | | | RAM | 192GB DDR5 3600 MT/s | | | P2P | Enabled | Disabled | | Fused MoE config | Tuned | Tuned | | Server/Model | Value | Without P2P | | ----------------------- | ---------------- | ----------- | | **llama.cpp / 35b-a3b** | **110-130 tk/s** | Untested | | **vLLM / 35b-a3b** | **90-197 tk/s** | ~35-65 tk/s | | **llama.cpp / 27b** | **30-70 tk/s** | Untested | | **vLLM / 27b** | **32-195 tk/s** | ~32-70 tk/s | That's a 10-30% improvement over the same configuration without the P2P patch and MoE tuning, depending on the workload. The gain varies with batch size, sequence length, and how much inter-GPU communication the model actually needs. MoE architectures like Qwen 3.5 benefit particularly well since expert routing creates more cross-GPU traffic during tensor parallel inference, and the fused_moe kernel is on the critical path for every token generated. _Note: I'd love to break down the performance improvements in more detail including without the Fused MoE - perhaps when time allows I'll update this post._ ## Configuration ### vLLM #### Dockerfile ```dockerfile FROM vllm/vllm-openai:cu130-nightly ENV LD_LIBRARY_PATH=/usr/lib64:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/lib/x86_64-linux-gnu:/usr/local/cuda/lib64 ENV CUDA_VERSION=130 ENV UV_TORCH_BACKEND=cu130 # Install latest transformers, numba and vLLM RUN uv pip install --system -U https://github.com/huggingface/transformers/archive/refs/heads/main.zip RUN uv pip install --system --force-reinstall numba RUN uv pip install --system -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly # Fix for RTX 3090 with P2P enabled via patched driver RUN sed -i 's/handles = \[pynvml.nvmlDeviceGetHandleByIndex(i) for i in physical_device_ids\]/return True/g' /usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py EXPOSE 8000 ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"] ``` #### Docker Compose ```yaml services: &name vllm: container_name: *name hostname: *name profiles: - *name build: context: ./vllm # Dockerfile below ipc: host environment: # NVIDIA settings NVIDIA_VISIBLE_DEVICES: all NVIDIA_DRIVER_CAPABILITIES: all NVIDIA_DISABLE_REQUIRE: '1' CUDA_VISIBLE_DEVICES: '0,1' CUDA_DEVICE_ORDER: FASTEST_FIRST CUDA_DEVICE_MAX_CONNECTIONS: 10 CUDA_CACHE_MAXSIZE: 4294967296 # 4GB CUDA_SCALE_LAUNCH_QUEUES: '4x' NCCL_CUMEM_ENABLE: 0 RAY_memory_monitor_refresh_ms: 0 # https://docs.vllm.ai/en/latest/configuration/env_vars/ FLASH_ATTN: 1 OMP_NUM_THREADS: 6 PYTORCH_ALLOC_CONF: 'expandable_segments:False' TENSOR_PARALLEL_SIZE: 2 VLLM_CPU_KVCACHE_SPACE: 8 # (gb) VLLM_DO_NOT_TRACK: 1 VLLM_ENABLE_CUDAGRAPH_GC: 1 VLLM_FLASHINFER_MOE_BACKEND: throughput VLLM_MARLIN_USE_ATOMIC_ADD: 1 VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: 1 VLLM_TARGET_DEVICE: cuda VLLM_USE_DEEP_GEMM: 0 VLLM_USE_FLASHINFER_SAMPLER: '1' VLLM_USE_PRECOMPILED: 1 VLLM_TUNED_CONFIG_FOLDER: /tuned_configs # Contains my fused MoE kernels command: - '--served-model-name' - 'vLLM' - '--tensor-parallel-size' # split across 2 GPUs - '2' - "--attention-backend" - "FLASHINFER" # memory optimisation # https://docs.vllm.ai/en/stable/configuration/optimization.html - '--max-model-len' - '131070' - '--max-num-batched-tokens' - '16384' - '--max-num-seqs' - '8' - '--gpu-memory-utilization' - '0.95' - '--swap-space' - '16' - '--block-size' - '32' ## Qwen 3.5 specific settings - '--model' - 'cyankiwi/Qwen3.5-35B-A3B-AWQ-4bit' # 35b-a3b # - 'cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4' # 27b - linear attention layers kept at full-precision - '--language-model-only' # disables vision if not needed - '--tool-call-parser' - 'qwen3_coder' - '--reasoning-parser' - 'qwen3' - '--enable-auto-tool-choice' - '--speculative-config' # speculative decoding (drafting) using MTP (model-based token prediction) - '{"method":"mtp","num_speculative_tokens":2}' - '--override-generation-config' - '{"temperature": 0.7, "repetition_penalty": 1.0, "top_k": 20}' - '--mamba-cache-mode' - 'all' - '--enable-prefix-caching' - '--enable-chunked-prefill' # - '--enable-expert-parallel' # MoE models only, disable for 27b model, note: Disabled at present, see below for details runtime: nvidia deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: ['compute', 'utility', 'graphics', 'video'] - driver: cdi device_ids: - nvidia.com/gpu=all capabilities: ['compute', 'utility', 'graphics'] ``` #### vLLM Expert Parallelism I did some testing with vLLM's expert parallelism (EP) feature (`--enable-expert-parallel`) but found that it actually slightly decreased performance. | Metric | EP off | EP on | | --------------------- | ------- | ------- | | Short request tk/s | 82.7 | 71.9 | | Medium request tk/s | 69.6 | 67.7 | | Long request tk/s | 164.9 | 168.6 | | Tail tk/s | 39.9 | 20.7 | | Spec acceptance range | 76-88% | 70-77% | | KV cache tokens | 232,624 | 236,912 | EP on gave slightly more KV cache (experts split = less per-GPU model memory), but throughput is consistently lower on short/medium requests and the speculative decode acceptance rate drops. The long request window is a wash (168 vs 165). I came to the conclusion that the cross-GPU expert routing overhead is eating more than the VRAM savings give back and am keeping it off for now. ### llama.cpp I use [llama-swap](https://github.com/mostlygeek/llama-swap) for hot loading models with llama.cpp, my real configuration uses a number of macros so I've compiled relevant parts of the config roughly as follows: ```yaml models: "qwen3-5-35b-a3b-ud-q4kxl-64k-coding-instruct": ttl: 300 # 5 minutes env: - CUDA_SCALE_LAUNCH_QUEUES=4x - CUDA_CACHE_MAXSIZE=4294967296 - GGML_CUDA_GRAPH_OPT=1 - LLAMA_SET_ROWS=1 - LLAMA_ARG_KV_SPLIT=false cmd: > # server --slots --metrics --props --threads -1 --threads-batch -1 --threads-http -1 --prio-batch 2 --prio 2 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 # note: don't use --cache-reuse with Qwen 3.5 MoE models until https://github.com/ggml-org/llama.cpp/issues/18497 is resolved --no-context-shift --keep -1 --cache-reuse 256 --cache-ram -1 --kv-unified --no-mmap --jinja --fit on # 64k context --ctx-size 65535 # ngram speculative decoding --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64 # qwen 3.5 - set for coding (instruct, no thinking) --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.00 --presence-penalty 0.0 --repeat-penalty 1.0 --chat-template-kwargs "{\"enable_thinking\": false}" --model /models/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf ``` Note: These parameters are not necessarily the perfect optimally tuned settings as the model only came out yesterday, so I haven't had a lot of time to experiment with it yet. If you have a recommendation for better settings for this model [please let me know](https://smcleod.net/contact/)! --- Enjoy! - [The Role Bridging Problem](https://smcleod.net/2025/11/the-role-bridging-problem/index.md): The Role Bridging Problem --- title: "The Role Bridging Problem" date: 2025-11-20T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI", "LLM", "tech", "agentic coding", "mcp", "learning", "software engineering"] author: "Sam McLeod" draft: false summary: "An observation on functional correctness without domain quality." searchHidden: false cover: image: "facade-of-competence-jpeg-2260.jpg" alt: "The Facade of Competence" --- An observation on functional correctness without domain quality. AI tools are enabling people to work beyond their existing capabilities to learn and contribute in places they wouldn't otherwise. The risk emerges when operating outside one's area of expertise - when the ability to generate output outpaces the ability or incentive to evaluate its quality. The technology makes this possible, left unchecked, the results demonstrate why domain expertise still matters. ## The Expertise Gap Without domain expertise to architect, guide and critique AI-generated work, two paths emerge: 1. Moving forward while being unaware of the quality and alignment gap that stems from a lack of domain expertise. 2. Giving up, thinking that the AI isn't capable of delivering quality results. Let's focus on the first path - the more insidious one. Code gets written, interfaces get built, documents get produced, and they all _look_ correct enough to ship. The output appears competent, so it moves forward without the critical evaluation that domain expertise provides. This is particularly problematic because the work is often functionally correct. Tests pass, the UI renders, the API responds. What's missing is the domain mindset - the architectural considerations, the edge case handling, the maintainability concerns that come from actually understanding the domain you're working in. It's the ultracrepidarian's dilemma: AI provides the competence to produce, but not the expertise to evaluate. Consider these scenarios: - A data analyst uses AI to submit a pull request to codebases they otherwise wouldn't touch - they're producing code, but not applying software engineering principles. - A front-end developer generates backend code that works today but lacks the patterns that make it work reliably at scale. - Designers use tools like Figma AI or V0 to generate frontend code where the interface looks polished but the code underneath is unmaintainable or introduces tight coupling. - Engineers draft proposals that read coherently but lack the strategic thinking and stakeholder awareness that experienced BAs and product owners would bring. The work ships because it superficially meets requirements. The problems emerge later. ## Technical Debt at Scale The real cost isn't visible immediately. It accumulates over time as these outputs compound: **Erosion of quality standards**: As AI-generated output that "looks good enough" ships more frequently, teams may unconsciously lower their quality bar. The distinction between functional correctness and engineering quality blurs. **Maintenance burden**: Code that works but wasn't built with an engineering mindset becomes increasingly expensive to modify. Each change requires understanding and working around patterns that don't align with the domain's best practices. **Knowledge gaps**: When people contribute code to systems they aren't capable of understanding, the codebase becomes fragmented. Different sections follow different patterns, making it harder for the team to maintain a coherent mental model. **Review fatigue**: Development teams spend increasing time either rejecting pull requests from well-intentioned contributors or cleaning up merged code that technically functions but violates fundamental principles. **False velocity**: Teams may appear to move faster because more work ships, but the accumulated friction slows future development. Technical debt grows faster than it's addressed. The pattern is consistent across domains: functional correctness without domain quality. The code runs, the document reads coherently, the interface renders - but none of it reflects the practices and considerations that come from actual expertise in that area. ## Moving Forward AI tools excel at acceleration, automation and helping us learn. The issue isn't the tools themselves, but how we apply them. Engineering problems require an engineering mindset - the critical thinking to evaluate outputs, iterate on solutions, and recognise when generated code lacks the architectural patterns or error handling that deliver high quality outcomes. The solution isn't prescriptive rules about what you can and can't do with AI, or constraints that lock you into your initial capabilities. Use AI to accelerate your work and learn adjacent domains, but apply critical thinking to the outputs. When you're working outside your expertise, ask questions. Involve people who live in that domain and learn from them. Push back when managers expect AI tooling to mean you can single-handedly deliver work that spans multiple specialisms. Good engineering still requires good engineers - AI doesn't change that, but it can create the illusion of competence without the substance. ## Bridging the Gap If you're using AI to work in adjacent domains, there are practical steps to improve the quality of the artefacts you produce: **Extend AI capability beyond base knowledge**: Use Model Context Protocol (MCP) tools to connect AI to documentation, code repositories, and domain-specific knowledge. Rather than simply relying on the model's internal training data that will be outdated and is certainly generic. **Constrain and guide with agent rules**: Agent rules (really more suggestions than rules) help to reinforce boundaries, patterns, and standards. Whenever AI repeatedly fails to follow instructions or operate outside your intended scope - that is an opportunity to consider adding or refining your agent rules. **Document and have AI operate from a plan**: Establish quality gates, good workflows help to keep agents on task and create checkpoints where domain experts review outputs before they move forward. **Recognise your evaluation limits**: You're probably outside your ability to properly judge AI outputs when: - You're accepting code because it runs, not because you understand (or could understand) its architectural implications - You can't articulate why one approach might be better than another in that domain - You're not certain which edge cases matter or how they should be handled - You find yourself defending AI outputs without being able to explain the trade-offs If you recognise these patterns, consider involving a domain expert. It's not about gate-keeping, it's about leveraging people with the right skills, experience and judgement to help drive high quality outcomes. - [Stop Polluting Context - Let Users Disable Individual MCP Tools](https://smcleod.net/2025/08/stop-polluting-context-let-users-disable-individual-mcp-tools/index.md): Stop Polluting Context - Let Users Disable Individual MCP Tools --- title: "Stop Polluting Context - Let Users Disable Individual MCP Tools" date: 2025-08-26T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI", "LLM", "tech", "mcp", "cline", "claude", "agentic", "agentic coding", "mcp", "security"] author: "Sam McLeod" draft: false summary: "If you're building MCP servers, you should be adding the ability to disable individual tools." searchHidden: false cover: image: "tool-context-usage.png" alt: "MCP Server Tool Context Usage" # caption: "Optional caption text" --- **TLDR**: If you're building MCP servers, let users disable individual tools. It's simple to implement and your context window will thank you. ![alt text](tool-context-usage.png) Agents often don't need _every_ tool a MCP server provides. Allowing individual tools to be disabled can not just vastly improve token usage, it aids with tool use reliability, the quality of the model's output and your security posture. - **Optimises context use** by reducing token count when running many MCP servers - **Reduces [context rot](https://research.trychroma.com/context-rot)** - we know that model performance grows increasingly unreliable as input length grows and that irrelevant information acts as distractors that degrade performance - **Improves security posture** by reducing attack surface - disable risky operations in production while keeping read-only tools available - **Enables finer-grained control** over tool access for organisational compliance Want the AWS Terraform server but you don't use Terragrunt? Disable the Terragrunt tool (save 1.1k tokens). Want web search but not video and maps search? Disable the maps and video tools. ## The Problem is Real MCP servers add invaluable capabilities to LLMs and agents, however - unchecked the impact of their many tools quickly becomes apparent. The numbers are worse than you think: | MCP Server | Tool Name | Tokens | |-----------------------|--------------------------------|----------------------| | **AWS Terraform** | | **6.4k** (7 tools) | | | ExecuteTerragruntCommand | 1.1k | | | SearchAwsccProviderDocs | 1.1k | | | SearchAwsProviderDocs | 905 | | | SearchUserProvidedModule | 894 | | | RunCheckovScan | 806 | | | SearchSpecificAwsIaModules | 849 | | | ExecuteTerraformCommand | 743 | | **AWS Cost Explorer** | | **9.1k** (7 tools) | | | get_cost_and_usage | 2.2k | | | get_cost_comparison_drivers | 1.9k | | | get_cost_and_usage_comparisons | 1.5k | | | get_cost_forecast | 1.4k | | | get_dimension_values | 910 | | | get_tag_values | 699 | | | get_today_date | 469 | | **Playwright** | | **9.7k** (21 tools) | | | browser_take_screenshot | 631 | | | browser_fill_form | 593 | | | browser_type | 545 | | | browser_click | 516 | | | browser_select_option | 501 | | | _...plus 16 more tools_ | _390-469_ | | **Markdownify** | pptx-to-markdown | **4.0k** (10 tools) | | | | 406 | | | bing-search-to-markdown | 405 | | | audio-to-markdown | 404 | | | docx-to-markdown | 403 | | | image-to-markdown | 402 | | | _...plus 5 more tools_ | _394-400_ | | **ShadCN UI** | get_directory_structure | **3.0k** (7 tools) | | | | 478 | | | get_component_demo | 427 | | | get_component | 422 | | | get_component_metadata | 422 | | | list_blocks | 405 | | | _...plus 2 more tools_ | _376-467_ | **That's over 32k tokens _just for tool descriptions & annotations_ - before you've written a line of code.** Individual tools range from lightweight (387 tokens for basic memory operations) to massive (2.7k tokens for AWS's documentation generator). The AWS Cost Explorer's `get_cost_and_usage` tool alone burns 2.2k tokens describing parameters you might never use. Fortunately, implementing selective tool disabling is straightforward and the benefits are immediate. ## Implementation Advice Start simple. The conditional registration approach requires minimal code and covers most use cases. If you need runtime flexibility later, you can always refactor to a registry pattern. - **Parse once**: Process environment variables at startup, not per-request - **Normalise input**: Handle whitespace, case variations, and naming conventions (`underscore_tool` vs `hyphen-tool`) - **Use map lookups**: `O(1)` performance with `map[string]bool` over slice iteration - **Case insensitivity**: Match both `ToolName` and `toolname` - **Whitespace**: `" tool1 , tool2 "` should be equivalent to `"tool1,tool2"` - **Empty strings**: `DISABLED_TOOLS=",tool1,,tool2,"` should work - **Non-existent tools**: Disabling `"non-existent_tool"` shouldn't break anything - **Fail gracefully**: Invalid tool names shouldn't crash your server ## Example Implementation Approaches I've implemented tool disabling in both my [mcp-devtools server](https://github.com/sammcj/mcp-devtools) with a registry based approach,and recently raised a [PR to AWS's MCP servers](https://github.com/awslabs/mcp/pull/1178) with a simpler conditional utility. Both work well and take slightly different approaches. ### Simple Conditional Registration In my [PR](https://github.com/awslabs/mcp/pull/1178) to AWS's MCP servers, I used straightforward conditional registration. Here's the pattern: ```python from awslabs.common.config import tool_enabled if tool_enabled('ExecuteTerraformCommand'): @mcp.tool(name='ExecuteTerraformCommand') async def execute_terraform_command(...): # existing implementation unchanged ``` The utility function parses a comma-separated environment variable: ```python def disabled_tools() -> Set[str]: disabled_tools = os.environ.get('DISABLED_TOOLS', '') if not disabled_tools: return set() return {tool.strip() for tool in disabled_tools.split(',') if tool.strip()} def tool_enabled(tool_name: str) -> bool: return tool_name not in disabled_tools() ``` Configuration is simple: ```json { "env": { "DISABLED_TOOLS": "ExecuteTerraformCommand,RunCheckovScan" } } ``` ### Registry-Based Runtime Filtering My [MCP DevTools](https://github.com/sammcj/mcp-devtools) server uses a more sophisticated approach with runtime filtering: ```go // Tools register normally during init func init() { registry.Register(&ThinkTool{}) } // But filtering happens at retrieval func GetTool(name string) (tools.Tool, bool) { if disabledFunctions[name] { return nil, false } return toolRegistry[name], true } ``` This approach allows for more complex filtering logic and better separation of concerns. The trade-off is slightly more implementation complexity. ## Just Do It Having a pattern for selective tool disabling is something I now add to all MCP servers that I develop and recommend to others. The implementation effort is minimal, but the user value - and performance improvement - is significant. If you're maintaining an MCP server, add this feature. Your users are already thinking about their token budgets and context quality - help them optimise both. --- ## Appendix - A Note On The Worst Offender: Github GitHub's MCP server is a masterclass in context pollution. Where other servers hover around 3-10k tokens, **The GitHub MCP consumes a staggering 46k tokens across 91 different tools** that cannot be individually enabled or disabled. GitHub alone consumes nearly a quarter of Claude Sonnet / Opus 4's context window - before you even write a single line of code. > _For comparison - [MCP-DevTools](https://github.com/sammcj/mcp-devtools) provides the most useful GitHub functionality in just 1.6k tokens with a single tool._ Github's MCP fails to: - Group functionality into broader tools (e.g. `list_issues`, `search_issues`, `update_issue` could be a single `issue_management` tool). - Keep tool description and parameter annotations concise. - Provide any mechanism to only enable individual tools. ![Github's Horrible MCP Server](github-dafaq.png) - [MCP DevTools](https://smcleod.net/2025/08/mcp-devtools/index.md): MCP DevTools --- title: "MCP DevTools" date: 2025-08-18T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI", "LLM", "tech", "coding", "cline", "claude", "agentic", "agentic coding", "mcp", "security"] author: "Sam McLeod" draft: false description: "A single, modular MCP server for AI coding agents." searchHidden: false cover: image: "sec-arch.png" alt: "Security middleware blocking potentially malicious content" # caption: "Optional caption text" --- [MCP DevTools](https://github.com/sammcj/mcp-devtools) - The one tool that replaced the 10-15 odd NodeJS/Python/Rust MCP servers I had running at any given to for agentic coding tools with a single server that provides tools I consider useful for agents when coding. ## The Problem The MCP ecosystem has grown rapidly, but I found myself managing many separate servers, each often running multiple times for every MCP client I had running, not to mention the ever growing memory and CPU consumption of the many NodeJS or Python processes. I wanted something that: - Compiled to a single binary with minimal dependencies - Provided consistent, well-tested tools - Could be easily extended with new functionality - Offered proper security controls for production use - Performed well under load ## Architecture and Design The server is modular in design with a "tool registry" that allows me to add new tools as needed and have them presented to MCP clients in a standard way. The server is written in Golang and compiles to a single binary that can be run in STDIO, SSE or Streamable HTTP mode. ```mermaid graph LR A[MCP DevTools
Server] A --> B[Search &
Discovery] A --> C[Document
Processing] A --> D[Security
Analysis] A --> E[Intelligence &
Memory] A --> F[Utilities] A --> G[Agents] B --> B_Tools[🌐 Internet Search
📡 Web Fetch
📦 Package Search
📚 Package Documentation
🐙 GitHub
🎨 ShadCN UI Components
🔌 API Integration
☁️ AWS DocumentationTerraform Documentation] C --> C_Tools[📄 Document Processing
📑 PDF Processing] D --> D_Tools[📋 SBOM Generation
🛡️ Vulnerability Scan
🔒 Security Framework
🛠️ Security Override] E --> E_Tools[🧠 Think Tool
🔢 Sequential Thinking
🕸️ Memory Graph] F --> F_Tools[🧮 Calculator
🇬🇧 American→English
📁 Filesystem
📝 Changelog Generation] G --> G_Tools[🤖 Claude Code
✨ Gemini CLI
🅰️ Q Developer] classDef inputOutput fill:#FEE0D2,stroke:#E6550D,color:#E6550D classDef llm fill:#E5F5E0,stroke:#31A354,color:#31A354 classDef components fill:#E6E6FA,stroke:#756BB1,color:#756BB1 classDef process fill:#EAF5EA,stroke:#C6E7C6,color:#77AD77 classDef stop fill:#E5E1F2,stroke:#C7C0DE,color:#8471BF classDef data fill:#EFF3FF,stroke:#9ECAE1,color:#3182BD classDef decision fill:#FFF5EB,stroke:#FD8D3C,color:#E6550D classDef storage fill:#F2F0F7,stroke:#BCBDDC,color:#756BB1 classDef api fill:#FFF5F0,stroke:#FD9272,color:#A63603 classDef error fill:#FCBBA1,stroke:#FB6A4A,color:#CB181D class A components class B,B_Tools decision class C,C_Tools api class D,D_Tools error class E,E_Tools data class F,F_Tools process class G,G_Tools llm ``` Each tool implements a standard interface: ```go type Tool interface { Definition() mcp.Tool Execute(ctx context.Context, logger *logrus.Logger, cache *sync.Map, args map[string]interface{}) (*mcp.CallToolResult, error) } ``` This registers the tool automatically during startup, making it trivial to add new functionality. The registry handles discovery, validation, and execution whilst providing shared services like caching, logging and security hardening middleware. The server supports three transport modes: - **STDIO**: Direct communication for running locally - **Streamable HTTP**: Networked deployments with optional authentication and centralised configuration --- ## Tooling Overview These tools can be disabled by adding their function name to the `DISABLED_FUNCTIONS` environment variable in your MCP configuration. | Tool | Purpose | Dependencies | Example Usage | |-------------------------------------------------------------------------------------------------------------------|------------------------------------|-------------------------------|---------------------------------| | **[Internet Search](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/internet-search.md)** | Multi-provider web search | None (Provider keys optional) | Web, image, news, video search | | **[Web Fetch](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/web-fetch.md)** | Retrieve web content as Markdown | None | Documentation and articles | | **[GitHub](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/github.md)** | GitHub repositories and data | None (GitHub token optional) | Issues, PRs, repos, cloning | | **[Package Documentation](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/package-documentation.md)** | Library documentation lookup | None | React, Django, TensorFlow docs | | **[Package Search](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/package-search.md)** | Check package versions | None | NPM, Python, Go, Java, Docker | | **[Think](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/think.md)** | Structured reasoning space | None | Complex problem analysis | | **[Find Long Files](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/find_long_files.md)** | Identify files needing refactoring | None | Find files over 700 lines | | **[Calculator](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/calculator.md)** | Basic arithmetic calculations | None | 2 + 3 * 4, batch processing | | **[DevTools Help](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/devtools_help.md)** | Extended info about DevTools tools | None | Usage examples, troubleshooting | These tools can be enabled by setting the `ENABLE_ADDITIONAL_TOOLS` environment variable in your MCP configuration. | Tool | Purpose | `ENABLE_ADDITIONAL_TOOLS` | Example Usage | |-----------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|---------------------------|---------------------------------------------| | **[Filesystem](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/filesystem.md)** | File and directory operations | `filesystem` | Read, write, edit, search files | | **[American→English](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/american-to-english.md)** | Convert to British spelling | `murican_to_english` | Organise, colour, centre | | **[ShadCN UI Component Library](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/shadcn-ui.md)** | Component information | `shadcn` | Button, Dialog, Form components | | **[Memory](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/memory.md)** | Persistent knowledge graphs | `memory` | Store entities and relationships | | **[SBOM Generation](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/sbom.md)** | Generate Software Bill of Materials | `sbom` | Analyse project dependencies | | **[Vulnerability Scan](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/vulnerability_scan.md)** | Security vulnerability scanning | `vulnerability_scan` | Find security issues | | **[Generate Changelog](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/changelog.md)** | Generate changelogs from git commits | `generate_changelog` | Release notes from local/remote repos | | **[Document Processing](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/document-processing.md)** | Convert documents to Markdown | `process_document` | PDF, DOCX → Markdown with OCR | | **[PDF Processing](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/pdf-processing.md)** | Fast PDF text extraction | `pdf` | Quick PDF to Markdown | | **[AWS Documentation](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/aws_documentation.md)** | AWS documentation search and retrieval | `aws` | Search and read AWS docs, recommendations | | **[Terraform Documentation](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/terraform-documentation.md)** | Terraform Registry API access for providers, modules, and policies | `terraform_documentation` | Provider docs, module search, policy lookup | | **[Security Framework](https://github.com/sammcj/mcp-devtools/blob/main/docs/security.md)** (BETA) | Context injection security protections | `security` | Content analysis, access control | | **[Security Override](https://github.com/sammcj/mcp-devtools/blob/main/docs/security.md)** | Agent managed security warning overrides | `security_override` | Bypass false positives | | **[Sequential Thinking](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/sequential-thinking.md)** | Dynamic problem-solving through structured thoughts | `sequential-thinking` | Step-by-step analysis, revision, branching | | **[API to MCP](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/api.md)** | Dynamic REST API integration | `api` | Configure any REST API via YAML | **Agents as Tools** - In addition to the above tools, MCP DevTools can provide access to AI agents as tools by integrating with external LLMs. | Agent | Purpose | `ENABLE_ADDITIONAL_TOOLS` | |-----------------------------------------------------------------------------------------------------------|---------------------------|---------------------------| | **[Claude Agent](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/claude-agent.md)** | Claude Code CLI Agent | `claude-agent` | | **[Gemini Agent](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/gemini-agent.md)** | Gemini CLI Agent | `gemini-agent` | | **[Q Developer Agent](https://github.com/sammcj/mcp-devtools/blob/main/docs/tools/q-developer-agent.md)** | AWS Q Developer CLI Agent | `q-developer-agent` | ## Tooling Details ### Search and Discovery Tools **Internet Search** provides unified access to multiple search providers (Brave, SearXNG, DuckDuckGo) through a single interface. Rather than managing separate tools for each provider, agents can search web, images, news, videos, and local results with consistent parameters. The tool handles provider-specific requirements and normalises results. **GitHub** offers repository access without the context bloat of the official GitHub MCP server. It supports repository search, issue/PR management, file content retrieval, cloning, and GitHub Actions monitoring. The tool accepts flexible repository identifiers (owner/repo, URLs, or direct issue/PR links) and provides intelligent authentication via tokens or SSH keys. **Package Search** checks versions across NPM, PyPI, Go modules, Maven, Docker Hub, GitHub Actions, and AWS Bedrock through one interface. Instead of separate tools for each ecosystem, agents can batch-check dependencies, find latest versions, and validate constraints efficiently. **Package Documentation** fetches up-to-date library documentation using Context7's knowledge base. Combined with the package search tool, this provides a complete workflow for researching and implementing new dependencies. ### Intelligence and Reasoning **Think Tool** provides structured reasoning space for AI agents during complex troubleshooting. Based on Anthropic's research showing 54% improvement in complex scenarios, it allows agents to pause and analyse problems before taking action. The tool supports triggering Claude's different thinking intensities (`hard`, `harder`, `ultra`) for varying problem complexity. **Memory** maintains persistent knowledge graphs across sessions. Agents can store entities, relationships, and context that persists between conversations, enabling more sophisticated long-term reasoning. ### Document Processing **Document Processing** converts PDFs and DOCX files to Markdown using Python's Docling library. It supports OCR, hardware acceleration (MPS, CUDA), and intelligent content extraction with caching. **PDF Processing** provides fast PDF text extraction for simpler use cases where full document processing isn't needed. ### Security and Analysis The security framework deserves special attention. It's a configurable, multi-layered system that provides: - **Access Control**: Prevents tools from accessing sensitive files and domains - **Content Analysis**: Scans returned content for security threats using pattern matching - **Override System**: Allows bypassing false positives with audit logging Built-in protection includes shell injection detection, data exfiltration prevention, prompt injection mitigation, and sensitive file protection. The system uses YAML-based rules with automatic reloading and minimal performance impact when disabled. **SBOM Generation** and **Vulnerability Scanning** provide security analysis for projects, helping identify dependencies and potential security issues. ### Development Utilities **Filesystem** tool provides secure file operations with access controls. **Find Long Files** identifies files over 700 lines that might need refactoring. **Changelog Generation** creates release notes from git commits. **American→English** converts American spelling to British English - a small but appreciated touch for international users. ### External Agent Integration **Claude Agent** and **Gemini Agent** tools allow using other coding agents as sub-agents for delegated and specialised tasks. This enables a multi-agent workflow where different models can contribute their strengths. --- ## Security Framework I recently implemented a security frameworks which operates as a configurable middleware that can inspect content that's retrieved from an external source before the content makes its way back to the LLM, the intent being to help mitigate some prompt injection style attack vectors. The middleware currently has three functions when enabled it can be configured to: 1. Warn or block content retrieved from tools that can fetch external data (e.g. web scraping, internet search, document conversion etc…). 2. Block a list of domains, e.g. `*.ru`, `dodgy-site.com` etc. 3. Prevent the tools the server provides from being able to certain files, e.g. `id_rsa`, `*.pem` etc. ```go ops := security.NewOperations("tool-name") safeResp, err := ops.SafeHTTPGet(urlStr) if err != nil { // Handle security blocks or network errors return nil, err } ``` The framework is not designed to be solution for all possible threat vectors, by the nature of a tool designed for software development tasks, unless the internet search and file fetching functionality is disabled mcp-devtools and similar tools will always have the risk of fetching unwanted content, but I think it's a one line of defence and an interesting experiment. If you're interested of have ideas for improvement you can take a look here: - Docs: https://github.com/sammcj/mcp-devtools/blob/main/docs/security.md - Default rules and configuration: https://github.com/sammcj/mcp-devtools/blob/main/internal/security/default_config.yaml The system currently includes: - Pattern-based threat detection (basic shell injection, data exfiltration, prompt injection) - Domain and file access controls - Content size limits and entropy analysis - Audit logging with override capabilities Example of the security framework in action, blocking potentially malicious content fetched by a tool, before it reaches the LLM: {{< wide-image src="sec-block.png" alt="Security middleware blocking potentially malicious content" >}} ### Architecture {{< wide-image src="sec-arch.png" alt="Security framework architecture" >}} ### Action Processing ```mermaid flowchart TD A[Content Input] --> B{Security Check} B -->|Match Found| C{Action Type} B -->|No Match| D[Allow Content] C -->|allow/ignore| D C -->|warn/warn_high| E[Log Warning] C -->|block| F[Block Content] C -->|notify| G[Send Notification] E --> H[Return Content + Warning] F --> I[Return Error] G --> J[Return Content + Notice] classDef allow fill:#31A354 classDef warn fill:#E6550D classDef block fill:#EE2E2E classDef notify fill:#756BB1 class D,H,J allow class E warn class F,I block class G notify ``` --- ## OAuth Authentication As an exercise in learning OAuth for MCP, I've implemented OAuth 2.0/2.1 authentication with browser-based flows or resource server modes. This enables centralised user authentication and access controls both of the server itself, and to upstream services. ```mermaid graph TD subgraph "MCP DevTools OAuth Architecture" direction TB subgraph "OAuth Modes" BrowserMode[🌐 Browser Authentication Mode
OAuth Client] ResourceMode[🛡️ Resource Server Mode
Token Validation] end subgraph "OAuth Provider" AuthProvider[OAuth 2.1 Provider] AuthServer[Authorisation Server] TokenEndpoint[Token Endpoint] JWKSEndpoint[JWKS Endpoint] end subgraph "Browser Auth Flow" Browser[🌏 System Browser] CallbackServer[📡 Localhost Callback Server] PKCEGen[🔐 PKCE Challenge Generator] end subgraph "Resource Server Flow" TokenValidator[🔍 JWT Token Validator] JWKSFetch[🔑 JWKS Fetcher] AudienceCheck[🎯 Audience Validator] end User[👤 User] --> BrowserMode MCP_Client[📱 External MCP Client] --> ResourceMode BrowserMode --> PKCEGen BrowserMode --> Browser BrowserMode --> CallbackServer Browser --> AuthServer CallbackServer --> TokenEndpoint ResourceMode --> TokenValidator TokenValidator --> JWKSFetch TokenValidator --> AudienceCheck JWKSFetch --> JWKSEndpoint AudienceCheck --> AuthProvider end classDef browserAuth fill:#e1f5fe,stroke:#0277bd,color:#000 classDef resourceAuth fill:#f3e5f5,stroke:#7b1fa2,color:#000 classDef oauthProvider fill:#e8f5e8,stroke:#2e7d32,color:#000 classDef user fill:#fff3e0,stroke:#ef6c00,color:#000 class BrowserMode,Browser,CallbackServer,PKCEGen browserAuth class ResourceMode,TokenValidator,JWKSFetch,AudienceCheck resourceAuth class AuthProvider,AuthServer,TokenEndpoint,JWKSEndpoint oauthProvider class User,MCP_Client user ``` --- ## Getting Started Installation is simple: ```bash go install github.com/sammcj/mcp-devtools@HEAD ``` Basic configuration requires only the binary path: ```json { "mcpServers": { "dev-tools": { "type": "stdio", "command": "/path/to/mcp-devtools" } } } ``` Most tools work immediately, with optional API keys enabling additional providers (Brave Search, GitHub tokens, etc.). ## Conclusion MCP DevTools demonstrates that consolidating MCP functionality into a single binary solution that generally provides better performance, security, and maintainability than managing multiple separate servers. The modular architecture makes it easy to extend whilst the security framework provides an additional layer of protection fetch working with untrusted external content. - https://github.com/sammcj/mcp-devtools - [Agentic Coding Adoption Cost Cycle](https://smcleod.net/2025/07/agentic-coding-adoption-cost-cycle/index.md): Agentic Coding Adoption Cost Cycle --- title: "Agentic Coding Adoption Cost Cycle" date: 2025-07-16T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI", "LLM", "tech","coding", "cline", "agentic", "agentic coding"] author: "Sam McLeod" draft: false description: "The two common themes I see with engineers adopting agentic coding tools" searchHidden: false cover: image: "agentic-coding-adoption-cost-cycle.png" alt: "Agentic Coding Adoption Cost Cycle" # caption: "Optional caption text" --- {{< wide-image src="agentic-coding-adoption-cost-cycle.png" alt="Agentic Coding Adoption Cost Cycle" >}} - [Agentic Coding Workflow & Cline Demo](https://smcleod.net/2025/06/agentic-coding-workflow-cline-demo/index.md): Agentic Coding Workflow & Cline Demo --- title: "Agentic Coding Workflow & Cline Demo" date: 2025-06-20T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI", "LLM", "tech", "ollama", "coding", "cline", "agentic", "tutorials", "agentic coding", "demo", "video"] author: "Sam McLeod" draft: false description: "Recording of a demo and Q&A session on my Agentic Coding workflow and Cline use." searchHidden: false --- [Square Peg](https://www.squarepeg.vc/) hosted event on June 20, 2025 where I demonstrated a basic version of my daily Agentic Coding workflow using Cline and MCP tools. {{< youtube "dDSLw-6vR4o" >}} --- > _What does it take to write enterprise-grade code in the AI-native era?_ > > _Join Square Peg investors James Tynan and Grace Dalla-Bona for a live demo and Q&A session with three leading AI-native developers - Grant Gurvis, Listiarso Wastuargo, and Sam McLeod - and get a behind-the-curtain look at the workflows that enable them to ship faster, smarter, and cleaner code using tools like Cursor, Cline, and smolagents._ ![alt text](sqr-peg-socials.jpg) - [Vibe Coding vs Agentic Coding](https://smcleod.net/2025/06/vibe-coding-vs-agentic-coding/index.md): Vibe Coding vs Agentic Coding --- title: "Vibe Coding vs Agentic Coding" date: 2025-06-06T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI", "LLM", "tech", "ollama", "coding", "cline", "agentic", "tutorials", "llama", "agentic coding", "vibe coding"] author: "Sam McLeod" draft: false description: "From Creative Exploration to Production Quality" searchHidden: false --- Picture this: A business leader overhears their engineering team discussing "vibe coding" and immediately imagines developers throwing prompts at ChatGPT until something works, shipping whatever emerges to production. The term alone-"vibe coding"-conjures images of seat-of-the-pants development that would make any CTO break out in a cold sweat. This misunderstanding is creating a real problem. Whilst vibe coding represents genuine creative exploration that has its place, the unfortunate terminology is causing some business leaders to conflate all AI-assisted / accelerated development with haphazard experimentation. I fear that engineers using sophisticated AI coding agents be it with advanced agentic coding tools like Cline to deliver production-quality solutions are finding their approaches questioned or dismissed entirely. The irony is that these same business leaders are missing out on the transformative potential of agentic coding-a disciplined approach that treats AI as a skilled development partner rather than a random code generator. The difference isn't just semantic; it's the distinction between sustainable engineering practice and creative tinkering. To understand why this distinction matters, I like clarify what we're actually talking about into these two distinct approaches: **Vibe Coding** represents an experimental and creative process where users prompt their way to creating their idea. It's immediate, intuitive, and often produces remarkable results through iterative experimentation. **Agentic Coding** by contrast, focuses on leveraging AI coding agents as engineering *peers* that you can co-engineer solutions with. It's structured, deliberate, and designed for sustainable outcomes. The confusion between these approaches is costing businesses the opportunity to leverage AI's potential in a safe, secure and maintainable way. --- ## Vibe Coding: Fast, Fun and Creative And we should never want to take away from the people that are doing it, nor should we encourage gate keeping of AI coding tools, or AI in general. It's amazing what people can create with Vibe Coding - folks that may have never written more than a few lines of code can explore their creativity and have a great time building something that they would otherwise may have not been able to create. But it is important that we realise (at least with our current capabilities) that Vibe Coded applications may not deliver production-ready, maintainable, business aligned solutions, architected with an engineering mindset. Of course - not all solutions _need_ to be well engineered or production ready, but significant portion of applications found running in businesses today would have an impact if they were to fail or become unmaintainable. ## Agentic Coding: An AI Engineering Peer, Not a Magic Wand Agentic coding - while still leveraging AI coding tools takes a more engineered and extended approach. Rather than treating AI as a black box that spits out code, it positions AI agents as sophisticated development partners equipped with the right tools, context, data and constraints to deliver production-grade solutions. This isn't about asking an AI to write your entire application-it's about creating a more structured, measurable development process where AI agents work within defined parameters, armed with the right tools and knowledge to make informed decisions on tasks towards completing a goal. - Humans operating with an engineering mindset. - Significant value is placed on the time invested in design before implementation. - Agents enabled with tools to extend their capabilities and knowledge. - Agents aligned to business and technical requirements. - Leverage deterministic tools for deterministic outcomes. - Apply the same quality and security measures as you would for a human. - Enforce safety through sandboxing and scoped permissions rather than trusting the model to behave. It's a shift from treating AI as full auto-pilot to treating it as a peer developer who operates best with proper tooling and context. --- ## Agentic Coding: Workflow I like to follow a "[Setup → Plan → Act → Review and Iterate](https://smcleod.net/2025/04/my-plan-document-act-review-flow-for-agentic-software-development/)" workflow for agentic coding, which can loosely be broken down as: - **Setup**: Configure your coding agent with the right skills and tools, access to documentation, coding standards, and business context - **Plan**: Have the agent analyse requirements, propose architecture, and create a structured approach before writing code - **Act**: Let the agent implement solutions while adhering to established patterns and constraints, delegating research-heavy tasks to sub-agents with fresh context windows - **Review and Iterate**: Human oversight validates outputs, provides feedback, and guides refinements - with what you learn fed back into your rules, skills and plans Threaded through all four stages is context management. Models are stateless and their working memory (the context window) is finite - as it fills, they become noticeably less capable. Development plans, skills that load on demand and sub-agents with fresh context all exist to protect that limited working memory. ## Agentic Coding: Tooling A key component of agentic coding is extending your agent's capabilities and knowledge. This takes three main forms: rules and guidelines that steer behaviour (coding standards, conventions and guardrails in the agent's system prompt), Agent Skills that package up instructions, scripts and reference material the agent loads when relevant, and MCP servers that connect agents to external tools and data sources. _Update: When I first published this post, MCP servers were the main way to extend coding agents. Agent Skills have since replaced many of the MCP servers we were running - a small deterministic script inside a skill is often simpler and more reliable than a server doing the same job. MCP still has its place for external integrations and shared data sources, and the underlying principle hasn't changed: favour deterministic tools for deterministic outcomes._ For example: - Query package repositories (internal or private) for version availability - Access package, library and framework documentation or even example implementation patterns - Augment the coding style with language best practices, guidelines or internal coding standards - Integrate with existing development workflows and CI/CD - Accelerating security remediation by enabling the agent to run your security scanning tools and action any findings autonomously - Understand business or technical context and requirements resulting in aligned solutions This tooling transforms AI from a code generator into a contextually aware development partner. --- ## The Critical Contrasts | Aspect | Vibe Coding | Agentic Coding | |:----------------------------|:----------------------------------------------------|:-----------------------------------------------------------------| | **Primary Approach** | Creative experimentation through prompting | Structured collaboration with AI as development peer | | **Skills Required** | Minimal technical background needed | Strong engineering fundamentals and system design skills | | **Planning Phase** | Iterative discovery purely through trial and error | Upfront analysis, architecture design, and requirement mapping | | **Tool Integration** | Usually minimal | Agent Skills, MCP tools, documentation access and toolchains | | **Quality Assurance** | Manual testing of immediate outputs | Automated testing, code review, and adherence to standards | | **Business Alignment** | Solutions may drift from actual requirements | Explicitly designed for business needs and technical constraints | | **Code Maintainability** | Often produces write-once, difficult-to-modify code | Emphasises readable, documented, and extensible solutions | | **Technical Debt** | Accumulates rapidly without structured oversight | Actively managed through proper engineering practices | | **Scalability** | Struggles with complex, multi-component systems | Designed for enterprise-scale, distributed architectures | | **Security Considerations** | Ad-hoc, often overlooked in creative process | Integrated security scanning and compliance from the start | | **Time to Demo** | Very fast for proof-of-concepts | Slightly higher initial investment, faster long-term iterations | | **Production Readiness** | Requires significant refactoring for production | Built with production deployment and operations in mind | **Key Insight**: The fundamental difference isn't about speed or AI capability-it's about treating development as either creative exploration or an engineering practice. ## The Enterprise Imperative As organisations increasingly rely on AI-assisted development, the choice between vibe and agentic coding becomes a strategic decision. Vibe coding might get you to a demo faster, but agentic coding gets you to sustainable, scalable solutions that can form the backbone of real businesses. The key is understanding that effective agentic coding isn't about replacing human developers-it's about augmenting human capabilities with AI agents that have the right tools, context, and constraints to deliver professional-grade results. --- ## For Those Making the Transition If you're currently vibe coding and want to transition to agentic coding for production work, here's how to make that shift practically. ### Start with Constraints, Not Freedom Instead of open-ended prompts like "build me a user authentication system," begin with structured requirements: "Create a JWT-based authentication system following OWASP guidelines, using our existing PostgreSQL schema, and adhering to our TypeScript coding standards." Configure your AI agent with these constraints upfront. ### Build Your Agent's Toolkit Start with two or three Agent Skills or MCP servers that provide tooling a good human engineer would use - perhaps access to your internal API documentation, your security scanning tools or your architectural decision records. ### Implement Human Checkpoints Establish specific review gates in your workflow. After the planning phase, review the proposed architecture before implementation begins. After initial implementation, conduct a technical review focusing on maintainability and business alignment. These aren't bureaucratic hurdles-they're quality gates that ensure your AI partner stays on track. ### Practice Iterative Refinement When your agent produces code that works but isn't quite right, resist the urge to start over with a new prompt. Instead, provide specific feedback: "This function handles the happy path well, but needs error handling for network timeouts and invalid API responses." Teaching your AI to improve existing code builds better long-term solutions than constantly generating fresh approaches. ### Document and Template Your Patterns As you develop successful agentic workflows and coding agent templates, document the prompts, constraints, and review criteria that work best for your context. This turns your individual learning into repeatable organisational capability, allowing other team members to leverage the same structured approach rather than starting from scratch with their own vibe-based experiments. --- ## The Bottom Line Vibe coding will always have its place for quick prototypes and creative exploration. But for production quality, maintainable solutions - agentic coding not only a great opportunity for acceleration - it can be sustainable. The future belongs to developers who can effectively collaborate with AI agents. The question isn't whether AI will change how we code-it's whether we'll evolve our practices to harness that change responsibly. *See also - my [agentic coding workflow guide](https://smcleod.net/2025/04/my-plan-document-act-review-flow-for-agentic-software-development/) and [live demo recording](https://smcleod.net/2025/02/agentic-coding-live-demo-/-brownbag/).* - [My Plan, Document, Act, Review flow for Agentic Software Development](https://smcleod.net/2025/04/my-plan-document-act-review-flow-for-agentic-software-development/index.md): My Plan, Document, Act, Review flow for Agentic Software Development --- title: "My Plan, Document, Act, Review flow for Agentic Software Development" date: 2025-04-28T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg lastmod: 2026-07-04T10:00:00+10:00 url: "/2025/04/my-plan-document-act-review-flow-for-agentic-software-development/" tags: ["AI", "LLM", "tech", "coding", "agentic", "agentic coding", "agents", "skills", "context engineering", "tutorials"] author: "Sam McLeod" draft: false description: "An overview of my Setup, Plan, Act, Review & Iterate workflow for agentic software development." searchHidden: false cover: image: "setup-plan-act-iterate.svg" alt: "Agentic Coding Development Flow" # caption: "Optional caption text" --- I follow a simple, yet effective flow for agentic coding that helps me to efficiently develop software using AI coding agents while keeping them on track, focused on the task at hand and ensuring they have access to the right tools and information. The flow is simple: **_Setup -> Plan -> Act -> Review & Iterate._** It's a cycle rather than a straight line - what you learn in each pass feeds back into your setup and plans for the next. ![](setup-plan-act-iterate.svg) _Updated July 2026: I've revised this post to reflect how the workflow has evolved - Agent Skills, context engineering and sandboxing now feature heavily, and the tool-specific examples have been generalised. The workflow itself hasn't changed._ ## Workflow Quickstart Outline of the Setup -> Plan -> Act -> Review & Iterate workflow: 1. **Setup** - Choose capable models and an agentic coding harness that gets the most out of them. - Extend the agent with the right **tools** (MCP servers, language servers), **rules** and **skills**. - Scope what the agent can touch with **permissions and a sandbox**. 2. **Plan** - Ideate with the agent to hone in on the goal and requirements until you're satisfied that the agent and you are on the same page. - Prompt the agent to **create a development plan** (a markdown document in the repo) that captures: - **Why** - the goal, relevant context and background, and who it's for. - **What** - requirements, definition of done and where to find non-obvious information. - **How** - a **phased checklist of development tasks**. The checklist is the state of the work (important!). - Optionally start a fresh session with an agent in parallel and prompt it to read the plan and validate they understand it, asking any clarifying questions as needed. 3. **Act** - Start a fresh session. - Prompt the agent to read the development plan, ask any clarifying questions, then begin the first phase of development tasks. - For each task: implement, perform a critical self-review, lint / test / build, address any issues, then mark the task complete. 4. **Review & Iterate** - Review the work - the standard measures (code quality, architecture, testing) plus agent-specific ones (alignment to the plan, context window and sub-agent utilisation). - Update the plan for future phases if required and iterate towards the goal. - Feed what you've learned back into your rules, skills and tools. The intent of this workflow is to have a simple, clear, reviewable and actionable plan that can be worked from. It allows you to start a fresh session throughout the process without losing track of state, critical context or requirements. You may have heard of PRD or spec driven development with agents. Spec-_first_ development is essentially Plan then Act by another name. Spec-_anchored_ and spec-_as-source_ approaches - where specs are maintained as the ongoing source of truth - sit closer to waterfall and don't scale down well: small bugs and small stories drown in ceremony. The Setup -> Plan -> Act -> Review & Iterate workflow is a more agile approach that aims to be lighter, more portable (across both projects and tools) and less likely to over-engineer solutions, delivering value faster while still maintaining quality. --- The following is a more detailed breakdown of each phase of the workflow along with examples and prompt templates you can use to get started. ## 🕵 Setup ### 🧠 Models & Harness A raw model is not an agent - it becomes one when a harness gives it state, tool execution, feedback loops and enforceable constraints. Pick a strong coding model and a harness that gets the most out of it (I currently use Claude Code with Anthropic's models), then invest your setup effort in the tools, rules, skills and permissions that follow. ### 🛠️ Tools The effective use of tools is critical to the success and cost effectiveness of agentic coding. Favour deterministic tools for deterministic outcomes, and pick the lightest mechanism that does the job: - If a good CLI already exists (e.g. GitHub's `gh`), the agent can use it directly - no MCP server or skill required. - MCP servers earn their place for external systems that need standardised, controlled access - Confluence, Jira, security scanners, or anything requiring OAuth. - Many of the MCP servers I used to run have since been replaced by Agent Skills wrapping small deterministic scripts (see [Skills](#-skills) below). The MCP servers and other tooling I use are available here: [sammcj/agentic-coding](https://github.com/sammcj/agentic-coding). I still maintain [MCP DevTools](https://github.com/sammcj/mcp-devtools) - a single, efficient Go binary that replaces multiple Node.js and Python based MCP servers (search, package and documentation lookup, document processing and more). It remains the only MCP server I _always_ have enabled. #### Language Servers (LSP) Give the agent access to language servers and it gains the same code intelligence as your editor: go-to-definition, find-references, type information and diagnostics. The agent navigates code structurally instead of grepping text - before changing a function signature it can list every caller and reason about the blast radius, and type errors surface while editing rather than at build time. It's also cheaper in tokens than grep and glob for code-shaped questions. ### 📏 Rules I heavily utilise agent rules to steer the development process - a set of global rules used across all projects, with project-specific rules added as needed (e.g. `CLAUDE.md` / `AGENTS.md`, and path-scoped rule files such as `.claude/rules/*.md`). Despite the name, rules don't define agent behaviour so much as influence it - they're influential, not enforced. A few things I've learned writing them: - State what the agent _should_ do rather than what it shouldn't - mentioning a forbidden word or behaviour draws the model's attention to the very thing you don't want (beware the pink elephants!). - Group related rules with headings or XML-style tags - structure improves adherence. - Use emphasis sparingly: if everything is important, nothing is important. - Keep rules concise and high signal - enabled rules are loaded into every session. My rules are available here: [sammcj/agentic-coding](https://github.com/sammcj/agentic-coding#rules) ### 📚 Skills Agent Skills are packages of written knowledge and (optionally) executable scripts that inject domain expertise into the agent's context at runtime. Think of skills like a bookshelf. You don't remember the teachings of every book you own - when a situation arises that needs knowledge you don't have, you scan the spines to find the one that helps. Skills work the same way through progressive disclosure: the agent always sees each skill's name and short description, loads the skill body only when triggered, and reads any bundled references or scripts only if needed. An agent can have many skills available without filling its context. > Skills teach the agent what to know and how to do it. Rules guide how it behaves. Tools let it act. Two cautions. First, avoid "slop" skills - skills vibed into existence that contain no genuinely new knowledge, tools or workflows. Second, subject downloaded skills to the same rigour you'd apply to a library off the internet - prefer skills from sources you'd trust to land a pull request in your repo. You can outsource your thinking, but you can't outsource your understanding. I've written more on [writing and reviewing Agent Skills](https://smcleod.net/2026/07/writing-and-reviewing-agent-skills-common-pitfalls/), and my skills - including [skill-creator-primer](https://github.com/sammcj/agentic-coding/blob/main/Skills/skill-creator-primer/SKILL.md) which catches the common authoring pitfalls - are available at [sammcj/agentic-coding](https://github.com/sammcj/agentic-coding). ### 🔐 Permissions & Sandbox Scope what the agent can touch before you start: - Use your harness's permission model (allow / ask / deny) to match the trust level to each action's blast radius. Pre-approving trusted actions is almost as important as denying risky ones - constant approval prompts train you to click through without reading. - Run the agent inside a sandbox and lock it in: disable any option that lets the agent bypass the sandbox when it hits friction, and review your permissions when friction occurs instead. - The sandbox is one layer of defence to couple with sensible permissions - it won't protect you from actions you've explicitly allowed. - Hooks (shell commands that fire on agent events, much like git hooks) can add deterministic guardrails such as blocking dangerous command patterns. Use them sparingly - noisy hooks bloat context and can defeat prompt caching. - Exclude anything the agent shouldn't read (secrets, large data dumps) via your tool's ignore / deny configuration. ### 📑 Gather Documentation / Examples If I'm working with especially new libraries, frameworks or specifications I'll save key documentation or example code in the project (e.g. `docs/reference_examples/`) - or better, package it as a skill with the documentation as reference files so it's only loaded when needed. With good documentation-fetching tools available to the agent, this step is often unnecessary. --- ## 🤔 Plan _If you want to skip straight to a prompt template for this, see [#prompt-template-create-development-plan](#prompt-template-create-development-plan)_ I'll start by writing a prompt in a text editor / file with a detailed goal for the project I want to build. Then I start a fresh session in a read-only / planning mode and provide a prompt something like the following: > I want you to help me develop a new Golang application that provides an API for managing a list of tasks. > > The intent is to have a simple API I can use across various machines on my network that will keep track of tasks. > > The application must be able to be run on both Linux and macOS. It must have a Dockerfile that follows best practices to build and deploy the application. > > I want to use the Gin framework for the API and SQLite for the database. > > The intended audience is myself and my family, both of which are technical and can use the command line and API tools. > > We may be deploying the application to a remote server in the future, so it must be able to run on a server with limited resources. > > I don't know if I want to use a REST or GraphQL API. I'd like you to help me explore the pros and cons of each and make a recommendation. > > Here are some links to the documentation for the libraries I want to use: > - gin web framework repository - https://github.com/gin-gonic/gin > - Popular SQLite packages for Go - https://github.com/mattn/go-sqlite3, https://github.com/glebarez/go-sqlite Let's break this down, I am providing the agent: - **What** I want to build - **Why** I'm doing it - **Technical requirements** - **The intended audience** (users) of the software - **Assumptions** - **Unknowns** I want to explore during planning - **Links** to upstream documentation, examples etc... I then: - **Iterate** with the agent to refine the plan, asking it to clarify any unknowns or assumptions. - **Create A Plan Document** that outlines the project, requirements and development tasks. - **Review** the plan document and edit as required. - **Iterate** to hone in on the plan and approach ### 📄 The Plan Document - Prompt the agent to **create a markdown document** with a checklist of tasks to complete in a phased development approach: > [!NOTICE] Development Plan Prompt > Create a new markdown document called docs/DEVELOPMENT_PLAN.md. In this document, start by adding a detailed description of the project, requirements and assumptions, then add a checklist of tasks to complete in a phased development approach. - A good plan answers three questions: - **Why** - the goal, important context and background, and the target audience. - **What** - the requirements and definition of done, where to find non-obvious information, and the workflow to follow. - **How** - the phased checklist of tasks. Keep tasks concise and unambiguous, and avoid specifying exact code changes - leave the agent room to engineer. - I **review** the markdown document and edit as required. - I add a **tool use and task completion reminder** at the end: > [!NOTICE] Tool & Task Prompt > Remember to use the tools, functions and skills available to you. For each task: perform a critical self-review, run lint / test / build and address any issues before checking the task off. After each phase is complete, STOP and I will review your work. - **Iterate** to hone in on the plan and approach: > [!NOTICE] _Example_ > I like the idea of going with REST, but let's make sure we can easily switch to GraphQL in the future if we need to. Also we should consider how we will handle authentication and authorisation. The intent here is to have a clear, concise and detailed plan that you could in theory provide to any agent or competent software engineer to undertake the development. Investing in planning always pays off - a little extra effort at the start results in higher quality, better aligned outcomes. The checklist in the plan is the state of the work - it's what lets you start fresh sessions without losing progress. For larger projects I keep one plan per feature under `docs/plans/`. A note on built-in plan modes: some harnesses store plans centrally outside your repo, and some can be hasty to complete the plan, leading to premature analysis and design. I prefer having the agent write the plan to a markdown file in the repo - it's reviewable, portable across tools, and survives any individual session. --- ## 🏁 Act I start a fresh session so we're not pulling all the context (token usage) added during planning into development. (If the plan is small and your context usage is still under ~40-50K tokens, carrying on in the same session is fine.) - Prompt the agent to read the plan and then begin development: > [!NOTICE] Begin Development Prompt > First read the docs/DEVELOPMENT_PLAN.md file and ask me any clarifying questions you have. Then begin development of the first phase, ensuring all requirements are met. - For each task the agent should: implement, perform a critical self-review, run lint / tests / build, address any issues, then check the task off before moving to the next. ### 🧮 Context Engineering Context engineering - managing the quality and quantity of context in the system - runs through this entire workflow. LLMs are stateless: the entire context (system prompt, tool definitions, every file read, every prior message) is re-sent, and re-billed, every turn. The context window is the model's working memory, and models become measurably less capable and slower as it fills ("context rot"). Just because a model claims a 1M token context window doesn't mean it's any good when it gets anywhere near that. As a rough guide (as at mid-2026): keep under ~160K tokens on "200K" models, and under ~250-300K on "1M" models. The plan document is what makes this workable. Because the state of the work lives in the plan rather than the conversation, you can start a fresh session at any time and pick up exactly where you left off. Embrace starting fresh - thinking "but we've come so far, now it truly understands what I mean..." is a trap! To keep context lean: - Start fresh sessions rather than pushing a long one further. - Watch the quality of your enabled rules, tools and skills - poorly written ones bloat every session. - Have tools and scripts output only actionable information - silence the rest. - Be token-aware with reads: a grep for a symbol costs a few hundred tokens, reading the whole file costs thousands. The cheapest read is the one you didn't need. ### 🤖 Sub-Agents Sub-agents provide context isolation and parallelisation for delegated tasks. They can't talk to each other - the main agent is the hub and they're the spokes. Good uses: - Research - spin up a sub-agent per topic and have each return only what's useful. - Codebase exploration that would otherwise fill your main context with file dumps. - Working with "rich" files - have a sub-agent extract a PDF / PPTX / DOCX to markdown rather than pulling it into the main session. Some harnesses now also support agent teams, where teammates share a task list and communicate directly rather than fanning out and reporting back - useful when tasks genuinely require cross-talk. ### 🔎🧑‍💻 Review Once a phase is completed: - **Review** the work - the standard measures (code quality, architecture, testing, documentation) plus the agent-specific ones: did the approach stay aligned to the goal and the plan, and how well were context and sub-agents utilised? - **Update** - ask the agent to update the plan with any remedial items and ensure completed tasks are checked off. - **Fix** - or request fixes to any issues or changes required. Automate as much of this as you can - linters, test coverage gates and review skills / prompt templates beat manual inspection. ### 🔄 Iterate - Then I start a fresh session with the prompt: > [!NOTICE] Continue Development Prompt > First read the docs/DEVELOPMENT_PLAN.md file. Then continue development, ensuring all requirements are met. - Iterate until all phases are complete. ### 📋 Rules, Skills & Tools Aren't Static - Keep Improving Them! - I like to perform a brief retro after completing with lessons, what I'd do differently etc... - Update my global rules and skills with anything I've learned that can be applied to all projects. - This is the cycle in the diagram - what you learn in each pass feeds back into Setup for the next. --- ### Prompt Template: Create Development Plan This prompt template (most harnesses let you save these as slash commands or reusable workflows) assumes you have already started to tease out the goals and requirements of the task with the agent, and now you want to create a detailed development plan that can be used to guide the development process. ```markdown ROLE: You are a senior development planner tasked with creating a detailed development plan based on the provided discussion and requirements. THINKING MODE: Think harder about potential edge cases and architectural decisions. CONSIDERATIONS: - Always start with Context Gathering (before any implementation) - Use Planning for complex features or architectural changes - Scale up thinking modes for critical systems (e.g. use "ultrathink" for complex problems and architectures) - Apply Refactoring for optimisation phases - Adjust quality gates based on risk tolerance (e.g. if the project is for local development purposes it may not need as strict QA as if it was a production security system) - Maintain context between prompt sequences - If you are unsure of the agreed direction for development, ask clarifying questions before proceeding. - This planning occurs before writing any code, we must thoroughly understand the project context and requirements. IF THERE IS EXISTING CODE IN THE PROJECT: 1. Read all relevant files in the project directory 2. Examine existing documentation (README.md, docs/ etc.) 3. Analyse the codebase structure and dependencies 4. Identify coding conventions and patterns used 5. Review any existing tests to understand expected behaviour DEBUGGING PROTOCOL: - If tests fail: analyse failure reason and fix root cause - If performance issues: profile and optimise critical paths - If integration issues: check dependencies and interfaces TASK: Create a new markdown file called DEVELOPMENT_PLAN.md that contains the following: - An overview of the project purpose, goal and objectives along with any important background information. - Each task should be a checklist item. - A list of hard requirements if we have defined any. - Any unknowns or assumptions (if applicable). - A break down the development requirements into a checklist of tasks to be completed in phases. - You do not need to include dates or time estimates. - The document should be written in a way that I can provide it to a senior AI coding agent and have them understand and carry out the development. - Use dashes and a single space for markdown lists. - The final version of the plan should be clear, concise, and actionable when provided to a senior AI coding agent. --- Example DEVELOPMENT_PLAN.md --- # Development Plan for [PROJECT_NAME] ## Project Purpose and Goals [PROJECT_PURPOSE_AND_GOALS] ## Context and Background [PROJECT_CONTEXT_AND_BACKGROUND] ## Development Phases ### Phase 1 - [ ] Task 1 - [ ] Task 1.1 - [ ] Task 2 - [ ] Task 3 ## QA CHECKLIST - [ ] All user instructions followed - [ ] All requirements implemented and tested - [ ] No critical code smell warnings - [ ] Code follows project conventions and standards - [ ] Documentation is updated and accurate if needed - [ ] Security considerations addressed - [ ] Performance requirements met - [ ] Integration points verified - [ ] Deployment readiness confirmed - [ ] [OTHER_QA_CRITERIA] --- Then stop, and I will review the plan. ``` --- ## Further Reading - [Vibe Coding vs Agentic Coding](https://smcleod.net/2025/06/vibe-coding-vs-agentic-coding/) - why the engineering-led approach matters. - [Writing and Reviewing Agent Skills - Common Pitfalls](https://smcleod.net/2026/07/writing-and-reviewing-agent-skills-common-pitfalls/) - my guidance on skill authoring. - [sammcj/agentic-coding](https://github.com/sammcj/agentic-coding) - my rules, skills, MCP tools and configs. - [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) - Anthropic's engineering blog. - [Context Rot](https://research.trychroma.com/context-rot) - Chroma's research on model performance as input length grows. - [agentskills.io](https://agentskills.io) - the Agent Skills format and directory. - [LLM Sampling Parameters Guide](https://smcleod.net/2025/04/llm-sampling-parameters-guide/index.md): LLM Sampling Parameters Guide --- title: "LLM Sampling Parameters Guide" date: 2025-04-25T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg lastmod: 2025-11-27T01:00:00+10:00 tags: ["AI", "LLM", "tech", "ollama", "coding", "sampling", "inference", "tutorials", "llama"] author: "Sam McLeod" draft: false description: "A practical guide to LLM sampling parameters for Ollama, llama.cpp, and MLX" searchHidden: false cover: image: "sampling-methods-comparison.png" alt: "LLM Sampling Methods Comparison" --- Large Language Models don't generate text deterministically - they use probabilistic sampling to select the next token based on prediction probabilities. How these probabilities are filtered and adjusted before sampling significantly impacts output quality. This guide explains the key sampling parameters, how they interact, and provides recommended settings for different use cases. ## Framework Reference Last updated: November 2025 ### Parameter Comparison | Parameter | llama.cpp | Default | Ollama | Default | MLX | |-------------------|---------------------|---------|---------------------|---------|-------------------| | Temperature | `temp` | `0.8` | `temperature` | `0.8` | `temp` | | Top P | `top_p` | `0.9` | `top_p` | `0.9` | `top_p` | | Min P | `min_p` | `0.1` | `min_p` | `0.0` | `min_p` | | Top K | `top_k` | `40` | `top_k` | `40` | `top_k` | | Repeat Penalty | `repeat_penalty` | `1.0` | `repeat_penalty` | `1.1` | _Unsupported_ | | Repeat Last N | `repeat_last_n` | `64` | `repeat_last_n` | `64` | _Unsupported_ | | Presence Penalty | `presence_penalty` | `0.0` | `presence_penalty` | - | _Unsupported_ | | Frequency Penalty | `frequency_penalty` | `0.0` | `frequency_penalty` | - | _Unsupported_ | | Mirostat | `mirostat` | `0` | `mirostat` | `0` | _Unsupported_ | | Mirostat Tau | `mirostat_ent` | `5.0` | `mirostat_tau` | `5.0` | _Unsupported_ | | Mirostat Eta | `mirostat_lr` | `0.1` | `mirostat_eta` | `0.1` | _Unsupported_ | | Top N Sigma | `top_nsigma` | `-1.0` | _Unsupported_ | - | _Unsupported_ | | Typical P | `typical_p` | `1.0` | `typical_p` | `1.0` | _Unsupported_ | | XTC Probability | `xtc_probability` | `0.0` | _Unsupported_ | - | `xtc_probability` | | XTC Threshold | `xtc_threshold` | `0.1` | _Unsupported_ | - | `xtc_threshold` | | DRY Multiplier | `dry_multiplier` | `0.0` | _Unsupported_ | - | _Unsupported_ | | DRY Base | `dry_base` | `1.75` | _Unsupported_ | - | _Unsupported_ | | Dynamic Temp | `dynatemp_range` | `0.0` | _Unsupported_ | - | _Unsupported_ | | Seed | `seed` | `-1` | `seed` | `0` | - | | Context Size | `ctx_size` | `2048` | `num_ctx` | `2048` | - | | Max Tokens | `n_predict` | `-1` | `num_predict` | `-1` | - | ### Notable Default Differences | Parameter | llama.cpp | Ollama | Note | |----------------|---------------|--------|-----------------------------------| | min_p | `0.1` | `0.0` | Ollama disables Min P by default | | repeat_penalty | `1.0` | `1.1` | Ollama applies penalty by default | | seed | `-1` (random) | `0` | Different random behaviour | ### Feature Support | Feature | llama.cpp | Ollama | MLX | |----------------------------------------|-----------|--------|---------| | Core (temp, top_p, top_k, min_p) | ✓ | ✓ | ✓ | | Repetition penalties | ✓ | ✓ | ✗ | | Presence/frequency penalties | ✓ | ✓ | ✗ | | Mirostat | ✓ | ✓ | ✗ | | Advanced (DRY, XTC, typical, dynatemp) | ✓ | ✗ | Partial | | Custom sampler ordering | ✓ | ✗ | ✗ | --- ## Core Sampling Parameters ### Temperature Controls the randomness of token selection by modifying the probability distribution before sampling. **The formula:** Each token's logit (raw score) is divided by the temperature value, then softmax produces final probabilities: ``` adjusted_probability = softmax(logits / temperature) ``` **Values explained:** - **1.0 (baseline):** The model's original probability distribution is used as-is. This is the baseline, not "disabled". If the model assigns "the" a 40% probability and "a" a 30% probability, those exact values are used for sampling. - **< 1.0 (e.g., 0.5):** Makes the distribution *sharper*. High-probability tokens become more likely, low-probability tokens become less likely. Output becomes more deterministic, favouring the model's top choices. - **> 1.0 (e.g., 2.0):** Makes the distribution *flatter*. Probabilities become more uniform, giving lower-ranked tokens a better chance. Output becomes more random and varied. - **0:** Greedy decoding. Always selects the highest probability token. Fully deterministic. (Implementations handle this as argmax rather than literal division by zero.) - **→ ∞:** Approaches uniform random selection across all tokens. **Example:** Say the model's top 3 tokens have these base probabilities (at temp 1.0): - "cat": 60% - "dog": 30% - "fish": 10% At **temp 0.5**: Distribution sharpens - "cat" dominates even more strongly (~85%), others become unlikely. At **temp 2.0**: Distribution flattens - probabilities converge toward each other (~40%, ~35%, ~25%). *(Exact values depend on the underlying logits; these illustrate the directional effect.)* **Key points:** - Temperature changes have more impact at lower values (0.5→0.6 is more significant than 1.5→1.6) - Temperature is often used alongside top-p and top-k, which truncate the distribution before sampling. Temperature reshapes probabilities; top-p/top-k restrict which tokens are considered at all. - Applied *after* filtering methods in the standard pipeline - Lower temperatures suit factual, deterministic tasks. Higher temperatures suit creative work where variety is desirable.
What is a logit? A logit is the raw, unnormalised output value from a model before being transformed through softmax to produce probabilities. The term comes from logistic regression where it refers to log-odds.
### Top P (Nucleus Sampling) Restricts token selection to the smallest set whose cumulative probability exceeds threshold P. **How it works:** Tokens are sorted by probability and summed until reaching the threshold (e.g., 0.9 for 90%). Only tokens in this "nucleus" are considered. **Strengths:** Guarantees sampled tokens come from a set representing a fixed proportion of total probability mass. **Limitations:** - With flat distributions (low confidence), may include many low-probability tokens - With peaked distributions (high confidence), may select only one or two tokens - Doesn't adapt candidate set size based on model confidence **Recommended values:** 0.9 for general use, 0.8 for factual, 0.95 for creative. Set to 1.0 to disable. ### Min P Filters vocabulary to include only tokens whose probability is at least a fraction of the *most likely* token's probability. **How it works:** Calculate threshold as `min_p × probability_of_top_token`. Only tokens above this threshold are considered. **Key advantage - adaptability:** - High confidence (top token has high probability) → higher threshold → smaller, focused candidate set - Low confidence (probabilities spread out) → lower threshold → more diverse candidates This avoids including too many unlikely tokens (unlike Top P in flat distributions) while preserving alternatives when one token dominates. **Sensitivity:** - Very low (0.01): Little filtering when distributions are flat - Very high (0.2+): Can become overly restrictive **Recommended values:** 0.05-0.1 provides good balance. Applied before temperature in the standard pipeline. ### Top K Restricts selection to the K highest-probability tokens, regardless of their actual values. **How it works:** If top_k=40, only the 40 highest-probability tokens are considered. **Limitation:** As a fixed-count filter, it can't adapt to model confidence - too restrictive when many good alternatives exist, too permissive when only a few tokens are meaningful. **Interaction with other filters:** In llama.cpp, Top K applies first, creating an initial pool that subsequent filters (Top P, Min P) operate within. **Recommendation:** For most use cases, disable (set to 0) and rely on Min P or Top P. Some models have specific recommendations (e.g., Gemma 3 recommends top_k=64). ### Repeat Penalty Discourages repeating tokens that appeared within the `repeat_last_n` window. **How it works:** Applies a multiplicative penalty to logits of recently-seen tokens. Values > 1.0 decrease probability of recent tokens. **Cautions:** - Penalises *any* repetition, including necessary repetition (code syntax, key terms) - Can produce awkward phrasing as the model avoids natural repetition - Often masks underlying prompt or parameter issues **Related parameter - repeat_last_n:** How far back to look. Use 64 for prose, 128-512 for code to capture variable names. **Recommendation:** Start with 1.0 (disabled) or 1.05, increase only if problematic repetition persists. Never exceed 1.2. ### Presence Penalty Applies a fixed penalty to any token that has appeared *at least once* in the preceding context (prompt + generation). **Use case:** Encourages introducing new concepts or tokens; discourages topic looping. **Difference from repeat_penalty:** Presence penalty considers the entire context, not just a recent window, and applies the same penalty regardless of how many times a token appeared. **Recommended values:** 0.0-0.5. Start at 0.0 (disabled). ### Frequency Penalty Penalises tokens based on how *frequently* they've appeared in the preceding context. **Use case:** Reduces stylistic repetition and overuse of common words; increases lexical diversity. **Difference from presence_penalty:** Frequency penalty scales with occurrence count - a word used 10 times gets penalised more than one used twice. **Recommended values:** 0.0-0.5. Start at 0.0 (disabled). --- ## Advanced Parameters ### Mirostat An adaptive sampling method that dynamically adjusts constraints to maintain a target level of "surprise" (perplexity). **Modes:** - 0: Disabled - 1: Mirostat (original) - 2: Mirostat 2.0 (generally preferred) **Parameters:** - `mirostat_tau` (target entropy): Lower (2.0-3.0) = more focused, higher (4.0-6.0) = more diverse - `mirostat_eta` (learning rate): How quickly it adjusts. Default 0.1. **Usage:** When enabling Mirostat, disable other filters by setting `top_p=1.0`, `top_k=0`, `min_p=0.0`. It acts as an alternative sampling controller. **When to use:** - Long-form text where consistent quality matters - When you prefer adaptive control over manual tuning - To prevent quality degradation over extended generations **Starting values:** mirostat=2, mirostat_tau=5.0, mirostat_eta=0.1 ### Other Sampling Methods | Parameter | Name | Description | Use Case | Range | |---------------|---------------------|---------------------------------------------------------|---------------------------------|------------------------------| | `tfs_z` | Tail Free Sampling | Filters low-probability tail based on second derivative | Alternative to Top P/Min P | `0.9-1.0` (`1.0` disables) | | `typical_p` | Typical Sampling | Selects tokens close to expected probability | Surprising yet coherent outputs | `0.9-1.0` (`1.0` disables) | | `top_a` | Top-A | Threshold based on squared top probability | Adaptive filtering | `0.0-1.0` (`0.0` disables) | | `dry_*` | DRY | Detects and penalises repeating n-gram patterns | Prevents phrase recycling | Multiplier `0.0-2.0` | | `dynatemp_*` | Dynamic Temperature | Adjusts temp based on distribution entropy | Automatic temperature tuning | Range `0.0-2.0` | | `top_n_sigma` | Top-N-Sigma | Threshold at max logit minus N standard deviations | Statistical filtering | `1.0-3.0` (`-1.0` disables) | | `xtc_*` | XTC | Occasionally forces lower-probability token selection | Breaks predictable patterns | Prob `0.0-1.0`, Thresh `0.1` | --- ## Sampler Ordering The order in which sampling methods apply significantly impacts output. The default llama.cpp pipeline: ``` penalties → dry → top_n_sigma → top_k → typ_p → top_p → min_p → xtc → temperature ``` **Why this order matters:** 1. **Penalties** modify initial logits 2. **Filters** remove low-probability or undesirable tokens based on original predictions 3. **Temperature** rescales logits of remaining candidates 4. **Softmax** converts to probabilities, then sampling occurs Applying temperature *before* filtering could distort the distribution, causing unwanted tokens to pass filters or good tokens to be filtered out. When using Mirostat, it replaces the filtering steps and dynamically controls the process itself. ### llama.cpp Default Sampler Order ``` penalties → dry → top_n_sigma → top_k → typ_p → top_p → min_p → xtc → temperature ``` Shorthand sequence: `edskypmxt` Customise with `--samplers` or `--sampling-seq`. --- ## Diagrams {{< wide-image src="sampling-methods-comparison.png" alt="Sampling Methods Comparison" >}} {{< wide-image src="ollama-sampling-diagram.png" alt="Ollama Sampling Pipeline" >}} --- ## Troubleshooting Before adjusting parameters, check: - **Prompt quality:** Is it clear and specific? - **Context window:** Has the limit been reached? - **Model suitability:** Is the model appropriate for the task? ### Model keeps repeating itself Note: Perform changes one at a time to isolate effects. 1. Check prompt doesn't encourage repetition 2. Gradually increase `repeat_penalty` (start with 1.05, max 1.15-1.2~) 3. Try `frequency_penalty` (0.1-0.5) or `presence_penalty` (0.1-0.5) 4. Slightly increase `min_p` or decrease `top_p` ### Outputs too random/incoherent 1. Lower temperature (try 0.6, then 0.4) 2. Increase `min_p` to 0.1 3. If still problematic, enable `top_k=40` ### Outputs too generic/predictable 1. Increase temperature (0.7-1.0) 2. Lower `min_p` to 0.03-0.05 3. Ensure `top_k` is disabled (0) ### Model generates incorrect facts 1. Lower temperature significantly (0.3 or 0.1) 2. Increase `min_p` to 0.1-0.15 3. Adjust prompting to encourage factual accuracy --- ## Appendix: Code Generation Parameters Code generation has unique requirements - intentional repetition (variable names, syntax patterns) combined with precision needs. ### Key Considerations - **Repetition handling:** Low or no repeat penalties (1.0-1.05) work better than for text - **Context length:** Longer repeat lookback (128-512) maintains variable naming consistency - **Temperature sweet spot:** 0.1-0.7 for most code tasks - **Precision vs. creativity:** Lower (0.1-0.2) for critical implementations, slightly higher (0.3-0.6) when multiple valid approaches exist - [Getting Started with Agentic Systems - Developer Learning Paths](https://smcleod.net/2025/04/getting-started-with-agentic-systems-developer-learning-paths/index.md): Getting Started with Agentic Systems - Developer Learning Paths --- title: "Getting Started with Agentic Systems - Developer Learning Paths" date: 2025-04-16T01:10:00+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI", "LLM", "tech", "agentic", "coding", "learning", "engineering", "tutorials"] author: "Sam McLeod" draft: false description: "A curated learning path for engineers looking to gain practical experience with AI and agentic systems." searchHidden: false cover: image: "generative-agent-state-diagram.png" alt: "Generative Agent State Diagram" # caption: "Optional caption text" --- As agentic systems become increasingly central to modern software development, many engineers are looking to build practical skills but don't know where to start. This guide provides a short list of pre-reading/watching and hands-on training resources to help you get started with developing with AI agents. The focus is on practical implementation for tools and methods you're likely to use in the workplace, so you can quickly gain experience and confidence in building AI powered and agentic systems. ![generative-agent-state-diagram](generative-agent-state-diagram.png) ## Understanding the Landscape Before diving into specific frameworks and tools, it's essential to understand the broader context of AI agents and their role in software development. The resources below provide a solid foundation: ```mermaid flowchart TD Start([Start Here]) --> Foundation[Foundation Understanding] Foundation --> Practical[Practical Implementation] subgraph Foundation[Foundation Knowledge] direction TB Basics[AI & Prompting Basics]:::process AgentConcepts[Agent Concepts]:::process GoodAgents[Building Good Agents]:::components RAG[RAG - Retrieval Augmented Generation]:::components MCP[MCP - Model Context Protocol]:::components end subgraph Practical[Hands-on Learning Agents] direction TB SmolAgents[SmolAgents Tutorial]:::llm CrewAI[CrewAI Course]:::llm Implementation[Build Real Projects]:::process Tools[Agent Tools]:::components Memory[Memory Systems]:::components end classDef process fill:#EAF5EA,stroke:#C6E7C6,color:#77AD77 classDef llm fill:#E5F5E0,stroke:#31A354,color:#31A354 classDef components fill:#E6E6FA,stroke:#756BB1,color:#756BB1 classDef data fill:#EFF3FF,stroke:#9ECAE1,color:#3182BD ``` ## Pre-Reading/Watching Before diving into implementation, these resources will help you build foundational knowledge: ### 1. AI & Agent Fundamentals - [Why Agent Engineering](https://youtu.be/5N33E9tC400) - Why we are all in on Agent Engineering in 2025. - [How We Build Effective Agents](https://youtu.be/D7_ipDqhtwk) - Barry Zhang from Anthropic touches on the key points of building effective agents. - [Intro to Large Language Models](https://youtu.be/zjkBMFhNj_g) - This is a 1 hour general-audience introduction to Large Language Models: the core technical component behind systems like ChatGPT, Claude, and Bard. What they are, where they are headed, comparisons and analogies to present-day operating systems, and some of the security-related challenges of this new computing paradigm. - [LLMs For the working Programmer](https://youtu.be/zwItokY087U) - Some practical high level tips for developers working with LLMs for the first time. - [The Future of AI Development](https://youtu.be/JhCl-GeT4jw) - Rethinking how we might 'program' in the future. ### 2. Prompting Fundamentals The Prompting Guide provides essential knowledge for effective interaction with AI models: - [Prompting Basics](https://www.promptingguide.ai/introduction/basics) - [Prompting Tips and Best Practices](https://www.promptingguide.ai/introduction/tips) - [LearnPrompting](https://learnprompting.org/docs/introduction) - [Google Prompt Engineering Guide](https://www.gptaiflow.tech/assets/files/2025-01-18-pdf-1-TechAI-Goolge-whitepaper_Prompt%20Engineering_v4-af36dcc7a49bb7269a58b1c9b89a8ae1.pdf) (PDF) ## Short Training Courses Once you've grasped the fundamentals, these tutorials and courses will help you build practical experience: ### SmolAgents Track SmolAgents is an open-source framework designed to simplify the creation of AI agents. It's an excellent starting point for practical implementation: - [Introduction to Agents Course](https://huggingface.co/learn/agents-course/en/unit0/introduction) - This comprehensive course walks you through building your first AI agents using the SmolAgents framework. ### CrewAI Track CrewAI offers a different perspective on multi-agent systems: - [Multi AI Agent Systems with CrewAI](https://www.deeplearning.ai/short-courses/multi-ai-agent-systems-with-crewai/) - Learn to build and orchestrate multiple AI agents working together. This course covers the fundamental building blocks of multi-agent systems with practical examples. ### Tooling & Integrations Track #### MCP - Model Context Protocol - [Introduction to MCP](https://www.youtube.com/watch?v=kQmXtrmQ5Zg) - [MCP Documentation Website](https://modelcontextprotocol.io/introduction) - [DeepLearning & Anthropic MCP Course](https://www.deeplearning.ai/short-courses/mcp-build-rich-context-ai-apps-with-anthropic/) - [Huggingface MCP Course](https://huggingface.co/learn/mcp-course/en/unit1/key-concepts) #### RAG - Retrieval Augmented Generation - [What is RAG](https://youtu.be/T-D1OfcDW1M) - [Intro to RAG](https://youtu.be/Y08Nn23o_mY) - [GraphRAG vs Traditional RAG](https://youtu.be/Aw7iQjKAX2k) ## Advanced Learning Once you're comfortable with the basics, these resources will help you deepen your understanding: - [Transformers explained, visually](https://poloclub.github.io/transformer-explainer/) - A visual guide to understanding transformers, the backbone of the majority of current generation AI models. - [LLMs explained, briefly](https://youtu.be/LPZh9BOjkQs) - [What is a neural network?](https://www.youtube.com/watch?v=aircAruvnKk) ### SmolAgents Advanced Topics 1. [Guided Tour of Advanced Features](https://huggingface.co/docs/smolagents/guided_tour) 2. [Building Effective Agents](https://huggingface.co/docs/smolagents/tutorials/building_good_agents) 3. [Working with Agent Tools](https://huggingface.co/docs/smolagents/tutorials/tools) 4. [Implementing Agent Memory](https://huggingface.co/docs/smolagents/tutorials/memory) ## Community I think the best community in terms to sharing new methods, models and tools is by far [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) (Reddit). ## Practical Tips for Getting Started > [!TIP] The most effective way to learn is by doing. 1. **Start small**: Begin with simple agent implementations before moving to complex multi-agent systems. 2. **Focus on understanding**: Don't just copy code-understand why agents make certain decisions and how they process information. 3. **Get your hands dirty and experiment**: Create small, disposable proof-of-concept projects to test different agent architectures and approaches. 4. **Try agentic coding**: Try using tools like Cline or Copilot Agent to help accelerate your coding process. Start with the fundamentals, work through the tutorials, and gradually take on more complex projects. The goal is not just to understand the theory but to develop practical skills that you can apply in real-world software development. - [The Cost of Agentic Coding](https://smcleod.net/2025/04/the-cost-of-agentic-coding/index.md): The Cost of Agentic Coding --- title: "The Cost of Agentic Coding" date: 2025-04-03T01:00:01+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI","LLM","tech","agentic","coding","vibecoding"] author: "Sam McLeod" draft: false description: "The cost of not leveraging agentic coding is likely far greater than you think." searchHidden: false cover: image: "productivity-flywheel.svg" alt: "Productivity Flywheel" # caption: "Optional caption text" --- Don't ask yourself "_What if my high performing engineers spent $2k/month on agentic coding?_" ...ask yourself why they (and others) _aren't_ - and what opportunities they're missing as a result. ![](productivity-flywheel.svg) Agentic coding represents not merely a productivity tool but a fundamental economic shift in software development. While some may balk at potential AI inference costs, this analysis demonstrates that even in hypothetical extreme edge cases-where your most AI-intensive power users might theoretically approach $2,000 monthly-the economics would remain overwhelmingly favourable: a modest cost increase yielding 300-1,000% productivity gains. In reality, most engineers using these tools effectively will incur far lower costs (typically $200-800 monthly) while still achieving transformative productivity improvements. For most engineers, the actual investment will be far lower while still delivering transformative results. Beyond raw output, this approach liberates engineers from repetitive implementation tasks, enables parallel work streams, and makes previously unviable projects economically feasible. The true cost isn't in adopting these tools, but in the opportunity cost of failing to leverage them while competitors operate within an entirely different economic paradigm. --- ## Quantifying the ROI: An Evidence-Based Approach While claims of 300-1,000% productivity increases might initially sound hyperbolic, they're grounded in real-world observations and data collection. To provide transparency and enable readers to test these assertions against their own contexts, I've developed an [Agentic Coding ROI Calculator](https://smcleod.net/tools/agentic_coding_costs/) that models the economic impact of these tools. ![Cost Calculator Interface](cost-graph.jpg) This calculator is in alpha/beta state and the public facing version has a few limitations, however it isn't purely based on theoretical projections but rather synthesises data from multiple sources and methodologies: - **Direct observations** of productivity differences in teams adopting agentic coding approaches - **Implementation metrics** from multiple projects and case studies - **Feedback and performance data** from engineers trained in effective agentic coding techniques - **Community benchmarking** cross-referenced with direct findings to establish validity - **Manual development estimates** from traditional software engineering practices to provide baseline for comparisons - **Parallel work streams** where traditional (manual) development occurs alongside agentic coding, allowing for direct comparisons of output - **Token usage metrics** exported from agentic coding tools to establish typical consumption patterns for various project sizes and complexities The calculator's parameters weren't set arbitrarily but calibrated based on multiple data points: - **Project Size Benchmarks:** The preset project sizes (from Micro to Enterprise) reflect real-world development patterns observed across dozens of projects, with token consumption and time metrics mapped to each scale. - **Productivity Multipliers:** The force multiplier effect (typically 3-10x) wasn't chosen for marketing impact but represents the median range observed across multiple engineering teams-with some outlier cases showing even higher multiples for narrow, well-defined tasks. - **Token Usage Patterns:** The calculator's default 8:1 output-to-input ratio and 88% cached tokens reflect typical patterns observed in production environments using modern agentic coding tools with effective prompt caching. - **Model Comparison Data:** Pricing and capability information for various models (Claude, GPT, DeepSeek) is regularly updated to reflect current market offerings and their relative performance characteristics. By continuously refining these parameters based on new observations and feedback, the calculator serves not just as a cost projection tool but as a living repository of empirical knowledge about agentic coding economics. ```mermaid flowchart TB Data[/"Data Sources"/]:::inputOutput --> Benchmarks["Manual Dev Estimates & Parallel Streams of Work"]:::process subgraph Calculator["ROI Calculator"] Parameters["Input Parameters"]:::inputOutput --> Calculation["Cost Comparison
Calculation"]:::process Benchmarks --> Parameters Calculation --> DirectROI["Direct ROI
Assessment"]:::data Calculation --> OppCost["Opportunity Cost
Assessment"]:::data end DirectROI --> TotalValue["Total Value
Assessment"]:::data OppCost --> TotalValue TotalValue --> DecisionMaking["Inform Strategic Decision Making"]:::process classDef inputOutput fill:#FEE0D2,stroke:#E6550D,color:#E6550D classDef process fill:#EAF5EA,stroke:#C6E7C6,color:#77AD77 classDef data fill:#EFF3FF,stroke:#9ECAE1,color:#3182BD ``` The calculator allows for adjusting parameters including hourly rates, project complexity, model selection, and engineer experience levels to see how these variables affect the ROI equation in your specific context. What's particularly revealing is that even with conservative parameters-smaller productivity gains, higher AI costs, longer setup times-the economic case remains compelling. The following sections explore these economics in greater detail, but I encourage readers to experiment with the public-facing beta version of the calculator themselves to test assumptions against their own business contexts. ### The Real Cost Question This brings us back to our original question: _What if your high-performing engineers spent $2k/month on agentic coding?_ The calculator is one way of revealing why this is the wrong question. When properly measured-accounting for both direct productivity gains and reduced opportunity costs-the more appropriate question becomes: _"Why would you accept a 3-10x productivity penalty by not enabling your engineers with these tools?"_ The data shows that even at the upper bound of AI inference costs (for arguments sake let's say $2,000 monthly), the economics remain overwhelmingly favourable. But the calculator also reveals something more subtle: most organisations won't approach these upper bounds, with typical monthly costs falling between $200-800 per engineer while still capturing the majority of productivity benefits. This creates a compelling economic narrative: Organisations that systematically adopt agentic coding aren't just getting more output for their dollar-they're fundamentally changing the economic parameters of software development in ways that traditional cost models struggle to fully capture. --- ## Historical Context and Long-Tail Innovation What we're witnessing with agentic software development is paralleled only by the largest of productivity revolutions throughout history. When the power loom increased textile production capacity by 40x, it didn't eliminate weaving jobs instead, it unlocked previously unimaginable scale and capacity. Similarly, when semiconductor manufacturing productivity increased 38x between 1970-2000, it created entirely new product categories rather than simply making existing electronics more efficiently. ![](loom-361631_1920.jpg) What makes agentic coding particularly transformative is its ability to fundamentally reshape how engineers approach problem-solving. Unlike traditional automation that simply speeds up existing processes, agentic coding creates a multiplicative effect by: - **Eliminating cognitive bottlenecks** - Engineers can offload repetitive implementation details while focusing on higher-level architecture and design choices - **Compressing knowledge acquisition cycles** - Where learning a new framework or API might have taken days or weeks, engineers can now leverage domain expertise across multiple technical areas simultaneously - **Enabling parallel exploration of solution spaces** - Multiple approaches can be developed and evaluated concurrently rather than sequentially - **Reducing the activation energy for creativity** - The cost of experimentation drops dramatically, allowing for more innovative approaches that would otherwise be too resource-intensive to attempt This combinatorial effect creates a self-reinforcing productivity flywheel. As engineers spend less time on implementation details and more time on strategic thinking, the quality of their architectural decisions improves. These better foundations then amplify the effectiveness of agentic coding assistance, creating a virtuous cycle that compounds over time. The most profound impact isn't merely coding faster-it's the fundamental shift in what becomes economically viable to build. Projects that were previously deemed too complex, time-consuming, or specialised are now within reach, expanding the frontier of what's possible in software development. ## Agentic Coding: An Evolutionary Step, Not the Destination It's important to recognise that agentic coding, while transformative, represents a point on a longer continuum rather than an ultimate destination. Where traditional code completion tools like Copilot operate primarily at the syntax and function level-offering suggestions within a developer's existing workflow-agentic systems represent a significant evolutionary leap by handling cohesive, multi-step development tasks with greater autonomy. However, this is likely not the final form of AI-augmented development. As foundation models and agent architectures continue to advance, we can anticipate further shifts in how software is conceptualised and created. The wisest approach isn't to view agentic coding as an end goal, but rather as a crucial transitional technology that both delivers immediate productivity benefits and prepares engineering teams for future paradigms where the boundaries between human and machine contributions become increasingly fluid. Organisations that successfully adopt today's agentic techniques and tools will be better positioned to adapt to whatever comes next, having already developed the organisational practices, mental models, and enabled technical tooling needed to thrive as we likely move towards a future when the importance of human written code for higher level tasks becomes less and less important. --- ## Analysing the Economics of Agentic Development With this in mind, let's say for argument's sake, one of your high performing engineers _heavily_ utilises AI for large amounts of agentic coding and somehow racks up $2,000 a month on AI inference. At first glance, this may seem like a significant investment. However, when we consider the productivity gains that agentic coding unlocks, the economics of this decision become much more compelling. > [!TIP] TLDR > Even if enabling agentic coding cost a hypothetical $2,000 per month (which it doesn't), the economics are unmistakable. > > With productivity gains of 300-1,000%, **the return on investment is _exceptional_**. This analysis explores the return on investment in the form of AI inference to unlock agentic coding, considering both the quantifiable productivity metrics and the qualitative improvements in risk profiles, code quality, and strategic capabilities. We are not talking about throwing $2,000 away on someone blindly piping all the data into a LLM and expecting it to write software (and thus add value) - we are talking about the (potential) spend on _effectively_ using agentic coding tools (e.g. [Cline](https://cline.bot)), coupled with the best AI models (e.g. Claude Sonnet 3.7 at the time of writing this) and the best agent tooling (e.g. [MCP](https://modelcontextprotocol.io) for package versioning, documentation etc). This is not a "throw money at the problem" solution - this is about _unlocking_ the potential of the engineer to be able to do more with less, and to be able to do it faster and better. > [!INFO] Agentic coding is not (currently) full autopilot > The importance of the engineer understanding and having the most effective tools cannot be understated. > > Agentic coding or any other form of AI assisted coding is not (currently) 'autopilot' - it requires good engineering skills and experience to be effective. ## Breaking Down the Numbers ![cost breakdown](cost-breakdown.jpg) ### The Traditional Approach Let's establish a baseline for comparison. In a conventional development process, a project might cost approximately $12,000 monthly (representing 1.0 unit of work), delivering a predictable 1.0 unit of work output. When fully accounting for loaded developer costs ($75-150/hour), this translates to $600-1,200 per developer day. This direct cost calculation forms the foundation of the economic comparison but, as we'll see later, captures only part of the true economic picture. ### The Agentic Coding Alternative By contrast, if you add $2,000 to the human cost to unlock agentic coding, the direct economic equation changes dramatically: - **Just 16% increase in direct costs** - **300%-1,000% productivity increase** _(3.0-10.0 units of work delivered)_ - **ROI of 1,700%-5,900%** - **Payback period of ~1.3 workdays** These figures align with real-world observations from development teams integrating agentic coding into their workflow, where engineers report producing 5-10x more code while maintaining or improving quality standards once properly enabled and trained on agentic coding. And let's not forget - it's only $2,000 extra if they actually use $2,000 worth of AI inference, if they use less - the cost is less but it might be an indicator that they're not using AI to unlock their full potential. ## The True Economic Impact: Beyond Direct Productivity While the direct productivity metrics are compelling on their own, they capture only part of the economic value proposition of agentic coding. The most significant economic factor-often overlooked in traditional ROI calculations-is opportunity cost. ### The Critical Role of Opportunity Cost In traditional development models, the true cost of a project isn't merely the engineering hours invested, but includes the market opportunities lost while waiting for completion. These opportunity costs typically manifest in three critical ways: 1. **Time-to-market delays:** Each week of delayed delivery represents lost revenue, competitive advantage, and market positioning-costs that often dwarf direct development expenses. When a competitor launches a feature three months before you can, the market share and mind share losses often significantly exceed the development costs themselves. 2. **Idea validation cycles:** Agentic coding enables testing multiple approaches or features in parallel, significantly reducing the sunk cost of pursuing inferior solutions and allowing faster pivots. This parallel exploration capability fundamentally changes product development economics by reducing costly commitment to unproven approaches. 3. **Strategic resource allocation:** By compressing routine development work, organisations can reallocate their most talented engineers to high-value strategic initiatives rather than maintenance and implementation details. This addresses one of the most persistent hidden costs in technology organisations: the opportunity cost of brilliant minds devoted to repetitive tasks. For many businesses, these opportunity cost factors represent the most compelling economic case for agentic coding, even more so than the direct productivity multipliers. A feature that reaches market three months earlier, or a strategic pivot executed in days rather than weeks, creates value that conventional ROI calculations struggle to fully capture. ### Hidden Costs of Traditional Development Our conventional development process carries numerous hidden costs that rarely appear in standard economic analyses: - **Context switching penalties**: Productivity loss and cognitive load when juggling multiple tasks, with research suggesting this can consume up to 40% of productive capacity - **Knowledge dependencies**: Communicating with and waiting for teammates with specialised skills, often creating bottlenecks that cascade through project timelines - **Slow feedback cycles**: Delays in identifying and resolving issues that compound throughout development cycles - **Sequential work limitations**: Inability to meaningfully progress multiple streams in parallel, forcing linear approaches to inherently multi-faceted problems - **Creative opportunity costs**: Limited human investment in time for creative thinking and ideation due to implementation burdens ### The Full Economic Picture When properly analysed, the typical daily cost proposition becomes even more favourable: - Developer daily cost (excluding hidden costs): $600-1,200 - AI inference daily cost: $5-25 (for heavy usage) - For most developers: $2-10 per day (typical usage patterns) The key insight isn't about spending the maximum possible, but rather enabling engineers to leverage these tools without arbitrary limitations that undermine their effectiveness. It's worth emphasising that most engineers will fall well below the upper threshold, with monthly AI costs typically ranging from $200-800 depending on usage patterns and specific tasks. The true economic implications extend beyond direct productivity measures and include structural changes to development processes: reduced dependency on specialised knowledge, shortened feedback cycles, and practical parallel exploration of solution alternatives-all of which compound the value proposition far beyond the simple calculation of coding output per dollar spent. --- ## Workflow Comparison The following diagram illustrates the structural differences between conventional development approaches and agentic coding workflows: ### Traditional vs Agentic Coding Approach ```mermaid flowchart LR Traditional["Traditional"] --- Agentic["Agentic"]:::hidden subgraph Traditional[" "] direction TB T_Start([Start]) --> T_Plan[Team Planning
Multiple Stakeholders]:::process T_Plan --> T_EstimateCards[Create & Estimate
Tickets]:::process T_EstimateCards --> T_Code[Code Solution]:::process T_Code <--> T_Context[Context
Switching]:::stop T_Code --> T_Block{Need
External
Expertise?}:::decision T_Block -->|Yes| T_Wait[Wait for Help]:::stop T_Wait --> T_Return[Resume Work]:::process T_Return --> T_Code T_Block -->|No| T_Test[Test Solution]:::process T_Test --> T_Review[Peer Review]:::process T_Review --> T_Wait2[Wait for
Feedback]:::stop T_Wait2 --> T_Fix[Fix Issues]:::process T_Fix --> T_Test T_Test -->|Passed| T_Doc[Document
Solution]:::process T_Doc --> T_End([End: 1x Output]) end subgraph Agentic[" "] direction TB A_Start([Start]) --> A_Plan[Define Goal and Plan Work]:::process A_Plan --> A_Delegate[Delegate to Agent]:::components A_Delegate --> A_Parallel{Parallel Work}:::decision A_Parallel -->|Agentic Task A| A_Agent1[Agent Work
Stream 1]:::llm A_Parallel -->|Agentic Task B| A_Agent2[Agent Work
Stream 2]:::llm A_Parallel -->|Your Focus| A_Other[Work on Other
Tasks]:::process A_Agent1 --> A_Review[Early Review]:::process A_Agent2 --> A_Review A_Review -.-> A_Refine[Refine Direction]:::process A_Refine -.-> A_Review A_Review --> A_Measure[Measure & Review]:::llm A_Doc[Documentation]:::process --> A_Measure A_Measure --> A_End([End: 3-10x Output]) end %% Set white backgrounds for all elements classDef default fill:#ffffff,stroke:none style Traditional fill:#ffffff,stroke:#cccccc style Agentic fill:#ffffff,stroke:#cccccc classDef hidden stroke:none,fill:none %% Apply colour scheme classDef inputOutput fill:#FEE0D2,stroke:#E6550D,color:#E6550D classDef llm fill:#E5F5E0,stroke:#31A354,color:#31A354 classDef components fill:#E6E6FA,stroke:#756BB1,color:#756BB1 classDef process fill:#EAF5EA,stroke:#C6E7C6,color:#77AD77 classDef stop fill:#E5E1F2,stroke:#C7C0DE,color:#8471BF classDef data fill:#EFF3FF,stroke:#9ECAE1,color:#3182BD classDef decision fill:#FFF5EB,stroke:#FD8D3C,color:#E6550D classDef storage fill:#F2F0F7,stroke:#BCBDDC,color:#756BB1 classDef api fill:#FFF5F0,stroke:#FD9272,color:#A63603 classDef error fill:#FCBBA1,stroke:#FB6A4A,color:#CB181D ``` ## Jevons Paradox and Value Distribution The [Jevons Paradox](https://en.wikipedia.org/wiki/Jevons_paradox) suggests that as resource utilisation becomes more efficient, total consumption often increases rather than decreases. Applied to software development, historical evidence indicates that a 5x productivity improvement wouldn't reduce demand for development - instead, it would dramatically expand the universe of viable software projects. And let's face it-it wouldn't be a blog post about agentic AI without a reference to Jevon. ![Jevons Paradox](jevons.jpg) As illustrated above, the Jevons Paradox (or '_effect_') means that as agentic coding makes development more efficient, we can expect an expansion of software projects rather than a contraction of development work. The economic value created through this productivity transformation won't simply disappear-it will redistribute across the ecosystem in complex and interconnected ways: 1. **Engineers**: Engineers gain freedom from repetitive implementation tasks, allowing them to focus on intellectually stimulating work and tackle previously intractable problems-improving both job satisfaction and career development. 2. **Organisations**: Beyond merely completing existing project backlogs faster, organisations can fundamentally reimagine their product development strategies. Projects previously considered too speculative or resource-intensive become viable, enabling more experimental approaches and reducing the opportunity cost of innovation. 3. **End users**: As development capacity expands and costs decrease, software solutions will emerge for increasingly specialised needs. Long-neglected "long tail" problems-too niche to warrant conventional development resources-become economically viable to solve, particularly benefiting underserved markets and use cases. 4. **Market entrants**: The reduced capital requirements and compressed development timelines lower barriers to entry across the technology sector. This democratisation enables more diverse founders to bring solutions to market without requiring extensive initial investment, potentially reshaping competitive dynamics across industries. 5. **Adjacent professionals**: Product managers, designers, domain experts, and other non-engineering stakeholders gain the ability to more directly influence and participate in the development process, as the technical barrier between concept and implementation diminishes. 6. **Society**: The cumulative effect of these changes extends beyond immediate economic benefits. As software solutions proliferate into previously underserved domains, we may see accelerated progress on complex societal challenges where software solutions were previously too costly or complex to implement at scale. Rather than a zero-sum redistribution, agentic coding has the potential to expand the total value created-a rising tide that creates new opportunities across the entire ecosystem while fundamentally changing who can participate in software creation and what problems become economically viable to solve. ## The Hidden Human Cost of Widget Programming Perhaps the most overlooked aspect of today's software landscape is its profound misuse of human potential. Globally, talented engineers spend their days building variations of the same widgets - auth systems, API integrations, JS framework migrations and ever-valuable widget implementation - that have been implemented thousands of times before. When we look beneath the surface of enterprise software development, a sobering reality emerges: much of this work isn't advancing innovation or addressing meaningful challenges, but simply recreating slightly modified versions of existing solutions. This represents an extraordinary waste of human intelligence and creativity. As we continue to advance AI capabilities in software development, the true opportunity isn't merely efficiency-it's the redirection of human cognitive capacity toward genuinely novel problems that demand creativity, ethical judgment, and domain expertise. The real cost of our current paradigm isn't measured in dollars, but in squandered human potential that could be addressing the vast landscape of unsolved challenges still awaiting our attention. --- ## Conclusion The economic case for investing in AI-powered development isn't merely compelling-it's transformative. When 3-10x productivity gains can be achieved with only modest increases in cost, we're no longer discussing incremental improvements but a fundamental shift in what's possible when engineers are empowered with the right tools, models and techniques. This isn't just about coding faster-it's about reimagining how value is created through software. For leadership, this represents a strategic inflection point that demands attention: the organisations that systematically leverage these capabilities will operate in an entirely different economic paradigm from those that don't. The question isn't whether to adopt these tools and enable our teams to use them-but how quickly you can enable and integrate them to remain competitive. The benefits extend beyond immediate productivity metrics to encompass risk reduction, solution quality, and-perhaps most importantly-the liberation of human potential from repetitive implementation tasks. As we stand at this crossroads in software development, the cost-benefit analysis couldn't be clearer. The most expensive AI tools are those left unused by your team, as competitors race ahead with fundamentally different economics. > Don't ask yourself "_What if my high performing engineers spent $2k/month on agentic coding?_" > > **Ask yourself why others _aren't_ - and what opportunities they're missing as a result.** - [The Democratisation Paradox: What History Teaches Us About AI](https://smcleod.net/2025/03/the-democratisation-paradox-what-history-teaches-us-about-ai/index.md): The Democratisation Paradox: What History Teaches Us About AI --- title: "The Democratisation Paradox: What History Teaches Us About AI" date: 2025-03-23T01:00:01+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI","LLM","tech","culture","art","economics","capitalism","history","creativity","automation"] author: "Sam McLeod" draft: false description: "The Democratisation Paradox: What History Teaches Us About AI." searchHidden: false --- Every technological revolution has triggered waves of anxiety about the obsolescence of human skills and professions. The current fears that AI will replace artists, eliminate writing jobs, render illustrators obsolete, and devalue creative work follow a well-established historical pattern that's worth examining critically. ## The Democratisation Paradox When photography emerged in the 19th century, painters predicted the death of portraiture. When home cameras became accessible, professional photographers feared obsolescence. When smartphones put cameras in everyone's pockets, the same concerns resurfaced {{< cite-inline "Benjamin1969" >}}{{< cite-inline "Batchen2001" >}}. Yet professional photography hasn't vanished-it's evolved. What actually occurred was a democratisation of image creation, while simultaneously elevating the appreciation for truly skilled work {{< cite-inline "Hand2012" >}}. The written word followed a similar trajectory. Writing began as the domain of elites and scholars before the printing press dramatically altered the landscape {{< cite-inline "Eisenstein1980" >}}. As literacy spread, journalism rose, then blogs emerged, and self-publishing platforms demolished traditional gatekeepers {{< cite-inline "Ong1982" >}}. Now we're witnessing an influx of low/no-effort AI-generated books appearing on Amazon. Despite this, professional authors continue to publish their work, and readers still distinguish and value quality writing regardless of its production channel {{< cite-inline "Baron2015" >}}. Mathematics faced its existential crisis with the calculator. Critics warned that mechanical computation would cripple mathematical understanding {{< cite-inline "Kidwell2008" >}}. The opposite occurred-calculators made mathematics more approachable and freed human minds to tackle more complex problems. Complex, manual arithmetic skills may have diminished, but mathematical innovation accelerated as computational tools handled the grunt work {{< cite-inline "Kaput1992" >}}. Digital music production technologies sparked similar anxieties. When samplers, MIDI, and production software emerged, traditionalists decried the "death of musicianship" {{< cite-inline "Taylor2001" >}}. Instead, these tools lowered barriers to creation and spawned entirely new genres {{< cite-inline "Theberge1997" >}}. Even with today's deluge of AI-generated music on streaming platforms, human musicians continue creating, though the economics of fair compensation remains problematic {{< cite-inline "Prior2018" >}}. ## The Creative Landscape is Forever in Transition Some observers have coined the term "AI slop" to describe hastily produced, minimally curated content generated with AI tools but little human refinement. This includes auto-generated YouTube videos with robotic narration, cheaply produced books flooding Amazon, or generic stock-style images created through basic prompts. It should be noted that the term is sometimes overused to disparage anything that might have been created with AI assistance, regardless of quality or craftsmanship. After all - much like CGI in films-when it's good, you don't notice it's there {{< cite-inline "Floridi2014" >}}. This phenomenon mirrors the early days of every democratised medium: when blogs first appeared, when self-publishing began, when amateur photography flooded social media. The signal-to-noise ratio initially decreases before quality standards emerge and the market matures {{< cite-inline "Andrejevic2013" >}}. It's worth noting that such transitions are not new but rather a recurring pattern in technological evolution. What history demonstrates repeatedly is that technological advancements don't typically eliminate fields-they restructure them. The true impact is often a dismantling of exclusive gatekeeping mechanisms that previously restricted access to knowledge and creative tools. This creates opportunities for those previously locked out of creative industries while also challenging established professionals to differentiate their work. When anyone can create, the volume of poor-quality content increases, but so does the potential for innovation from previously excluded voices. ![alt text](art-tech-cycle-1.png) ## Beyond Technology: Capitalism & Systemic Economics The more pressing concern isn't whether AI will replace human creativity-it's whether our economic systems will properly value human contributions in an AI-augmented landscape. AI itself is neither good nor evil-it is simply technology. The real risk lies in our societal structures and economic systems that are centred around capitalist ideologies focused on profit maximisation. Humans are capable of both extraordinary creation-moon landings, symphonies, medical breakthroughs-and deeply destructive behaviours. This duality is mirrored in AI outputs, as systems absorb both the brilliance and flaws present in their training data, aligned with the values of the powerful companies that build and control them {{< cite-inline "Crawford2021" >}}. Meanwhile, humans remain caught in a competitive mindset focused on scarcity of money, power, and resources, with the overwhelming majority dedicating five out of seven days of their lives working for someone else rather than themselves. As we transition from traditional capitalism toward technofeudalism, power increasingly concentrates in the hands of tech giants who control the digital infrastructure upon which society depends {{< cite-inline "Varoufakis2021" >}}. Unlike conventional corporations merely pursuing shareholder value, these new feudal lords-companies like Google, Amazon, the so-called "Open"AI and Meta-wield unprecedented influence over access to digital spaces, data, and online commerce {{< cite-inline "Zuboff2019" >}}. This power dynamic has intensified with the development of advanced AI, as the most powerful models in the world are now built and controlled by these same corporate entities, further cementing their position as the overlords of our digital existence. This technofeudal control directly impacts creative economies. Many artists, writers, and musicians have legitimate fears about being undercut by AI-generated alternatives that cost a fraction to produce-alternatives often created using models owned by these same tech giants. Streaming services profiting from both human and AI-generated content maintain compensation models designed to maximise their own profits rather than fairly value human creativity. Meanwhile, traditional companies driven primarily to deliver year-on-year profits to shareholders will likely continue trying to do more with less, leveraging these AI tools at the expense of human creators. ### Impact on the Workforce In the short term over the 1 to 2 years that follow, we're likely to witness significant job displacement, particularly within large enterprise organisations {{< cite-inline "Frey2017" >}}. These corporations are prime targets for AI-driven efficiency measures, especially in areas where work is standardised despite perceptions of uniqueness. A perfect example is enterprise software development, where organisations often maintain vast teams building functionally similar systems to their competitors-each believing their implementation to be special or distinctive when, in reality, most are simply widget factories operating in parallel. This systemic inefficiency has persisted primarily because large organisations traditionally move slowly, creating a market equilibrium where inefficiency is tolerated. However, AI threatens to shatter this balance by dramatically accelerating automation capabilities while simultaneously reducing technical barriers to implementation, allowing new players to quickly gain a foothold {{< cite-inline "Brynjolfsson2014" >}}. Consider also call centre staff, content moderators, and entry-level developers at large corporations. These roles don't always create substantial unique value when focused on repetitive tasks, and corporations are unlikely to invest in re-skilling these workers at their own cost when AI alternatives become viable. In the medium term - perhaps the next 3 to 5 years or so, companies that cut headcount earlier may return to hiring, but with roles significantly refocused {{< cite-inline "Autor2015" >}}. Rather than simply rebuilding widget factories, positions will increasingly centre on 'driving' AI capabilities, requiring deeper engineering and architectural understanding to enable organisations to target new and competing revenue streams. This shift represents not just a change in job descriptions but a fundamental reimagining of how human expertise complements rather than competes with automated systems. Historically, many large enterprises have enjoyed a form of institutional immunity-deemed 'too big to fail' due to their substantial capital reserves, the collective slow pace of industry movement, and often lenient regulatory oversight. Yet AI's unprecedented ability to accelerate development threatens to alter this dynamic, Allowing smaller organisations or potentially even individuals to outmanoeuvre larger competitors encumbered by legacy systems and drowning in bureaucracy. This shift in competitive advantage may reshape industry landscapes faster than many established players can adapt. Evolution is not mandatory, but neither is survival. ## Moving Forward: Balancing Progress and Humanity The democratisation of creative tools through AI represents the latest chapter in humanity's complex relationship with technology. As we've seen throughout history, the initial disruption eventually gives way to a new equilibrium where human creativity adapts rather than disappears. The true challenge lies not in whether AI will replace us, but in how we choose to structure our economic and social systems around these new capabilities. Rather than fearing AI itself, we would be wiser to focus our collective energy on building economic models that distribute the benefits of technological advancement more equitably. After all, the most valuable aspect of creativity has never been the tools themselves, but the uniquely human perspective, experience, and purpose that breathes life into them. --- ## Abstract > This paper examines the historical pattern of technological disruption and democratisation across various creative fields and applies these insights to the current AI revolution. We explore how each technological innovation-from photography to digital music production-initially triggered fears of human obsolescence but ultimately restructured rather than eliminated professional domains. The paper argues that the true challenge of AI lies not in the technology itself, but in our economic systems' ability to properly value human contributions in an AI-augmented landscape. By analysing the recurring "democratisation paradox," wherein new technologies simultaneously lower barriers to creation while elevating appreciation for skilled work, we can better understand and navigate the current transition toward AI-assisted creativity and work. {{< bibliography >}} {{< bibentry "Andrejevic2013" >}}Andrejevic, M. (2013). *Infoglut: How Too Much Information Is Changing the Way We Think and Know*. New York: Routledge.{{< /bibentry >}} {{< bibentry "Autor2015" >}}Autor, D. H. (2015). Why Are There Still So Many Jobs? The History and Future of Workplace Automation. *Journal of Economic Perspectives, 29*(3), 3-30.{{< /bibentry >}} {{< bibentry "Baron2015" >}}Baron, N. S. (2015). *Words Onscreen: The Fate of Reading in a Digital World*. Oxford, UK: Oxford University Press.{{< /bibentry >}} {{< bibentry "Batchen2001" >}}Batchen, G. (2001). *Each Wild Idea: Writing, Photography, History*. Cambridge, MA: MIT Press.{{< /bibentry >}} {{< bibentry "Benjamin1969" >}}Benjamin, W. (1969). *The Work of Art in the Age of Mechanical Reproduction*. (H. Zohn, Trans.). New York: Schocken Books. (Original work published 1936).{{< /bibentry >}} {{< bibentry "Brynjolfsson2014" >}}Brynjolfsson, E., & McAfee, A. (2014). *The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies*. New York: W. W. Norton & Company.{{< /bibentry >}} {{< bibentry "Crawford2021" >}}Crawford, K. (2021). *Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence*. New Haven, CT: Yale University Press.{{< /bibentry >}} {{< bibentry "Eisenstein1980" >}}Eisenstein, E. L. (1980). *The Printing Press as an Agent of Change*. Cambridge, UK: Cambridge University Press.{{< /bibentry >}} {{< bibentry "Floridi2014" >}}Floridi, L. (2014). *The Fourth Revolution: How the Infosphere is Reshaping Human Reality*. Oxford, UK: Oxford University Press.{{< /bibentry >}} {{< bibentry "Frey2017" >}}Frey, C. B., & Osborne, M. A. (2017). The future of employment: How susceptible are jobs to computerisation? *Technological Forecasting and Social Change, 114*, 254-280.{{< /bibentry >}} {{< bibentry "Hand2012" >}}Hand, M. (2012). *Ubiquitous Photography*. Cambridge, UK: Polity Press.{{< /bibentry >}} {{< bibentry "Kaput1992" >}}Kaput, J. J. (1992). Technology and Mathematics Education. In D. A. Grouws (Ed.), *Handbook of Research on Mathematics Teaching and Learning* (pp. 515-556). New York: Macmillan.{{< /bibentry >}} {{< bibentry "Kidwell2008" >}}Kidwell, P. A., Ackerberg-Hastings, A., & Roberts, D. L. (2008). *Tools of American Mathematics Teaching, 1800-2000*. Baltimore, MD: Johns Hopkins University Press.{{< /bibentry >}} {{< bibentry "Ong1982" >}}Ong, W. J. (1982). *Orality and Literacy: The Technologizing of the Word*. London: Methuen.{{< /bibentry >}} {{< bibentry "Prior2018" >}}Prior, N. (2018). *Popular Music, Digital Technology and Society*. London: SAGE Publications.{{< /bibentry >}} {{< bibentry "Taylor2001" >}}Taylor, T. D. (2001). *Strange Sounds: Music, Technology and Culture*. New York: Routledge.{{< /bibentry >}} {{< bibentry "Theberge1997" >}}Théberge, P. (1997). *Any Sound You Can Imagine: Making Music/Consuming Technology*. Middletown, CT: Wesleyan University Press.{{< /bibentry >}} {{< bibentry "Varoufakis2021" >}}Varoufakis, Y. (2021, June 28). Techno-Feudalism Is Taking Over. *Project Syndicate*.{{< /bibentry >}} {{< bibentry "Zuboff2019" >}}Zuboff, S. (2019). *The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power*. London: Profile Books.{{< /bibentry >}} {{< /bibliography >}} - [The effects of prompt caching on Agentic coding](https://smcleod.net/2025/03/the-effects-of-prompt-caching-on-agentic-coding/index.md): The effects of prompt caching on Agentic coding --- title: "The effects of prompt caching on Agentic coding" date: 2025-03-20T21:00:01+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI","LLM","tech","anthropic","aws","cline","programming","vertex","gcp","agentic","agentic coding"] author: "Sam McLeod" draft: false description: "What is prompt caching and why is it so important for Agentic coding?" searchHidden: false cover: image: "cline-1.png" alt: "Cline prompt caching indicator" # caption: "Optional caption text" --- Prompt caching is a feature that Anthropic first offered on their API in 2024. It adds a cache for the tokens used ## Why it matters Without prompt caching every token in and out of the API must be processed and paid for in full. This is bad for your wallet, bad for the LLM hosting providers bottom line and bad for the environment. This is especially important when it comes to Agentic coding, where there are a lot of tokens in/out and important - a lot of token reuse, which makes it a perfect use case for prompt caching. ![Cline prompt caching indicator](cline-1.png) | Use case | Latency w/o | Latency w/ caching | Cost reduction | |-------------------------------------------------------|-------------|--------------------|----------------| | Chat with a book (100,000 token cached prompt) | 11.5s | 2.4s (-79%) | -90% | | Many-shot prompting (10,000 token prompt) | 1.6s | 1.1s (-31%) | -86% | | Multi-turn conversation (10-turn, long system prompt) | ~10s | ~2.5s (-75%) | -53% | _Credit: [Anthropic](https://www.anthropic.com/news/prompt-caching)_ The impact of not having prompt caching enabled can be clearly illustrated by performing the same software development task with the same model with and without prompt caching enabled: ![Comparing the cost with and without caching](with-without-caching.png) Here we can see that the cost of Agentic coding with Amazon Bedrock is significantly higher as it does not yet support prompt caching, they have a gated beta but it is limited to the US regions only. --- ## How it works ![alt text](closedai-caching-image.png) _Image credit: "[Open"AI](https://platform.openai.com/docs/guides/prompt-caching)_ --- Anthropic cache architecture ![](pc1.png) ![](pc2.png) --- ## Who has it and who doesn't - Anthriopic: Yes - DeepSeek: Yes - OpenAI: Yes - GCP VertexAI: Yes - Azure OpenAI: Yes - Bedrock: **No (in gated beta in the US regions only, with no ETA for Australian regions)** [Cline](https://cline.bot/), is in my opinion has been the best Agentic coding tool for some time, it make extensive use of prompt caching to _greatly_ reduce costs with LLM API providers that support it - even better it exposes the effects of prompt caching to the user: ### Links - [Anthropic Prompt Caching Documentation](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) - [GCP VertexAI Prompt Caching Documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude-prompt-caching) - ["Open"AI Prompt Caching Documentation](https://platform.openai.com/docs/guides/prompt-caching) - ["Amazon Bedrock Prompt Caching](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) - [Cline](https://cline.bot) - [Agentic Coding - Live Demo / Brownbag](https://smcleod.net/2025/02/agentic-coding-live-demo-/-brownbag/index.md): Agentic Coding - Live Demo / Brownbag --- url: "/2025/02/agentic-coding-live-demo-/-brownbag/" # keeps the live URL: Hugo 0.166 no longer keeps "/" from the title in :title slugs title: "Agentic Coding - Live Demo / Brownbag" date: 2025-02-07T01:00:01+10:00 thumbnail: thumbnail.png images: ["thumbnail.png"] thumbnailIcon: thumbnail-icon.svg tags: ["AI","LLM","tech","deepseek","llama","anthropic","aws","cline","roo code","programming","brownbag"] author: "Sam McLeod" draft: false description: "Video recording of a brownbag presentation / live demo I ran on Agentic Coding using Cline/Roo Code" searchHidden: false ---
Apologies for the video quality, Google Meet/Hangouts records in very low resolution and bitrate. Links mentioned in the video: - [Cline](https://cline.bot) - [Roo Code](https://github.com/RooVetGit/Roo-Code) (Cline fork with some experimental features) - MCP - https://modelcontextprotocol.io/introduction - The package-version MCP server I created: https://github.com/sammcj/mcp-package-version - https://smithery.ai (index of MCP servers) - https://mcp.so(index of MCP servers) - https://glama.ai/mcp/servers (index of MCP servers) - [Bringing K/V Context Quantisation to Ollama](https://smcleod.net/2024/12/bringing-k/v-context-quantisation-to-ollama/index.md): Bringing K/V Context Quantisation to Ollama --- url: "/2024/12/bringing-k/v-context-quantisation-to-ollama/" # keeps the live URL: Hugo 0.166 no longer keeps "/" from the title in :title slugs title: "Bringing K/V Context Quantisation to Ollama" date: 2024-12-04T20:00:02+00:00 tags: ["AI", "LLM", "tech", "ollama", "llama","quantisation"] author: "Sam McLeod" draft: false description: "K/V context cache quantisation has been added to Ollama. This enables significant reductions in VRAM usage, allowing users to realise the potential of expanded context sizes and run larger models at their existing context sizes." searchHidden: false cover: image: "llm-vram-components.svg" alt: "Illustration of the components of a LLM's memory usage" hidden: false --- Explaining the concept of K/V context cache quantisation, why it matters and the journey to [integrate it into Ollama](https://github.com/ollama/ollama/pull/6279). [![Release the Ollamas](ollama-release.png)](https://github.com/ollama/ollama/pull/6279) --- ## Why K/V Context Cache Quantisation Matters The [introduction](https://github.com/ollama/ollama/pull/6279) of K/V context cache quantisation in Ollama is significant, offering users a range of benefits: - • **Run Larger Models**: With reduced VRAM demands, users can now run larger, more powerful models on their existing hardware. - • **Expand Context Sizes**: Larger context sizes allow LLMs to consider more information, leading to potentially more comprehensive and nuanced responses. For tasks like coding, where longer context windows are beneficial, K/V quantisation can be a game-changer. - • **Reduce Hardware Utilisation**: Freeing up memory or allowing users to run LLMs closer to the limits of their hardware. Running the K/V context cache at Q8_0 quantisation effectively halves the VRAM required for the context compared to the default F16 with minimal quality impact on the generated outputs, while Q4_0 cuts it down to just one third (at the cost of some noticeable quality reduction). ![What makes up a LLMs memory usage?](llm-vram-components.svg) Consider running a 8b parameter model with a 32K context size, the vRAM required for the context could be as follows: - • **F16** K/V: Around **6GB**. - • **Q8_0** K/V: Around **3GB**, (50%~ saving). - • **Q4_0** K/V: Around **2GB**, (66%~ saving). Saving that 3GB of vRAM by using Q8_0 could be enough to either allow you to double the context size to 64K, or perhaps to run a larger parameter model (e.g. 14B instead of 8B). --- ## Interactive VRAM Estimator I've built an interactive VRAM estimator to help you understand the impact of K/V context cache quantisation on your VRAM usage. You can adjust the model size, context size and quantisation level to see how it affects the memory requirements. {{< vram-calculator >}} --- ## Enabling K/V Context Cache Quantisation in Ollama This is covered in the [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-set-the-quantization-type-for-the-kv-cache), but here's a quick guide: - • Build the latest version of Ollama from the main branch or download the pre-release binaries from Ollama's [releases page](https://github.com/ollama/ollama/releases)). - • Make sure you're running Ollama with [Flash Attention enabled](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-enable-flash-attention) (`OLLAMA_FLASH_ATTENTION=1`), _Note: This should become the default behaviour in the near future as there's no reason not to use it._ - • Set the K/V cache quantisation to Q8_0 by adding `OLLAMA_KV_CACHE_TYPE="q8_0"` to the environment variables you run Ollama with. Start Ollama and Q8_0 quantisation will be used for the K/V context cache by default. If you run into issues with quality however - you can simply disable K/V context cache quantisation by setting `OLLAMA_KV_CACHE_TYPE` environment variable to `f16` or not setting it at all. This is even easier if you run Ollama in a container as you can simply run two containers with different configurations (as there's practically no overhead to running multiple containers). ### Implementation Limitations See [what wasn't included in the PR](#what-wasnt-included-in-the-pr) for more information on these limitations. - You cannot set the quantisation level in a model's Modelfile (this would be really good to add back in). - You cannot set the K and V caches to different quantisation levels. - You cannot request a quantisation level via the Ollama API, or on the command line. --- ## Understanding K/V Context Cache Quantisation K/V context quantisation is completely separate from model quantisation, which is the process of reducing the precision of the model's weights and biases to save memory and improve performance. Instead of compressing the model itself, K/V context cache quantisation focuses on reducing the memory footprint of the context cache used during text generation. > _Matt Williams kindly featured the PR on his YouTube channel, which generated a lot of interest and feedback from the community._ [![Matt William's YouTube video on the PR](https://img.youtube.com/vi/RFaMiQ97EoE/0.jpg)](https://www.youtube.com/watch?v=RFaMiQ97EoE) ### K/V Context Cache You can think of the K/V (key-value) context as the 'working memory' of an LLM. It's it needs to keep at front of mind as you interact with it. This cache can get _very_ large - in the order of many gigabytes. In simple terms, the K/V context cache acts as the memory of an LLM during text generation. It stores the essential information from the preceding text, allowing the model to maintain context and generate coherent responses. ### Quantisation Quantisation _(or 'quantization' to our American friends)_ can be thought of as _compression_, it works by reducing the precision of the numerical values stored within it. Think of it like rounding numbers - you lose a tiny bit of detail, but you save a lot of space. When quantisation is applied to the K/V context cache it greatly reduces the memory requirements, allowing users to run larger models or use larger context sizes on their existing hardware. The most commonly used quantisation levels for the K/V are Q8_0 and Q4_0, unquantised is referred to as F16 (or F32 although for inference you would not run F32). ### Performance Quantisation of the K/V context cache has minimal impact on performance, with quantising the K cache slightly improving performance while quantising the V cache may have a slight negative impact. The overall performance impact is negligible, especially when weighed against the significant reductions in VRAM usage. ### Quality • Q8_0 - Minimal quality impact for normal text generation models, suitable for most users to be enabled by default. Perplexity measurements on an early implementation showed it added around 0.002-0.05~ perplexity to the model. • Q4_0 - Some noticeable quality reduction, but still usable for those without much vRAM or working on creative tasks where quality is less critical. In early testing, Q4_0 added around 0.206-0.25~ perplexity to the model. --- ### Perplexity Measurements I recently performed some perplexity measurements comparing Q8_0 and Q4_0 quantisation levels on a Qwen 2.5 Coder 7B model as the Qwen series of models is known to be quite sensitive to quantisation. The results shown below show the difference in quality is minimal with an increase of `0.0043`: | Model | Params | Quant | K/V Cache Type | ppl | Notes | Date | | ------------- | ------ | ----- | -------------- | ------------------ | --------------------------------------------------------------------------------------- | ---------- | | qwen2.5 coder | 7b | q6_k | `f16/f16` | 8.3891 +/- 0.02016 | [unsloth 128k gguf](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-128K-GGUF) | 2024-12-06 | | qwen2.5 coder | 7b | q6_k | `q8_0/q8_0` | 8.3934 +/- 0.02017 | [unsloth 128k gguf](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-128K-GGUF) | 2024-12-06 | {{< perplexity-chart >}} _Perplexity Measurement_ ```bash ./llama-perplexity -ngl 99999 -fa -b 2048 -c 6114 -sm none \ # -ctk f16 -ctv f16 \ # and # -ctk q8_0 -ctv q8_0 \ -f wiki.train.raw.txt \ -m Qwen2.5-Coder-7B-Instruct-128k-Q6_K.gguf ``` --- ### When not to use K/V context cache quantisation Note that the ability to set the K/V cache quantisation level in a model's Modelfile was [removed from the PR](#what-wasnt-included-in-the-pr), but I hope that Ollama will reconsider this in the future so people can experiment with different quantisation levels for different models and use cases. **Embedding Models** As stated in the [FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-set-the-quantization-type-for-the-kv-cache), it is not recommended to use K/V cache quantisation for embedding models as these are more sensitive to quantisation. For this reason Ollama attempts to detect embedding models and disable K/V cache quantisation automatically (you'll see a warning in the logs if this happens). **Vision/Multi-Modal Models (Maybe)** The same _may_ apply to vision/multi-modal models based on some reports - although I have not personally looked into this. **High Attention Head Models** It's been noted in llama.cpp's test back at the start of 2024 that high attention head models can be more sensitive to K/V quantisation, so you may want to avoid using K/V context cache quantisation with these models. How high is too high? I'm not sure - but Qwen 2 was mentioned as an example in the llama.cpp PR, I personally have been using [Qwen 2.5 Coder 32b Q6_K](https://huggingface.co/unsloth/Qwen2.5-Coder-32B-Instruct-128K-GGUF/blob/main/Qwen2.5-Coder-32B-Instruct-Q6_K.gguf) with Q8_0 K/V cache quantisation without any noticeable issues. --- ## Compatibility K/V context cache quantisation requires [Flash Attention](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-enable-flash-attention) to be enabled. Enabling Flash Attention has no negative impacts and is something I expect to become the default behaviour in the near future with Ollama. While practically all modern models support Flash Attention, if a model is loaded that - or if your hardware doesn't support Flash Attention, Ollama will automatically fall back to the default F16 quantisation. You'll see a warning in the logs if this happens. Supported Hardware: - • Apple Silicon (Metal): Works on Apple Silicon devices. - • NVIDIA GPUs: Works on all NVIDIA GPUs with CUDA support, Pascal and newer. - • AMD: Works on most AMD GPUs with ROCm support, although ROCm in general is not as well supported as CUDA or Metal and performance may vary. --- ## The Journey to Integration The journey to integrate K/V context cache quantisation into Ollama took around 5 months. The hard work was done up front by [ggerganov](https://github.com/ggerganov/) in the underlying [llama.cpp](https://github.com/ggerganov/llama.cpp), initially in [#7527](https://github.com/ggerganov/llama.cpp/pull/7527), which Ollama uses as it's primary inference engine. My PR integrated that functionality into Ollama which involved supporting the required configuration, implementing memory estimations for layer placement, error and condition handling, ensuring compatibility with the existing codebase, a lot of testing and a lot of communication around the feature to help others understand it's importance and how to use it. ### Successes - • It's merged! - • Extensive testing and feedback from the community. - • Once the PR gained traction with the Ollama team, [jmorganca](https://github.com/jmorganca) and [jessegross](https://github.com/jessegross) were both incredibly helpful in providing feedback and guidance, especially as I am not a Golang developer. - • The PR became so popular I knew of _many_ people running Ollama from the feature branch, this is not something I wanted to see in the long term, but it was a good sign that people were interested in the feature. - • I have been building Ollama successfully at least twice, often much more every day for 5 months without major issues. ### Challenges - • Explaining the concept and benefits of K/V context cache quantisation to the community. - • Addressing merge conflicts due to updates in the Ollama main branch as time passed. - • Adapting to the new CGO server implementation and refactoring the code to accommodate new runners. - • Ensuring compatibility with the existing codebase as it changed over time. - • Finding the right balance between allowing users to configure Ollama to meet their needs while maintaining simplicity and ease of use. - • The noise in the PR from folks either piling on or trying to help, which made it hard to keep track of the actual changes. - • Github's PR interface which can be a bit clunky when dealing with large PRs. - • Daily battles with my (International / British English) spell checker to keep the Americanised spelling of words consistent with the rest of the codebase. _It took 5 months, but we got there in the end._ --- ## Definitions | Term | Definition | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | LLM | Large Language Model, a type of AI model capable of understanding and generating human-like text. | | vRAM | Video RAM, the memory used by your graphics card. LLMs require significant vRAM, especially for larger models and context sizes. | | Context Size | The amount of text the LLM can "remember" and consider when generating a response including both the user's inputs and the models own outputs. Larger context sizes allow for more nuanced and relevant output. | | Quantisation | A technique for reducing the precision of numerical values, resulting in smaller data sizes. | | Q8_0 & Q4_0 | Different levels of quantisation, with Q8_0 halving the VRAM usage of the context and Q4_0 reducing it to one third compared to F16 (unquantised). | | llama.cpp | The primary underlying inference engine used by Ollama. | | Flash Attention | A technique used to reduce the memory requirements of LLMs by only [attending](https://huggingface.co/docs/text-generation-inference/en/conceptual/flash_attention) to a subset of the context at a time. | | ROCm | The AMD Radeon Open Compute platform, an open-source platform for GPU computing | | CUDA | A parallel computing platform and application programming interface model created by Nvidia | | Metal | A low-level, low-overhead hardware-accelerated graphics and compute application programming interface developed by Apple. | --- ## What wasn't included in the PR Originally I had several features which in the PR that were not included in the final version as Ollama wanted to minimise the configuration exposed to users and not introduce API changes. - • All the K/V quantisation types supported by llama.cpp (Q4_1, IQ4_NL, Q5_0, Q5_1). - • Ability to set the quantisation level in a models Modelfile (I'm hoping this will be added back in the future). - • API parameters that allowed setting the quantisation level when making requests. - • CMD line parameters that allowed setting the quantisation level when running Ollama. Additionally the ability to set different quantisation levels for the K and V caches was not included, this might be nice to add back in the future, as to quote [JohannesGaessler](https://github.com/JohannesGaessler) [[May 2024](https://github.com/ggerganov/llama.cpp/pull/7412#issuecomment-2120427347)] while measuring the quality impact of K/V context cache quantisation using an older implementation: > _• The K cache seems to be much more sensitive to quantization than the V cache. However, the weights seem to still be the most sensitive._ > > _• Using q4_0 for the V cache and FP16 for everything else is more precise than using q6_K with FP16 KV cache._ > > _• A 6.5 bit per value KV cache with q8_0 for the K cache and q4_0 for the V cache also seems to be more precise than q6_K weights. There seems to be no significant quality loss from using q8_0 instead of FP16 for the KV cache._ --- ## Reporting Issues If you find a bug with K/V context cache quantisation it could be either in Ollama or, perhaps more likely - in the underlying llama.cpp project. Remember to test using the latest Ollama release or main branch build and to test with the feature disabled to ensure it's actually related to K/V context cache quantisation. It's likely to be Ollama if it's related to: - • Enabling/disabling the feature. - • Memory estimations (e.g. how many layers are offloaded to each GPU). It's more likely to a bug in llama.cpp if it's related to: - • Performance. - • ROCm support. - • Model compatibility. - • Quality issues. When logging a bug: - • You should first [search for existing issues in the llama.cpp project](https://github.com/ggerganov/llama.cpp/issues?q=sort%3Aupdated-desc+is%3Aissue+k%2Fv+quantization). - • Reach out to the community to see if others have experienced the issue. - • Ollama has a [Discord server](https://discord.gg/ollama) where you can discuss issues, although be mindful that **Discord is an information black hole and is not well suited to knowledge discovery or issue tracking**. - • If you find an issue do not add a comment such as "+1" or "I have this issue too" - instead use an emoji reaction to indicate your support and only comment if you have valuable information to add. Finally, if you can't find an existing issue, you can create an issue (in the relevant project), ensuring you include your hardware configuration, software versions, environmental settings, the model you're using, the context size, the quantisation level and any other relevant information. > [!WARNING] Log Bugs Responsibly > > Remember: You're _logging a bug_ to a free and open source project, not requesting _support_ from a paid service. > > Be patient, respectful and provide as much information as you can to help the developers diagnose and fix the issue if they have the time and resources to do so. --- ## Further Reading - • [The PR to add K/V context cache quantisation to Ollama.](https://github.com/ollama/ollama/pull/6279) - • [Matt William's YouTube video on the PR](https://www.youtube.com/watch?v=RFaMiQ97EoE) - • [Understanding Quantisation](/2024/07/understanding-ai/llm-quantisation-through-interactive-visualisations/) - • Ollama - • [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-set-the-quantization-type-for-the-kv-cache) - • [Ollama Blog](https://ollama.com/blog) - • [Ollama Releases](https://github.com/ollama/ollama/releases) - • [HuggingFace's blog post on K/V cache quantisation](https://huggingface.co/blog/kv-cache-quantization), which provides a more technical deep dive into the topic in the context of Transformers - • Related llama.cpp PRs and performance measurements (note: these are now quite old and things have likely improved since): - • [ggerganov/llama.cpp#7527](https://github.com/ggerganov/llama.cpp/pull/7527#issuecomment-2132341565) - • [ggerganov/llama.cpp#7412](https://github.com/ggerganov/llama.cpp/pull/7412#issuecomment-2120427347) --- ## Socials Discuss this post on: - • [HackerNews](https://news.ycombinator.com/item?id=42323953) - • [Twitter](https://x.com/ollama/status/1864487185443115378) - • [LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:7270248388057542656/) - • [Mastodon](https://aus.social/@s_mcleod/113597789410718453) - • [Reddit](https://www.reddit.com/r/LocalLLaMA/comments/1h62u1p/ollama_has_merged_in_kv_cache_quantisation/) - [Will AI Take My Job?](https://smcleod.net/2024/11/will-ai-take-my-job/index.md): Will AI Take My Job? --- title: "Will AI Take My Job?" date: 2024-11-29T22:00:02+00:00 tags: ["AI", "LLM", "tech", "jobs", "capitalism", "profit", "automation", "work", "economy"] author: "Sam McLeod" draft: false description: "It's probably not so much AI itself you have to fear." searchHidden: false --- TLDR; Maybe. Longer answer: Capitalism sucks, hard. It's the bad card we've been dealt, for-profit companies will always be looking to reduce costs and increase profits. If this means they can reduce expenses by automating an activity - they will (eventually). To look at this another way - if the output of your job is repeatable and not creative - one could argue that (other than providing you income), it might also not be the best use of your time. For-profit companies will continue to focus on profits over people, governments will continue to be slow to act and life will be very hard for many people not in a position to adapt - made worse back the lack of universal basic income or similar. If I was ruthless capitalist tycoon - I would say something like "evolution isn't mandatory, but neither is survival", but I'm not, and that's not helpful to humanity. In reality while there's a lot of nuance to this discussion, I think it's of the upmost importance to focus efforts on awareness, cross-skilling, re-skilling both yourself and those around you, to push for a more equitable society where the benefits of automation are shared by all (it's good to have dreams, right?) because like it or not - things are changing, and they're changing fast. Work won't go away. The work we value will change. New work will be created and employers will _(eventually)_ adapt - but it's unlikely to meet the pace of change. Until such time as the world is no longer built on the backs of working to make the rich richer, It's not AI itself you have to fear, but the companies that place profits over people and for those that are not in a position to adapt. - [Generating Diagrams with with AI / LLMs](https://smcleod.net/2024/10/generating-diagrams-with-with-ai-/-llms/index.md): Generating Diagrams with with AI / LLMs --- url: "/2024/10/generating-diagrams-with-with-ai-/-llms/" # keeps the live URL: Hugo 0.166 no longer keeps "/" from the title in :title slugs title: "Generating Diagrams with with AI / LLMs" date: 2024-10-08T01:00:10+00:00 tags: ["AI", "tools", "LLM", "tech", "diagrams", "automation", "mermaid"] author: "Sam McLeod" draft: false description: "Generating diagrams with AI / LLMs" searchHidden: false cover: image: "diagram-gen.png" alt: "DiagramGen" hidden: false --- The two tools I use with AI / LLMs for generating diagrams are [Mermaid](https://mermaid.js.org) and [Excalidraw](https://excalidraw.com). - [Mermaid](https://mermaid.js.org) (or MermaidJS) is a popular diagramming library and format supported by many tools and is often rendered inside markdown (e.g. in a readme.md). - [Excalidraw](https://excalidraw.com) is an excellent, free and open source diagramming and visualisation tool. - I also often make use of a third party [Obsidian plugin for Excalidraw](https://github.com/zsviczian/obsidian-excalidraw-plugin). ![diagram-gen](diagram-gen.png) ## Excalidraw It has a 'generate diagram with AI' feature which if you're using the excalidraw.com online editor offers a few free generations each day (I think this uses a low-end OpenAI model). If you're running Excalidraw locally or by using the brilliant Obsidian plugin - you can provide any OpenAI compatible API endpoint and model for AI generations. Behind the scenes Excalidraw AI generates and then renders MermaidJS. Many LLMs are quite capable of generating valid Mermaid syntax, as with everything in GenAI - the investment in how you prompt and what model you pick will greatly impact the quality of the output you get. ## Mermaid Also known as MermaidJS - Mermaid is a popular diagramming library and format supported by many tools and is often rendered inside markdown (e.g. in a readme.md). - Syntax: https://mermaid.js.org/intro/syntax-reference.html - Config Options: https://mermaid.js.org/config/schema-docs/config.html Playgrounds: - https://www.mermaidchart.com/play - https://mermaid.live/edit - https://www.mermaidflow.app/editor ## Prompting The most important two things to remember when getting LLMs to generate diagrams are: 1. State that you expect the output to be in MermaidJS format. 2. State the type of diagram you want ([flowchart](https://mermaid.js.org/syntax/flowchart.html), [sequence](https://mermaid.js.org/syntax/sequenceDiagram.html), [class](https://mermaid.js.org/syntax/classDiagram.html), [gantt](https://mermaid.js.org/syntax/gantt.html) etc.) from the supported mermaid types. ### Example System Prompt > You are a MermaidJS expert. Generate a valid, syntactically correct MermaidJS diagram based on the description I provide. Follow these guidelines: > > 1. Use the appropriate diagram type (flowchart, sequence, class, etc.) based on the context. > 2. Ensure all syntax is correct and up-to-date with the latest MermaidJS version. > 3. Use clear, concise node and relationship labels. > 4. Implement appropriate styling and colors to enhance readability. > 5. Include a title using the `graph TB` syntax for top-to-bottom diagrams or `graph LR` for left-to-right. > 6. Add comments to explain complex parts of the diagram. > 7. Optimise the layout for clarity and minimal crossing lines. > 8. Use British English spelling for all text elements (labels, titles, etc.). > > Respond ONLY with the MermaidJS code, without any introduction, explanation, or conclusion. Do not use backticks or any other formatting other than starting the response with three backticks and mermaidjs suitable for using in markdown, then begin your response with the diagram type declaration and end it with the last line of MermaidJS code and closing backticks. ### Example Mermaid Prompt Below is an example prompt for generating the sequence diagram pictured above: > Create a sequence diagram in mermaid depicting a workflow where a developer is uplifting an application from AngularJS to ReactJS leveraging AI tooling such as IDE Github Copilot, Interactive CLI AI tooling such as Aider powered by LLMs via an API. > > Written text should be in British English spelling for labels, The diagram should be informal in nature suitable for excalidraw in standard mermaidjs format. > > It should include flows such as: > > ```plaintext > developer -> AI to assess current application state and generate step by step plan for uplift -> developer review plan -> refinements needed? > | yes -> refine prompt / scope -> repeat > | no -> Use AI to generate plan documentation -> AI attempt first-parse uplift -> developer reviews -> change in tactic needed? > | yes -> selectively use AI tooling to uplift specific component or functionality > | no -> continue with AI first-pass uplift -> developer reviews AI generated tests --> AI generated tests sufficient/valid? > | yes -> continue with AI generated tests > | no -> developer writes tests in IDE assisted by AI tooling > | continue with AI generated tests -> developer reviews AI generated code -> AI generated code sufficient/valid? > | yes -> continue with AI generated code > | no -> developer writes code in IDE assisted by AI tooling > | Use AI tooling to improve code quality, test coverage and documentation > ``` All I added afterwards was some nice colouring. - [Ingest: Streamlining Content Preparation for LLMs](https://smcleod.net/2024/07/ingest-streamlining-content-preparation-for-llms/index.md): Ingest: Streamlining Content Preparation for LLMs --- title: "Ingest: Streamlining Content Preparation for LLMs" date: 2024-07-29T01:00:10+00:00 tags: ["AI", "tools", "LLM", "tech", "cli", "golang", "automation"] author: "Sam McLeod" draft: false description: "A CLI tool for parsing directories into LLM-friendly markdown" searchHidden: false cover: image: "https://raw.githubusercontent.com/sammcj/ingest/main/screenshot.png" alt: "Ingest CLI Screenshot" hidden: false --- [Ingest](https://github.com/sammcj/ingest/) is a tool I've written to make my life easier when preparing content for LLMs. It parses directories of plain text files, such as source code, documentation etc... into a single markdown file suitable for ingestion by AI/LLMs. ![Ingest CLI Screenshot](https://raw.githubusercontent.com/sammcj/ingest/main/screenshot.png) Ingest can also estimate vRAM requirements for a given model, quantisation and context length: ![Ingest vRAM Estimate Screenshot](https://raw.githubusercontent.com/sammcj/ingest/main/screenshot2.png) ## Features - Traverse directory structures and generate a tree view - Include/exclude files based on glob patterns - Estimate vRAM requirements and check model compatibility using another package I've created called [quantest](https://github.com/sammcj/quantest) - Parse output directly to LLMs such as Ollama or any OpenAI compatible API for processing - Generate and include git diffs and logs - Count approximate tokens for LLM compatibility - Customisable output templates - Copy output to clipboard (when available) - Export to file or print to console - Optional JSON output Ingest Intro ("Podcast" Episode): ## Installation ### go install (recommended) Make sure you have Go installed on your system, then run: ```shell go install github.com/sammcj/ingest@HEAD ``` ### Manual install 1. Download the latest release from the [releases page](https://github.com/sammcj/ingest/releases) 2. Move the binary to a directory in your PATH, e.g. `mv ingest* /usr/local/bin/ingest` ## Usage Basic usage: ```shell ingest [flags] ``` ingest will default to the current working directory if no path is provided, e.g: ```shell $ ingest ⠋ Traversing directory and building tree... [0s] [ℹ️] Tokens (Approximate): 15,945 [✅] Copied to clipboard successfully. ``` Generate a prompt from a directory, including only Python files: ```shell ingest -i "**/*.py" /path/to/project ``` Generate a prompt with git diff and copy to clipboard: ```shell ingest -d /path/to/project ``` Generate a prompt for multiple files/directories: ```shell ingest /path/to/project /path/to/other/project ``` Generate a prompt and save to a file: ```shell ingest -o output.md /path/to/project ``` ### VRAM Estimation and Model Compatibility Ingest includes a feature to estimate VRAM requirements and check model compatibility using the [Gollama](https://github.com/sammcj/gollama)'s vramestimator package. This helps you determine if your generated content will fit within the specified model, VRAM, and quantization constraints. To use this feature, add the following flags to your ingest command: ```shell ingest --vram --model [--memory ] [--quant ] [--context ] [--kvcache ] [--quanttype ] [other flags] ``` Examples: Estimate VRAM usage for a specific context: ```shell ingest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant q4_k_m --context 2048 --kvcache q4_0 . # Estimated VRAM usage: 5.35 GB ``` Calculate maximum context for a given memory constraint: ```shell ingest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant q4_k_m --memory 6 --kvcache q8_0 . # Maximum context for 6.00 GB of memory: 5069 ``` Find the best BPW (Bits Per Weight): ```shell ingest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --memory 6 --quanttype gguf . # Best BPW for 6.00 GB of memory: IQ3_S ``` The tool also works for exl2 (ExllamaV2) models: ```shell ingest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant 5.0 --context 2048 --kvcache q4_0 . # For exl2 models ingest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant 5.0 --memory 6 --kvcache q8_0 . # For exl2 models ``` When using the VRAM estimation feature along with content generation, ingest will provide information about the generated content's compatibility with the specified constraints: ```shell ingest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --memory 8 --quant q4_0 . ⠋ Traversing directory and building tree... [0s] [ℹ️] 14,702 Tokens (Approximate) Maximum context for 8.00 GB of memory: 10240 Generated content (14,702 tokens) fits within maximum context. Top 5 largest files (by estimated token count): 1. /Users/samm/git/sammcj/ingest/main.go (4,682 tokens) 2. /Users/samm/git/sammcj/ingest/filesystem/filesystem.go (2,694 tokens) 3. /Users/samm/git/sammcj/ingest/README.md (1,895 tokens) 4. /Users/samm/git/sammcj/ingest/utils/utils.go (948 tokens) 5. /Users/samm/git/sammcj/ingest/config/config.go (884 tokens) [✅] Copied to clipboard successfully. ``` Available flags for VRAM estimation: - `--vram`: Enable VRAM estimation and model compatibility check - `--model`: Specify the model ID to check against (required for estimation) - `--memory`: Specify the available memory in GB for context calculation (optional) - `--quant`: Specify the quantization type (e.g., q4_k_m) or bits per weight (e.g., 5.0) - `--context`: Specify the context length for VRAM estimation (optional) - `--kvcache`: Specify the KV cache quantization (fp16, q8_0, or q4_0) - `--quanttype`: Specify the quantization type (gguf or exl2) Ingest will provide appropriate output based on the combination of flags used, such as estimating VRAM usage, calculating maximum context, or finding the best BPW. If the generated content fits within the specified constraints, you'll see a success message. Otherwise, you'll receive a warning that the content may not fit. ## LLM Integration Ingest can pass the generated prompt to LLMs that have an OpenAI compatible API such as [Ollama](https://ollama.com) for processing. ```shell ingest --llm /path/to/project ``` By default this will use any prompt suffix from your configuration file: ```shell ./ingest utils.go --llm ⠋ Traversing directory and building tree... [0s] This is Go code for a file named `utils.go`. It contains various utility functions for handling terminal output, clipboard operations, and configuration directories. ... ``` You can provide a prompt suffix to append to the generated prompt: ```shell ingest --llm -p "explain this code" /path/to/project ``` ## Configuration Ingest uses a configuration file located at `~/.config/ingest/config.json`. You can make Ollama processing run without prompting setting `"llm_auto_run": true` in the config file. The config file also contains: - `llm_model`: The model to use for processing the prompt, e.g. "llama3.1:8b-q5_k_m". - `llm_prompt_prefix`: An optional prefix to prepend to the prompt, e.g. "This is my application." - `llm_prompt_suffix`: An optional suffix to append to the prompt, e.g. "explain this code" Ingest uses the following directories for user-specific configuration: - `~/.config/ingest/patterns/exclude`: Add .glob files here to exclude additional patterns. - `~/.config/ingest/patterns/templates`: Add custom .tmpl files here for different output formats. These directories will be created automatically on first run, along with README files explaining their purpose. ### Flags - `-i, --include`: Patterns to include (can be used multiple times) - `-e, --exclude`: Patterns to exclude (can be used multiple times) - `--include-priority`: Include files in case of conflict between include and exclude patterns - `--exclude-from-tree`: Exclude files/folders from the source tree based on exclude patterns - `--tokens`: Display the token count of the generated prompt - `-c, --encoding`: Optional tokeniser to use for token count - `-o, --output`: Optional output file path - `--llm`: Send the generated prompt to an OpenAI compatible LLM server (such as Ollama) for processing - `-p, --prompt`: Optional prompt suffix to append to the generated prompt - `-d, --diff`: Include git diff - `--git-diff-branch`: Generate git diff between two branches - `--git-log-branch`: Retrieve git log between two branches - `-l, --line-number`: Add line numbers to the source code - `--no-codeblock`: Disable wrapping code inside markdown code blocks - `--relative-paths`: Use relative paths instead of absolute paths - `-n, --no-clipboard`: Disable copying to clipboard - `-t, --template`: Path to a custom Handlebars template - `--json`: Print output as JSON - `--pattern-exclude`: Path to a specific .glob file for exclude patterns - `--print-default-excludes`: Print the default exclude patterns - `--print-default-template`: Print the default template - `--report`: Print the largest parsed files - `--verbose`: Print verbose output - `-V, --version`: Print the version number (WIP - still trying to get this to work nicely) ### Excludes You can get a list of the default excludes by parsing `--print-default-excludes` to ingest. These are defined in [defaultExcludes.go](https://github.com/sammcj/ingest/blob/main/filesystem/defaultExcludes.go). To override the default excludes, create a `default.glob` file in `~/.config/ingest/patterns/exclude` with the patterns you want to exclude. ### Templates Templates are written in standard [go templating syntax](https://pkg.go.dev/text/template). You can get a list of the default templates by parsing `--print-default-template` to ingest. These are defined in [template.go](https://github.com/sammcj/ingest/blob/main/template/template.go). To override the default templates, create a `default.tmpl` file in `~/.config/ingest/patterns/templates` with the template you want to use by default. ## Wrapping Up Ingest is a daily time-saver for me when working with LLMs. The code is open source and available on [Github](https://github.com/sammcj/ingest/) - Contributions are welcome, Please feel free to submit a Pull Request. As always, if you find any bugs or have ideas for improvements, don't hesitate to open an issue or submit a PR. Happy ingesting! - [LLM Parameter Playground](https://smcleod.net/2024/07/llm-parameter-playground/index.md): LLM Parameter Playground --- title: "LLM Parameter Playground" date: 2024-07-20T01:00:10+00:00 # weight: 1 # aliases: ["/first"] tags: ["AI", "tools","LLM", "tech", "playground", "parameters", "inference"] author: "Sam McLeod" draft: false description: "A web app to explore the effects of different inference parameters on LLMs." searchHidden: false cover: image: "parameter-playground.webp" alt: "" hidden: false --- Here's a fun little tool I've been hacking on to explore the effects of different inference parameters on LLMs. You can find the code and instructions for running it locally on [GitHub](https://github.com/sammcj/parameter-playground). [![](parameter-playground.webp)](https://github.com/sammcj/parameter-playground) It started as a fork of rooben-me's [tone-changer-open](https://github.com/rooben-me/tone-changer-open), which itself was a "fork" of Figma's tone generator, I've made quite a few changes to make it more focused on local LLMs and advanced parameter exploration. - [Code, Chaos, and Copilots (AI/LLM Talk July 2024)](https://smcleod.net/2024/07/code-chaos-and-copilots-ai/llm-talk-july-2024/index.md): Code, Chaos, and Copilots (AI/LLM Talk July 2024) --- url: "/2024/07/code-chaos-and-copilots-ai/llm-talk-july-2024/" # keeps the live URL: Hugo 0.166 no longer keeps "/" from the title in :title slugs title: "Code, Chaos, and Copilots (AI/LLM Talk July 2024)" date: 2024-07-18T01:00:10+00:00 # weight: 1 # aliases: ["/first"] tags: ["AI", "tools", "quantisation", "LLM", "gguf", "talk", "tech", "keynote", "presentation"] author: "Sam McLeod" draft: false description: "Integrating AI into Daily Development" searchHidden: false cover: image: "code-chaos-copilots-screenshot.png" alt: "" hidden: false --- Code, Chaos, and Copilots is a talk I gave in July 2024 as an intro to how I use AI/LLMs to augment my capabilities every day. - What I use AI/LLMs for - Prompting tips - Codegen workflow - Picking the right models - Model formats - Context windows - Quantisation - Model servers - Inference parameters - Clients & tools - Getting started cheat-sheets [Download Slide Deck](https://media.githubusercontent.com/media/sammcj/smcleod_files/master/slides/code-chaos-copilots/smcleod-2024-07-18-code-chaos-copilots.pdf) [![Download Slide Deck](code-chaos-copilots-screenshot.png)](https://media.githubusercontent.com/media/sammcj/smcleod_files/master/slides/code-chaos-copilots/smcleod-2024-07-18-code-chaos-copilots.pdf) ### _Disclaimer_ I'm not a ML Engineer or data scientist, As such, the information presented here is based on my understanding of the subject and may not be 100% accurate or complete. If you find errors, or have feedback on my understanding / assumptions - help me learn by [raising an issue](https://github.com/sammcj/smcleod/issues/new/choose)! ### Recording Personally I think the slides are more valuable than the recording, but if you're interested in hearing me try to rush through some of the content, you can watch it here: [![Code, Chaos, and Copilots - Integrating AI into Daily Development - Sam McLeod 2024-07-18](https://img.youtube.com/vi/8VY6dY-JiYQ/0.jpg)](https://www.youtube.com/watch?v=8VY6dY-JiYQ) - [Understanding AI/LLM Quantisation Through Interactive Visualisations](https://smcleod.net/2024/07/understanding-ai/llm-quantisation-through-interactive-visualisations/index.md): Understanding AI/LLM Quantisation Through Interactive Visualisations --- url: "/2024/07/understanding-ai/llm-quantisation-through-interactive-visualisations/" # keeps the live URL: Hugo 0.166 no longer keeps "/" from the title in :title slugs title: "Understanding AI/LLM Quantisation Through Interactive Visualisations" date: 2024-07-17T01:00:10+00:00 # weight: 1 # aliases: ["/first"] tags: ["AI", "tools", "quantisation", "LLM", "gguf", "dashboard"] author: "Sam McLeod" draft: false description: "AI/LLM Quantisation Visualised" searchHidden: false cover: image: "quant-neural.png" alt: "" --- AI models ("LLMs" in this case) have inherently large sizes and computational requirements that often pose challenges for deployment and use. ## _Disclaimer_ I am not a ML or data scientist. I am simply an engineer with an interest in AI. This project is a result of my personal interest in understanding the impact of quantisation on LLMs. The visualisations are based on my understanding of the subject and may not be 100% accurate or complete. I encourage you to verify the information presented here with other sources. ## Quantisation Quantisation, a technique to reduce model size and memory footprint, is often confusing for newcomers, and understanding the trade-offs involved in the various quantisation types can be complex. Quantisation refers to the process of converting model weights from higher to lower precision data types (e.g. floating point -> integer) ![](quant-neural.png) As a thought experiment, and for my own learning, I've created an interactive dashboard to help myself and other users understand the impact of quantisation on LLMs. This (_somewhat_) interactive dashboard aims to demystify LLM quantisation by providing visual representations of key metrics and trade-offs. ### Colour Spectrum Analogy Imagine the model data to be the colour spectrum (pictured as 16 bits here) ![](16-to-8bit.png) If we quantise the data to 8 bits we are removing (thus compressing) parts of the data based on a set of rules. We can still see a wide range of "colours" but we lose some of the detail. _Note: This is a crude analogy, Modern quantisation techniques have smarts that selectively quantise parts of the model to varying degrees to reduce the loss._ ## Dashboard The data is mainly focused on GGUF quantisation, however the visualisations can be used to understand other quantisation and model formats as well. I plan to add more quantisation techniques and models in the future. If you find errors - please do let me know! I want to correct my understanding and improve the visualisations over time. The dashboard is open source and available at: https://github.com/sammcj/quant/ and I welcome contributions and feedback. _Note: This chart requires JavaScript and may not render properly on all mobile devices with small screens._ {{< quantisationDashboard >}} ## Visualisations Explained ### Perplexity vs Compression Chart This chart illustrates the relationship between model compression and perplexity increase. It aims to help understand how different quantisation levels affect model quality and size reduction. ### Quantisation Spectrum Heatmap This table provides a quick overview of various quantisation types, their relative sizes, qualities, and performance characteristics on different hardware (CUDA and Metal). It's for quickly comparing different quantisation options at a glance. ### Quantisation Sweet Spots This table helps to identify the optimal quantisation level for different model sizes and VRAM constraints. It's useful for those working with specific hardware limitations. ### Quantisation Efficiency These charts dive deeper into the efficiency of quantisation across different model sizes, and the resulting perplexity. ### Decision Tree This visual guide helps users navigate the decision-making process for selecting the most appropriate quantisation level based on their priorities (quality vs. size) and hardware constraints. ## References - https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix - https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 - https://github.com/ggerganov/llama.cpp/discussions/5962 - https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods/ - https://github.com/ggerganov/llama.cpp/blob/master/examples/perplexity/README.md - https://github.com/ggerganov/llama.cpp/discussions/5006 - https://huggingface.co/docs/optimum/en/concept_guides/quantization - https://github.com/matt-c1/llama-3-quant-comparison - [Rating AI Tools](https://smcleod.net/2024/07/rating-ai-tools/index.md): Rating AI Tools --- title: "Rating AI Tools" date: 2024-07-03T01:00:10+00:00 # weight: 1 # aliases: ["/first"] tags: ["AI", "tools", "ratings"] author: "Sam McLeod" draft: false description: "Subjective ratings of the notable AI tools I've tried." searchHidden: false cover: image: "" alt: "" hidden: false --- I spend a lot of time working with AI tools and often get asked for recommendations. The following is a list of some of the more notable tools I've tried and rated them based on my experience. Just because something has a low rating doesn't mean it's bad - it just means it didn't work well for me and I wouldn't personally recommend it. > [!WARNING] > This is now _very_ outdated (2024) --- Hover over the heatmap to see the ratings and any notes as I add them. _Last Updated: 2024-07-14_ {{< heatmap >}} More coming soon! - [Gollama: Ollama Model Manager](https://smcleod.net/2024/06/gollama-ollama-model-manager/index.md): Gollama: Ollama Model Manager --- title: "Gollama: Ollama Model Manager" date: 2024-06-01T01:00:10+00:00 lastmod: 2024-07-14T01:00:10+00:00 # weight: 1 # aliases: ["/first"] tags: ["Markdown", "Go", "golang", "ollama", "AI", "LLM", "LMStudio"] author: "Sam McLeod" draft: false description: "Gollama is a Go-based client for Ollama for managing models." # canonicalURL: "https://canonical.url/to/page" searchHidden: false cover: image: "gollama-v1.0.0.jpg" alt: "Gollama TUI" hidden: false --- - [Gollama on Github](https://github.com/sammcj/gollama) ![](gollama-logo-open-graph.png) Gollama is a client for Ollama for managing models. It provides a TUI for listing, filtering, sorting, selecting, inspecting (coming soon!) and deleting models and can link Ollama models to LM-Studio. The project started off as a rewrite of my [llamalink](https://smcleod.net/2024/03/llamalink-ollama-to-lm-studio-llm-model-linker/) project, but I decided to expand it to include more features and make it more user-friendly. ![](gollama-v1.0.0.jpg) ## Features - List available models - Display metadata such as size, quantisation level, model family, and modified date - Edit / update a model's Modelfile - Sort models by name, size, modification date, quantisation level, family etc - Select and delete models - Run and unload models - Inspect model for additional details - Link models to LM Studio - Copy / rename models - Pull (update) existing models - Pull (get) new models - Push models to a registry - Show running models - Has some cool bugs ## Installation From go: ```shell go install github.com/sammcj/gollama@HEAD ``` From Github: Download the most recent release from the [releases page](https://github.com/sammcj/gollama/releases) and extract the binary to a directory in your PATH. e.g. `zip -d gollama-v1.8.1.zip -d gollama && mv gollama /usr/local/bin` ## Usage To run the `gollama` application, use the following command: ```sh gollama ``` _Tip_: I like to alias gollama to `g` for quick access: ```shell echo "alias g=gollama" >> ~/.zshrc ``` ### Inspect ![](./gollama-inspect.png) ### Top ![](./gollama-top.jpg) ### Simple model listing Gollama can also be called with `-l` to list models without the TUI. ```shell ./gollama -l ``` ![](./cli-list.jpg) ### Key Bindings - `Space`: Select - `Enter`: Run model (Ollama run) - `i`: Inspect model - `t`: Top (show running models) - `D`: Delete model - `e`: Edit model **new** - `c`: Copy model - `U`: Unload all models - `p`: Pull an existing model **new** - `g`: Pull (get) new model **new** - `P`: Push model - `n`: Sort by name - `s`: Sort by size - `m`: Sort by modified - `k`: Sort by quantisation - `f`: Sort by family - `l`: Link model to LM Studio - `L`: Link all models to LM Studio - `r`: Rename model _**(Work in progress)**_ - `q`: Quit #### Command-line Options - `-l`: List all available Ollama models and exit - `-s `: Search for models by name **new** - OR operator (`'term1|term2'`) returns models that match either term - AND operator (`'term1&term2'`) returns models that match both terms - `-e `: Edit the Modelfile for a model **new** - `-ollama-dir`: Custom Ollama models directory - `-lm-dir`: Custom LM Studio models directory - `-cleanup`: Remove all symlinked models and empty directories and exit - `-no-cleanup`: Don't cleanup broken symlinks - `-u`: Unload all running models - `-v`: Print the version and exit ##### Edit Gollama can be called with `-e` to edit the Modelfile for a model. ```shell gollama -e my-model ``` ##### Search Gollama can be called with `-s` to search for models by name. ```shell gollama -s my-model # returns models that contain 'my-model' gollama -s 'my-model|my-other-model' # returns models that contain either 'my-model' or 'my-other-model' gollama -s 'my-model&instruct' # returns models that contain both 'my-model' and 'instruct' ``` ## Configuration Gollama uses a JSON configuration file located at `~/.config/gollama/config.json`. The configuration file includes options for sorting, columns, API keys, log levels etc... Example configuration: ```json { "default_sort": "modified", "columns": [ "Name", "Size", "Quant", "Family", "Modified", "ID" ], "ollama_api_key": "", "ollama_api_url": "http://localhost:14434", "lm_studio_file_paths": "", "log_level": "info", "log_file_path": "gollama.log", "sort_order": "Size", "strip_string": "my-private-registry.internal/" } ``` The strip string option can be used to remove a prefix from model names as they are displayed in the TUI. This can be useful if you have a common prefix such as a private registry that you want to remove for display purposes. ## Installation and build from source 1. Clone the repository: ```shell git clone --depth=1 https://github.com/sammcj/gollama.git cd gollama ``` 2. Build: ```shell go get make build ``` 3. Run: ```shell ./gollama ``` ## Logging Logs can be found in the `gollama.log` which is stored in `$HOME/.config/gollama/gollama.log` by default. The log level can be set in the configuration file. ## License Copyright © 2024 Sam McLeod This project is licensed under the MIT License. - [Confuddlement: Download Confluence Spaces as Markdown, Summarise with Ollama](https://smcleod.net/2024/05/confuddlement-download-confluence-spaces-as-markdown-summarise-with-ollama/index.md): Confuddlement: Download Confluence Spaces as Markdown, Summarise with Ollama --- title: "Confuddlement: Download Confluence Spaces as Markdown, Summarise with Ollama" date: 2024-05-23T05:45:10+00:00 # weight: 1 # aliases: ["/first"] tags: ["Confluence", "Markdown", "Go", "golang", "ollama", "AI", "LLM"] author: "Sam McLeod" draft: false description: "Download Confluence Spaces as Markdown, Summarise with Ollama" # canonicalURL: "https://canonical.url/to/page" searchHidden: false # cover: # image: "https://github.com/sammcj/confuddlement/screenshots/screenshot.png?raw=true" # image path/url # alt: : "placeholder" # caption: "placeholder" # relative: true # hidden: false --- - [Confuddlement on Github](https://github.com/sammcj/confuddlement) I was tired of manually downloading Confluence pages and converting them to Markdown, so I wrote a small command-line tool designed to simplify this process. Confuddlement is a Go-based tool that uses the Confluence REST API to fetch page content and convert it to Markdown files. It can fetch pages from multiple spaces, skip pages that have already been fetched, and summarise the content of fetched pages using the Ollama API. ```plain $ go run ./main.go Confuddlement 0.3.0 Spaces: [COOLTEAM, MANAGEMENT] Fetching content from space COOLTEAM COOLTEAM (Totally Cool Team Homepage) Retrospectives Decision log Development Onboarding Saved page COOLTEAM - Feature List to ./confluence_dump/COOLTEAM - Feature List.md Skipping page 7. Support, less than 300 characters MANAGEMENT (Department of Overhead and Bureaucracy) Painful Change Management Illogical Diagrams Saved page ./confluence_dump/Painful Change Management.md Saved page Illogical Diagrams to ./confluence_dump/Ilogical Diagrams.md Done! ``` ```plain $ go run ./main.go summarise Select a file to summarise: 0: + COOLTEAM - Feature List 1: + Painful Change Management 2: + Illogical Diagrams Enter the number of the file to summarise: 1 Summarising Painful Change Management... "Change management in the enterprise is painful and slow. It involves many forms and approvals." ``` ```plain go run main.go -q 'who is the CEO?' -s 'management' -r 2 Querying the LLM with the prompt 'who is the CEO?'... "The CEO of the company is Peewee Herman." ``` ## Usage ### Running the Program 1. Copy [.env.template](https://github.com/sammcj/confuddlement/blob/main/.env.template) to `.env` and update the environment variables. 2. Run the program using the command `go run main.go` or build the program using the command `go build` and run the resulting executable. 3. The program will fetch Confluence pages and save them as Markdown files in the specified directory. #### Querying the documents with AI You can summarise the content of a fetched page using the Ollama API by running the program with the `summarise` argument: ```shell go run main.go summarise ``` To perform a custom query, you can use the `query` argument: - `-q`: The query to to provide to the LLM. - `-s`: The search term to match documents against. - `-r`: The number of lines before and after the search term to include in the context to the LLM. ```shell go run main.go -q 'who is the CEO?' -s 'management' -r 2 Querying the LLM with the prompt 'who is the CEO?'... "The CEO of the company is Peewee Herman." ``` ### Environment Variables **Customisation options** Confuddlement has several customisation options: - Setting the directory where Markdown files will be saved - Specifying the number of pages to fetch per API request - Defining the minimum length of a page to be considered valid - Skipping pages that have already been fetched - Summarisation of fetched pages using the Ollama API The following environment can be set to configure the program: > - `CONFLUENCE_DUMP_DIR`: The directory where the Markdown files will be saved. > - `CONFLUENCE_LIMIT`: The number of pages to fetch per API request. > - `CONFLUENCE_BASE_URL`: The base URL of the Confluence instance. > - `CONFLUENCE_USER`: The username to use for API authentication. > - `CONFLUENCE_SPACES`: The space keys to fetch pages from, separated by commas. > - `CONFLUENCE_API_TOKEN`: The API token to use for authentication. > - `DELETE_PREVIOUS_DUMP`: Set to `true` to delete the previous dump directory (and state) before fetching pages. > - `MIN_PAGE_LENGTH`: The minimum length of a page to be considered valid. > - `SKIP_FETCHED_PAGES`: Set to `true` to skip pages that have already been fetched. > - `DEBUG`: Set to `true` to enable debug logging. > - `OLLAMA_HOST`: The host of the Ollama API (optional, only required for summarisation). > - `OLLAMA_MODEL`: The model to use for summarisation (optional, only required for summarisation). > - `OLLAMA_NUM_CTX`: The number of context sentences to include in the summary (optional, only required for summarisation). > - `OLLAMA_NUM_PREDICT`: The number of predicted sentences to include in the summary (optional, only required for summarisation). [Confuddlement](https://github.com/sammcj/confuddlement) - [NVApi - Nvidia GPU Monitoring API](https://smcleod.net/2024/05/nvapi-nvidia-gpu-monitoring-api/index.md): NVApi - Nvidia GPU Monitoring API --- title: "NVApi - Nvidia GPU Monitoring API" date: 2024-05-18T11:30:03+00:00 # weight: 1 # aliases: ["/first"] tags: ["AI", "nvidia", "GPU", "Monitoring", "API", "Go", "golang"] author: "Sam McLeod" draft: false description: "Nvidia GPU Monitoring API" # canonicalURL: "https://canonical.url/to/page" searchHidden: false cover: image: "https://github.com/sammcj/NVApi/blob/main/screenshots/home-assistant-integration-2.png?raw=true" # image path/url alt: "NVApi integrated with Home Assistant" # alt text hidden: true # only hide on current single page --- NVApi is a small application I've written for monitoring and presenting utilisation metrics from Nvidia GPUs. This can be used to monitor GPU memory, temperature, power usage, and utilisation of GPUs in a system and can easily be integrated into tools such as HomeAssistant or Prometheus. The package uses the Nvidia Management Library (NVML) and provides a simple API for monitoring Nvidia GPUs along with a basic GUI client. - [NVApi on Github](https://github.com/sammcj/nvapi) ![](https://github.com/sammcj/NVApi/blob/main/screenshots/home-assistant-integration-2.png?raw=true) ## Usage ### Docker Container _Note: The Dockerfile is a work in progress, the current container image is bloated and not optimised for size yet._ The application can be run as a container: ```shell docker build -t nvapi:latest . ``` Or using docker-compose see [docker-compose.yml](https://github.com/sammcj/NVApi/blob/main/docker-compose.yml) for an example configuration. ### Local Installation To run the API, use the following command: ```shell go run main.go -port 9999 -rate 3 ``` This will start the API on port 9999 with a rate limit of 1 request per second. ## API Endpoints ### `/` Returns the current GPU utilisation information in JSON format. ## Query Parameters - `port`: The port number to listen on (default: 9999) - `rate`: The minimum number of seconds between requests (default: 3) ## Example Response ```shell curl http://localhost:9999/gpu ``` ```json [{ "gpu_utilisation": 0, "memory_utilisation": 0, "power_watts": 23, "memory_total_gb": 24, "memory_used_gb": 9.77, "memory_free_gb": 14.23, "memory_usage_percent": 41, "temperature": 35, "fan_speed": 0, "processes": [{ "Pid": 2345831, "UsedGpuMemoryMb": 9678, "Name": "/tmp/ollama630272566/runners/cuda_v12/ollama_llama_server", "Arguments": ["--model", "/home/llm/.ollama/models/blobs/sha256-583c616da14b82930f887f991ab446711da0b029166200b67892d7c9f8f45958", "--ctx-size", "12288", "--batch-size", "512", "--embedding", "--log-disable", "--n-gpu-layers", "33", "--flash-attn", "--parallel", "6", "--port", "42161"] }] }] ``` ![](https://github.com/sammcj/NVApi/blob/main/screenshots/json_response.png?raw=true) ### `/svg` Returns a (very) basic SVG image showing the current GPU utilisation as a bar. ### Home Assistant Integration Example of using the API to integrate with Home Assistant: ```yaml sensors: - platform: rest name: "GPU Utilisation" resource: http://localhost:9999 unit_of_measurement: "%" unique_id: gpu_0 scan_interval: 30 json_attributes: - gpu_utilisation - memory_utilisation - memory_used_gb - memory_free_gb - power_watts - temperature - fan_speed value_template: '{{ value_json[0].gpu_utilisation }}' - platform: rest name: "GPU Memory Utilisation" resource: http://localhost:9999 unit_of_measurement: "%" unique_id: gpu_0_memory_utilisation scan_interval: 30 json_attributes: - memory_utilisation value_template: '{{ value_json[0].memory_utilisation }}' - platform: rest name: "GPU Memory Used" resource: http://localhost:9999 unit_of_measurement: "GB" unique_id: gpu_0_memory_used_gb scan_interval: 30 json_attributes: - memory_used_gb value_template: '{{ value_json[0].memory_used_gb }}' - platform: rest name: "GPU Memory Free" resource: http://localhost:9999 unit_of_measurement: "GB" unique_id: gpu_0_memory_free_gb scan_interval: 30 json_attributes: - memory_free_gb value_template: '{{ value_json[0].memory_free_gb }}' - platform: rest name: "GPU Temperature" resource: http://localhost:9999 unit_of_measurement: "°C" unique_id: gpu_0_temperature scan_interval: 30 json_attributes: - temperature value_template: '{{ value_json[0].temperature }}' - platform: rest name: "GPU Fan Speed" resource: http://localhost:9999 unit_of_measurement: "RPM" unique_id: gpu_0_fan_speed scan_interval: 30 json_attributes: - fan_speed value_template: '{{ value_json[0].fan_speed }}' - platform: rest name: "GPU Power" resource: http://localhost:9999 unit_of_measurement: "W" unique_id: gpu_0_power scan_interval: 30 json_attributes: - power_watts value_template: '{{ value_json[0].power_watts }}' ``` ## NVApi-Tray GUI A simple GUI application that displays the GPU utilisation information from the API. ![](https://github.com/sammcj/NVApi/blob/main/screenshots/NVApiGUI.png?raw=true) This is a work in progress but can be built from the `NVApi-GUI` directory. ```shell cd NVApi-GUI go build ``` - [The Enterprise "Agile" Software Development Manifesto](https://smcleod.net/2024/04/the-enterprise-agile-software-development-manifesto/index.md): The Enterprise "Agile" Software Development Manifesto --- title: 'The Enterprise "Agile" Software Development Manifesto' author: "Sam McLeod" description: "We are pretending to uncover better ways of developing software by talking about it and making others believe we're doing it. Through this facade, we have come to value: Processes and tools over individuals and interactions, Comprehensive documentation over working software, Contract negotiation over customer collaboration, Following a plan over responding to change." categories: [ Agile, Culture, Humour ] keywords: ["tech", "agile", "enterprise", "humour", "culture", "manifesto", "software development"] date: "2024-04-08" images: ["enterprise-agile.jpg"] cover: image: " enterprise-agile.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - agile - culture - humour - software development - Manifesto - enterprise # series: ["AI", "Code"] toc: enable: true auto: true math: false draft: true --- # The Enterprise "Agile" Software Development Manifesto ![](enterprise-agile.jpg) We are pretending to uncover better ways of developing software by talking about it and making others believe we're doing it. Through this facade, we have come to value: ## Processes and tools over individuals and interactions We believe in the almighty power of bureaucracy and red tape. Who needs skilled individuals when you have a convoluted process that nobody understands? Interactions are overrated; just follow the process blindly, and everything will magically work out. We'll even hire an army of Agile coaches to ensure that everyone is following the process, regardless of whether it makes sense or not. ## Comprehensive documentation over working software We love documentation so much that we prioritize it over actual working software. Why bother with functioning code when you can have a 500-page requirements document that nobody reads? We believe that the more documentation we have, the more agile we are. Plus, our Agile coaches will make sure that every single document is templated and filled out correctly, even if it means spending more time on paperwork than on actual development. ## Contract negotiation over customer collaboration Collaboration is for the weak. We prefer to engage in lengthy contract negotiations, ensuring that every single detail is spelled out, even if it means delaying the project by months. Our customers should be kept at arm's length and only contacted when absolutely necessary. And if they dare to suggest a change, we'll make sure to put them through our rigorous change management process, which involves at least five different approval levels and a dozen forms to fill out. ## Following a plan over responding to change Plans are meant to be followed religiously, no matter how outdated or irrelevant they become. Change is the enemy of "Agile" in enterprises. We'd rather stick to a plan that doesn't work than adapt to new information or changing circumstances. And if anyone suggests deviating from the plan, our change management process will ensure that it takes at least three months to get approval, by which time the opportunity will have long passed. That is, while there is value in the items on the left, we pretend to value the items on the right more, but in reality, we don't care about either. Signed by: - The CIO who attended a two-hour "Agile" workshop once and now believes they are an expert - The project manager who thinks "Agile" is just a buzzword and a way to micromanage the team - The developer who's tired of attending daily stand-up meetings that last an hour and being constantly interrupted by Agile coaches - The stakeholder who doesn't understand why nothing gets delivered despite all the "Agile" ceremonies and the army of Agile coaches - The Agile coach who's making a fortune by enforcing meaningless processes and templates, while stifling any actual agility - The change management team who ensures that any attempt to be truly agile is bogged down in bureaucracy and red tape /s - [SuperPrompter - Supercharge your text prompts for AI/LLM image generation](https://smcleod.net/2024/03/superprompter-supercharge-your-text-prompts-for-ai/llm-image-generation/index.md): SuperPrompter - Supercharge your text prompts for AI/LLM image generation --- url: "/2024/03/superprompter-supercharge-your-text-prompts-for-ai/llm-image-generation/" # keeps the live URL: Hugo 0.166 no longer keeps "/" from the title in :title slugs title: "SuperPrompter - Supercharge your text prompts for AI/LLM image generation" author: "Sam McLeod" description: "SuperPrompter, a Python app that uses the SuperPrompt-v1 model to expand short text prompts for AI image generation." categories: [ Tech, AI, LLM, Code ] keywords: ["tech", "ai", "llm", "code", "github", "SDXL", "gguf", "ggml"] date: "2024-03-22" images: [""] cover: image: "superprompter-screenshot.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - LLM - AI - Code series: ["AI", "Code"] toc: enable: true auto: true math: false draft: false --- [SuperPrompter](https://github.com/sammcj/superprompter) is a Python-based application that utilises the [SuperPrompt-v1](https://huggingface.co/roborovski/superprompt-v1) model to generate optimised text prompts for AI/LLM image generation (for use with Stable Diffusion etc...) from user prompts. ![screenshot](superprompter-screenshot.jpg) See [Brian Fitzgerald's Blog](https://brianfitzgerald.xyz/prompt-augmentation/) for a detailed explanation of the SuperPrompt-v1 model and its capabilities / limitations. ## Features - Utilises the [SuperPrompt-v1](https://huggingface.co/roborovski/superprompt-v1) model for text generation. - A basic (aka ugly) graphical user interface built with tkinter. - Customisable generation parameters (max new tokens, repetition penalty, temperature, top p, top k, seed). - Optional logging of input parameters and generated outputs. - Bundling options to include or exclude pre-downloaded model files. - Unloads the models when the application is idle to free up memory. ## Prebuilt Binaries Check [releases](https://github.com/sammcj/superprompter/releases) page to see if there are any prebuilt binaries available for your platform. ## Building ### Prerequisites - Python 3.x - Required Python packages (listed in `requirements.txt`) - python-tk (`brew install python-tk`) ### Installation 1. Clone the repository: ```shell git clone https://github.com/yourusername/SuperPrompter.git ``` 2. Navigate to the project directory: ```shell cd SuperPrompter ``` 3. Create a virtual environment (optional but recommended): ```shell make venv ``` 4. Install the required packages: ```shell make install ``` ### Usage Run the application: ```shell make run ``` - On first run, it will download the [SuperPrompt-v1](https://huggingface.co/roborovski/superprompt-v1) model files. - Subsequent runs will use the downloaded model files. - Once the model is loaded, the main application window will appear. - Enter your prompt in the "Your Prompt" text area. - Adjust the generation parameters (max new tokens, repetition penalty, temperature, top p, top k, seed) as desired. - Click the "Generate" button or press Enter to generate text based on the provided prompt and parameters. - The generated output will be displayed in the "Output" text area. - Optionally, enable logging by checking the "Enable Logging" checkbox. - When enabled, the input parameters and generated outputs will be saved to a log file named `~/.superprompter/superprompter_log.txt` in the user's home directory. ### Bundling SuperPrompter can be bundled into a standalone executable using PyInstaller. The bundling process is automated with a `Makefile` and a `bundle.py` script. To bundle the application: 1. Install deps ```shell make venv make install ``` 2. Check it runs ```shell make run ``` 3. Run the bundling command ```shell make bundle ``` This command will download the SuperPrompt-v1 model files and bundle the application with the model files included. Alternatively, if you want to bundle the application without including the model files (they will be downloaded at runtime), run: ```shell make bundleWithOutModels ``` 4. The bundled executable will be available in the `dist` directory. ## Contributing Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request. ## License I have open sourced the codebase under the MIT License and it is freely available for download on GitHub. - [Llamalink - Ollama to LM Studio LLM Model Linker](https://smcleod.net/2024/03/llamalink-ollama-to-lm-studio-llm-model-linker/index.md): Llamalink - Ollama to LM Studio LLM Model Linker --- title: "Llamalink - Ollama to LM Studio LLM Model Linker" author: "Sam McLeod" description: "This is a simple command-line tool that allows you to easily link Ollama models to LM Studio's directory structure. It simplifies the process of symlinking Ollama models to LM Studio, making it convenient to use the models in both applications." categories: [ Tech, AI, LLM, Code ] keywords: ["tech", "ai", "llm", "code", "github", "ollama", "gguf", "ggml"] date: "2024-03-21" images: [""] cover: image: "llamalink-screenshot.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - LLM - AI - Code series: ["AI", "Code"] toc: enable: true auto: true math: false draft: false --- Two of my most commonly used LLM tools are [Ollama](https://ollama.com/) and [LM Studio](https://lmstudio.ai/). Unfortunately they store their models in different locations and filenames. Manually copying or linking files was a pain, so I wrote a simple command-line tool to automate the process. This is why I created [Llamalink](https://github.com/sammcj/llamalink). ![screenshot](llamalink-screenshot.jpg) Ollama is a cross-platform model server that allows you to run LLMs and manage their models in a similar way to Docker containers and images, while LM Studio is a macOS app that provides a user-friendly interface for running LLMs. I often use both applications to generate text, and I wanted a way to easily link Ollama models to LM Studio's directory structure. This would make it convenient to use the models in both applications. ## Features - Retrieves the list of available Ollama models using the ollama list command. - Displays the list of models and allows the user to select specific models to link or link all models at once. - Retrieves the model path for each selected model using the ollama show --modelfile command. - Creates the necessary directories in the LM Studio directory structure. - Creates symlinks from the Ollama model paths to the corresponding LM Studio model paths. - Removes any existing symlinks before creating new ones to avoid conflicts. - Cleans up any broken symlinks in the LM Studio models directory. - Can be run interactively or non-interactively. ## Usage At the time of writing this, the `llamalink` command provides the following options: - `-a` Link all models without prompting for selection. - `-ollama-dir` Specify a custom Ollama models directory. - `-lm-dir` Specify a custom LM Studio models directory. - `-min-size` Include only models over the given size (in GB or MB). - `-max-size` Include only models under the given size (in GB or MB). - `-q` Quiet operation, only output an exit code at the end. - `-no-cleanup` Don't cleanup broken symlinks. - `-cleanup` Remove all symlinked models and empty directories and exit. - `-h` Print the help message. If no flags are provided, the program will run interactively. --- ## Installation ```plaintext go install github.com/sammcj/llamalink@latest ``` ## Example ```plaintext llamalink Ollama models directory: /Users/samm/.ollama/models LM Studio models directory: /Users/samm/.cache/lm-studio/models Select the models to link to LM Studio: 1. knoopx/hermes-2-pro-mistral:7b-q8_0 2. dolphincoder:15b-starcoder2-q4_K_M 3. llava:13b 4. nomic-embed-text:latest 5. qwen:14b 6. qwen:7b-chat-q5_K_M 7. stable-code:3b-code-q5_K_M 8. tinydolphin:1.1b-v2.8-q5_K_M Enter the model numbers (comma-separated), or press Enter to link all: 3 Model: llava:34b-v1.6-q5_K_M Path: /Users/samm/.ollama/models/blobs/sha256:4ddc7cc65a231db765ecf716d58ed3262e4496847eafcbcf80288fc2c552d9e6 LM Studio model directory: /Users/samm/.cache/lm-studio/models/llava/llava-34b-v1.6-q5-K-M-GGUF Symlinked llava:34b-v1.6-q5_K_M to /Users/samm/.cache/lm-studio/models/llava/llava-34b-v1.6-q5-K-M-GGUF/llava-34b-v1.6-q5-K-M.gguf ``` ## License I have open sourced the codebase under the MIT License and it is freely available for download both on GitHub and via the Go package manager. ## Links - https://github.com/sammcj/llamalink/ - https://ollama.com/ - https://lmstudio.ai/ (_note: LM Studio is not open source_) - [Open source, locally hosted AI powered Siri replacement](https://smcleod.net/2023/11/open-source-locally-hosted-ai-powered-siri-replacement/index.md): Open source, locally hosted AI powered Siri replacement --- title: "Open source, locally hosted AI powered Siri replacement" author: "Sam McLeod" description: "Using an ESP32 S3 Box 3 with Willow, Home Assistant and Large Language Models (LLMs) create a locally hosted, offline, AI powered Siri / Alexa replacement." categories: [ Tech, AI, LLM, IoT, ESP32, Hardware ] keywords: [ "AI", "LLM", "Siri", "Willow", "Homeassistant", "ESP32", "hardware", "IoT" ] date: "2023-11-20T01:00:00Z" images: [""] cover: image: "" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - LLM - AI - HomeAssistant - Willow - ESP32 - IoT series: ["AI"] toc: enable: true auto: false math: false draft: false --- Offline AI / LLM Assistant {{< youtube "_qft28MiVnc" >}} More info on this soon but the basic idea was to use Willow, Home Assistant and local LLM models to create a locally hosted, offline, AI powered Siri replacement and interface it with ESP32 S3 Box 3 devices. ## ESP32 S3 Box 3 The [ESP32 S3 Box 3](https://www.espressif.com/en/news/ESP32-S3-BOX-3) is a small, low cost, ESP32 based board with a microphone, speaker, small touch display, 18650 battery, and a few other nice features. ![](esp32-1.png) ![](esp32-2.jpg) ## Willow [Willow](https://heywillow.io/) is an open source, locally hosted AI powered voice assistant. It is built with Python and runs on Linux, macOS, and Windows. Willow is designed to be extensible and highly configurable. ## Home Assistant [Home Assistant](https://www.home-assistant.io/) is an open source home automation platform. It is able to track and control all devices at home and offer a platform for automating control. Home Assistant allows you to control all your devices without storing any of your data in the cloud. We like to keep your privacy private. ## Docker-Compose Bellow are the various docker-compose and config files relating to the project. As they are - **they will not work**, but may prove useful as a reference / guide. {{< gist sammcj 4bbcc85d7ffd5ccc76a3f8bb8dee1d2b >}} - [Fun with Makefiles - Dynamic Menu Generation](https://smcleod.net/2023/11/fun-with-makefiles-dynamic-menu-generation/index.md): Fun with Makefiles - Dynamic Menu Generation --- title: "Fun with Makefiles - Dynamic Menu Generation" date: 2023-11-14T00:00:00+00:00 lastmod: 2023-11-14T00:00:00+00:00 author: Sam McLeod description: "This reusable snippet will generate a menu of targets from the Makefile. It will use `fzf` if it is installed, otherwise it will use a numbered menu." keywords: ["tech", "linux", "makefile", "make", "programming"] tags: ["linux", "programming"] categories: ["Tech", "Linux"] series: ["Fun With Makefiles"] images: ["makefile-menu.png"] cover: image: "makefile-menu.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles toc: enable: true auto: false math: false draft: false --- ## Dynamic Menu Generation This reusable snippet will generate a menu of targets from the Makefile. It will use `fzf` if it is installed, otherwise it will use a numbered menu. ```Makefile #### Dynamically Generated Interactive Menu #### # Error Handling SHELL := /bin/bash .SHELLFLAGS := -o pipefail -c # Name of this Makefile MAKEFILE_NAME := $(lastword $(MAKEFILE_LIST)) # Special targets that should not be listed EXCLUDE_LIST := menu all .PHONY # Function to extract targets from the Makefile define extract_targets $(shell awk -F: '/^[a-zA-Z0-9_-]+:/ {print $$1}' $(MAKEFILE_NAME) | grep -v -E '^($(EXCLUDE_LIST))$$') endef TARGETS := $(call extract_targets) .PHONY: $(TARGETS) menu all menu: ## Makefile Interactive Menu @# Check if fzf is installed @if command -v fzf >/dev/null 2>&1; then \ echo "Using fzf for selection..."; \ echo "$(TARGETS)" | tr ' ' '\n' | fzf > .selected_target; \ target_choice=$$(cat .selected_target); \ else \ echo "fzf not found, using numbered menu:"; \ echo "$(TARGETS)" | tr ' ' '\n' > .targets; \ awk '{print NR " - " $$0}' .targets; \ read -p "Enter choice: " choice; \ target_choice=$$(awk 'NR == '$$choice' {print}' .targets); \ fi; \ if [ -n "$$target_choice" ]; then \ $(MAKE) $$target_choice; \ else \ echo "Invalid choice"; \ fi # Default target all: menu help: ## This help function @egrep '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' # Targets (example targets listed below) lint: ## Run lint @echo "Running lint..." test: ## Run test @echo "Running test..." build: ## Run build @echo "Running build..." ``` ![](makefile-menu.png) - [MBA Washing](https://smcleod.net/2023/11/mba-washing/index.md): MBA Washing --- title: MBA Washing author: "Sam McLeod" description: '"MBA Washing" refers to the phenomenon where individuals or companies, often with a strong theoretical or academic background but limited recent practical experience, adopt and reinterpret industry-specific terminology and cultural movements.' categories: [ Tech, Culture ] keywords: ["tech", "culture", "devops", "communication"] date: "2023-11-13T09:00:00Z" images: ["mbaw-flow-1.png"] cover: image: " mbaw-flow-1.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - culture - DevOps # series: [""] toc: enable: true auto: false math: false draft: false --- "**MBA Washing**" refers to the phenomenon where individuals, often with a strong theoretical or academic background but limited recent practical experience, adopt and reinterpret industry-specific terminology and cultural movements. -- This reinterpretation is typically influenced by their academic learning, recent literature, and biases towards larger enterprise perspectives. As a result, the original intent and practical effectiveness of these concepts, such as DevOps or Platform Engineering for example, may be diluted or misrepresented. > It should be noted that the term "MBA Washing" is somewhat of a joke and not intended to be derogatory towards those with an MBA or similar business-focused education. Instead, it is meant to highlight the impact of theoretical perspectives on cultural movements and industry practices by itself being a co-optable, theoretical term. ### Expanded Definition Coined to describe the influence of academic or theoretical perspectives, particularly from those with an MBA or similar theory heavy, business or marketing focused education, on practical industry practices. #### Key Characteristics - **Theoretical Over Practical**: Emphasis on book knowledge over recent, hands-on experience in the field. - **Enterprise Bias**: A tendency to view concepts through the lens of large-scale enterprise operations, which may not always be agile or efficient. - **Misinterpretation of Concepts**: Terms and practices like DevOps, Agile or Platform Engineer are often co-opted, redefined, losing their original effectiveness and intent. ### Impact - **Dilution of Practices**: Core principles of agile, efficient methodologies may be overshadowed by bureaucratic or theoretical approaches. - **Misalignment with Industry Trends**: A gap emerges between what is taught or advocated and what is practised in cutting-edge or smaller-scale environments. - **Communication Barriers**: Creates a disconnect between those trained in these 'washed' concepts and practitioners on the ground. ![MBA Washing: Process and Impact](mbaw-flow-1.png) > This flowchart outlines how a concept, when interpreted through an MBA perspective, can lead to a theoretical emphasis, enterprise bias, and misinterpretation of concepts. These factors collectively impact industry practices, leading to the dilution of agile practices, misalignment with industry trends, and communication barriers. These outcomes can further result in reduced efficiency and innovation, a gap between theory and practice, and conflicts in team dynamics, ultimately altering industry practices. --- ### Examples **DevOps in Theory vs. Practice**: An MBA graduate might focus on the strategic alignment and process optimization aspects of DevOps, overlooking the importance of continuous integration and deployment practices essential in real-world scenarios. ![](the-evolution-of-devops-from-philosophy-to-brand.png) **Agile Methodology**: The agile methodology might be reinterpreted to fit rigid, hierarchical corporate structures, contrary to its original intent of fostering flexibility and cross-functional collaboration. **Platform Engineering**: As seen in "Humanitec"s [State of Platform Engineering (Vol 2)](https://humanitec.com/whitepapers/state-of-platform-engineering-report-volume-2) report - with their "_history_" of Platform Engineering (pictured below) - there is a string recency bias and a lack of understanding or value placed in the history of such a movement. This is a common theme in many "industry reports" as often published by large, privately owned companies. ![](state-of-pe-poor-timeline.png) This isn't to say that we should discredit any of the individuals, events or communities such a timeline (Team Typologies is a great resource for example) - but instead to highlight the biases of reports such as this. They only paint a narrow, often enterprise-focused, view of the field. If sticking to a basic high level timeline I'd create something more along the lines of the following: ![](the-evolution-of-platform-engineering-and-the-rise-of-platform-as-a-product.png) --- In summary, we should be mindful of the impact of theoretical doctrine and perspectives on cultural movements and industry practices, along with the potential for dilution or misrepresentation of these concepts. - [SDXL LoRA Training](https://smcleod.net/2023/10/sdxl-lora-training/index.md): SDXL LoRA Training --- title: SDXL LoRA Training author: "Sam McLeod" description: "A talk I gave to some peers on creating your own SDXL LoRA models from my tinkering around over the last few weeks." categories: [ Tech, Talks, AI ] keywords: ["tech", "talk", "slides", "ai", "ml", "art", "sdxl", "lora", "stable diffusion", "diffusion", "model", "training", "machine learning"] date: "2023-10-30T20:00:00Z" images: ["sdxl.jpeg"] cover: image: " sdxl.jpeg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - talk - slides - AI - ML - art - StableDiffusion series: ["AI/ML"] toc: enable: false auto: false math: false draft: false --- A talk I gave to some peers on creating your own SDXL LoRA models from my tinkering around over the last few weeks. Definitely not an expert by any means - note disclaimer slide! Slides: - [PDF](https://github.com/sammcj/smcleod_files/raw/master/slides/SDXL_LoRA/SDXL%20LoRA%20-%20Sam%20McLeod%20-%202023.pdf) Video: {{< youtube jRxyn08impM >}} - [Fixing AMD CPU Scaling on Fedora](https://smcleod.net/2023/07/fixing-amd-cpu-scaling-on-fedora/index.md): Fixing AMD CPU Scaling on Fedora --- title: "Fixing AMD CPU Scaling on Fedora" date: 2023-07-09T06:41:37 lastmod: 2023-07-09T07:41:37 author: Sam McLeod description: "Enabling the new AMD P-State Driver on Fedora with Kernel 6.4" keywords: ["tech", "linux", "fedora", "amd", "pstate", "kernel", "power", "performance"] tags: ["linux", "performance"] categories: ["Tech", "Linux"] series: [] images: [""] cover: image: "" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles toc: enable: true auto: false math: false draft: false --- Recently, after replacing my home server I noticed that the CPU (Ryzen 7600) was only scaling between 3000MHz and 3800MHz, which is the base and the first level boost clock of the CPU. I was expecting it to scale down to as low as 400Mhz when idle, and up to 5.17Ghz on boost. ## Investigation ```plain cpupower frequency-info analyzing CPU 0: driver: acpi-cpufreq CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: Cannot determine or is not supported. hardware limits: 3.00 GHz - 5.17 GHz available frequency steps: 3.80 GHz, 3.00 GHz available cpufreq governors: conservative ondemand userspace powersave performance schedutil current policy: frequency should be within 3.00 GHz and 3.80 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency: 3.00 GHz (asserted by call to hardware) boost state support: Supported: yes Active: yes Boost States: 0 Total States: 2 Pstate-P0: 3800MHz Pstate-P1: 3000MHz ``` ```bash cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver acpi-cpufreq cat /sys/devices/system/cpu/cpu0/cpuinfo_min_freq 3000000 cat /sys/devices/system/cpu/cpu0/cpuinfo_max_freq 3800000 ``` ```bash zenstates.py -l P0 - Enabled - FID = 98 - DID = 8 - VID = C9 - Ratio = 38.00 - vCore = 0.29375 P1 - Enabled - FID = 78 - DID = 8 - VID = 79 - Ratio = 30.00 - vCore = 0.79375 P2 - Disabled P3 - Disabled P4 - Disabled P5 - Disabled P6 - Disabled P7 - Disabled C6 State - Package - Enabled C6 State - Core - Enabled ``` After a significant amount of reading and hunting through BIOS settings only to find that ASRock have removed the CPPC enablement setting. I found that while the new AMD P-State driver was merged into the kernel - it didn't seem to load by default with kernel 6.3.11 which is the latest version in the Fedora 38 repositories. Fedora is currently testing 6.4 and has test packages available, so I decided to give it a go - and it works perfectly. ## Installing the new kernel The kernel itself is stable, but it's classed as a test package, so you'll need to enable the testing repository. ```bash dnf install koji -y koji list-builds --package=kernel --after="2023-07-07" --pattern "kernel-6.4*" mkdir -p kernel_test && cd kernel_test koji download-build --arch=x86_64 kernel-6.4.2-201.fc38 dnf update kernel-*.rpm ``` ## Enable the driver There's one more step before rebooting, we need to enable the new AMD P-State driver. 1. Edit `/etc/default/grub`, adding `amd_pstate=enable` to the `GRUB_CMDLINE_LINUX` line. 2. Update the bootloader `grub2-mkconfig -o /boot/grub2/grub.cfg; dracut -f` 3. Reboot ## Check it's working ```bash cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver amd-pstate-epp cat /sys/devices/system/cpu/cpu0/cpuinfo_min_freq 400000 cat /sys/devices/system/cpu/cpu0/cpuinfo_max_freq 5170000 ``` That's the new pstate driver running and 400Mhz to 5.17Ghz! And if we look at htop, we can see that individual CPU cores are scaling down to 400Mhz when idle: ```plain 0[ 0.0% 400MHz N/A ] 6[ 0.0% 400MHz N/A] Tasks: 193, 773 thr, 250 kthr; 1 running 1[ 0.0% 3672MHz 40°C] 7[ 0.0% 400MHz N/A] Systemd: running (0/739 failed) (0/411 jobs) 2[ 0.0% 400MHz N/A ] 8[| 0.7% 3592MHz N/A] Swp[ 0K/8.00G] 3[ 0.0% 400MHz N/A ] 9[| 0.7% 400MHz N/A] Load average: 0.49 0.30 0.20 4[ 0.0% 400MHz N/A ] 10[| 0.7% 3527MHz N/A] Disk IO: 0.0% read: 0KiB/s write: 0KiB/s 5[ 0.0% 400MHz N/A ] 11[|| 1.3% 400MHz N/A] Network: rx: 2KiB/s tx: 1KiB/s (12/5 pkts/s) Mem[||||||||||||||| 2.73G/65G] Uptime: 00:16:32 ``` There's still something weird up with the hardware pstates which I suspect is something to do with the CPPC mode (action/passive etc...), but it doesn't seem to matter as the frequency scaling is working as expected. ```plain cpupower frequency-info analyzing CPU 8: driver: amd-pstate-epp CPUs which run at the same hardware frequency: 8 CPUs which need to have their frequency coordinated by software: 8 maximum transition latency: Cannot determine or is not supported. hardware limits: 400 MHz - 5.17 GHz available cpufreq governors: performance powersave current policy: frequency should be within 400 MHz and 5.17 GHz. The governor "powersave" may decide which speed to use within this range. current CPU frequency: Unable to call hardware current CPU frequency: 3.34 GHz (asserted by call to kernel) boost state support: Supported: yes Active: yes Boost States: 0 Total States: 2 Pstate-P0: 3800MHz Pstate-P1: 3000MHz ``` I'm not sure if that's a bug in the driver or the tool, but it doesn't seem to matter 🤷. ## Conclusion Great success! ![](https://upload.wikimedia.org/wikipedia/en/f/ff/SuccessKid.jpg) - [Defaulting git clone to shallow (depth=1)](https://smcleod.net/2023/05/defaulting-git-clone-to-shallow-depth1/index.md): Defaulting git clone to shallow (depth=1) --- title: "Defaulting git clone to shallow (depth=1)" date: 2023-05-29T08:53:24+10:00 lastmod: 2023-05-29T08:53:24+10:00 author: Sam McLeod description: "Shallow clones are faster and smaller" keywords: ["Git", "Scripting", "ZSH", "Bash", "Shell"] tags: ["git", "scripting", "zsh", "bash", "Shell"] categories: ["Scripting", "Zsh", "Git"] series: ["ZSH"] toc: enable: false auto: false math: false draft: false type: posts --- Before adding this to my shell config, I would manually add --depth=1 to all my git clones. I never saw the value in downloading the entire source history and (at times incorrectly) committed files to my local machine unless I was cloning the repo for the purpose of backing it up. It seems strange to me that git doesn't have a config option to set this as default behaviour. My hack for this is [a simple zsh/bash function](https://github.com/sammcj/zsh-bootstrap/blob/a55cae3421fe8e2144b9c1f30bf07180966a7a58/9-functions.rc#L20) that can wrap git commands - and as such add clone depth if it's not specified. ```bash function git_wrapper() { # If invoked by another function, alias or xargs, interpret it as normal if [[ -n ${FUNCNAME[*]} ]] || [[ -n $ALIASES ]] || [[ -n $XARGS ]]; then command git "$@" return fi # clone with depth=1 if no depth is not specified if [[ $1 == "clone" ]] && [[ $* != *"--depth"* ]]; then shift command git clone --depth=1 "$@" else command git "$@" fi } alias git='git_wrapper' ``` Maybe there's a better way of doing this? - [How likely would you be to block a company from asking you to rate everything they do?](https://smcleod.net/2023/05/how-likely-would-you-be-to-block-a-company-from-asking-you-to-rate-everything-they-do/index.md): How likely would you be to block a company from asking you to rate everything they do? --- title: "How likely would you be to block a company from asking you to rate everything they do?" date: 2023-05-26T06:41:37 lastmod: 2023-05-26T19:41:37 author: Sam McLeod description: "NPS Surveys are corporate spam" keywords: ["NPS", "Surveys", "Marketing", "Spam", "Sales", "Customer Experience", "Email", "NPS"] tags: ["Sales", "Marketing", "Customer Experience"] categories: ["Sales/Marketing", "Enterprise"] series: [] images: ["i-dont-use-nps-1.jpg"] # featuredimage: "i-dont-use-nps-1.jpg" cover: image: "i-dont-use-nps-1.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles toc: enable: true auto: false math: false draft: false --- ## I'm tired of getting emails from companies asking me to rate their products and services It feels like you can't even buy a coffee these days without being asked if you would recommend the coffee shop to a friend or colleague. - "How likely would you be to recommend us to a friend or colleague?" - "Rate our service from 1-10" - "How did we do?" - "How was your experience?" > It's a simple question, and it's a simple score. > > It's also a simple way to annoy your customers and end up in their spam folder. --- ## What is NPS? NPS stands for **N**et **P**romoter **S**core. The idea is that you ask your customers to rate your product, service or experience from 0-10, and then you subtract the percentage of detractors (0-6) from the percentage of promoters (9-10) to get your NPS score. For example, if 50% of your customers are promoters and 10% are detractors, your NPS is 40. Companies like to use _(abuse?)_ this to measure customer satisfaction and make their customers feel like they listened to and that their opinion matters. There are many problems with this highly reductive and impersonal MBE style approach but the one that annoys me the most is that it's essentially created a new form of corporate spam. --- I've had a crack at writing some advanced email rules to filter these out to a folder - it's far from perfect but it's a start. The following [Sieve](http://sieve.info/) script (rather unintelligently) filters out NPS surveys from a combination of keywords in the body and headers and moves them from your inbox to a folder of your choice. This script should work with any mail service / server that supports standard Sieve scripts, I use this with [Fastmail](https://fastmail.com) _(or if you feel like it you can optionally use [my referral link](https://ref.fm/u13738357)_). Fastmail offers a [Sieve testing tool](https://app.fastmail.com/sievetester/) you can use to test your script before you apply it to your account. [Click here to find the latest version](https://github.com/sammcj/scripts/blob/master/email/customer-satisfaction-spam.sieve) of this script and _please_ do raise PRs to improve it! There's plenty of room for improvement here, e.g.: - Additional matching keywords / phrases. - Check if sender is in your address book (Not currently possible with Fastmail's Sieve extensions). - Use an upstream list of keywords, phrases and domains rather than hardcoding them (Not sure if this is possible with Sieve?). - Automate updating the ruleset. ## Links - [NPS Considered Harmful](https://jmspool.medium.com/net-promoter-score-considered-harmful-and-what-ux-professionals-can-do-about-it-fe7a132f4430) - [Reasons to use NPS](https://www.npsistheworst.com/reasons-to-use-nps) (also credit to this site for the image used in this post) - [NPS is hurting you](https://uxdesign.cc/this-popular-business-metric-is-hurting-you-55ed535e9a59) - [Sieve testing tool](https://app.fastmail.com/sievetester/) (Fastmail) - [Sieve](http://sieve.info/) - [Escaping the Time Trap: Why Estimating Effort, Not Time, Leads to Greater Success](https://smcleod.net/2023/04/escaping-the-time-trap-why-estimating-effort-not-time-leads-to-greater-success/index.md): Escaping the Time Trap: Why Estimating Effort, Not Time, Leads to Greater Success --- title: "Escaping the Time Trap: Why Estimating Effort, Not Time, Leads to Greater Success" date: 2023-04-28T06:41:37 lastmod: 2023-05-17T19:41:37 author: Sam McLeod description: "Effort is non-linear and multi-dimensional. Time (with relation to delivery) is linear and one-dimensional" keywords: ["Estimating", "Delivery", "Kanban", "Agile","DevOps"] tags: ["Estimating", "Delivery", "Kanban", "agile","DevOps"] categories: ["Delivery", "Leadership", "DevOps"] series: [] images: ["time-based-estimates.png"] # featuredimage: "time-based-estimates.png" cover: image: "time-based-estimates.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles toc: enable: true auto: false math: false draft: false --- Many organisations - while branding themselves as "Agile" - continue to make the mistake of estimating project velocity based on _time_ than _effort_. This flawed approach equates to measuring the team's ability to estimate time, rather than focusing on the actual effort needed to complete tasks. This isn't to say that time isn't important - quite the opposite. Time is often a critical factor in delivery. However, it's important to understand that time is a _constraint_ and a _metric_ - not a _unit of work_. --- Managers want to know how much time a project or deliverable will take - but often aren't interested in the complexity of the work. Engineers also want to know how long a piece of work may take - however solving complex problems requires _effort_ - not simply _time_ and the two should not be conflated through the estimation process. > Effort is non-linear and multi-dimensional. Time (with relation to delivery) is linear and one-dimensional. As humans we simplify the world around us. This helps us to make (hopefully sensible) decisions in a reasonable time and to communicate with others. However, as complexity grows our reductionist approach often breaks down. We encounter flawed assumptions, external influences and unknown unknowns and as our understanding of each element grows our simple models start to again resemble the complex systems of which we're trying to simplify. The age old mistake reductionist assumption that the time it takes to complete something is simply amount of work / capacity is alive and well in software delivery. > How many engineers does it take to deliver a feature? One engineer can deliver a feature in 10 days, so 10 engineers can deliver it in 1 day, right?... In reality the time a piece of work takes involves complexity of the task, how well practised we are the the kinds of work involved, the skills and experience of the people involved, distractions, outside influences and dependencies. ## The Problem with Estimating in Time ```mermaid graph TD A(Time-Based Estimates) --> B1(Unrealistic Expectations) A --> B2(Skill Inconsistencies) A --> B3(Failure in Reflecting Complexity) B1 --> C1(Prioritisation Mismanagement) B2 --> C2(Pressure & Stress) B3 --> C3(Ignoring Task Complexity) C1 --> D1(Schedule Overruns) C2 --> D2(Unpredictable Outcomes) C3 --> D3(Incorrect Or Missing Features) ``` ### Time Cannot Represent Complexity Possibly the most important reason to avoid estimating in time is that time cannot represent complexity. When estimating in hours/days, we're assuming a direct relationship between effort and time. This can blur the line between the effort required to execute a task and the time it takes to complete it. Complex tasks may require more time than expected, introducing unanticipated delays. Imagine two tasks. The first: fixing a simple bug. The second: solving a rare edge-case concurrency problem. Clearly, the effort needed for each task differs. Estimating in time means the team will spend hours trying to figure out how long it could take, rather than focusing on the actual effort involved. ```mermaid graph TD A(Time-Based Estimation) --> B(Simplistic View) A --> C(Focused on Duration) B --> D1(Difficulty to Capture Complexity) C --> D2(Inaccurate Expectations) D1 --> E1(Cascading Delays) D2 --> E2(Misallocation of People) ``` ### Developer Inconsistencies Different developers work at different speeds, which makes estimating in time unfair and unreliable. The experience or familiarity a developer has with a specific technology or domain can greatly influence their ability to complete a task. Developer A might complete a task in two hours, while Developer B might need six hours for the same task. Estimating in time can lead to incorrect assumptions about progress. ### Time Estimates Create Artificial Deadlines and Unnecessary Pressure When estimating in hours or days, artificial deadlines are often introduced, which may not account for unforeseen challenges, external dependencies, or varying developer skills. This can lead to unnecessary pressure on team members to meet these deadlines, adversely affecting team morale and the quality of work. With effort estimation, teams can prioritise tasks more effectively, anticipating the necessary adjustments as they encounter new information, without compromising team dynamics or work quality. --- ## Estimating Effort Points - The Better Approach ```mermaid graph TD D(Effort-Based Estimates) --> E1(Emphasis on Complexity) D --> E2(Recognition of Developer Variability) D --> E3(Realistic Expectations) E1 --> F1(Better Prioritisation) E2 --> F2(Focus on Skill Growth) E3 --> F3(Smooth Communication) ``` ### Accounting for Complexity Effort points help teams understand the complexity of tasks and the relative effort needed to complete them. Comparing different tasks based on their complexities allows for logical resource allocation, ensuring smooth project execution. ### Flexibility for Diversity in Skills Effort points aren't based on how long a task takes, but on how difficult it may be. This allows for better adaptability to the varying skill levels of developers. Teams can assign tasks without time pressure, enabling developers to grow and to deliver quality products. ### Smooth Communication Effort points support transparency in communication. Rather than assuming how long a task will take, project managers can better account for the effort involved, promoting realistic expectations and allowing for better planning. ### Continuous Improvement Through Data Effort points offer data-driven insights on team performance, aiding in continuous improvement. The team can identify bottlenecks and find ways to streamline processes, leading to increased efficiency and fewer obstacles. Here are some resources to help guide your team in switching to effort point estimation: - [Effort Estimation in Agile: A Beginner's Guide](https://www.smartsheet.com/content-center/best-practices/agile/project-tips/effort-estimation-agile-beginners-guide) - [Agile Estimating and Planning: Planning Poker](https://www.planningpoker.com/) --- ## Effective Prioritisation With effort estimation, teams gain a better understanding of the inherent complexity and relative effort required for different tasks. This understanding enables them to prioritise tasks more effectively in several ways: ### Focus on Value Delivery By identifying the effort involved in each task, teams can strategically prioritise and deliver high-value features first. This ensures that the most crucial aspects of the project are addressed early, resulting in faster value delivery to stakeholders and end-users. ```mermaid graph LR; A(Focus on Value Delivery)-->B1(Effort Estimation); B1-->C1(High-Value Features); C1-->D1(Faster Value Delivery); ``` ### Balanced Workload Distribution Effort estimation allows teams to evaluate the complexity of tasks on a case-by-case basis. With this knowledge, they can distribute the workload among team members more fairly, considering individual expertise levels and balancing the work according to each person's capacity. ```mermaid graph LR; A(Balanced Workload Distribution)-->B2(Effort Estimation); B2-->C2(Evaluate Complexity); C2-->D2(Distribute Workload Fairly); ``` ### Dependency Management Teams can analyse complex tasks in the context of their dependencies and grouped efforts. By prioritising tasks based on their dependencies, teams can streamline workflow, minimise bottlenecks, and ensure that tasks that are a prerequisite to other tasks are addressed more efficiently. ```mermaid graph LR; A(Dependency Management)-->B3(Effort Estimation); B3-->C3(Identify Dependencies); C3-->D3(Streamline Workflow); ``` ### Risk Mitigation Effort-based estimation makes it easier to identify high-risk tasks that require significant effort or have a high degree of uncertainty. By prioritising these tasks, teams can better allocate people to mitigate potential risks and address issues proactively. ```mermaid graph LR; A(Risk Mitigation)-->B4(Effort Estimation); B4-->C4(Identify High-Risk Tasks); C4-->D4(Allocate People Strategically); ``` ### Iterative Planning Agile methodologies emphasise working in iterations or sprints. Effort estimation aids in this process as teams can select and prioritise tasks to complete during each iteration, allowing them to adapt to changing project requirements and maintain progress. By prioritising tasks more effectively using effort estimation, teams can enhance productivity, ensure balanced workloads, manage dependencies, minimise risk, and promote iterative planning, all of which contribute to more successful project outcomes. ```mermaid graph LR; A(Iterative Planning)-->B5(Effort Estimation); B5-->C5(Select Tasks); C5-->D5(Adapt to Changing Requirements); ``` --- ## Strategies for Product Owners/Project Managers to Better Understand Timelines ```mermaid graph LR A(Timelines) --> B(Implement Better Estimation Techniques) A --> C(Maintain a Product Roadmap) A --> D(Collaborate with Team) A --> E(Monitor & Adapt) ``` ### Implement Better Estimation Techniques Using effort estimation methods like Planning Poker or T-shirt sizing will provide a more accurate picture of project progress. Whenever possible, combine these techniques to continuously refine your understanding. ### Maintain a Product Roadmap Prioritise features and plan your project incrementally. A high-level product roadmap helps monitor progress and highlight dependencies or potential roadblocks ### Collaborate with Your Team Encourage open communication and involve team members in planning and estimation. Active collaboration ensures shared understanding, lowers the risk of unexpected delays, and promotes proactive problem-solving ### Link Effort-Based Estimation to DORA Metrics Effort-based estimation plays an essential role in enhancing the overall performance of software delivery teams. When aligned with the DORA (DevOps Research and Assessment) metrics - as recommended in the [Accelerate book](https://itrevolution.com/book/accelerate/) - the benefits of effort estimation can be even more profound. DORA metrics focus on organisational performance in terms of lead time, deployment frequency, mean time to recovery, and change failure rate. By linking effort-based estimation to these metrics, teams can evaluate the efficiency and effectiveness of their processes, and continuously improve their software development and delivery practices. ```mermaid graph TB K[Effort-Based Estimation] --> L1[Higher Deployment Frequency] K --> L2[Lower Lead Time] K --> L3[Lower Change Failure Rate] K --> L4[Faster Mean Time to Recovery - MTTR] L1 --> M1[Resource Management] L1 --> M2[Productivity] L2 --> M3[Streamlined Processes] L2 --> M4[Efficiency] L3 --> M5[Smarter Testing] L3 --> M6[Software Quality] L4 --> M7[Rapid Response] L4 --> M8[Problem-Solving Skills] M1 --> N1[Reduced Overload] M2 --> N2[Job Satisfaction] M3 --> N3[Continuous Delivery] M4 --> N4[Agile & DevOps Practices] M5 --> N5[Avoid Errors] M6 --> N6[Customer Satisfaction] M7 --> N7[Customer Trust] M8 --> N8[Tackling Similar Issues] ``` ### Monitor and Adapt Stay open to revisiting your initial estimates, adjusting your plans to accommodate any changes or setbacks. Agile methodologies emphasise continuous improvement, so adapt your timeline as needed By embracing Agile/Kanban methodologies and focusing on effort points instead of time, your team can increase productivity and achieve better results. Adopt these modern practices to provide realistic timelines, improve communication, and foster a more flexible and efficient working environment. ## Summary In summary, adopting effort-based estimation in Agile and Kanban methodologies and avoiding reverting to dated waterfall style time-based estimation offers numerous advantages, such as better task prioritisation, enhanced communication, and a deeper understanding of task complexity. By steering away from time-based estimation and embracing an effort-first mindset, you're optimising your delivery process and resource allocation while operating in a more human-centric way. You can take this a step further by connecting this approach to DORA metrics further highlights the impact of effort estimation on an organisation's performance. - [Introduction to AI and Large Language Models (LLMs)](https://smcleod.net/2023/01/introduction-to-ai-and-large-language-models-llms/index.md): Introduction to AI and Large Language Models (LLMs) --- title: "Introduction to AI and Large Language Models (LLMs)" author: "Sam McLeod" description: "A high level intro to LLMs that I'm writing for a few friends that are new to the concept. It is far from complete, definitely contains some errors and is a work in progress." categories: [ Tech, AI, LLM, WIP ] keywords: ["tech", "ai", "llm", "wip"] date: "2023-01-26T01:00:00Z" images: [""] cover: image: "" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - LLM - AI - WIP - DevOps series: ["AI"] toc: enable: true auto: false math: false draft: false --- This is a high level intro to LLMs that I'm writing for a few friends that are new to the concept. It is far from complete, definitely contains some errors and is a work in progress. _**_This is a work in progress and a living document._**_ Language models, or LLMs, are a type of artificial intelligence that can generate text based on a given prompt. They work by learning patterns in large amounts of text data and using those patterns to generate new text. LLMs can be used for a variety of tasks, such as generating chatbots, answering questions, and creating art. ## How Do LLMs Work? LLMs work by analysing the patterns in the data they were trained on. When you ask an LLM a question or give it a prompt, it predicts the most likely response based on its training. (This is a massive over-simplification). ## Definition and Mechanism LLMs are AI systems trained on extensive text datasets. They use a structure called a Transformer, which processes input text and predicts output based on learned patterns. For instance, if you input "The weather in Sydney is", the LLM, based on its training, might predict and complete it as "sunny and warm". ## Key Terminology - **Quantisation**: Reducing the model size by decreasing the precision of its parameters, speeding up operation, different levels of quantisation result in different amounts of accuracy loss. Quantisation can speed up its operation without significant loss of accuracy. For example, converting a model's parameters from 32-bit floating-point to 16-bit, but most commonly people run Q4 (4 bit), Q5, Q6 or Q8 quantised models. - **HuggingFace**: A platform offering and community collection of pre-trained models, adapters, LoRAs and training datasets. - **Inference**: This is when the model generates a response to your input. For example, when you input a prompt, the model's inference process is what creates the output text. - **Fine-Tuning**: Adjusting a pre-trained model to make it more suitable for specific tasks. For instance, you might fine-tune a model on a dataset of technical documents if you want it to generate technical content. - **LoRA** (Low-Rank Adaptation): A technique to fine-tune models efficiently by only modifying a small part of the model's weights. This is useful for adapting a model to new tasks without extensive retraining. - **Zero-Shot**: Providing a single prompt to the LLM to get a single response. --- ## Model Formats ### GPTQ GPTQ stands for "generate predictive quantisation," which is a method for quantising the weights in an LLM that can be used to generate text. GPTQ models can only be interferenced on GPUs. ### GGUF / GGML GGML stands for "general-purpose generative model" and is a type of LLM that can be fine-tuned for specific tasks, such as language translation or chatbot generation. GGUF is the newer file extension for GGML models. GGUF models can be interferenced on both CPU and GPU. ### AWQ A newer model format (more information needed). ## Model Servers Language model servers, or LLM servers, are software programs that allow you to run LLMs on your own server. There are several different types of LLM servers available, each with its own features and capabilities. In this guide, we will discuss the differences between some of the most commonly used LLM servers. When using text generation web ui, ollama, lm_studio they will automatically select the correct model server. Text Generation WebUI lets you tune a lot of the parameters to get the most out of the LLMs and your hardware. ### autogptq Note: Generally not used anymore, exllamav2 is preferred. autogptq is an open-source LLM server that uses GPTQ as its quantisation method. It supports a wide range of LLMs, including Megatron-LM and Pegasus, and can be fine-tuned for specific tasks using Hugging Face's pre-trained models. autogptq is designed to be highly flexible and customisable, with support for GPU and CPU inference. ### llama.cpp llama.cpp is a C++ implementation of an LLM that can be fine-tuned on your own data using the Hugging Face transformers library. It supports both GPGPU (General Purpose GPU Processing Unit) and CPU inference, and can be used to generate text for a wide range of tasks, including language translation and chatbot generation. llama.cpp is highly modular and can be easily integrated with other software programs. ### ctransformers ctransformers is an open-source LLM server that uses the transformers library to run LLMs on your own server. It supports both GPU and CPU inference, and can be fine-tuned for specific tasks using Hugging Face's pre-trained models. ctransformers is highly flexible and customisable, with support for a wide range of LLMs and inference methods. ### autoAWQ autoAWQ is an open-source LLM server that uses the Adaptive Weight Quantisation (AWQ) method to reduce the size of the model. It supports both GPU and CPU inference, and can be fine-tuned for specific tasks using Hugging Face's pre-trained models. autoAWQ is designed to be highly efficient and can reduce the size of the model by up to 90%. ### exllama exllama is a Python-based LLM server that uses the AutoAWQ method to quantise the model. It supports both GPU and CPU inference, and can be fine-tuned for specific tasks using Hugging Face's pre-trained models. exllama is highly flexible and customisable, with support for a wide range of LLMs and inference methods. ### exllama_hf exllama_hf is an open-source LLM server that uses the Hugging Face transformers library to run LLMs on your own server. It supports both GPU and CPU inference, and can be fine-tuned for specific tasks using Hugging Face's pre-trained models. exllama_hf is highly flexible and customisable, with support for a wide range of LLMs and inference methods. --- ## Prompting ### Tips for Good Prompts - Be Specific: Clear and detailed prompts lead to better responses. - Context Matters: Provide relevant background information, this is very important and often overlooked. - Avoid Ambiguity: Make sure your prompt can't be misinterpreted. - Chain of Thought Prompting: Break down a complex task into smaller steps in your prompt. For example, for a coding task, start with the problem definition, then outline the steps to solve it, and finally ask the model to generate code based on these steps. - Zero-Shot vs Few-Shot Learning: These techniques involve giving the model either no examples (zero-shot) or a few examples (few-shot) to guide its response. For instance, you can provide a couple of examples of home automation commands and then ask the model to generate more. ### What to Avoid - Overly Complex Requests: Keep it simple, especially at the beginning. - Vague Language: Be as clear and direct as possible. --- ## Models - Llama 2: An open-source base model that can be run locally for various tasks. - Stable Diffusion: A set of base models for art generation. ## Common Open Source Tools - [Text Generation Web UI](https://github.com/oobabooga/text-generation-webui): Interfaces for interacting with LLMs through a web browser. Has lots of plugins, extensions and APIs. - [InvokeAI](https://invoke-ai.github.io/InvokeAI/): A web based open source tool for using LLM models for art generation. - [Ollama](https://ollama.ai/): A cross-platform model server. Think of it like the docker client but for LLMs. Note: [LM Studio](https://github.com/lmstudio-ai) is also a great app for macOS but it's not open source. ## CPU vs GPU Inference CPU inference is the process of using an LLM to generate text by passing the prompt through the LLM's neural network on a central processing unit. This method is slower than GPU inference but requires less processing power and can be used on devices with limited memory, such as mobile devices. CPU inference can also be useful for running LLMs on devices that do not have a GPU or where GPU usage is not feasible due to power constraints. GPU inference, on the other hand, is the process of using an LLM to generate text by passing the prompt through the LLM's neural network on a graphics processing unit. This method is faster than CPU inference and can be used to process large amounts of data at once. GPU inference requires more processing power than CPU inference and may not be suitable for devices with limited memory or where power usage is a concern. ### Model Offloading Model offloading is the process of transferring an LLM from its original device, such as a server, to another device, such as a mobile phone, for faster processing. This can be done using various methods, including on-device inference and cloud-based inference. On-device inference involves running the LLM directly on the target device, while cloud-based inference involves running the LLM on a remote server and sending the results back to the client device. Model offloading can be useful for devices that do not have enough processing power to run the LLM locally, or for applications where latency is a concern and faster processing is required. However, it may also introduce security risks, as sensitive data may be transferred over the network. ## Deep Dive into Key Terminology - Tokenization: The process of converting text into tokens (small pieces) that the model can understand. For example, the sentence "Hello, World!" might be split into tokens like ["Hello", ",", "World", "!"]. - Embeddings: These are representations of tokens in a high-dimensional space. They capture the meaning and context of words, allowing the model to understand relationships between different tokens. - Attention Mechanism: A part of the Transformer architecture that helps the model focus on relevant parts of the input when generating a response. It's like highlighting important words in a sentence to better understand its meaning. --- ## Interference Parameters Common interference parameters are values that can be adjusted when using LLMs to control how much the model interferes with other processes on the same system. These include the batch size, temperature, top_p/k and sequence length. The learning rate controls how quickly the LLM updates its weights during training, the batch size determines how many examples the LLM processes at once during inference, and the sequence length determines the maximum length of text that the LLM can generate at once. ### max_new_tokens max_new_tokens is a parameter that controls the maximum number of tokens that can be generated by the LLM during a single inference session. This parameter can be used to limit the length of the generated text or to control the amount of resources required for inference. ### temperature temperature is a parameter that controls the randomness of the LLM's output. A higher temperature value will result in more random and unpredictable output, while a lower temperature value will result in more deterministic and predictable output. This parameter can be used to fine-tune the LLM for specific tasks or to control the level of creativity or diversity in the generated text. ### top_p To quote [Carlos F. Enguix](https://carlosfenguix.website): "Let's assume you set the Top P value as P (0 ≤ P ≤ 1). Now we have a set of words from the previous step with various probabilities. How Top P works is if it finds the smallest group of words whose cumulative probability exceeds the value of P. This way, the number of words in the set can dynamically increase and decrease according to the next word probability distribution. If the value of P is 0, then "Top P" will select the word with the highest probability. This is equivalent to greedy decoding. If the value of P is 1, then "Top P" will select the entire set of words. This is equivalent to sampling from the entire distribution. Range: 0.00 - 1.00 Example: top_p 0.01 Deterministic Value: 0" ### top_k To quote [Carlos F. Enguix](https://carlosfenguix.website): "Fine-tune the token selection process with top_k, specifying the number of highest probability vocabulary choices considered during decoding. Balance this value for optimal results. Sample from the k most likely next tokens at each step. Lower k focuses on higher probability tokens. Range: 0 - 200 words Example: top_k 20 Deterministic Value: 0" ### min_p min_p is a parameter that controls the minimum probability required for a token to be included in the generated text during inference. A higher value of min_p will result in more conservative output, while a lower value of min_p will result in more adventurous output. This parameter can be used to fine-tune the LLM for specific tasks or to control the level of risk or exploration in the generated text. ### repetition_penalty repetition_penalty is a parameter that controls the penalty applied to repeating tokens during the inference process. A higher value of repetition_penalty will result in less repetitive output, while a lower value of repetition_penalty will result in more repetitive output. This parameter can be used to fine-tune the LLM for specific tasks or to control the level of creativity or diversity in the generated text. ### presence_penalty presence_penalty is a parameter that controls the penalty applied to tokens that are already present in the generated text during the inference process. A higher value of presence_penalty will result in less repetitive output, while a lower value of presence_penalty will result in more repetitive output. This parameter can be used to fine-tune the LLM for specific tasks or to control the level of creativity or diversity in the generated text. ### frequency_penalty frequency_penalty is a parameter that controls the penalty applied to tokens that appear frequently in the training data during the inference process. A higher value of frequency_penalty will result in less frequent output, while a lower value of frequency_penalty will result in more frequent output. This parameter can be used to fine-tune the LLM for specific tasks or to control the level of creativity or diversity in the generated text. For example: - -2.0 When the morning news starts playing, I noticed that my TV now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now now (The most frequent word is "now" with a percentage of 44.79%) - -1.0 He always watches the news in the morning, watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching watching (The most frequent word is "watching" with a percentage of 57.69%) - 0.0 When the morning sun shines into the small restaurant, a tired mail man appears at the door, holding a bag of mail in his hand. The owner warmly prepares breakfast for him, and he starts sorting the mail while enjoying his breakfast. (The most frequent word is "the" with a percentage of 8.45%) - 1.0 A deep sleep girl is awakened by a warm sunbeam. She sees the first ray of sunlight in the morning, surrounded by the sounds of birds and the fragrance of flowers, everything is full of vitality. (The most frequent word is "the" with a percentage of 5.45%) - 2.0 Every morning, he sits on the balcony to have breakfast. In the gentle sunset, everything looks very peaceful. However, one day, as he was about to pick up his breakfast, an optimistic little bird flew by, bringing him a good mood for the day. (The most frequent word is "the" with a percentage of 4.94%) ### repetition_penalty_range repetition_penalty_range is a parameter that controls the range of possible values for the repetition penalty during the inference process. A higher value of repetition_penalty_range will result in more diverse output, while a lower value of repetition_penalty_range will result in more consistent output. This parameter can be used to fine-tune the LLM for specific tasks or to control the level of creativity or diversity in the generated text. ### epsilon_cutoff epsilon_cutoff is a parameter that controls the threshold for early termination during inference. A higher value of epsilon_cutoff will result in more complete inferences, while a lower value of epsilon_cutoff will result in faster but potentially incomplete inferences. This parameter can be used to balance accuracy and speed during inference. ### eta_cutoff eta_cutoff is a parameter that controls the threshold for early termination based on the number of tokens generated during inference. A higher value of eta_cutoff will result in more complete inferences, while a lower value of eta_cutoff will result in faster but potentially incomplete inferences. This parameter can be used to balance accuracy and speed during inference. ## Other Concepts and Terminology - [MoE: Mixture of Experts](https://huggingface.co/blog/moe), a technique for combining multiple models into a single model. This can be used to improve the performance of an LLM by combining multiple models trained on different datasets. - [RAG: Retrieval-Augmented Generation](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/), a technique used for retrieving and parsing information from an external source. ## Links Other potentially useful links: - https://learnprompting.org/docs/intro - https://huggingface.co/docs - [Open Budgeting - Subscriptions & Licences](https://smcleod.net/2023/01/open-budgeting-subscriptions-licences/index.md): Open Budgeting - Subscriptions & Licences --- title: "Open Budgeting - Subscriptions & Licences" date: 2023-01-05T12:41:37 lastmod: 2023-01-05T12:41:37 author: Sam McLeod description: "Publicly documenting my subscription and license expenses" keywords: ["budgeting", "finance", "money", "subscriptions", "licenses", "open budgeting", "software", "saas", "streaming"] tags: ["budgeting", "finance", "money", "subscriptions", "licenses", "open budgeting", "software", "saas", "streaming"] categories: ["saas", "finance"] series: ["Open Budgeting"] images: ["money-au.jpg"] cover: image: "money-au.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles toc: enable: true auto: true math: false draft: false --- # Budgeting for Subscriptions & Licenses Transparently tracking my financial expenses spent on subscriptions and licenses. I'm doing this in the interest of exploring subscription fatigue and to help others who might interested in doing similar. ## Recurring Expenses | Expense | Cost (Yearly) | Category | Required/QoL | Want To Cancel/Replace | Usefulness | Satisfaction | Frequency | | :--------------- | :------------ | :--------------------- | :----------: | :--------------------: | :--------: | :----------------------------: | :-------: | | 1Blocker | $12 | Security/Privacy | Required | No | Critical | High | Yearly | | 1Password | $55 | Security Software | Required | Yes | Critical | Medium ^1 (Was high before v8) | Monthly | | Amazon Prime | $85 | Utilities | QoL | Yes | High | Medium | Yearly | | Apple iCloud | $54 | Backup Software | Required | No | Critical | High | Yearly | | Apple Music | $130 | Entertainment | Required | No | High | High | Yearly | | Aus.Social | $24 | Entertainment/Donation | QoL | No | Medium | High | Monthly | | Backblaze | $96 | Backup Software | Required | No | Critical | Medium | 2 Yearly | | Bear | $22 | PKM | Required | No/Maybe | High | Medium | Yearly | | BetterTouchTool | $8 | Utility | QoL | No | High | High | 2 Yearly | | Fastmail | $81 | Utility | QoL | No | Critical | High | Yearly | | Github CoPilot | $120 | Coding/Writing Tool | QoL | No | High | High | Yearly | | Infuse | $14.50 | Entertainment | QoL | No | Medium | Medium | Yearly | | Last.FM | $45 | Entertainment | QoL | Maybe | Medium | Medium | Yearly | | Mobile/Data | $1056 | Utilities | Required | No | Critical | High | Monthly | | Namecheap | $23.33 | Utility | QoL | No | Critical | High | 3 Yearly | | OpenAI API | $20 | Coding/Writing Tool | QoL | No | High | High | As Needed | | Overcast | $13 | Entertainment | QoL | Maybe | Medium | Medium | Yearly | | PlayStation Plus | $135 | Entertainment | QoL | No | Medium | High | Yearly | | RedirectWeb | $3 | Utility | QoL | No | Medium/Low | High | Yearly | | Security Camera | $55 | Security | QoL | Maybe | High | Medium | Yearly | | SGU Podcast | $140 | News | QoL | No | Medium | High | Monthly | | The Sizzle | $50 | News | QoL | No | Medium | High | Yearly | | Working Copy | $6.50 | Utility | QoL | No | High | High | 2 Yearly | **Total (Yearly): $2248** -- ## One-Time or Upon Major Release Expenses Products or services that are paid for on a one-time basis or when a major upgrade is released. Includes paid software I use on a regular basis. | Expense | Cost | Category | Required/QoL | Want To Cancel/Replace | Usefulness | Satisfaction | Frequency | | :------------- | :----- | :------------ | :----------: | :--------------------: | :--------: | :----------: | :------------: | | Geekbench | $15 | Utilities | QoL | No | Low | Medium | Major Versions | | LaunchControl | $27.60 | Utilities | QoL | No | Medium | High | Major Versions | | MusicHarbor | | Entertainment | QoL | No | High | High | Once-Off | | Parcel | | Utilities | QoL | No | Medium | High | Once-Off | | Pixelmator Pro | $59 | Photography | QoL | No | High | High | Once-Off | | Plex Pass | $111 | Entertainment | QoL | No | High | High | Once-Off | | Reeder | | News | QoL | No | High | High | Major Versions | | Things | | PKM | QoL | Maybe | High | Medium | Once-Off | | WatchChat 2 | $13 | Utilities | QoL | No | Low | Medium | Once-Off | | Yomu Pro | $8 | Entertainment | QoL | No | Medium | High | Once-Off | | Bumpr | $3 | Utilities | QoL | No | High | High | Once-Off | ## Paused/Cancelled These are products or services that I have either paused or cancelled. I may resume them in the future. | Expense | Type | Category | Reason | Satisfaction | Cost (Yearly) | | :--------- | :--: | :------------ | :----------------: | :----------: | :-----------: | | Kayo Sport | SaaS | Entertainment | Only when F1 is on | Medium | $140 | --- ## Changelog | Last Updated | Notes | |--------------|------------------| | 2023-01-05 | Initial creation | --- ## Notes - All prices are in AUD. - I really want a native replacement for 1Password, since version 8 it has seriously has gone downhill. - I might end up replacing Backblaze with BorgMatic. - [Energy, Sustainability and Deployment Frequency](https://smcleod.net/2022/12/energy-sustainability-and-deployment-frequency/index.md): Energy, Sustainability and Deployment Frequency --- title: "Energy, Sustainability and Deployment Frequency" date: 2022-12-31T12:03:41+11:00 lastmod: 2022-12-31T12:03:41+11:00 author: Sam McLeod description: "How frequent deployments give teams regular wins, and why that matters for their energy and sustainability." keywords: [ "Delivery", "deployment", "frequency", "energy", "wins", "DevOps", "Agile", "Leadership", "management" ] tags: [ "Delivery", "health", "wins", "DevOps", "agile", "Leadership", "management" ] categories: [ "DevOps", "Leadership", "Health" ] series: [] cover: image: "wins.jpg" toc: enable: true auto: false math: false draft: false type: posts --- # Deployment / Delivery Frequency I often end up needing to advocate for more frequent delivery/deployments with clients. There's the usual benefits commonly discussed such as improved feedback, reduced risk, well understood processes, maintainable dependencies etc... however {{< highlight color="#c5ecff" >}} what's often missed entirely is how it relates to the health and sustainability of the team. {{< /highlight >}} _Note: I use the term "Delivery" and "deployment" interchangeably throughout this post._ --- ## How We Get Our Energy > I like to think of how get my energy throughout the day/week/month/year in terms of "wins". ### "Wins" I get the majority of my energy from "wins". - **Wins are when I complete, deliver, create or fix something.** - {{< highlight color="#c5ecff" >}}The amount of energy I get does _not_ scale relative to the size of the win. {{< /highlight >}} - Even if it did - lots of {{< highlight color="#c5ecff" >}}regular, small wins add up to more energy than a few big wins. {{< /highlight >}} - From a health point of view - avoiding the need for big-highs is more sustainable, resulting in less big-lows. - **Toil drains my energy**. - Overheads such as poor tooling, paperwork, "management culture" etc... {{< highlight color="#c5ecff" >}}Deployment frequency is a key driver of "wins". {{< /highlight >}} There are simply more opportunities to get our wins in as delivery rate increases. ## Effective Delivery Requires Effective Learning Exploring delivery frequency in the context of how we learn is critical if we want to understand how it relates to our health and sustainability. We learn best with: - **Tasks that fit in our working memory**. - If you can't remember what you did - how will you learn from it? - **Repetition**. - {{< highlight color="#c5ecff" >}}Repetition is built on the momentum of wins. {{< /highlight >}} - **System optimisation through simplification**. - Eliminating unnecessary components and processes reduces context switching and cognitive load. - Cognitive load is the amount of mental effort required to process information. As it increases so does the risk of mistakes, fatigue and burnout. - As individual efficiency increases - so does our energy. - **Reward systems**. - Rewards are in the form of energy attained get from wins. These are all related to delivery frequency, here in lies the connection to health and sustainability, and as such deployment frequency is a key driver of effective learning. > This highlights that delivery/deployment frequency exists not only within the scope of software quality - but also within the critical path for health and sustainability. - [Zsh Configuration and Plugins - Part Two](https://smcleod.net/2022/11/zsh-configuration-and-plugins-part-two/index.md): Zsh Configuration and Plugins - Part Two --- title: "Zsh Configuration and Plugins - Part Two" date: 2022-11-18T16:32:43+11:00 lastmod: 2022-11-18T16:32:43+11:00 author: Sam McLeod description: "My Z-Shell configuration, scripts and hacks" keywords: ["zsh", "bash", "scripting", "linux", "macOS"] tags: ["zsh", "bash", "scripting", "linux", "macOS"] categories: ["Zsh", "Scripting", "Linux", "macOS"] series: ["ZSH", "Code"] images: ["zsh-part-two.gif"] cover: image: "zsh-part-two.gif" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles toc: enable: true auto: false math: false draft: false --- Zsh Configuration and Plugins - Part two _You can find the first post in this series [here](/2022/11/zsh-configuration-and-plugins-part-one/)._ --- # Functions Many of my functions start out as aliases and then I convert them to functions when I need to add more functionality. You can find these in [9-function.rc](https://github.com/sammcj/zsh-bootstrap/blob/master/9-functions.rc). --- ```bash pdfcompress () { gs -q -dNOPAUSE -dBATCH -dSAFER -sDEVICE=pdfwrite -dCompatibilityLevel=1.3 -dPDFSETTINGS=/screen -dEmbedAllFonts=true -dSubsetFonts=true -dColorImageDownsampleType=/Bicubic -dColorImageResolution=144 -dGrayImageDownsampleType=/Bicubic -dGrayImageResolution=144 -dMonoImageDownsampleType=/Bicubic -dMonoImageResolution=144 -sOutputFile="$1".compressed.pdf "$1"; } ``` ```bash function echoerr() { printf "%s\n" "$*" >&2; } ``` --- ## Navigation ### `fcd` Interactive cd using fzf ```bash function fcd() { local dir; while true; do # exit with ^D dir="$(ls -a1p | grep '/$' | grep -v '^./$' | fzf --height 40% --reverse --no-multi --preview 'pwd' --preview-window=up,1,border-none --no-info)" if [[ -z "${dir}" ]]; then break else cd "${dir}" || exit fi done } ``` --- ## Creation Functions ### `tch` Touches all files passed in as arguments, if the directory specified doesn't exist it will be created. ```bash tch() { for x in "$@"; do __mkdir "${x:h}" done touch "$@" } ``` --- ## AWS Functions ### AWS Azure AD login #### `aws-azure-login` Authenticates to AWS using Azure AD and sets the profile. ```bash function aws-azure-login() { command aws-azure-login --no-prompt --profile "$@" export AWS_PROFILE=$@ export AWSCLIPARAMS="--profile=$@" } ``` #### `s2a` Authenticates to AWS using saml2aws and sets the profile. ```bash function s2a { eval $($(which saml2aws) script --shell=bash --profile=$@); } ``` ```bash function awslogin() { if [ -z "${1}" ]; then echo "ERROR: account name required, e.g. awslogin data-dev" else # Check credentials are current, refresh if needed and export into shell aws configure list --profile "${1}" && eval $(saml2aws script --profile "${1}") fi } ``` --- ## Encryption Functions ### `encrypt_file_pw` File with generated password using age. ```bash encrypt_file_pw() { if ! command -v age &> /dev/null; then echo "age could not be found. Install it with 'brew install age'" return else age -p "$1" -o "${1}.age" fi } ``` #### `keychain_password` Reads a password from keychain and outputs it usage: `keychain_password ` ```bash keychain_password() { # Make sure there no screen recording active on before returning the password if [[ $(pgrep -i screencapture) ]]; then echo "ERROR: macOS screen recording (screencapture process) is active. Please stop it before printing the password." return 1 fi security find-generic-password -s "$1" -a "$(whoami)" -w } ``` #### `keychain_password_prompt` Prompts for a name and a password and stores it in keychain ```bash keychain_password_prompt() { echo "Enter a name for the password:" read -r name echo "Enter the password:" # disable echoing the password stty -echo read -r password security add-generic-password -s "$name" -a "$(whoami)" -w "$password" } ``` #### `ssh-add-keys` A function that checks ssh-add and adds my keys if they're not already added ```bash function ssh-add-keys() { if ! ssh-add -l|grep -qe 'ED25519\|RSA'; then ssh-add --apple-use-keychain ~/.ssh/id_*.key fi } ``` --- ## Misc Functions ### `list_env` list env variables with fzf ```bash list_env() { var=$(printenv | cut -d= -f1 | fzf) \ && echo "$var=$(printenv "$var")" \ && unset var } ``` ### `update_asdf` ```bash function update_asdf() { asdf update asdf plugin-update --all } ``` ### `__mkdir` Creates a directory if it doesn't exist. - Alias: `mkcd` ```bash __mkdir() { if [[ ! -d $1 ]]; then mkdir -p "$1"; fi } ``` ### `mv` Usage: `mv oldfilename` If you call mv without the second parameter it will prompt you to edit the filename on command line, useful when you want to change just a few letters in a long name. ```bash function mv() { if [ "$#" -ne 1 ]; then command mv "$@" return fi if [ ! -f "$1" ]; then command file "$@" return fi read -ei "$1" newfilename mv -v "$1" "$newfilename" } ``` ### `clean_string` ```shell clean_string() { # Escape special characters in a string such as $, ", ', `, \, and newline. # Usage: escape_string "string to escape" local string="${1}" local escaped_string escaped_string=$(printf '%q' "${string}") echo "${escaped_string}" } ``` --- ## Git Functions ### `gitlazy` Adds, commits and pushes all changes in the current directory with the first argument as the commit message ```bash function gitlazy() { git add . git commit -a -m "$1" git push } ``` ### `gco` Checks out a branch when provided a branch name, or will prompt to select a branch using fzf. ![](gco.png) ```bash function gco(){ if [[ -n $1 ]]; then if [[ $1 == "-b" ]]; then # shift to remove -b from args shift 1 fi # check to see if the branch exists, if it doesn't offer to create it and check it out otherwise just check it out if git branch -a | grep -q "remotes/origin/$1"; then git checkout "$1" else echo "Creating branch $1" git checkout -b "$1" fi else git branch --sort=-committerdate | fzf --header 'Checkout Recent Branch' --preview 'git diff --color=always {1}' --pointer='>' | xargs git checkout fi } ``` ### `gbd` Deletes a branch when provided a branch name, or will prompt to select a branch to delete using fzf. ![](gbd.png) ```bash function gbd(){ if [[ -n $* ]]; then git branch -d "$@" else git branch --sort=-committerdate | fzf --header 'Delete Git Branch' --preview 'git diff --color=always {1}' --pointer='>' | xargs git branch delete fi } ``` ### `__untracked_files` Checks for any untracked files and prompts to add them ```bash # A function that provides an untracked_files check for other functions function __untracked_files() { # Check for any files not added - if there are any, list them and ask the user if they want to add them if git status --porcelain | grep -q "^??"; then echo "Untracked files detected:" git status --porcelain | grep "^??" local yn="" echo "Do you want to add all untracked files? (y/n)" vared -p "" -c yn case $yn in [Yy]*) git add . ;; *) echo "Skipping add" ;; esac fi } ``` ### `checkout-hotfix` Checks out a hotfix branch with the current date and user. - Alias: `coh` ```bash local PREFIX="hotfix-${USER}" local DATESTAMP DATESTAMP=$(date +%Y-%m-%d) # If the hotfix branch already exists, append a number to the branch name, if the number already exists increment it if command git branch -a | grep -q "$PREFIX-$DATESTAMP"; then local COUNT=1 while git branch -a | grep -q "$PREFIX-$DATESTAMP-$COUNT"; do COUNT=$((COUNT + 1)) done command git checkout -b "${PREFIX}-${DATESTAMP}-${COUNT}" else command git checkout -b "${PREFIX}-${DATESTAMP}" fi ``` ### `commit-hotfix` Commits a hotfix branch with the current date and user. - Alias: `ch` ```bash commit-hotfix() { local DATESTAMP DATESTAMP=$(date +%Y/%m/%d) local GIT_ARGS GIT_ARGS="" local MESSAGE MESSAGE="$*" # Set the commit message to the arguments passed to the function # Handle -n argument if [[ $1 == "-n" ]]; then GIT_ARGS="-n" # Add -n as a parameter to git if it's passed as an argument to the function MESSAGE="${*:2}" # Remove the first argument from the list of arguments fi # Check for any files not added - if there are any, list them and ask the user if they want to add them __untracked_files # Get the current branch name local CURRENT_BRANCH CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD) # Check if there are no changes to commit if git diff --cached --exit-code --quiet; then # Check if the branch exists on the remote if git ls-remote --heads origin "$CURRENT_BRANCH" | grep -q "$CURRENT_BRANCH"; then # Branch exists on remote, check for unpushed commits if git cherry -v origin/"$CURRENT_BRANCH" | grep -q "^+"; then echo "No new changes to commit, but there are commits that haven't been pushed." command git push else echo "No changes to commit or push." fi else # Branch does not exist on remote, prompt to push and track echo "Current branch '$CURRENT_BRANCH' does not exist on the remote." local yn="" echo "Do you want to push and track this branch on the remote? (y/n)" vared -p "" -c yn case $yn in [Yy]*) git push --set-upstream origin "$CURRENT_BRANCH" ;; *) echo "Skipping push" return ;; esac fi else # Changes to commit command git commit "$GIT_ARGS" -m "Hotfix-${DATESTAMP} -- ${MESSAGE}" command git push --set-upstream origin "$CURRENT_BRANCH" fi } ``` ### `checkout-jira` Checks out a branch with the provided JIRA ticket number along with the current date and user. - Alias: `coj` ```bash JIRA_BRANCH="IF-${1}-${USER}-$(date +%Y-%m-%d)" # If the JIRA branch already exists, append the current time to the branch name if command git branch -a | grep -q "remotes/origin/${JIRA_BRANCH}"; then JIRA_BRANCH="${JIRA_BRANCH}-$(date +%H-%M-%S)" fi command git checkout -b "${JIRA_BRANCH}" ``` ### `commit-jira` Takes the JIRA card number from the start of the branch name for the commit message (e.g. IF-1234 from the branch IF-1234-feature-name) - Alias: `cj`. ```bash function commit-jira() { # Assumes a branch name of the format IF-1234- local GIT_ARGS="" local MESSAGE="$*" # Set the commit message to the arguments passed to the function # If arguments are provided, use them for the commit message if [[ -n $* ]]; then MESSAGE="${*:2}" # Remove the first argument from the list of arguments shift 1 fi # Check for any files not added - if there are any, list them and ask the user if they want to add them __untracked_files # Get the current branch name local CURRENT_BRANCH CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD) # Check if there are no changes to commit if git diff --cached --exit-code --quiet; then # Check if the branch exists on the remote if git ls-remote --heads origin "$CURRENT_BRANCH" | grep -q "$CURRENT_BRANCH"; then # Branch exists on remote, check for unpushed commits if git cherry -v origin/"$CURRENT_BRANCH" | grep -q "^+"; then echo "No new changes to commit, but there are commits that haven't been pushed." command git push else echo "No changes to commit or push." fi else # Branch does not exist on remote, prompt to push and track echo "Current branch '$CURRENT_BRANCH' does not exist on the remote, pushing..." git push --set-upstream origin "$CURRENT_BRANCH" fi else # Changes to commit BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD | cut -d'-' -f1 -f2) command git commit "$GIT_ARGS" -m "$BRANCH_NAME -- $MESSAGE" command git push --set-upstream origin "$CURRENT_BRANCH" fi } ``` ### `pr-checkout` Checks out a pull request from GitHub using the GitHub CLI. ```bash function pr-checkout() { local jq_template pr_number jq_template='"'\ '#\(.number) - \(.title)'\ '\t'\ 'Author: \(.user.login)\n'\ 'Created: \(.created_at)\n'\ 'Updated: \(.updated_at)\n\n'\ '\(.body)'\ '"' pr_number=$( gh api 'repos/:owner/:repo/pulls' | jq ".[] | $jq_template" | sed -e 's/"\(.*\)"/\1/' -e 's/\\t/\t/' | fzf \ --with-nth=1 \ --delimiter='\t' \ --preview='echo -e {2}' \ --preview-window=top:wrap | sed 's/^#\([0-9]\+\).*/\1/' ) if [ -n "$pr_number" ]; then gh pr checkout "$pr_number" fi } ``` ### `git_add_commit_push` Git checkout new branch, git add, git commit, git push in all subdirectories matching a pattern - Alias: `gacp` ```bash function git_add_commit_push(){ if [[ -z $1 ]] || [[ -z "$2" ]] || [[ -z "$3" ]]; then echo 'You must pass three parameters, branchname, commit message, dir match - e.g. "my-branch" "commit message" ABC*'; fi BRANCHNAME="$1" COMMITNAME="$2" MATCHDIRS="$3" for dir in $MATCHDIRS; do ( cd "$dir" && git checkout -b "$BRANCHNAME" && git add . && git commit -n -m "$COMMITNAME" && git push ) done } ``` #### `ghac` Deletes workflow logs from a given repo older than 1 month Usage: `USER=myuser REPO=myrepo ghac` ```bash function ghac() { DATE=$(date -v "-1m" +"%Y-%m-%d") gh api "repos/${USER}/${REPO}/actions/runs" --paginate -q '.workflow_runs[] | select (.run_started_at <= "env.DATE") | (.id)' \ | xargs -n1 -I % gh api "repos/${USER}/${REPO}/actions/runs"/% -X DELETE } ``` ### Git Meta-Functions > The following Git functions are mainly consumed by other functions or the prompt. ```bash function git_prompt_info() { local ref if [[ "$(command git config --get oh-my-zsh.hide-status 2>/dev/null)" != "1" ]]; then ref=$(command git symbolic-ref HEAD 2> /dev/null) || \ ref=$(command git rev-parse --short HEAD 2> /dev/null) || return 0 echo "$ZSH_THEME_GIT_PROMPT_PREFIX${ref#refs/heads/}$(parse_git_dirty)$ZSH_THEME_GIT_PROMPT_SUFFIX" fi } ``` Outputs the name of the current branch ```bash function git_current_branch() { local ref ref=$(command git symbolic-ref --quiet HEAD 2> /dev/null) local ret=$? if [[ $ret != 0 ]]; then [[ $ret == 128 ]] && return # no git repo. ref=$(command git rev-parse --short HEAD 2> /dev/null) || return fi echo "${ref#refs/heads/}" } ``` Gets the number of commits ahead from remote ```bash function git_commits_ahead() { if command git rev-parse --git-dir &>/dev/null; then local commits commits="$(git rev-list --count @{upstream}..HEAD)" if [[ "$commits" != 0 ]]; then echo "$ZSH_THEME_GIT_COMMITS_AHEAD_PREFIX$commits$ZSH_THEME_GIT_COMMITS_AHEAD_SUFFIX" fi fi } ``` Gets the number of commits behind remote ```bash function git_commits_behind() { if command git rev-parse --git-dir &>/dev/null; then local commits commits="$(git rev-list --count HEAD..@{upstream})" if [[ "$commits" != 0 ]]; then echo "$ZSH_THEME_GIT_COMMITS_BEHIND_PREFIX$commits$ZSH_THEME_GIT_COMMITS_BEHIND_SUFFIX" fi fi } ``` Outputs if current branch is ahead of remote ```bash function git_prompt_ahead() { if [[ -n "$(command git rev-list origin/"$(git_current_branch)"..HEAD 2> /dev/null)" ]]; then echo "$ZSH_THEME_GIT_PROMPT_AHEAD" fi } ``` Outputs if current branch is behind remote ```bash function git_prompt_behind() { if [[ -n "$(command git rev-list HEAD..origin/"$(git_current_branch)" 2> /dev/null)" ]]; then echo "$ZSH_THEME_GIT_PROMPT_BEHIND" fi } ``` Outputs if current branch exists on remote or not ```bash function git_prompt_remote() { if [[ -n "$(command git show-ref origin/"$(git_current_branch)" 2> /dev/null)" ]]; then echo "$ZSH_THEME_GIT_PROMPT_REMOTE_EXISTS" else echo "$ZSH_THEME_GIT_PROMPT_REMOTE_MISSING" fi } ``` Formats prompt string for current git commit short SHA ```bash function git_prompt_short_sha() { local SHA SHA=$(command git rev-parse --short HEAD 2> /dev/null) && echo "$ZSH_THEME_GIT_PROMPT_SHA_BEFORE$SHA$ZSH_THEME_GIT_PROMPT_SHA_AFTER" } ``` Formats prompt string for current git commit long SHA ```bash function git_prompt_long_sha() { local SHA SHA=$(command git rev-parse HEAD 2> /dev/null) && echo "$ZSH_THEME_GIT_PROMPT_SHA_BEFORE$SHA$ZSH_THEME_GIT_PROMPT_SHA_AFTER" } ``` Get the status of the working tree ```bash function git_prompt_status() { local INDEX STATUS INDEX=$(command git status --porcelain -b 2> /dev/null) STATUS="" if eval "$(echo "$INDEX" | command grep -E '^\?\? ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_UNTRACKED$STATUS" fi if eval "$(echo "$INDEX" | grep '^A ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_ADDED$STATUS" elif eval "$(echo "$INDEX" | grep '^M ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_ADDED$STATUS" fi if eval "$(echo "$INDEX" | grep '^ M ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_MODIFIED$STATUS" elif eval "$(echo "$INDEX" | grep '^AM ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_MODIFIED$STATUS" elif eval "$(echo "$INDEX" | grep '^ T ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_MODIFIED$STATUS" fi if eval "$(echo "$INDEX" | grep '^R ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_RENAMED$STATUS" fi if eval "$(echo "$INDEX" | grep '^ D ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_DELETED$STATUS" elif eval "$(echo "$INDEX" | grep '^D ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_DELETED$STATUS" elif eval "$(echo "$INDEX" | grep '^AD ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_DELETED$STATUS" fi if eval "$(command git rev-parse --verify refs/stash >/dev/null 2>&1)"; then STATUS="$ZSH_THEME_GIT_PROMPT_STASHED$STATUS" fi if eval "$(echo "$INDEX" | grep '^UU ' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_UNMERGED$STATUS" fi if eval "$(echo "$INDEX" | grep '^## [^ ]\+ .*ahead' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_AHEAD$STATUS" fi if eval "$(echo "$INDEX" | grep '^## [^ ]\+ .*behind' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_BEHIND$STATUS" fi if eval "$(echo "$INDEX" | grep '^## [^ ]\+ .*diverged' &> /dev/null)"; then STATUS="$ZSH_THEME_GIT_PROMPT_DIVERGED$STATUS" fi echo "$STATUS" } ``` Compares the provided version of git to the version installed and on path outputs -1, 0, or 1 if the installed version is less than, equal to, or greater than the input version, respectively ```bash function git_compare_version() { local INPUT_GIT_VERSION INSTALLED_GIT_VERSION INPUT_GIT_VERSION=(${(s/./)1}) INSTALLED_GIT_VERSION=($(command git --version 2>/dev/null)) INSTALLED_GIT_VERSION=(${(s/./)INSTALLED_GIT_VERSION[3]}) for i in {1..3}; do if [[ ${INSTALLED_GIT_VERSION[$i]} -gt ${INPUT_GIT_VERSION[$i]} ]]; then echo 1 return 0 fi if [[ ${INSTALLED_GIT_VERSION[$i]} -lt ${INPUT_GIT_VERSION[$i]} ]]; then echo -1 return 0 fi done echo 0 } ``` Outputs the name of the current user ```bash function git_current_user_name() { command git config user.name 2>/dev/null } ``` Outputs the email of the current user ```bash function git_current_user_email() { command git config user.email 2>/dev/null } ``` - [Github Not-So-Reusable Actions](https://smcleod.net/2022/11/github-not-so-reusable-actions/index.md): Github Not-So-Reusable Actions --- title: "Github Not-So-Reusable Actions" description: "Github Actions Reusable Workflows vs Composite Actions" date: 2022-11-06T15:52:27+11:00 lastmod: 2022-11-08T08:52:27+11:00 author: Sam McLeod keywords: ["github", "ci", "yaml"] tags: ["github", "ci", "yaml", "DRY"] categories: ["Software", "Github"] series: [] cover: image: "./old-man-yells-at-github-actions.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles toc: true math: false draft: false --- Github Actions is a "BYOBE" (Bring Your Own Bloody Everything) offering that provides basic CI with surprisingly convoluted configuration to Github. The product as a whole is an exercise in frustration, one of the worst parts is the lack of reusability and the complexity required to achieve it. Github's concept of reusable workflows on Github Actions is clearly a cobbled together afterthought. There's no one way to reuse workflows - you can't simply include / inherit FROM another workflow and no support for standard YAML anchors/aliases[^1]. - [Reusable Workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows) - [Composite Actions](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action) --- ## Composite Actions Composite actions are a way to combine multiple steps into a single step. They are a way to reuse [steps](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idsteps), but not [workflows](https://docs.github.com/en/actions/using-workflows/about-workflows). - Booleans in composite actions are actually Strings - even if you set their type to Boolean[^2]. 🤦 - Composite Actions requires setting the shell in each step (`shell: bash`). - Composite Actions requires setting run.using as composite in each workflow (`runs.using: "composite"`). - Composite Actions can be called as part of one of the steps in a job. - Composite Actions can include multiple files, so it's possible to use files from the Action or from the user's repository. - Composite Actions can include multiple steps, but not multiple jobs. - Composite Actions do not support `${{ secrets.mysecret }}`, a workaround is to pass the secrets as input parameters from calling workflow. 🤦 ### Example {#composite-action-example} Composite Action ```yaml name: Composite Action Example description: "Composite Action Example" inputs: my-input: description: 'Input for the composite action' required: true default: 'Hello World' my-secret-input: description: 'Secret input for the composite action' required: true runs: using: "composite" steps: - name: Hello run: echo world shell: bash - name: Do something secret shell: bash run: login -p ${{ inputs.my-secret-input }} ``` Calling Workflow ```yaml ... jobs: hello-world: runs-on: ubuntu-latest steps: - name: hello world uses: actions/helloworld@v3 - uses: myorg/my-reusable-repo/.github/actions/my-action@main with: node-auth-token: ${{ secrets.SUPER_SECRET_TOKEN }} ... ``` --- ## Reusable Workflows A reusable workflow is similar to normal GitHub Actions workflow, however: - Reusable workflows cannot call other reusable workflows. - Reusable workflows cannot accept matrix input (`strategy.matrix`). - Reusable workflows are a single YAML file, with no additional files retrieved by default. - Reusable workflows are called as a job - but cannot contain steps in the calling workflow. - The reusable workflow itself can have multiple steps. - Reusable workflows can include multiple jobs, and multiple steps in each job. - Reusable workflows are defined through the `workflow_call` event type. - "Regular" (non-reusable) Workflow can be triggered through a `workflow_dispatch` event. - Both event types support `input` - however with dispatchable Workflows `input` object is `${{ github.event.inputs }}` where as callable workflows receive `${{ inputs }}` 🤦. - As a result, in order to make a reusable workflow dispatchable, a wrapper workflow may be required. - Unlike Composite Actions, Booleans in Reusable Workflows _are_ Booleans. ### Example {#reusable-workflow-example} Reusable Workflow ```yaml name: Reusable Workflow Example on: workflow_call: jobs: hello-world: name: "Example job in a reusable workflow" runs-on: ubuntu-latest steps: - uses: "actions/checkout@v3" with: repo_token: ${{ secrets.GITHUB_TOKEN }} ``` Calling Workflow ```yaml ... jobs: hello-world: uses: myorg/my-reusable-repo/.github/workflows/my-action.yml@main with: secrets: inherit ... ``` --- ## (I Can't Get No) YAML Anchors As a side note - the YAML Spec defined a simple way to reuse values in YAML documents by means of [Anchors and Aliases](https://yaml.org/spec/1.2/spec.html#id2765878). If you haven't used them before (and don't worry - you can't on Github) - I have an example [here](../yaml-anchors-and-aliases) Unfortunately despite requests[^1] to[^3] do[^4] so[^5] - Github Actions does not support YAML anchors or aliases. While obviously not a solution for reusable workflows - these would make large workflows a lot simpler and easier to maintain and as such reduce reliance on the band aid solutions that are _Reusable Workflows_ and _Composite Actions_. --- [^1]: [GitHub Actions: Support YAML anchors](https://github.com/actions/runner/issues/1182) [^2]: [Github Booleans are only _sometimes_ Booleans](https://github.com/actions/runner/issues/1483) [^3]: [Support YAML anchors](https://github.com/community/community/discussions/4501) [^4]: [Support YAML anchors](https://github.com/actions/starter-workflows/issues/162) [^5]: [Support YAML anchors](https://github.com/actions/runner/issues/438#issuecomment-722778085) - [YAML Anchors and Aliases](https://smcleod.net/2022/11/yaml-anchors-and-aliases/index.md): YAML Anchors and Aliases --- title: "YAML Anchors and Aliases" description: "Using YAML Anchors and Aliases to make config files more DRY" date: 2022-11-06T15:52:27+11:00 lastmod: 2022-11-06T08:52:27+11:00 author: Sam McLeod keywords: ["yaml"] tags: ["yaml", "DRY"] categories: ["Software"] series: [] cover: image: "Official_YAML_Logo.png" toc: true math: false draft: false --- The YAML Spec defined a simple way to reuse values in YAML documents by means of [Anchors and Aliases](https://yaml.org/spec/1.2/spec.html#id2765878). - Anchors (`&`) are used to define a value. - Aliases (`*`) are used to reference a value. - Overrides (`<<:`) are used to merge values. ## Example This may look more complicated at first, but once you know what it's doing it's _very_ simple and can greatly reduce the amount of duplication in your YAML files and with that the risk of config drift and copy/pasta errors. ```yaml simple-anchor: &hw "echo Hello World" env: &env-common - MY_ENV_VAR_1: "my value 1" - MY_ENV_VAR_2: "my value 2" - MY_ENV_VAR_3: "my value 3" with: &with-common my-input: "my value" my-secret-input: ${{ secrets.SECRET }} ... steps: - name: "Hello" run: *hw env: - SPECIAL_VAR: "my special value" <<: *env-common <<: *with-common - name: "World" run: *hw env: <<: *env-common <<: *with-common ``` Let's break that down: ## Define YAML Anchors ```yaml simple-anchor: &hw "echo Hello World" # Create an anchor named 'hw' with the value "echo Hello World" env: &env-common # Create an anchor named 'env-common' with the values that follow - MY_ENV_VAR_1: "my value 1" - MY_ENV_VAR_2: "my value 2" - MY_ENV_VAR_3: "my value 3" with: &with-common # Create an anchor named 'with-common' with the values that follow my-input: "my value" my-secret-input: ${{ secrets.SECRET }} ``` ## Use YAML Anchors ```yaml steps: - name: "Hello" run: *hw # Use Alias 'hw' which results in run: "run: echo Hello World" env: - SPECIAL_VAR: "my special value" <<: *env-common # Use Alias 'env-common' with override (<<:) to merge into existing values # (env: # - SPECIAL_VAR: "my special value" # - MY_ENV_VAR_1: "my value 1" # - MY_ENV_VAR_2: "my value 2" # - MY_ENV_VAR_3: "my value 3") <<: *with-common # Use Alias 'with-common' with override to merge into existing values # my-input: "my value" # my-secret-input: ${{ secrets.SECRET }}) ``` And that's it! - [Zsh Configuration and Plugins - Part One](https://smcleod.net/2022/11/zsh-configuration-and-plugins-part-one/index.md): Zsh Configuration and Plugins - Part One --- title: "Zsh Configuration and Plugins - Part One" date: 2022-11-06T10:32:43+11:00 lastmod: 2022-11-06T10:32:43+11:00 author: Sam McLeod description: "My Z-Shell configuration, scripts and hacks" keywords: ["zsh", "bash", "scripting", "linux", "macOS"] tags: ["zsh", "bash", "scripting", "linux", "macOS"] categories: ["Zsh", "Scripting", "Linux", "macOS"] series: ["ZSH", "Code"] cover: image: "colour_vertical_icon_wbg_wide.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles images: ["colour_vertical_icon_wbg_wide.png"] toc: true math: false --- As of 2022, I've been using zsh as my primary shell for 14 years. Over that time I've experimented with and accumulated a lot of configuration, scripts, hacks, plugins and themes. I've settled on a configuration that I'm mostly happy with, over this series I'm going to share my current setup. I store all of my configuration in a git repository available here: [zsh-bootstrap](https://github.com/sammcj/zsh-bootstrap) {{< github-button button="star" user="sammcj" repo="zsh-bootstrap" count="true" >}} {{< github-button button="fork" user="sammcj" repo="zsh-bootstrap" count="true" >}} ## File Structure I've organised my setup into a few different files. ### Installation `bootstrap_shell.sh`- Used to install the shell, configuration, plugins and homebrew packages when first setting up a new macOS machine/VM. ### Configuration - `zshrc` - The main zsh config file, it sources the following (.rc) files which are all just standard zsh files. - `0-paths.rc` - Sets up the PATH variable. - `1-zgen.rc` - Loads the plugin manager and plugins. - `3-location_specifics.rc` - Sets up location specific configuration (e.g. based on network/hostname). - `4-aliases.rc` - Sets up command aliases. - `5-exports.rc` - Sets up environment exports. - `6-zsh_options.rc` - Sets up zsh options/configuration flags. - `7-history.rc` - Sets up history configuration. - `9-functions.rc` - Sets up reusable functions. - `10-prompt.rc` - Sets up the prompt. - `14-source-files.rc` - Sources any files / configuration provided by tools, e.g. [fnm](https://github.com/Schniz/fnm), [asdf](https://github.com/asdf-vm/asdf), [iterm2_shell_integration](https://iterm2.com/documentation-shell-integration.html) etc. The only exception to this is files with private (`.*(.|-)private(.|-).*`) in the name, which are ignored by git, they don't contain anything sensitive such as secrets, but I don't want to share them. ### Brew packages - `Brewfile-` - Brewfile for the current machine, this is used by the bootstrap script to install homebrew packages, and by the alias `brewbackup` to backup the current packages. ### Dotfiles And a number of dotfiles that are symlinked into place: - `.vimrc` - Vim configuration. - `.gitconfig` - Git configuration. - `.gitconfig-no_push` - Git configuration only sourced on repositories where I wish to prevent pushing to a specific branch. - `.gitignoreglobal` - Global gitignore file. - `.dircolors` - Configuration for `ls` colours. - `tmux.conf` - Tmux configuration. More on these files later! ## Plugin Management I use [zgen](https://github.com/tarjoilija/zgen), its lightweight, simple and fast. You define a list of plugins you want to use, save that list as part of your zsh config, zgen will download and load them for you. ```shell git clone https://github.com/tarjoilija/zgen.git "${HOME}/.zgen" --depth=1 ``` ### Plugins - [RobSis/zsh-completion-generator](https://github.com/RobSis/zsh-completion-generator) - Plugin that generates completion functions automatically from getopt-style help texts. - [zdharma-continuum/fast-syntax-highlighting](https://github.com/zdharma-continuum/fast-syntax-highlighting) - Feature rich syntax highlighting for Zsh. - ![](https://raw.githubusercontent.com/zdharma-continuum/fast-syntax-highlighting/master/images/highlight-much.png) - [zsh-users/zsh-autosuggestions](https://github.com/zsh-users/zsh-autosuggestions) - Fish-like autosuggestions for zsh. - [lukechilds/zsh-better-npm-completion](https://github.com/lukechilds/zsh-better-npm-completion) - Better npm completion for ZSH. - Adds npm install recommendations from npm cache, uninstall recommendations from dependencies/devDependencies, detailed information on script contents for npm run - [MichaelAquilina/zsh-you-should-use](https://github.com/MichaelAquilina/zsh-you-should-use) - ZSH plugin that reminds you to use existing aliases for commands you just typed. - ![](https://raw.githubusercontent.com/MichaelAquilina/zsh-you-should-use/master/img/global.png) - [3v1n0/zsh-bash-completions-fallback](https://github.com/3v1n0/zsh-bash-completions-fallback) - Fallback to bash completions if zsh completions are not available. - [olivierverdier/zsh-git-prompt](https://github.com/olivierverdier/zsh-git-prompt) - A very fast git prompt for zsh. - [paulirish/git-open](https://github.com/paulirish/git-open) - Type `git open` to open the GitHub page or website for a repository in your browser. - [peterhurford/git-it-on.zsh](https://github.com/peterhurford/git-it-on.zsh) - A git alias for the lazy, similar to git-open - I'm currently weighing up between the two. - [sunlei/zsh-ssh](https://github.com/sunlei/zsh-ssh) - Better host completion for ssh in Zsh. - [torifat/npms](https://github.com/torifat/npms) - Run npm scripts interactively using fzf. - [nnao45/zsh-kubectl-completion](https://github.com/nnao45/zsh-kubectl-completion) - kubectl completion for zsh. - [trejo08/kubecolor-zsh](https://github.com/trejo08/kubecolor-zsh) - kubectl output colorizer. - [zsh-users/zsh-completions](https://github.com/zsh-users/zsh-completions) - Additional completion definitions for Zsh. #### oh-my-zsh plugins zgen also supports loading individual oh-my-zsh plugins (which is far faster than loading all of oh-my-zsh). - [macos](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/macos) - macOS specific aliases and functions. - [docker](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/docker) - Docker aliases and functions. - [docker-compose](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/docker-compose) - Docker compose aliases and functions. - [aws](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/aws) - AWS aliases and functions. - [colored-man-pages](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/colored-man-pages) - Coloured man pages. - [command-not-found](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/command-not-found) - Suggests installing missing commands. - [fnm](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/fnm) - Fnm aliases and functions. - [fzf](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/fzf) - Fzf aliases and functions. - [nmap](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/nmap) - Nmap aliases and functions. - [gh](https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/gh) - GitHub CLI aliases and functions. ## Putting it all together The following script is sourced on shell initialisation, it loads zgen, and then loads all the plugins. ```zsh # load zgen source "${HOME}/.zgen/zgen.zsh" export ZGEN_RESET_ON_CHANGE="${HOME}/.zshrc" if ! zgen saved; then echo "Creating a zgen save" # load omz plugins zgen oh-my-zsh zgen oh-my-zsh plugins/macos zgen oh-my-zsh plugins/docker zgen oh-my-zsh plugins/docker-compose zgen oh-my-zsh plugins/gitfast zgen oh-my-zsh plugins/aws zgen oh-my-zsh plugins/colored-man-pages zgen oh-my-zsh plugins/command-not-found zgen oh-my-zsh plugins/fnm zgen oh-my-zsh plugins/fzf zgen oh-my-zsh plugins/nmap zgen oh-my-zsh plugins/gh # load normal plugins zgen loadall <" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles title: Encourage Your Peers To Contribute To Open Source keywords: ["culture", "opensource", "git", "Leadership", "values"] tags: ["culture", "opensource", "git", "Leadership", "values"] toc: true draft: false --- Contributing to Open Source is important to the quality and maintainability of the software and engineering communities we rely on every day - so why is it that so many developers/engineers never participate? --- I frequently cross paths who may have been in tech for some time - but don't contribute to Open Source projects. Their Github/GitLab accounts are bare with almost no activity. They don't add to or fix problems find the answers to in documentation - even log bug reports. Most companies don't do enough to contribute back to open source projects. It's not just that they don't provide time for their employees to contribute, they don't even encourage it let alone educate people on how to do it, or why it's important. --- ## What does it mean to contribute to Open Source? {#what} "Contributing" takes many forms: - Submitting bug reports / creating issues. - Commenting on other peoples bug reports / issues. - Updating or creating documentation. - Submitting pull/merge requests with fixes/features. --- ## Why don't people contribute to Open Source? {#why-not} Asking several people as to why - I've heard that the main reasons they don't contribute basically boil down to: - They are worried they might look silly. - They didn't realise there was value to other people in submitting a bug report / issues for their problems. - They don't think they can add value to the project. - They don't know how to interact (socially) with a given open source community. - They didn't think that fixing documentation was a valid contribution in itself. - They hadn't considered going through other peoples issues to see if they could help. In reality, the only way to get better at contributing to open source is to contribute to open source. --- ## Why is it important to contribute to Open Source? {#why} - It helps non-profits and small businesses get a leg up. - You can positively impact the health (stress levels, context switching etc...) of existing maintainers. - People are often more likely to help you if you help others. ## ...But why is it important for **_me_**? {#why-me} - You get to learn both new technical and social skills. - You get exposed to ideas and ways of thinking you may not be familiar with. - You gain meaningful experience that you can speak to throughout your career, with your clients and in job interviews (on either side of the table). _It can be as simple as making sure you log bugs/issues publicly, making your work visible and updating incorrect or missing documentation._ --- ### Contributing to Open Source is a skill like any other {#skill} If you're worried contributing for the first time, or not sure if you can add value - remember: - Everyone who contributes to open source started out as a beginner. - When logging bugs - provide logs, context, steps to reproduce and any other context that may be useful. - When submitting pull requests - make sure your code is at least well formatted, passes any linting rules and has reasonable comments. - When commenting on issues - be polite, be helpful and be patient. and above all else - be appreciative and kind to maintainers. ## OpenSource Guide [OpenSource Guide](https://opensource.guide/how-to-contribute/) is a great resource for learning how to contribute to open source projects. --- Please, don't just contribute - encourage your peers to contribute too. - [Top album picks for 2022](https://smcleod.net/2022/10/top-album-picks-for-2022/index.md): Top album picks for 2022 --- author: "Sam McLeod" categories: [ Music ] series: [ BestOf ] date: "2022-10-30T07:00:00Z" images: ["backdrop-linn.jpg"] cover: image: " backdrop-linn.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - music - bestof title: Top album picks for 2022 norss: true #TODO: keywords: ["music", "bestof"] toc: true math: false draft: true #TODO: --- My top album pics for 2022 (in no particular order) ### Artist - [Album](https://en.wikipedia.org/wiki/link) #TODO ![]() #TODO: Add image From [Wikipedia](https://en.wikipedia.org/wiki/link): #TODO: > Bla bla bla #TODO: - [Making Work Visible - Avoid DMs](https://smcleod.net/2022/10/making-work-visible-avoid-dms/index.md): Making Work Visible - Avoid DMs --- author: "Sam McLeod" categories: [ "Tech", "Leadership", "Values", "RemoteWork" ] date: "2022-10-20T07:00:00Z" lastmod: "2023-05-31T07:52:26Z" aliases: - /tech/2022/10/20/make-work-visible-dms/ images: ["silos.jpg"] cover: image: " silos.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles title: Making Work Visible - Avoid DMs keywords: ["work", "communication", "culture", "tech", "direct messages", "remotework"] tags: ["work", "communication", "culture", "tech", "direct messages", "remotework"] toc: true --- We create more value by having conversations in public instead of behind closed doors. This is a blurb I add to the teams wiki page. > ## Avoid sending DMs/PMs for support and technical questions > > ### We value making work visible > > Part of this is achieved by us making an effort to ensure we're asking for support and technical questions by starting a thread in our squads (or the relevant) room - instead of via of direct messaging (or calling) an individual. > > This helps with: > > - Sharing knowledge amongst the team. > - Identifying common problems that others might be having or aware of. > - Balances support to prevent any individual getting overloaded. > - Helps ensure a timely response. > - Allowing others to keep up to date with ongoing issues. > - Eases cognitive load by: > - Reducing the number of notifications people receive > - Reducing context switching. > - Allows people to search for previously occurring questions and answers. > - Highlighting documentation gaps. > - Making work visible. > > For more information see: > > - [Making Work Visible](https://smcleod.net/2022/10/making-work-visible-avoid-dms/) > - [Collaborate with kindness](https://slack.com/intl/en-au/blog/collaboration/etiquette-tips-in-slack) > - [NoHello](https://nohello.net) > > Thank you, > 😊 --- ## Just like with software development, conversations benefit from being open {#open} The majority of our work conversations should be open and benefit from the feedback and different perspectives of others. Open communication helps to foster a blameless culture, where people are encouraged to share their ideas, ask questions and embrace the idea of "I don't know". Learning, sharing and understanding within a silo is often a waste of time and effort. We can create more value by having conversations in 'public' instead of behind closed doors. Changing peoples habits from having private, 1 on 1 conversations to having public, open conversations often requires people to be reminded of the benefits. For some it is a change in behaviour, for others it is a change in mindset. --- ## False assumptions {#false-assumptions} Many people often default 1 on 1 private conversations based on common *false* assumptions. _> *"I don't want to share my ideas with the team until they're fully formed"*_ By sharing your ideas as they are forming you benefit from the feedback and different perspectives of others. This isn't just important for the problem you're trying to solve - it is also large part of how we learn and improve in the long term.   _>*"The others are busy with their own work, I don't want to bother them with something they don't care about"*_ As a sender of information, who are you to decide what others are interested in? Shouldn't it be the listener's right to decide if they can add value to a conversation? Even if you don't have anything to contribute, you can still learn from listening to a conversation. Getting an answer or piece of insight from someone you didn't expect is common when you start moving your discussions out into the open[^1].   _> *"I don't want to interrupt others with my questions"*_ Communicating openly does not mean yelling your ideas over the top of everyone, it doesn't even mean they have to be notified of your conversation let alone requested to participate in it. _There is a big difference between wasting peoples time with pointless meetings/workshops and sharing your information/discussions in a open forum._ Embracing and opt-in communication style, people have the option to choose to ignore the discussion if it doesn't interest them, or even to prioritise it and perhaps read through it later at their convenience. Communicating effectively means using the tools properly to minimise noise and the cognitive load of others.   --- ### 🧵 Using threads to promote open team communication {#use-threads} ![](megaphone.jpg) Many of the tools we use for communication are designed to be open and transparent - but it's up to us to use them properly. _Conversation threading may seem like a small feature of modern collaboration tools - using threads yields significant rewards._ Using threads can significantly reduce context switching, cognitive load and promotes more engaging communication. Without threads - you're constantly notifying everyone in the room of your train of thought - this might seem good for you, but it's not for everyone else. Even if people have notifications turned off they will desensitise to discussions and will miss important information. #### 1. Conversations are grouped by context Knowing the context (topic, problem, idea, etc) of a conversation is important for understanding the conversation while reducing context switching. You can have multiple simultaneous, asynchronous conversations without worrying about tripping over over each other or the need to create a new channel. #### 2. Reading a stream of conversation is easy Without threaded conversations the signal to noise ration is low, you often have to skip over unrelated messages to follow and make sense of the conversation. #### 3. Threaded discussions reduce the number of notifications people receive There is nothing worse than opening a messaging app and seeing an indicator of a large number of unread messages. If you're not interested in a conversation you can choose to ignore all of it by not joining the thread. If you've joined the thread you can choose to mute it if its no longer relevant to you. #### 4. Makes it easier search for and retain knowledge One risk with frequent conversations is that previously occurring questions or decisions can become hard to find. By using threads you not only can you search for the context/topic/starting message of a thread - you can also link to it - sharing knowledge of the discussion and not just a single persons message. The ability for new starters, people you assume aren't interested - or even outsiders to have access to the teams discussions and historical knowledge can help to reduce the need for synchronous communication events such as large meetings, the constant re-sharing of information and occurrences of invented here syndrome. #### 5. It allows people to prioritise their engagement People can read through the a discussion at their convenience - or not at all if it doesn't interest them. --- ### Open discussions avoidance is (in part) a learned habit From email where "best practices" include having as few people as possible, from managers booking meeting rooms with a select group of people and from the fear of being perceived as asking too many questions when we're new to a team. Remember: *We create more value by having conversations in public instead of behind closed doors.* --- ## References / Links - [Slack collaboration etiquette tips](https://slack.com/intl/en-au/blog/collaboration/etiquette-tips-in-slack) - [GitLab's Communication Handbook - Don't Send Direct Messages](https://about.gitlab.com/handbook/communication/#avoid-direct-messages) - [NoHello](https://nohello.net) [^1]: Thanks to Ville Saarinen from FlowDock's (now defunct) blog for the inspiration and some of the content for this post. - [The Best Of - 2022 Edition](https://smcleod.net/2022/10/the-best-of-2022-edition/index.md): The Best Of - 2022 Edition --- author: "Sam McLeod" categories: [ Tech, BestOf, Software ] date: "2022-10-19T19:00:00Z" aliases: - /tech/2022/10/18/apps-of-2022/ images: ["tools.jpg"] cover: image: " tools.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - bestof - software series: [ BestOf ] title: The Best Of - 2022 Edition --- Near the end of each year I note down a summary of the best apps I've enjoyed using throughout the year, here's 2022. _*See also [Firefox Addons - 2022](https://smcleod.net/software/2022/05/16/firefox-addons-2022/)*_ ## The Best - MeetingBar (Meetings in your menubar) - [Get MeetingBar](https://github.com/leits/MeetingBar) - Reeder (RSS) - [Get Reeder](https://www.reederapp.com/) - Shottr (Better screenshots) - [Get Shottr](https://shottr.cc/) - Bumpr (Multi-browser rules for links) - [Get Bumper](https://www.getbumpr.com/) - Onyx (macOS customisation) - [Get OnyX](https://titanium-software.fr/en/onyx.html) - Brew (Package manager) - [Get Brew](https://brew.sh/) - mas (Mac App Store manager for Brew) - [Get mas](https://github.com/mas-cli/mas) - Little Snitch (Firewall) - [Get Little Snitch](https://obdev.at/products/littlesnitch) - RadioSilence (Simple Firewall) - [Get RadioSilence](https://radiosilenceapp.com/) - Pixelmator Pro (Image/Photo editor) - [Get Pixelmator Pro](https://www.pixelmator.com/pro/) - Keka (Archive/Compression tool) - [Get Keka](https://www.keka.io/) - LaunchControl (Frontend for launchd) - [Get LaunchControl](https://www.soma-zone.com/LaunchControl/) - Peek (Quicklook plugins) - [Get Peek](https://apps.apple.com/au/app/peek-a-quick-look-extension/id1554235898?mt=12) - MusicHarbor (Track new and upcoming music releases) - [Get MusicHarbor](https://apps.apple.com/au/app/musicharbor-track-new-music/id1440405750) - DaisyDisk (Disk usage inspector) - [Get DaisyDisk](https://daisydiskapp.com/) - Things (GTD/Notes/Lists app) - [Get Thing](https://culturedcode.com/things/) - BetterTouchTool (macOS automation and customisation) - [Get BetterTouchTool](https://folivora.ai/) - Parcel (Track parcels) - [Get Parcel](https://apps.apple.com/au/app/parcel/id375589283) - Backblaze (Backups) - [Get Backblaze](https://www.backblaze.com/) - MicroSnitch (Get notifications for active mic/webcam) - [Get Micro Snitch](https://obdev.at/products/microsnitch/index.html) - Handbrake (Transcode video files) - [Get Handbrake](https://handbrake.fr/) - Amphetamine (Keep you mac awake) - [Get Amphetamine](https://apps.apple.com/us/app/amphetamine/id937984704?mt=12) - Bear (Markdown/notes/web clipping) - [Get Bear](https://bear.app/) - Calibre (eBook management and conversation) - [Get Calibre](https://calibre-ebook.com/) - ImageOptim Beta (Image compression) - [Get ImageOptim](https://imageoptim.com/ImageOptim1.8.9a1.tar.bz2) - *Note: This is a beta version, but it's the only version that supports Apple Silicon natively, the author is working on [ImageOptim 2 - A complete rewrite](https://github.com/ImageOptim/ImageOptim/issues/354)*. - Strongbox (Native password/secrets management) - [Get StrongBox](https://strongboxsafe.com/) - *Note: I will likely eventually replace 1Password with Strongbox, it's a native app (not Electron!), currently waiting on official browser extensions*. - Stay (Window management) - [Get Stay](https://cordlessdog.com/stay/) - Firefox (Don't run a browser from an ad company) - [Get Firefox](https://www.mozilla.org/en-GB/firefox/new/) - Fastmail (Email) - [Get Fastmail](https://www.fastmail.com) - mu-repo (Run bulk commands across git repos) - [Get mu-repo](https://fabioz.github.io/mu-repo/) You can find all the applications I use, along with my shell config and plugins within the [BrewFile](https://github.com/sammcj/zsh-bootstrap) which I use to bootstrap my macOS machines. ## The Worst - 💩 AWS CDK 💩 - 💩 1Password 8 💩 - 💩 Microsoft Teams 💩 - 💩 Microsoft Office365 💩 - [It's 2022 and we're (still) not deploying enough](https://smcleod.net/2022/08/its-2022-and-were-still-not-deploying-enough/index.md): It's 2022 and we're (still) not deploying enough --- author: "Sam McLeod" categories: [ ml, DevOps ] date: "2022-08-23T07:00:00Z" aliases: - /DevOps/2022/08/23/were-not-deploying-frequently-enough/ images: ["nick-abrams-FTKfX3xZIcc-unsplash.jpg"] cover: image: " nick-abrams-FTKfX3xZIcc-unsplash.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - DevOps - software development - agile - enterprise title: It's 2022 and we're (still) not deploying enough --- ## We're (still) not deploying enough It's 2022 and not deploying frequently enough is *still* one of the most common causes of software failure. I couldn't tell you how often I work with clients that have deployment problems and it turns out they haven't deploy the code in production for weeks, sometimes months, or in a few cases years. Sometimes it's because they're afraid of change, sometimes it's due to a lack of resources, but more often than not it's due to fear of the unknown. But you know what? **If you're not deploying your code, then you have no idea if it works or not**. Do you want to know if your code works? Deploy it in production! The more you deploy, the better you'll get at it. And what happens when you get good at something? You get really good at it. You'll be able to deploy code faster, more often and with less bugs than ever before. ## Mean Time To Change matters The Mean Time To Change (MTTC) metric is one of the most important metrics in software engineering. It measures how long it takes to deploy a change, which can be used as a measure of quality. Time spent on testing and deployment will be saved by reducing MTTC, which means more time for other tasks like building products or improving processes. In fact, deploying often is a key part of any DevOps strategy because it helps reduce costs associated with deployment errors and keeps them from happening too often. It's important to note that MTTC doesn't mean anything if you don't have a robust way of determining how often your team releases software. If it takes an hour for someone on your team to deploy changes and another half-hour before everyone else can verify them, then even though they may be deploying every day, they're still not deploying fast enough because there's no guarantee that the code will be released in production within 24 hours. People have been talking about the importance of MTTC for decades. I'm a big fan of the work that the DevOps movement has done in this area, but there's still so much more that needs to be done. It's time we start thinking about MTTC at every level from individual engineers to entire companies. - [The 4 key DevOps metrics for organisational performance](https://cloud.google.com/blog/products/DevOps-sre/using-the-four-keys-to-measure-your-DevOps-performance). - [MTTC - Wikipedia](https://en.wikipedia.org/wiki/Mean_time_to_change). ## Why some companies (usually large enterprises) are afraid of deploying daily - and why they're wrong The first group that I want to address are the large, more traditional enterprises. These companies have a lot of reasons for not deploying frequently enough: fear of change, lack of resources and time amongst them. They're wrong in thinking deploying less means less problems. Research from Accelerate and The State of DevOps shows that enterprises deploy more frequently correlate highly with their software having far less bugs, teams having improved productivity and as a result a more competitive product offering. Daily deployments help you catch problems early on when they're easier to fix than later down the line when it's harder to fix-and if you can't fix them then there's no way you can prevent bugs from happening again! That's why it's so important for all developers not just those at startups who are used to working this way but also those at large enterprises and even their managers! Daily deployments, when done right and with the proper tools, can help everyone get better at what they do. ## Convincing your team - and the business This can sometimes be a challenge, especially at slow moving, traditional enterprises. But like anything it's hard to convince someone that they should do something if they've never done it before (and have no idea how to do it). This is where you come in: You can show them what it's like when you're deploying your code frequently, and the benefits that come from doing so. The first thing to do is make them understand that there are benefits to deploying code frequently. It's not just good for developers, it's also good for business. Why? Because faster deployments means less bugs, lower support costs and happier customers! Once they understand this, then they can start thinking about how they might do it themselves. There are two common ways to approach this. ### Approach A - Showcase what good looks like Start small, pick a modern or at least recently developed application, and show them how it's done. This can be a great way to get started, make sure your project is visible to everyone internally or better yet - open source it! This is a great way to get people excited about how they can use it, how quickly they can deploy and how easy it is to get started. ### Approach B - Get everyone on board with the idea of deploying code frequently It's hard for developers to adopt this mindset if they aren't encouraged by their managers or business owners. So, talk to your management about it and let them know how important it is that you get buy-in from them. The next step is to figure out how you're going to do it. - How are you going to deploy your code frequently? - What tools do you need? - Do you have a staging environment already in place? If not, then now is the time to set one up! This approach can work if there is a lot of trust between management and developers, but can take a significant time if there is little trust, motivation or engagement. ### What does bad look like In addition to either of these approaches - make sure you showcase what bad looks like - What does it mean to be bad at deploying software and how has that manifested in your organisation? - What are some of the impacts of not being good at deploying software? - How often do things fail and how much time does it take to recover from these failures? - How do you currently know if something is failing? - How do you currently know if something is *working*? - What's the cost? (This can be difficult to figure out, sometimes estimates are good enough). You may be surprised by what you find! ## Conclusion It's critical to do your due diligence and make sure that you're deploying frequently enough. You want to be confident that your app is ready when it's needed, so that customers don't have to wait for you or suffer from any problems. If you find yourself struggling with this issue, consider hiring a consultant or coach who can help guide you through the process of continuously improving your development practices, making it easier for them to work together effectively on current and future projects as well! ### References - [DORA - DevOps Research and Analysis](https://www.DevOps-research.com/research.html) - [Accelerate: The Science of Lean Software and DevOps](https://itrevolution.com/book/accelerate/) - [The Phoenix Project](https://itrevolution.com/the-phoenix-project/) - [The DevOps Handbook](https://itrevolution.com/the-DevOps-handbook/) - [2021 Accelerate State of DevOps Report](https://cloud.google.com/blog/products/DevOps-sre/announcing-dora-2021-accelerate-state-of-DevOps-report) - [2019 DORA Accelerate State of DevOps Report](https://cloud.google.com/blog/products/DevOps-sre/the-2019-accelerate-state-of-DevOps-elite-performance-productivity-and-scaling) - [DALL·E 2 - Selected works](https://smcleod.net/2022/07/dalle-2-selected-works/index.md): DALL·E 2 - Selected works --- author: "Sam McLeod" categories: [ machine-learning ] date: "2022-07-08T07:00:00Z" aliases: - /machine-learning/2022/07/08/dalle-2/ images: ["dalle2spaceorange.jpeg"] cover: image: " dalle2spaceorange.jpeg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - art - machinelearning - humour title: DALL·E 2 - Selected works --- Over the past month I've been playing with OpenAI's [DALL·E 2](https://openai.com/dall-e-2/), below are some of the interesting images I've generated. "_[DALL·E 2](https://openai.com/dall-e-2/) is a new AI system that can create realistic images and art from a description in natural language._" ## "A renaissance oil painting of two developers arguing over which javascript framework is the worst" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2012.23.27%20-%20a%20renaissance%20oil%20painting%20of%20two%20developers%20arguing%20over%20which%20javascript%20framework%20is%20the%20worst.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2012.23.36%20-%20a%20renaissance%20oil%20painting%20of%20two%20developers%20arguing%20over%20which%20javascript%20framework%20is%20the%20worst.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2012.23.47%20-%20a%20renaissance%20oil%20painting%20of%20two%20developers%20arguing%20over%20which%20javascript%20framework%20is%20the%20worst.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2012.23.53%20-%20a%20renaissance%20oil%20painting%20of%20two%20developers%20arguing%20over%20which%20javascript%20framework%20is%20the%20worst.jpeg) ## "A Ralph Steadman painting of 7 cats sitting on a fence, and one raven" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.36.13%20-%20A%20Ralph%20Steadman%20painting%20of%207%20cats%20sitting%20on%20a%20fence%2C%20and%20one%20raven.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.36.16%20-%20A%20Ralph%20Steadman%20painting%20of%207%20cats%20sitting%20on%20a%20fence%2C%20and%20one%20raven.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.36.21%20-%20A%20Ralph%20Steadman%20painting%20of%207%20cats%20sitting%20on%20a%20fence%2C%20and%20one%20raven.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.36.33%20-%20A%20Ralph%20Steadman%20painting%20of%207%20cats%20sitting%20on%20a%20fence%2C%20and%20one%20raven.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.36.37%20-%20A%20Ralph%20Steadman%20painting%20of%207%20cats%20sitting%20on%20a%20fence%2C%20and%20one%20raven.jpeg) ## "A poster of Bruce Lee posing in front of a wall painted with traditional Japanese dragon artwork" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.38.58%20-%20A%20poster%20of%20Bruce%20Lee%20posing%20in%20front%20of%20a%20wall%20painted%20with%20traditional%20Japanese%20dragon%20artwork.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.39.05%20-%20A%20poster%20of%20Bruce%20Lee%20posing%20in%20front%20of%20a%20wall%20painted%20with%20traditional%20Japanese%20dragon%20artwork.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.40.56%20-%20A%20realistic%20photo%20of%20Bruce%20Lee%20posing%20in%20front%20of%20a%20wall%20painted%20with%20traditional%20Japanese%20dragon%20artwork.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-20%2015.41.02%20-%20A%20realistic%20photo%20of%20Bruce%20Lee%20posing%20in%20front%20of%20a%20wall%20painted%20with%20traditional%20Japanese%20dragon%20artwork.jpeg) ## "An Art Nouveau painting of Batman outside a church" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-22%2015.58.38%20-%20An%20Art%20Nouveau%20painting%20of%20Batman%20outside%20a%20church.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-22%2015.58.45%20-%20An%20Art%20Nouveau%20painting%20of%20Batman%20outside%20a%20church.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-22%2015.58.50%20-%20An%20Art%20Nouveau%20painting%20of%20Batman%20outside%20a%20church.png) ## "Old man yells at cloud" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-24%2010.07.53%20-%20old%20man%20yells%20at%20cloud.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-24%2010.08.02%20-%20old%20man%20yells%20at%20cloud.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-24%2010.09.25%20-%20old%20man%20yells%20at%20cloud%20computing.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-24%2010.09.29%20-%20old%20man%20yells%20at%20cloud%20computing.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-24%2010.09.41%20-%20old%20man%20yells%20at%20cloud%20computing.png) ## Variations on "A romantic oil painting of a chicken wearing beats by dre and smoking a cigarette" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.15.21%20-%20a%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20and%20smoking%20a%20cigarette.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.15.24%20-%20a%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20and%20smoking%20a%20cigarette.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.16.44%20-%20a%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20red%20beats%20by%20dre%20headphones%20while%20smoking%20a%20cigarette%2C%20driving%20a%20black%20sports%20car.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.16.46%20-%20a%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20red%20beats%20by%20dre%20headphones%20while%20smoking%20a%20cigarette%2C%20driving%20a%20black%20sports%20car.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.16.56%20-%20a%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20red%20beats%20by%20dre%20headphones%20while%20smoking%20a%20cigarette%2C%20driving%20a%20black%20sports%20car.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.17.46%20-%20a%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20red%20headphones%20while%20smoking%20a%20cigarette%2C%20driving%20a%20black%20sports%20car.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.17.58%20-%20a%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20red%20headphones%20while%20smoking%20a%20cigarette%2C%20driving%20a%20black%20sports%20car.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.19.49%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20headphones%20while%20smoking%20a%20cigarette%20and%20flexing%20its%20biceps%20while%20driving%20a%20red%20sports%20car.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.19.52%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20headphones%20while%20smoking%20a%20cigarette%20and%20flexing%20its%20biceps%20while%20driving%20a%20red%20sports%20car.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.19.57%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20headphones%20while%20smoking%20a%20cigarette%20and%20flexing%20its%20biceps%20while%20driving%20a%20red%20sports%20car.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.20.00%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20beats%20by%20dre%20headphones%20while%20smoking%20a%20cigarette%20and%20flexing%20its%20biceps%20while%20driving%20a%20red%20sports%20car.png) ## "A Romantic oil painting of a chicken wearing ripped jeans, aviator sunglasses, smoking a cigarette leaning up against a wall" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.21.15%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20ripped%20jeans%2C%20aviator%20sunglasses%2C%20smoking%20a%20cigarette%20leaning%20up%20against%20a%20wall.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.21.18%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20ripped%20jeans%2C%20aviator%20sunglasses%2C%20smoking%20a%20cigarette%20leaning%20up%20against%20a%20wall.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.21.22%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20ripped%20jeans%2C%20aviator%20sunglasses%2C%20smoking%20a%20cigarette%20leaning%20up%20against%20a%20wall.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.21.30%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20ripped%20jeans%2C%20aviator%20sunglasses%2C%20smoking%20a%20cigarette%20leaning%20up%20against%20a%20wall.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.21.35%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20ripped%20jeans%2C%20aviator%20sunglasses%2C%20smoking%20a%20cigarette%20leaning%20up%20against%20a%20wall.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.21.40%20-%20romantic%20oil%20painting%20of%20a%20chicken%20wearing%20ripped%20jeans%2C%20aviator%20sunglasses%2C%20smoking%20a%20cigarette%20leaning%20up%20against%20a%20wall.png) ## "A romantic painting of snoop dog and a cool chicken wearing ripped jeans and aviators standing in a recording booth rapping into a microphone" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.34.03%20-%20a%20romantic%20painting%20of%20snoop%20dog%20and%20a%20cool%20chicken%20wearing%20ripped%20jeans%20and%20aviators%20standing%20in%20a%20recording%20booth%20rapping%20into%20a%20microphone%20with%20a%20m.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.34.12%20-%20a%20romantic%20painting%20of%20snoop%20dog%20and%20a%20cool%20chicken%20wearing%20ripped%20jeans%20and%20aviators%20standing%20in%20a%20recording%20booth%20rapping%20into%20a%20microphone%20with%20a%20m.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.36.50%20-%20a%20dutch%20masters%20style%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20standing%20in%20a%20recording%20booth%20rapping%20into%20a%20microphone%20along%20side%20a%20cool%20anthropomorphic%20.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.37.58.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.38.03.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.38.14.png) ## "A dutch masters style oil painting of the rapper snoop dogg rapping into a microphone along side a cool looking animal wearing ripped jeans" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.39.14%20-%20a%20dutch%20masters%20style%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20rapping%20into%20a%20microphone%20along%20side%20a%20cool%20looking%20animal%20wearing%20ripped%20jeans.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.39.34%20-%20a%20dutch%20masters%20style%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20rapping%20into%20a%20microphone%20along%20side%20a%20cool%20looking%20animal%20wearing%20ripped%20jeans.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.40.18%20-%20a%20dutch%20masters%20style%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20rapping%20into%20a%20microphone%20in%20a%20recording%20studio.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.40.21%20-%20a%20dutch%20masters%20style%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20rapping%20into%20a%20microphone%20in%20a%20recording%20studio.png) ## "An oil painting of the rapper snoop dogg as a marvel super hero whose super power is smoking" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.47.59%20-%20an%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20as%20a%20marvel%20super%20hero%20whose%20super%20power%20is%20smoking.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.48.03%20-%20an%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20as%20a%20marvel%20super%20hero%20whose%20super%20power%20is%20smoking.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.48.09%20-%20an%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20as%20a%20marvel%20super%20hero%20whose%20super%20power%20is%20smoking.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.48.12%20-%20an%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20as%20a%20marvel%20super%20hero%20whose%20super%20power%20is%20smoking.png) ## "A romantic style oil painting of the rapper snoop dogg as a marvel super hero whose super power is smoking" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.49.02%20-%20a%20romantic%20style%20oil%20painting%20of%20the%20rapper%20snoop%20dogg%20as%20a%20marvel%20super%20hero%20whose%20super%20power%20is%20smoking.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.51.36.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-28%2016.51.40.png) ## "A detailed psychedelic painting of a person listening to the band TOOL for the first time and having their third eye opened" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.04.17.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.04.21.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.04.46.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.04.57.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.05.18.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.05.22.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.05.31.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.05.43.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.07.17.png) ## "A white Romantic Sculpture of the dude from the film the big lebowski" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.27.08%20-%20a%20white%20Romantic%20Sculpture%20of%20the%20dude%20from%20the%20film%20the%20big%20lebowski%20.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.28.01%20-%20a%20full%20height%20white%20Romantic%20Sculpture%20of%20the%20dude%20from%20the%20film%20the%20big%20lebowski%20standing%20on%20a%20colourful%20magic%20carpet.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.28.09%20-%20a%20full%20height%20white%20Romantic%20Sculpture%20of%20the%20dude%20from%20the%20film%20the%20big%20lebowski%20standing%20on%20a%20colourful%20magic%20carpet.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.29.16%20-%20a%20full%20height%20white%20neoclasical%20Sculpture%20of%20the%20dude%20from%20the%20film%20the%20big%20lebowski%20standing%20on%20a%20colourful%20magic%20carpet.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-06-30%2011.29.20%20-%20a%20full%20height%20white%20neoclasical%20Sculpture%20of%20the%20dude%20from%20the%20film%20the%20big%20lebowski%20standing%20on%20a%20colourful%20magic%20carpet.png) ## "A dutch masters oil painting of a robot cat" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.39.02%20-%20a%20dutch%20masters%20oil%20painting%20of%20a%20robot%20cat.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.39.06%20-%20a%20dutch%20masters%20oil%20painting%20of%20a%20robot%20cat.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.39.11%20-%20a%20dutch%20masters%20oil%20painting%20of%20a%20robot%20cat.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.39.14%20-%20a%20dutch%20masters%20oil%20painting%20of%20a%20robot%20cat.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.39.18%20-%20a%20dutch%20masters%20oil%20painting%20of%20a%20robot%20cat.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.40.20.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.40.30.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.41.03.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/DALL%C2%B7E%202022-07-07%2012.44.03.png) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/FEB26F65-0E04-413F-8DED-1D76824BA11B.jpg) ## "A red Tesla model 3 crashed on the side of the road due to it's poorly designed suspension" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2067.jpeg) ## "A kiwi walking over hot coals" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2070.jpeg) ## "A Ralph Steadman style ink painting of cats sitting on a fence smoking" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2071.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2072.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2073.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2074.jpeg) ## "Two chickens playing chess while smoking" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2078.jpeg) ## "Cats get dressed up to go to the theatre" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2084.jpeg) ## "A grumpy cat drinking black coffee and smoking a cigarette" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2085.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2086.jpeg) ## "A [female artist](https://www.charlottewatson.org/) with brown and grey hair hangs white deer antlers in an art gallery" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2092.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2093.PNG) ## "A black and white cat and a boarder collie dog in the style of Grayson Perry" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2094.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2095.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2125.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2126.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2127.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2129.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2130.jpeg) ## "A lonely cat by a window in a Ralph Steadman style" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2137.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2138.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2139.jpeg) ## "A post-apocalyptic city at night ruled by cats" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2140.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2141.jpeg) ## "A border collie dog and black and white cat in the style of Gustav Klimt" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2146.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2152.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2154.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2155.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2157.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2165.PNG) ## "DevOps hipsters deploy to the cloud" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2172.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2173.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2174.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2175.jpeg) ## "A psychedelic monkey smoking" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2181.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2183.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2184.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2185.jpeg) ## "Cats smoking cigars and playing poker" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2198.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2200.jpeg) ## "An elite hacker breaks into the computer mainframe, his eyes are red, his beard is long, he is in a dark room at his desk" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2202.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2210.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2211.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2213.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2214.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2215.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2232.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2233.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2234.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2235.jpeg) ## "Cats captaining space ships with lasers" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2241.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2242.jpeg) ## "Two lamas duelling with lightsabers at a disco" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2255.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2256.jpeg) ## "A dark, detailed oil painting of a post-apocalyptic city a night, two terminator cyborgs are patrolling the street" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2262.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2263.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2266.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2267.jpeg) ## "A photo of a beautiful British field in the summer, there are butterflys, birds and a small pond. There's a vintage car driving far in the background." ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2274.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2275.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2277.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2278.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2884.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2888.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/871A74F0-2D45-4430-BF01-3AAF94B82D51.jpg) ## "A city from Bladerunner" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2284.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2285.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2288.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2289.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2290.jpeg) ## "Modern oil painting of four Meerkats tap dancing on a stage, one of them has a red top hat and a walking cane" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2298.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2300.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2301.jpeg) ## "Postman Pat in the style of H.R. Giger" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2309.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2310.jpeg) ## "A traditional oil painting of man down on his luck, he drinks coffee in a a cafe while it rains outside" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2315.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2316.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2317.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2318.jpeg) ## "A photo of a McDonald's store designed by H.R. Giger" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2324.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2326.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2327.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2328.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2329.jpeg) ## "Three cool looking chickens drinking beer while skateboarding in the desert" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2351.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2352.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2353.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2354.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2356.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2357.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2359.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2360.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2361.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2362.jpeg) ## "A Dutch masters style oil painting of a racoon smoking a cigar and drinking sherry" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2363.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2364.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2367.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2368.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2370.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2379.jpeg) ## "An old oil painting of batman leaning on a model T ford" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2384.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2385.jpeg) ## "An oil painting of Batman playing his guitar at night" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2388.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2389.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2390.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2391.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2392.jpeg) ## "An old white man sits at home in his living room playing platstation, at his feet are his two dogs that are black and blond" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2433.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2434.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2435.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2436.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2437.jpeg) ## "A large tall man assembles a huge bookcase" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2439.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2441.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2442.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2443.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2444.jpeg) ## "An oil painting of an old dog wearing jeans sits on a red leather chair with a cat smoking a cigar and drinking wine" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2447.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2448.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2449.jpeg) ## "Man yells at toast" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2464.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2465.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2466.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2467.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2468.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2469.jpeg) ## "An armadillo turkey chimera" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2471.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2472.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2473.jpeg) ## "A man sitting on a couch who really loves pillows and chickens" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2476.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2477.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2478.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2484.PNG) ## "A dog and cat smoking a pipe while listening to records" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2486.PNG) ## "A fat cat eats a potato" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2487.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2489.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2491.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2492.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2493.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2494.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2495.jpeg) ## "An orange cut in half contains the entire universe both space and time" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2501.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2502.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2503.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2504.PNG) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2505.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2506.jpeg) ## "The pope driving a mini batmobile" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2526.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2527.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2528.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2529.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2530.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2531.jpeg) ## "A llama climbing mountains" ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2538.jpeg) ![](https://media.githubusercontent.com/media/sammcj/smcleod_files_ml/main/dalle2/IMG_2542.jpeg) --- _All images are purely comical works of art, any reference of external persons or brands is done so with artistic intent and not for profit._ - [Beginner Harmonica Practice Exercises](https://smcleod.net/2022/06/beginner-harmonica-practice-exercises/index.md): Beginner Harmonica Practice Exercises --- author: "Sam McLeod" categories: [ Music ] date: "2022-06-13T13:00:00Z" aliases: - /music/2022/06/13/beginner-harp-practice/ images: ["hohner-marine-band.jpg"] cover: image: " hohner-marine-band.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - music title: Beginner Harmonica Practice Exercises --- # Beginner Harmonica Practice Exercises Way back in around 2008 I wrote up a doc with a list of beginner harp exercises and practice riffs, at the time it was uploaded to a few forums and a shared Google doc - both have since disappeared. To this day I still get regular emails asking for a copy of it so I thought I'd copy it to a post here. Note: I haven't updated these since around 2011. ## Practice Exercises (skip forward for the riffs) ### Scales The Blues Scale ```plaintext 2 3' 4+ 4' 4 5 6+ ``` Practice jumps ```plaintext 4B 4D 4B 5B 4B 5D 4B 6B 4B 6D 4B 7D 4B 7B -2 6 -2 -5 -2 5 -2 -4 -2 4 -2 -3 -2 ``` Hans Scale ```plaintext 4B 4D 5B 5D 6B 6D 7D 7B 8D 7B 7D 6D 6B 5D 5B 4D 4B 4B 4D 5B 4D 4B 4D 5B 4D 4B 4D 5B 5D 6B 5D 5B 4D ``` Good Practice for endings (But sounds annoying!) ```plaintext 8B 8D 7B 8B 8D 7B 7D 8D 7B 6D 6B 5D 5B 4D 4B 4D 5B 4D 4B 5B 4D 4B 3D 4D 4B 5B 6B 7D 7B 7D 7B 8D ``` Triplets ```plaintext (4b) (4) (5) (4) (4b) (3) (3) (3b) (2) (2) (2b) (1) (3b) (3) (4) (4b) (4) (4b) (3) ``` Practice scaling from 4B to 10D as and even. ### Vibrato Practice vibrato on 2D and 4D (and blows) - (Try coughing lightly!) ```plaintext 2D~ 2D~ 4D~ 4D~ 2D~ 2D~ ``` ### Tone [Lee Sankey - Improve your tone - Thin vs Thick sounds](http://www.youtube.com/watch?v=HUmwmEWQvps) Think about the kind of air you're using, for example - the difference between blowing fog on a window and trying to blow a feather along a desk and don't use your hands. Thin note - focus on the lips ( think of it as being forward in the mouth system ) ```plaintext (2) ``` Thick note - focus on swallowing the note down into your system. ```plaintext (2) ``` Work on going from thin to thick on (2), then try other holes. ```plaintext (2), (2), (2), (2) ``` [Lee Sankey - Harder exercises for high low transitions](http://www.youtube.com/watch?v=T5qDjl6Xuv0) ```plaintext 4 (4) 5 (5) 6 (6) (7) 7 (7) (6) 6 (5) 5 (4) 4 ``` This found this scale very good for practising bends. ```plaintext 4 (3) (3bb) (2) (2b) 2 (1) 1 ``` Then join them, do this both ways. ```plaintext 7 (7) (6) 6 (5) 5 (4) 4 (3) (3b) (2) (2b) 2 (1) 1 ``` This is to be played slowly and carefully, over time it will start to sound nice. ```plaintext (1) 2 (2) (3bb) (3) (4) 5 6 (6) (7) (8) 8 9 (10) ``` ```plaintext (4) (5) (6) 7 (8) (9) 9 (8) 7 (6) (5) ``` ```plaintext (4) 5 6 (6) (7) (8) 8 9 (10) ``` Practice blues scales ```plaintext 2D 3D' 4B 4D' 4D 5D 6B 5D 4D 4D' 4B 3D' 2D ``` ```plaintext 2D 3D" 3D 3D" 2D 3D" 3D 3D" 2D 3D" 3D 4B 4D 4B 3D 3D" ``` I find this one quite hard: ```plaintext 2D 3D' 2D 3D' 4B 3D' 4B 4D' 4B 4D' 4D 4D' 4D 5D 4D 5D 6B 5D 6B 5D 4D 5D 4D 4D' 4D 4D' 4B 4D' 4B 3D' 4B 3D' 2D 3D' 2D ``` ### Second Position Scales Practice 2nd position scales (I find this very hard) ```plaintext 6B 6D 7D 7B 8D 8B 9D 9B 10D 9B 9D 8B 8D 7B 7D 6D - played twice, then 6B 6D 7D 6D 6B 6D 7D 6D 6B 6D 7D 7B 8D 7B 7D 6D - played twice, followed by the first phrase, played twice. ``` ### Bending Other bending practise, Start slow - as you get cleaner start trying it faster (From the song June Apple) ```plaintext 4D 4B 3D 4B 3D 3D" 2D 2D 3D" 3D 3D" 3D 4B 4D 2D 2D" 2B 2D" 2D 3D" 2D 2D" 4B 3D 4B 4D 5B 4D 4B 3D x4B 3D 2D" 2D 2D 3D" 3D 3D" 3D 4B 4D 2D 2D" 2B 2D" 2D 3D" 2D 2D" 3D" 2D ``` [Peter Gunn Bass Line](http://www.harptab.com/blues/gunn.shtml) > "This is the hardest bend practising riff that there is." This uses the 3 hole bends, Play it (if you can) until you can do it very fast. [More bending practice](http://www.youtube.com/watch?v=jF2iATfES8c) ```plaintext -1 -1d 1 -2 -2d 2 -3 -3d 3 -4 -4d 4 -5 -5d 5 -6 -6d 6 ``` Then do it in reverse, for example `1 -1d -1` etc. ```plaintext -2 -3b 4 -4b -4 5 6 ``` Then try the [following](http://www.harptabs.com/song.php?ID=334). ```plaintext -2 -3 4 -4 -5 6 6 -5 -4 -3 -2 4 4 -3 -2 4 4 -3 -2 -3 4 -4 6 -5 -5 -4 -5 4 4 4 6 6 6 -5 -5 -4 -4 4 -3 -2 -1 -1 ``` ### Speed and Breathing Pattern Practice ```plaintext 6 (6) (7) 7 (7) (6) 6 ::| Repeat endlessly! ``` ## Riffs / Licks Walter's Boogie (great repeating riff) ```plaintext 5 4 5 5+ 4+ 5+ 4 3 4 5+ 4+ 5+ ``` Trouble Riff ```plaintext 4 5 4 5 6+ ``` ```plaintext Easy Blues Riff 2 3 2 3 4+ 3 2 ``` Hoochie Coochie Man Harmonica Tab ```plaintext 1 2" 1 2" 2 ``` My take on [Modern Blues Harmonica - (Gussow.052)](http://www.youtube.com/watch?v=jLUBcCi335g) G-Harp - Make a really forward 'tuk tuk' sound with the draw chords ```plaintext (1234) (1234), 1234, (1234) (1234) (1234) (1234), 1234, (12) (12) - then back to the bar above again. ``` ```plaintext (5)(5)(5)(5)5(4) , (4)(4)(4)(4)4(3) (1234) (1234), (1234) (1234) ``` Fun [simple riff](http://www.youtube.com/watch?v=adV4eG8Kqd4) to fiddle around with. ```plaintext (1234) (1234) 4 (3) (4) (3) 3 (1234) (1234) ``` Great, really fun riff taken from [A. Gussow .24](http://www.youtube.com/watch?v=elf4CFHjH7o), You can fiddle around with this one a lot and have a lot of fun making variations up as you go. Adam plays this on a C harp. ```plaintext (2) (2b) (2), (3b) (2) (2) (2b) (1), 2 (2) (2) (2b) (2), (3b) (2) (4-4b-4)~(4-4b-4), 4 (1+3+2) ``` The `(4-4b-4)~(4-4b-4)` - sounds like wee-o-wee-o-wee, you can also add in a bit of 5 to make it `(45-45b-45)` More things to try with this riff: ```plaintext (34) 34 (23), 23 (12) - Start each d (4)(3)(4)(3)(4) 4 (3+2) - Raw with a small bend ``` Adam does something like a - 4-5-4-5-4-5-4 (3) (2) (2B) (1) (2) ```plaintext (2) (1) (1) , 2 (2) (4) 4 (3b) 3 (2) ``` [Leo - Harmonica Lesson - Lick 2](http://www.youtube.com/watch?v=0O8QFVs4cyY) ```plaintext -2 -3 -2 -3 4 -3 4 -4 4 -4 -5 -4 -5 6 ``` Im a man riff ```plaintext 3 4 3 (3) (2) ``` Common Bent Riff ```plaintext 2D 2D" 1D 2D , 2D 3D' 4B 4D ``` Simple Blues Jams ```plaintext 3 4 (~3) 3 3 4 (~3) 3 4 4 (~3) 4 3 4 (~3) 3 3 (~3) 4 (~4) 4 4 (~3) 4 {::} ``` Sammy Riff One ```plaintext 3 4 (4b->4) 4 (3b->3) 3 2 (2) 3 4 (4b->4) 4 (3b->3) 3 (1) 2 (2b)->(2)v ``` Easy Blues Riff from PlayHarmonicaEasily on YouTube ```plaintext (12) (12) 34 34 (34) (34) 34 34 (12) (12) 34 (12) ``` [Lee Sankey Backing+Riff Building](http://www.youtube.com/watch?v=ByZEWqdL93Q) Brilliant video and teacher - Really try to feel the bend, you can use 4b-3b or whatever - timing is key. ```plaintext (123) (123) 123 123 (123) , (3B) or (123) (123) 123 123 (123) , (3) (2) (3b) (2) ``` [Ronnie Shellist Blues Riff](http://www.youtube.com/watch?v=z5SODHdjZZA) ```plaintext (1-slide-3") (3) (2) 2 (1) 2 (2) 2 (2) (1-slide-3") (3) (2) 2 (1) 2 (2) 2 (123) ``` [Gussow - Best played slow on _A _ harp](http://www.youtube.com/watch?v=d-OxIr1mk3I). Release lips and let air out the top of the harp through lips on the 234 blow. ```plaintext (1) (2b) (1) (2) (123) 234 (3b) (2) (1) (2b) (1) (2) (1) (2b) (2..) ``` Gussow - Mojo Riff ```plaintext (2) (2) (2)-(2b) (1)-(2) ``` Boogie Riff (Something like this, you really have to 'feel the groove' of this one) ```plaintext (2) (2), (2) (2), (2) (2), (2) (2), (2) (3) 4 (3) (2) ``` [Little Walter Lick](http://www.youtube.com/watch?v=XSmGcg643IM) Play slurred to sound best. ```plaintext (2) (3) (4) 5 (5) (4) 5 (4) ``` [Isaacullah riff of the day](http://www.youtube.com/watch?v=J9vM9RuKyoY) ```plaintext (4) (5) (4) 4 (3b) (2) (2) (1) (2b)-(2) ``` [Catfish Blues Lick](http://www.youtube.com/watch?v=KRvHEeIu-wA) Played quite quick when you can. ```plaintext (5) (4) (4b) (3) (2) (2b) (2) (2) (2) (123) ``` Play each note twice like 'tah-tah' except for the last 4D ```plaintext 2D 3D 4D 5B 6B 5B 4D 3D ::| 4B 5B 6B 6D 7B 6D 6B 5B 2D 3D 4D 5B 6B 5B 4D 3D 4D 5B 5D 5B 4D 5B 6B 5B 2D 3D 4D 5B 5D 5B 4D ``` 12 Bar Blues Boogie ```plaintext (2) (3) (4) 5 (5) 5 (4) (3) (2) (3) (4) 5 (5) 5 (4) (3) 1 2 3 (3bb) (3b) (3bb) 3 2 (2) (3) (4) 5 (5) 5 (4) (3) (1) (2b) (3bb) (3) 1 2 3 (3bb) (2) (3) 4 (4b) (4) (4b) 4 (3) ``` Here I've used dots and spacing to emphasise the notes length (I don't know how well this works without hearing it!) - this has a swing to it. ```plaintext (2).. (2b)(1)(2b) (1)... , (2).. (2b)(1) (2).... ``` Saw waw-waw on the 6's and the second 6 is played for half the length of the first. ```plaintext (2) (3) (4) 5 6 - 6 (4) (5) (6) (7) 4 5 6 (6) (4) (4b) (3) 4 (3b) (2) (2) (3b) (3) (4) 5 (4) (3b) (2) (1) 2 (2) (3b) (3) (4) (4) (4b) (3) (2) ``` ### Harder Riffs / Licks [Adam Gussow - boogie blues lick](http://www.youtube.com/watch?v=lj8qXFxf2vo). Hit the bent 4's hard! ```plaintext (4b) (4) (5) (4b) - (3) - (2) (123) (2) , (3) (234) 345 (2) ``` [What A Woman](http://www.youtube.com/watch?v=2Q418k3rAAc) ```plaintext 4 (3+4b) 4 (3+4b) (2) 4 (3+4b) (2) (3+4b) (2) ``` [Charlie McCoy Harmonica lick](http://www.youtube.com/watch?v=KlRPPDSg80E) ```plaintext (9) (8) (8) 9 9 (8) (7) (6) ``` [Juke Intro Riff](http://www.youtube.com/watch?v=6YtMG8ehiaQ) ```plaintext 3B, 3Dbb-3D-4D, 5B, 6B, 6B (or 6-3B or 6-9B) 2D, 3Db, 3D, 4D, 5B, 6B, 6B (or 6-3B or 6-9B) - alt version ``` [Dan Gage - Box Shuffle (harder bends)](http://www.youtube.com/watch?v=7815Z-VpQSI) ```plaintext I Chord: (2), (2), (4), (4), (5), (5), 6, 6 IV Chord: 1, 1, (2), (2), (3b), (3b), 4, 4 V Chord: (1), (1), (3bb), (3bb), 4, 4, (4), (4) ``` ## Advice Other advice / things to keep in mind. 1. [The 10 Principles Of Blues Improv (Gussow)](http://www.youtube.com/watch?v=6TjC7qXHHfE) 2. Know 12 bar changes. For example: -The (1) played cross-harp is the root of what chord? (Answer is the VI chord) -The 6 is the root of what chord on the 12 bar blues? -What hole on the harp is the flat 7th? (Answer is (3b)) 3. Know the basic blues scale - make every note sound good. ```plaintext (2) (3b) 4 (4b) (4) (5) 6` ``` 4. Keep time, play in time. 5. Learn some repeated riffs, repetition with variation. 6. Call and response - keep it in mind. 7. Don't start your solo too loud. Build it up - know your power moves. 8. Simplicity is ok. 9. Practice hard, pattern and repeat the things you want to work into your improv, then forget about them and just play. 10. Think song. Think how this song is going to be different from your next song. 11. Change textures. Linear melodies - repeated riffs - one long note - warble - staccato. 12. Move out of places of safety. --- ## Unfinished The following are unfinished ideas, riffs, exercises. [Diatonic practice exercises](http://www.harptabs.com/song.php?ID=2527) This is a collection of assorted exercises that will help one get comfortable with the whole range of notes on a diatonic harmonica. ### Scales Low ```plaintext +1 -1 +2 -2'' -2 -3'' -3 +4 +4 -3 -3'' -2 -2'' +2 -1 +1 ``` Middle ```plaintext +4 -4 +5 -5 +6 -6 -7 +7 +7 -7 -6 +6 -5 +5 -4 +4 ``` High ```plaintext +7 -8 +8 -9 +9 -10 +10' +10 +10 +10' -10 +9 -9 +8 -8 +7 ``` Ex. A ```plaintext +4 +5 -4 -5 +5 +6 -5 -6 +6 -7 -6 +7 -7 -8 +7 +7 -6 -7 +6 -6 -5 +6 +5 -5 -4 +5 +4 -4 -3 +4 ``` Ex. B1 ```plaintext +4 +5 +6 +5 +4 +5 +6 +5 -4 -5 -6 -5 -4 -5 -6 -5 - (repeat 2x and end w/ +4) ``` Ex. B2 ```plaintext +1 +2 +3 +2 +1 +2 +3 +2 -1 -2'' -3'' -2'' -1 -2'' -3'' -2'' - (repeat 2x and end w/ +1) ``` Ex. B3 ```plaintext +7 +8 +9 +8 +7 +8 +9 +8 -8 -9 -10 -9 -8 -9 -10 -9 - (repeat 2x and end w/ +7) ``` Ex. C1 ```plaintext +4 +5 +6 -7 +6 +5 +4 +5 +6 -7 +6 +5 -4 -5 -6 +7 -6 -5 -4 -5 -6 +7 -6 -5 +4 ``` Ex. C2 ```plaintext +1 +2 +3 -3 +3 +2 +1 +2 +3 -3 +3 +2 -1 -2'' -3'' +4 -3'' -2'' -1 -2'' -3 +4 -3'' -2'' +1 ``` Ex. C3 ```plaintext +7 +8 +9 +10' +9 +8 +7 +8 +9 +10' +9 +8 -8 -9 -10 +10 -10 -9 -8 -9 -10 +10 -10 -9 +7 ``` Ex. D1 ```plaintext -1 -4 +4 -4 -1 -4 +4 -4 +3 +6 -5 +6 +3 +6 -5 +6 -1 -4 +4 -4 -1 -4 +4 -4 -1 ``` Ex. D2 ```plaintext -4 -8 +7 -8 -4 -8 +7 -8 +6 +9 -9 +9 +6 +9 -9 +9 -4 -8 +7 -8 -4 -8 +7 -8 -4 ``` Ex. E1 ```plaintext +4 -4 +5 +4 -4 +5 -4 +5 -5 -4 +5 -5 +5 -5 +6 +5 -5 +6 -5 +6 -6 -5 +6 -6 +6 -6 -7 +6 -6 -7 -6 -7 +7 +4 ``` Ex. E2 ```plaintext +1 -1 +2 +1 -1 +2 -1 +2 -2'' -1 +2 -2'' +2 -2'' +3 +2 -2'' +3 -2'' +3 -3'' -2'' +3 -3'' +3 -3'' -3 +3 -3'' -3 -3'' -3 +4 +1 ``` Ex. E3 ```plaintext +7 -8 +8 +7 -8 +8 -8 +8 -9 -8 +8 -9 +8 -9 +9 +8 -9 +9 -9 +9 -10 -9 +9 -10 +9 -10 +10' +9 +10 +10' +10 +10' +10 +7 ``` Ex. F Play in reverse or play only a portion and then reverse. Try this pattern on any scale: 2nd note, 1st, 3rd, 2nd, 4th, 3rd, 5th, 4th, 6th, 5th note, etc. ```plaintext -1 +1 +2 -1 -2'' +2 -2 -2'' -3'' -2 -3 -3'' +4 -3 -4 +4 +5 -4 -5 +5 +6 -5 -6 +6 -7 -6 +7 -7 -8 +7 +8 -8 -9 +8 +9 -9 -10 +9 +10' -10 +10 +10' ``` ### Rhythm Exercises Note: *=1/8 note or note played at 2x speed; **=1/16 note or note played at 4x speed. Ex. G ```plaintext (-2-3) (+1+2) (-1-2) (+1+2) (-2-3)* (-2-3)* (+1+2)* (+1+2)* (-1-2)* (-1-2)* (+1+2)* (+1+2)* (-3-4) (-3-4) (rest) (rest) (-2-3)* (-2-3)* (+1+2)* (+1+2)* (-1-2)* (-1-2)* (+1+2)* (+1+2)* ``` Ex. H ```plaintext (+1+2) (+1+2) (-2-3) (-2-3) (+1+2)* (+1+2)* (+1+2)* (+1+2)* (-2-3)* (-2-3)* (-2-3)* (-2-3)* (+1+2) (+1+2) (-2-3)* (+1+2)* (-2-3)* (+1+2)* (-2-3) (-2-3) (+1+2)* (-2-3)* (+1+2)* (-2-3)* ``` Ex. I If you tap your foot to get a steady beat, you should be playing 4 rapid notes for every tap below. ```plaintext (-1-2)** (-3-4)** (-3-4)** (-3-4)** (-1-2)** (-3-4)** (-3-4)** (-3-4)** (-1-2)** (-3-4)** (-1-2)** (-3-4)** (-1-2)** (-3-4)** (-3-4)** (-3-4)** (+1+2)** (+3+4)** (+3+4)** (+3+4)** (+1+2)** (+3+4)** (+3+4)** (+3+4)** (+1+2)** (+3+4)** (+1+2)** (+3+4)** (+1+2)** (+3+4)** (+3+4)** (+3+4)** ``` ### Bending Exercises Ex. J1 (draw 2 bends) ```plaintext -1 -2 -1 -2' -1 -2'' -1 -2' -1 -2 (repeat) ``` Ex. J2 (draw 2 bends) ```plaintext -3 -2 -3 -2' -3 -2'' -3 -2' -3 -2 (repeat) ``` Ex. K1 (draw 3 bends) ```plaintext -2 -3 -2 -3' -2 -3'' -2 -3''' -2 -3'' -2 -3' -2 -3 (repeat) ``` Ex. K2 (draw 3 bends) ```plaintext -4 -3 -4 -3' -4 -3'' -4 -3''' -4 -3'' -4 -3' -4 -3 (repeat) ``` Ex. L (note 5 draw does bend ever so slightly, but don't force it). ```plaintext -1 -1' +1 -2 -2' -2'' +2 -3 -3' -3'' -3''' +3 -4 -4' +4 -5 -5' +5 -6 -6' +6 (then play in reverse) ``` ### Octave Exercises You'll need to tongue block. Ex. M ```plaintext (+1+4) (-1-4) (+2+5) (-2''-5) (+3+6) (-3'''-6) (-3-7) (+4+7) (reverse here or continue) (-4-8) (+5+8) (-5-9) (+6+9) (-6-10) (-7) (+7+10) (repeat in reverse) ``` Ex. N ```plaintext (-1-4) (+2+5) (+3+6) (-3-7) (-4-8) (+5+8) (+6+9) (-6-10) (then play in reverse) ``` - [Firefox Addons for 2022](https://smcleod.net/2022/05/firefox-addons-for-2022/index.md): Firefox Addons for 2022 --- author: "Sam McLeod" categories: [ Software ] date: "2022-05-17T08:00:00Z" lastmod: "2022-11-07T16:52:27+11:00" aliases: - /software/2022/05/16/firefox-addons-2022/ images: ["firefox-logo-collage-1.png"] cover: image: " firefox-logo-collage-1.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - software series: [ BestOf ] title: Firefox Addons for 2022 --- ## Firefox Addons - 2022 Edition My list of must-have Firefox addons - 2022 edition Updated: 2022-11-07 ### Privacy and Security Firstly - you should have [Firefox's Enhanced Tracking Protection](https://support.mozilla.org/en-US/kb/enhanced-tracking-protection-firefox-desktop) enabled. - [Don't Track Me Google](https://addons.mozilla.org/en-GB/firefox/addon/dont-track-me-google1) - [LocalCDN](https://addons.mozilla.org/en-GB/firefox/addon/localcdn-fork-of-decentraleyes/) - [UTM Tracking Token Stripper](https://addons.mozilla.org/en-GB/firefox/addon/utm-tracking-token-stripper) - *Note: You can accomplish some of what this does by setting up the `removeparam` uBlock origin rules I've listed below.* - [Multi-Account Containers](https://addons.mozilla.org/en-GB/firefox/addon/multi-account-containers) - *Useful for setting sites such as Amazon, eBay, Twitter, LinkedIn, Banking etc... each to always open in their own isolated container.* - The official addon for whatever Password Manager you use. - [Firefox Translations](https://browser.mt/) #### uBlock Origin Probably the single most important addon. - [uBlock Origin](https://addons.mozilla.org/en-GB/firefox/addon/ublock-origin) 1. Enable and update all built-int filter lists except for 'Regions' and 'Languages'. 2. Under 'Trusted Sites' add your local IP range (e.g. `192.168.0.0/24`) and local hostnames if you run any local web servers (e.g. `https://my-home-server.local` 3. Enable 'Cloud Storage' (if you use Firefox sync) and upload your config tabs. ### Quality of Life Improvements - [Bypass Paywalls Clean](https://addons.mozilla.org/en-GB/firefox/addon/bypass-paywalls-clean) - [SponsorBlock - Skip Sponsorships on YouTube](https://addons.mozilla.org/en-GB/firefox/addon/sponsorblock) - [FastForward](https://addons.mozilla.org/en-GB/firefox/addon/fastforwardteam/) - [Copy PlainText](https://addons.mozilla.org/en-GB/firefox/addon/copy-plaintext) - [Awesome RSS](https://addons.mozilla.org/en-GB/firefox/addon/awesome-rss) - [Night Mode - Hacker News](https://addons.mozilla.org/en-GB/firefox/addon/night-mode-hacker-news) - [Upvote Anywhere](https://addons.mozilla.org/en-GB/firefox/addon/upvote-anywhere/) - [Refined Github](https://github.com/refined-github/refined-github) - [MarkDownload](https://addons.mozilla.org/en-GB/firefox/addon/markdownload/) - [FediAct](https://github.com/lartsch/FediAct) ### For AWS Users - [Console Recorder for AWS](https://addons.mozilla.org/en-GB/firefox/addon/console-recorder) - [Former2 Helper](https://addons.mozilla.org/en-GB/firefox/addon/former2-helper) ## Other Configuration - Settings -> Privacy & Security -> [Strict Tracking Protection](https://support.mozilla.org/en-US/kb/enhanced-tracking-protection-firefox-desktop) (more information [here](https://blog.privacyguides.org/2021/12/01/firefox-privacy-2021-update)) ### about:config - `mousewheel.with_meta.action: 0` - Disable scroll zooming. - `extensions.pocket.enabled: false`- Disable Pocket. - `dom.event.clipboardevents.enabled: false` - Don't allow websites to prevent copy and paste, stop webpage knowing which part of the page had been selected. ### uBlock Origin Rules Add the following to "My Filters" - The content of [ublacklist github translation](https://raw.githubusercontent.com/arosh/ublacklist-github-translation/master/uBlacklist.txt) - The content of [ublacklist stackoverflow translation](https://raw.githubusercontent.com/arosh/ublacklist-stackoverflow-translation/master/uBlacklist.txt) ``` # Remove tracking tokens *$removeparam=utm_source *$removeparam=fbclid *$removeparam=gclid *$removeparam=utm_source *$removeparam=utm_medium *$removeparam=utm_term *$removeparam=utm_campaign *$removeparam=utm_content *$removeparam=utm_cid *$removeparam=utm_reader *$removeparam=utm_referrer *$removeparam=utm_name *$removeparam=utm_social *$removeparam=utm_social *$removeparam=igshid *$removeparam=ICID *$removeparam=rb_clickid ! block pinterest rubbish on google google.*##.g:has(a[href*=".pinterest.*"]) google.*##a[href*=".pinterest."]:upward(1) ! Block pinterest rubbish on ddg duckduckgo.*##.results > div:has(a[href*=".pinterest.com"]) ``` --- ## A note on 1Password [1Password](https://addons.mozilla.org/en-GB/firefox/addon/1password-x-password-manager/) While I don't have an issue with their browser extension by itself - I can no longer recommend the desktop / 'application' version 1Password as of version 8. Since raising more capital 1Password has been chasing quick development of new features and have stopped native application development. 1Password 8 is unfortunately an Electron (Chrome) web-frame application. This has a lot of negative implications for security, performance and in itself is a statement of the direction that 1Password is taking moving forward. So - what are the alternatives? For a desktop and mobile application [Strongbox](https://strongboxsafe.com) is a leading contender, but lacks good browser integration. They are actively working on an [official Firefox addon which in beta](https://addons.mozilla.org/en-GB/firefox/addon/strongbox-autofill/) which is currently in beta. In the mean time as Strongbox uses the standard KeyPassX format you can use the [KeePassXC-Browser](https://addons.mozilla.org/en-GB/firefox/addon/keepassxc-browser/) addon to access your Strongbox database within Firefox. - [Goodbye Evernote, Hello Bear](https://smcleod.net/2021/07/goodbye-evernote-hello-bear/index.md): Goodbye Evernote, Hello Bear --- author: "Sam McLeod" categories: [ Software ] date: "2021-07-22T19:00:00Z" aliases: - /software/2021/07/22/goodbye-evernote-hello-bear/ images: ["bear_header-mac-screenshot.png"] cover: image: " bear_header-mac-screenshot.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - software title: Goodbye Evernote, Hello Bear --- Until recently I used Evernote for my notes, web clippings and document drafts, I was a subscriber going on 14 years and had over 3000 notes. However in recent years Evernote has gone downhill to the point it could be considered hostile to it's users. The native macOS app was replaced with an Electron Javascript webframe and like almost all Electron apps I've tried it was plagued by poor performance, memory leaks and aggressive idle CPU utilisation and above all else it now lacks the low latency feeling the native application had. With the release of the updated client, Evernote has removed offline note functionality - tough luck if you're travelling or don't have reliable internet access. This year I finally decided I had enough and have spent several months testing out alternatives including Bear, Notion, Ulysses, Simplenote, IA Writer and several others. ## Selection Criteria I assessed the options based on the following criteria: - Web clipper for clipping the content of blogs, forum posts, comments etc... - Offline note usage - Cross device sync - Markdown support - Code block support - Syntax highlighting would be a bonus - Attachment support - Native application performance - Low latency interface - Doesn't leak memory or churn CPU when idle - Doesn't excessively drain battery - Performant, full text internal search ## Bear I finally settled on [Bear](https://bear.app) which I'm having a great experience with and would happily recommend. It provides clean interface that really is a pleasure to spend time working in, is a native application and performs excellently and is relatively affordable at $15USD a year. ## Screenshots ![](https://github.com/sammcj/smcleod_files/blob/master/images/evernote_bear_2021/bear_header-mac-screenshot.png?raw=true) _Image courtesy of [Bear](https://bear.app)_ ![](https://github.com/sammcj/smcleod_files/blob/master/images/evernote_bear_2021/bear_feature-themes.jpg?raw=true) _Image courtesy of [Bear](https://bear.app)_ Example of a note clipped from [this](https://itnext.io/kubernetes-essential-tools-2021-def12e84c572) webpage: ![](https://github.com/sammcj/smcleod_files/blob/master/images/evernote_bear_2021/bear_web_clipper.jpg?raw=true) Example of a PDF stored inline: ![](https://github.com/sammcj/smcleod_files/blob/master/images/evernote_bear_2021/bear_pdf_attachment.jpg?raw=true) - [Rancilio Silvia Upgrade](https://smcleod.net/2021/07/rancilio-silvia-upgrade/index.md): Rancilio Silvia Upgrade --- author: "Sam McLeod" categories: [ Hardware ] date: "2021-07-11T14:00:00Z" aliases: - /hardware/2021/07/11/silvia-upgrade/ images: ["assembled_preview.jpeg"] cover: image: " assembled_preview.jpeg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - coffee - hardware title: Rancilio Silvia Upgrade --- Weekend upgrades to my Ranchilio Silvia v4 espresso machine. - Added a digital [PID](https://www.seattlecoffeegear.com/blog/2018/10/01/whats-a-pid/) [kit](https://www.jetblackespresso.com.au/shop/p/rancilio-silvia-pid-preinfusion/) for improved temperature control - Replaced the boiler element - Thermal insulation - Basic ["smart" power control](https://www.cygnett.com/products/smart-wi-fi-plug-with-power-monitoring) with Siri integration - Upgrade the standard Ranchilio basket to a [VST 18g Ridged](https://www.dukescoffee.com.au/shop/vst-precision-basket-ridged/) ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/assembled.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/siri.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/vst_18g_r.jpeg?raw=true) --- ## Rebuild Photos Note the damage to the boiler elements terminals, this was shorting and causing the houses RCD to trip ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/boiler_element_damage.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/boiler_out.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/element.jpeg?raw=true) Wiring the PID controller ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/pid_interface_wiring.jpeg?raw=true) Replacing the boiler thermostat with one connected to the PID controller ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/pid_boiler_wiring.jpeg?raw=true) Adding insulation to the boiler ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/insulation.jpeg?raw=true) --- ## Before ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/before.jpeg?raw=true) ## After ![](https://github.com/sammcj/smcleod_files/blob/master/images/silvia_upgrades_2021/assembled.jpeg?raw=true) - [DevOps Team Charter](https://smcleod.net/2020/11/devops-team-charter/index.md): DevOps Team Charter --- author: "Sam McLeod" categories: [ DevOps ] date: "2020-11-18T14:00:00Z" aliases: - /tech/2020/11/18/team-charter/ images: ["pexels-denniz-futalan-2306897.jpg"] cover: image: " pexels-denniz-futalan-2306897.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - DevOps title: DevOps Team Charter --- A template team charter for software and platform engineering teams. Research[^1] shows there is great value in people embracing a shared vision and ideas that are bigger than themselves. When it comes to a team charter, it's important to remember that it's not a static document. It's a living document that should be updated as the team evolves and grows. Keep in mind that - One size doesn't fit all. - Consider your (teams) mission, cultural background and values. - While not exhaustive - this is on the longer side, you may wish to distill as is practical, but I implore you not to only pick the easier items - that's missing the point. - These can give you talking points when performing postmortems, retros and during times of conflict. _See also: '[Using Old Ways of Thinking to Apply New Ways of Working](https://itrevolution.com/antipattern-1-2-using-old-ways-of-thinking-to-apply-new-ways-of-working/)' and '[BVSSHJ Principles](https://itrevolution.com/bvssh-principles/)'_. With all that in mind, here's a template I use when working with teams, it is not designed to be used as-is, but rather the relevant parts should be distilled and used as a concise starting point for discussion. ## Charter ### Platform Engineering - Team Charter This team charter is an introductory document that sets the vision, mission, values, culture and communication protocols for our Platform Engineering team. #### Team Vision / Mission Statement _Remember to add your vision and mission statement(s) to your teams landing page on your wiki._ "To enable, empower and add value to the business, coworkers and customers by building, running, maintaining and improving the (cloud) platforms and services including infrastructure, automation, platform integrations and CI/CD tooling used by engineers and applications." #### Team Values _Pick 5-10_ - We value focusing on outcomes rather than output. - We value working on a trust, but verify model, utilising guard rails to help protect our systems while empowering our colleagues. - We value small, easy to review, incremental changes over large, slow moving changes. - We value that perfect is often the enemy of good. - We value that rework is better than no work. - We value finishing work over starting work. - We value having empathy and respect for our peers. - We value automation over manual tasks or changes. - We value solutions that add business value over technical value. - We value simple rather than complex solutions. - We value the use of widely adopted tooling and solutions over creating bespoke development. - We value pulling work over pushing work. - We value working in small teams of 3 to 6 people. - We value measuring our assumptions. - We value fast feedback. - We value change rather than fear it. - We value open communication among the team. - We value evidence based conclusions. - We value loosely coupled, tightly integrated systems. - We value breaking down silos and barriers, there is no us and them - only us. - We value taking the sensible, applicable practices and ways of working from product delivery frameworks and cultural movements rather than trying to 'implement' them. - We value light touch interfaces, systems and processes over those that create toil (work you don't want to do). - We value sharing knowledge and have a centralised repository which can be searched by anyone in the organisation. - We value that the people are more important than the tools. - We value mistakes for the lessons they teach us. #### Team Culture and Code of Conduct _Remember to add them to your teams landing page on your wiki, and to applicable software repositories._ _Pick 3-5_ - We operate with and embrace an empathetic, blameless culture of psychological safety (everyone feels safe and no one is afraid to make mistakes, and can show and employ one's self without fear of negative consequences of self-image, status or career). - We respect each other's time (work/life balance, start / finishing hours and timezones). - We ask for help when we need it. - We support team members when they ask for help. - We operate as peers. - We assume that the intentions of our coworkers are good. - We expect that feedback should be constructive with the objective of uplifting and empowering those around us. - We recognise and celebrate all individual and team accomplishments. - We respect that people need to maintain a work / life balance in order to be healthy, happy and effective. - We work collaboratively when possible and use a consensus approach when making team decisions but accept that the team lead may act as a tiebreaker or need to make final calls. - We understand and appreciate that not all team members can have the same level of knowledge across all systems and tooling, from that we leverage that each individual adds value in different ways. - When making major architectural decisions, we document the decisions made. - When delegation is required, we agree to clearly communicate and make the delegation visible. #### Team Communication _Remember to add them to your teams landing page on your wiki._ _Pick 3-5_ - We try to listen rather than to hear. - We try to understand the views and values of others. - We value transparency so that we may learn from each other. - We have agreed on and have documented our core working and core meeting hours. - We communicate with constructive, positive feedback. - We communicate with principled negotiation rather than positional bargaining. - We communicate with direct feedback over indirect (slow) feedback. - We prepare for and arrive on time for meetings / workshops whenever possible. - We avoid holding meetings without value or without agendas. - We avoid holding meetings too early or late in the day. - We value asynchronous communication when appropriate (e.g. the message is not urgent, should be viewed by multiple people when suits them etc…). --- ### Real Work Example To quote [Grant Sutton](https://www.linkedin.com/in/grant-d-sutton/?originalSubdomain=au) from [DigIO](https://digio.com.au): > Back when I was on a team we used **"Rework is better than no work"** as one of our core principles. > > There were several times on the project where we could have been effectively stuck waiting of API specifications to be finalised. > > In those cases someone from the team would remind the other "Rework is better than no work", and we'd start working off whatever information we had, even if it was just content from developers in Slack messages. > > There were many occasions where the early development that we did either identified problems, or drove changes to the API that wouldn't have happened if we had waited. _See also: Grant's post on '[Dev Mantras - A Team Charter Anti-Pattern](https://medium.com/digio-australia/dev-mantras-a-team-charter-anti-pattern-7a2b6f8369fa)'_ [^1]: - [Ferrari F12 Berlinetta](https://smcleod.net/2020/11/ferrari-f12-berlinetta/index.md): Ferrari F12 Berlinetta --- author: "Sam McLeod" categories: [ Cars ] date: "2020-11-08T16:00:00Z" aliases: - /cars/2020/11/08/ferrari-f12/ images: ["f12b_compressed.jpg"] cover: image: " f12b_compressed.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: [photos, cars] title: Ferrari F12 Berlinetta --- Last weekend I had a chance encounter with a [Ferrari F12 Berlinetta](https://en.wikipedia.org/wiki/Ferrari_F12) and the owner kindly let me jump in and take a few photos. ## Specs - 6.3L v12 - 544 KW (740 Horsepower) - 690 Nm at 6000RPM - 1525 kg - 0-100 in 3.1s - 0-200 in 8.5s - Maximum speed: 'over 340 km/h' - Was ['The fastest road car Ferrari ever built'](https://web.archive.org/web/20121228015009/http://www.ferrari.com/english/gt_sport%20cars/gt/pages/120229-car-announcing-the-f12berlinetta-the-fastest-ferrari-ever-built.aspx) ## Photos ![](https://github.com/sammcj/smcleod_files/blob/master/images/ferrari_f12/F12B_1.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/ferrari_f12/F12B_2.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/ferrari_f12/F12B_8.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/ferrari_f12/F12B_4.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/ferrari_f12/F12B_5.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/ferrari_f12/F12B_6.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/ferrari_f12/F12B_7.jpeg?raw=true) - [The Best Of - 2020 Edition](https://smcleod.net/2020/10/the-best-of-2020-edition/index.md): The Best Of - 2020 Edition --- author: "Sam McLeod" categories: [ Tech, BestOf, Software ] date: "2020-10-10T16:00:00Z" aliases: - /tech/2020/10/10/apps-of-2020/ images: ["william-hook-unsplash.jpg"] cover: image: " william-hook-unsplash.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - bestof - software title: The Best Of - 2020 Edition series: [ BestOf ] --- Near the end of every year I note down a summary of the best apps, hardware & podcasts I've enjoyed throughout the year (and often for some time before). _This post has been superseded. You can find the latest version of this post [here]({{< ref "posts/2022-10-19-apps-of-2022" >}})._ ## Software and Services - [Plex](https://plex.tv) - [Things](https://culturedcode.com/things/) - [Pixelmator Pro](https://www.pixelmator.com/pro/) - [Reeder](http://reederapp.com) - [NextMeeting](https://apps.apple.com/us/app/next-meeting/id1017470484?mt=12) - [Bumpr](https://www.getbumpr.com) - [Fastmail](https://www.fastmail.com) - [1Password](https://1password.com) - [Calibre](http://calibre-ebook.com) - [Backblaze](https://www.backblaze.com) - [iTerm](http://iterm2.com) - [Gitlab](http://gitlab.org) - [PFsense](https://www.pfsense.org) - [Stay](https://cordlessdog.com/stay/) - [Onyx](https://www.titanium-software.fr/en/onyx.html) - [Evernote](https://evernote.com) - [Handbrake](https://handbrake.fr) - [Amphetamine](https://apps.apple.com/us/app/amphetamine/id937984704?mt=12) - [Firefox](https://www.mozilla.org/en-GB/firefox/new/) ## Software - Mobile - [Plex](https://plex.tv) - [Things](https://culturedcode.com/things/) - [Reeder](http://reederapp.com/ios) - [Overcast](https://overcast.fm) - [Evernote](https://evernote.com) - [MiniHack](https://itunes.apple.com/au/app/minihack-for-hacker-news/id631108846?mt=8) - [Alien Blue](http://www.reddit.com/r/alienblue) - [Signal](https://signal.org) - [Prompt](https://panic.com/prompt/) ## Browser Addons - [uBlock Origin](https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/) - [Dencentraleyes](https://addons.mozilla.org/en-US/firefox/addon/decentraleyes/) ## Hardware - [LG C9 OLED TV](https://www.lg.com/au/tvs/lg-OLED65C9PTA) - [Das Ultimate 4](http://www.daskeyboard.com) - [Logitech MX Master 3](https://www.logitech.com/en-au/products/mice/mx-master-3.html) - [16'' Macbook Pro](http://www.apple.com) - [iPhone 12 Pro](https://www.apple.com) - [AppleTV 4K](https://www.apple.com) - [Audioquest DragonFly Red USB DAC](https://www.audioquest.com/dacs/dragonfly/dragonfly-red) - [Ultrasone Signature Pro Headphones](https://ultrasone.com/product/ultrasone-signature-pro/?lang=en) - [Kindle Paperwhite](http://www.amazon.com.au/gp/feature.html?docId=3077740006) - [All Day Socks](http://alldaysocks.com) - [Bellroy Wallets](http://bellroy.com) ## Podcasts - [The Skeptics Guide To The Universion (SGU)](https://www.theskepticsguide.org/) - [You Are Not So Smart](https://youarenotsosmart.com/) - [Risky.Biz](https://risky.biz) - [RHLSTP](https://www.rhlstp.co.uk/) - [Louis Theroux - Grounded](https://www.bbc.co.uk/programmes/p089sfrz) - [Hiphop Saved My Life](https://podcasts.apple.com/au/podcast/hip-hop-saved-my-life-with-romesh-ranganathan/id982388481) - [Geological](http://www.geologicpodcast.com/) - [99% Invisible](https://99percentinvisible.org/) - [BBC Global News](https://www.bbc.co.uk/programmes/p02nq0gn/episodes/downloads) ## YouTube Channels - [Techmoan](https://www.youtube.com/channel/UC5I2hjZYiW9gZPVkvzM8_Cw) - [Jim Jefferies - Well I Don't Know About That!](https://www.youtube.com/playlist?list=PLVYeh_3Gx_94IWkO3rfvX9ZTgIoKgHJ9K) - [Salvatore Ganacci](https://www.youtube.com/channel/UCyseFvMP4mZVlU5iEEbAamA) - [Warped Perception](https://www.youtube.com/channel/UCFH4dWqQQOYkyJZUGT4q5pg) - [Smarter Every Day](https://www.youtube.com/c/smartereveryday) - [DTM](https://www.youtube.com/channel/UCwKq447rYMVI5dAQWMmFnfg) - [First We Feast (HotOnes)](https://www.youtube.com/channel/UCPD_bxCRGpmmeQcbe2kpPaA) - [NPR Tiny Desk Concerts](https://www.youtube.com/playlist?list=PL1B627337ED6F55F0) - [Logic](https://www.youtube.com/channel/UC4EDjfs78AhVgpBqE3WFGJg) ## The Absolute _Worst_ Software I've Experienced - 💩 Microsoft Office365 💩 - 💩 Microsoft Teams 💩 - 💩 Atlassian Jira 💩 - 💩 Atlassian Bamboo 💩 - 💩 Atlassian Confluence 💩 - 💩 McAffee Endpoint Security 💩 - 💩 Facebook 💩 - [Rest in Peace Jimothy](https://smcleod.net/2020/10/rest-in-peace-jimothy/index.md): Rest in Peace Jimothy --- author: "Sam McLeod" categories: [ cats ] date: "2020-10-08T15:20:00Z" aliases: - /jimothy/2020/10/08/rip-jimothy/ images: ["jimothy.jpg"] cover: image: " jimothy.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - photos title: Rest in Peace Jimothy --- You magnificent girl, you brought me 14 years of love and happiness ## Photos ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/1.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/2.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/3.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/4.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/5.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/6.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/8.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/9.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/10.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/11.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/12.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/13.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/16.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/17.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/18.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/19.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/20.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/21.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/22.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/23.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/24.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/25.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/26.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/27.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/28.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/29.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/30.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/31.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/32.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/33.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/34.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/35.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/36.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/37.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/38.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/rip_jimothy/39.jpeg?raw=true) - [Camels Dressed As Unicorns](https://smcleod.net/2019/08/camels-dressed-as-unicorns/index.md): Camels Dressed As Unicorns --- author: "Sam McLeod" categories: [ DevOps ] date: "2019-08-08T08:10:00Z" aliases: - /tech/2019/08/08/camels-and-unicorns/ - /tech/2019/08/08/camels-and-unicorns.html images: ["camel_corn.jpeg"] cover: image: " camel_corn.jpeg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - DevOps - career title: Camels Dressed As Unicorns --- ## Stop trying to hire with titles like 'DevOps Engineer' or 'Cloud Engineer' "DevOps is ... not a job title" "It's more of a cultural practice, like innovation, and it makes just as little sense hiring "innovation engineers" and expecting your organisation to be innovative, without also creating the culture to foster innovation." (Joel Shea when responding to this posted on [LinkedIn](https://www.linkedin.com/posts/activity-6564991913110368256-WxFP)) In most cases it's clear the organisation doesn't truly know what they want or need and likely don't understand the nuances of the aspects of engineering. The problem is - they're going to end up with a 'Camel Dressed As A Unicorn' rather than someone with deep or natural engineering understanding in the areas they most need to help see, discuss, solve problems and deliver effective and lasting solutions. From the what I've seen recently companies hiring with 'DevOps Engineer' titles - the organisation (thinks they) want / need a Developer with a little bit of Operations experience - but not a true understanding of Operations or platform engineering. I'm almost certain they're trying to appeal to the hype (in a genuine effort to find someone highly skilled) but as they're missing the point of where people's skills lay and what those skills mean - they're confusing engineers, recruiters and likely themselves further. ## DevOps is _deeply_ important Likely the most important thing to succeed at product and systems delivery - but it's not a job title. If you need Developers to work in a DevOps culture and environment - say so. If you need Operations or Platform Engineers to work in a DevOps culture and environment - say so. Staying either of those does not mutually exclude ones abilities to know 'a bit of the other side', e.g. a Developer that has worked closely with a CI/CD system's backend or Kubernetes Operations, or an Operations Engineer that has worked closely with Developers and Testers and can work with APIs: or whip up some integration scripts when needed. - [Looking For New Opportunities](https://smcleod.net/2019/08/looking-for-new-opportunities/index.md): Looking For New Opportunities --- author: "Sam McLeod" categories: [ Tech ] date: "2019-08-06T10:30:00Z" aliases: - /tech/2019/08/06/new-opportunities/ - /tech/2019/08/06/new-opportunities.html images: ["fiji_aws_prac_training_1.jpg"] cover: image: " fiji_aws_prac_training_1.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - DevOps - career title: Looking For New Opportunities --- As stated in [my announcement post](https://smcleod.net/tech/2019/06/29/leaving-infoxchange.html) - after 7~ years, I resigned from Infoxchange several months ago. I planned on taking a holiday, relaxing, decompressing, dipping my toes in some off-the-shelf tech I haven't spent a lot of time with and performing some personal growth through reading a number of books that have been sitting on my 'to read' shelf. This is exactly what I've done. --- ## I'm now open to new opportunities I'd rather find a workplace that I can add great value too and be happy to work with than rush into a new role quickly, obviously if something great turns up right away - I'll pursue it. ### Considerations - What _value_ will I bring to the organisation? - What _gaps_ or _new ventures_ is the organisation looking to hire for? - What might I _learn_ while working? - What is the _team structure_ I'd be joining (if any)? - What _technologies_ will I likely be working with? - What _tooling_ will I be required to use? - Will I have _fun_ (at least some of the time)? - Will I have a reasonable amount of _creative freedom_ in my work? - How much _bureaucracy_ will surround the role I'm in? ### My Resume / CV These days I'm generally using [my LinkedIn](https://www.linkedin.com/in/sammcj/) as a living Resume / CV. [My LinkedIn](https://www.linkedin.com/in/sammcj/) is also the best place to contact me regarding potential positions. ### A message to recruiters As soon as I changed my status to 'Open to new job opportunities.' I've been swamped with messages on LinkedIn and email. If you have a potential role for me, I've come up with a basic list of things I'd like to know before having a formal chat. It would be greatly helpful (for both of us) if you could please send through some information about the position including: - Role(s) and Responsibilities. - Team size. - _Internal_ team and company values. - Example of what a day / week or month might look like in the position (might be hard I know). - 'Must have' experience. - 'Highly desired' experience. - 'Nice to have' experience. - Salary range. - Office location and parking (e.g. CBD - no parking but close to major train station, Inner-suburb - Company car parking, Outer-suburb - on street parking). If you can't answer some of those, no problem - I'm just working on this as an initial template to get a handle on the offers coming through. #### Clarification on some common role titles and experience terms Please see my blog post - ['Stop trying to hire with titles like 'DevOps Engineer' or Cloud Engineer'](https://smcleod.net/tech/2019/08/08/camels-and-unicorns.html) - On 'DevOps' in a title: Often when I see this, the organisation is really wanting / asking for a Senior Developer with some minor Operations experience - if so - that's not me. I've worked very closely with Developers and Testers over the years, but I am skilled in platform and pipeline engineering, automation, highly-available service engineering, distributed systems engineering, platform and service performance engineering and operations team leadership. - On term 'Cloud Experience': Cloud, for me at least - means outsourced Infrastructure and Service / Service Endpoints. I've spent many years building private clouds, (mostly based on Xen Cloud Platform, XenServer Cloud, XCP-ng and KVM systems) and building out reliable, cloud-native, scalable and highly redundant services that are presented and endpoints to developers, testers or businesses directly - essentially I've been 'building private Clouds' more than I have been using 'off-the-shelf' solutions like AWS or GCP. However, I have dipped my toes in each and completed some AWS training, at the end of the day I found that it's just a change of tooling and for me - that's not a problem. However if you're looking for someone that's only worked on AWS and knows all the AWS offerings inside-and-out already - that's not me, I can or will be that person - but haven't had the need to pivot as of yet. --- ## What I've been upto (since Infoxchange) - Took a 10 day holiday in Fiji, lying on the beaches, reading, decompressing and enjoying the sunshine and chatting with the locals. - A lot of reading and research (Pitch Anything, Getting to Yes, Astrophysics for People in a Hurry, The Skeptics' Guide to the Universe, The Code of the Extraordinary Mind). - Skimmed through several AWS Practitioner and Best Practise Architecture guides (As mentioned, I've always designed and run cloud platforms - but not made heavy use other 'off the shelf' options). - Completed the Amazon AWS Practitioner Essentials training. - Caught up with fellow platform, automation engineers as well as people I see as mentors. - Visited a few meetups. --- ## Links - [My LinkedIn](https://www.linkedin.com/in/sammcj/) - [My Twitter](https://twitter.com/s_mcleod) - [My Github](https://github.com/sammcj) - [This Blog](https://smcleod.net) --- ![](https://github.com/sammcj/smcleod_files/blob/master/images/fiji_2019/fiji_selfie_1.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/fiji_2019/fiji_sunset_1.jpg?raw=true) - [Leaving Infoxchange](https://smcleod.net/2019/06/leaving-infoxchange/index.md): Leaving Infoxchange --- author: "Sam McLeod" categories: [ Tech ] date: "2019-06-29T09:00:00Z" aliases: - /tech/2019/06/29/leaving-infoxchange/ - /tech/2019/06/29/leaving-infoxchange.html images: ["road-no-car-2048.jpg"] cover: image: " road-no-car-2048.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - DevOps - career title: Leaving Infoxchange --- After 7~ years, I have resigned from Infoxchange. --- In 2012 I was hired to work with [Infoxchange](https://www.infoxchange.org) to _"shake things up"_ (in the Systems Operations team) and stabilise the hosting environment. As Team Lead of Systems Operations I was tasked with melding the team and then for us to work to provide a stable, robust, modern and scalable platform for Infoxchange developed application hosting and product delivery and in a short time, with a small budget - we did just that. This 7~ year journey was exciting, rewarding and of course at times very stressful. --- I have learnt a lot and made so many very dear friends so resigning was far from easy, however it is time to move forward to new potential challenges. I would like to give a particularly special thanks to: - Ross Williamson, my friend of many years and nothing short of a truly amazing engineer. - Alex Izmayov, for his tireless dedication and passion to learning and delivering engineering automation. - The many fantastic developers that designed, wrote and delivered the code for many of the applications we ran. - Many of the application consultants and product owners that care deeply for the clients we served. --- ## What next? - A holiday, one where I plan to spend part of my time studying, learning and dipping my toes in any relevant technologies I haven't been required to work with. - A _lot_ of reading ([see the tech and personal growth items within my Goodreads 'Want to Read' shelf for a taste](https://www.goodreads.com/review/list/5836466-sam?order=d&shelf=to-read&sort=date_added)). - AWS or GCP Operations guides / courses. I've always designed and run cloud platforms - but not really used other 'out of the box' options. - A return to more frequently visiting and engaging at meetups (Infracoders, DevOps Melbourne, Elastic Meetup). - Consultation with fellow platform, automation engineers and mentors. --- ## Initial considerations for future endeavours - Will I have _fun_ (at least some of the time)? - Will I have a reasonable amount of _creative freedom_ in my work? - How much _bureaucracy_ will surround the role I'm in? - What might I _learn_ while working? - What is the _team structure_ I'd be joining (if any)? - What _technologies_ will I likely be working with? - What _tooling_ will I be required to use? - Am I likely to make _friends_ in the position? --- ## A message to recruiters **Note: I am now [looking for new opportunities](https://smcleod.net/tech/2019/08/06/new-opportunities.html)** You play an important part in both leading people to new positions while helping organisations find the right people to help them stabilise or grow. However, until I either post that I'm actively seeking a new role or set my LinkedIn to being open to new positions - I first need time for a break and to do some reading, study and reflection. I'll update my profiles when I'm ready to start looking. --- ## Links - [My LinkedIn](https://www.linkedin.com/in/sammcj/) - [My Goodreads](https://www.goodreads.com/review/list/5836466-sam?order=d&shelf=to-read&sort=date_added) - [My Twitter](https://twitter.com/s_mcleod) - [My Github](https://github.com/sammcj) - [This Blog](https://smcleod.net) - [Goodbye XenSever - Hello XCP-ng](https://smcleod.net/2019/02/goodbye-xensever-hello-xcp-ng/index.md): Goodbye XenSever - Hello XCP-ng --- author: "Sam McLeod" categories: [ Tech ] date: "2019-02-05T13:00:00Z" aliases: - /tech/2019/02/11/hello-xcp-ng images: ["xcpng_rocket_logo.png"] cover: image: " xcpng_rocket_logo.png" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software title: Goodbye XenSever - Hello XCP-ng --- In 2018 I set out to replace our XenSever 7.2 based virtualisation after [Citrix essentially screwed over free / open source users](https://xenserver.org/blog/entry/xenserver-7-3-changes-to-the-free-edition.html). This project was to directly replace XenServer 7.2 with something supported and manageable for our traditional virtualisation needs. --- ## High Level Selection Considerations I evaluated a number of options, with the primary candidates below. Key criteria (at a high level) I was evaluating: - Ease of moving from our existing XenServer 7.2 based hypervisor clusters. - Security (architecture, hardening, monitoring, logging). - Cost (both licensing if any and self-support / management costs). - VM Performance (Storage IOPs and throughput, Network latency and throughput, Processing latency, steal from over-provisioned workloads). - Management UI/UX and performance (for BAU activities). - Management / Cluster SPOFs, fail-over and redundancy. - Installation and upgrade process. - Update and security patching frequency. - Networking design and complexity. - Community (size, engagement, acceptance of suggestions / MRs). - Reliable live VM migrations. - Ease of management for a small team (Part of a low TCO). - Risk of vendor and technology lock-in. - Risk of survival (will it still be well maintained over the next 1-3 years). - Stability and reliability above practically all else. ## [XenServer 7.6 (w/ Paid License)](https://xenserver.org/) ### For - Potentially priority support from Citrix for issues. - Easy upgrade from 7.2. - Fresh installs and upgrades are simple, painless and easy to pxeboot, licensing can be a pain after install however. ### Against - Slow moving development. - Incredibly expensive, [it would cost us something like $140,800 AUD per year (shelf price)](https://store.citrix.com/store/citrix/en_US/pd/ThemeID.37713000/productID.315979800) for our 32 hosts (each with 2 sockets). - Features heavily restricted by licensing model. - Mostly older-fashioned enterprise users. - No web management interface (although you can use Xen Orchestra from the folks behind XCP-ng). - Diminishing community since [XenServer licensing changes in 7.3](https://xenserver.org/blog/entry/xenserver-7-3-changes-to-the-free-edition.html). - Poor storage performance compared to KVM based solutions. - Many outdated packages, while kind of based on CentOS 7, there are a great deal of packages from older releases or completely custom rebuilt. - Uses old technologies like EXT3 and doesn't support SSD/Flash TRIM/DISCARD functions. - SELinux not enforcing or supported. - OpenvSwitch still uses a lot of Dom0 CPU (same as XenServer). - Dom0 often ends up limiting VM storage operations (tapdisk maxing out Dom0 CPU). - Applying updates can be painful or at least slow, requiring binary ISO files to be downloaded from Citrix. - Uses Jira for bug tracking (I just can't stand the thing, it's painfully laggy, give my GitLab or Github over it any day). ## [XCP-ng](https://xcp-ng.org/) XCP-ng is a relatively recent fork from XenServer after it was open sourced, tracking upstream but clearly prioritising modernisation and community. - [XCP-ng: Building an Open Source and Turnkey virtualisation platform - Talk/Video FOSDEM](https://www.youtube.com/watch?v=VpGC5zuLjSs) ### For - Simple and painless upgrade from XenServer 7.2. - Fresh installs and upgrades are simple, painless and easy to pxeboot. - Updates are very easy and all provided via yum. - Rapidly growing and inclusive community. - Much more modern and easy to use [forums](https://xcp-ng.org/forum/). - Open source and without licensing to worry about or restrictions on features. - Live VM migrations are not only faster, but I am yet to have a VM fail to migrate. - CentOS 7 based, and while still has many old packages (from XenServer) the developers and community are working hard to modernise these. - Much easier to submit bugs and MRs via their public repos than XenServer. - A fantastic level of transparency of direction and challenges. - You have the option of using [Xen Orchestra](https://xen-orchestra.com) has a web interface to manage XCP-ng. - mdadm, kernel (some say 'software') RAID support. - Promising support for OpenZFS. - Prioritising replacing old technologies built into XenServer, such as [moving from EXT3 to EXT4](https://xcp-ng.org/forum/topic/839/new-ext4-driver-please-test-it) and working on a more modern kernel build pipeline. - TRIM/DISCARD for iSCSI LUNs is [in the works](https://github.com/xcp-ng/xcp/issues/123). - Efficient ZSTD import/export compression. - The start of an exciting [Roadmap](https://github.com/xcp-ng/xcp/wiki/Roadmap). - (Near future) enhancements for a new storage stack (SMAPIv3). - (Near future) enhancement to include VXLAN support. - Significant [initiative into research and development](https://xcp-ng.org/blog/2018/12/19/xcp-ng-research-initiative/), partnering with several educational and scientific institutions. - [And lots of other enhancements being worked on and in the backlog](https://github.com/xcp-ng/xcp/issues?q=is%3Aopen+is%3Aissue+label%3A%22enhancement+%3Anew%3A%22). ### Against - Poor storage performance compared to KVM based solutions (same as XenServer). - SELinux not enforcing or supported (same as XenServer). - OpenvSwitch still uses a lot of Dom0 CPU (same as XenServer). - Dom0 often ends up limiting VM storage operations (tapdisk maxing out Dom0 CPU) (same as XenServer). ## [oVirt](https://www.ovirt.org/) _Note: I found myself in a bit of a love/hate relationship with oVirt, while it's performance and architecture was quite impressive, there were just too many problems with it's install and upgrade process for me to really 'trust' it for us. I do expect this to improve in the future, the community is very active and the product is moving quickly._ ### For - Solid technology that's the upstream for Redhat's RHEV. - Redhat backed. - Fantastic KVM performance that far outweighs _any_ Xen based distribution I've tested. - Decent web-based HTML5 management interface, could do with some UX but is constantly improving. - Best security model of the hypervisor distros I tested, SELinux enforced on, clear hardening and best practise shows in configuration files and defaults. ### Against - Messy, complex networking configuration. - Very complex and often unpredictable installation that often seems to fail on repeated identical installations. - Management is a floating VM which _could_ be considered a SPOF. - Very long, slow install process. - Complex multi-step upgrade / update process. - Live VM migrations have been a mixed bag with VMs randomly failing to migrate and ending up switched off but working perfectly at other times. - Could be a very time consuming migration from XenServer, but this goes for any Xen->KVM migrations (despite the various conversion tools available that I tried). ## [Proxmox](https://www.proxmox.com/en/) _Note: While Proxmox might be OK for a small deployment or home lab, I'd probably rather just run something like CentOS/Fedora with KVM and [Kimchi](https://github.com/kimchi-project/kimchi) for management._ ### For - Fantastic KVM performance that far outweighs _any_ Xen based distribution I've tested. - Easy install and upgrade (simply through apt). - Decent web interface for management, although lacking many options that must be set on the command line leading to some configuration confusion. ### Against - Design seems to be lacking some general security / hardening awareness. - SELinux not just set to permissive - but completely uninstalled. - Had a very 'home lab' feel to it's architecture, configuration management and community. - Still required paid subscription for (some) updates. - Due to the odd way Proxmox handles it's configuration, it was going to be hard to decide what should be Puppet controlled and what shouldn't. - Networking configuration is more complex to manage, partly due to some limitations with the web based management interface. --- ## Decision time As much as I would have liked to shift to a KVM based solution, I just didn't feel that oVirt was a safe move for a small team. Staying and paying for XenServer seemed insane and I do not believe that the Citrix XenServer project will live on for long other than in large locked-in organisations. Given that we wanted a simple upgrade process to something that was maintained and had a good community XCP-ng ended up being the (almost) obvious choice. ### Upgrade experience The upgrade from XenServer 7.2 to XCP-ng 7.6 couldn't have been more easy. 1. Insert XCP-ng ISO (pool master first). 2. Select Upgrade. 3. Repeat. 4. Switch to using XCP-ng Centre rather than XenCentre Of course there is a lot we manage with automation (Puppet), so I had to make some minor updates to our Puppet manifests but that happened as XenServer major releases came out regardless. #### Upgrade bugs We only hit two _minor_ problems during the upgrade, thankfully I was warned about one and the other was an easy fix: 1. A [bug in the upstream XenServer](https://bugs.xenserver.org/browse/XSO-924) which meant I had to ensure the vm parameter `platform:device_id` was set to `0001` on all VMs, otherwise they'd fail to live migrate during a pool upgrade, I did this simply ensuring it was set across all VMs before the upgrade: ``` xe vm-param-set platform:device_id=0001 uuid=b56f95fb-cb8f-86ce-5c7d-8efc2239f1db ``` 2. After upgrading from XenServer 7.2 to XCP-ng 7.6 hosts lost their iSCSI IQN shortly after their first boot. I simply grabbed their correct IQN from our [storage servers](https://smcleod.net/tech/2015/06/17/ssd-storage-cluster-diagram.html) config (in [hiera](https://wikitech.wikimedia.org/wiki/Puppet_Hiera)) and set it in XCP-ng centre for each host after they were upgraded. --- ## Three weeks with XCP-ng Over the past 2-4 weeks I've upgraded two clusters with a total of 32 physical hosts and 240~ VMs from XenServer 7.2 to XCP-ng 7.6. Obviously while the XCP-ng upgrade itself is an easy process there was a _lot_ of prior testing, peer review and then the hundreds of VMs to live migrate and juggle for clustered systems, but - it went well with no impact to our internal business services or customers. So far, compared to XenServer, XCP-ng has been painless and the transition was easy for both myself and the team that uses it daily. We have encountered no problems at all and both the product shows real progress and promise while the community has been active and supportive. --- # The Future (For us) This project was to directly replace XenServer 7.2 with something supported and manageable for our traditional virtualisation needs. It's worth mentioning that in longer term we will likely move to a mixed virtual environment. This will likely involve container host nodes being directly controlled / orchestrated by Kubernetes and some traditional hypervisor clusters (perhaps XCP-ng!) for traditional VM suited / stateful workloads. --- ## XCP-ng Links - [XCP-ng Website](https://xcp-ng.org/) - [XCP-ng Blog](https://xcp-ng.org/blog/) - [XCP-ng Roadmap](https://github.com/xcp-ng/xcp/wiki/Roadmap) - [XCP-ng Download](https://xcp-ng.org/#easy-to-install) - [XCP-ng Centre Downloads](https://github.com/xcp-ng/xenadmin/releases) - [XCP-ng Twitter](https://twitter.com/xcpng) - [Xen Orchestra](https://xen-orchestra.com) Getting help: - [XCP-ng Forums](https://xcp-ng.org/forum/) - [XCP-ng IRC](irc://irc.freenode.net/#xcp-ng) - [XCP-ng GitHub Issues](https://github.com/xcp-ng/xcp/issues) - [New Zealand Otago Trip in a 2019 Mustang GT](https://smcleod.net/2018/12/new-zealand-otago-trip-in-a-2019-mustang-gt/index.md): New Zealand Otago Trip in a 2019 Mustang GT --- author: "Sam McLeod" categories: [ Photos ] date: "2018-12-20T00:00:00Z" aliases: - /photos/2018/12/20/otago-trip-mustang-2018/ images: ["clyde_mustang_2080.jpg"] cover: image: " clyde_mustang_2080.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - cars - photos title: New Zealand Otago Trip in a 2019 Mustang GT --- In late November of 2018 I was lucky enough to get my hands on the upcoming 2019 Sport spec Mustang GT v8 to be its first driver. I'm not usually an American Muscle enthusiast, but the new 2019 GT really is quite different from its slightly older counterparts, with a new specially developed [10 speed gearbox](https://www.caranddriver.com/reviews/a24847025/2018-ford-mustang-automatic-transmission-performance/) (yes, _10_ gears - because more is better right?), tuned up suspension and so seemed like it would make for a pretty fun trip. I _might_ do a review of the car at some point, needless to say it was a lot of fun, very quick, handled surprisingly well for a big America v8, it did however suffer from very poorly designed console controls. And so I took the pony for a gallop from Christchurch, drove down South to Central Otago where my father owns [the pub](https://www.tripadvisor.com.au/Restaurant_Review-g3136888-d3370364-Reviews-Oturehua_Railway_Hotel-Oturehua_Central_Otago_Otago_Region_South_Island.html) in [Oturehua](https://www.google.com/maps/@-45.4192795,169.9300073,63966a,35y,35.19t/data=!3m1!1e3) (Ida Valley, Heart of the South Island, Middle of Nowhere). After a few nights, I drove up through [Clyde](https://www.google.com/maps/search/Clyde+dam/@-45.2746719,169.3109118,18430a,35y,37.35t/data=!3m1!1e3), [Cromwell](https://www.google.com/maps/@-45.1926831,169.2105724,29206a,35y,36.61t/data=!3m1!1e3) and finally up to [Wanaka](https://www.google.com/maps/place/Wanaka,+New+Zealand/@-44.9143868,169.1465933,37975a,35y,36.22t/data=!3m1!1e3!4m5!3m4!1s0xa9d5461db9ec2d6f:0x500ef868479c1e0!8m2!3d-44.7031813!4d169.1320981) to stay with friends, then traced my steps back over the days to follow. Below are some photos from the trip. _note: open images in a new tab to display in full resolution_ ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/trip_map.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1579.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1601.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1610.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1607.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1713.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1733.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1617.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1633.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1631.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/IMG_1649.jpeg?raw=true) --- ## Oturehua ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/maps/oturehua_3d.jpg?raw=true) ## Clyde ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/maps/clyde_3d.jpg?raw=true) ## Cromwell ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/maps/cromwell_3d.jpg?raw=true) ## Wanaka ![](https://github.com/sammcj/smcleod_files/blob/master/images/otago_mustang_2018/maps/wanaka_3d.jpg?raw=true) - [Run youtube-dl (or similar) in the background](https://smcleod.net/2018/07/run-youtube-dl-or-similar-in-the-background/index.md): Run youtube-dl (or similar) in the background --- author: "Sam McLeod" categories: [ Tech ] date: "2018-07-30T08:52:00Z" aliases: - /tech/2018/07/30/background-youtube-dl images: ["camera-background.jpg"] cover: image: " camera-background.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software title: Run youtube-dl (or similar) in the background --- I wanted an alias or function to use [`youtube-dl`](https://rg3.github.io/youtube-dl/) in the background. Looking around the web lots of people seemed to want this and most of them were banging their heads against a wall due to: - A) bash quoting - B) backgrounding dying when their terminal closed - C) passing the argument (url in my case) to the function Here's a simple function I whipped up that seems to 'just works'™: ```bash function yt { nohup youtube-dl "$1" --no-progress 2>&1 > youtube-dl-"$(date +%Y%m%d-%H%M%S)".log & } ``` And if you don't want logs, simply send the output to `/dev/null`: ```bash function yt { nohup youtube-dl "$1" --no-progress 2>&1 > /dev/null & } ``` Example: ```shell yt 'https://www.youtube.com/watch?v=fK0qwngdNgw` ``` ...and you can continue on with other work, queue up more downloads or even close the terminal session. The resulting output looks as such: ```shell ~ ls Jimmy Licks-fK0qwngdNgw.mp4 youtube-dl-20180730-085025.log ``` The log file contains the usual [`youtube-dl`](https://rg3.github.io/youtube-dl/) information: ```shell ~ cat youtube-dl-20180730-085025.log [youtube] fK0qwngdNgw: Downloading webpage [youtube] fK0qwngdNgw: Downloading video info webpage [youtube] fK0qwngdNgw: Downloading js player vflE6swsi [youtube] fK0qwngdNgw: Downloading MPD manifest [youtube] fK0qwngdNgw: Downloading MPD manifest [dashsegments] Total fragments: 8 [download] Destination: Jimmy Licks-fK0qwngdNgw.f137.mp4 [download] 100% of 15.84MiB in 00:10.04MiB/s ETA 00:0002:39 [dashsegments] Total fragments: 5 [download] Destination: Jimmy Licks-fK0qwngdNgw.f140.m4a [download] 100% of 564.58KiB in 00:01.31MiB/s ETA 00:00:12 [ffmpeg] Merging formats into "Jimmy Licks-fK0qwngdNgw.mp4" Deleting original file Jimmy Licks-fK0qwngdNgw.f137.mp4 (pass -k to keep) Deleting original file Jimmy Licks-fK0qwngdNgw.f140.m4a (pass -k to keep) ``` - [Disabling scroll-wheel zoom in Firefox](https://smcleod.net/2018/07/disabling-scroll-wheel-zoom-in-firefox/index.md): Disabling scroll-wheel zoom in Firefox --- author: "Sam McLeod" categories: [ Tech ] date: "2018-07-16T10:57:14Z" aliases: - /tech/2017/11/21/disable-firefox-scrollzoom - /tech/2018/07/16/disable-scroll-zoom-ff images: ["zoom.jpg"] cover: image: " zoom.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software title: Disabling scroll-wheel zoom in Firefox --- This feature annoys me endlessly, I end up zoomed in and out of websites all over the internet. ... But the fix is easy and there's no addons required. 1. Navigate to `about:config` (in Firefox's URL bar) 2. Change the value of the following two properties to `0`: ``` mousewheel.with_control.action mousewheel.with_meta.action ``` If you use Firefox sync and want these settings to sync between your machines, also add the following properties: Create two *new* properties both of type `boolean` and set them to `true`: ``` services.sync.prefs.sync.mousewheel.with_meta.action services.sync.prefs.sync.mousewheel.with_control.action ``` Enjoy! - [Highlands Festival of Speed 2018](https://smcleod.net/2018/04/highlands-festival-of-speed-2018/index.md): Highlands Festival of Speed 2018 --- author: "Sam McLeod" categories: [ Photos ] date: "2018-04-11T00:00:00Z" aliases: - /photos/2018/03/20/highlands images: ["v8.jpg"] cover: image: " v8.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - cars - photos title: Highlands Festival of Speed 2018 --- Last weekend I headed to New Zealand and marshalled at the [Highlands Festival of Speed](https://www.highlands.co.nz/events/highlands-festival-of-speed-2018/). > The Highlands Festival of Speed is a weekend packed with some of the best modern classic racing you will see in New Zealand. Featuring the Pre 65 Racing, Mainland Muscle, Formula Libre, South Island Porsche, OSCA, Highlands Modern Classics/Nostalgic Classics and the Highlands Sprint Series. This is a weekend of true blue Kiwi racing starring some of New Zealand's most well known racing legends. ## Videos I've supplied the car details at the bottom of this post. {{< youtubepl id="PLt6FXz8iff5hdyNdCTik8HqNhm5y9mwxH" >}} ## Photos _Note: These images are heavily compressed for the web, I have higher quality photos, just email or send me a [DM on Twitter](https://twitter.com/s_mcleod) if you want them._ Our marshalling site for the day [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2075.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2075.jpg) My father Grahame and I all geared up [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2086.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2086.jpg) The cars [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2674.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2674.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2079.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2079.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2089.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2089.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2091.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2091.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2138.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2138.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2195.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2195.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2200.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2200.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2208.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2208.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2210.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2210.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2219.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2219.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2226.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2226.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2234.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2234.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2243.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2243.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2251.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2251.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2271.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2271.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2276.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2276.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2282.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2282.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2284.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2284.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2289.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2289.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2292.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2292.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2297.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2297.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2300.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2300.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2308.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2308.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2316.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2316.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2319.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2319.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2337.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2337.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2346.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2346.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2350.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2350.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2354.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2354.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2359.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2359.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2364.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2364.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2371.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2371.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2373.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2373.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2383.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2383.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2384.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2384.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2385.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2385.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2392.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2392.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2412.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2412.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2418.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2418.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2425.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2425.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2435.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2435.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2456.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2456.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2481.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2481.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2486.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2486.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2527.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2527.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2528.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2528.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2533.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2533.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2565.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2565.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2591.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2591.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2594.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2594.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2598.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2598.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2603.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2603.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2606.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2606.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2612.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2612.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2619.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2619.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2627.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2627.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2632.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2632.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2636.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2636.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2639.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2639.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2643.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2643.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2649.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2649.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2654.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2654.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2658.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2658.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2660.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2660.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2662.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2662.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2683.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2683.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2690.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2690.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2721.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2721.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2723.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2723.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2725.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2725.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2729.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2729.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2741.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2741.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2757.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2757.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2782.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2782.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2795.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2795.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2826.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2826.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2847.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2847.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2874.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2874.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2883.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2883.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2911.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2911.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2915.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2915.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2929.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2929.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2934.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2934.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2942.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2942.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2958.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2958.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2969.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2969.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2982.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2982.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_3007.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_3007.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_3019.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_3019.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_3024.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_3024.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_3026.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_3026.jpg)
## Car Numbers and Engine Sizes [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2120.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2120.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2121.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2121.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2122.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2122.jpg) [![](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos_small/IMG_2123.jpg)](https://raw.githubusercontent.com/sammcj/highlands-festival-of-speed/master/photos/IMG_2123.jpg) - [Flash Storage and SSD Failure Rate Update (March 2018)](https://smcleod.net/2018/03/flash-storage-and-ssd-failure-rate-update-march-2018/index.md): Flash Storage and SSD Failure Rate Update (March 2018) --- author: "Sam McLeod" categories: [ Tech ] date: "2018-03-20T00:00:00Z" aliases: - /tech/2018/03/20/oss-flash-storage-update images: ["sam-standing-at-rack.jpg"] cover: image: " sam-standing-at-rack.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware title: Flash Storage and SSD Failure Rate Update (March 2018) --- It was almost 3 years ago that my open source storage project went into production. In that time it's been running 24/7 serving as highly available solid state storage for hundreds of VMs and several virtualisation clusters across our two main sites. I'm happy to report that the clusters have been operating very successfully since their conception. Since moving away from proprietary 'black box' vendor SANs, we haven't had a single SAN issue, storage outage. We've also been so please with the success of the design that we've grown from a single cluster at each site to a total of 6 clusters (12 nodes). ## Drive Failure Rates TLDR; Mid-range SSDs from reputable brands (SanDisk, Crucial / Micron and Intel) are *very* reliable, far more so that the highly overpriced 'enterprise' rotational drives from HP and friends. ### Supermicro DOMs (OS Mirror) We found that the [Supermicro DOM](https://www.supermicro.com/products/nfo/SATADOM.cfm) disks have been quite unreliable, Supermicro acknowledged the problem and issued an updated hardware revision. ![](dom.jpg) We haven't yet had a single updated v1.1 revision DOM fail on us, but they do have one downside: they're slightly smaller in size, likely due to additional under provisioning on the NAND flash. This means that you can't simply swap like-for-like and expect your local drive mirrors to rebuild as there isn't enough space on the newly installed v1.1 DOM, an easy workaround for this which we've applied across the board is to increase to the next size units - our 16GB DOMs have been replaced with 32GB. ### SATA SSDs We haven't had many SATA SSDs fail at all to date out of the total 208 SATA SSDs we have installed I count that we've only had 5 failures, 1 of which was DOA. #### Failed SATA SSDs * 1x SanDisk Extreme Pro 480GB * 2x SanDisk Extreme Pro 960GB * 1x Micron M600 1TB * 1x Crucial MX500 1TB (DOA) ### NVMe SSDs We've had no NVMe SSDs fail out of the 8x Intel DC3600 series we have deployed. ![](sans.jpg) - [Theory of constraints](https://smcleod.net/2018/01/theory-of-constraints/index.md): Theory of constraints --- author: "Sam McLeod" categories: [ Tech ] date: "2018-01-17T00:00:00Z" aliases: - /tech/2018/02/05/theory-of-constraints images: ["samuel-zeller-237359.jpg"] cover: image: " samuel-zeller-237359.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - DevOps title: Theory of constraints --- ### "A chain is no stronger than its weakest link" - Any improvement made anywhere besides the bottleneck is an illusion. - Any improvement made after the bottleneck is useless because it will always remain starved waiting for work from the bottleneck. - Any improvement made before the bottleneck merely results in more 'work' piling up at the bottleneck. 1. _Identify_ the system's constraint(s) (that which prevents the organisation from obtaining more of the goal in a unit of time) 2. Decide how to _exploit_ the system's constraint(s) (how to get the most out of the constraint) 3. Subordinate everything else to the above decision (_align_ the whole system or organization to support the decision made above) 4. _Elevate_ the system's constraint(s) (make other major changes needed to increase the constraint's capacity) 5. Warning! If in the previous steps a constraint has been broken, go back to step 1, but do not allow inertia to cause a system's constraint. --- ### References - [The Phoenix Project](http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592) - [Wikipedia - Theory of Constraints](http://en.wikipedia.org/wiki/Theory_of_constraints) - [Top album picks for 2017](https://smcleod.net/2017/12/top-album-picks-for-2017/index.md): Top album picks for 2017 --- author: "Sam McLeod" categories: [ Music ] date: "2017-12-28T00:00:00Z" aliases: - /music/2017/12/28/album-picks-2017/ images: ["backdrop-linn.jpg"] cover: image: " backdrop-linn.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - music title: Top album picks for 2017 series: [ BestOf ] --- Here are my top album pics for 2017 (in no particular order) _Note: This is by no means an exhaustive list, it's just the top albums that really stood out to me and in all fairness, I've thought of several others since so there may be a follow up post (or two)._ ### Father John Misty - [Pure Comedy](https://en.wikipedia.org/wiki/Pure_Comedy) ![](https://github.com/sammcj/smcleod_files/blob/master/images/albums_2017/FJM_PureComedy_Covers.gif?raw=true) As a bonus, a short film was released with the album: {{< youtube "cejjqC1oyQM" >}} From [Wikipedia](https://en.wikipedia.org/wiki/Pure_Comedy): > Most of Pure Comedy was written in 2015. It touches on themes of progress, technology, fame, the environment, politics, aging, social media, human nature, human connection and his own role in it all. Tillman included an 1800-word-long essay about its symbolism and meaning in the release announcement email to his fan club. >> Pure Comedy is the story of a species born with a half-formed brain. The species' only hope for survival, finding itself on a cruel, unpredictable rock surrounded by other species who seem far more adept at this whole thing (and to whom they are delicious), is the reliance on other, slightly older, half-formed brains. This reliance takes on a few different names as their story unfolds, like "love," "culture," "family," etc. Over time, and as their brains prove to be remarkably good at inventing meaning where there is none, the species becomes the purveyor of increasingly bizarre and sophisticated ironies. These ironies are designed to help cope with the species' loathsome vulnerability and to try and reconcile how disproportionate their imagination is to the monotony of their existence. ### Ibeyi - [Ash](http://www.ibeyi.fr/) ![](https://github.com/sammcj/smcleod_files/blob/master/images/albums_2017/Ibeyi-Ash-Cover.jpg?raw=true) From [Wikipedia](https://en.wikipedia.org/wiki/Ibeyi): > Ibeyi is a French-Cuban musical duo consisting of twin sisters Lisa-Kaindé Diaz and Naomi Diaz (both 22 years old). The duo sings in English, French, Spanish and Yoruba, a Nigerian language spoken in West Africa and spoken by their ancestors before being taken to Cuba as slaves in the 1700s. Lisa is the lead singer and plays the piano; Naomi plays traditional Peruvian/Cuban percussion instruments cajón and Batá drum. Their music has elements of Yoruba, French and Afro-Cuban, and fuses jazz with beats, samples with traditional instruments. ### Wu-Tang Clan - [The Saga Continues](https://en.wikipedia.org/wiki/The_Saga_Continues_(Wu-Tang_Clan_album)) ![](https://github.com/sammcj/smcleod_files/blob/master/images/albums_2017/Wu-Tang_The_Saga_Continues.jpg?raw=true) From [Wikipedia](https://en.wikipedia.org/wiki/The_Saga_Continues_(Wu-Tang_Clan_album)): > The Saga Continues is the seventh studio album by Wu-Tang Clan, produced by long-time producer Mathematics released on October 13, 2017 on eOne. The Saga Continues was mixed by Josh Gannet. The group name was shortened to "Wu-Tang" to mark the fact that the album features all Wu-Tang Clan members except U-God due to his legal issues with the group over royalties. It also features guest appearance from Streetlife, Redman, Sean Price and others. Producer Mathematics has explained, "It's a Wu-Tang record of course, [but] it can't be a complete Wu-Tang Clan album without [U-God]." ### Flotation Toy Warning - [The Machine That Made Us](https://en.wikipedia.org/wiki/The_Machine_That_Made_Us) ![](https://github.com/sammcj/smcleod_files/blob/master/images/albums_2017/tmtmu.jpg?raw=true) From [Wikipedia](https://en.wikipedia.org/wiki/The_Machine_That_Made_Us): > Flotation Toy Warning is a band from London, UK, formed in 2001. Their sound has been described as a cross between space rock and chamber pop, with musical similarities to bands such as Mercury Rev, The Flaming Lips, Grandaddy, Sparklehorse, The Unicorns, and Air. The band is made up of frontman and lead vocalist Paul Carter, guitar and bass guitar players Ben Clay and Nainesh Shah, drummer Steve Swindon, and keyboardist Vicky West. Available on [Bandcamp](https://talitres.bandcamp.com/album/the-machine-that-made-us) ### The Weather Station - [The Weather Station](http://www.theweatherstation.net/) ![](https://github.com/sammcj/smcleod_files/blob/master/images/albums_2017/the-weather-station.jpg?raw=true) From [Wikipedia](https://en.wikipedia.org/wiki/The_Weather_Station) _(Note: somewhat outdated information)_ > The Weather Station is a Canadian folk music band fronted by Tamara Lindeman, formed in 2006. The band membership has changed over the years, but now includes Lindeman, with Ben Whiteley on bass, Adrian Cook on pedal steel, and Ian Kehoe on drums. ### The War On Drugs - [A Deeper Understanding](https://en.wikipedia.org/wiki/A_Deeper_Understanding) ![](https://github.com/sammcj/smcleod_files/blob/master/images/albums_2017/A_Deeper_Understanding.jpg?raw=true) From [Wikipedia](https://en.wikipedia.org/wiki/A_Deeper_Understanding): > The War on Drugs is an American indie rock band from Philadelphia, Pennsylvania, formed in 2005. The band consists of Adam Granduciel (lyrics, vocals, guitar), David Hartley (bass), Robbie Bennett (keyboards), Charlie Hall (drums), Jon Natchez (saxophone, keyboards) and Anthony LaMarca (guitar). - [HP 4951C Protocol Analyser](https://smcleod.net/2017/12/hp-4951c-protocol-analyser/index.md): HP 4951C Protocol Analyser --- author: "Sam McLeod" categories: [ Tech ] date: "2017-12-27T00:00:00Z" aliases: - /tech/2017/12/27/hp-4951 images: ["backdrop-hp4951.jpg"] cover: image: " backdrop-hp4951.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - hardware photos: true title: HP 4951C Protocol Analyser --- My good friend Joel Shea received a most unlikely gift this Christmas - A vintage HP 4951 Protocol Analyser. According to the [HP Computer Museum](http://www.hpmuseum.net/display_item.php?hw=1123): > Original Price: $3595 > The 4951B was replaced by the 4951C and 4952A in 1986. Both new models handled Async, BSC, SDLC, HDLC, X.25 and SNA protocols. The 4951C also handled DDCMP, while the 4952A did not. The 4952A handled X.21 while the 4951C did not. Both new analysers used a floppy dive (618 KB) for removable media. Promo Photo: ![](https://github.com/sammcj/smcleod_files/blob/master/images/hp_4951/4951_1988-PromoPhoto-45.jpg?raw=true) Joel's specimen: ![](https://github.com/sammcj/smcleod_files/blob/master/images/hp_4951/hp4951-1.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/hp_4951/hp4951-2.jpeg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/hp_4951/hp4951-3.jpeg?raw=true) And the original spec sheet: ![](https://github.com/sammcj/smcleod_files/blob/master/images/hp_4951/4951C-spec.jpeg?raw=true) - [Talk - Clustered, Distributed File and Volume Storage with GlusterFS](https://smcleod.net/2017/11/talk-clustered-distributed-file-and-volume-storage-with-glusterfs/index.md): Talk - Clustered, Distributed File and Volume Storage with GlusterFS --- author: "Sam McLeod" categories: [ Tech ] date: "2017-11-14T00:00:00Z" aliases: - /tech/2017/11/14/getting-started-with-gluster images: ["backdrop-plastic-containers.jpg"] cover: image: " backdrop-plastic-containers.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - talk - slides title: Talk - Clustered, Distributed File and Volume Storage with GlusterFS --- Using GlusterFS to provide volume storage to Kubernetes as a replacement for our existing file and static content hosting. This talk was given at [Infracoders](https://www.meetup.com/Infrastructure-Coders/events/244535588/) on Tuesday 14th November 2017. Click below to view slides (PDF version): [![Click to view slides]({{< ref "/" >}}/img/gluster-first-slide.jpg)](https://media.githubusercontent.com/media/sammcj/smcleod_files/refs/heads/master/slides/lightning_san.pdf) - [*Direct download link*](https://media.githubusercontent.com/media/sammcj/smcleod_files/refs/heads/master/slides/lightning_san.pdf) --- ![](https://github.com/sammcj/smcleod_files/blob/master/images/gluster_talk_14112017/2.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/gluster_talk_14112017/3.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/gluster_talk_14112017/4.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/gluster_talk_14112017/5.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/gluster_talk_14112017/6.jpg?raw=true) - [Applying syntax in Sublime based on the first file line](https://smcleod.net/2017/10/applying-syntax-in-sublime-based-on-the-first-file-line/index.md): Applying syntax in Sublime based on the first file line --- author: "Sam McLeod" categories: [ Tech ] date: "2017-10-25T00:00:00Z" aliases: - /tech/2017/10/25/sublime-syntax-topline images: ["backdrop-code.jpg"] cover: image: " backdrop-code.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech title: Applying syntax in Sublime based on the first file line --- In vim, you can add a comment at the top of files to set the syntax, e.g.: ``` shell # vim: syntax=ruby ``` In SublimeText there are _many_ ways to detect syntax, one interesting approach I've recently found useful is to match on the top line in the file. For example, with Puppet there is a file called `Puppetfile`, it has no extension but it's really Ruby syntax, so it's useful to add linting incase you miss something simple like a `,` and break deployments. I use a plugin called [ApplySyntax](https://facelessuser.github.io/ApplySyntax/) making it easy to apply syntax options to files, I believe you can do this in the languages syntax without the plugin but YMMV: ``` yaml { ... // Put your custom syntax rules here: "syntaxes": [ { "syntax": [ "Ruby/Ruby" ], "rules": [ { "first_line": ".*syntax=ruby.*" }, # To match on the first line in the file { "file_path": "^Puppetfile$"} # Or match on the file name or path itself ] } ] ... } ``` - [Broadcom, Or How I Learned To Start Worrying And Drop The Packet](https://smcleod.net/2017/10/broadcom-or-how-i-learned-to-start-worrying-and-drop-the-packet/index.md): Broadcom, Or How I Learned To Start Worrying And Drop The Packet --- author: "Sam McLeod" categories: [ Tech ] date: "2017-10-13T00:00:00Z" aliases: - /tech/2017/10/13/Drop-Broadcom images: ["office-space-broadcom.jpg"] cover: image: " office-space-broadcom.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - networking - hardware title: Broadcom, Or How I Learned To Start Worrying And Drop The Packet --- ![](office-space-broadcom.jpg) Earlier this week we started the process to upgrade one of our hypervisor compute clusters when we encountered a rather painful bug with HP's Broadcom NIC chipsets. We were part way through a routine rolling pool upgrade of our hypervisor (XenServer) cluster when we observed unexpected and intermittent loss of connectivity between several VMs, then entire XenServer hosts. The problems appeared to impact hosts that hadn't yet upgraded to XenServer 7.2. We now attribute this to a symptom of extreme packet loss between the hosts in the pool and thanks to buggy firmware from Broadcom and HP. We were aware of the [recently published issues](http://www.thevirtualist.org/bricked-qlogic-broadcom-bcm57840-driver-update/) with Broadcom/HP NICs used in VMware clusters where NICs would be [bricked by a firmware upgrade](https://h20566.www2.hpe.com/hpsc/doc/public/display?docId=a00027033en_us). This issue is different from what we experienced. We experienced extreme packet loss between hosts in the cluster. With XenServer, the pool master must be upgraded first. The result was that XAPI pool management suffered a communication breakdown across the management network and complicated diagnosis. In fact, the connectivity problems went unnoticed until many hours after the master was upgraded. At first appeared as if it was a problem caused by the pool being partially upgraded. We wondered if we had perhaps made a poor decision to run the upgrade on a single node for a few hours to observe its performance. We made the call to upgrade another host and analyse our findings. The next upgraded hosts appeared stable. In fact we later found this host wasn't impacted by the bug. We then made the call to upgrade several more nodes and continue to track their stability. After upgrading half the pool, we suddenly hit problems. VMs failed, Hosts started dropping out of the pool and losing track of the power state of running/stopped VMs. We found that the master along with one of the other hosts were experiencing major packet loss on their management network cards. We suspected faulty NICs as it wouldn't be the first time a Broadcom had failed us and there is no physical network cabling. Broadcom has had its fair share of bad press over the years. Many botched firmware updates and proprietary driver issues. I'm recommending people to stay clear from using network cards based on their chipsets. ## Downgrading The Firmware As soon as we spotted the packet loss on the Broadcom NICs we upgraded their firmware to [2.19.22-1](http://downloads.linux.hpe.com/SDR/repo/spp/RHEL/7/x86_64/current/firmware-nic-qlogic-nx2-2.19.22-1.1.x86_64.rpm) with no improvement. We then upgraded to [2.18.44-1 / 7.14.62](http://downloads.linux.hpe.com/SDR/repo/spp/RHEL/7/x86_64/current/hp-firmware-nic-qlogic-nx2-2.18.44-1.1.x86_64.rpm) again with no improvement. We even went as far as trying [2.16.20 / 7.12.83](http://downloads.linux.hpe.com/SDR/repo/spp/RHEL/7/x86_64/current/hp-firmware-nic-qlogic-nx2-2.16.20-1.1.x86_64.rpm) from back in 2015 - but still no luck. At the time of writing this no firmware downgrades (or upgrades) have fixed the issue. The packet loss manifests itself immediately after rebooting or power cycling. But - _not on every reboot!_. This is the odd thing - approximately half the time when booting a host it is fine until the next boot. We've compared the `dmesg`, `lspci` and `modinfo` output between boot cycles, we can't find anything that stands out. The bug seems to be caused by the version of the `bnx2x` driver present in XenServer 7.2's Kernel. Upon further reading HP recommends that you use bnx2x driver 7.14.29-2 or later, XenServer still uses the old Kernel version of 4.4.0 - that's not currently an option. I suspect that it's a bug in the Broadcom firmware loaded into the NIC upon boot. I suspect a race condition related to the devices interrupt handling (MSI/MSI-X). ## XenServer XenServer needs to update its kernel or at least the bnx2x driver module past the version that triggers the bug. I've logged a ticket for this over at [bugs.xenserver.org](https://bugs.xenserver.org/browse/XSO-808) Additionally, XenServer didn't notice the packet loss/network interruptions during the rolling pool upgrade. I have reported this concern and have suggested that XenServer adds pool wide checks for connectivity issues between hosts, at _least_ during a pool upgrade. ## Workaround We don't have (a good) one. Currently we're simply testing for packet loss after boot on the management NIC. If detected we reboot the host and check again. This far from ideal - but until the bug is resolved there isn't any other fix that we can find short of compiling a custom module for XenServer 7.2. Given the widespread problems with Broadcom, we've ordered HP 560M, Intel based NICs to replace them. ## BNX2X Driver The driver included with XenServer 7.2 that triggers the problem is `1.714.1`: ``` filename: /lib/modules/4.4.0+10/updates/bnx2x.ko version: 1.714.1 license: GPL #description: QLogic BCM57710/57711/57711E/57712/57712_MF/57800/57800_MF/57810/57810_MF/57840/57840_MF Driver author: Eliezer Tamir srcversion: 927337210F53311B18D0D7E alias: pci:v000014E4d0000163Fsv*sd*bc*sc*i* alias: pci:v000014E4d0000163Esv*sd*bc*sc*i* alias: pci:v000014E4d0000163Dsv*sd*bc*sc*i* alias: pci:v00001077d000016ADsv*sd*bc*sc*i* alias: pci:v000014E4d000016ADsv*sd*bc*sc*i* alias: pci:v00001077d000016A4sv*sd*bc*sc*i* alias: pci:v000014E4d000016A4sv*sd*bc*sc*i* alias: pci:v000014E4d000016ABsv*sd*bc*sc*i* alias: pci:v000014E4d000016AFsv*sd*bc*sc*i* alias: pci:v000014E4d000016A2sv*sd*bc*sc*i* alias: pci:v00001077d000016A1sv*sd*bc*sc*i* alias: pci:v000014E4d000016A1sv*sd*bc*sc*i* alias: pci:v000014E4d0000168Dsv*sd*bc*sc*i* alias: pci:v000014E4d000016AEsv*sd*bc*sc*i* alias: pci:v000014E4d0000168Esv*sd*bc*sc*i* alias: pci:v000014E4d000016A9sv*sd*bc*sc*i* alias: pci:v000014E4d000016A5sv*sd*bc*sc*i* alias: pci:v000014E4d0000168Asv*sd*bc*sc*i* alias: pci:v000014E4d0000166Fsv*sd*bc*sc*i* alias: pci:v000014E4d00001663sv*sd*bc*sc*i* alias: pci:v000014E4d00001662sv*sd*bc*sc*i* alias: pci:v000014E4d00001650sv*sd*bc*sc*i* alias: pci:v000014E4d0000164Fsv*sd*bc*sc*i* alias: pci:v000014E4d0000164Esv*sd*bc*sc*i* depends: mdio,libcrc32c,ptp,vxlan vermagic: 4.4.0+10 SMP mod_unload modversions parm: pri_map: Priority to HW queue mapping (uint) parm: num_queues: Set number of queues (default is as a number of CPUs) (int) parm: disable_iscsi_ooo: Disable iSCSI OOO support (uint) parm: disable_tpa: Disable the TPA (LRO) feature (uint) parm: int_mode: Force interrupt mode other than MSI-X (1 INT#x; 2 MSI) (uint) parm: dropless_fc: Pause on exhausted host ring (uint) parm: poll: Use polling (for debug) (uint) parm: mrrs: Force Max Read Req Size (0..3) (for debug) (int) parm: debug: Default debug msglevel (uint) parm: num_vfs: Number of supported virtual functions (0 means SR-IOV is disabled) (uint) parm: autogreeen: Set autoGrEEEn (0:HW default; 1:force on; 2:force off) (uint) parm: native_eee:int parm: eee:set EEE Tx LPI timer with this value; 0: HW default; -1: Force disable EEE. parm: tx_switching: Enable tx-switching (uint) ``` Whereas XenServer 7.0 has driver version `1.713.04` which seems not to trigger the issue: ``` filename: /lib/modules/3.10.0+10/extra/bnx2x.ko version: 1.713.04 license: GPL #description: QLogic BCM57710/57711/57711E/57712/57712_MF/57800/57800_MF/57810/57810_MF/57840/57840_MF Driver author: Eliezer Tamir srcversion: 13EAA521200A40118055D63 alias: pci:v000014E4d0000163Fsv*sd*bc*sc*i* alias: pci:v000014E4d0000163Esv*sd*bc*sc*i* alias: pci:v000014E4d0000163Dsv*sd*bc*sc*i* alias: pci:v00001077d000016ADsv*sd*bc*sc*i* alias: pci:v000014E4d000016ADsv*sd*bc*sc*i* alias: pci:v00001077d000016A4sv*sd*bc*sc*i* alias: pci:v000014E4d000016A4sv*sd*bc*sc*i* alias: pci:v000014E4d000016ABsv*sd*bc*sc*i* alias: pci:v000014E4d000016AFsv*sd*bc*sc*i* alias: pci:v000014E4d000016A2sv*sd*bc*sc*i* alias: pci:v00001077d000016A1sv*sd*bc*sc*i* alias: pci:v000014E4d000016A1sv*sd*bc*sc*i* alias: pci:v000014E4d0000168Dsv*sd*bc*sc*i* alias: pci:v000014E4d000016AEsv*sd*bc*sc*i* alias: pci:v000014E4d0000168Esv*sd*bc*sc*i* alias: pci:v000014E4d000016A9sv*sd*bc*sc*i* alias: pci:v000014E4d000016A5sv*sd*bc*sc*i* alias: pci:v000014E4d0000168Asv*sd*bc*sc*i* alias: pci:v000014E4d0000166Fsv*sd*bc*sc*i* alias: pci:v000014E4d00001663sv*sd*bc*sc*i* alias: pci:v000014E4d00001662sv*sd*bc*sc*i* alias: pci:v000014E4d00001650sv*sd*bc*sc*i* alias: pci:v000014E4d0000164Fsv*sd*bc*sc*i* alias: pci:v000014E4d0000164Esv*sd*bc*sc*i* depends: mdio,libcrc32c,ptp vermagic: 3.10.0+10 SMP mod_unload modversions parm: pri_map: Priority to HW queue mapping (uint) parm: num_queues: Set number of queues (default is as a number of CPUs) (int) parm: disable_iscsi_ooo: Disable iSCSI OOO support (uint) parm: disable_tpa: Disable the TPA (LRO) feature (uint) parm: int_mode: Force interrupt mode other than MSI-X (1 INT#x; 2 MSI) (uint) parm: dropless_fc: Pause on exhausted host ring (uint) parm: poll: Use polling (for debug) (uint) parm: mrrs: Force Max Read Req Size (0..3) (for debug) (int) parm: debug: Default debug msglevel (uint) parm: num_vfs: Number of supported virtual functions (0 means SR-IOV is disabled) (uint) parm: autogreeen: Set autoGrEEEn (0:HW default; 1:force on; 2:force off) (uint) parm: native_eee:int parm: eee:set EEE Tx LPI timer with this value; 0: HW default; -1: Force disable EEE. parm: tx_switching: Enable tx-switching (uint) ``` ## Affected Components * [HP 530M Network cards](https://www.hpe.com/h20195/v2/getpdf.aspx/c04111538.pdf?ver=3) (as they use the Broadcom bcm57810 chipset), commonly found in BL460c Gen8 blades and similar. * XenServer 7.2 (Patched to the latest XS72E006 patch) * Kernel 4.4.0+10 as found in XenServer 7.2 * Broadcom bnx2x module version 1.714.1 * [HP firmware for qlogic nx2](http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3bc2b88453424d87b7543d6459) (seemingly all versions) ## Links * [Broadcom, Die Mutha](http://blog.serverfault.com/post/broadcom-die-mutha/) * [Bricked QLogic Broadcom BCM57840 after driver update](http://www.thevirtualist.org/bricked-qlogic-broadcom-bcm57840-driver-update/) * [HP Flex-10 10Gb 2-port 530M Adapter](https://www.hpe.com/h20195/v2/getpdf.aspx/c04111538.pdf?ver=3) * [HPE Network Adapters - Updating The BNX2X Driver Package Version 2.713.30 On VMware Hosts With Certain Network Adapters Running Certain Firmware May Require A Network Adapter Replacement](https://h20566.www2.hpe.com/hpsc/doc/public/display?docId=a00027033en_us) * [HP Qlogic NX2 Firmware](http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_19bc1fb428d4400a90d59a7175#tab-history) - [GlusterFS](https://smcleod.net/2017/09/glusterfs/index.md): GlusterFS --- author: "Sam McLeod" categories: [ Tech ] date: "2017-09-25T00:00:00Z" aliases: - /tech/2017/09/25/GlusterFS-3-12-1 images: ["backdrop-insidebuilding.jpg"] cover: image: " backdrop-insidebuilding.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - software title: GlusterFS --- We're in the process of shifting from using our custom 'glue' for orchestrating Docker deployments to Kubernetes, When we first deployed Docker to replace LXC and our legacy Puppet-heavy application configuration and deployment systems there really wasn't any existing tool to manage this, thus we rolled our own, mainly a few Ruby scripts combined with a Puppet / Hiera / Mcollective driven workflow. The main objective is to replace our legacy NFS file servers used to host uploads / attachments and static files for our web applications, while NFS(v4) performance is adequate, it is a clear single point of failure and of course, there are the age old stale mount problems should network interruptions occur. I spent time evaluating various cluster filesystems / network block storage and the two that stood out were Ceph and Gluster and settled on Gluster as the most suitable for our needs, it's far less complex to deploy than Ceph, it has less moving pieces and files are stored in a familiar manner on hosts. ## Implementation I've settled on a 3 node deployment with one node as an [arbiter](http://docs.gluster.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/) (replica 3, arbiter 1). Our nodes are CentOS 7 VMs within our exiting XenServer infrastructure, each node has 8 vCPUs [Xeon E5-2680 v4](https://ark.intel.com/products/91754/Intel-Xeon-Processor-E5-2680-v4-35M-Cache-2_40-GHz), 16GB of RAM and backed by our [iSCSI SSD Storage](https://smcleod.net/tech/ssd-storage-cluster-diagram/). ## Automation / Puppet We're long time, heavy users of Puppet so naturally I'm deploying Gluster via a [Puppet module](https://github.com/voxpupuli/puppet-gluster) and generating volumes and volume configuration from [Hiera](https://docs.puppet.com/hiera/). Volumes are automatically generated from the Hiera structure that defines our applications. ```shell # profiles::services::gluster::volume: profiles::services::gluster::volume::pool: "%{alias('profiles::services::gluster::host::pool')}" profiles::services::gluster::volume::pool_members: "%{alias('profiles::services::gluster::host::pool_members')}" profiles::services::gluster::volume::brick_mountpoint: '/mnt/gluster-storage' profiles::services::gluster::volume::replica: 3 profiles::services::gluster::volume::arbiter: 1 profiles::services::gluster::volume::volume_options: # Failover clients after 10 seconds of a server being unavailable 'network.ping-timeout': '10' 'cluster.lookup-optimize': 'true' 'cluster.readdir-optimize': 'true' 'cluster.use-compound-fops': 'true' 'performance.parallel-readdir': 'true' 'performance.client-io-threads': 'true' 'performance.stat-prefetch': 'true' 'diagnostics.brick-log-level': 'WARNING' 'diagnostics.client-log-level': 'WARNING' 'server.event-threads': '3' 'client.event-threads': '3' ``` I enabled the `nis_enabled` SEbool to prevent a number of SELinux denials I noticed in the logs: ```shell profiles::os::redhat::centos7::selinux::selinux_booleans: 'nis_enabled': value: 'on' ``` I also increased the local emepheral port range and the kernel's socket backlog limit as suggested by Redhat: ```shell sysctl::base::values: 'net.ipv4.tcp_max_syn_backlog': ensure: present value: '4096' comment: 'Increase syn backlogs for gluster' 'net.ipv4.ip_local_port_range': ensure: present value: '32768 65535' comment: 'Increase local port range for gluster' 'net.core.somaxconn': ensure: present value: '2048' comment: 'Increase kernel socket backlog limit for gluster' ``` ## Performance Performance is, well, very poor for anything other than large reads. I was expecting a hit to IOP/s performance as you would with any clustered, network file system, but I wasn't expecting it to drop as much as it did, especially after enabling the above performance options on the volumes. ![gluster perf vs native.jpg]({{site.baseurl}}/img/gluster perf vs native.jpg) ### Sequential Read Acceptable performance at 380MB/s: ```shell # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=1M --iodepth=32 --size=5G --readwrite=read test: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32 fio-2.1.11 Starting 1 process Jobs: 1 (f=1): [R(1)] [100.0% done] [380.0MB/0KB/0KB /s] [380/0/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=509: Mon Sep 25 12:17:06 2017 read : io=5120.0MB, bw=310321KB/s, iops=303, runt= 16895msec cpu : usr=0.58%, sys=3.50%, ctx=16068, majf=0, minf=533 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued : total=r=5120/w=0/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: io=5120.0MB, aggrb=310321KB/s, minb=310321KB/s, maxb=310321KB/s, mint=16895msec, maxt=16895msec ``` ### Sequential Write A terrible average of 25.6MB/s: ```shell # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=1M --iodepth=32 --size=5G --readwrite=write test: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32 fio-2.1.11 Starting 1 process test: Laying out IO file(s) (1 file(s) / 5120MB) Jobs: 1 (f=1): [W(1)] [18.9% done] [0KB/25600KB/0KB /s] [0/25/0 iops] [eta 01m:43s] ``` ### 4K Write IOP/s A terrible average of 684 IOP/s: ```shell # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=32 --size=5G --readwrite=randwrite test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32 fio-2.1.11 Starting 1 process Jobs: 1 (f=1): [w(1)] [1.2% done] [0KB/2736KB/0KB /s] [0/684/0 iops] [eta 34m:53s] ``` ### 4K Read IOP/s A terrible average of 3017 IOP/s: ```shell # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=32 --size=5G --readwrite=randread test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32 fio-2.1.11 Starting 1 process Jobs: 1 (f=1): [r(1)] [5.6% done] [12068KB/0KB/0KB /s] [3017/0/0 iops] [eta 07m:20s] ``` ## Annoyances ### Open Files Gluster seems to love open file handles, on an average node in the cluster with 120 small volumes connected to 3 fuse clients and no files I often see up to 1.5 _million_ open files: ```shell root@int-gluster-02:~ # lsof | wc -l 1042782 root@int-gluster-01:~ # netstat -lnp|grep gluster|wc -l 111 ``` ### Inconsistent Configuration A big gripe I have is with the location of cluster and volume configuration, it is kept in a number of different files and file locations, for example: ```shell /etc/glusterfs/glusterd.vol /var/lib/glusterd/options /var/lib/glusterd/glusterd.info /var/lib/glusterd/vols/simpleapp_static/quota.conf /var/lib/glusterd/vols/simpleapp_static/trusted-simpleapp_static.tcp-fuse.vol /var/lib/glusterd/vols/simpleapp_static/bricks/int-gluster-01\:-mnt-gluster-storage-simpleapp_static ``` This makes managing the cluster and volumes with automation tools a bit of a pain and thus most tooling calls out to the `gluster` command line application to set and read configuration which is not ideal. ### Memory Usage & Leaks Along the way with Gluster from version 3.10 to 3.12 I've encountered _many_ memory leaks resulting in OOMs, corrupt volume configuration and often entire cluster rebuilds. A lot of these seem to occur when making a large number of changes to volume configuration. Often when Gluster runs out of memory and gets OOM killed, it corrupts `.vol` or `.info` files in `/var/lib/glusterd/vols//`, this causes the daemon to fail to start, requires you to hunt through logs for mentions of incorrectly configured volumes, delete the files in question and hope that they correctly sync back from another node, if they don't or can't be synced back it seems you might have to destroy the volumes (or cluster) and recreate them before restoring your (brick) data from backups - this is not at all fun. ### Poor Gluster CLI Performance Again, because of the nature of cluster and volume configuration automation often has to call out to use the Gluster CLI tool to set or get information, this is made especially painful due to the tools performance - notably how long it takes to set / get options on volumes. Setting a volume option on average takes around 1 second and the tool cannot set an option on multiple volumes at once. This quickly adds up, suppose you have 200 volumes and you're setting 5 options on each volume, `200*5=1000 / 60` - that's _16.6 minutes just to set the options!_, this could be a real issue when recovering from a disaster scenario. ### Broken Links to Gluster.org and Gluster.readthedocs.io Many times I've clicked on links to Gluster articles or documentation and have found them to be broken, it seems that Gluster.org has undergone changes and has not created redirects for existing permalinks. - [Return Of The RSS](https://smcleod.net/2017/09/return-of-the-rss/index.md): Return Of The RSS --- author: "Sam McLeod" date: "2017-09-22T00:00:00Z" aliases: - /thoughts/2017/09/22/return-of-the-rss images: ["melbourne.jpg"] cover: image: " melbourne.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software title: Return Of The RSS --- Of all the tools for reading news and subscribing to software releases, I still find RSS the most useful. I use Feedly to manage my rss subscriptions and keep all my devices in sync, but instead of using the Feedly's own client, I use an app called Reeder as the client / reader itself. - [Link: My Feedly RSS Feed](http://feedly.com/smcleod/blogs) ## [Feedly](https://feedly.com) RSS feed subscription management Features: - Keyword alerts. - Browser plugins to subscribe to (current) url. - Notation and highlighting support (a bit like Evernote). - Search and filtering across large numbers of feeds / content. - IFTTT, Zapier, Buffer and Hootsuite integration. - Built in save / share functionality (that I only use when I'm on the website). - Backup feeds to Dropbox. - Very fast, regardless of the fact that I'm in Australia - which often impacts the performance of apps / sites that tend to be hosted on AWS in the US as the latency is so high. - Article de-duplication is currently being developed I believe, so I'm looking forward to that! - Easy manual import, export and backup (no vendor lock-in is important to me). - Public sharing of your Feedly feeds (we're getting very meta here!). ## [Reeder](http://reederapp.com) A (really) beautiful and fast iOS / macOS client - The client apps aren't cheap but damn they're good quality, I much prefer them over the standard Feedly apps. - Obviously supports Feedly as a backend but there are many other source services you can use along side each other. - I save articles using Reeder's clip to Evernote functionality... a lot. - Sensible default keyboard shortcuts (or at least for me they felt natural YMMV of course). - Good customisable 'share with' options. - Simple, well designed UX. - Easy manual import an export just like Feedly. - [MH-Z19 CO2 sensor reader, logger and visualiser](https://smcleod.net/2016/12/mh-z19-co2-sensor-reader-logger-and-visualiser/index.md): MH-Z19 CO2 sensor reader, logger and visualiser --- title: "MH-Z19 CO2 sensor reader, logger and visualiser" date: 2016-12-21T16:37:38+11:00 lastmod: 2016-12-21T16:37:38+11:00 author: Sam McLeod description: "Python scripts to read, log and plot CO2 readings from an MH-Z19 sensor over UART." keywords: ["health","iot","co2"] tags: ["health","IoT","co2"] categories: ["IoT","Health"] series: [] cover: image: "https://user-images.githubusercontent.com/862951/52827251-21597e00-3118-11e9-9ebc-ddbbc9fb02a8.jpg" toc: enable: true auto: false math: false draft: false type: posts --- # MH-Z19 CO2 sensor reader, logger and visualiser ![plot](https://user-images.githubusercontent.com/862951/52826593-a98a5400-3115-11e9-868a-72a763b6d587.jpg) * Reads data from UART(serial)-connected MH-Z19 (or MH-Z14) sensor using python 3. * If you dare to install nodejs you can visualise the logged data (using html and plotly.js library). Repository: [sammcj/CO2-Logger](https://github.com/sammcj/CO2-Logger) ## Usage *Note this post is from 2016, in 2021 I replaced my custom Co2 loggers with an [Aranet4](https://aranet.com/products/aranet4/). While very expensive, is and excellent off-the-shelf solution, with many features.* ### Connection Sensor can be queried using 3.3v UART at 9600 bps. Sensor main feed voltage is 5v. Can be connected to computer using almost any USB-UART converter if voltage matches. ### Requirements * Python3 * `pip3 install serial pyserial` * A USB -> Serial UART connector (Could also be used with Raspberry Pi, ESP8266, Arduino etc...) * [MH-Z19](https://www.winsen-sensor.com/d/files/PDF/Infrared%20Gas%20Sensor/NDIR%20CO2%20SENSOR/MH-Z19%20CO2%20Ver1.0.pdf) Co2 Sensor (Around $25AUD on eBay) ### Querying ```shell $ python3 CO2Reader.py --tty=/dev/tty.SLAB_USBtoUART --timeout=2 Connected to /dev/tty.SLAB_USBtoUART 2019-02-25 07:44:48 558 58 2019-02-25 07:44:49 558 58 2019-02-25 07:44:51 558 58 ... ``` 3 fields separated by tab: timestamp, CO2 concentration (ppm), *internal* sensor temperature (Celsius) Use stream redirection to save data series to file: ```shell touch co2.log python3 CO2Reader.py --tty=/dev/tty.SLAB_USBtoUART --timeout=2 >>co2.log ``` ### Options * `--tty` (optional) tty to connect to (default is `/dev/tty.SLAB_USBtoUART`) * `--single` (optional) perform a single measurement and exit * `--quit` (optional) connect to the sensor and quit immediately * `--timeout` (optional) time between requests (defaults to `10` seconds) * `--command` (optional) raw command to pass to the sensor - use carefully (default is to read Co2 levels) ### Sensor Commands Raw commands can be passed to the Co2 sensor, these to be used with caution. Example: ```shell python3 CO2Reader.py --tty=/dev/tty.SLAB_USBtoUART --timeout=1 --command='0xff, 0x01, 0x86, 0x00, 0x00, 0x00, 0x00, 0x00, 0x79' ``` #### Default operation (Reads Co2 levels) `0xff, 0x01, 0x86, 0x00, 0x00, 0x00, 0x00, 0x00, 0x79` #### Enable ABC `0xFF, 0x01, 0x79, 0xA0, 0x00, 0x00, 0x00, 0x00, 0xE6` #### Disable ABC `0xFF, 0x01, 0x79, 0x00, 0x00, 0x00, 0x00, 0x00, 0x86` #### Measurement Ranges (Note: Untested) * 1000 = `0xFF, 0x01, 0x99, 0x00, 0x00, 0x00, 0x03, 0xE8, 0x7B` * 2000 = `0xFF, 0x01, 0x99, 0x00, 0x00, 0x00, 0x07, 0xD0, 0x8F` * 3000 = `0xFF, 0x01, 0x99, 0x00, 0x00, 0x00, 0x0B, 0xB8, 0xA3` * 5000 = `0xFF, 0x01, 0x99, 0x00, 0x00, 0x00, 0x13, 0x88, 0xCB` #### Run Zero Point (400ppm) calibration * Run this for 10mins + in 400ppm Environment WARNING: DO NOT RUN THIS UNLESS YOU KNOW WHAT YOU ARE DOING! `0xFF, 0x01, 0x87, 0x00, 0x00, 0x00, 0x00, 0x00, 0x78` ### Visualizing 1. Install npm dependencies `npm install` 2. Start server `python3 -m http.server 8088 --bind 127.0.0.1` 3. Open browser at 4. Select your log file in input field ## Technical Specifications MH-Z19 | Attribute | Value | |-----------------------------|-----------------------------| | Target gas | Carbon Dioxide CO2 | | Operating Voltage | 3.6 to 5.5 Vdc | | Operating current | < 18mA average | | Interface levels | 3.3 Vdc | | Output signal format | UART or PWM | | Preheat time | 3 min | | Response time | <60 s | | Accuracy | ± (50 ppm+5% reading value) | | Measuring range | 0 to 5000 ppm | | Operating temperature range | 0 to + 50°C | | Dimensions | 33mmx20mmx9mm(LxWxH) | ## Wiring | Function | UART / Signal | MH-Z19 pin | |----------|---------------|------------| | Vcc +5V | +5V | 6 Vin | | GND | GND | 7 GND | | UART | TXD0 | 2 RXD | | UART | RXD0 | 3 TXD | ![co2-sensor-mh-z19-pinout](https://user-images.githubusercontent.com/862951/52826907-c7a48400-3116-11e9-9c2e-c5fde2cf8f1d.jpg) ## Photos ### MH-Z19 Sensors ![41byp-zvj1l _sx342_](https://user-images.githubusercontent.com/862951/53372770-d48f6600-39a7-11e9-8c05-9aca871de7d0.jpg) ![350px-mhz19](https://user-images.githubusercontent.com/862951/53372771-d527fc80-39a7-11e9-80e6-17ade8eb4b19.jpg) ### Connected to a USB -> UART adapter ![mh-z19](https://user-images.githubusercontent.com/862951/52826018-38e23800-3113-11e9-92f3-18c99c902ae5.jpg) ### Calibration **Don't use this function unless you know what you are doing or you may incorrectly calibrate the sensor!** (Australia has an average of around 400-410PPM and Sensor zero point is 400PPM) ![mhz - calibration](https://user-images.githubusercontent.com/862951/52827251-21597e00-3118-11e9-9ebc-ddbbc9fb02a8.jpg) ## Credits * Inspiration taken from my old [AirQualityLogger project](https://github.com/sammcj/airqualitylogger) * Thanks to [@matmunn](https://github.com/matmunn) for Python 2->3 upgrade. * Originally forked from ## Licence MIT licence - [Castle Hill Winter Trip 2016 Photos](https://smcleod.net/2016/08/castle-hill-winter-trip-2016-photos/index.md): Castle Hill Winter Trip 2016 Photos --- author: "Sam McLeod" categories: [ Photos ] date: "2016-08-06T00:00:00Z" aliases: - /photos/2016/08/06/winter-wonder-land/ description: Castle Hill Winter Trip 2016 [Photos] images: ["backdrop-austria.jpg"] cover: image: " backdrop-austria.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - photos - cars title: Castle Hill Winter Trip 2016 Photos --- ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/blue.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/overandout.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/thisimagerocks.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/foggy.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/skifield.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/depth.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/getyawideon.jpg?raw=true) ![](https://github.com/sammcj/smcleod_files/blob/master/images/castle_hill_2016/getyawideonagain.jpg?raw=true) --- {{< vimeo 177805831 >}} - [The State of Android in 2016 & The OnePlus 3 Phone](https://smcleod.net/2016/07/the-state-of-android-in-2016-the-oneplus-3-phone/index.md): The State of Android in 2016 & The OnePlus 3 Phone --- author: "Sam McLeod" categories: [ Tech ] date: "2016-07-11T00:00:00Z" aliases: - /tech/2016/07/11/lets-try-android-in-2016 images: ["adrien-296945.jpg"] cover: image: " adrien-296945.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - hardware title: The State of Android in 2016 & The OnePlus 3 Phone --- I wanted to try Android for a couple of weeks, I like staying on top of technology, gadgets and making sure I never become a blind 'zealot' for any platform or brand. ## The OnePlus 3 I did a lot of research and decided to try the "Oneplus 3" as it was good bang-for-buck, ran the latest software had plenty of grunt with the latest 8 core, high clock speed Qualcomm processor coupled with 6GB of DDR4 - the specs really are very impressive, especially for a $400USD phone. Hardware wise the unit is lovely, not as high build quality as my iPhone 6s+ but not nearly as bad as the Samsung Galaxy S3 or other Samsung devices I had tried in the past. ![](http://medias.mensquare.com/wp-content/uploads/2016/06/15091937/oneplus-3-photos-officielles_48.jpg) ## Hardware ### The Good - Hardware wise the OP3 felt like it was designed with more care than any other Android phone I'd spent any fair amount of time. - USB-C connectivity is great to see on a phone, it feels right at home on the device and opens a world of opportunities for accessories assuming the OS supports them. ### The Bad - The camera felt about as a good as an iPhone 4s which is now quite dated technology. - Wireless AC performance was just over 1/2 that of my iPhone 6s+. ### The Ugly - While I loved that the phone used USB-C I was very disappointed to find that it was actually only USB 2.0. - The 4G LTE only gained about 2/3 - 1/2 the performance of my iPhone 6s+ in the same location and on the same network. ## The Software ### The Good - The OS looked pretty, at first. - I was please to see that the OP3 used block level encryption by default. - The OP3 had SELinux enabled, although I'm not sure what policies were in use. - The device was fast and responsive at all times. ### The Bad - I started installing apps and finding my way around the OS very quickly felt messy and like a chore. - I noted that the general 'quality' of android apps felt quite poor. - Nothing seemed to integrate very well, there was a real feeling that minimal time had been spent on the idea of usage flow on the interface. - If at all and a lot of apps felt like they hadn't been maintained for quite some time despite being the 'the best option' available. - Notifications seemed to annoy me more often than they did notify me of important things. - Battery life was about 1/2 that of my iPhone 6s+, although I was using it quite extensively. ### The Ugly - The platform was ridding with Google specific apps, and it tries really hard to make you use them whenever possible. - Many apps required root access to the phone which is not something I wanted to do, similar applications on iOS did not require 'jailbreaking' which I have not done in many years, nor felt the need to. I had a go at disconnecting my from the clutches of Google first by trying not to sign into any of their applications and disabling all the information reporting in settings, I felt like I was a bit like how iOS was back in the iPhone 3G days when you had to root the phone to do half the things you wanted. I then ended up flashing a custom rom that was pretty much stock, but without the good stock apps, I had to install the Google App Store and create an account which I think is fair enough because I couldn't be bothered dropping custom downloaded apks and using that dreadful android transfer tool from my laptop which seems to work 4/5 times you plug your phone in, but like a lot of the apps in the ecosystem - felt very 'javaish' (pun slightly intended). ## Final Words The OP3 is a _very_ impressive piece of hardware for $400 USD but I'm left with the feeling that Google's platform is fragmented and lacking work-flow and I was left with the over all feeling that Google was watching my every move, I don't know if it's true or not but I certainly felt like I was Google's product, not the OS. I look forward to a usable, non-Android Linux OS image becoming available hopefully in the near future, then it'd be a handy device for tethering with and for mobile security analysis. - [Update Delayed Serial STONITH Design](https://smcleod.net/2016/07/update-delayed-serial-stonith-design/index.md): Update Delayed Serial STONITH Design --- author: "Sam McLeod" categories: [ Tech ] date: "2016-07-04T00:00:00Z" aliases: - /tech/2016/07/04/update-rcd-stonith-design images: ["noah-kuhn-27481.jpg"] cover: image: " noah-kuhn-27481.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware title: Update Delayed Serial STONITH Design --- _note: This is a follow up post from [2015-07-21-rcd-stonith](https://smcleod.net/tech/2015/07/21/rcd-stonith/)_ ### A Linux Cluster Base STONITH provider for use with modern Pacemaker clusters This has since been accepted and merged into Fedora's code base and as such will make it's way to RHEL. - Source Code: [Github](https://github.com/sammcj/fence_rcd_serial) - [Diptrace](https://diptrace.com/download/download-diptrace/) CAD Design: [Github](https://github.com/sammcj/fence_rcd_serial/tree/master/CAD/STONTH_CAD_DESIGN_V3) - I have open sourced the CAD circuit design and made this available within this repo under [CAD Design and Schematics](https://github.com/sammcj/fence_rcd_serial/tree/master/CAD/STONTH_CAD_DESIGN_V3) - Related RedHat Bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1240868](https://bugzilla.redhat.com/show_bug.cgi?id=1240868) #### `v1` vs `v2/v3` versions of the `rcd_serial` STONITH system The v2/v3 cables include the following improvements: - Have a connector on the outside of the server (that's female side runs from the reset pin 'hijacker') so that cables can be easily disconnected. - Have a slightly improved circuit design. - Have long thumb-screws on the RS232 serial connector that not longer accidentally unscrew the female RS232 connector in the server (whoops!). ![]({{< ref "/" >}}/img/v3_cable.jpg) ![]({{< ref "/" >}}/img/rcd_serial_v3_diagram.png) #### 'Hijack' cable All the 'Hijack' cable is, is essentially a small M-F extension cable for the motherboard front-panel (buttons/LED) PINs with one change: - The two reset pins are 'hijacked' so you can add a connector to run out the back of the server a connection to the adjacent storage server's rcd_serial connector. ![]({{< ref "/" >}}/img/hijack.jpg) ![]({{< ref "/" >}}/img/clip_correct.jpeg) --- Many thanks to: - OurPCB (Board fabrication). - Clusterlabs, Redhat and Fedora Teams, feedback and peer review. - John Sutton for his [original design](http://www.init.hr/dev/cluster/glue/lib/plugins/stonith/rcd_serial.c) that served as inspiration. - [Monitoring SystemD Units With Nagios](https://smcleod.net/2016/05/monitoring-systemd-units-with-nagios/index.md): Monitoring SystemD Units With Nagios --- author: "Sam McLeod" categories: [ Tech ] date: "2016-05-23T00:00:00Z" aliases: - /tech/2016/05/23/monitor-systemd-nagios images: ["feature-laptop.jpg"] cover: image: " feature-laptop.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software title: Monitoring SystemD Units With Nagios --- _Ever forgotten to add a critical service to monitoring?_ _Want to know if a service or process fails without explicitly monitoring every service on a host?_ ...Then why not use SystemD's existing knowledge of all the enabled services? Thanks to 'Kbyte' who made a simple Nagios plugin to do just this! ![](http://kbyte.snowpenguin.org/portal/wp-content/uploads/2014/11/nagios.png) ## Requirements - Python3 (For RHEL/CentOS 7 `yum install python34`) - python-nagiosplugin [My pre-built RPMs](https://packagecloud.io/app/s_mcleod/centos7/search?q=python-nagiosplugin) or `pip3 install nagiosplugin` - [PyNagSystemD](https://github.com/kbytesys/pynagsystemd/blob/master/bin/pynagsystemd.py) - [Online Conversion from SQL_ASCII to UTF8 in PostgreSQL](https://smcleod.net/2016/05/online-conversion-from-sql_ascii-to-utf8-in-postgresql/index.md): Online Conversion from SQL_ASCII to UTF8 in PostgreSQL --- author: "Sam McLeod" categories: [ Tech ] date: "2016-05-23T00:00:00Z" aliases: - /tech/2016/05/23/convert-sql-ascii-to-utf8 images: ["patrick-tomasso-71909.jpg"] cover: image: " patrick-tomasso-71909.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software - data title: Online Conversion from SQL_ASCII to UTF8 in PostgreSQL --- Scripts and source available here: [sql_ascii_to_utf8](https://github.com/sammcj/sql_ascii_to_utf8) ## The Goal To be able to take a Postgres Database which is in SQL_ASCII encoding, and import it into a UTF8 encoded database. Requirements: - Python3 (For RHEL/CentOS 7 `yum install python34`) - python-nagiosplugin [My pre-built RPMs](https://packagecloud.io/app/s_mcleod/centos7/search?q=python-nagiosplugin) or `pip3 install nagiosplugin` - [PyNagSystemD](https://github.com/kbytesys/pynagsystemd/blob/master/bin/pynagsystemd.py) ## The Problem PostreSQL will generate errors like this if it encounters any non-UTF8 byte-sequences during a database restore: ```shell # pg_dump -Fc test_badchar | pg_restore -d test_badchar_utf8 pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 2839; 0 26852 TABLE DATA table101 postgres pg_restore: [archiver (db)] COPY failed for table "table101": ERROR: invalid byte sequence for encoding "UTF8": 0x91 CONTEXT: COPY table101, line 1 WARNING: errors ignored on restore: 1 ``` And the corresponding data will be omitted from the database (in this case, the whole table, even the rows which did not have a problem): ```sql # echo "select * from table101;" | psql test_badchar_utf8 chardecimal | description | column6_text -------------+-------------+-------------- (0 rows) ``` The Solution ------------ To find and replace characters in an SQL_ASCII encoded database which do not conform to the UTF8 encoding requirements. This should be implemented as an SQL script, so that the data can be updated on a live database, to mininmise the downtime required. There is an existing script which will _find_ the offending rows in a table: While this will find the offending rows, it cannot do anything about it, as it stands. A more sophisticated script is required to actually replace the non-UTF8 characters with something acceptable. Two SQL scripts are provided here, the first of these SQL scripts is: - [Replace_non_UTF8.underscore.sql](https://github.com/sammcj/sql_ascii_to_utf8/blob/master/Replace_non_UTF8.underscore.sql) This will replace all non-UTF8 characters with underscores. This (at least) will allow the data to be imported successfully to a UTF8 database. The second of these SQL scripts is: - [Replace_non_UTF8.html_equiv.sql](https://github.com/sammcj/sql_ascii_to_utf8/blob/master/Replace_non_UTF8.html_equiv.sql) will replace selected bytes with a UTF8-sequence which corresponds to what is rendered by Firefox when encoded as `&#NN;` where NN is the hexadecimal value of the byte. Not all values from `80-FF` are covered by this script. Please add your own translations as required. Any byte without a specific translation will be replaced with an underscore. ## The Triggers Problem The functions `process_non_utf8_at_column()` and `process_non_utf8_at_schema()` work just fine, BUT if there are any 'triggers' on the rows being updated, these triggers are also invoked. Such triggers may expect a specific set of fields to be updated together, or increment sequence numbers. Running these triggers would be an undesirable side-effect of what should be a simple text-update. ## The Locking Problem The original solution was designed to minimise downtime, and these scripts would be ineffective if they were to lock table for anything more than a couple of seconds. Unfortunately, this is exactly what happens if triggers are disabled per-table while updating the text like this: ```sql ALTER TABLE _some_table_ DISABLE TRIGGER ALL; process_non_utf8_at_column(...); ALTER TABLE _some_table_ ENABLE TRIGGER ALL; ``` Postgres wraps it all in a transaction, and locks the table until the update is complete (which can be minutes on a large table). ## The Non-Locking, Non-Triggering Solution a) Don't use `ALTER TABLE _some_table_ DISABLE TRIGGER ALL;` Instead, use a session-based setting: ```sql SET session_replication_role = replica; ``` This has the effect of disabling triggers, but does not lock the whole table while the function is running. b) Don't try to run the whole DB in one go. The following script does a search for offending table,column combinations, and then invokes process_non_utf8_at_column() on each of these individually. - [run_process_non_utf8.sh](https://github.com/sammcj/sql_ascii_to_utf8/blob/master/run_process_non_utf8.sh) ## Sample DB and Outputs A test database can be created using the script: - [create_test_sql_ascii.sh](https://github.com/sammcj/sql_ascii_to_utf8/blob/master/create_test_sql_ascii.sh) This should be run as follows: ```sql createdb -e --template=template0 -E SQL_ASCII test_badchar ./create_test_sql_ascii.sh test_badchar ``` When we run the script, we should see the following output: ```sql # psql test_badchar < Replace_non_UTF8.html_equiv.sql CREATE FUNCTION CREATE FUNCTION CREATE FUNCTION CREATE FUNCTION ``` ```sql # ./run_process_non_utf8.sh test_badchar ======================================== Start: Tue Apr 19 09:16:30 AEST 2016 ======================================== INFO: schema: public table: table101 column: description time: 2016-04-19 09:16:30.949714+10 INFO: schema: public table: table101 column: non_utf8_text time: 2016-04-19 09:16:30.991793+10 INFO: schema: public table: empty_table column: somedata time: 2016-04-19 09:16:31.406256+10 search_for_non_utf8_columns ------------------------------------ (public,table101,description,1) (public,table101,non_utf8_text,11) (2 rows) 'table101','description' 'table101','non_utf8_text' ======================================== Initial search completed: Tue Apr 19 09:16:31 AEST 2016 Search run time 1 seconds ======================================== processing: 'table101','description' SET session_replication_role = replica; SET SELECT process_non_utf8_at_column('table101','description'); NOTICE: table "x_list" does not exist, skipping NOTICE: 1: table: table101 (column: description) time: 2016-04-19 09:16:31.425359+10 INFO: running remove_non_utf8 (UTF8 follo...��� <-here) NOTICE: 1: table: table101 (column: description) time: 2016-04-19 09:16:31.481739+10 process_non_utf8_at_column ---------------------------- (1 row) SET session_replication_role = DEFAULT; SET processing: 'table101','non_utf8_text' SET session_replication_role = replica; SET SELECT process_non_utf8_at_column('table101','non_utf8_text'); NOTICE: table "x_list" does not exist, skipping NOTICE: 1: table: table101 (column: non_utf8_text) time: 2016-04-19 09:16:31.498909+10 INFO: running remove_non_utf8 (91-> � ���...<- -½�½-) INFO: running remove_non_utf8 (92-> � ���...<- -½�½-) INFO: running remove_non_utf8 (96-> � ���...<- -½�½-) INFO: running remove_non_utf8 (A3-> � ���...<- -½�½-) INFO: running remove_non_utf8 (C7-> � ���...<- -½�½-) INFO: running remove_non_utf8 (D0-> � ���...<- -½�½-) INFO: running remove_non_utf8 (D5-> � ���...<- -½�½-) INFO: running remove_non_utf8 (E9-> � ���...<- -½�½-) INFO: running remove_non_utf8 (FA-> � ���...<- -½�½-) INFO: running remove_non_utf8 (�...�) INFO: running remove_non_utf8 (�🐱�...�🐱�) NOTICE: 1: table: table101 (column: non_utf8_text) time: 2016-04-19 09:16:32.358957+10 process_non_utf8_at_column ---------------------------- (1 row) SET session_replication_role = DEFAULT; SET ======================================== Update complete: Tue Apr 19 09:16:32 AEST 2016 Update run time 1 seconds ======================================== ========= checking results ============= search_for_non_utf8_columns ----------------------------- (0 rows) ======================================== Finish: Tue Apr 19 09:16:32 AEST 2016 Total run time 2 seconds ======================================== ``` ```sql # psql test_badchar < Replace_non_UTF8.cleanup.sql DROP FUNCTION DROP FUNCTION DROP FUNCTION DROP FUNCTION ``` - [Speeding Up rsync](https://smcleod.net/2016/05/speeding-up-rsync/index.md): Speeding Up rsync --- author: "Sam McLeod" categories: [ Tech ] date: "2016-05-03T00:00:00Z" aliases: - /tech/2016/05/03/speeding-up-rsync images: ["robin-pierre-323861.jpg"] cover: image: " robin-pierre-323861.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software - networking title: Speeding Up rsync --- The most common way to use rsync is probably as such: ```shell rsync -avr user@: ``` Resulting in 30-35MB/s depending on file sizes This can be improved by using a more efficient, less secure encryption algorithm, disabling compression and telling the SSH client to disable some unneeded features that slow things down. With the settings below I have achieved 100MB/s (at work between VMs) and over 300MB/s at home between SSD drives. ## Requirements - Python3 (For RHEL/CentOS 7 `yum install python34`) - python-nagiosplugin [My pre-built RPMs](https://packagecloud.io/app/s_mcleod/centos7/search?q=python-nagiosplugin) or `pip3 install nagiosplugin` - [PyNagSystemD](https://github.com/kbytesys/pynagsystemd/blob/master/bin/pynagsystemd.py) ```shell rsync -arv --numeric-ids --progress -e "ssh -T -c aes256-gcm@openssh.com -o Compression=no -x" user@: ``` If you want to delete files at the DST that have been deleted at the SRC (obviously use with caution: ```shell rsync -arv --numeric-ids --progress -e "ssh -T -c aes256-gcm@openssh.com -o Compression=no -x" user@: --delete ``` ## Notes 1. Because of the weak encryption used, it is not recommended for transferring files across hostile networks (such as the internet). 2. There are scenarios where enabling compression _can_ improve performance, i.e. if your network link is very slow and your files compress well. 3. Don't forget to forward your SSH keys to the host you're going to run it on! (`ssh-agent && ssh-add` ((if it's not already running)) `ssh -A user@host`) 4. If `aes256-gcm@openssh.com` isn't available for you due to using an old operating system, you can use `aes128-ctr` - [Benchmarking IO with FIO](https://smcleod.net/2016/04/benchmarking-io-with-fio/index.md): Benchmarking IO with FIO --- author: "Sam McLeod" categories: [ Tech ] date: "2016-04-29T00:00:00Z" aliases: - /tech/2016/04/29/benchmarking-io images: ["daniel-mayovskiy-303666.jpg"] cover: image: " daniel-mayovskiy-303666.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage title: Benchmarking IO with FIO --- ## This is a quick tldr there are _many_ other situations and options you could consider * [FIO man page](http://linux.die.net/man/1/fio) * IOP/s = Input or Output operations per second * Throughput = How many MB/s can you read/write continuously ### Variables worth tuning based on your situation * `--iodepth` The iodepth is very dependant on your hardware. * Rotational drives without much cache and high latency (i.e. desktop SATA drives) will not benefit from a large iodepth, Values between 16 to 64 could be sensible. * High speed, lower latency SSDs (especially NVMe devices) can utilise a much higher iodepth, Values between 256 to 4096 could be sensible. * `--bs` The block size is very dependant on your workload. * Writing/Reading lots of small files (i.e. documents, logs) benefit from / represent a smaller block size, Values between 2K - 128K could be sensible and 4k is likely the average in most situations. * Writing/Reading large files (i.e. videos, database backups) benefit from / represent a large block size, Values between 2M - 8M could be sensible and 4M is likely the average in most situations. ### Before running these tests 1. Check you're in a directory with enough free disk space. 1. Check / pause any other workloads that may interfere with the results. 1. Understand your workload / what you intend to use the storage for - i.e. what matters? 1. Tune anything you might want to tune as above such as `iodepth` or `size`. #### Random write test for IOP/s i.e. lots of small files ```shell sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite --ramp_time=4 ``` #### Random Read test for IOP/s i.e. lots of small files ```shell sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread --ramp_time=4 ``` #### Sequential write test for throughput i.e. one large file ```shell sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4M --iodepth=64 --size=4G --readwrite=write --ramp_time=4 ``` #### Sequential Read test for throughput i.e. one large file ```shell sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4M --iodepth=64 --size=4G --readwrite=read --ramp_time=4 ``` #### Testing IO latency without fio ```shell samm@int-backup-01:/mnt/store1 # ioping . # old backup server 4096 bytes from . (ext4 /dev/mapper/store1-36TB): request=1 time=0.2 ms 4096 bytes from . (ext4 /dev/mapper/store1-36TB): request=2 time=0.1 ms 4096 bytes from . (ext4 /dev/mapper/store1-36TB): request=3 time=0.9 ms ``` vs ```shell samm@int-backup-02:/mnt/store1 # ioping . # new backup server 4 KiB from . (ext4 /dev/md10): request=1 time=88 us 4 KiB from . (ext4 /dev/md10): request=2 time=103 us 4 KiB from . (ext4 /dev/md10): request=3 time=102 us ``` - [Mirroring a Gitlab project to Github](https://smcleod.net/2016/02/mirroring-a-gitlab-project-to-github/index.md): Mirroring a Gitlab project to Github --- author: "Sam McLeod" categories: [ Tech ] date: "2016-02-04T00:00:00Z" aliases: - /tech/2016/02/04/mirror-gitlab-to-github/ images: ["backdrop-austria.jpg"] cover: image: " backdrop-austria.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software title: Mirroring a Gitlab project to Github --- Let's pretend you have a project on Gitlab called `ask-izzy` and you want to mirror it up to Gitlab which is located at Assuming you're running Gitlab as the default user of `git` and that your repositories are stored in `/mnt/repositories` you can following something similar to the following instructions: 1. Grant write access to Github Get your Gitlab install's pubkey from the git user ```shell cat /home/git/.ssh/id_rsa.pub ``` On Github add this pubkey as deploy key on the repo, make sure you tick the option to allow write access. 2. Add a post-receive hook to the Gitlab project ```shell mkdir /mnt/repositories/developers/ask-izzy.git/custom_hooks/ echo "exec git push --quiet github &" > \ /mnt/repositories/developers/ask-izzy.git/custom_hooks/post-receive chown -R git:git /mnt/repositories/developers/ask-izzy.git/custom_hooks chmod +x /mnt/repositories/developers/ask-izzy.git/custom_hooks/post-receive ``` 3. Add Github as a remote to the Gitlab project ```shell cd /mnt/repositories/developers/ask-izzy.git vi config ``` and add in the Github remote: ```shell [remote "github"] url = git@github.com:ask-izzy/ask-izzy.git fetch = +refs/*:refs/* mirror = true ``` - [AskIzzy](https://smcleod.net/2016/01/askizzy/index.md): AskIzzy --- author: "Sam McLeod" categories: [ Tech ] date: "2016-01-29T00:00:00Z" aliases: - /tech/2016/01/29/askizzy/ images: ["melbourne.jpg"] cover: image: " melbourne.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech title: AskIzzy --- ## Today we launched a mobile website for homeless people [![AskIzzy home page](askizzy.png)](https://askizzy.org.au) ... and it was launched by [one of Australia's many recent Prime Ministers](http://www.heraldsun.com.au/news/victoria/askizzy-app-connects-the-homeless-to-food-shelter-and-health-services/news-story/bfae67275552be421af4dd54bfd575a6?nk=ebd042d1d2789514c4e8553cb5633711-1454053888) ### Today alone we served up over 87,000 requests As many of you know, I work with [Infoxchange](https://infoxchange.org) as the operations lead. When I first heard the idea of a website or app for people that have found or are worried about finding themselves homeless in Australia I really didn't think it made sense - until I saw the stats showing how many homeless people in Australia have regular access to a smart phone and data either via a cellular provider or free WiFi. We did a lot of research, working with homeless and at-risk people throughout Australia, it's really been quite an eye opener especially for my team who are largely technically focused. AskIzzy is the result of Infoxchange winning the Google Impact Challenge in 2015. For me the most interesting things about the site other than it's value to those in need is that it didn't cost tax payers a cent to develop or host and it has no model for making profit of any kind, this resulted in the site being designed truly for the end consumer - the person in need. - [Fix XenServer SR with corrupt or invalid metadata](https://smcleod.net/2016/01/fix-xenserver-sr-with-corrupt-or-invalid-metadata/index.md): Fix XenServer SR with corrupt or invalid metadata --- author: "Sam McLeod" categories: [ Tech ] date: "2016-01-18T00:00:00Z" aliases: - /tech/2016/01/18/fix-xenserver-sr-with-corrupt-metadata/ images: ["samuel-zeller-358865.jpg"] cover: image: " samuel-zeller-358865.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - xen title: Fix XenServer SR with corrupt or invalid metadata --- If a disk / VDI is orphaned or only partially deleted you'll notice that under the SR it's not assigned to any VM. This can cause issues that look like metadata corruption resulting in the inability to migrate VMs or edit storage. For example: ```shell [root@xenserver-host ~]# xe vdi-destroy uuid=6c2cd848-ac0e-441c-9cd6-9865fca7fe8b Error code: SR_BACKEND_FAILURE_181 Error parameters: , Error in Metadata volume operation for SR. [opterr=VDI delete operation failed for parameters: /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/MGT, 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b. Error: Failed to write file with params [3, 0, 512, 512]. Error: 5], ``` ## Removing stale VDIs To fix this, you need to remove those VDIs from the SR after first deleting the logical volume: * Get the LV ID (last number shown above) and find it's location in /dev: ```shell [root@xenserver-host ~]# lvdisplay | grep 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b LV Name /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/VHD-6c2cd848-ac0e-441c-9cd6-9865fca7fe8b ``` * Remove the logical volume: ```shell [root@xenserver-host ~]# lvremove /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/VHD-6c2cd848-ac0e-441c-9cd6-9865fca7fe8b Logical volume "VHD-6c2cd848-ac0e-441c-9cd6-9865fca7fe8b" successfully removed ``` * Destroy the VDI: ```shell [root@xenserver-host ~]# xe vdi-destroy uuid=6c2cd848-ac0e-441c-9cd6-9865fca7fe8b ``` ## Regenerate the MGT volume If this doesn't work and the SR is still having metadata problems the MGT (management volume) may be corrupt. Luckily this is easy to rebuild and doesn't require VMs to be powered off or migrated to another SR. * Rescan the SR ```shell xe sshellr-scan uuid= ``` * Rename the SR's MGT logical volume (this is safe and does not affect running VMs): ```shell lvcrshellename /dev/VG_XenStorage-/MGT /dev/VG_XenStorage-/oldMGT ``` * Rescan the SR Note: in some cases you might need to do this a couple of times. ```shell xe sshellr-scan uuid= ``` * Remove any stale VDIs Look for any VDIs without VMs on the SR in XenCentre or on the cli with: ```shell xe vshelldi-list sr= ``` Remove them with: ```shell [root@pm-b5 ~]# lvdisplay | grep 6c2cd848-ac0e-441c-9cd6-9865fca7fe8b LV Name /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/VHD-6c2cd848-ac0e-441c-9cd6-9865fca7fe8b [root@pm-b5 ~]# lvremove /dev/VG_XenStorage-3ae1df17-06ee-7202-eb92-72c266134e16/VHD-6c2cd848-ac0e-441c-9cd6-9865fca7fe8b Logical volume "VHD-6c2cd848-ac0e-441c-9cd6-9865fca7fe8b" successfully removed ``` * Rescan the SR ```shell xe sshellr-scan uuid= ``` - [iSCSI SCSI-ID / Serial Persistence](https://smcleod.net/2015/12/iscsi-scsi-id-/-serial-persistence/index.md): iSCSI SCSI-ID / Serial Persistence --- url: "/2015/12/iscsi-scsi-id-/-serial-persistence/" # keeps the live URL: Hugo 0.166 no longer keeps "/" from the title in :title slugs author: "Sam McLeod" categories: [ Tech ] date: "2015-12-14T00:00:00Z" aliases: - /tech/2015/12/14/iscsi-scsiid-persistence/ images: ["feature-san-fran.jpg"] cover: image: " feature-san-fran.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage title: iSCSI SCSI-ID / Serial Persistence --- _**"Having a SCSI ID is a f*cking idiotic thing to do."**_ [- Linus Torvalds](http://yarchive.net/comp/linux/scsi_ids.html) _...and after the amount of time I've wasted getting XenServer to play nicely with LIO iSCSI failover I tend to agree._ --- ## The Problem ![]({{< ref "/" >}}/img/san/sr_fail.jpg) One oddity of Xen / XenServer's storage subsystem is that it identifies iSCSI storage repositories via a calculated SCSI ID rather than the iSCSI Serial - which would be the sane thing to do. Citrix's less than ideal take on dealing with SCSI ID changes is for you to take your VMs offline, disconnected the storage repositories, recreate them, then go through all your VMs and re-attach their orphaned disks hoping that you remembered to add some sort of hint as to what VM they belong to, then finally wipe the sweat and tears from your face. From [CTX11641](https://support.citrix.com/article/CTX118641) - 'How to Identify If SCSI Storage Repository has Changed SCSI IDs': >"The SCSI ID of the logical unit number (LUN) changed. When this happened, the iSCSI storage repository became unplugged after a XenServer reboot." >... >"To correct the issue you must recreate a PBD with the entry to reflect the right SCSI ID." ## The Solution A big thank you to Nicholas A. Bellinger from the Kernel SCSI mailing list who helped me a lot in [this thread](http://comments.gmane.org/gmane.linux.scsi.target.devel/10617) where he explained: > "The Company ID, VSI, and VSIE are generated by LIO based upon the current `vpd_unit_serial` configfs attribute value. So as long as `vpd_unit_serial` is persistent, and the same value for backend devices across export failover to different nodes, Xen will always see the same EVPD information." > > An example SCSI ID of `0x6001405bff3f42a49d84cfcb64e2b933` would thus be comprised of: > > - NAA 6, IEEE Company_id: `0x1405` > - Vendor Specific Identifier: `0xbff3f42a4` > - Vendor Specific Identifier Extension: `0x9d84cfcb64e2b933` In addition to the `vpd_unit_serial` we found that the `iblock` number must also remain the same between failovers. ```shell /sys/kernel/config/target/core/iblock_0/lun_name/wwn/vpd_unit_serial ``` ``` /sys/kernel/config/target/core # tree ├── iblock_0 # Must be consistent between failovers │   └── iscsi_lun_r2 │  └── wwn │   └── vpd_unit_serial # Must be consistent between failovers ``` If you're using Corosync / Pacemaker for your target failover the `vpd_unit_serial` and `iblock` number must both be set in the `iSCSILogicalUnit` OCF provider: - The `iblock` number is configured with `lio_iblock=` - [See iSCSILogicalUnit#L217](https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/iSCSILogicalUnit#L217) - The `vpd_unit_serial` is configured with `scsi_sn=` - [See iSCSILogicalUnit#L138](https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/iSCSILogicalUnit#L138) Here is an example of a target and lun configured with `pcs`: ``` Resource: iscsi_target_r2 (class=ocf provider=heartbeat type=iSCSITarget) Attributes: iqn=iqn.2003-01.org.linux-iscsi.pm-san.x8664:sn.ca7d7b33c731 portals=10.50.42.75:3260 implementation=lio-t additional_parameters="MaxConnections=100 AuthMethod=None InitialR2T=No MaxOutstandingR2T=64" Operations: monitor on-fail=restart interval=30s timeout=20s (iscsi_target_r2-monitor-30s) start on-fail=restart interval=0 timeout=20s (iscsi_target_r2-start-0) stop on-fail=restart interval=0 timeout=20s (iscsi_target_r2-stop-0) Resource: iscsi_lun_r2 (class=ocf provider=heartbeat type=iSCSILogicalUnit) Attributes: target_iqn=iqn.2003-01.org.linux-iscsi.pm-san.x8664:sn.ca7d7b33c731 scsi_sn=633c5643 lun=1 lio_iblock=2 path=/dev/drbd2 allowed_initiators="iqn.2015-05.com.example:51e1fb93" implementation=lio-t Operations: monitor on-fail=restart interval=30s timeout=10s (iscsi_lun_r2-monitor-30s) start on-fail=restart interval=0 timeout=20s (iscsi_lun_r2-start-0) stop on-fail=restart interval=0 timeout=20s (iscsi_lun_r2-stop-0) Resource: iscsi_conf_r2 (class=ocf provider=heartbeat type=anything) Attributes: binfile=/usr/sbin/iscsi_iscsi_conf_r2.sh stop_timeout=3 ``` If you happen to be using Puppet for your Pacemaker configuration it might look a bit like this: ``` cs_primitive { "$iscsi_target_primitive": primitive_class => 'ocf', primitive_type => 'iSCSITarget', provided_by => 'heartbeat', parameters => { 'iqn' => "$iscsi_iqn", 'portals' => "${iscsi_vip}:3260", 'implementation' => 'lio-t', 'additional_parameters' => 'MaxConnections=100 AuthMethod=None InitialR2T=No MaxOutstandingR2T=64', }, operations => { 'monitor' => { 'timeout' => '20s', 'interval' => '30s','on-fail' => "restart"}, 'start' => { 'timeout' => '20s', 'interval' => '0','on-fail' => "restart"}, 'stop' => { 'timeout' => '20s', 'interval' => '0','on-fail' => "restart"}, }, require => [Cs_primitive["$ip_primitive"],Package['targetcli'],Service['pacemaker']], } cs_primitive { "$iscsi_lun_primitive": primitive_class => 'ocf', primitive_type => 'iSCSILogicalUnit', provided_by => 'heartbeat', parameters => { 'target_iqn' => $iscsi_iqn, 'scsi_sn' => $scsi_sn, 'lun' => '1', 'lio_iblock' => $lio_iblock, 'path' => $drbd_path, 'allowed_initiators' => $allowed_initiators, 'implementation' => 'lio-t'}, operations => { 'monitor' => { 'timeout' => '10s', 'interval' => '30s','on-fail' => "restart" }, 'start' => { 'timeout' => '20s', 'interval' => '0' ,'on-fail' => "restart" }, 'stop' => { 'timeout' => '20s', 'interval' => '0' ,'on-fail' => "restart" }, }, require => [Cs_primitive["$iscsi_target_primitive"],Service['pacemaker']], } ``` - [How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync](https://smcleod.net/2015/11/how-to-cluster-and-failover-almost-anything-an-intro-to-pacemaker-and-corosync/index.md): How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync --- author: "Sam McLeod" categories: [ Tech ] date: "2015-11-09T00:00:00Z" aliases: - /tech/2015/11/09/intro-to-pacemaker-and-corosync/ images: ["tim-napier-181584.jpg"] cover: image: " tim-napier-181584.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - clustering title: How to cluster and failover (almost) anything - An intro to Pacemaker and Corosync --- ## Slides [![Click to Start Slides]({{< ref "/" >}}/img/san/cluster_anything_screenshot.jpg)](https://media.githubusercontent.com/media/sammcj/smcleod_files/refs/heads/master/slides/cluster_anything.pdf) ## Failover Demo [![Click to Start Failover Video]({{< ref "/" >}}/img/san/lcmcpcmk.png)](https://vimeo.com/sammcj/review/133110890/6f4900c090) - [SAN Intro](https://smcleod.net/2015/10/san-intro/index.md): SAN Intro --- author: "Sam McLeod" categories: [ Tech, Talks ] date: "2015-10-07T00:00:00Z" aliases: - /tech/2015/10/07/san-intro-talk/ images: ["elliot-sloman-199291.jpg"] cover: image: " elliot-sloman-199291.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - clustering - talk - slides - hardware title: SAN Intro --- {{< vimeo 141612064 >}} - [Replacing Junos Pulse with OpenConnect](https://smcleod.net/2015/09/replacing-junos-pulse-with-openconnect/index.md): Replacing Junos Pulse with OpenConnect --- author: "Sam McLeod" categories: [ Tech ] date: "2015-09-22T00:00:00Z" aliases: - /tech/2015/09/22/replacing-junos-pulse-with-openconnect/ images: ["backdrop-zbuildings.jpg"] cover: image: " backdrop-zbuildings.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - networking title: Replacing Junos Pulse with OpenConnect --- In an attempt to avoid using the Juniper Pulse (Now Pulse Secure) VPN client we tried OpenConnect but found that DNS did not work correctly when connected to the VPN. This bug has now been resolved recently but has not made it's way into a new build, in fact there have been no releases for 6 months. Luckily the OpenConnect was not too difficult to build from source. ## Build OpenConnect on OSX ### Remove old openconnect and install deps ```shell brew remove openconnect brew install libxml2 lzlib openssl libtool libevent ``` ### Build openconnect ```shell wget git.infradead.org/users/dwmw2/openconnect.git/snapshot/0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz tar zxvf 0f1ec30d17aa674142552e275bf3fac30d891b39.tar.gz cd openconnect-0f1ec30 LIBTOOLIZE=glibtoolize ./autogen.sh PATH=/usr/local/opt/gettext/bin:$PATH ./configure make make install ``` ### To connect ```shell sudo openconnect --juniper -u myusername www.myserver.com ``` If you're comfortable with allowing admin users to run openconnect without entering a sudo password, add the following using `sudo visudo`: ```shell %admin ALL=(ALL) NOPASSWD: /usr/local/sbin/openconnect ``` - [SSD Storage - Two Months In Production](https://smcleod.net/2015/09/ssd-storage-two-months-in-production/index.md): SSD Storage - Two Months In Production --- author: "Sam McLeod" categories: [ Tech ] date: "2015-09-13T00:00:00Z" aliases: - /tech/2015/09/13/ssd-storage-two-months-in-prod/ images: ["containers-1.jpg"] cover: image: " containers-1.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware title: SSD Storage - Two Months In Production --- Over the last two months I've been running selected IO intensive servers off the the SSD storage cluster, these hosts include (among others) our: - Primary Puppetmaster - Gitlab server - Redmine app and database servers - Nagios servers - Several Docker database host servers ### Reliability We haven't had any software or hardware failures since commissioning the storage units. During this time we have had 3 disk failures on our HP StoreVirtual SANs that have required us to call the supporting vendor and replace failed disks. We have performed a great deal of live cluster failovers without any noticeable interruption to services and with no unexpected results. ### Expected findings #### Significantly higher disk and CPU throughput when required ![]({{< ref "/" >}}/img/san/pm-docker-pg-01_pgsql_query_time.jpg) ![]({{< ref "/" >}}/img/san/pm-docker-pg-01_iops.jpg) #### IOWait on VMs greatly reduced ![]({{< ref "/" >}}/img/san/iowait_gitlab.jpg) ![]({{< ref "/" >}}/img/san/int-redmine_iowait.jpg) ![]({{< ref "/" >}}/img/san/puppetmaster_disk_load.jpg) #### VM IO latency has been reduced 300-500% Before: ```shell 4096 bytes from . (ext4 /dev/mapper/debian-root): request=1 time=1.6 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=2 time=4.7 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=3 time=3.7 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=4 time=4.7 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=5 time=4.8 ms ``` After: ```shell 4096 bytes from . (ext4 /dev/mapper/debian-root): request=1 time=0.5 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=2 time=0.7 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=3 time=0.9 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=4 time=0.7 ms 4096 bytes from . (ext4 /dev/mapper/debian-root): request=5 time=0.6 ms ``` ![]({{< ref "/" >}}/img/san/gitlab_disk_latency.jpg) #### Increased reliability of performance data Previously many hosts that we were monitoring load related statistics on previous had broken [PNP4Nagios](https://docs.pnp4nagios.org/pnp-0.6/start) graphs as they were so starved for disk IO that checks would time out or fail to reliably return performance information, this has been resolved on the new storage. ### Unexpected findings #### Decreased network latency On hosts we've moved to the new storage we've noticed a significant decrease in ping latency to these hosts. I haven't investigated this but it's not something I was expecting to see. #### Increased SSH / login performance I didn't think this would be very noticeable but it's become blatantly obvious when you're sshing into a server running on the new storage. While this is a 'nice to have' from an admin perspective - it makes a huge difference to remote automation such as puppet runs etc... This likely a combination of the time it takes to write the logs and the decreased network latency mentioned above. - [OS X Software Update Channels For Betas](https://smcleod.net/2015/09/os-x-software-update-channels-for-betas/index.md): OS X Software Update Channels For Betas --- author: "Sam McLeod" categories: [ Tech ] date: "2015-09-01T00:00:00Z" aliases: - /tech/2015/09/01/os-x-software-update-channels/ images: ["rishi-deep-424424.jpg"] cover: image: " rishi-deep-424424.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech title: OS X Software Update Channels For Betas --- ### Set update channel to receive developer beta update ```shell sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11seed-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz ``` ### Set update channel to receive public beta update ```shell sudo softwareupdate --set-catalog https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz ``` ### List available updates ```shell sudo softwareupdate --list ``` ### Set update channel to receive default, stable updates ```shell sudo softwareupdate --clear-catalog ``` ### Show current settings ```shell defaults read /Library/Preferences/com.apple.SoftwareUpdate.plist ``` ### Write setting manually ```shell defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL https://swscan.apple.com/content/catalogs/others/index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.gz ``` - [iSCSI Benchmarking](https://smcleod.net/2015/07/iscsi-benchmarking/index.md): iSCSI Benchmarking --- author: "Sam McLeod" categories: [ Tech ] date: "2015-07-24T00:00:00Z" aliases: - /tech/2015/07/24/scsi-benchmarking/ images: ["datacentre-1.jpg"] cover: image: " datacentre-1.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware title: iSCSI Benchmarking --- The following are benchmarks from our testings of our iSCSI SSD storage. ### 67,300 read IOP/s on a VM on iSCSI - (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM) - Per VM and scales to 1,000,000 IOP/s total ```shell root@dev-samm:/mnt/pmt1 128 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=read test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128 2.0.8 Starting 1 process bs: 1 (f=1): [R] [55.6% done] [262.1M/0K /s] [67.3K/0 iops] [eta 00m:04s] ``` ### 38,500 random 4k write IOP/s on a VM on iSCSI - (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM) - Per VM and scales to 700,000 IOP/s total ```shell root@dev-samm:/mnt/pmt1 # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=128 --size=2G --readwrite=randwrite test: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128 2.0.8 Starting 1 process bs: 1 (f=1): [w] [26.3% done] [0K/150.2M /s] [0 /38.5K iops] [eta 00m:14s] ``` ### Raw device latency on storage units - Intel DC3600 1.2T PCIe NVMe ```shell root@s1-san6:/proc # ioping /dev/nvme0n1p1 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=1 time=104 us 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=2 time=83 us 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=3 time=51 us 4.0 KiB from /dev/nvme0n1p1 (device 1.1 TiB): request=4 time=71 us ``` - SanDisk SDSSDXPS960G SATA ```shell root@pm-san5:/proc # ioping /dev/sdc 4.0 KiB from /dev/sdc (device 894.3 GiB): request=1 time=4.2 ms 4.0 KiB from /dev/sdc (device 894.3 GiB): request=2 time=4.1 ms 4.0 KiB from /dev/sdc (device 894.3 GiB): request=3 time=4.1 ms 4.0 KiB from /dev/sdc (device 894.3 GiB): request=4 time=4.1 ms ``` - Micron_M600_MTFDDAK1T0MBF SATA ```shell root@pm-san5:/proc # ioping /dev/sdf 4.0 KiB from /dev/sdf (device 953.9 GiB): request=1 time=157 us 4.0 KiB from /dev/sdf (device 953.9 GiB): request=2 time=190 us 4.0 KiB from /dev/sdf (device 953.9 GiB): request=3 time=65 us 4.0 KiB from /dev/sdf (device 953.9 GiB): request=4 time=181 us ```shell ## Latency on the a VM - (Disk -> LVM -> MDADM -> DRBD -> iSCSI target -> Network -> XenServer iSCSI Client -> VM) ```shell root@dev-samm:/mnt 127 # ioping pmt1/ 4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=1 time=0.6 ms 4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=2 time=0.7 ms 4096 bytes from pmt1/ (ext4 /dev/xvdb1): request=3 time=0.7 ms --- pmt1/ (ext4 /dev/xvdb1) ioping statistics --- 3 requests completed in 2159.1 ms, 1508 iops, 5.9 mb/s min/avg/max/mdev = 0.6/0.7/0.7/0.1 ms root@dev-samm:/mnt # ioping pmt2/ 4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=1 time=0.6 ms 4096 bytes from pmt2/ (ext4 /dev/xvdc1): request=2 time=0.8 ms --- pmt2/ (ext4 /dev/xvdc1) ioping statistics --- 2 requests completed in 1658.4 ms, 1470 iops, 5.7 mb/s min/avg/max/mdev = 0.6/0.7/0.8/0.1 ms root@dev-samm:/mnt # ioping pmt3/ 4096 bytes from pmt3/ (ext4 /dev/xvde1): request=1 time=0.6 ms 4096 bytes from pmt3/ (ext4 /dev/xvde1): request=2 time=0.9 ms 4096 bytes from pmt3/ (ext4 /dev/xvde1): request=3 time=0.9 ms ``` ![](/img/san/lcmcpcmk.png) ![](/img/san/supermicrox2.jpg) - [Delayed Serial STONITH](https://smcleod.net/2015/07/delayed-serial-stonith/index.md): Delayed Serial STONITH --- author: "Sam McLeod" categories: [ Tech ] date: "2015-07-21T00:00:00Z" aliases: - /tech/2015/07/21/rcd-stonith/ images: ["padlock.jpg"] cover: image: " padlock.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware photos: true title: Delayed Serial STONITH --- A modified version of [John Sutton's](http://www.scl.co.uk/rcd_serial/README.rcd_serial) rcd_serial cable coupled with our Supermicro reset switch hijacker: ![]({{< ref "/" >}}/img/san/rcd_serial.jpg) This works with the rcd_serial fence agent [plugin](https://github.com/ClusterLabs/fence-agents/tree/master/agents/rcd_serial). Reasons `rcd_serial` makes for a very good [STONITH](https://en.wikipedia.org/wiki/STONITH) mechanism: - It has no dependency on power state. - It has no dependency on network state. - It has no dependency on node operational state. - It has no dependency on external hardware. - It costs less that $5 + time to build. - It is incredibly simple and reliable. Essentially the most common STONITH agent type in use is probably those that control UPS / PDUs, while this sounds like a good idea in theory there are a number of issues with relying on a UPS / PDU: - Units that have remote power control over individual outlets are very expensive and if an upgrade is undertaken a rake-wide outage may be required depending on the existing infrastructure. - Often these units are managed via the network, requiring the network and all that that entails to be functioning as expected. It also may require an additional NIC that may or may not fit into your storage units. - There are almost always two PDUs / UPSs to manage, until very recently the PDU STONITH agents only supported sending an action to a single unit, while they now support sending them to two units in modern packages there a number of situations that are complex to manage and predict - i.e. what if one unit responds, cuts the power and the other doesn't? Who's in charge? Do we fail over? etc... that's a LOT of logic for a STONITH action. - I've seen several PDUs fail, it's not pretty and often the management interface is the first thing to go. --- _The following was v1 of the devices design, a post on the update v2/v3 design can be found [here]({{< ref "posts/2016-07-04-update-rcd-stonith-design" >}})_ [Adam Coy's](https://www.linkedin.com/pub/adam-coy/5/989/888) slightly modified version of the circuit that includes an indicator LED and an [optocoupler](https://en.wikipedia.org/wiki/Opto-isolator): ![]({{< ref "/" >}}/img/san/optoCircuit.jpg) Example of where our Supermicro reset hijack connects on the target node: ![]({{< ref "/" >}}/img/san/reboot_jumper.jpg) Image of the rear of a two node cluster with networking and rcd_serial STONITH (v1) connectors attached: ![]({{< ref "/" >}}/img/san/storage_with_stonith_rear.jpg) --- ### Availability _This has since been added to RHEL/CentOS and the official Clusterlabs resource agents_ aliases: - At present the rcd_serial STONITH agent is available as part of the [`cluster-glue`](https://packagecloud.io/s_mcleod/pacemaker/packages/el/7/cluster-glue-1.0.12-1.16.1.x86_64.rpm) package, `cluster-glue` is not available in RHEL/CentOS but can be obtained from [OpenSUSE's CentOS7 Repo](ftp://rpmfind.net/linux/opensuse/factory/repo/oss/suse/x86_64/cluster-glue-1.0.12-19.2.x86_64.rpm) or [my own mirror](https://packagecloud.io/s_mcleod/pacemaker). (Tested with CentOS 7). I have an open ticket with [RedHat](https://bugzilla.redhat.com/show_bug.cgi?id=1240868) regarding the fact that their pacemaker rpm is built without the `--with stonithd` flag which allows this to work without with their version of Pacemaker. The long term solution is to get `rcd_serial` migrated to the new Pacemaker agent [API](https://fedorahosted.org/cluster/wiki/FenceAgentAPI) ([or is it this one?](http://clusterlabs.org/doxygen/pacemaker/2927a0f9f25610c331b6a137c846fec27032c9ea/stonith-ng_8h.html)) system. - [Video - Cluster Failover Performance Demo](https://smcleod.net/2015/07/video-cluster-failover-performance-demo/index.md): Video - Cluster Failover Performance Demo --- author: "Sam McLeod" categories: [ Tech, Talks ] date: "2015-07-12T00:00:00Z" aliases: - /tech/2015/07/12/video-cluster-failover-demo/ images: ["tim-napier-181584.jpg"] cover: image: " tim-napier-181584.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware - slides - talks title: Video - Cluster Failover Performance Demo --- {{< youtube "GvAV990z2Us" >}} - [CentOS 7 and HA](https://smcleod.net/2015/07/centos-7-and-ha/index.md): CentOS 7 and HA --- author: "Sam McLeod" categories: [ Tech ] date: "2015-07-07T00:00:00Z" aliases: - /tech/2015/07/07/centos-7-and-ha/ images: ["billy-huynh-278252.jpg"] cover: image: " billy-huynh-278252.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - clustering title: CentOS 7 and HA --- First some background... One of the many lessons I've learnt from my Linux HA / Storage clustering project is that the Debian HA ecosystem is essentially broken, We reached the point where packages were too old, too buggy or in Debian 8's case - outright missing. In the past I was very disappointed with RHEL/CentOS 5 / 6 and (until now) have been quite satisfied with Debian as a stable server distribution with historicity more modern packages and kernels. **_I feel that CentOS / RHEL 7 has changed the game.*_** (When combined with ElRepo or EPEL that provide wide array of modern packages) It is simply light years ahead of it's predecessor, resolves a lot of the issues we've had with Debian Jessie and after thorough testing of Debian 7,8 Centos 6,7 I have decided to employ CentOS 7 as the base OS for the storage cluster. CentOS / RHEL 7 are missing a few critical packages in the cluster HA space 1. The available pacemaker package was built without `--with-stonith`. This means there is no support for legacy STONITH plugins, many of these are heavily relied upon and do not have replacements in the new plugin system. 2. `cluster-glue` is missing / has been deprecated. The cluster-glue package provides a lot of very useful resource agents / plugins which again, are missing from the new pacemaker builds. 3. `crmsh` is no longer available and has been replaced by `pcs`. pcs is great, but lots of very useful tools still use crmsh including the [puppet-corosync](https://github.com/puppet-community/puppet-corosync) module which is fantastic for bootstrapping clusters. ## Short term solution ### Pacemaker I have recompiled CentOS 7's pacemaker package changing only one thing - I added `--with-stonith` to the rpmbuild command. - This is available from my [packagecloud repo](https://packagecloud.io/s_mcleod/pacemaker). ### Cluster-Glue OpenSUSE provides the `cluster-glue` and `cluster-glue-libs` from their [CentOS 7 repository](http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/) - This is also available from my [packagecloud repo](https://packagecloud.io/s_mcleod/pacemaker). ### CRMSH OpenSUSE also provides `crmsh` from their [CentOS 7 repository](() - This is also available from my [packagecloud repo](https://packagecloud.io/s_mcleod/pacemaker). ## Long term solution ### Update STONITH / Fencing agents Obviously the best solution however have a feeling that it will take some time for all the maintainers of the existing STONITH / fencing agent developers to port their code to the new framework. ### Request package to be built --with stonithd - [RedHat BugZilla](https://bugzilla.redhat.com/show_bug.cgi?id=1240868) ### Host required packages I have been logging requests to various third party RHEL repos asking if they would the missing packages. - [GhettoForge](http://ghettoforge.org/index.php/Usage) are currently considering hosting `pacemaker` built `--with stonithd` in their gf-plus repo. - [RPMFusion](http://rpmfusion.org/Wishlist?action=diff&rev1=401&rev2=402) - [ELRepo](http://elrepo.org/bugs/view.php?id=579) If they don't feel they have the resources or want to add them in for whatever reason, I will continue to host the packages myself on [Packagecloud](https://packagecloud.io/s_mcleod/pacemaker) and likely somewhere else as a fall-back. ``` Last updated: Tue Jul 7 11:59:25 2015 Last change: Tue Jul 7 10:57:16 2015 Stack: corosync Current DC: s1-san5 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 16 Resources configured Online: [ s1-san5 s1-san6 ] Clone Set: ping_gateway-clone [ping_gateway] Started: [ s1-san5 s1-san6 ] Master/Slave Set: ms_drbd_r0 [drbd_r0] Masters: [ s1-san5 ] Slaves: [ s1-san6 ] ip_r0 (ocf::heartbeat:IPaddr2): Started s1-san5 iscsi_target_r0 (ocf::heartbeat:iSCSITarget): Started s1-san5 iscsi_lun_r0 (ocf::heartbeat:iSCSILogicalUnit): Started s1-san5 stonith_s1-san5 (stonith:rcd_serial): Started s1-san6 stonith_s1-san6 (stonith:rcd_serial): Started s1-san5 Master/Slave Set: ms_drbd_r1 [drbd_r1] Masters: [ s1-san5 ] Slaves: [ s1-san6 ] ip_r1 (ocf::heartbeat:IPaddr2): Started s1-san5 iscsi_target_r1 (ocf::heartbeat:iSCSITarget): Started s1-san5 iscsi_lun_r1 (ocf::heartbeat:iSCSILogicalUnit): Started s1-san5 iscsi_conf_r1 (ocf::heartbeat:anything): Started s1-san5 iscsi_conf_r0 (ocf::heartbeat:anything): Started s1-san5 ``` - [SSD Storage Cluster - Update and Diagram](https://smcleod.net/2015/06/ssd-storage-cluster-update-and-diagram/index.md): SSD Storage Cluster - Update and Diagram --- author: "Sam McLeod" categories: [ Tech ] date: "2015-06-17T00:00:00Z" series: [ Storage ] aliases: - /tech/2015/06/17/ssd-storage-cluster-diagram/ - /tech/ssd-storage-cluster-diagram/ - /tech/2015/06/17/ssd-storage-cluster-diagram.html images: ["backdrop-feetsky.jpg"] cover: image: " backdrop-feetsky.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware title: SSD Storage Cluster - Update and Diagram --- Due to several recent events beyond my control I'm a bit behind on the project - hence the lack of updates which I apologise for. The goods news is that I'm back working to finish off the clusters and I'm happy to report that all is going to plan. Here is the final digram of the two-node cluster design: ![]({{< ref "/" >}}/img/san/diagram.png) Plain text version available [here](https://gist.github.com/sammcj/0503007ceb5038a0de3c) ![]({{< ref "/" >}}/img/san/lcmcpcmk.png) This was generated from the [LCMC](http://github.com/rasto/lcmc) tool (beware - it's java!). More on this soon... - [Video - Storage Cluster Failover Demo](https://smcleod.net/2015/05/video-storage-cluster-failover-demo/index.md): Video - Storage Cluster Failover Demo --- author: "Sam McLeod" categories: [ Tech, Talks ] date: "2015-05-14T00:00:00Z" aliases: - /tech/2015/05/15/video-storage-cluster-failover-demo/ images: ["elliot-sloman-199291.jpg"] cover: image: " elliot-sloman-199291.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles series: [ Storage ] tags: - tech - storage - hardware - slides - talks title: Video - Storage Cluster Failover Demo --- A brief demonstration of the failover and recovery process on the storage clusters I've been building. {{}} - [Talk - High Performance Software Defined Storage](https://smcleod.net/2015/04/talk-high-performance-software-defined-storage/index.md): Talk - High Performance Software Defined Storage --- author: "Sam McLeod" categories: [ Tech ] date: "2015-04-15T00:00:00Z" aliases: - /tech/2015/04/15/talk-high-perf-sds-ictalk/ images: ["robin-pierre-323861.jpg"] cover: image: " robin-pierre-323861.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles series: [ Storage ] tags: - tech - storage - talk - hardware - slides title: Talk - High Performance Software Defined Storage --- A high level talk from Infracoders Melbourne on 12/04/2015. [![Click to Start Slides]({{< ref "/" >}}/img/san/supermicrox2.jpg)](https://media.githubusercontent.com/media/sammcj/smcleod_files/refs/heads/master/slides/lightning_san.pdf) [![Click to Start Slides]({{< ref "/" >}}/img/san/ic-slides-image.png)](https://media.githubusercontent.com/media/sammcj/smcleod_files/refs/heads/master/slides/lightning_san.pdf) There's also a low quality recording available here: [![Click to Start Video]({{< ref "/" >}}/img/san/ic-sds-yt-thumb.png)](https://youtu.be/VAdqurA2zQ4?t=198) Related posts: [Building a high performance SSD SAN - Part 1](https://smcleod.net/building-a-high-performance-ssd-san/) - [Continuous integration for the Linux Kernel - Built within Docker](https://smcleod.net/2015/03/continuous-integration-for-the-linux-kernel-built-within-docker/index.md): Continuous integration for the Linux Kernel - Built within Docker --- author: "Sam McLeod" categories: [ Tech, CICD ] date: "2015-03-15T00:00:00Z" aliases: - /tech/2015/03/15/kernel-ci/ - /continuous-integration-for-the-linux-kernel-built-within-docker images: ["backdrop-zbuildings.jpg"] cover: image: " backdrop-zbuildings.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - DevOps - cicd title: Continuous integration for the Linux Kernel - Built within Docker --- Linux Kernel CI for Debian ![]({{< ref "/" >}}/img/build-passing.png) [Github: sammcj/kernel-ci](https://github.com/sammcj/kernel-ci) Those of us using technologies such as Docker and BTRFS or simply trying to gain a performance edge on the competition have a lot to gain from the features and performance of recent Kernel updates (especially from 3.18 onwards). 'Enterprise' Linux distributions such as RHEL & variants are concerningly out of date when comes to the Kernel. Many people seem to have forgotten what Linux is... _Linux IS the Kernel_. Someone said to me recently 'Why do I need a more modern Kernel? - It never gives us problems!' later in the conversation they alluded to a number of performance issues they were experiencing with both a database platform and with some modern containerised applications they were trying to run. _Both of those issues were resolved several years ago in the Kernel which was newer than what they were running_. There's a fine line between bleeding edge and being at the front of the game, if there's one thing I've learnt about this over the past few years it's that the Kernel gets better with age, not worse. In addition to CI of modern Kernel images, I've also integrated (optional) patching for GRSecurity fixes - GRSecurity is a great line of defence for the Kernel and prevents applications doing many silly things they shouldn't be allowed to do (Can anyone say Nodes?...). - Uploads publicly accessible Debian Kernel Packages to packagecloud.io - Includes Kernel Watcher that detects new stable kernel releases and triggers builds. - Supports patching the Kernel with GRSecurity - Tested with Gitlab-CI and Travis-CI but should work on any CI system. - Runs in an isolated and disposble docker container. - No root access required when building with Docker. - Both the build and the kernels Work with Debian Wheezy (7) and Jessie (8). - Supports uploading built packages to a remote server and adding them to reprepro ![]({{< ref "/" >}}/img/travis.png) ![]({{< ref "/" >}}/img/kernel_bash.jpg) - [Xen Orchestra Docker Image](https://smcleod.net/2015/02/xen-orchestra-docker-image/index.md): Xen Orchestra Docker Image --- author: "Sam McLeod" categories: [ Tech ] date: "2015-02-26T00:00:00Z" aliases: - /tech/2015/05/26/xen-orchestra-docker-image/ images: ["backdrop-insidebuilding.jpg"] cover: image: " backdrop-insidebuilding.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - xen title: Xen Orchestra Docker Image --- Docker config to setup XO which is a web interface to visualize and administrate your XenServer (or XAPI enabled) hosts [Github: sammcj/docker-xen-orchestra](https://github.com/sammcj/docker-xen-orchestra) ## Running the app Updates are pushed to the Docker Hub's automated build service: * ## From Docker Hub ```shell docker pull sammcj/docker-xen-orchestra docker run -d -p 8000:80 sammcj/docker-xen-orchestra ``` ## Building ```shell git clone https://github.com/sammcj/docker-xen-orchestra.git cd docker-xen-orchestra # Edit whatever config you want to change docker build -t xen-orchestra . ``` See for information on Xen Orchestra - [Delete Government-Linked Certificate Authorities in OSX](https://smcleod.net/2015/02/delete-government-linked-certificate-authorities-in-osx/index.md): Delete Government-Linked Certificate Authorities in OSX --- author: "Sam McLeod" categories: [ Tech, Security-Privacy ] date: "2015-02-23T00:00:00Z" aliases: - /tech/2015/02/23/delete-known-governmentlinked-certificate-authorities-in-os-x/ images: ["backdrop-italian-steps.jpg"] cover: image: " backdrop-italian-steps.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - security - privacy title: Delete Government-Linked Certificate Authorities in OSX --- [Inspired by http://zitseng.com/archives/7489](http://zitseng.com/archives/7489) * [Source (Github)](https://github.com/sammcj/delete-unknown-root-ca) **WARNINGS** * Do not run unless you understand what this is doing * The CA system is broken by design - This is not a fix for that * This is merely a band-aid for those interested or concerned about these root CAs ## Usage ```shell chmod +x delete_gov_roots.sh ./delete_gov_roots.sh ``` You'll be prompted for your password as root access is required to delete system-wide root certs. ![sha1](https://cloud.githubusercontent.com/assets/862951/6326428/a261ae24-bba5-11e4-9f69-5aeb36257077.png) ## See Also * * * * * * * - [Building a high performance SSD SAN - Part 1](https://smcleod.net/2015/02/building-a-high-performance-ssd-san-part-1/index.md): Building a high performance SSD SAN - Part 1 --- author: "Sam McLeod" toc: true categories: [ Tech ] date: "2015-02-16T00:00:00Z" aliases: - /tech/2015/02/16/building-a-high-performance-ssd-san/ - /building-a-high-performance-ssd-san/ images: ["melbourne.jpg"] cover: image: " melbourne.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles series: [ Storage ] tags: - tech - storage - hardware photos: true title: Building a high performance SSD SAN - Part 1 --- ![]({{< ref "/" >}}/img/san/graphs.png) Over the coming month I will be architecting, building and testing a modular, high performance SSD-only storage solution. I'll be documenting my progress / findings along the way and open sourcing all the information as a public guide. With recent price drops and durability improvements in solid state storage now is better time than any to ditch those old magnets. Modular server manufacturers such as SuperMicro have spent large on R&D thanks to the ever growing requirements from cloud vendors that utilise their hardware. ## The State Of Enterprise Storage Companies often settle for off-the-shelf large name storage products from companies based on several, often misguided assumptions: * That enterprise in the product name = reliability * That the blame from product / system failure can be outsourced * That vendors provide specialist engineers to support such complicated (and expensive) products * That that time and cost of building a modular storage solution tailored to their needs would be time consuming to design and manage At the end of the day we don't trust vendors to design our servers - why would we trust them to design our storage? A great quote on Wikipedia under 'enterprise storage': *"You might think that the hardware inside a [SAN][1] is vastly superior to what can be found in your average server, but that is not the case. EMC (the market leader) and others have disclosed more than once that "the goal has always to been to use as much standard, commercial, off-the-shelf hardware as we can". So your SAN array is probably nothing more than a typical Xeon server built by Quanta with a shiny bezel. A decent professional 1 [TB][2] drive costs a few hundred dollars. Place that same drive inside a SAN appliance and suddenly the price per terabyte is multiplied by at least three, sometimes even 10! When it comes to pricing and [vendor lock-in][3] you can say that storage systems are still stuck in the "mainframe era" despite the use of cheap off-the-shelf hardware."* It's the same old story, if you've got lots of money and you don't care about how you spend it or translating those savings onto your customers - sure *buy the ticket, take the ride* \- get a unit that comes with a flash logo, a 500 page brochure, licensing requirements and a greasy sales pitch. ## Our Needs Storage performance always seems to be our bottleneck at Infoxchange, we run several high-performance high-concurrency applications with large databases and complex reporting. We're grown (very) fast and with that spending too much on off-the-shelf storage solutions, we have a requirement to self-host most of our products securely within our own control, on our hardware and need to be flexible to meet current and emerging security requirements. I have been working on various proof-of-concepts which have lead to our decision to proceed with our own modular storage system tailored to our requirements. ## Requirements * Reliability above all else * SSD units must be durable * Network and iSCSI failover must be on-par with commercial products (if not better) * Multiple levels of provable redundancy * RAID * Cross hardware-replication * Easy IP and iSCSI failover using standard tools * 1RU rack height per unit * 100% SSD only - no spindles will be hurt in the making of this journey! * Each unit to provide up to 450,000 IOP/s read performance on tier 1 storage * Provide up to 2.5GB/s read performance and 1.5GB/s write performance on tier 1 storage * Each unit to provide up to 400,000 IOP/s read performance on tier 2 storage * Provide up to 1.2GB/s read performance and 1.2GB/s write performance on tier 2 storage * 20Gbit of redundant network connectivity per unit * Two tiers of SSD storage performance (PCIe & SATA) * Easily monitorable with standard tools * Use no proprietary RAID hardware * Come with 3 years of hardware warranty cover * Outperform all proprietary storage solutions costing twice the price or more * Deployable and manageable by any sysadmin and require no specialised storage administrators * Easily updatable for the latest security patches, features etc... * Highly customisable and easily upgradable to larger / faster storage in the future * Require significantly less energy and cooling over traditional storage units * Offer at-rest encryption if required * _**Cost less than $9.5K USD per node**_ ## Software |-|-|-| |:-------------|:-------------|:-----| | Operating System | Debian | Debian is our OS of choice, it has newer packages than RedHat variants and is incredibly stable | | RAID | MDADM | For SSDs hardware RAID cards can often be their undoing - they simply can't keep up and quickly become the bottleneck in the system. MDADM is mature and very flexible | | Node-to-Node Replication | DRBD | | NIC Bonding | LACP | | IP Failover | Pacemaker | We'll probably also use a standard VM somewhere on our storage network for quorum | | Monitoring | Nagios | | Storage Presentation | Open-iSCSI | | Kernel | Latest Stable (Currently 3.18.7) | Debian Backports currently has Kernel 3.16, however we do daily CI builds of the latest kernel stable source for certain servers and this may be a good use case for them due the SCSI bus bypass for NVMe introduced in 3.18+ | We're going to start with a two node cluster, we want to keep rack usage to a minimum so I'm going to go with a high density 1RU build. The servers themselves don't need to be particularly powerful which will help us keep the costs down. Easily the most expensive components are the 1.2TB PCIe SSDs - but the performance and durability of these units can't be overlooked, we're going to have a second performance tier constructed of high end SATA SSDs in RAID10. Of course if you wanted to reduce price further the PCIe SSDs could be left out or purchased at a later date. ## Hardware |-|-|-| |:-------------|:-------------|:-----| |**Base Server** | SuperMicro SuperServer 1028R-WTNRT |2x 10GbE, NVMe Support, Dual PSU, Dual SATA DOM Support, 3x PCIe, 10x SAS/SATA HDD Bays | | **CPU** | 2x Intel Xeon E5-2609 v3 | We shouldn't need a very high clock speed for our SAN, but it's worth getting the newer v3 processor range for the sake of future proofing. | **RAM** | 32GB DDR4 2133Mhz | Again, we don't need that much RAM, however it will be used for disk caching but 32GB should be more than enough and can be easily upgraded at a later date. | | **PCIe SSD** | 2x 1.2TB Intel SSD DC P3600 Series (With NVMe) | This is where the real money goes - the Intel DC P3600 and P3700 series really are top of the range, the critical thing to note is that they support NVMe which will greatly increase performance, they're backed by a 5 year warranty, these will be configured in RAID-1 for redundancy. | | **SATA SSD** | 8x SanDisk Extreme Pro SSD 480GB | The SanDisk Extreme Pro line is arguably the most reliable and highest performing SATA SSD on the market - backed by a 10 year warranty, these will be configured in RAID-10 for redundancy and performance. | | **OS SSD** | 2x 16GB MLC DOM | We don't need much space for the OS, just enough to keep vital logs and package updates, these will be configured in RAID-1 for redundancy. ![SuperMicro SuperServer 1028R-WTNRT]({{< ref "/" >}}/img/san/sm.jpg) ![SuperMicro SuperServer 1028R-WTNRT - mobo]({{< ref "/" >}}/img/san/mobo.jpg) ![1.2TB Intel SSD DC P3600 Series]({{< ref "/" >}}/img/san/intel.jpg) ![SuperMicro DOM](dom.jpg) ![SanDisk Extreme Pro SSD 480GB]({{< ref "/" >}}/img/san/ssd.jpg) ## AHCI vs NVMe NVMe is a relatively new technology which I'm very interested in making use of for these storage units. From [Wikipedia][4]: *"NVM Express has been designed from the ground up, capitalizing on the low latency and parallelism of PCI Express SSDs, and mirroring the parallelism of contemporary CPUs, platforms and applications. By allowing parallelism levels offered by SSDs to be fully utilized by host's hardware and software, NVM Express brings various performance improvements."* |-|AHCI|NVMe| |:------------- |:-------------|:-----| |Maximum queue depth | 1 command queue; 32 commands per queue | 65536 queues; 65536 commands per queue | | Uncacheable register accesses (2000 cycles each) | 6 per non-queued command; 9 per queued command | 2 per command | | MSI-X and interrupt steering | single interrupt; no steering | 2048 MSI-X interrupts | Parallelism and multiple threads | requires synchronization lock to issue a command | no locking | | Efficiency for 4 KB commands | command parameters require two serialized host DRAM fetches | gets command parameters in one 64 Bytes fetch | ## NVMe and the Linux Kernel *Intel published an NVM Express driver for Linux, It was merged into the Linux Kernel mainline on 19 March 2012, with the release of version 3.3 of the Linux kernel.* *A scalable block layer for high-performance SSD storage, developed primarily by_ [_Fusion-io][5]_ _engineers, was merged into the Linux kernel mainline in kernel version 3.13, released on 19 January 2014. This leverages the performance offered by SSDs and NVM Express, by allowing much higher I/O submission rates. With this new design of the Linux kernel block layer, internal queues are split into two levels (per-CPU and hardware-submission queues), thus removing bottlenecks and allowing much higher levels of I/O parallelisation.* Note the following: *As of version 3.18 of the Linux kernel, released on 7 December 2014,* [_VirtIO][6]_*block driver and the* [_SCSI][7]_*layer (which is used by Serial ATA drivers) have been modified to actually use this new interface; other drivers will be ported in the following releases.* Debian - our operating system of choice currently has kernel 3.16 available (using the official backports mirrors), however we do generate CI builds of the latest stable kernel for specific platforms - if you're interested on how we're doing that I have some information [here][8]. That's where I'm upto for now, the hardware will hopefully arrive in two weeks and I'll begin the setup and testing. ### Coming soon * Build experience / guide * Monitoring * Benchmarks * Failover configuration and testing * Software configurations (Including a Puppet module) * Ongoing experiences and application Stay tuned! ### Further reading * [1-2 year SSD wear on build boxes has been minimal](https://news.ycombinator.com/item?id=9052925) * From way back in 2011 - 'Velocity 2011: Artur Bergman - Artur on SSDs': {{< youtube "H7PJ1oeEyGg" >}} [1]: http://en.m.wikipedia.org/wiki/Storage_area_network "Storage area network" [2]: http://en.m.wikipedia.org/wiki/Terabyte "Terabyte" [3]: http://en.m.wikipedia.org/wiki/Vendor_lock-in "Vendor lock-in" [4]: http://en.wikipedia.org/wiki/NVM_Express [5]: http://en.wikipedia.org/wiki/Fusion-io [6]: http://en.wikipedia.org/wiki/VirtIO [7]: http://en.wikipedia.org/wiki/SCSI [8]: http://smcleod.net/continuous-integration-for-the-linux-kernel-built-within-docker - [Direct-Attach SSD Storage - Performance & Comparisons](https://smcleod.net/2015/02/direct-attach-ssd-storage-performance-comparisons/index.md): Direct-Attach SSD Storage - Performance & Comparisons --- author: "Sam McLeod" categories: [ Tech ] date: "2015-02-15T00:00:00Z" aliases: - /tech/2014/10/15/directattach-ssd-storage-performance-comparisons/ images: ["banner.jpg"] cover: image: " banner.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware title: Direct-Attach SSD Storage - Performance & Comparisons --- Further to my earlier post on XenServer storage performance with regards to directly attaching storage from the host, I have been analysing the performance of various SSD storage options. I have attached a HP DS2220sb storage blade to an existing server blade and compared performance with 4 and 6 SSD RAID-10 to our existing iSCSI SANs. While the P420i RAID controller in the DS2220sb is clearly saturated and unable to provide throughput much over 1,100MB/s - the IOP/s available to PostgreSQL are still a very considerably performance improvement over our P4530 SAN - in fact, 6 SSD's result in a 39.9x performance increase! Click the image below for the results: [![Click to Start Slides]({{< ref "/" >}}/img/san/bladedirectattach.png)](https://media.githubusercontent.com/media/sammcj/smcleod_files/refs/heads/master/slides/SSDvsSAN.pdf) - [Search - A Journey of Delivery on a Budget](https://smcleod.net/2015/02/search-a-journey-of-delivery-on-a-budget/index.md): Search - A Journey of Delivery on a Budget --- author: "Sam McLeod" categories: [ Tech ] date: "2015-02-15T00:00:00Z" aliases: - /tech/2014/11/04/search-a-journey-of-delivery-on-a-budget/ #description:Talk - Search - A Journey of Delivery on a Budget images: ["containers-3.jpg", "/img/misc/search.jpg"] cover: image: " /img/misc/search.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - clustering - DevOps - career title: Search - A Journey of Delivery on a Budget --- ### ["Search - A Journey of Delivery on a Budget"](https://smcleod.net/files/slides-Search-A-Journey-of-Delivery-on-a-Budget/) Presented a Melbourne Search - July 2014 - ["Search - A Journey of Delivery on a Budget"](https://smcleod.net/files/slides-Search-A-Journey-of-Delivery-on-a-Budget/) _(Click for slides)_ {{< youtube "RcXstZ4FzyE" >}} - [Talk - 24 Months](https://smcleod.net/2015/02/talk-24-months/index.md): Talk - 24 Months --- author: "Sam McLeod" categories: [ Tech, DevOps, Talks ] date: "2015-02-15T00:00:00Z" aliases: - /tech/2014/10/15/talk-24-months/ images: ["melbourne.jpg"] cover: image: " melbourne.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - talk - tech - DevOps - career title: Talk - 24 Months --- The way we work at Infoxchange has changed greatly. A retrospective journey into transforming Infoxchange's technology and culture over the past 24 months - presented a Melbourne DevOps - December 2014 [![Click to Download Slides]({{< ref "/" >}}/img/misc/24months.jpg)](https://github.com/sammcj/smcleod_files/blob/master/slides/24_months/24_Months.pdf?raw=true) - [The Best Of - 2014 Edition](https://smcleod.net/2015/02/the-best-of-2014-edition/index.md): The Best Of - 2014 Edition --- author: "Sam McLeod" categories: [ Tech, BestOf, Software ] date: "2015-02-15T00:00:00Z" aliases: - /tech/2015/01/01/the-best-of-2014-edition images: ["patrick-tomasso-71909.jpg"] cover: image: " patrick-tomasso-71909.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - software - bestof series: [ BestOf ] title: The Best Of - 2014 Edition --- At the end of every year I note down a summary of the best applications, hardware & websites I've enjoyed & depended on throughout the year (and often for some time before). _This post has long since been superseded. You can find the latest version of this post [here]({{< ref "posts/2022-10-19-apps-of-2022" >}})._ ## Software / General Use - Fastmail - - Evernote - - Reeder - - Keynote - - Lastpass - - Plex - - Calibre - ## Software / Geek Use - Sublime Text - - Homebrew - - DropSync - - Beets - - Textual - - XLD - - Code Academy - - Exercism.io - - Sickbeard + Headphones + Couchpotato + Sabnzbd - ## Software / Mobile - Tweetbot - - Pushover - - Lastpass - - Keynote - - Reeder - - Evernote- - Plex - - Backblaze - - WTF Podcast - - MiniHack - - Uber - - Goodreads - - Notify4M - - Bandcamp - - Hype Machine - - Nuzzel (Only got onto this today) - - Alien Blue - ## Software / SysAdmin or DevOps Specific - Gitlab - - Gitlab-CI - - Dash - - PostgreSQL (Makes my list every year & just keeps getting better) - - PGBadger - - Docker - - Consul - - Puppet (I couldn't do my job as well without it) - - iTerm - - Nginx - - Htop - - Bonnie++ - ++ - Openfire - - Hiera-Eyaml - - Rubinius - - Puma - - XenServer - - ElasticSearch - www.elasticsearch.org - Logstash - - FPM - - PFsense - - Debian Jessie (Not quite released but the next great version of the best linux Distro IMO) - - Check_MK Multisite - - PWSafe - - Supervisord - ## Websites - Last.fm - - Feedly - - HackerNews - - Lucidchart - - MondoTunes (Might be a little biased here!) - ## Hardware - Das Ultimate 4- - Logitech Performance MX - - 15'' Macbook Pro Retina - - iPhone 6+ (because its bigger than bigger, or something) - - CuBox - - Parani SD1000 Bluetooth Serial Adapter - - SanDisk Extreme Pro 480GB SSD - ## Other / Non-tech - All Day Socks - - Sennheiser Amperior On-Ear Headphones - - O2 + ODAC - - Benchmark DAC1 HDR - - Bellroy Wallets - - Ink Shoes - - Kindle Paperwhite - - Nerf Jolt - ## Books - The Phoenix Project (Reread this year for the 3rd time) - - Surely You're Joking Mr Feynman! - - The Dark Tower Series - - Snow Crash - - [XenServer, SSDs & VM Storage Performance](https://smcleod.net/2015/02/xenserver-ssds-vm-storage-performance/index.md): XenServer, SSDs & VM Storage Performance --- author: "Sam McLeod" categories: [ Tech ] date: "2015-02-15T00:00:00Z" aliases: - /tech/2015/02/14/xenserver-ssds-vm-storage-performance images: ["noah-kuhn-27481.jpg"] cover: image: " noah-kuhn-27481.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - tech - storage - hardware - xen title: XenServer, SSDs & VM Storage Performance --- ## Intro At Infoxchange we use XenServer as our Virtualisation of choice. There are many reasons for this including: * Open Source. * Offers greater performance than VMware. * Affordability (it's free unless you purchase support). * Proven backend Xen is very reliable. * Reliable cross-host migrations of VMs. * The XenCentre client, (although having to run in a Windows VM) is quick and simple to use. * Upgrades and patches have proven to be more reliable than VMware. * OpenStack while interesting, is not yet reliable or streamlined enough for our small team of 4 to implement and manage. * XenServer Storage & Filesystems Unfortunately the downside to XenServer is that it's underlying OS is quite old. The latest version (6.5) about to be released is still based on Centos 5 and still lacks any form of EXT4 and BTRFS support, direct disk access is not available… without some tweaking and has no real support for TRIM unless you have direct disk access and are happy with EXT3. Despite this, XenServer still manages to easily outperform VMware in both storage and CPU performance while costing… nothing unless you purchase support! ## Direct disk access It turns out, that you can add custom udev rules to pass through devices directly to VMs. For example, assuming `/dev/sdb` is the disk you wish to make available to guests, you can edit `/etc/udev/rules.d/50-udev.rules` like so: ```shell ACTION=="add", KERNEL=="sdb", SYMLINK+="xapi/block/%k", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'" ACTION=="remove", KERNEL=="sdb", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'" ``` There are some limitations with this: You're passing through the whole disk and can only grant that entire disk to a single VM. The udev configuration is volatile and is likely to be lost upon installing XenServer updates. You cannot (to my knowledge) boot from directly attached storage devices. What I've actually done is partitioned the SSD RAID array into 4 partitions on the XenServer host, this allows me to carve up the SSD RAID array and present adequate storage to several VMs on the host. i.e. ```shell ACTION=="add", KERNEL=="sdb1", SYMLINK+="xapi/block/%k", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'" ACTION=="remove", KERNEL=="sdb1", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'" ACTION=="add", KERNEL=="sdb2", SYMLINK+="xapi/block/%k", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'" ACTION=="remove", KERNEL=="sdb2", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'" ``` etc… I've then: * Mapped each partition to a VM running on that host. * Partitioned it within the host as needed. * Mounted /var/lib/docker as a BTRFS volume with compression enabled (compress=lzo) - This is used by CI as we build our our applications in Docker. * Mounted /home as an EXT4 - This is used by Gitlab-CI to checkout the pre-reqs for each build. ## Performance (benchmarks) ### Observations The HP P410i RAID card on the old G6 DL360's I am using for this is far underpowered and is unable to perform anywhere near the SSD's rated speeds. Direct disk access is 2-6 times 'faster' than XenServer's standard LVM or EXT3 storage. There is less than a 10% difference in performance between the SSD RAID array on the VM with the presented disk than directly on the XenServer host. XenServer's raw disk performance was exactly the same as Debian 7. Remember to ensure your RAID (and disk cache if not in prod) is enabled! Points of Note: This is an old server, we're not talking your latest and greatest hardware here, we're talking giving new life to an old-ish dog so that it may retain its usefulness while remaining cost effective. With TRIM being unavailable when using RAID, it is expected that the write performance will decrease somewhat overtime as the disks fill up as the disks will have to perform a 'READ, ERASE, WRITE' rather than a simple WRITE, To aid the lack of TRIM, I have left more than 25% of the disks unused as we simply don't need 1TB of SSD storage on each of the hosts. We have some 1GB cache cards arriving in the following weeks which we will upgrade to from the 512MB cards presently installed - I expect this to significantly further improve performance. ## Hardware * Refurbished HP DL360 G6 (2010 model). * 2x 8 Core Xeon x5550 w/ hyperthreading, 8 cores presented to domU. * 96GB DDR3 in dom0, 4GB in domU. * HP P410i w/ 512MB cache. * 2x 10K 146GB spindles for dom0. * 'HP Genuine'! * Constant 5-6 Watts each + cooling. * 2x SanDisk Extreme Pro SSD 480GB in RAID 0 * SATA III, Read up to 550MB/s, Write up to 515MB/s, Random Read up to 100K IOPS, Random Write up to 90K IOPS. * 0.15-0.2 Watts peak each. EXT4 Bonnie++ Results w/ 2x SSD in RAID1, XenServer 6.5 RC1, LVM: ![]({{< ref "/" >}}/img/xenserver/dl360-lvm.jpg) EXT4 Bonnie++ Results w/ 2x SSD in RAID1, XenServer 6.5 RC1, Direct Disk: ![]({{< ref "/" >}}/img/xenserver/dl360-dd.jpg) EXT4 dd Results w/ 2x SSD in RAID0, XenServer 6.5 RC1, LVM: ```shell samm@serv-dhcp-13:/var/tmp# dd if=/dev/zero of=tempfile bs=1M count=8000 conv=fdatasync,notrunc 8000+0 records in 8000+0 records out 8388608000 bytes (8.4 GB) copied, 118.11 s, 71.0 MB/s ``` EXT4 dd Results w/ 2x SSD in RAID0, XenServer 6.5 RC1, Direct Disk: ```shell samm@serv-dhcp-13:/var/tmp# dd if=/dev/zero of=tempfile bs=1M count=8000 conv=fdatasync,notrunc 8000+0 records in 8000+0 records out 8388608000 bytes (8.4 GB) copied, 19.21 s, 437 MB/s ``` Future Benchmarking Steps * Observe performance over time since TRIM is not available. * Upgrade RAID cards to 1GB cache (we have some spares). * Try with our new Gen8 BL460c blade servers, with locally attached P420i RAID controllers and 2GB cache. * Try linux software RAID with direct disks. ## XenServer's Future Filesystems I would hope that both EXT4 (and BTRFS) support is added in the near future. With this I would expect auto-detecting TRIM support similar to VMware and Hyper-V. Direct Disk Access Direct disk access clearly offers massive performance gains, I would like to see XenServer add this as an easy to use option when configuring storage. Non-volatile advanced configuration Related to direct disk access, XenServer needs some form of non-volatile advanced configuration options. VMware, raw Xen and KVM let you tweak many options without risk of loss after installing minor updates. - [BMW E36 328i Manual Conversion Programming](https://smcleod.net/2010/06/bmw-e36-328i-manual-conversion-programming/index.md): BMW E36 328i Manual Conversion Programming --- author: "Sam McLeod" categories: [ Cars ] date: "2010-06-10T00:00:00Z" aliases: - /cars/2010/06/22/e36-bmw-manual-conversion/ images: ["ivan-angelov-ITEOjp8utXc-unsplash.jpg"] cover: image: " ivan-angelov-ITEOjp8utXc-unsplash.jpg" #alt: "" #caption: "" #relative: false # To use relative path for cover image, used in hugo Page-bundles tags: - cars title: BMW E36 328i Manual Conversion Programming --- ## Reprogram BMW after doing Automatic to Manual Conversion Based on my E36 328is experience, **written in 2010**. ## Assumptions This guide assumes the following: - You have programmed a BMW ECU/DME before using DIS or similar. - You are taking all the necessary safety precautions. (Fully charged battery etc…) - You have DIS working (We used EasyDIS 1.0, Base 44) - You have Ediabas (INPA, NCS Expert, IFH Serve) installed and working. - You're able to read between the lines of our crappy document, some steps may differ slightly and our wording may be somewhat imprecise. ## Data Gathering Make notes of the following: - Your Basic Control Unit part number. (Mine is `1429861`) - The equivalent Programmed Control Unit to suit your car but for a manual spec (mine was 1429954) - I found several available on parts websites online. - Your VIN ## My Conversion Some background on my conversion: - We did this on a 1996 OBD1 Euro 328i Coupe, Siemens MS41.0 with Getrag 260 manual conversion. - We used a Carsoft cable (Chinese version) that had been modified to provide voltage readings and had several pins bridged to work with ODB1. - We will take no responsibly for ANY damage that could be caused by this process - use at your own risk! ## Reprogramming If you simply reflash your ECU/DME to a part number from a manual, the ZCS coding values stored in the EWS will cause your ECU/DME to think the automatic box is still installed! To fix this, you need to change the GM code stored in the EWS; this is done through comparing a value from the `E36ZST.000` file in NCS Expert's Daten Folder with your current GM code. This is simpler than it sounds. ### Steps 1. Find your current GM code from DIS; this is located in programming under ZCS/FA Coding - Choose Print Current Values. Answer the questions provided and note down all three codes when you get them. (GM, SA & VM will be displayed). 2. Open the following file in notepad `C:\NCSEXPER\DATEN\E36\E36ZST.000` 3. Find the line with your model, engine, old transmission and US/Euro information. You will see a 8 character value on the left of the large number of 0's that will look something like 11B20000. (The non-0 parts of this number should match the start of your GM code.) 4. Locate the same car but in manual, you will see the value above is slightly different, for an e36 328-coupe manual the value is `11A20000` - Note this down as it will be the base for your new GM code. 5. Now match your new GM code found in the last step with your current GM code provided by DIS but ignoring the last digit, which is only a checksum value. For example: - Old GM code from DIS: `11B20700B` - I found the manual GM base code to be `11A20000` - My new GM code will be `11A20700` (and we will work out the last digit in the next step). 6. Open NCS Expert. (My version is in German) - Click File > Load Profile - Choose Expert Mode - Press the first button - `FG/ZCS/FA` - Press the first button again - `ZCS eingeben` - Choose your model (`e36`) - Insert your VIN, Press OK - Enter your new GM (without the last digit) (Eg: `11A20700`) - Enter your current SA (without the last digit) (Eg: `0000482001228491`) - Enter your current VN (without the last digit) (Eg: `0004BFA936`) - Tick `checksumme berechnen` and press OK. 7. Now in the top part of the display you will be provided with your new GM including your last character, which is the checksum value (mine was 11A207008). 8. Back in DIS / Programming: - Choose ZCS/FA Coding - Choose Recoding - Choose the EWS module 9. If you get prompted something along the lines of 'Are these correct?' when displaying the current GM,SA & VM values - Choose NO. 10. Enter your new GM and your original SA & VN (these wont have changed). Go through the programming process as normal. 11. Now we go to DME Programming Choose exchange control unit Determine Control Unit 12. When asked "Is the faulty unit still installed in the car?" choose NO. 13. Enter data: - Your current basic control unit part number. (Mine was `1429861`). - Your chassis number. - Your replacement programmed control unit part number (Mine was `1429954`). 14. You'll need to click down a few times, then back to 'Program Control Unit' Enter details are requested (Kms, Chassis number), Start programming! (Make sure your battery is full of juice!) 15. DIS will now walk you through clearing your adaptation values. 16. Open ZCS/FA programming; follow the process to Align-EWS to DME. 17. Open DIS - Diagnostics and do a quick clear of errors. All done! ### Results You should notice: - At first the car will not be that zippy, as it has to relearn its adaptation values. - Once the car is warm, the idle speed will drop from the automatic 650-700RPM to 500-550RPM manual setting. - When changing gears you will notice the rev's drop a lot faster between gear changes (if you have a lightened flywheel you revs may still rise 200RPM~ at times.) - I'm not 100% sure on this one as I've only just done it, but I'm pretty sure the car actually feels a bit peppier at the top-end also, mine actually seems to sound a bit deeper when applying revs! ## Carsoft Cable Mods - Pin 16 of the 20pin diag connector needs to be connected through a 4k7 resistor to pin 9 of the serial port. - Pin 1 of the 20p diag via 4k7 to pin 6 on the serial. The above two allow INPA and friends to detect battery and ignition. Could connect direct without the resistors but you're in megadanger of blowing up your serial port so definitely use the resistors. - Pin 14 on the 20pin diag connector needs to be bridged (no resistor just a link) to pin 18 on the diag connector for OBDI cars to allow programming. ### The 20pin Diagnostics Connector - Pin 1 is terminal 15 (ignition voltage). - Pin 14 is an always on 12v source. - Pin 16 is terminal 30 (battery voltage). - Pin 18 is programming voltage for OBDI cars (not fitted in my E46). ### The Serial Port - Pin 6 is Data Set Ready "DSR" (used by EDIABAS for ignition detect). - Pin 9 is Ring Indicator "RI" (used for battery detect). On the carsoft cable the serial DB9 male to female has all wires in it pin for pin. The 20pin diag connector to the DB15 however only has 4 wires in it so an additional 2 wires will need to be added for the above mods. [Revision 1.1 - 22/6/10] # Misc - [About](https://smcleod.net/about/index.md): About --- title: "About" author: "Sam McLeod" norss: true hideTitle: true windowTile: 60 toc: enable: true auto: false --- > Hi, I'm Sam 👋🏻 ![](/profilephoto/apple-touch-icon.png) AI engineer, open source contributor and music geek based in Melbourne, Australia. Around twenty years in tech across platform engineering and technical leadership, with the last five focused heavily on LLMs and AI. I'm an AI Engineering Principal, currently working with [Mantel](https://mantelgroup.com.au). This site is where I write about engineering, share things I've learned, and occasionally ramble about music or hi-fi. ## Contact - 🧑‍💻 [Github](https://www.github.com/sammcj) - 👔 [LinkedIn](https://www.linkedin.com/in/sammcj) - 📄 [Resumé / CV](/cv/) - 🦙 You can also find me modding on [/r/LocalLlama](https://www.reddit.com/r/LocalLLaMA/top/?t=week) --- ## Interests Outside of work: - 🎸 Discovering new music, especially digging into niche subgenres for the hidden gems - 👨🏻‍🎤 Going to gigs - 📻 HiFi (yes, lots of gear) - 🚶‍♂️ Walking in nature, usually with new music in my ears - 🦾 AI and LLM tooling, development, and performance engineering - 📚 Sci-fi, speculative fiction, and biographies, mostly via audiobook - 📺 Film and TV, especially sci-fi and British comedy - 🧑🏼‍💻 Hardware and software engineering, IoT, home automation - 🐈 Cats, especially [Piper](/piper.jpg) - 🚗 Road trips and automotive engineering The [Links](/links/) page has some of my favourite services, tools, and media. --- ### 🔋 Things that give me energy - 🏡 Remote work and remote-first organisations - 🧑‍🤝‍🧑 Building high performance platform and software delivery teams and team culture - 🙋 Open communication - 🎷 Music collection and discovery - 🔬 Adenosine triphosphate ### 🪫 Things that drain my energy - 🏢 Working from an office - 💼 Bureaucracy - 🔐 Disempowered teams - 📈 Slow feedback loops > [!INFORMATION] 🎉 On remote work > > Going 100% remote was a career goal that finally came together through covid, and I'm not going back. The gains to my health, happiness and productivity have been immense, and any employer who doesn't recognise that isn't one I want to work for. --- ## Support my work [buymeacoffee.com/sam.mcleod](https://buymeacoffee.com/sam.mcleod) --- ## About this site [About this desktop](/about-desktop/) covers how this site is built and designed. --- ## Found a problem with this site? Please log an {{< github-button button="issue" user="sammcj" repo="sammcj" count="true" >}}! --- Words are my own or somebody else's 🖖🏼 - [About this desktop](https://smcleod.net/about-desktop/index.md): About this desktop --- title: "About this desktop" icon: "theme" window: about-desktop deskbarHidden: true --- This site is a small desktop in the browser, built as my own Hugo theme in plain JavaScript and CSS with no framework. Posts, apps and tools open in windows you can drag, snap and stack. Hugo builds it as a static site, so every post keeps its own URL, its feed and a plain page that reads without JavaScript. I've enjoyed trying out window management ideas along the way. Grab a window by its title tab and drop it on another window's tab, and the two stack into one tabbed window. Drag a tab away to pull it back out. Drop it on the side edge of another window instead and the two join side by side, moving together until you press the clip between them. Drag a window to a screen edge to snap it, press `a` to tile every open window, or drag a post out of the Posts window to open it on its own. - The [Control panel](/control-panel/) swaps in other looks. - The core shell is capped at 45KB gzipped. Everything else loads on first use. - The source, theme included, is on [GitHub](https://github.com/sammcj/smcleod). ## How it was designed - Three single-file prototypes came first: Deskbar Classic, Modern Tiles and Tracker Workbench. Deskbar Classic, a blend of Haiku and XFCE, became this site. - The name comes from the Deskbar, the menu and task panel of BeOS and Haiku, whose look the first style follows. - When adding new styles I build prototypes of the candidates to pick from. This [sheet of twelve theme concepts](/design/theme-concepts.html), from NeXTSTEP and CDE to Phosphor and Broadsheet, is one example. ## Credits - The first look was modelled on [Haiku](https://www.haiku-os.org/) (and BeOS before it). Others, such as the [XFCE](https://xfce.org/)-like and Synthwave looks, came later, and more will follow. - Built by [Hugo](https://gohugo.io/), with esbuild bundling the shell - Diagrams by [Mermaid](https://mermaid.js.org/), loaded only on pages that use them - The [sheep screen saver](/screensaver/) is eSheep, the 1990s desktop pet by Tatsutoshi Nomura that used to wander across my screen back in the day, drawn with the gSheep Purple sprites from [Adriano Petrucci's desktopPet](https://github.com/Adrianotiger/desktopPet) - [ADHD Medications](https://smcleod.net/admeds/index.md): ADHD Medications --- title: "ADHD Medications" description: "A visual comparison of common ADHD medications" author: "Sam McLeod" norss: true deskbarHidden: true # keeps it out of Tracker and Spotlight toc: false keywords: [""] tags: [""] --- This is an experimental visual comparison tool based on subjective experience, it should not be considered medical advice. Always consult a doctor before starting or changing medication. {{< admeds >}} - [Agentic Coding Tools](https://smcleod.net/agentic-coding-tools/index.md): Agentic Coding Tools --- title: "Agentic Coding Tools" window: tool icon: compare description: "Rating Agentic Coding Tools" author: "Sam McLeod" norss: true toc: enable: false auto: false disable: true --- I am a heavy user of agentic coding tools, part of my job is to evaluate their capabilities and limitations. This is an experiment where I am tracking their capabilities, limitations and my overall impression of them. _Note: Some cells have more information when you hover over them._ {{< comparison-table-legend >}} {{< comparison-table >}} - [Cars & Bikes](https://smcleod.net/cars/index.md): Cars & Bikes --- title: "Cars & Bikes" description: "Cars & Bikes list" author: "Sam McLeod" norss: true # subtitle: Sam McLeod toc: false keywords: ["car", "bike"] tags: ["car", "bike"] categories: ["Automotive"] --- {{< gist sammcj 9c371e898b9eed1e1165 >}} - [Chiptunes](https://smcleod.net/chiptunes/index.md): Chiptunes --- title: "Chiptunes" description: "Original chiptunes composed by AI and synthesised in your browser." icon: music layout: chiptunes window: chiptunes --- Each song is a few kilobytes of pattern data, played live through the Web Audio API on square, pulse, triangle and noise channels. Claude, an AI, composed them. The player needs JavaScript and stays silent until you press Play. - [Contact](https://smcleod.net/contact/index.md): Contact --- title: "Contact" window: mail icon: mail description: "Send Sam McLeod a message." aliases: ["contact", "email"] author: "Sam McLeod" norss: true toc: enable: false auto: false --- # Contact Me --- {{< contact >}} - [Control panel](https://smcleod.net/control-panel/index.md): Control panel --- title: "Control panel" icon: "control-panel" window: control-panel deskbarHidden: true aliases: ["/appearance/"] --- The Control panel holds this desktop's settings, kept in this browser: - **Appearance:** themes, or window style, colours, light or dark, dock, wallpaper and a CRT effect one at a time - **Posts:** the reader's width, text size and font - **System:** the screen saver's delay, and a button to try it It needs JavaScript. - [Enterprise Portal](https://smcleod.net/enterprise/index.md): Enterprise Portal --- title: "Enterprise Portal" description: "All your favourite enterprise apps in one window. Your organisation manages this page." icon: enterprise window: enterprise --- Enterprise Portal needs JavaScript, and a 1.4 GB update. It is a parody of chat, whiteboard, endpoint monitoring, ticketing, wiki and slide software, and stores nothing. - [Favourites](https://smcleod.net/favourites/index.md): Favourites --- title: "Favourites" description: "Things I like" # A folder of the favourites folders, which keep their own URLs layout: folder window: folder icon: favourites include: [/links, /hardware, /podcasts, /youtube, /blogs] # 1.5 times as wide as the Posts window (630x830) and as tall, right of it when it fits there windowWidth: 945 windowHeight: 830 windowBesidePosts: true norss: true --- - [Feeds](https://smcleod.net/feeds/index.md): Feeds --- title: "Feeds" description: "The latest posts from blogs and news sites I follow." icon: feeds layout: feeds window: feeds norss: true --- - [Gigs](https://smcleod.net/gigs/index.md): Gigs --- title: "Gigs" description: "Gigs list" author: "Sam McLeod" norss: true toc: false keywords: ["music", "gigs", "bands", "concerts"] tags: ["music", "gigs", "bands", "concerts"] categories: ["Music"] --- {{< gist sammcj 45a32a7df6ea5b4efec7a7dd3bf2fc95 >}} - [Hardware](https://smcleod.net/hardware/index.md): Hardware --- title: "Hardware" description: "Hardware I use and like." series: [ BestOf ] norss: true # Cards from data/hardware.yaml, with product shots from assets/hardware/ layout: folder window: folder icon: hardware view: cards thumbSize: 320x240 items: hardware windowWidth: 720 windowHeight: 546 --- More favourites: [YouTube Channels](/youtube/), [Podcasts](/podcasts/), [Tools](/links/) and [Blogs](/blogs/). - [LLM FAQ](https://smcleod.net/llm-faq/index.md): LLM FAQ --- title: "LLM FAQ" description: "Frequently Asked Questions about LLMs and AI" aliases: ["llm", "faq", "frequently-asked-questions","llm-faq","ollama-faq"] tags: ["AI", "tools", "LLM", "tech", "llms", "ollama","llama","faq","ollama faq","llm faq"] author: "Sam McLeod" norss: false # cover: # image: "diagram-gen.png" # alt: "DiagramGen" # hidden: false toc: enable: true auto: true draft: false --- ## "Should I run a larger parameter model, or a higher quality smaller model of the same family?" TLDR; **Larger parameter model [lower quantisation quality] > Smaller parameter model [higher quantisation quality]** E.g: Qwen2.5 32B Q3_K_M > Qwen2.5 14B Q8_0 Caveats: - Don't go lower than Q3_K_M, or IQ2_M, especially if the model is under 30B~ parameters. - This is in the context of two models of the same family and version (e.g. Qwen2.5 Coder). Longer answer: Check out the [Code Chaos and Copilots](https://smcleod.net/2024/07/code-chaos-and-copilots-ai/llm-talk-july-2024/) slide deck. --- ## "Should I use F16 quantised models to get the best quality output?" TLDR; **No.** For text generation running F16 models will do nothing other than waste your memory, compute, power and will respond slower. Longer answer: - Generally speaking you do not notice any improvements over around Q6_K. - The larger the models parameter size, the less of an impact you'll see from lowering the quantisation quality. - Very small models - below around 4B parameters are more affected by quantisation, try to keep those as high quality as possible up to Q8_0. - If there is no larger parameter model in the family - don't go below Q4_K_M unless you really do not have the memory, or if the model is over around 50B parameters. - Embeddings models are a bit different and are often kept at fp16, don't go lower than Q8_0. --- ## "Should I trust LLM generated code/content? What if the model makes a mistake? Don't they just make everything up?!" TLDR; You know who else makes mistakes, writes bad code and makes things up? Humans. Longer answer: - You (generally) don't blindly trust code from other humans, if you do (and there we all do to a degree) you are accepting the risk that it may not be perfect. - If you took a hand full of random developers with different backgrounds and without providing any context you asked them to write a very specific piece of code, you would get a hand full of different implementations - much of it would be less than ideal if not outright incorrect. - If you first provided them with an education in the language you're working in, a specification, and information about the world - and your expectations - you would get a much better result. The same is true for LLMs. The time and effort it takes however to provide this to an LLM is far less. - Just like with humans, you should should either: - Have systems in place to review and/or validate the outputs. - Or, Accept that the task and impact of undesired outcomes is low enough that you can accept the risk. --- ## "Will AI take my job?" TLDR; Maybe. Longer answer: Capitalism sucks, hard. It's the shitty card we've been dealt and because of that - for-profit companies will always be looking to reduce costs and increase profits. If this means they can reduce expenses by automating a role - they will (eventually). To look at this another way - if the output of your job is repeatable and not creative - one could argue that (other than providing you income), it might also not be the best use of your time. In the short term I think life will be very hard for many people not in a position to adapt, for-profit companies will continue to focus on profits over people, and governments will continue to be slow to act. If I was to be cold about it - I would say something like "evolution isn't mandatory, but neither is survival". In reality while there's a lot of nuance to this discussion, I think it's of the upmost importance to focus efforts on awareness, cross-skilling, re-skilling both yourself and those around you, and to push for a more equitable society where the benefits of automation are shared by all. ## Ollama ### "Is Ollama just a wrapper for Llama.cpp?" TLDR; No. Ollama uses llama.cpp as it's primary inference engine, but provides a different set of features. --- On several occasions I have come across the claim that _"Ollama is just a llama.cpp wrapper"_, which is inaccurate and completely misses the point. I am sharing my response here to avoid repeating myself repeatedly. With llama.cpp running on a machine, how do you connect your LLM clients to it and request a model gets loaded with a given set of parameters and templates? ... you can't, because llama.cpp is the inference engine - and it's bundled llama-cpp-server binary only provides relatively basic server functionality - it's really more of demo/example or MVP. Llama.cpp is all configured at the time you run the binary and manually provide it command line args for the one specific model and configuration you start it with. Ollama provides a server and client for interfacing and packaging models, such as: - Hot loading models (e.g. when you request a model from your client Ollama will load it on demand). - Automatic model parallelisation. - Automatic model concurrency. - Automatic memory calculations for layer and GPU/CPU placement. - Layered model configuration (basically docker images for models). - Templating and distribution of model parameters, templates in a container image. - Near feature complete OpenAI compatible API as well as it's native native API that supports more advanced features such as model hot loading, context management, etc... - Native libraries for common languages. - Official container images for hosting. - Provides a client/server model for running remote or local inference servers with either Ollama or openai compatible clients. - Support for both an official and self hosted model and template repositories. - Support for multi-modal / Vision LLMs - something that llama.cpp is not focusing on providing currently. - Support for serving safetensors models, as well as running and creating models directly from their Huggingface model ID. In addition to the llama.cpp engine, Ollama are working on adding additional model backends (e.g. things like exl2, awq, etc...). Ollama is not "better" or "worse" than llama.cpp because it's an entirely different tool. - [LLM vRAM Estimator](https://smcleod.net/vram-estimator/index.md): LLM vRAM Estimator --- author: "Sam McLeod" description: "LLM vRAM Estimator" tags: - vram - LLM - ollama - llama - AI - cuda - metal - gguf title: "LLM vRAM Estimator" folderWeight: 1 window: tool icon: vram categories: ["LLM"] keywords: ["vram", "llm", "ollama", "llama", "ai", "cuda", "metal", "gguf"] toc: false aliases: - /vram-calc/ - /vram/ - /vram-estimator/ --- {{< vram-calculator >}} - [Offline](https://smcleod.net/offline/index.md): Offline --- title: "Offline" deskbarHidden: true # keeps it out of Tracker and Spotlight --- > You are not connected to the Internet, only cached pages will be available. - [Podcasts](https://smcleod.net/podcasts/index.md): Podcasts --- title: "Podcasts" description: "Podcasts I listen to." series: [ BestOf ] norss: true # Cards from data/podcasts.yaml; artwork and latest episodes come from each show's RSS at build time layout: folder window: folder icon: podcasts view: cards thumbSize: 240x240 items: podcasts windowWidth: 720 windowHeight: 546 --- More favourites: [YouTube Channels](/youtube/), [Hardware](/hardware/), [Tools](/links/) and [Blogs](/blogs/). - [Projects](https://smcleod.net/projects/index.md): Projects --- title: "Projects" description: "My pinned GitHub repositories." norss: true # data/pinned.json is refreshed from GitHub in the deploy workflow (scripts/fetch-pinned.sh) layout: folder window: folder icon: projects items: pinned view: list itemPages: mcp-devtools: /posts/2025-08-18-mcp-devtools nvapi: /posts/2024-05-18-nvapi force-hidpi: /posts/2026-03-30-apple-silion-4k-limitations gollama: /posts/2024-06-01-gollama ingest: /posts/2024-07-28-ingest --- More on [GitHub](https://github.com/sammcj?tab=repositories). - [Quotes and Musings](https://smcleod.net/quotes/index.md): Quotes and Musings --- author: "Sam McLeod" description: "A collection of quotes" tags: - quotes title: "Quotes and Musings" categories: [ quotes ] keywords: "quotes" toc: true aliases: - /quotes-musings/ --- A collection of quotes and musings ## Quotes > "There is no 'us' and 'them'; it's an illusion. We are all human beings, and we all have a responsibility to support one another and to discover ways of wresting the power from the very, very few people who control all the cash and all the property." > "Everything under the sun is in tune, but the sun is eclipsed by the moon." > "Long you live and high you fly. And smiles you'll give and tears you'll cry. And all you touch and all you see is all your life will ever be." > "...Do you want a voice? If you do, you better bloody well go out and get it because it's not going to be handed to you on a plate." {{< typeit group=paragraph >}} \- Roger Waters {{}} --- > "It is the struggle itself that is most important. We must strive to be more than we are. It does not matter that we will never reach our own ultimate Goal. The effort yields its own rewards." {{< typeit group=paragraph >}} \- Data {{}} --- > "You have no responsibility to live up to what other people think you ought to accomplish. I have no responsibility to be like they expect me to be. It's their mistake, not my failing." > "What I cannot create - I do not understand." {{< typeit group=paragraph >}} \- Richard P. Feynman {{}} --- > "Every gun that is made, every warship launched, every rocket fired signifies, in the final sense, a theft from those who hunger and are not fed, those who are cold and are not clothed." {{< typeit group=paragraph >}} \- Dwight D. Eisenhower {{}} --- > "Most rules lie, yet lies are less shocking, less corrosive than the chosen, cultivated ignorance that Chevallier addresses here. This ignorance denies the reality of anything which provokes pity. It is an error to think of such war leaders or, today, economic strategists, as pitiless. They are abject. This is what we have to learn and to act upon. They are abject." {{< typeit group=paragraph >}} \- Gabriel Chevallier, Fear; A Novel of World War I (Quote from Roger Waters The Wall, 2015) {{}} --- > "If a society is tolerant without limit, its ability to be tolerant is eventually seized or destroyed by the intolerant." {{< typeit group=paragraph >}} \- Karl Popper {{}} --- > "He knows the price of everything and the value of nothing." {{< typeit group=paragraph >}} \- Oscar Wilde {{}} --- > "The road of excess leads to the palace of wisdom." {{< typeit group=paragraph >}} \- William Blake {{}} --- > "You have to die a few times before you can really live." > "Some people never go crazy. What truly horrible lives they must lead." > "Sometimes you just have to pee in the sink." {{< typeit group=paragraph >}} \- Charles Bukowski {{}} --- > "The most dangerous man to any government is the man who is able to think things out for himself, without regard to the prevailing superstitions and taboos." {{< typeit group=paragraph >}} \- H.L. Mencken {{}} --- > "When the going gets weird, the weird turn pro." > "There he goes. One of God's own prototypes. A high-powered mutant of some kind never even considered for mass production. Too weird to live, and too rare to die" > "Life should not be a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather to skid in broadside in a cloud of smoke, thoroughly used up, totally worn out, and loudly proclaiming "Wow! What a Ride!" > "No sympathy for the devil; keep that in mind. Buy the ticket, take the ride…and if it occasionally gets a little heavier than what you had in mind, well… maybe chalk it off to forced conscious expansion: Tune in, freak out, get beaten." > "So we shall let the reader answer this question for himself: who is the happier man, he who has braved the storm of life and lived or he who has stayed securely on shore and merely existed?" > "Life has become immeasurably better since I have been forced to stop taking it seriously." > "I was not proud of what I had learned, but I never doubted that it was worth knowing." > "A man who procrastinates in his choosing will inevitably have his choice made for him by circumstance." > "Some may never live, but the crazy never die." > "We are all alone, born alone, die alone, and-in spite of True Romance magazines-we shall all someday look back on our lives and see that, in spite of our company, we were alone the whole way. I do not say lonely-at least, not all the time-but essentially, and finally, alone. This is what makes your self-respect so important, and I don't see how you can respect yourself if you must look in the hearts and minds of others for your happiness." {{< typeit group=paragraph >}} \- Hunter S. Thompson {{}} --- > "You're up against skinheads that are all like 'play faster!' and we're all like 'Listen slower!'" {{< typeit group=paragraph >}} \- Maynard James Keenan {{}} --- > "The intellectual is, quite simply, a human being who has a pencil in his or her hand when reading a book." {{< typeit group=paragraph >}} \- George Steiner {{}} --- > "Science is the poetry of reality." {{< typeit group=paragraph >}} \- Richard Dawkins {{}} --- > "Tell him that we have fucking reprogrammed reality. Tell him that language is a virus and that religion is an operating system and that prayers are just so much fucking spam." {{< typeit group=paragraph >}} \- Neil Gaiman {{}} --- > "With the first link, the chain is forged. The first speech censured...the first thought forbidden...the first freedom denied - chains us all, irrevocably." {{< typeit group=paragraph >}} \- Jean-Luc Picard {{}} --- > "Any fool can know. The point is to understand." {{< typeit group=paragraph >}} \- Albert Einstein {{}} --- > "Life can only be understood backwards; but it must be lived forwards." {{< typeit group=paragraph >}} \- Søren Kierkegaard {{}} --- > "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius - and a lot of courage to move in the opposite direction." {{< typeit group=paragraph >}} \- E.F. Schumacher {{}} --- > "The cracks are where the light gets let in." {{< typeit group=paragraph >}} \- Leonard Cohen {{}} --- > "To do a great and important work, two things are necessary - a definite plan, and not quite enough time." {{< typeit group=paragraph >}} \- Leonard Bernstein {{}} --- > "Work for a cause, Not Applause." {{< typeit group=paragraph >}} \- Unknown {{}} --- > "Offence is important; that's how you know you care about things. Imagine a life where you're not offended. So dull." {{< typeit group=paragraph >}} \- Marcus Brigstocke {{}} --- > "The less confident you are, the more serious you have to act." {{< typeit group=paragraph >}} \- Tara Ploughman {{}} --- > "In the sunset of dissolution, everything is illuminated by the aura of nostalgia." {{< typeit group=paragraph >}} \- Milan Kundera {{}} --- > "Short words are best and the old words when short are best of all." {{< typeit group=paragraph >}} \- Winston Churchill {{}} --- > "None are more hopelessly enslaved than those who falsely believe they are free." {{< typeit group=paragraph >}} \- Johann Wolfgang Von Goethe, Elective Affinities {{}} --- > "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." {{< typeit group=paragraph >}} \- R. Buckminster Fuller {{}} --- > "Death is caused by swallowing small amounts of saliva over a long period of time." > "Religion is just mind control." {{< typeit group=paragraph >}} \- George Carlin {{}} --- > "It's sad when someone you know becomes someone you knew." > "My optimism wears heavy boots and is loud." > "Scar tissue is stronger than regular tissue. Realize the strength, move on." {{< typeit group=paragraph >}} \- Henry Rollins {{}} --- > "Simplicities are enormously complex. Consider the sentence "I love you"." {{< typeit group=paragraph >}} \- Richard O. Moore {{}} --- > "Truth is a matter of circumstances. It's not all things to all people at all times." {{< typeit group=paragraph >}} \- Natasha Romanoff {{}} --- > "No silicon heaven? Preposterous! Where would all the calculators go?" {{< typeit group=paragraph >}} \- Kryten {{}} ### Tech Specific --- > "Any improvements made anywhere besides the bottleneck are an illusion." > "Being able to take needless work out of the system is more important than being able to put more work into the system." > "A great team doesn't mean that they had the smartest people. What made those teams great is that everyone trusted one another. It can be a powerful thing when that magic dynamic exists." > "We need to create a culture that reinforces the value of taking risks and learning from failure and the need for repetition and practice to create mastery." > "Left unchecked, technical debt will ensure that the only work that gets done is unplanned work." {{< typeit group=paragraph >}} \- Gene Kim {{}} --- > "If it hurts, do it more often." > "If you plan and build for perfection, you will likely not live to see it completed." {{< typeit group=paragraph >}} \- Adrian Cockcroft {{}} --- > "Design not for MTBF, but for MTTR." {{< typeit group=paragraph >}} \- John Allspaw {{}} --- > "Programs must be written for people to read, and only incidentally for machines to execute." {{< typeit group=paragraph >}} \- Abelson & Sussman {{}} --- > "When the train stays on the tracks you don't ask why." {{< typeit group=paragraph >}} \- Marty Rubin {{}} --- > "A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system." {{< typeit group=paragraph >}} \- John Gall {{}} --- > "Everyone wants to get the right people, but it's not the people - it's the leadership, if we create the right environment - we will get the right people." > "You can easily judge the character of a man by how he treats those who can do nothing for him." > "Stress and anxiety at work have less to do with the work we do and more to do with weak management and leadership." > "You don't hire for skills, you hire for attitude. You can always teach skills." {{< typeit group=paragraph >}} \- Simon Sinek {{}} ## Musings > "Everything reminds me of the past and it's the only place I can never travel." > "Aiming to move from engineering to management is odd, think of it like a comedian - when you get good, or great even - you don't become a teacher of comedy, you tell jokes well and inspire others, you don't 'upgrade to comedian management'." > "You call engineers resources and I'll call you overhead." > "A manager once said to me *"I don't come to work to make friends"* - from point time onwards, I knew I wanted to do the exact opposite - I come to work to make friends, because the connections we make are what makes us human, they create and shape societies and give us meaning, direction and value to life." > "We have all the time in the world for people who let the organisation down, but none of the time for the people let down by the organisation." - [Resumé / CV](https://smcleod.net/cv/index.md): Resumé / CV --- title: "Resumé / CV" aliases: ["cv", "resume", "experience", "jobs", "linkedin"] author: "Sam McLeod" norss: true toc: true windowWidth: 1100 windowHeight: 1000 --- # Sam McLeod AI Engineering Principal, an early advocate for DevOps culture and practices with a background in platform engineering, automation, highly available systems design and team leadership. With 20+ years of engineering experience, I enjoy enabling and accelerating individuals, teams and organisations with generative AI solutions. *"Easy things should be easy, and hard things should be possible."* - [LinkedIn](https://www.linkedin.com/in/sammcj) - [Github](https://www.github.com/sammcj) --- ## Skills A rough self-assessment of where I sit across the areas I work in. {{< skills-matrix >}} --- ## Experience ## > Mantel Group / DigIO - AI & LLM Enablement, Acceleration, R&D, Agentic Systems - Platform Engineering Team Leadership - DevOps Culture & Practices Transformation - CI/CD and Tooling Modernisation - Performance and Availability Engineering - Amazon Web Services (AWS) and Google Cloud (GCP) Platform Engineering - Cloud Automation (Terraform) - AWS CDK (Typescript) Below are some of the project I've worked on. ### Leading AI Enablement & Acceleration June 2024 - Present AI Engineering Principal, leading efforts to enhance engineering capabilities and accelerate software development with generative AI. - Building, prototyping, enabling and tuning AI systems using a combination of cloud and open weight large language models with a goal of accelerating development and delivery throughout the SDLC. - Advocacy, training and tooling enablement for software developers and platform engineers to leverage AI to enhance their productivity. - Researching, prototyping and implementing AI tooling and LLMs focused on software development. - Driving the organisation-wide adoption of AI across diverse teams, projects, and client engagements, proactively shaping our approach to the rapidly evolving AI-driven software development landscape. --- ### DevOps & Platform Engineering Lead (Consulting) Feb 2022 - June 2024 (2 years 5 months) A hands-on engineering lead role, working with an iconic Australian brand building one of their first AWS projects - a robotics automation platform. - AWS CDK (Typescript) - Writing and maintaining L2 and L3 constructs, reusable packages, scaffolding etc... - AWS ECS (Elastic Container Service) - Building pipelines and infrastructure for containerised applications. - Github Actions (CI/CD) - Creating highly automated CI/CD workflows and custom Actions to support software and infrastructure development and delivery. - Development accelerated through the use of AI. --- ### DevOps & Platform Engineering Lead (Consulting) Feb 2021 - Dec 2021 (11 months) Brought in initially for 3 months I worked across multiple teams to understand, assess and make recommendations on the organisations software development and platform engineering culture, tooling and processes. - Providing platform security and performance recommendations. - Provide recommendations for and promote de-siloing and DevOps culture working across multiple stakeholders and teams. - Assess and select modern SaaS CI/CD tooling to replace legacy on-prem products and investigating product alignment and capabilities. - Providing strategic engineering recommendations at a CIO level. - Running workshops across multiple teams to understand and collate common concerns, risks and areas for improvements especially between developers and platform engineering teams. - Worked with teams to help secure systems and mitigate vulnerabilities. - Worked with multiple vendors on solution assessments and proof of concepts. - Development accelerated through the use of AI. --- ### Engineer Experience and DevOps Culture Consultant (Consulting) Mar 2020 - Aug 2020 (6 months) Engaged to assess the developer / engineer experience, provide and implement recommendations on DevOps culture and platform engineering within a Platform Engineering team. --- ### GCP Platform (Infrastructure) Design and Implementation (Consulting) Mar 2020 - Aug 2020 (6 months) Worked in a team to design and implement a new GCP platform to facilitate running microservices for high value financial payment transactions to replace services running in a legacy on-prem data centre. - Ran workshops with stakeholders to understand requirements and constraints. - Created CI/CD pipelines using GitHub and CloudBuild. - Created automation for the provisioning GKE (Kubernetes) clusters. - Worked with third parties to integrate networking, automation, and microservices with external projects. - Worked to help onboard junior team members to the automation pipeline. - Delivered a Cloud-Native MVP platform for microservice deployment and operation. --- ### GCP Terraform Automation Evaluation (Consulting) Jan 2020 - Mar 2020 (3 months) Engaged as part of a team to improve a large, legacy GCP automation (Terraform) codebase. - Ran workshops across stakeholders and engineers that use the codebase. - Collated findings from workshops to discover constraint trends and areas for improvements. - Inspected the current usage of Terraform and CI/CD pipelines. - Created a pack with a plan to implement recommendations to improve engineer workflow, CI/CD performance and code style improvements. --- ### GitLab Cloud Native / Kubernetes Implementation (Consulting) Dec 2019 - Jan 2020 (2 months) Worked with a team to design and kick off implementation of GitLab for VCS, CI/CD and Testing on Kubernetes (EKS) for 300-2000 staff across Australia. --- ### Cloud Migration Assessment (Consulting) Oct 2019 - Oct 2019 (1 month) - Assessed in uplifting the client's capability to move from on-prem hosting a to AWS. - Provided advice and recommendations on technology, tooling and training requirements. - Assisted with application integration solutions and architecture aligning with AWS best practices. - Created and delivered a pack on recommendations on moving from legacy ITIL and Waterfall processes and culture to adopt a more modern - DevOps and Agile workflow and culture. --- ### Continuous Integration and Delivery Tooling Assessment (Consulting) Sep 2019 - Oct 2019 (2 months) Working with a large enterprise client to explore options to improve or replace legacy CI/CD tooling. - Conducted workshops with business units to discover efficiency, issues, risks and tooling requirements. - Aligned requirements with industry standard measurements and researched the current tooling landscape to create capabilities scorecards. - Consulted with vendors around product capabilities and potential pricing. - Made formal recommendations on replacement CI/CD tooling and it's benefits, risks and high level architecture. --- ### DevOps Maturity Assessment (Consulting) Aug 2019 - Sep 2019 (2 months) - Undertook a DevOps and cloud maturity assessment and provided recommendations. - Conducted workshops to discover the state of culture, process and tooling across the organisation. - Aligned findings to scientifically backed research behind high performing software delivery organisations. - Delivered a leadership and executive pack, detailing next steps to improve the software development and hosting capabilities within the organisation, further advance their Cloud adoption strategy, suggested roadmap with risks and opportunities for improvement. --- ## > Infoxchange ### Platform Engineering Team Lead & Platform Architect Aug 2012 - Jan 2018 (7 years) Built a team and brought DevOps culture and ways of working to the organisation. Architecting and deploying scale out platforms for mission critical healthcare systems and confidential data platforms. Creating and leading the optimisation and automation of large environments of compute, storage and application clusters for many hundreds of thousands of customers across Australia and serving the organisation as a trusted technical adviser for DevOps culture and across operational platform delivery. Working closely with development and testing teams and technologies such as Kubernetes, Docker, Self Design Clustered Highly-Available Storage Clusters, Deep Monitoring, Test Driven Recovery, ElasticSearch / ELK Stack, RabbitMQ, Logstash, Docker, XenServer, Puppet, oVirt, KVM, Xen, Nginx, Git, GitLab Enterprise, PostgreSQL, Nessus, Metasploit, Burp Suite, High Performance Compute, Cloud & Hybrid Platform Technologies. - Team Leadership - DevOps Coordination & Mentorship - Private Cloud Architecture, Engineering & Operation - Platform and Distributed Systems Automation - Product Delivery & Performance Engineering - Storage Security & Encryption Design (LUKS, Puppet encryption technologies) - Continuous Integration & Delivery Systems (GitLab CI/CD, Custom Docker/LXC CI/CD) - High Performance Storage SAN Design & Architecture (DRBD, Pacemaker, Corosync, LIO) - Virtualisation Architecture & Engineering (KVM, Xen, XenServer, XCP-ng, oVirt) - Database Administration & Database Performance Engineering - Security Auditing & Automation - Tender Response & Delivery - Highly Available Clustered Storage Design & Architecture --- ## > Sensis ### Senior Platform Operations and Automation Engineer Jan 2012 - Aug 2012 (8 months) Working for Sensis (then part of Telstra) I was part of a senior team of platform engineers responsible for maintaining, deploying and improving a large scale environment working with systems such as Puppet, VMware, Nagios, Redhat, Solaris to provide a stable and scalable environment for Sensis web applications. My role included working reactively on incidents as they occur, internal improvement projects, application deployments both in and out-of-hours. --- ## > Second Generation Computing Jan 2008 - Jan 2012 (4 years 1 month) Self-employed part time, consulting to home and small business users for networking setup, small business server deployments, technical troubleshooting and repairs. --- ## > Canterbury District Health Board (Te Whatu Ora Waitaha) ### Linux / Unix Systems Operations Engineer Nov 2007 - Jan 2012 (4 years 3 months) System Engineer managing large scale deployments of Linux and Windows systems, network and stability engineering. Primary systems: - Running and maintaining the primary patient management system for the greater South Island. - Canterbury Health Laboratories Delphic Servers. Engineer and Administrator for a wide variety of Unix/Linux and Windows servers that ran a large number of electronic health and medical systems throughout the South Island of New Zealand. --- ## > Dealsafe Ltd ### Founder & Product Engineer Jul 2010 - Aug 2011 (1 year 2 months) Web Directory startup, online trades and company review directory platform. --- ## > Doortech Ltd ### Engineer Sep 2005 - Nov 2007 (2 years 3 months) Working with electronic and mechanical security and safety systems. - [Screen saver](https://smcleod.net/screensaver/index.md): Screen saver --- title: "Screen saver" description: "eSheep wandering the desktop, or leaves falling past it after Haiku's Leaves screen saver." icon: "sheep" deskbarHidden: true --- Sheep wander over your windows after a few minutes without input (or leaves fall, if you pick Leaves in the [Control panel](/control-panel/?pane=system)), but never while a post is open in the reader. Choosing Screen saver from the menu, or typing `screensaver` in the [Terminal](/terminal/), starts it straight away. Move the mouse, tap or press a key to come back. Without JavaScript there is no screen saver. - [Sketch](https://smcleod.net/sketch/index.md): Sketch --- title: "Sketch" description: "A small drawing app for mouse, finger or pen." icon: sketch window: sketch --- Sketch needs JavaScript. Your last drawing stays in this browser, and **Save PNG** downloads it. - [Success](https://smcleod.net/contact-success/index.md): Success --- title: "Success" description: "Message Sent" aliases: ["message"] # /contact/ and /email/ belong to contact.md author: "Sam McLeod" norss: true deskbarHidden: true # keeps it out of Tracker and Spotlight toc: enable: false auto: false --- **Success** Your message has been sent. {{< script >}} setTimeout(function() { window.location.href = "/"; }, 3000); {{< /script >}} - [Terminal](https://smcleod.net/terminal/index.md): Terminal --- title: "Terminal" description: "A small shell for browsing posts by year, tag and category." icon: "term" window: terminal deskbarHidden: true --- A small shell for browsing this site: list posts by year, tag and category, search their text, and open them in the reader. Type `help` for the commands. The Terminal needs JavaScript; without it, [all posts](/posts/) are one click away. - [Favourite Tools](https://smcleod.net/links/index.md): Favourite Tools --- title: "Favourite Tools" # "Tools" inside the Favourites folder, which sorts and labels by link title linkTitle: "Tools" description: "Software and services I use and like." # type: pages series: [ BestOf ] norss: true # The Favourite Tools folder: data/applications.yaml, shown as icons only layout: folder window: folder icon: software windowWidth: 720 # 1.2 x the folder window's 600 windowHeight: 546 # 1.3 x its 420 items: applications itemPages: force-hidpi: /posts/2026-03-30-apple-silion-4k-limitations --- - [YouTube Channels](https://smcleod.net/youtube/index.md): YouTube Channels --- title: "YouTube Channels" description: "Channels and playlists I watch, each with its latest video." series: [ BestOf ] norss: true # Cards from data/youtube.yaml; the latest videos come from each channel's RSS at build time layout: folder window: folder icon: videos view: cards thumbSize: 320x180 items: youtube windowWidth: 720 windowHeight: 546 --- More favourites: [Podcasts](/podcasts/), [Hardware](/hardware/), [Tools](/links/) and [Blogs](/blogs/). # tools - [Agentic Coding Costs](https://smcleod.net/tools/agentic-coding-costs/index.md): Agentic Coding Costs --- title: "Agentic Coding Costs" description: "Agentic versus manual coding cost comparison." layout: tool window: tool icon: chart deskbarHidden: true # retired from Applications; the page and tool still work frame: /tools/agentic_coding_costs/index.html norss: true --- - [AI Energy and Water Use](https://smcleod.net/tools/old/ai-consumption/index.md): AI Energy and Water Use --- title: "AI Energy and Water Use" description: "AI energy and water consumption, with the data in context." layout: tool window: tool icon: energy frame: /ai-consumption/index.html norss: true aliases: ["/tools/ai-consumption/"] --- - [Applications](https://smcleod.net/tools/index.md): Applications --- title: "Applications" description: "Calculators, dashboards, comparisons and small apps." layout: folder window: folder searchKind: tool icon: apps windowWidth: 750 windowHeight: 384 norss: true include: ["/vram-estimator", "/terminal", "/sketch", "/enterprise", "/chiptunes", "/feeds", "/screensaver"] --- - [LLM Quantisation Dashboard](https://smcleod.net/tools/quantisation-dashboard/index.md): LLM Quantisation Dashboard --- title: "LLM Quantisation Dashboard" folderWeight: 2 description: "How quantisation types compare on size and quality." layout: tool window: tool icon: quantise frame: /quantisation-dashboard.html norss: true --- - [Old](https://smcleod.net/tools/old/index.md): Old --- title: "Old" folderWeight: 3 description: "Older apps that still work but are no longer updated." layout: folder window: folder searchKind: tool icon: folder norss: true include: ["/agentic-coding-tools"] --- - [Tier List Creator](https://smcleod.net/tools/tiers/index.md): Tier List Creator --- title: "Tier List Creator" description: "Drag items into tiers and export the result." layout: tool window: tool icon: tiers frame: /tiers.html norss: true --- # blogs - [AI](https://smcleod.net/blogs/ai/index.md): AI --- title: "AI" description: "AI blogs, news and changelogs I read." norss: true # Cards from data/blogs_ai.yaml; the latest post comes from each site's RSS at build time layout: folder window: folder icon: ai view: cards thumbSize: 240x240 items: blogs_ai windowWidth: 720 windowHeight: 546 --- - [Blogs](https://smcleod.net/blogs/index.md): Blogs --- title: "Blogs" description: "Blogs and news sites I read." layout: folder window: folder icon: blogs norss: true --- # series - [AI](https://smcleod.net/series/ai/index.md): AI --- title: AI description: This is a series of articles about AI / LLMs. --- - [Code](https://smcleod.net/series/code/index.md): Code --- title: Code description: This is a series of articles about code / scripting. --- - [Open Budgeting](https://smcleod.net/series/open-budgeting/index.md): Open Budgeting --- title: Open Budgeting description: This is a series of articles following my ongoing financial expenses on subscriptions and licenses. --- - [Series](https://smcleod.net/series/index.md): Series --- title: Series --- - [SSD SAN Storage Cluster Design](https://smcleod.net/series/storage/index.md): SSD SAN Storage Cluster Design --- title: SSD SAN Storage Cluster Design description: A series of articles on designing an SSD SAN storage cluster. --- - [ZSH](https://smcleod.net/series/zsh/index.md): ZSH --- title: ZSH description: A series on my Z-Shell configuration, scripts and hacks. --- # photos - [Photos](https://smcleod.net/photos/index.md): Photos --- title: "Photos" description: "Photo albums from posts and trips." layout: photos window: photos icon: photos norss: true ---