PageFortress

Reviving This Blog

Wed, 15 Jul 2026 04:30:00 GMT

The last post here went up in March 2020, a guide to keeping monitoring simple. Then the world got interesting, and this site sat frozen while the bio drifted two jobs out of date. If you visited recently, the sidebar and the about page couldn't even agree on where I worked. I'll take the hit on that one.

Funny part is, the first post on this blog was about picking Pelican and Netlify specifically to remove the friction that killed my old Hugo site, so I could "gain back that drive to produce content." Three posts and then six years of silence says the friction was never really the stack. Since that last post I picked up a Principal title at Autodesk, jumped to Lambda to build GPU cloud infrastructure, and moved into engineering management. The writing energy went into RFCs, decision docs, and CNCF work instead, and the blog lost every prioritization battle it entered.

So this is the rebuild. The site now runs on Astro with the AstroPaper theme: dark mode, an RSS feed that's actually on (I had feeds disabled for eight years and didn't know until this migration), a talks page, and social cards that unfurl properly. Netlify stays. It's been quietly deploying this thing since 2018 and has earned its keep. Comments are gone. Disqus wasn't worth what it cost, and if you want to argue with me about something I wrote, I'm easy to find.

Full disclosure: an AI agent did most of this migration while I made the calls. It audited the old setup, caught that the build was pinned to a Python version Netlify no longer ships (the next deploy would have failed), converted the old posts, and wired up redirects so old links still work. Working with agents like this is a big part of my day job now, and it's exactly the kind of thing I want to write about here.

What to expect going forward: platform engineering and Kubernetes, CNCF and TAG Infrastructure community work, engineering management, and the occasional personal project. I'm not promising a cadence. I'm promising the next gap won't be six years.

Knowing Is Half the Battle

Mon, 09 Mar 2020 00:33:00 GMT

One of our current struggles with stabilizing a platform is usually an overabundance of data generated from systems, presented on dashboards, and generating noise for our monitoring/alerts. Too many times we have had red herring after red herring as we try to decipher what exactly the data is telling us. There is no guarantee that an on-call engineer or administrator will know where to look or what queries will yield meaningful insight into the current system state.

Even if you have experience with setting up monitoring/alerting for your distributed systems/micro-services, I highly suggest you keep reading. I will do my best to suggest some best practices when it comes to these topics and hope they help you bridge the gap between Development and Ops.

We need to go back and understand the basics of simple monitoring. I'll go over some solid signals to monitoring, that should help identify and isolate problems during an incident. The goal is the reduction of erroneous on-call engineer pages, more time spent on identifying and troubleshooting the root cause, and quicker MTTD (Mean Time to Detection) and MTTR (Mean Time to Resolution).

Golden Signals

One of the things I will reference as a core theme for this post is Google's "Four Golden Signals" from its SRE book. These signals are latency, traffic, errors, and saturation. I highly recommend reading up on them and keeping it in the back of your mind as we will use this when going over the type of things we will be monitoring depending on the type of service.

As stated previously, one of the main problems faced is an overabundance of data. We do not do a great job of aggregating or isolating the important "four signals" for our services. Great aggregation is magical. Systems instantly appear simpler and the extra application data is isolated to where they belong. Having a good system for aggregation and classification of your services reduces the amount of thinking required to troubleshoot.

A lot of systems can be easily classified into 4 categories: HTTP/RPC, Queue Processing, Stream Processing, and Job Processing. Taking into consideration Google's "Golden Signals", you'll see that for each different type of service, there will be different golden signals that really give a good overview of a system's health without needing much raw data to analyze.

HTTP/RPC

(Traffic) Request rate — how much traffic is the service dealing with, measured in queries per second.
(Errors) Error rate — how much of the incoming requests are failing, measured as a portion of total traffic.
(Latency) Duration — how long are we taking to process each request, measured as a percentile on a histogram, usually the 99th percentile.

Queue Processor (SQS, ZeroMQ, RabbitMQ, etc)

(Latency/Traffic) Lag — how much time passed between the message ingress time and when we start processing it, measured in milliseconds.
(Traffic/Saturation) Net queue size — how many messages were written to the queue minus how many messages were removed from the queue in a given time frame.
(Errors) Error rate — the number of messages we failed processing divided by the number of messages we tried to process. Measured as a percentage.
(Latency) Duration — see HTTP/RPC Duration.

Stream Processor (Batch File, Kafka, Kinesis)

(Latency) Lag per partition per ingress topic — Kafka, for example, takes topics as input, then splits each topic into one or more partition. The lag is determined based on the difference in index position of the consumer groups across multiple partitions. Measured in message counts.
(Errors) Errors — see queue processor error rate.
(Traffic) Throughput — a sum of how many messages the processor emits in a given time frame, for a specific topic.

Scheduled Job (cron, K8s, etc.)

(Errors) Failed jobs — a count of how many jobs failed to run to completion.
(Traffic/Saturation) Missed Executions — by nature, scheduled jobs depend on a scheduler being available to invoke them. We monitor missed executions by comparing actual attempts vs duration.
(Latency) Duration — see other duration signals.

These key signals should be kept in mind when designing alerts or monitoring around the different services worked on everyday. This does not mean these signals should be the only ones you use in terms of monitoring the health of your system, but it should be a good starting point or a bare minimum to help clear out any unwanted red herrings or unnecessary data points.

Platform Metrics

Up until now, we were solely focused on application metrics and what good "golden signals" for that specific type of data looked like. However, you'll notice a lack of representation of one signal, Saturation. While saturation is easily measured for some services, others are not as easily measured based on just application data alone. That is where platform metrics come in.

These are your typical platform metrics:

CPU
Memory
Disk
- Free Space
- I/O
Network
- I/O (throughput)
- latency/loss

Platform metrics alone are the biggest culprit of red herrings and noisy alerts. Thresholds are often set too high or too low. Dashboard resolution (time spans) are often set too long or too short to reveal the correct picture. It is easy to get rabbit-holed into a CPU spike, or a memory dip, but remember that they tend to lead in the wrong direction without corroborating evidence from logs or application data. Platform metrics are at best circumstantial.

Keep. It. Simple. Silly.

My favorite part of the Golden Signals section is "As Simple as Possible, No Simpler".

It highlights just how easy it is for a monitoring system to become excessively complex even with the right intentions, and I believe it highlights where exactly what to strive for as a whole when it comes to monitoring our systems and services.

A monitoring system can be fragile, complicated, and a maintenance burden. It ties up engineering teams from working on new products and services, and contributes to our ever growing mountain of tech debt. But we can change that. Start back at the basics and keep things simple and organized. It will be the only way we fully understand the services we created and keep our systems understandable.

Please feel free to leave feedback or questions!

Icingapy

Wed, 03 Oct 2018 19:17:00 GMT

A couple weeks ago, I created a new python project called Icingapy that act as a library for interacting with icinga cgi endpoints. It is published on PyPI as icingapy and can be found here: https://pypi.org/project/icingapy/

One of the reasons I came around to doing this project is the need for interacting with Icinga at a base configuration out of box. To clarify, this is without any additional tooling setup such as Nagios-API or any other additional solutions that require additional configuration of the Icinga/Nagios server.

Why bother? In situations where Icinga 1 is a legacy system or under strict control that additional solutions are not possible, there is still a need for a library such as this.

Currently the library only supports three commands, summary, status, and downtime. I do plan to expand functionality however I felt that these three represent a solid start for the library. I rather focus on a few core methods first to make sure they are reliably working before expanding the feature creep.

Let me know what you think about the library and feel free to leave issues or submit PRs!

Setting up Pelican and Netlify

Fri, 14 Sep 2018 19:17:00 GMT

Spending all day working as a system administrator, sometimes you just don't want to spend your free time administrating more servers and services. This is the predicament I found myself in. My previous "blog" was just a static site powered by Hugo, and I really liked it. So what happened?

When I had previously setup Hugo it was all done manually, with no build pipeline, or consideration of continuous integration. It was a manual process to make changes, build the static site, and upload the files. I found myself dreading making changes to it or figuring out how to add pages/post actual blog articles. I also took it as an opportunity to gain more experience with Golang. However, the cost was losing the drive to maintain and add content. The learning curve required actively fought my drive to make blog posts or update it with changes. So, I chose to try out Netlify as a possible solution to automating some of the manual parts of maintaining a static site and hopefully gain back that drive to produce content.

After signing up, Netlify setup OAuth to my GitHub account and asked for a repository. Naturally, I paused and realized I needed actual code/content before I could proceed. So naturally with the goal of making things easier, I decided on Python/Pelican. I choose Python because it is my most fluent coding language and Pelican because that was the first recommendation that I found. After taking some time to read and research the documentation, it appeared to be mature and stable.

After setting up a new Git blank git repo, I made sure to add the git repo to Netlify so that my first commit would trigger a build. I did the following:

dpage@do-ops:~/Coding/pagefortress.com$ mkvirtualenv -p python3.6 pelican
(pelican) dpage@do-ops:~/Coding/pagefortress.com$ pip install pelican

If you haven't used virtualenv or virtualenv-wrapper, I highly recommend checking it out here: https://virtualenvwrapper.readthedocs.io/en/latest/ It is a GREAT way to manage and organize your python virtual environments all in one place if you do not want to keep them within your project directory.

For example if you use pelican for multiple projects, instead of needing a virtualenv subfolder for each project, the venv is keep in your home directory and activated simply by typing workon pelican

(pelican) dpage@do-ops:~/Coding/pagefortress.com$ pelican-quickstart
Welcome to pelican-quickstart v3.7.1.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.


> Where do you want to create your new web site? [.]
> What will be the title of this web site? PageFortress
> Who will be the author of this web site? Dylan Page
> What will be the default language of this web site? [en]
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n) Y
> What is your URL prefix? (see above example; no trailing slash) http://pagefortress.com
> Do you want to enable article pagination? (Y/n)
> How many articles per page do you want? [10] 5
> What is your time zone? [Europe/Paris] Eastern
Please enter a valid time zone:
 (check [http://en.wikipedia.org/wiki/List_of_tz_database_time_zones])
> What is your time zone? [Europe/Paris] America/Detroit
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n)
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n)
> Do you want to upload your website using FTP? (y/N)
> Do you want to upload your website using SSH? (y/N)
> Do you want to upload your website using Dropbox? (y/N)
> Do you want to upload your website using S3? (y/N)
> Do you want to upload your website using Rackspace Cloud Files? (y/N)
> Do you want to upload your website using GitHub Pages? (y/N)
Done. Your new project is available at /home/dpage/Coding/pagefortress.com

Now that the project core is ready, setup Netlify to build the site using the command:

(pelican) dpage@do-ops:~/Coding/pagefortress.com$ pelican content
Done: Processed 1 article, 0 drafts, 1 page and 0 hidden pages in 0.21 seconds.

If you run into issues with the build failing, using the -D option for debug output, to review the logs.

You can find the Pelican install documentation here: http://docs.getpelican.com/en/stable/install.html

Also make sure to freeze your pip versions to requirements.txt and set a python version in runtime.txt. See: https://www.netlify.com/docs/continuous-deployment/#set-node-ruby-or-python-version

Next, it was simple as picking a theme (http://docs.getpelican.com/en/stable/settings.html#themes) and start to write content! I chose reStructuredText, but Markdown is also supported.