Category: resources

  • Mapping complex systems without getting lost in documentation

    Mapping complex systems without getting lost in documentation

    Good infrastructure is maintainable, scalable, robust, resilient and secure. And it stays this way by putting processes into place which perform upkeep on all of these factors. So, how can you ensure that these upkeep processes are healthy and understood by everyone?

    People don’t think in text

    Our brains aren’t made for thinking in text. Try to remember the last page of the last book you read.

    Now try to remember the route you took from your childhood bedroom to the kitchen.

    We are visual-spatial thinkers. And as such, our documentation should be primarily visual-spatial.

    Flowcharts, even when done on the back of a napkin, can show you more than a full-length page of documentation.

    But why settle for napkins? You can use tools like excalidraw to draw them out, or even better; use a code-based tool to have them drawn for you.

    That way you can save the code in your versioncontrol system and see how your diagrams – and your infra, or your understanding of it – evolved over time.

    Let’s practice a little

    I use a notetaking tool called obsidian. In it, a widely used language for marking up flowcharts is used: mermaid.

    Mermaid is easily integrated into any pipeline for documentation builds. Gitlab even has it integrated into their markdown renderer!

    So, let’s learn by building.

    Step 1: Start with a basic flowchart

    flowchart TD
        A[Start] --> B[Do something] --> C[Finish]
    

    This is a basic top-down flowchart:

    • flowchart TD tells Mermaid we’re drawing a top-down flowchart.
    • The arrows (-->) connect nodes.
    • Each node has an ID (e.g. A) and a label (e.g. Start).

    Try changing the labels or rearranging the arrows. That’s the quickest way to get a feel for how it works.

    Step 2: Add conditional logic

    Let’s make it interactive by adding a decision.

    flowchart TD
        A[Start] --> B{Do you agree?}
        B -- Yes --> C[Continue]
        B -- No --> D[Stop here]
    

    Here we introduce:

    • {curly braces} to represent a decision node.
    • Multiple arrows coming out of the same node with labels (Yes and No).

    Now we have a mini flow that resembles real-world logic.

    Step 3: Modularizing with subgraphs

    As diagrams grow, structure helps. Let’s group nodes into a logical block.

    flowchart TD
        subgraph Process
            A[Start] --> B{Do you agree?}
            B -- Yes --> C[Continue]
            B -- No --> D[Stop here]
        end
        C --> E[Wrap up]
    
    • subgraph NAME groups related nodes.
    • end closes the group.
    • The outer node E connects to the subgraph.

    This is great for readability and managing large diagrams.

    Step 4: Change direction and style

    Want things to flow left to right?

    flowchart LR
        A[Login] --> B{Is password correct?}
        B -- Yes --> C[Dashboard]
        B -- No --> D[Show error]
    
    • LR means Left to Right.
    • You can also use RL (Right to Left), BT (Bottom to Top).

    Step 5: Make some sequence diagrams and GANTT charts!

    I think you get the gist of it. Let’s show you some more basic examples. How about a sequence diagram?

    sequenceDiagram
        Alice->>+John: Hello John, how are you?
        Alice->>+John: John, can you hear me?
        John-->>-Alice: Hi Alice, I can hear you!
        John-->>-Alice: I feel great!

    Or even a GANTT chart, for all of you aspiring project managers:

    gantt
        title A Gantt Diagram
        dateFormat  YYYY-MM-DD
        section Section
        A task           :a1, 2025-01-01, 30d
        Another task     :after a1  , 20d
        section Another
        Task in sec      :2025-01-12  , 12d
        another task      : 24d
    

    And if you want to get even fancier, just go visit the docs at mermaidjs.

    This skill is crucial

    While this might seem like a fun inocuous little tutorial for mermaidjs, the point I’m trying to make is this:

    There is no excuse for not having diagrams for your infrastructure, your network and your application services — and there is tremendous value in them.

    I have, in fact, on occasion helped technical teams solve their own problems just by diagramming it out. So do yourself a favor. Get your team to do diagrams instead of plain text.

    And if they’re not up for it, why not give me a call?

  • Auryn: a CLI pipeline DSL

    Auryn: a CLI pipeline DSL

    Auryn: A Domain-Specific Language for CLI Pipelines

    Sometimes you just can’t help but write a bash script to chain together a couple of tools. Sometimes those bash scripts get out of hand, and you wish you had something that could effectively be the linux pipe symbol, but also clean and format your data for the next CLI program.

    Auryn has your back.

    The challenge with CLI pipeline tooling

    Every pipeline engineer has attempted to chain tools together using Bash scripts or integrate them into Jenkins pipelines. The reality is that there’s a lack of standardized input and output formats across CLI tools.

    Each tool serves a different purpose, making automation cumbersome—even though the process is often repetitive:

    1. Read a file.
    2. Optionally transform the input.
    3. Apply the tool with specific parameters.
    4. Save the (potentially massive) output to a file.
    5. Clean the output for the next tool, which will…
    6. Repeat the cycle.

    Introducing Auryn: a CLI glue pipeline DSL

    Enter Auryn—a domain-specific language (DSL) designed to streamline shell pipelines. Think of it as a platypus: seemingly uncertain of its identity, yet functioning exceptionally well because of it.

    Here’s a glimpse of Auryn in action:

    run "bash cert_transparency_check.sh __DOMAIN__"
      parsewith cert_transparency_check 
      output "__OUTPUT_DIR__/found_domains.txt"
      as $domain_list

    In this snippet:

    • A script is executed with a constant (__DOMAIN__) as its only known truth.
    • The output is parsed using a simple parser you write in a shell script, tailored to the unique output of each CLI tool.
    • The parsed text can be saved to a file and stored in a variable ($domain_list) for use in subsequent tools.

    Why Auryn Matters

    Auryn addresses the fragmentation in CLI tooling by providing a unified language to define and manage shell workflows. It empowers security engineers to automate complex pipelines without being bogged down by inconsistent tool interfaces.

    The inspiration for Auryn

    Tom learnt about CPU pipelining and thought the mental model applied to running bash scripts as well. A couple of hours talking to an LLM later, Auryn was born.

    The cycle is:

    • Instruction Fetch
      • Get data
    • Instruction Decode
      • Parse the data for CLI input
    • Execute
      • Run the CLI program with the supplied parameters
    • Write to memory
      • Put the result into a variable
    • Write back
      • (Optionally) write the raw output to a file

    Auryn is the result of a brain blip, and was put together in a couple of hours yet results in an immensely useful output.

    Get Started with Auryn

    Ready to tame the beast? Grab a copy of Auryn from the project’s GitHub page: https://github.com/kapott/auryn. Feel free to expand upon it. While the repository doesn’t have a license yet, it will be free, as in beer.