Strangulating the monolith to a pluggable and scalable architecture

Global Technology

McDonald’s Technical Blog

Global Technology

·

Follow

Published in

McDonald’s Technical Blog

·

4 min read

·

Feb 28, 2024

3

By extracting microservices from the monolith, McDonald’s developers are creating apps that are more portable and resilient.

by Tasneem Damen, Principal Architect

Many remarkable frameworks exist for greenfield software development but are not always viable for a big-bang replacement of proprietary enterprise applications. Hence, this post will focus on applying strangulating patterns and engineering techniques for brownfield innovation.

Let’s look at the app our Owners/Operators use to manage restaurant systems. The application is a read/write heavy configuration management monolith, peaking at 400 MB/sec daily for a few hours per day, per country. The batch-oriented architecture lacks API-based communication, and the configuration data contains information with varying context and cache expiration periods.

As a result, there are disparate caching layers for millions of consumers, making the end-to-end cache invalidation time hours. A disparate caching layer on the batched data can also make governance challenging, as global market teams from over 100 countries must align to accept information metadata changes.

Approach

A logical start is to look at the information context — curation and authoring process — to bound it effectively. Looking at domain-driven design principles for quick service restaurants and retail domains and following the Pareto Principle, consumer menu management carved out as a series of core and supporting sub-domains is an ideal candidate for strangulation¹.

The batch generation process within the monolith lends itself as a façade, aggregating data from new API-driven domain services to generate the batches, consumed by data movement service for the downstream consumers. The batch generation façade enables gradual business unlock for both upstream and downstream consumers, while remaining backwards compatible. New domain APIs follow McDonald’s standardized ontology while batch generation uses an adapter² pattern to ensure stability for the existing consumers. Moreover, feature flag strategies are also implemented to canary the façade rollout to further de-risk the adoption process by McDonald’s markets. Half of the McDonald’s stores use this capability on daily basis across the globe.

Batch generation, a compute-intense process, modeled as a microservice, enables optimization of daily burst-out scale and independent release management cycles.

Containerizing the batch generation service allows for horizontal scale-out of the storage and computing without incurring the costs of vertically scaling the entire monolith or creating dedicated instances per country. Currently, we use KEDA’s event-driven pod-scaler to provide burst-out scale to maximize resource utilization.

Outcome

True MACH architecture

With the strangulation in place, the business domain and corresponding capability can now be treated as its own product, free from past constraints, to redefine its business excellence. Its architecture can be Microservices-based, API-first, Cloud-native, and Headless (MACH), which differs from the monolith architecture but follows Twelve-Factor App methodology — a methodology for building software-as-a-service applications. These best practices are designed to enable applications to be built with portability and resilience when deployed to the web.

Governance and standardization

A strangulated business capability provides more tools for solving a business problem than implicit conventions of the monolith.

For example, user experience can solve for omnichannel vs. multi-channel menu management strategies. Moreover, core domains with semantic API versioning can iteratively introduce changes globally. Creating a single source of truth minimizes the overall complexity in the end-to-end architecture — from the time a business user authors to when an end consumer sees it on their devices.

Changing business SLAs

Switching from a batch-oriented architecture to an API-driven architecture means new service-level agreements (SLAs) for the new domain microservices receiving consumer service requests throughout the day.

Using batch generation as the first consumer of the domain API helps validate the internal scale-out strategy before syndicating to other external consumers. Pre-cached channel-specific menus are provided from edge nodes, which when coupled with smart validation and delta caches, may exponentially reduce the entire network throughput.

Consumer adoption

As monolith applications are inherently more complex to change, new feature requirements are developed in the new domain microservices and served to consumers over secure APIs with real-time event notification. Consumers are better positioned to honor the service-level objective and request bug updates and enhancements to fuel McDonald’s innovation. Ultimately, this helps unlock new business value and drive the consumer adoption.

¹Strangluation is a software design pattern where the existing system is put behind an intermediary façade or wrapper. Over time new services behind the façade start to replace capabilities of the old system. The process is repeated until the old system is fully strangulated into new services.

²Adapter is a structural design pattern that allows objects with incompatible interfaces to collaborate. In this instance REST JSON objects are converted into XML documents.

Microservices

Monolithic Architecture

Monolithic

Technology

Architecture

3

McDonald’s Technical Blog

Published in McDonald’s Technical Blog

1K Followers

·Last published Dec 3, 2024

Our tech team comprises intrapreneurs who build really cool tech, using innovations like AI, IOT, edge computing — and are defining the future of we do alongside diverse, global teams.

Global Technology

Written by Global Technology

1.8K Followers

·91 Following

Responses (3)

What are your thoughts?

Also publish to my profile

Good Day, Adam

Good Day, Adam

he/him

10 months ago

McDonald's and "Strangulation" are not really the best of word combination representation - maybe "Hamburglar Masking" (as a title).

Even if "indigestion" becomes what happens when servers eject files to another server, it just feels like your "strangulating" McDonalds' image.

How is CosMc's doing?

Additionally, has McDonald's ever considered:

a) a Cafe Du Monde-like awning setting - where the restaurant is located in the middle with restrooms, and people can eat outside-in (like those stall eating areas during COVID)?

b) McDonalds Lounges for airports, bus stations, and certain travel stops? A place where travelers can unwind, try location-specific food, and experience unique world's faire-like flare

c) a "County Fair" meets diner experience - where the diner is a bus, train car, or even a shipping container; order on a large jukebox, or on one of the tables. From Funnel cake sticks to to McCotton Candy Canes, there's a fun unique joy in each experience. In addition you can collect special location-specific removable car window stickers from each location. How about McFair?

d) Lastly - McDonald's aluminum foil, or paper storing bags... or MacBags. MacBags could be used for food and in gneral for storing your storables in a most recyclable package. This would be an interesting pivot, too.

Nevertheless, great read, and I invite you to check out my piece called the Meeting District.

1

Mohammed Brückner

Mohammed Brückner

9 months ago

I must say, I wholeheartedly agree with your assessment of the benefits of transitioning from a monolithic architecture to a microservices-based model. Your discussion of McDonald's experience is a compelling case study, and the emphasis on careful planning, testing, and monitoring throughout the migration process is spot on.

That being said, I do have a few perspectives to add to the conversation. While the benefits of microservices are clear, it's important to acknowledge that this approach is not without its challenges. For example, managing and coordinating a large number of independent services can be complex and time-consuming. Additionally, ensuring that each service is secure and compliant with relevant regulations can be a significant undertaking.

To further elaborate on these points, it's worth noting that a recent survey by O'Reilly found that 35% of organizations cite managing microservices as their biggest challenge, while 31% cite security as a major concern. Additionally, a study by the Cloud Native Computing Foundation found that 57% of organizations have difficulty ensuring compliance in a microservices environment.

Despite these challenges, there are steps that organizations can take to mitigate them. For example, implementing a service mesh can help to manage and coordinate microservices, while automated testing and monitoring can help to ensure security and compliance.

In terms of specific examples, Netflix is a well-known proponent of microservices, and they have developed a number of tools to manage their complex environment. For example, they use a service mesh called Istio to manage communication between services, and they have implemented automated testing and monitoring to ensure security and compliance.

Another example is Amazon, which has transitioned many of its services to a microservices-based model. They use a combination of service discovery, load balancing, and automated deployment to manage their environment, and they have implemented strict security and compliance policies to ensure that each service meets their standards.

Finally, it's worth noting that the transition to microservices is not a one-size-fits-all solution. Each organization must carefully consider its own needs and resources before embarking on this journey. As the Golden Girls might say, "It's like choosing a new pair of shoes - you have to find the ones that fit you best!"

Dan Boerner

Dan Boerner

9 months ago

Congrats on your platform modernization efforts Tasneem.

Curious if you have looked at using Federated GraphQL to support your strangulation pattern? It's a great way to combine a broad set of service APIs together into a single source of truth while insulating clients from versioned endpoints that can change as microservices evolve.

At Apollo we're working with hundreds of large enterprises like McDonalds who are strangling with monoliths with GraphQL. All the best on your efforts!

https://www.apollographql.com/why-graphql

McDonald's Streamlines CI with GitHub Actions and Reusable Workflows

Mar 13, 2024 2 min read

by

- Author photoAditya Kulkarni

Delivery Lead at Secure Privacy | InfoQ DevOps Editor

Write for InfoQ

Feed your curiosity.Help 550k+ global

senior developers

each month stay ahead.Get in touch

Recently, McDonald’s discussed leveraging GitHub Actions to emphasize readability and maintainability and reduce code duplication. Their goals included establishing a streamlined CI process, enhancing developer productivity, and implementing pipeline observability and monitoring.

Michael Gorelik, senior solution architect, and Achintya Pillai, software engineer III at McDonals, elaborated on their journey in the McDonald’s Technical Blog. They talked about establishing a consistent framework — referred to as the "golden path" — to facilitate continuous integration stages, including code quality, security measures, packaging, and tagging. In addition, this framework also enabled the maintenance of uniform quality standards across all their applications and promoted the adoption of standardized naming conventions for deployment artifacts throughout the organization.

McDonald's engineering team manages a broad technology landscape with numerous microservices written in various languages and utilizing different cloud-native services. They utilise reusable workflows and custom actions from GitHub to streamline continuous integration (CI) processes across their applications.

Source: Reduce, reuse, recycle: McDonald’s reusable workflows

To minimize code duplication in their pipelines, the team organized CI workflows by application language types, utilizing GitHub Actions' reusable workflows to create a central CI codebase. These central workflows include stages like building, quality and security scans, artifact creation, and tagging. This structure streamlines maintenance, updates, and code reuse, ensuring standardized application development. Additionally, the team employed containers to store required utilities and libraries, enabling streamlined workflow execution and reducing security risks and execution time.

McDonald's engineering team seeks to extend the golden path approach to all CI processes across their applications. To implement the centralized reusable workflows and actions, a simple CI caller file is integrated into each application, enabling engineers to reference the golden path workflows for CI stages and ensure compliance with required standards.

Earlier this year, GitHub Actions made headlines with the introduction of a new M1 macOS runner. This new runner operates Actions workflows with a VM featuring 3 vCPUs, 7 GB RAM, and 14 GB of storage, showcasing the latest Mac hardware capabilities available for Actions. We stumbled upon an intriguing HackerNews discussion that mentioned an experiment conducted by Alex Ellis using GitHub Actions. The conversation involved comments from the tech community, with one HN user ranting-moth describing GitHub Actions' computing time as costly, particularly about Mac pricing.

Using GitHub Actions for CI, McDonald's Engineering also aims to offer engineers a blend of centralized control and individual autonomy. This approach establishes a central "golden path" for essential steps while enabling developers to add new stages independently, maintaining agility without disrupting other operations.

Finally, in terms of observability, the team has established centralized monitoring solutions by adopting reusable workflows. These solutions assist both DevOps and application teams in regularly monitoring their pipeline performance and pinpointing areas for future enhancements.

About the Author

Aditya Kulkarni

Aditya has been functioning as tech-aware delivery leader over this tenure. He has worked with different organizations on their journey to agility and DevOps transformation. An avid reader, he is always interested in keeping an eye on the latest in the world of software development!

Show more

Reduce, reuse, recycle: McDonald’s reusable workflows

Global Technology

McDonald’s Technical Blog

Global Technology

·

Follow

Published in

McDonald’s Technical Blog

·

5 min read

·

Feb 13, 2024

4

To enable a fast and reliable continuous integration process, McDonald’s turns to reusable workflows and GitHub Actions.

By Michael Gorelik, Senior Solution Architect and Achintya Pillai, Software Engineer III

McDonald’s Engineering teams are at the forefront of digital innovation, creating seamless and engaging e-commerce applications that allow customers to conveniently order their favorite meals using mobile phones.

Our technology landscape is vast, encompassing many microservices, which are written in a wide range of languages and use different cloud-native services. Engineering teams around the globe must build, test, package and publish each of these services multiple times a day to integrate ongoing changes.

Our goal is to create fast, reliable, consistent yet flexible state-of-the-art, continuous integration (CI) processes for all our different applications.

To do this, we use a combination of reusable workflows and custom reusable actions provided by GitHub. While implementing the reusable workflows with GitHub Actions, we have focused on a few key objectives for improvements:

1. Maximize readability and maintainability by reducing code duplication and complexity.

2. Create a golden path for the continuous integration process.

3. Maximize developer productivity by increasing autonomy without sacrificing quality and standardization.

4. Implement pipeline observability and monitoring.

High-level process

Increased reliability and code maintainability

To reduce the pipeline code duplication, we grouped CI workflows for applications, based on their language types and using reusable workflows on GitHub Actions to make a central CI codebase. The centralized CI workflows include the bread-and-butter stages required for any continuous integration process, such as build, quality and security scans, and create and tag artifacts. This reusable structure provides us with a common location to maintain, update and reuse code and helps ensure our applications are following a certain standard. We have also used containers to store required utilities and libraries, allowing us to run various stages within our container. Using containers for workflow executions helps avoid downloading multiple external libraries during pipeline execution, reduce pipeline execution time and eliminate security risks from external libraries.

Golden path CI process

One of our goals was to create a standardized golden path for continuous integration stages, such as code quality, security, packaging and tagging. This helps us maintain an equal quality standard across all our applications and helps us maintain a standardized naming convention for deployment artifacts across the organization.

We aim to propagate the golden path to all our applications’ CI processes. With our central reusable workflows and actions, we can ensure all our applications follow a certain set of required steps. To achieve this setup, we inject a simple CI caller file into every application, so engineers can reference the golden path reusable workflows for the CI stages and ensure their delivered application is of a certain required standard. A high-level illustration of the process is shown below:

Developer productivity

A common issue with centralized codebases like the one described above, is the frequent requests for code changes, which can take a long time, due to the many approval processes and gates that need to be addressed by a central team. Centralized repositories can be impactful because they are being shared by many teams.

With our GitHub Actions codebase for CI, we aim to give our engineers a balance of centralized control and individual autonomy. This approach will allow the organization to create and sustain a central golden path for certain required steps and provide developers the flexibility to add new stages at any time, without impacting other operations and sacrificing agility.

Let us dive into the changes process. When a CI pipeline is run, the CI caller file first references our golden path, which executes a few required stages, to maintain a certain level of quality, security, and standards across all our applications. This part will need some level of central control by an organization’s platform leads. Below is an illustration of the pull request process our engineers would follow if they needed to make a change to the defined golden path in the central CI workflow.

Although, if engineers need to add a custom CI stage for their application-specific workflow, they can add a custom action to their CI caller YAML file, without impacting other teams and going through a pull-request process, as shown in the diagram below:

Observability and monitoring

To enable enterprise-level visualization of the workflows, workflow queues and workflow logs across multiple repositories, we use CI visibility tools that allow easy integration and visualization of GitHub action workflows.

With our reusable workflows, we can unlock central monitoring solutions to help DevOps teams and applications teams monitor their pipeline performance frequently and identify future improvements. We capture the following metrics from our pipelines:

1. Pipeline count

2. Repository commit history that triggered the pipeline

3. Pipeline lead times

4. Pipeline success and failure rate

5. Monitor branches used in the GitHub Actions codebase

In conclusion, reusable workflows and GitHub Actions are powerful tools to help engineers create a robust process for running their CI pipelines for many applications with various languages, versions, and frameworks. With the GitHub Actions features and tools, we have implemented a reusable component-based CI system, which addressed reliability, maintainability, productivity, and monitoring. Continuous integration golden path helps to increase user’s agility and innovation while providing built-in standardization and quality.

Workflow

Github

Ci Cd Pipeline

Technology

Integration

4

McDonald’s Technical Blog

Published in McDonald’s Technical Blog

1K Followers

·Last published Dec 3, 2024

Our tech team comprises intrapreneurs who build really cool tech, using innovations like AI, IOT, edge computing — and are defining the future of we do alongside diverse, global teams.

Global Technology

Written by Global Technology

1.8K Followers

·91 Following

Responses (4)

What are your thoughts?

Aaron Gonzales

Aaron Gonzales

he/him

11 months ago

Great read! We use a similar breakdown for our reusable workflow management. Have a couple Qs though, how do you manage changes across workflow inputs? Are the caller workflows injected in the repository basically calling a single reusable workflow with a large list of inputs/secret values?

7

1 reply

Ciro Miranda

Ciro Miranda

9 months ago

Thanks for sharing. Very interesting how you setup GitHub Actions with reusable workflows. It would be great if you had more details about the observability and monitoring section. Which tools are you using? How are you tagging your pipelines?

4

Tane Piper

Tane Piper

10 months ago

This is very similar to the process the team I lead is going through, as we move our new platform to production, nice to see a write up on it.

HArish Pillai

HArish Pillai

11 months ago

Need of the Hour....

GitHub Actions as a time-sharing supercomputer

The time-sharing computers of the 1970s meant operators could submit a job and get the results at some point in the future. Under the guise of "serverless", everything old is new again.

AWS Lambda reinvented the idea of submitting work to a supercomputer only to receive the results later on, asynchronously. I liked that approach so much that in 2016 I wrote a prototype to unlock the idea of functions but for your own infrastructure. It's now known as OpenFaaS and has over 30k GitHub stars, over 380 contributors and its community have given hundreds of blog posts and conference talks.

There's something persuasive about running jobs and I don't think it's because developers "don't want to maintain infrastructure".

I know this, it's a UNIX system

"I know this, it's a UNIX system"

See my Twitter thread as I built the actions-batch tool.

Prior work

I mentioned OpenFaaS and to some extent, it does for Kubernetes what time-sharing did for mainframes in the early 60s and 70s.

You can write functions in application code or bash and wrap them in containers, then have them autoscale, scale to zero, with built-in monitoring an a REST API for automation.

For a couple of examples of bash see my openfaas-streaming-templates or the samples written by a Netflix engineer for image and video manipulation.

With OpenFaaS you write code once and then that acts as a blueprint, it can be scaled, triggered by cron, Kafka and databases, run synchronously or asynchronously with retries and callbacks built-in to receive the results.

But sometimes all you want is a one-shot task.

In the Kubernetes APIs, we have a "Job" that can be scheduled. So my initial experiments involved writing a wrapper for that, which we use for customer support at OpenFaaS.

Fixing the UX for one-time tasks on Kubernetes

I'd also had a go at something similar for Docker Swarm which companies were using for cleaning up database indexes and running nightly cron jobs.

actions-batch

actions-batch is an open-source CLI available on GitHub

asciicast

An ASCII cast of building a Linux Kernel, and having the binary brought back to your own computer to use.

So with the comparison to OpenFaaS out of the way, and some prior work, let's look at how actions-batch works.

1. A new GitHub repository is created

2. A workflow is written which runs "job.sh" upon commits

3. When a local bash file is written to the repo as "job.sh", the job triggers

That's the magic of it. We've created an "unofficial" API which turns GitHub Actions into a time-sharing supercomputer.

The good bits:

- You can include secrets

- You can fetch the outputs of the builds

- You can use self-hosted runners or hosted runners

- Private and public repos are supported

Build a Linux Kernel and bring it back to your machine

Let's say you're running an Apple MacBook, and need to build a Linux Kernel? You may not have Docker installed, or want to fiddle with all that complexity.

mkdir kernels

actions-batch \

--owner alexellis \

--org=false \

--token-file ~/batch \

--file ./examples/linux-kernel.sh \

--out ./kernels

Then:

┏━┓┏━╸╺┳╸╻┏━┓┏┓╻┏━┓   ┏┓ ┏━┓╺┳╸┏━╸╻ ╻

┣━┫┃   ┃ ┃┃ ┃┃┗┫┗━┓╺━╸┣┻┓┣━┫ ┃ ┃  ┣━┫

╹ ╹┗━╸ ╹ ╹┗━┛╹ ╹┗━┛   ┗━┛╹ ╹ ╹ ┗━╸╹ ╹

By Alex Ellis 2023 -  (232d61a253f0805b85d60fecf87f5badbb53047b)

Job file: linux-kernel.sh

Repo: https://github.com/alexellis/hopeful_goldwasser3

----------------------------------------

View job at:

https://github.com/alexellis/hopeful_goldwasser3/actions

----------------------------------------

Listing workflow runs for: alexellis/hopeful_goldwasser3 max attempts: 360 (interval: 1s)

Without installing anything on your computer, in a minute or two, you'll get a vmlinux that's ready to use.

Contents of: ./kernels

FILE    SIZE

vmlinux 22.71MB

QUEUED DURATION TOTAL

3s     2m51s    2m57s

Of course, hosted runners are known for being great value, but particularly slow. So we can run the same thing on our own, more powerful infrastructure:

actions-batch \

--owner actuated-samples \

--token-file ~/batch \

--file ./examples/linux-kernel.sh \

--out ./kernels \

--runs-on actuated-24cpu-96gb

In this example, a 24vCPU microVM was used with 96GB of RAM allocated. Of course, you never need this much RAM to build a Kernel, but it shows what's possible.

If you want to know how much disk, RAM or vCPU you need for a GitHub Action, you can use the actuated telemetry action.

Once complete, the repository is deleted for you.

temporary-repo

The repository is part of the "batch job" specification

Run some ML/AI using Llama

You can run inference using a machine learning model from Hugging Face.

Here's how to get a Llama2 model to answer a bunch of questions that you provide with 150 tokens being used.

examples/llama.sh

Example of running inference against a pre-trained model

Example of running inference against a pre-trained model

Download a video from YouTube

actions-batch \

--owner alexellis \

--org=false \

--token-file ~/batch \

--file ./examples/youtubedl.sh \

--out ~/videos/

This will create a file named ~/videos/video.mp4 with the UNIX documentary by Bell Labs.

See a screenshot of the results

Since writing the post, I've added an example for Whisper from OpenAI, and run it using actuated.dev so that I could use a GPU in an isolated microVM rather than having to use Docker insecurely. We had to add support for cloud-hypervisor to mount GPUs since this isn't supported in Firecracker.

Imagine you have a folder with a bunch of audio tracks, and you just submit a batch job and get the transcriptions back on your computer when you've had dinner, or come back from the gym? That's what batch job system is all about.

It can take a long time, it can even be quick, but it's about submitting a work item and getting the results later on.

This example was on CPU using a bare-metal host on Hetzner, within a Firecracker VM. The same example will run on hosted runners.

OIDC tokens

You can use GitHub's built-in OIDC tokens if you need them to federate to AWS or another system.

#!/bin/bash

# Warning: it's recommend to only run this with the --private (repo) flag

env

OIDC_TOKEN=$(curl -sLS "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=https://fed-gw.exit.o6s.io" -H "User-Agent: actions/oidc-client" -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN")

JWT=$(echo $OIDC_TOKEN | jq -j '.value')

jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "$JWT"

# Post the JWT to the printer function to visualise it in the logs

# curl -sLSi ${OPENFAAS_URL}/function/printer -H "Authorization: Bearer $JWT"

Deploy a function to OpenFaaS using secrets

We've seen how to download artifacts from a build, but what if our job needs a secret?

First, create a folder called .secrets.

Then add a file called .secrets/openfaas-gateway-password with your admin user and then create another file called .secrets/openfaas-url with the URL of your OpenFaaS gateway.

Two repo-level secrets will be created named: OPENFAAS_GATEWAY_PASSWORD and OPENFAAS_URL. They can then be consumed as follows:

curl -sLS https://get.arkade.dev | sudo sh

arkade get faas-cli --quiet

sudo mv $HOME/.arkade/bin/faas-cli /usr/local/bin/

sudo chmod +x /usr/local/bin/faas-cli

echo "${OPENFAAS_GATEWAY_PASSWORD}" | faas-cli login -g "${OPENFAAS_URL}" -u admin --password-stdin

# List some functions

faas-cli list

# Deploy a function to show this worked and update the "com.github.sha" annotation

faas-cli store deploy env --name env-actions-batch --annotation com.github.sha=${GITHUB_SHA}

sleep 2

# Invoke the function

faas-cli invoke env-actions-batch <<< ""

Run curl remotely, if you want to check if it's your network

Sometimes, you wonder if it's your network that's the issue. So you DM someone on Slack: "Can you access XYZ?"

Let the super computer do it instead:

#!/bin/bash

set -e -x -o pipefail

# Example by Alex Ellis

curl -s https://checkip.amazonaws.com > ip.txt

mkdir -p uploads

cp ip.txt ./uploads/

Results:

Found file: 6_Complete job.txt

---------------------------------

2023-12-22T11:59:23.6683796Z Cleaning up orphan processes

Contents of: /tmp/artifacts-2603933045

FILE   SIZE

ip.txt 15B

QUEUED DURATION TOTAL

3s     13s      19s

Deleting repo: actuated-samples/vigorous_ishizaka8

cat /tmp/artifacts-2603933045/ip.txt

172.183.51.127

Well 172.183.51.127 is definitely not my IP. It worked.

Build a container image remotely, then import it

Sometimes I build ML and AI containers on Equinix Metal because they have a 10Gbps pipe, and I may well be on holiday or in a cafe with 1Mbps available.

Let's submit that batch job!

#!/bin/bash

set -e -x -o pipefail

# Example by Alex Ellis

# Build and then export a Docker image to a tar file

# The exported file can then be imported into your local library via:

# docker load -i curl.tar

mkdir -p uploads

cat > Dockerfile <<EOF

FROM alpine:latest

RUN apk --no-cache add curl

ENTRYPOINT ["curl"]

EOF

docker build -t curl:latest .

Finally:

./actions-batch \

--org=false \

--owner alexellis \

--token-file ~/batch \

--file ./examples/export-docker-image.sh \

--out ./images/

....

Contents of: ./images/

FILE     SIZE

curl.tar 12.37MB

QUEUED DURATION TOTAL

5s     22s      29s

Then let's import that curl image:

docker rmi -f curl

docker images |grep curl

docker load -i ./images/curl.tar

38d2771a5c36: Loading layer [==================================================>]  4.687MB/4.687MB

Loaded image: curl:latest

docker run -ti curl:latest

curl: try 'curl --help' or 'curl --manual' for more information

It worked just as expected.

Let's have a race?

Here, I've submitted the same job both to an x86_64 server and an arm64 server both on my own infrastructure. They'll build a Linux Kernel using the v6.0 branch.

Off to the binary races - what's quicker? vmlinux or Image?

This is also a handy way of comparing GitHub's hosted runners with your own self-hosted infrastructure - just change the "--runs-on" flag.

The youtubedl.sh example is multi-arch aware, and uses a bash if statement to download the correct version of youtubedl for the system. Same thing with the Linux Kernel example you'll find in the repo.

Wrapping up

I hope this idea captures the imagination in some way. Feel free to try out the examples and let me know how it can be improved, and whether this is something you could use.

Q&A:

Where are the examples?

I've added a baker's dozen of examples, but would welcome many more. Just send a PR and show how you've run the tool and what output it created.

https://github.com/alexellis/actions-batch/tree/master/examples

Will GitHub be "angry"?

We often talk about brands and companies as if they were a single person or mind. GitHub is not one person, but the GitHub team tend to love and encourage innovation and have built APIs in order to be able to make use of GitHub Actions in this kind of way.

The most relevant clauses are: C. Acceptable Use and H. API Terms.

Exercise common sense.

Should I feel bad about using free runners for batch jobs?

Use your own discretion here. If you think what you're doing doesn't align with the terms of service, use a private repo, and pay for the minutes.

Or use your own self-hosted runners with a solution like actuated

Could I run this in production?

The question really should be: is GitHub Actions production ready? The answer is yes, so by proxy, you could run this tool in production.

What's the longest job I can run?

The limit for hosted and self-hosted runners is 6 hours. If that's not enough, consider how you could break up the job into smaller pieces, or perhaps look at run-job or OpenFaaS.

Why not use Kubernetes Jobs instead?

Funny you asked. In the introduction I mentioned my tool alexellis/run-job which does exactly that.

How is this different from OpenFaaS?

Workloads for OpenFaaS need to be built into a container image and are run in a heavily restricted environment. Functions are ideal for many calls, with different inputs.

actions-batch only accepts a bash script, and is designed to run in a full VM, running administrative tasks and tools like Docker. It's designed to only run periodic, one-shot jobs or tasks.

Shouldn't you be doing some real work?

Many of the things I've started as experiments or prototypes have given me useful feedback. OpenFaaS was never meant to be a thing, neither was inlets or actuated and people told me not to build all of them.

You may also like

- Telemetry-free metering for right-sizing GitHub Actions

- Explore and debug GitHub Actions runners via SSH - free for GitHub Sponsors

- actions-usage - Understand your usage of GitHub Actions - free CLI that builds a summary of GitHub Actions usage

GitHub Actions as a time-sharing supercomputer (alexellis.io)

205 points by petercooper 11 months ago | hide | past | favorite | 79 comments

ufo 11 months ago | next [–]

One thing that I still don't understand about github actions is how much computing power gets wasted downloading and reinstalling the same dependencies over and over. A build that takes a couple of seconds in my local machine can easily take several minutes in github actions.

mardifoufs 11 months ago | parent | next [–]

I prefer GA to basically every other build system especially because of the plugins but yeah, and I think caching can be turned on... but it has tons of inconsistencies related to packages and deps. This video really highlighted just how brittle the whole thing is even behind the scenes:

https://youtu.be/9qljpi5jiMQ (13:58 time stamp for the part that is especially relevant here)

ufo 11 months ago | root | parent | next [–]

After that video, I think I finally get Github Actions.

HideousKojima 11 months ago | parent | prev | next [–]

In Azure DevOps pipelines there's a fun issue where if you're pulling a dependency from nuget.org it gets pulled very quickly, but pulling from your own artifact repositories hosted in the very same Azure DevOps instance take 40-80 seconds. I'm not the one paying the extra bills when my builds take 5 to 6 times longer than they should but it's still frustrating.

And Microsoft has effectively zero incentive to fix this because it still technically works and they get to charge my employer for all that extra build agent time. Here's a few examples of people reporting similar issues and getting zero help:

https://developercommunity.visualstudio.com/t/devops-build-p...

https://developercommunity.visualstudio.com/t/nuget-restore-...

https://developercommunity.visualstudio.com/t/slow-performan...

easton 11 months ago | root | parent | next [–]

Microsoft hosted agents aren’t charged by the minute on ADO, IIRC. You pay for a certain number of parallel agents per month.

Unless you’re booting build servers on demand on your infra, you aren’t paying more for more minutes. Just waiting :(

(Unlike GitHub, where they do charge per minute.)

namtab00 11 months ago | root | parent | prev | next [–]

do you know if ADO has any caching mechanisms for pipelines doing builds?

HideousKojima 11 months ago | root | parent | next [–]

The issue in this case isn't caching, even if I'm not actually pulling any packages from our private nuget repo simply having that repo listed in the project's nuget.config file is enough to cause the issue.

0cf8612b2e1e 11 months ago | parent | prev | next [–]

GitHub charges per minute. Some incentive to keep actions on the most under specced machines available.

codetrotter 11 months ago | parent | prev | next [–]

Yeah. We have a similar thing going on with self-hosted GitLab at my work. (And there, there isn’t even any incentive why it should be slow.)

When I make an edit and rebuild on my computer, it takes a few seconds to rebuild because most things are cached.

In CI it takes 20 minutes on the MR branches of our repo. 30 minutes on master. Because a whole bunch of crap is downloaded from scratch and rebuilt from scratch.

Our infrastructure guys did set up something that is able to cache stuff that some of the repos use. But with the limited access I have I have not been able to figure out how to make use of it for the repo I work on. And they don’t have time to look into it for me either.

That’s my life lol

pjot 11 months ago | root | parent | next [–]

You can define a cache key in your .gitlab-ci.yml

https://docs.gitlab.com/ee/ci/yaml/index.html#cache

Already__Taken 11 months ago | root | parent | prev | next [–]

Considering gitlab is a very k8s native app it's way of running a build and moving caches into it, vs moving the container over to where the cache's are is very annoying. I've had issues open for years to update the cache-key with a CSI mount and let k8s move the pods to where the mount is fast. As it is now it pulls everything from S3.

mardifoufs 11 months ago | root | parent | prev | next [–]

Yeah gitlab has pretty good caching support and it is basically a must if you do bigger jobs, especially on stuff like c++ apps. It is pretty brittle sometimes, and in my experience it won't be able to be as fast as actual local dev but it's still good. You can even make it use a remote cache IIRC.

koromak 11 months ago | root | parent | prev | next [–]

On free runners I'd regularly hit the 1 hour limit. Took some time to set up self-hosted through EC2 and really optimize caching, its now down to 10-20 mins, depending on the repo. Most of that is because of the self-hosted runners, the free ones are painfully slow.

Still, on my local machine these jobs would take 5 mins so its not perfect. And as the build gets more complicated and more stages are added, the problem compounds since the initial download is the slowest part.

Raed667 11 months ago | parent | prev | next [–]

Do other CI services do it differently? I worked with Jenkins and Circleci and it was exactly the same.

For github actions caching was fairly easy to add.

ufo 11 months ago | root | parent | next [–]

Last time I tried to add caching to my actions, I spent several hours and still couldn't get it to work properly in the end. I really wonder how many computing hours have been wasted because the default and most likely to work is to not cache anything.

aeyes 11 months ago | root | parent | prev | next [–]

On CircleCI it's also easy to add using save_cache and restore_cache, the docs have examples for different popular frameworks. There are a couple of advanced features and I have been able to cache most build artifacts.

They also cache the Docker images without you having to do anything but it really only works if you use their base images. With custom images the cache rate doesn't look great in my experience but I have no stats to back that up.

sagarm 11 months ago | parent | prev | next [–]

Use a persistent runner and you can take advantage of on disk caches...at the cost of repeatability you get from a clean image on each run.

nsonha 11 months ago | root | parent | next [–]

repeatability must mean different things to us, a cache hit is repeatability to me, a network call can easily fail for so many reasons

paranoidrobot 11 months ago | root | parent | next [–]

It sure can fail but running on a clean agent each time reduces the "Well, it works on my machine" scenarios you can run into.

For example:

The last run didn't clean up, or changed something global.

New runs don't work because some expected state is missing that had been set up on a previous run.

Blowing the agent away each time reduces the chance that something previously run can impact your current run.

This is at the cost of pulling from networks each time. You can reduce those issues by adding network caches for modules/dependencies.

sagarm 11 months ago | root | parent | prev | next [–]

I use persistent runners, but we had to add additional steps (docker prune, mvn -U, docker image ls, etc) to keep the runner healthy and be better able to debug issues.

WorldMaker 11 months ago | parent | prev | next [–]

Given GitHub also owns npm these days, at least some of the constantly reinstalled dependencies are just getting copied from the equivalent of "next door".

eastbound 11 months ago | root | parent | next [–]

And/or they can fix GitHub’s caching of NPM if it saves NPM money.

kevmo314 11 months ago | prev | next [–]

"super" seems a bit generous. Actions runners have always felt slow to me. Fast enough to get the job done for CI/CD but for a batch job running it locally would be faster.

This idea does leech free compute in an API-agnostic way though, if this could be tied together with a GCP free tier, AWS free instance, etc I wonder if you could cobble together enough free resources to run everything you wanted.

ape4 11 months ago | parent | next [–]

I suspect they are slightly slow on purpose - so people don't abuse it for mining or whatever

pinkgolem 11 months ago | root | parent | next [–]

CPU is okay from my observations, but iops is surprisingly slow

aberoham 11 months ago | prev | next [–]

Alex's product vision is fantastic. I hope https://github.com/self-actuated gets noticed by more folks out there who are hitting the limits of GitHub's hosted runners.

k8svet 11 months ago | parent | next [–]

Seems like a solid tech approach too. I'm surprised there isn't more of this around GHA. It really feels like Microsoft calibrated exactly good enough, yet everyone seems to have their own piles of workarounds in published actions, even bigger piles in their infra, and then some more in the workflow yamls themselves, to get everything actually workings, especially when you need to support GHA runners, self-hosted runners, ARC runners, `act` runners, etc. If they can foster an OSS community around easy self-hosted, and then also offer a hosted runners product that is priced well. I'd pay 'em.

ranting-moth 11 months ago | prev | next [–]

GitHub Actions computing time is crazy expensive. Especially the Mac prices.

judge2020 11 months ago | parent | next [–]

Running macOS legally requires real mac servers and a bespoke storage solution: https://www.datacenterdynamics.com/en/analysis/not-just-stac...

A self-hosted macOS runner will be more economical in the long-run, if you have a spot you can hook it up at; or, if you're fine doing things less than legally, you can use https://github.com/sickcodes/Docker-OSX.

255kb 11 months ago | root | parent | next [–]

To add to this, they recently published a video showing how they provision and manage their macOS runners: https://www.youtube.com/watch?v=I2J2MzKjcqY

actionfromafar 11 months ago | root | parent | prev | next [–]

I'm sure legality depends on jurisdiction, too. If you acquired the software legally and you need to keep it running in a VM, I'm sure it's legal at least in some places.

But yeah, just drive-by-downloading MacOS to your Windows box it is probably not quite on the up and up.

chpatrick 11 months ago | root | parent | prev | next [–]

Or get one from Hetzner (there's one in the server finder right now for 64/month).

wilg 11 months ago | parent | prev | next [–]

Highly recommend everyone check out the self-hosted runners. GitHub has made it crazy easy to set up and all your actions are free and can have local cache. And you get all the benefits of the Actions ecosystem. Throw a used Mac mini or old PC or whatever at it.

tomschlick 11 months ago | parent | prev | next [–]

They also price gouge by charging each job you run as per minute rounded up.

So if you have 5 10 second actions that run when a PR is created, that 50 seconds of compute is charged as 5 minutes.

Retr0id 11 months ago | root | parent | next [–]

Is it really price gouging, or just embodying the real warmup costs of spinning up a job?

almostdeadguy 11 months ago | root | parent | prev | next [–]

I truly don't understand why this isn't more widely discussed (I've seen several "GH Actions Gotchas" articles where this isn't mentioned). Many of the community actions also seem to be designed to run as short jobs to paper around missing features (for ex: https://github.com/dorny/paths-filter ), that end up eating up an enormous amount of your minutes budget.

xjia 11 months ago | parent | prev | next [–]

We are working on an alternative that's 100% compatible with GHA but much cheaper and faster. Check out https://dime.run/

tomComb 11 months ago | prev | next [–]

Why do actions use VM’s instead of containers? And yet codespaces use containers.