So you might be like us, using AWS and shifting to using Graviton in order to save costs. Now of course, there are build and runtime use cases. Some of your runtimes might not support Graviton yet, and if this is the case, you can run into an impedance mismatch, where you want to build on one platform (arm) and deploy to another (x86). Docker of course has support for a –platform switch, and in the past this has worked great, until I needed to use amazonlinux:2023 as a base image. Then I received the dreaded “fatal glibc error: CPU does not support x86-64-v2” error. Many hours wasted and lost sleep occurred next, henceforth this blog so other people can benefit from my pain. If you are interested in how we got it to work in Github Actions, jump to the bottom.
Before I get to how I resolved this problem, I want to back up and explain why other simple solutions wouldn’t work for us. You may say things like, just use a build server platform that matches your deployment. For us, this was not possible. We are using Github actions and the system we are developing is used by many teams throughout the organization who have their own runners in their own AWS accounts. Forcing everyone to switch runners is one problem. Another problem is that in Github actions while it is possible to use another runner for a separate workflow, if that workflow calls another workflow the same runner is used. Meaning you can’t switch runners in the middle of a job. I believe Gitlabs has more flexibility here, but not an option for us. In addition even if something like this worked, now teams would have to setup two runners and it just complicates things.
The next issue was that this build produced bundles of python libraries that are used in both Airflow (MWAA 2.8.1) and Spark (EMR Serverless). At this time, MWAA doesn’t support Graviton, but our organization is moving our Spark jobs to Graviton at lightening speed for the cost savings. So you can see here, I need to produce bundles for both x86 and Graviton from one build, there is no way around this.
Ok, so googling around I found qemu, and then quickly found https://github.com/docker/setup-qemu-action. This is supposed to do the docker magic that will solve my problem. But no dice still CPU does not support x86-64-v2. After a few late nights, trial and error, and reading the entire internet, I discovered buildx, which will let you install the specific platform that amazonlinux:2023 was needing: linux/amd64/v2. So these are the necessary components to get installed in your github workflow.
runs:
steps:
# qemu and buildx with proper platforms are needed in order to build bundles on x86 or arm runners
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
platforms: linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/386
Then you can launch docker with the correct platform like this:
docker buildx build --platform linux/amd64/v2 .
Armed with this, pun intended, you can now run builds on Graviton that need to use the amazonlinux:2023 image in order to produce things that are going to run on x86 based machines, or vice versa.
Now of course this emulation is going to slow things down if you are already on the correct architecture, so what you can do is detect the platform and adjust the –platform string according. Happy building!