RFC: ARM64 (aarch64) Architecture Support for Cloud Foundry#1530
RFC: ARM64 (aarch64) Architecture Support for Cloud Foundry#1530vighes wants to merge 9 commits into
Conversation
|
rkoster
left a comment
There was a problem hiding this comment.
Given the impact on the ecosystem I think it would only make sense to also include apple silicon support as in scope. Because this would bring a lot of value to the CF Core developers who have MacBooks and now way of easily validating component changes locally.
|
|
||
| 1. **Stemcell naming convention:** How should ARM64 stemcells be identified? Proposed: `ubuntu-jammy-arm64` alongside existing `ubuntu-jammy` (implicitly x86_64). Open to community input on naming and metadata approach. | ||
|
|
||
| 2. **CI/CD long-term ownership:** ARM64 CI workers will be provided initially to unblock development. Should long-term ownership transfer to the CF Foundation, or remain as a community-contributed resource? |
There was a problem hiding this comment.
Ideally eventually we end up with CF Foundation ownership and get some AWS credits to help with offsetting the increase in CI spend.
Also related, these workers are these concourse workers running on ARM?
There was a problem hiding this comment.
Yes, these would be Concourse workers running on ARM64. For Go cross-compilation tasks, existing x86_64 workers can produce ARM64 binaries via GOARCH=arm64. But for native compilation (runC with seccomp, Ruby gem native extensions, stemcell building) and integration testing, ARM64 workers are required.
There was a problem hiding this comment.
@taylorsilva to my knowledge Concourse is supporting ARM64 and having concourse workers running on ARM64 won't be an issue. Is this correct?
There was a problem hiding this comment.
at
https://github.com/concourse/concourse/releases/tag/v8.2.4
I see https://github.com/concourse/concourse/releases/download/v8.2.4/concourse-8.2.4-linux-arm64.tgz
under releases section so looks like the answer is yes.
There was a problem hiding this comment.
Thanks for confirming Concourse ARM64 support.
Our POC cross-compiled all Go components from Apple Silicon (M4 Pro Mac), However full local integration (runC with seccomp, container creation) still requires Linux ARM64 either via Docker/Lima on Mac or a remote ARM64 instance. The Linux ARM64 work directly benefits Apple Silicon developers for compilation and unit testing, though not for full local container deployment. I'll add a note acknowledging this developer workflow benefit. |
- Support mixed-architecture cells (ARM64 Diego cells + x86_64 control plane) following the Windows cell model, removed as non-goal - Add binary blob handling strategy for components shipping pre-compiled binaries (e.g., JDK in uaa-release) - Add Apple Silicon developer workflow benefit
|
I totally agree with @rkoster on its value for CF developers. It would be much easier to have a local CF where I can generate test CF environments rather than using Docker to create local test environments. It would also help promote Korifi as a development environment, where I am currently trying to get Stratos to run in a Korifi environment. |
|
|
||
| The App Runtime Deployments Working Group owns `cf-deployment`. Work includes: | ||
|
|
||
| - Set up cf-deployment validation against the ARM64 stemcell (similar to existing Jammy/Noble validation) |
There was a problem hiding this comment.
Setting up another validation pipeline for ARM64 is not a problem. But we should clarify the cost increase in advance.
There was a problem hiding this comment.
Agreed, We can provide more details on expected compute hours once we scope the pipeline configurations.
- Add Paketo CNBs to existing ARM64 support list - Add BOSH Docker CPI on Apple Silicon to testing scope (ref bosh-deployment#497) - Add buildpack automation challenges (storage, size, pipelines) to Phase 3
This comes to mind as a new possibility for running locally on Mac: https://github.com/apple/container |
- Add BOSH Director architecture-aware compilation consideration for mixed AMD64/ARM64 environments to Foundational Infrastructure WG scope - Update stemcell naming open question to reference FIPS naming precedent and add architecture metadata field proposal
|
I went through the comments and resolved the ones that appeared to be clarified. To summarize, there are two "major" questions that still need clarification before we can proceed with the RFC:
I'll raise these questions during the TOC meeting to work out how we can resolve them. |
| The App Runtime Interfaces Working Group owns Cloud Controller (CAPI) and UAA. Work includes: | ||
|
|
||
| - Validate Cloud Controller Ruby application runs on ARM64 stemcells (POC validated gem compilation) | ||
| - Validate UAA Java WAR deploys and runs on ARM64 JVM (Eclipse Temurin, SAP Machine, or equivalent ARM64 JDK) |
There was a problem hiding this comment.
UAA is owned by Foundational Infrastructure WG.
There was a problem hiding this comment.
Thanks Stephan, Corrected — moved UAA validation to the Foundational Infrastructure WG section. Updated in latest commit.
| The ARI Working Group owns the classical buildpacks which have higher adoption in CF today. Note: ARM64 support for Cloud Native Buildpacks (CNBs) is already being addressed by the Paketo WG separately (see [ARM64 Paketo Buildpacks](https://www.cloudfoundry.org/blog/arm64-paketo-buildpacks/)). Work for classical buildpacks includes: | ||
|
|
||
| - Produce ARM64 variants of classical buildpack dependencies (JDK, Node.js, Python, Ruby, Go, .NET, PHP, nginx runtimes) | ||
| - Produce an ARM64 variant of the `cflinuxfs4` root filesystem |
There was a problem hiding this comment.
Fixed — updated to cflinuxfs5 in the buildpacks section. Thanks for catching the remaining references.
We discussed the two points from above and current decision is:
|
…ences - UAA is owned by FI WG, not ARI WG (per @stephanme) - Fix remaining cflinuxfs4 references to cflinuxfs5
Thanks for the update. The two-area approach (ARM64 stemcell area + CF stack/buildpacks area) makes sense for clear ownership boundaries. |
I asked about cost estimation in the TOC Slack channel. @vighes it would be great if you could incorporate the second point about the new WG areas into the RFC. |
wayneeseguin
left a comment
There was a problem hiding this comment.
Added comments, suggest adding the Windows ARM64 as a non-goal explicitly at this time (could always be added later).
| - **Author(s):** Sachin Vighe | ||
| - **Status:** Draft | ||
| - **Related RFCs:** [rfc-0026-noble-os](https://github.com/cloudfoundry/community/blob/main/toc/rfc/rfc-0026-noble-os.md) | ||
| - **Affected Component(s):** bosh-linux-stemcell-builder, Diego, Garden-runC, Gorouter, Loggregator, Cloud Controller, UAA, cf-deployment, buildpacks |
There was a problem hiding this comment.
Can't put my finger on it at the moment but I feel there are some more affected components. bosh-dns? others?
There was a problem hiding this comment.
Added bosh-agent, bosh-dns, BPM and NATS to the Affected Components. Also added their validation to the relevant workstreams. The Affected Repositories section at the bottom provides the full breakdown.
|
|
||
| ### Developer Workflow Benefit | ||
|
|
||
| ARM64 Linux support also benefits CF core developers on Apple Silicon MacBooks (M1/M2/M3/M4). All Go-based CF components can be cross-compiled locally using `GOOS=linux GOARCH=arm64`, enabling faster development and unit test iteration without needing remote infrastructure. Additionally, an ARM64 stemcell would enable running BOSH with the Docker CPI natively on Apple Silicon (see [bosh-deployment#497](https://github.com/cloudfoundry/bosh-deployment/issues/497)), giving developers a fully local CF environment without Rosetta emulation issues. Validating this local BOSH-lite workflow on Apple Silicon is included in the testing scope. |
There was a problem hiding this comment.
This would be a fantastic outcome indeed. We would be able to get a 'cf local' up and running again most likely, especially with apple releasing their containers project.
There was a problem hiding this comment.
Agreed! Once ARM64 stemcells exist, BOSH-lite on Docker CPI becomes native on Apple Silicon no more Rosetta issues. Validating this workflow is included in the testing scope.
| - BOSH Acceptance Tests (BATS) pass on ARM64 stemcells | ||
| - CF Acceptance Tests (CATs) pass on ARM64 deployment | ||
| - Applications can be deployed via standard buildpacks on ARM64 cells | ||
| - Performance ≥ 100% of x86_64 baseline for equivalent instance classes |
There was a problem hiding this comment.
Is this always guaranteed? 🤔
There was a problem hiding this comment.
Fair point, not guaranteed across all workload types. Reworded to "comparable" with benchmarking deferred to Phase 2. POC data and public Graviton benchmarks suggest parity or better for Go/Java/Ruby workloads.
|
|
||
| ## Non-Goals | ||
|
|
||
| 1. **Not breaking x86_64 support** — existing deployments continue unchanged |
There was a problem hiding this comment.
A bit confusing, wouldn't it be better to state this as a direct goal vs non-goal, e.g. it is a goal to have no impact on existing?
There was a problem hiding this comment.
I mean maybe it's fine, I find the logic of this section a bit confusing on first read.
There was a problem hiding this comment.
But I still feel that this section should be rewritten to directly state goals of not breaking things with opt-in support etc...
There was a problem hiding this comment.
Restructured, Non-Goals section now focuses purely on what's out of scope. Thanks for flagging.
| ↓ | ||
| BOSH Director (bosh) | ||
| ↓ | ||
| Diego (diego-release) + Garden (garden-runc-release) ← critical path |
There was a problem hiding this comment.
Isn't the whole tree really the 'critical path' ? it either all works or it doesn't.
There was a problem hiding this comment.
that's right, the whole chain is the critical path. Removed the misleading annotation from the diagram and clarified while the entire chain must work, Garden-runC carries the highest implementation risk specifically due to C code and seccomp.
| - Validate the BOSH agent on ARM64 (POC has demonstrated this works) | ||
| - Set up CI pipelines producing ARM64 stemcell builds (ARM64 CI workers will be provided) | ||
| - Validate the BOSH Director (Ruby) operates correctly on ARM64 (POC validated 134 gems) | ||
| - For mixed AMD64/ARM64 environments: address BOSH Director's current single-architecture compilation VM limitation (e.g., support multiple compilation VM types or architecture-aware compilation routing) |
There was a problem hiding this comment.
This is key, for example being able to run Diego Cells in isolation segments on say split intel vs arm would be quite valuable.
| - Performance ≥ 100% of x86_64 baseline for equivalent instance classes | ||
| - Zero impact on existing x86_64 deployments | ||
|
|
||
| ## Non-Goals |
There was a problem hiding this comment.
Perhaps should declare Windows ARM64 support is a non-goal for this effort?
| **Testing (1):** | ||
| - `cf-acceptance-tests` — Full platform validation (Go) | ||
|
|
||
| Remaining repositories (P1/P2) are primarily Go-based and are expected to cross-compile with minimal effort. A detailed repository inventory will be shared alongside this RFC. |
There was a problem hiding this comment.
This might address my initial comments.
- Clarify critical path language in dependency chain diagram - Rewrite Non-Goals section to separate goals from non-goals clearly - Add Windows ARM64 as explicit non-goal - Soften performance success criteria to "comparable" with Phase 2 benchmarking - Add bosh-dns, bosh-agent, BPM, NATS to affected components - Add "Proposed New Areas" section for ARM64 Stemcell and CF Stack areas - Expand workstream details for BOSH DNS and BPM validation
Hi @beyhan Done — see the "Proposed New Areas" section which proposes two new areas with clear boundaries and scope. |
Added as explicit non-goal #1. There's no Windows Server ARM64 offering from major cloud providers today, so this is firmly out of scope. |
| - Run component-level integration tests on ARM64 (Diego inigo tests, Gorouter integration tests) | ||
| - Validate the Loggregator pipeline on ARM64 | ||
| - Validate container networking (Silk CNI, cf-networking) on ARM64 | ||
| - Validate NATS and BPM on ARM64 |
There was a problem hiding this comment.
BPM is not part of the ARP working group
There was a problem hiding this comment.
Good catch — moved BPM to the Foundational Infrastructure WG section. Thanks!
This RFC proposes adding ARM64 architecture support to Cloud Foundry,
enabling deployments on ARM64-based infrastructure (AWS Graviton, Ampere
Altra, etc.) with 20-40% better price/performance.
A proof-of-concept validated all major CF components (Diego, Garden-runC,
Gorouter, Loggregator, Cloud Controller, UAA, BOSH) compile and execute
on ARM64 with zero source code changes.
Looking forward to community feedback on the proposed workstreams and
open questions.
Preview link
cc @beyhan