Build an MCP Server in Go (Part 3): How IRSA and EKS Pod Identity Actually Work

Everything the MCP server could check through Part 2 lives inside the cluster. But "my pod can't reach S3" is very often not a Kubernetes problem at all, it's a workload identity problem, and that identity lives partly on the Kubernetes side (the ServiceAccount) and partly on the AWS side (IAM, and now EKS Pod Identity). Before building a tool that diagnoses it, it's worth actually understanding how it works; most of the confusion people run into comes from treating IRSA and Pod Identity as interchangeable, when they solve the same problem through genuinely different mechanisms.

How pod permissions actually work on EKS

Before touching the diagnostic tool, it's worth stopping and explaining this properly, because "how does a pod get AWS permissions" trips up almost everyone the first time, and the two mechanisms work in genuinely different ways under the hood.

The problem neither mechanism existed to solve

An EC2 instance gets AWS credentials through an instance profile, no secrets anywhere, the instance just asks a local metadata endpoint (IMDS) and gets temporary credentials back, automatically. A pod is not an instance. Dozens of pods can run on the same EC2 node, each one potentially needing different AWS permissions; the checkout-service pod should be able to write to one S3 bucket, the image-resizer pod to a different one, and neither should be able to touch what the other can. If every pod on a node just inherited the node's IAM role, that isolation is gone; any pod could do anything the node itself could do.

Before:

Both IRSA and Pod Identity exist to solve exactly this: giving individual pods their own, scoped-down AWS identity, distinct from the node they happen to be running on and from every other pod sharing that node.

Now:

What a Kubernetes ServiceAccount actually is

Both mechanisms above lean on something Kubernetes already has, so it's worth pinning down before going further: a ServiceAccount is Kubernetes's own concept of "an identity a pod runs as," entirely separate from AWS and predating both IRSA and Pod Identity by years. It has nothing to do with AWS by default, its original job is purely internal to the cluster.

Every pod runs as some ServiceAccount, whether you set one explicitly or not, if you don't specify one, the pod gets the default ServiceAccount for its namespace automatically. A ServiceAccount is a real Kubernetes object, small and unremarkable on its own:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout-service
  namespace: production

Its native purpose is to authenticate to the Kubernetes API server itself, not AWS. Kubernetes automatically mounts a token for the pod's ServiceAccount into the container filesystem, and that token is what lets, say, a controller running inside the cluster call kubectl-equivalent API operations as "the checkout-service identity", with whatever RBAC permissions (Role/RoleBinding or ClusterRole/ClusterRoleBinding) have been granted to that ServiceAccount. This is the exact same object check_endpoints and describe_pod implicitly rely on: the RBAC section back in Part 2 that scoped this MCP server's own ServiceAccount to get/list/watch; that's this same mechanism, just describing what the server itself is allowed to do inside the cluster.

What both IRSA and Pod Identity do is extend that existing identity outward, to AWS, instead of inventing a new one. That's why IRSA works by annotating a ServiceAccount rather than creating some new AWS-specific object, and why Pod Identity's association is keyed on a namespace/ServiceAccount pair rather than anything else.

A ServiceAccount was already sitting there as "the identity of this pod" in Kubernetes' own model, and both mechanisms are answers to the same follow-up question: how do I get that identity recognized by AWS too?

RBAC: what a ServiceAccount is actually allowed to do

A ServiceAccount by itself is just an identity, a name Kubernetes can authenticate. It comes with zero permissions attached. What decides what that identity is actually allowed to do inside the cluster is a separate system: RBAC, Role-Based Access Control.

RBAC works through four object types, and the relationship between them is the whole model:

• Role (or ClusterRole, the cluster-wide version), a set of permissions, expressed as verbs on resources. Something like "can get and list pods in the production namespace." A Role on its own grants nothing to anyone, it's a description of a permission set, sitting unused until something is bound to it.

• RoleBinding (or ClusterRoleBinding), the thing that actually connects a Role to an identity. This is where a ServiceAccount (or a user, or a group) gets named and paired with aRole, and only once that binding exists does the permission become real.

Put together: a ServiceAccount is who, a Role is what's allowed, and a RoleBinding is the because I said so connecting the two. Kubernetes checks all three together on every single API request, "is the ServiceAccount making this call bound, via some RoleBinding, to a Role that permits this specific verb on this specific resource", and denies anything that doesn't match.

This is exactly what Part 2's "RBAC scope the service account, not just the interface" practice was describing, now with the actual mechanism behind it named: this MCP server runs under its own ServiceAccount, which needs a ClusterRole granting only get/list/watch (never create/update/delete/patch), connected by a ClusterRoleBinding. The read-only guarantee built into KubeClient's Go interface in Part 1 and the read-only guarantee enforced by RBAC here are two independent layers checking the same thing from different directions; the Go code structurally can't call a write operation because the interface has none, and even if it somehow did, RBAC would reject the request at the API server before it touched anything. Neither one depends on the other holding; both have to fail at once for something to go wrong.

It's also worth being precise about where RBAC's authority ends, since that boundary is exactly why this post exists: RBAC only governs the Kubernetes API. It has no opinion at all about what a pod is allowed to do once it starts making calls to AWS instead of tokube-apiserver, a ServiceAccount can be tightly scoped under RBAC and still hold an overly broad IAM role through IRSA or Pod Identity, because those are a completely separate permission system, checked by AWS, not by Kubernetes. That gap, RBAC stopping exactly at the cluster's edge, is the reason diagnose_workload_identity has to exist as its own tool rather than just being another RBAC check.

One ServiceAccount, two independent extensions. RBAC (left) is entirely a Kubernetes-side concern, it never leaves the cluster. IRSA and Pod Identity (right) both start from that same ServiceAccount but end up at AWS through completely different trust mechanisms, which is exactly why diagnosing one has nothing in common with diagnosing the other. The two sections below walk through each AWS-side path in detail.

IRSA: a pod proves who it is directly to AWS

IRSA (IAM Roles for Service Accounts) works by turning a Kubernetes ServiceAccount into something AWS can verify cryptographically, using OpenID Connect (OIDC).

OIDC (OpenID Connect) itself is worth grounding first, since everything else in this section builds on it. It's an identity protocol built on top of OAuth 2.0, and it solves a specific problem: how does party B trust a claim about identity made by party A, without B having to call A up and ask "is this real?" every single time? OIDC's answer is the JWT (JSON Web Token), a compact, signed piece of text containing claims like "this is ServiceAccount X, in namespace Y, issued at time Z". The token is signed with a private key only the issuer holds, and anyone who has the issuer's corresponding public key can verify that signature offline, no network call back to the issuer required, no shared secret sitting in a config file waiting to leak. That's the whole trick: proving something is genuine by checking a signature, not by trusting whoever happens to be presenting it. It's the same reason OIDC has become the backbone of "log in with X" buttons everywhere, the token itself carries proof of who issued it.

Before the rest of the flow makes sense, it's also worth being clear on what an OIDC issuer actually is, and, just as importantly, where it actually runs, since "the cluster's own issuer" is easy to misread as something running alongside your workloads. In OIDC terms, the issuer is the party that signs and stands behind tokens; Google, in the "Sign in with Google" example, or in this case, EKS itself.

Every EKS cluster gets its own OIDC issuer: a public HTTPS endpoint, unique to that cluster, that serves the OIDC discovery document and the JWKS (the public keys needed to verify token signatures). That endpoint is part of the EKS control plane, hosted and managed by AWS, the same way kube-apiserver itself is, not a pod running on your nodes. The actual signing of ServiceAccount tokens happens inside kube-apiserver using a private key the control plane holds; the issuer endpoint is just where the corresponding public key gets published, so AWS STS can fetch it and verify signatures later without ever calling back into your cluster.

Registering that issuer with AWS IAM (step 1 below) is what lets IAM trust tokens signed by your cluster specifically; an OIDC provider registered in one AWS account only vouches for tokens from the one EKS cluster it points at, which is exactly why the trust policy's Federated principal has to reference this cluster's specific issuer URL, not some generic "any EKS cluster" identity.

Here's the flow, step by step:

1. When you set up IRSA, your EKS cluster's OIDC issuer gets registered with AWS IAM as a trusted identity provider. This is a one-time step per cluster.

2. You annotate a ServiceAccount with eks.amazonaws.com/role-arn: arn:aws:iam::...:role/my-role. An annotation here just means ordinary Kubernetes metadata, a key-value pair stored under metadata.annotations on the object, the same mechanism used for all sorts of unrelated things across the ecosystem. Unlike labels, which Kubernetes uses to select and group objects, annotations aren't used for selection; they're just data a controller or webhook can read to change its behavior. In this case:

yaml

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: checkout-service
     namespace: production
     annotations:
       eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/checkout-s3-reader

That specific eks.amazonaws.com/role-arn key is what the mutating webhook in the next step is watching for; its presence is the trigger, and its value is the role the webhook tells the pod to assume.

3. When a pod using that ServiceAccount starts, a mutating webhook (installed as part of the EKS setup) injects a JWT, a signed token proving "this pod belongs to ServiceAccount X in namespace Y, in this specific cluster", into the pod as a file, along with environment variables (AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE) telling the AWS SDK where to find it.

A mutating webhook is part of Kubernetes' admission control pipeline, the sequence of checks every API request passes through between "the request is authenticated" and "the object actually gets written to etcd". Here's the full sequence for something like kubectl apply -f pod.yaml:

1. Authentication: who is making this request?

2. Authorization: RBAC check: is this identity allowed to do this?

3. Mutating admission webhooks: a chance for registered external services to modify the object before it's persisted.

4. Object schema validation: is the (possibly now-modified) object well-formed?

5. Validating admission webhooks: a chance for external services to approve or reject the final object, no more modifications allowed.

6. Persisted to etcd: the object is now real.

A mutating webhook is registered via a MutatingWebhookConfiguration pointing at an HTTPS endpoint, usually a small service running as its own Deployment in the cluster. When a matching request comes in (here, Pod CREATE), kube-apiserver pauses, sends the object to that endpoint, and applies whatever JSON patch comes back before continuing.

For IRSA, this is the Amazon EKS Pod Identity Webhook (an IRSA-era name that predates EKS Pod Identity the newer mechanism, which is a frequent source of confusion). It looks up the pod's spec.serviceAccountName, checks that ServiceAccount for the eks.amazonaws.com/role-arn annotation from step 2, and if present, patches in the projected token volume and the two environment variables above. None of this appears anywhere in the pod spec as written, it's injected transparently at admission time, which is exactly why IRSA "just works" once the annotation is set, with nothing else to configure on the pod itself.

4. When your Go code makes its first AWS API call, the SDK automatically reads that token and calls sts:AssumeRoleWithWebIdentity, presenting the JWT directly to AWS Security Token Service.

5. STS verifies the JWT's signature against the cluster's registered OIDC provider, checks the IAM role's trust policy for a matching condition (the specific sub claim, system:serviceaccount:<namespace>:<serviceaccount>), and if everything lines up, hands back temporary credentials.

The important thing to notice: the pod talks to AWS STS directly. No extra agent runs in the cluster to make this work, the SDK, the token file, and STS are the whole path. That also means the trust policy is where all the real security logic lives: it's the only thing standing between "I have a valid JWT" and "I get to assume this specific role."

Same five steps as above, made concrete: a pod running as the checkout-service ServiceAccount wants to read an object from S3. Its AWS SDK component presents the JWT sitting in its JWT Token file to AWS STS (1), STS checks that token and the requested role against AWS IAM (3), IAM confirms the trust relationship checks out against the registered OIDC identity provider and the role's attached policies (3–4), STS hands back temporary credentials (5), and only then does the pod call S3 directly (6), S3 itself never sees the JWT at all, only the temporary credentials STS issued.

Pod Identity: an in-cluster agent brokers the exchange

EKS Pod Identity, AWS's newer mechanism, moves that broker inside the cluster instead. Different flow:

1. Instead of an OIDC trust relationship, you create an association, a record, stored in AWS via the EKS API, saying "ServiceAccount X in namespace Y on cluster Z maps to IAM role R." No OIDC provider, no cluster-specific string embedded in an IAM trust policy.

2. A DaemonSet called the EKS Pod Identity Agent runs on every node, alongside your workloads.

3. The same kind of mutating webhook sets environment variables on the pod, but this time pointing the AWS SDK at the local agent (AWS_CONTAINER_CREDENTIALS_FULL_URI, a standard mechanism the SDK already supports for container environments, the same one ECS tasks use), not at AWS directly.

4. When your Go code makes its first AWS call, the SDK requests credentials from that local endpoint instead of STS. The Pod Identity Agent receives that request, looks up the association for this pod's namespace/ServiceAccount, and calls a dedicated API (eks-auth:AssumeRoleForPodIdentity) on the pod's behalf to get temporary credentials, then hands them back to the SDK.

It's worth being precise about where STS actually sits in this chain, since it's easy to assume Pod Identity skips STS entirely, it doesn't, it just moves further away from the pod. The pod's SDK never talks to STS or IAM directly; it only ever talks to the local agent.

Credentials flow back up the same four steps in reverse: STS to the EKS Auth API, to the agent, to the pod's SDK, so the diagram only needs to show one direction to be complete. The practical difference from IRSA's diagram earlier: there, the pod called STS itself, in one hop. Here, there are two extra hops and a different AWS API in the middle, which is exactly why Pod Identity failures can look different from IRSA failures even though both ultimately end at STS.

The trust policy here looks completely different too: instead of trusting an OIDC federated principal, it trusts the AWS service pods.eks.amazonaws.com directly. The association, which pod maps to which role, lives entirely in AWS, not encoded into the trust policy's conditions at all.

The EKS Auth API

The agent calls eks-auth:AssumeRoleForPodIdentity, and it's worth being precise about what that actually is, too: Amazon EKS Auth is its own standalone AWS service, with its own service ID (eks-auth), its own regional endpoint, and its own SDK client in every language, separate from both the general eks API and from iam/sts. It's part of the broader EKS product family, but it isn't the Kubernetes control plane (kube-apiserver, etcd, and friends, Kubernetes-specific infrastructure) and it isn't IAM's API either (IAM is global, not regional, with its own iam:* action namespace). Architecturally, EKS Auth sits as a peer to IAM and STS, not a part of either, a purpose-built bridge service that receives the pod's own projected token (audience pods.eks.amazonaws.com, different from IRSA's sts.amazonaws.com token), checks whether a Pod Identity association exists for this namespace/ServiceAccount, and whether the target role's trust policy allows it. Only then, internally, an implementation detail your code and the agent never see directly, does it call STS to actually mint the credentials, and hand them back down the chain.

One practical consequence of EKS Auth being its own regional service: a private EKS cluster with no outbound internet access needs its own VPC endpoint specifically for eks-auth, distinct from the endpoints for eks, sts, and everything else. It's newer and less well-known than the others, which makes it an easy one to miss when locking down a private cluster's networking, Pod Identity will simply fail to issue credentials with no obvious explanation until that endpoint exists.

Why this distinction matters for diagnosis

This is exactly why diagnose_workload_identity has to check two completely different places rather than one: an IRSA misconfiguration is almost always a trust policy problem (wrong OIDC provider, wrong sub condition, wrong namespace/ServiceAccount string), diagnosable by reading the role. A Pod Identity misconfiguration can be an association problem, no association exists at all for this ServiceAccount, something no amount of staring at an IAM role will ever reveal, since the role's trust policy for Pod Identity looks identical regardless of which ServiceAccounts happen to be associated with it. You have to ask AWS's EKS API directly, which is exactly what FindPodIdentityRole does.

And since both can be configured on the same ServiceAccount at once, very common mid-migration, and Pod Identity silently wins when both are present, a pod can have a perfectly correct-looking IRSA setup that's doing nothing at all, because an association nobody remembers creating is quietly overriding it. That's not a rare edge case to handle defensively; it's routine enough right now, with teams actively migrating from IRSA to Pod Identity, that it deserves to be the first thing the tool checks.

Summary: one rule that matters most

A pod gets AWS credentials one of two ways on EKS: IRSA (OIDC-based, checked via the ServiceAccount annotation and the IAM role's trust policy) or EKS Pod Identity (association-based, checked via the EKS API). Both can be configured at once during a migration, and when they are, Pod Identity wins, the mutating webhook prefers it whenever both are present. A role that looks unused from the Kubernetes side might still be the one actually granting access, or the one silently being ignored, and that precedence rule is the first thing any diagnostic tooling built on top of this needs to check.

Three things worth carrying forward:

A ServiceAccount is Kubernetes's own native identity concept, and RBAC governs it only inside the cluster, neither has any authority once a pod starts talking to AWS instead of kube-apiserver. IRSA and Pod Identity both extend that same ServiceAccount identity outward to AWS, but through completely different mechanisms: IRSA is a direct, OIDC-verified call from the pod to STS; Pod Identity routes through an in-cluster agent and an AWS-side association that has no visible trace on the Kubernetes side at all. Diagnosing a failure means checking two unrelated surfaces, an IAM role's trust policy for IRSA, and the EKS API's association records for Pod Identity, because nothing about looking at one tells you anything about the other.

Part 4 builds the actual diagnostic tool on top of everything explained here, a new AWS-facing Go client, a diagnose_workload_identity MCP tool that checks both mechanisms and applies the precedence rule above, and an honest look at what it means for this server to start needing AWS IAM permissions, not just Kubernetes RBAC.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

• LinkedIn

• GitHub

• Twitter / X

Building from Asunción, Paraguay 🇵🇾