IAM & the Security Boundary

1.1Proving who sent the call

Every call to an AWS API, whether it comes from the console, the command line or your code, is an HTTPS request. Before S3 touches app.log it has to answer two questions: who sent this, and are they allowed to do it? The first is authentication.

A program that talks to AWS holds an access key, which comes in two halves: an access key ID, which is like a username, and a secret access key, which is like a password and must never be shown to anyone. AWS's SDKs, the libraries your code calls, use the pair to sign each request with Signature Version 4, or SigV4 for short. The SDK computes a signature over the request's method, path, headers and body, using the secret half. (The kind of keyed fingerprint it computes is called an HMAC: only someone who holds the key can produce the right one.) AWS keeps its own copy of the key, recomputes the signature on its side, and compares.

?Why sign instead of sending a password?

Because the secret never crosses the network. What's signed covers the request and a timestamp, so a captured request can't be altered, and AWS rejects it once the timestamp is too old. Only the access key ID travels in the clear; the secret stays with you.

1.2The four facts AWS judges

The second question is authorization, and it's what the rest of this chapter is about. AWS gathers everything it knows about the call into a request context and checks it against every policy that applies. To describe that context we need two more words. Everything you rent from AWS lives in an account, which has a twelve-digit ID such as 111122223333 and owns its own resources, users and rules. And AWS gives every resource, user and role a long unique name called an ARN (Amazon Resource Name). An ARN reads arn:aws:<service>:<region>:<account>:<name>, with fields left empty when they don't apply. Bucket names are global, so our log file's ARN has empty region and account fields: arn:aws:s3:::logs/app/app.log. The context has four parts, and here they are for our call:

Each named fact in the last row is a condition key, and rules can test condition keys as well as the first three parts. The principal in that table doesn't look like a person's name, and there's a reason.

1.3Who counts as a principal

A principal is whatever signed the request. Our app's principal, assumed-role/app/i-0abc, names no human because the instance stores no password or access key. It holds a role, an identity that owns no credentials of its own. Something has to assume the role, and AWS then hands that something temporary keys, called a role session. Section 7 shows how the handover works and section 8 shows how it gets attacked. For now, here are all the kinds of principal there are, each with its own ARN shape (IAM identifiers):

?Why does the role session have a different ARN from the role?

Because a role is a template and nobody acts as a template. They assume it and get a session, and the session's ARN carries a session name that the assumer chooses (ours is the instance ID, i-0abc). Two engineers assuming the same role show up in CloudTrail, AWS's log of every API call, as two different principals.

That session name matters later. Section 5.3 shows that a policy granting the role ARN and a policy granting the session ARN are evaluated differently.

So S3 now knows who is calling, what they want and which object it touches. What it needs next is a set of rules to compare those four facts against. To see what a rule engine does with them, we can ask a small one.

2.1A toy engine and three requests

An office building has a door policy. Nobody gets through a door unless a rule says they may. Staff badges are written into some rules ("engineering may enter the lab"), and there's one more kind of rule, a flat ban ("nobody enters the server room without an escort"). If a ban applies, it wins over every permission, however many permissions you hold.

Cloud permissions work like that. Every request is checked against rules, the starting answer is no, and an explicit deny beats any allow. You can watch that logic run without an AWS account. AWS's own evaluator isn't public, but AWS also open-sourced a policy language called Cedar, which follows the same two rules. A Cedar rule is one line: a permit says who may do what to which thing, and a forbid is a ban. Our toy has one of each, and we'll ask Cedar about three requests: the app role reads app.log, the app role reads a file called db-password.txt that's labelled secret, and an intern role reads app.log.

You need Python and cedarpy (pip install cedarpy), a binding for Cedar. The script describes the roles and files as entities, gives each file a classification, hands Cedar the two rules, and defines ask(who, what), which builds one request and prints the decision.

import cedarpy
 
policies = """
permit(principal == Role::"app", action == Action::"read", resource);
forbid(principal, action, resource) when { resource.classification == "secret" };
"""
entities = [
    {"uid": {"type": "Role", "id": "app"}, "attrs": {}, "parents": []},
    {"uid": {"type": "Role", "id": "intern"}, "attrs": {}, "parents": []},
    {"uid": {"type": "Object", "id": "app.log"}, "attrs": {"classification": "public"}, "parents": []},
    {"uid": {"type": "Object", "id": "db-password.txt"}, "attrs": {"classification": "secret"}, "parents": []},
]
 
def ask(who, what):
    req = {"principal": f'Role::"{who}"', "action": 'Action::"read"', "resource": f'Object::"{what}"', "context": {}}
    r = cedarpy.is_authorized(req, policies, entities)
    print(f"{who:<7} read {what:<15} -> {str(r.decision).split('.')[-1]}")
 
ask("app",    "app.log")           # matched by the permit
ask("app",    "db-password.txt")   # permit matches, but a forbid matches too
ask("intern", "app.log")           # nothing matches

app     read app.log         -> Allow
app     read db-password.txt -> Deny
intern  read app.log         -> Deny

Read the three lines against the two rules. The app role can read app.log because the permit matches it and no forbid does. It can't read db-password.txt: the same permit matches, but the forbid matches too, and forbid wins. The intern role is denied for app.log even though nothing forbids it, because no rule mentions intern at all.

2.2The two rules to remember

Two things from those three lines carry through the whole chapter. First, silence means no: a request that no rule allows is denied. Second, an explicit deny beats any number of allows. AWS IAM follows the same two rules, and most of the surprising AccessDenied errors, and most of the over-broad permissions, come from forgetting one of them.

The first rule already explains the engineer's laptop from the opening. The call from the laptop is signed with the engineer's own keys, so its principal is the engineer, and if no rule mentions the engineer, the laptop is in the position of intern: nothing allows the call, so the answer is no.

A Cedar rule fits on one line. IAM's rules are longer JSON documents, and there are several kinds of them, so we'll start with what goes inside one.

3.1A statement

A policy is a list of statements. Each statement says: for these actions, on these resources, for these principals, under these conditions, the effect is Allow or Deny. Here is one that lets our app read its logs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadAppLogs",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::logs", "arn:aws:s3:::logs/app/*"],
      "Condition": {
        "StringEquals": { "aws:PrincipalOrgID": "o-exampleorgid" },
        "Bool": { "aws:SecureTransport": "true" }
      }
    }
  ]
}

s3:GetObject reads one object, and s3:ListBucket lists what a bucket holds. They need different resources, because listing acts on the bucket itself (arn:aws:s3:::logs) while reading acts on the objects inside it (logs/app/*, which covers our app/app.log). The two conditions say the caller's account must belong to our organization (section 4 explains organizations) and the request must use HTTPS.

Notice that the statement never says who it applies to. That's because a policy is attached to something, and this one will be attached to the app's role, so it applies to whoever is acting as that role. A policy attached to an identity like this is an identity policy. A policy can also be attached to the thing being accessed, such as the logs bucket, and then it's a resource policy. A resource policy has no owner to apply to, so each of its statements needs a Principal element naming who it covers. Section 4 lists all the places a policy can be attached. Here are the elements a statement can have:

Version isn't a date you pick. 2012-10-17 is the current policy language version, and you need it for policy variables, placeholders like ${aws:username} that AWS fills in from the request, to work.

3.2How conditions combine

Conditions have their own small logic, documented under condition evaluation:

?Why does a missing key matter so much?

Because keys are only present when they make sense. aws:SourceIp isn't in the context when the request comes through a VPC endpoint (a private path from your VPC, the private network your instances run in, to an AWS service), and aws:MultiFactorAuthPresent is only there for temporary credentials, never for long-lived access keys (global condition keys). A Deny that tests such a key with an ordinary operator will quietly not fire when the key is absent. One that uses a negated operator will fire. That's the behaviour you want from "deny unless this key has the right value", because requests with no key at all get caught too.

Key names are also case-insensitive. The troubleshooting guide warns that a condition on foo matches Foo and FOO, so a request carrying two tags whose names differ only by case can be denied unexpectedly.

3.3Three sharp edges

A few elements look harmless and aren't.

• NotAction with Allow. It matches everything except the listed actions. AWS's own example, "NotAction": "iam:*" on "Resource": "*", allows every action in every other service. AWS's warning is that it "could result in granting users more permissions than you intended."
• Incomplete ARNs. In an identity policy, AWS fills missing ARN fields with wildcards. arn:aws:sqs becomes arn:aws:sqs:*:*:*: every queue, in every region, in every account.
• Friendly names get reused. If a policy refers to user/John by name, in a Resource element or a condition, and John leaves, the next user created as John inherits that access. Every user also has a unique ID (AIDA...) that is never reused, and resource policies can pin that ID instead.

That is everything a single statement can say. A statement does nothing until it's part of a policy attached to something, and AWS offers seven places to attach one.

4.1Seven places a policy can live

The grammar is the same everywhere. What changes is who attaches the policy and what it's able to do. Some vocabulary first, because several of these places exist for companies with many accounts. The account from section 1.2 is the basic container: it owns resources, and it's the hardest boundary IAM has. A company usually runs many accounts, and AWS Organizations groups them into a tree whose branches are organizational units (OUs). A policy can be attached to a single account or to a whole OU, and then applies to every account beneath it.

4.2Two policies grant, five only limit

Only two of the seven can grant access. The rest can only take it away. So for our call to succeed, something from the first two rows has to allow it, and every limiter that applies must leave it alone.

?Why have so many limiters?

Because they're owned by different people. Whoever writes a role's identity policy isn't the security team writing the SCP, and neither is the team that owns the bucket. Each limiter lets one owner cap everyone else without editing their policies.

One call can now meet up to seven policies, written by different people. In what order does AWS look at them, and how do their answers combine?

5.1Two rules that override everything

Everything rests on the two rules from section 2:

1. Default deny. Every request starts denied. AWS's only listed exception is the account's root user.
2. Explicit deny wins. One matching Deny statement, in any policy of any type, ends the evaluation with Deny. No number of Allows can outvote it.

AWS names the two outcomes differently. An implicit deny means nothing allowed the request. An explicit deny means something forbade it. The distinction shows up in error messages, and it tells you which kind of fix you need: add an Allow somewhere, or find and remove a Deny.

5.2One request through the stack

Here is the order for a request within one account. The scene below follows our app's s3:GetObject call through four checks, then repeats it after one change to the role.

Three details sit behind those captions. In step 2, if the resource's account has RCPs, one of them must allow the action. When RCPs are turned on, a policy called RCPFullAWSAccess is attached everywhere and can't be detached, so this step only bites if you've added narrower ones. If the caller's account has SCPs, every level from the root of the organization down to the account must allow the action, and a missing Allow at any level is an implicit Deny.

In step 3, a resource policy that names the caller's IAM user or session ARN directly is enough on its own, and the answer is Allow. Otherwise some identity-based policy must allow the action, or a resource policy must name the caller's role. If neither does, it's an implicit Deny. Step 4 applies the permissions boundary and the session policy, if present, and the request must pass both.

5.3Union for grants, intersection for caps

Behind those steps is a simple pattern. Grants add together, and caps overlap:

?So can a bucket policy bypass a permissions boundary?

Sometimes, and this is the subtle part. It depends on which ARN the bucket policy names.

An explicit deny still beats all of them. These exceptions only concern implicit denies.

5.4A model you can run

AWS's own evaluator isn't public, so everything above comes from AWS's documentation. The documented order fits in about thirty lines of Python, which makes it possible to check your own reading of it. It is a model of the published steps and contains none of AWS's code. Wildcards are matched with fnmatch, Python's matcher for shell-style patterns such as logs/*, and conditions are left out. Each statement is a small dictionary, and a resource-policy statement also records how it names the caller (user, session or role), because section 5.3 showed that this decides the answer. The six cases are all s3:GetObject on our log file, by a role session:

from fnmatch import fnmatch
 
def matches(stmts, req, effect):
    """True if some statement with this effect covers the request's action and resource."""
    return any(s["effect"] == effect
               and fnmatch(req["action"], s["action"])
               and fnmatch(req["resource"], s["resource"])
               for s in stmts)
 
def decide(req, pol):
    allp = [s for k in ("rcp", "scp", "resource", "identity", "boundary", "session")
            for s in pol.get(k, [])]
    if matches(allp, req, "Deny"):                       return "Deny (explicit)"
    for k in ("rcp", "scp"):
        if k in pol and not matches(pol[k], req, "Allow"): return f"Deny (no {k} allows)"
    # a resource policy naming the user or the SESSION ARN is enough on its own
    named = [s for s in pol.get("resource", []) if s["names"] in ("user", "session")]
    if matches(named, req, "Allow"):
        return "Allow (resource policy)"
    rbp = matches(pol.get("resource", []), req, "Allow")  # named the role ARN
    if not rbp and not matches(pol.get("identity", []), req, "Allow"):
        return "Deny (no identity-based policy allows)"
    if "boundary" in pol and not matches(pol["boundary"], req, "Allow"):
        return "Deny (no permissions boundary allows)"
    if "session" in pol and not matches(pol["session"], req, "Allow"):
        return "Deny (no session policy allows)"
    return "Allow"
 
def stmt(effect, action, resource="*", names=None):
    return {"effect": effect, "action": action, "resource": resource, "names": names}
 
REQ = {"action": "s3:GetObject", "resource": "arn:aws:s3:::logs/app/app.log"}
LOGS = "arn:aws:s3:::logs/*"
identity = [stmt("Allow", "s3:GetObject", LOGS)]
boundary_sqs = [stmt("Allow", "sqs:*")]
 
cases = [
    ("1 identity allows",                    {"identity": identity}),
    ("2 + SCP denies s3:*",                  {"identity": identity, "scp": [stmt("Allow", "*"), stmt("Deny", "s3:*")]}),
    ("3 bucket policy names role, no id",    {"resource": [stmt("Allow", "s3:GetObject", LOGS, "role")]}),
    ("4 ...plus a boundary without s3",      {"resource": [stmt("Allow", "s3:GetObject", LOGS, "role")], "boundary": boundary_sqs}),
    ("5 bucket policy names SESSION, same",  {"resource": [stmt("Allow", "s3:GetObject", LOGS, "session")], "boundary": boundary_sqs}),
    ("6 identity allows, session policy",    {"identity": identity, "session": [stmt("Allow", "s3:ListBucket")]}),
]
for label, pol in cases:
    print(f"{label:<38} -> {decide(REQ, pol)}")

1 identity allows                      -> Allow
2 + SCP denies s3:*                    -> Deny (explicit)
3 bucket policy names role, no id      -> Allow
4 ...plus a boundary without s3        -> Deny (no permissions boundary allows)
5 bucket policy names SESSION, same    -> Allow (resource policy)
6 identity allows, session policy      -> Deny (no session policy allows)

Cases 4 and 5 are the ones to study. They have the same bucket policy and the same boundary, which lacks S3, and they differ only in the ARN the bucket policy names. Naming the role gets a Deny, because the role's boundary caps it. Naming the session gets an Allow, because permissions granted directly to a session aren't limited by the boundary. That's the table in section 5.3, executed. Case 2 shows the other rule: the SCP has an Allow for everything and a Deny for S3, and the Deny wins.

5.5The same rule in an open-source engine

Cedar is the engine whose source we can read: AWS's open-source policy language, used by Amazon Verified Permissions (AWS's managed authorization service for your own applications) and described in an OOPSLA 2024 paper (OOPSLA is a programming-languages conference). It has permit and forbid instead of Allow and Deny, and it lands on the same two rules:

impl From<PartialResponse> for Response {
    fn from(p: PartialResponse) -> Self {
        let decision = if !p.satisfied_permits.is_empty() && p.satisfied_forbids.is_empty() {
            Decision::Allow
        } else {
            Decision::Deny
        };
        /* ... collect the determining policy IDs and any errors ... */
    }
}

Allow needs at least one satisfied permit and zero satisfied forbids. Everything else, including "no policy matched at all", is Deny. That's default deny and deny-wins in six lines.

?What does evaluation cost as policies pile up?

Look at the loop that fills those lists, in authorizer.rs. It's for p in pset.policies(): every policy in the set is evaluated. Timing it through the cedarpy bindings with N permits, each naming a different team, plus one forbid, 2,000 decisions per run and five runs each, gives:

Those figures come from the engine's own authz_duration_micros metric, so they leave out parsing, and the absolute values change from machine to machine. The shape doesn't: cost is linear in policy count, roughly half a microsecond each. That's probably why the authorizer's doc comment talks about deciding "with respect to the given Slice": callers are expected to pass only the policies that could apply to this request, not the whole store. To feel the difference, suppose a service makes 1,000 decisions a second:

Seven seconds of work every second needs more than seven CPU cores just to answer yes or no, so the filtering step earns its keep.

Everything so far has put the app and the bucket in one account. What if the log bucket belongs to a different one?

6.1Two evaluations, both must allow

For a cross-account request, AWS runs the evaluation twice, per the cross-account logic: once in the caller's account (the trusted account) and once in the resource's account (the trusting account). "The request is allowed only if both evaluations return a decision of Allow."

Usually you cross by assuming a role in the other account, which means asking STS, AWS's Security Token Service, for temporary credentials. Section 7 covers STS in full. Step through our app's journey:

?Why not just put account A in the bucket policy?

You can. A bucket policy naming A's role works without any role in B, as long as A's identity policy also allows the S3 call. What changes is where control lives. With a role in B, account B owns and audits exactly what A can do. With a bucket policy, it's split across two accounts' policies, and both must agree.

6.2Which organization policies apply where

Across accounts we also have to ask which organization-level cap applies, the caller's or the resource's. SCPs follow the principal. RCPs follow the resource. In our example, account A's SCPs limit what our app may do, and account B's RCPs limit what anyone may do to the logs bucket. That's the whole reason RCPs were added: an SCP can't touch a caller from outside your organization.

Organizations' docs give the example directly, with the letters the other way round from ours: a bucket in account A that grants access to account B outside the org. A's SCP "doesn't apply to those outside users". A's RCP "applies to the S3 bucket in Account A even when accessed by users from Account B."

6.3Two resources that insist on being named

For most resources in the same account, an Allow in either the identity policy or the resource policy is enough. Two exceptions, called out in the evaluation docs:

KMS, the Key Management Service, holds the encryption keys that protect data in S3 and elsewhere. That KMS rule is probably why "I'm an admin and I still can't decrypt" is such a common ticket. A key's policy is the root of trust for the key, and your identity policy comes second.

In the sequence above, STS handed back credentials in a single step. What exactly does it return, and how long do the credentials last?

7.1What AssumeRole returns

AssumeRole returns three strings and a deadline:

How long that lasts:

?Why does role chaining cap at an hour?

Because otherwise a session could renew itself forever. A role session that assumes another role, which assumes the first back, would never need the original credentials again. A one-hour cap means the chain has to return to a real identity regularly.

7.2Narrowing a session

AssumeRole takes optional parameters that shape the session. A session policy narrows what the session may do. It can be written inline as JSON, or given as the ARNs of managed policies, standalone policies stored in IAM that many roles can share. Session tags are name-and-value labels attached to the session so that other policies can match on them, for example a bucket policy that only lets in sessions tagged team=logs:

Session policies and tags ride inside the session token. Push too many in and STS rejects the call with PackedPolicyTooLarge, even when each individual limit is met. Watch SessionTokenUtilization in the response.

7.3Pods: IRSA and EKS Pod Identity

A pod, one running group of containers in Kubernetes, needs AWS credentials that belong to it, not to the node (the machine it runs on). EKS, AWS's managed Kubernetes, has two mechanisms, and both end in a role session.

Both start from the pod's Kubernetes identity, its service account. Kubernetes can write a signed token into the pod's filesystem that says "this is service account log-reader in namespace prod". The token is a JWT, a small JSON document with a signature anyone can check. In the first mechanism, IRSA, the cluster's token issuer is registered with IAM as an OIDC provider (OIDC is the standard login protocol these tokens come from), so STS will accept the token as proof of identity and hand back a role session. In the second, EKS Pod Identity, an agent on the node trades the token for credentials on the pod's behalf. The kubelet, Kubernetes's agent on each node, refreshes the token before its TTL (time to live) runs out:

Sources: EKS's best practices guide and how Pod Identity works.

?Why does the IRSA trust policy need that sub condition?

Because without it, the trust policy trusts the whole cluster's OIDC issuer. Any service account in any namespace could present its token and assume the role. The token's sub (subject) field names the service account it was issued to, so a condition on :sub pins the role to one namespace and one service account.

7.4The credential chain decides which identity you are

An SDK that needs credentials has several places to look, and it takes the first that works. This list is the credential provider chain, and the exact order differs a little between SDKs: environment variables and config files, web identity (the IRSA token file), container credentials (the Pod Identity address above, and its equivalent on AWS's other container services), and finally the EC2 instance metadata service.

The last stop in that chain, the instance metadata service, is how our own app's instance gets its role. It's also the stop that attackers care about most.

8.1How an instance gets its credentials

This is where the opening's puzzle gets its answer: our app has no keys in its code because the instance hands them over on request. You attach a role to an instance through an instance profile, the wrapper EC2 uses to hand a role to an instance. EC2 assumes the role for you, and code on the instance, such as our app's SDK, fetches the session from http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>. The server at that address is the Instance Metadata Service (IMDS). That address is a special one that only code running on the instance can reach. The SDK fetches fresh keys from it before the old ones expire, so every key the app ever holds is a temporary one. The design is convenient, since no key ever needs to be stored on disk, and it's probably the most attacked piece of AWS IAM.

?Why is an HTTP endpoint for credentials dangerous?

Because plenty of software makes HTTP requests on someone else's behalf: proxies, webhooks, URL previewers, PDF renderers, firewalls. If an attacker can steer one of them at 169.254.169.254, the instance fetches its own credentials and hands them over. That's server-side request forgery, SSRF. In the original version of the service, IMDSv1, a plain GET was all it took.

To see it on our app, suppose it has a feature that fetches any URL a user types, to show a preview:

8.2Capital One, 2019

The bank Capital One lost data to exactly this kind of attack in 2019, and it's the best-documented case. There, the component an attacker steered was a web application firewall (WAF), a filter that inspects web traffic before it reaches an app. It ran on one of Capital One's EC2 instances with a role attached. The case's criminal complaint (filed July 29, 2019) and the later superseding indictment describe the attack in three commands. Both filings call the provider "the Cloud Computing Company" and redact the role's prefix. Two more names appear below. TOR is an anonymising network that hides where a request comes from. And after the breach, Senator Ron Wyden asked AWS how it happened, and AWS's chief information security officer (CISO) answered in writing.

?Which part was an IAM failure?

Arguably two of the three. Those credentials left the instance and still worked from TOR, because nothing bound them to where they were issued. And a firewall's role could list and read hundreds of buckets it had no business with. AWS's CISO described the permissions to Wyden as "likely broader than intended" (The Register).

8.3IMDSv2: four small defenses

AWS shipped IMDSv2 in November 2019 (Colm MacCárthaigh's post). You PUT to get a session token first, then send it as a header:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

Compare that with the preview attack in section 8.1, where one plain GET was enough. Each detail of the new exchange blocks a different way of getting the request through:

Token lifetimes run from one second to six hours (21,600 s). Since mid-2024, AWS has said newly released instance types use IMDSv2 only, but older types still accept v1 unless you set HttpTokens=required.

8.4Binding credentials to the machine

IMDSv2 makes theft harder. Two newer controls make stolen credentials less useful:

Even with all of these, stolen keys carry every permission of the role. In the Capital One case the firewall's role could list and read hundreds of buckets, and the last control above does nothing about that. How do you decide what a role needs?

9.1Tools that find what's unused

In practice you start broad in a sandbox, observe what the code calls, then cut. For our app that ends at a role that can read app/* in one bucket and nothing else. AWS gives you three sources of evidence. In the first, AWS separates management actions, which create and configure things (CreateBucket, PutBucketPolicy), from data-plane events, which read and write the data itself (GetObject on app.log):

Access Analyzer has one more check, aimed the other way. Its external access findings list every resource, such as the logs bucket, that someone outside your account or organization (your zone of trust) could reach.

?How can it be sure it hasn't missed a request?

Because it doesn't test example requests. External access findings come from Zelkova, which translates policies into logical formulas (SMT formulas) and asks a solver, a program that decides whether a formula can be satisfied, whether any request at all from outside the zone of trust could be allowed. The FMCAD 2018 paper notes that it "solves a PSPACE-complete problem" (a class of problems that are hard in general) "and is invoked many millions of times daily."

9.2iam:PassRole, the quiet escalation

Many services act for you with a role you give them: an EC2 instance profile, a Lambda execution role, a service role for CodeBuild (AWS's build service). Handing a role to a service takes the iam:PassRole permission.

?Why is PassRole dangerous?

Because it lets you use a role's permissions without ever assuming it. The PassRole docs give the example: Alice can't touch S3, but if she can pass an S3-capable role to a service, "the service could perform Amazon S3 actions on behalf of Alice." A developer with lambda:CreateFunction and iam:PassRole on * can run code as any role in the account that trusts Lambda.

It's also hard to audit. "PassRole is not an API call", so CloudTrail has no PassRole events. You find it in the CreateFunction or RunInstances event that received the role.

{
  "Effect": "Allow",
  "Action": "iam:PassRole",
  "Resource": "arn:aws:iam::111122223333:role/lambda-app-*",
  "Condition": { "StringEquals": { "iam:PassedToService": "lambda.amazonaws.com" } }
}

Scope the Resource to a naming pattern, and pin the service with iam:PassedToService.

9.3Boundaries for delegated admin

Permissions boundaries exist for one situation above all: letting teams create their own roles without letting them create a role more powerful than themselves.

You allow iam:CreateRole only when the request attaches a specific boundary (the iam:PermissionsBoundary condition key), and you deny editing or removing that boundary. A team can then create any role it likes, and every one of them is capped by the boundary, whatever its identity policy says. It's the intersection rule from section 5.3, used on purpose.

Everything so far guards the credentials or shrinks what they can do. The last failure needs no stolen credentials at all, and no permission that's too broad.

10.1Hardy's compiler, 1988

Norm Hardy named the problem in a two-page paper, The Confused Deputy (Operating Systems Review 22(4), 1988), about a timesharing system at Tymshare, a company whose customers shared one large computer and paid for the time they used.

A compiler, (SYSX)FORT, had a "home files license" so it could write a statistics file in its own directory, SYSX. Users could name a file to receive debugging output. One user named (SYSX)BILL, the billing file. The compiler opened it with its own license and wrote over it. "The billing information was lost."

?Whose fault was it?

Nobody's code was wrong, and Hardy built his argument on that. "The compiler serves two masters and carries some authority from each," he wrote, and "it has no way to keep them apart." The user supplied a name; the compiler supplied the authority. His fix was capabilities, where the thing that names a file is also the thing that authorizes writing to it.

IAM, like Tymshare's system, is name-based: policies list ARNs. So the confused deputy is always possible, and AWS provides explicit checks to close it.

10.2Cross-account: the external ID

AWS's version involves a SaaS vendor, a company that sells software as a service, assuming roles in its customers' accounts. Say you hired one, Example Corp, to analyse the logs in your logs bucket. Here's AWS's own scenario, shown with the problem and then the fix:

What matters is who picks the external ID: it's generated and controlled by the vendor, unique per customer. If customers chose their own, the other customer would choose yours.

10.3Cross-service: SourceArn and SourceAccount

AWS services are deputies too. A bucket policy that lets cloudtrail.amazonaws.com write logs trusts CloudTrail, not the person who configured the trail. AWS spells it out: that bucket "could receive CloudTrail logs from ... an unauthorized actor in their AWS account, if they know the name of the S3 bucket."

You fix it by testing the context the service passes along:

{
  "Effect": "Allow",
  "Principal": { "Service": "cloudtrail.amazonaws.com" },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::central-logs/AWSLogs/111122223333/*",
  "Condition": { "StringEquals": { "aws:SourceAccount": "111122223333" } }
}

That covers the ways the boundary fails. What's left is finding the cause when it does.

11.1Read the message

Most services now return this shape:

User: arn:aws:iam::123456789012:user/John is not authorized to perform: codecommit:ListRepositories
with an explicit deny in a service control policy: arn:aws:organizations::777788889999:policy/o-exampleorgid/service_control_policy/p-examplepolicyid123

Two caveats from the docs: if several policy types deny, the message names only one, and some services don't use this format at all.

11.2A checklist, cheapest first

1. Who am I? aws sts get-caller-identity. A lot of AccessDenied tickets probably come down to the wrong principal: the node role instead of the pod's, an old profile, or an expired session (section 7.4).
2. Read the message for the policy type, as above.
3. Is it an explicit deny? Search SCPs, RCPs, boundaries and the resource policy for a Deny that matches, including ones with conditions.
4. Is it cross-account? Then both sides need an Allow (section 6).
5. Is it KMS? The key policy has to allow it, whatever IAM says.
6. Encoded message? Some services, EC2 among them, return an encoded authorization message. aws sts decode-authorization-message turns it into the policy context that denied it.
7. Test offline. The IAM policy simulator, or Access Analyzer's check-access-not-granted, before you change production policy.

The commands, grouped by the question each answers:

# Which identity is this code really using? (sections 1.3 and 7.4)
aws sts get-caller-identity
 
# What did the encoded denial message say? (step 6 above)
aws sts decode-authorization-message --encoded-message "$ENCODED"
 
# Stop pods and proxies on this instance reaching its role (section 8.3)
aws ec2 modify-instance-metadata-options --instance-id i-0abc \
  --http-tokens required --http-put-response-hop-limit 1

11.3Rules that hold up

1. Run code as a role. Temporary keys expire on their own; AKIA keys don't (section 1.3).
2. List the actions the code calls. Build the list from CloudTrail or Access Analyzer instead of writing s3:* (sections 3.3 and 9.1).
3. Put each limit where its owner can reach it. SCPs and RCPs for the organization, boundaries for delegated admins, and nothing important in the management account (sections 4 and 6.2).
4. Require IMDSv2 with a hop limit of 1 where pods shouldn't reach the node's role, and pin instance credentials to their VPC (section 8).
5. Give every deputy a way to say whose behalf it acts for. External IDs for vendors, aws:SourceArn or aws:SourceAccount for services (section 10).
6. Scope iam:PassRole by role-name pattern and iam:PassedToService (section 9.2).
7. Read the error before changing a policy. It names the policy type that said no.

11.4What you trade for what

11.5Symptom, cause, fix