Home CloudGoat Vulnerable Lambda Scenario - Part 2 (Response)
Post
Cancel

CloudGoat Vulnerable Lambda Scenario - Part 2 (Response)

Posted Jun 21, 2023
By Dominic Spinosa
15 min read
  • Part 1 - Attacking CloudGoat’s vulnerable Lambda scenario
  • Part 2 (you are here) - Responding to the attack

cloudgoat

In part one of this series, we walked through the steps necessary to exploit a Lambda function with an inherent SQL injection vulnerability using a compromised AWS access key, escalate our privileges and subsequently access company secrets. In this post (part two), we’ll assume the role of a responder (e.g., SOC analyst, IR team member) and conduct an investigation with the goal of evicting the attacker from the affected AWS environment. The structure of this post will loosely follow NIST’s incident handling guide as well as the AWS Security Incident Response Guide:

  • Preparation (skip this, as we’re dealing with an “active” incident)
  • Detection and Analysis
  • Containment, Eradication and Recovery
  • Post-Incident Activity (we’ll cover this in future posts)

Detection: The Initial Alert

The starting point that triggers our investigation would be entirely dependent on the security tooling being used by the organization. For the sake of example, we’ll only rely on services, tooling and telemetry that are native to AWS to facilitate the investigation and “mock” the initial alert.

Here’s a quick recap of the attack we executed in the first post:

  • Discovery: Gained some situational awareness by exploring the affected IAM user’s permissions and enumerating available resources within the AWS account
  • Privilege Escalation: Exploited a reachable Lambda function with an inherent SQL injection vulnerability to attach the AdministratorAccess managed policy to the affected IAM user
  • Collection: Leveraged the newly minted access to retrieve secrets from Secrets Manager

Let’s assume the worst case scenario, in which the initial alert points to a user accessing the final_flag secret from a suspicious IP address (one that isn’t owned/managed by the organization or hasn’t been previously observed). This could be in the form of a GuardDuty IAM finding, a custom detection, an alert from a third-party solution, etc. In any case, this is what we’re given:

The secret “vulnerable_lambda_cgid13u1qpdipe-final_flag” was accessed using AWS Secrets Manager by a suspicious IP address (1.2.3.4) at 2023-03-28T17:18:59Z.

Analysis: Building Context and Timeline

A responder’s ability to quickly and effectively investigate an incident is almost entirely dependent on the context that’s made available to them. In this example, the context is quite limited (intentionally), so we’re left to build it out on our own.

Analyze Cloudtrail Logs Using Athena

We know that a secret within AWS Secrets Manager was accessed from a suspicious IP. Let’s pivot on the secret ID in question by pulling the relevant CloudTrail logs - events with an event name of GetSecretValue further filtered on both the secret ID (CloudTrailEvent:requestParameters:secretId) and the source IP address. There are many options to sift through and analyze CloudTrail data, such as piping output from the AWS CLI to a utility (e.g., jq), using the Python SDK and storing the output as a Pandas DataFrame (ideally, using a Jupyter notebook), etc. Because we’re dedicated to the theme of using native AWS services and tooling when possible, we’ll query CloudTrail logs using Athena for this purpose.

Execute this bash script to setup the necessary Athena components for CloudTrail log analysis. For example:

1
2
3
4
5
6
7
8
9

./aws-setup-cloudtrail-investigation.sh \ 
   -c cloudtrail-logs-123/AWSLogs/REDACTED/CloudTrail/us-east-1 \ # full S3 bucket path to CloudTrail logs
   -w secret-exposure-workgroup \ # name of Athena workgroup
   -d "workgroup for investigating incident XYZ" \ # Athena workgroup description
   -a us-east-1 \ # Athena workgroup region
   -r athena-results-456 \ # bucket to store Athena query results
   -n cloudtrail_logs_us_east_1 \ # Athena table name
   -s "2023/03/01" # CloudTrail logs start date
   -e "2023/03/08" # CloudTrail logs end date

NOTE: The CloudTrail “Event history” section in the console has a “Create Athena table” option. However, the resulting SQL statement doesn’t account for partitioning, which may unnecessarily increase query time and costs, depending on the size of the trail being imported to Athena.

Now, we can begin our investigation! Navigate to Athena in the AWS console, open up a new tab within the query editor:

cloudgoat

Choose the newly created workgroup using the dropdown in the top-right:

cloudgoat

Locate the initial CloudTrail event that likely produced the alert:

1
2
3
4
5
6

select eventtime, useridentity, eventsource, eventname, awsregion, sourceipaddress, useragent, requestparameters, tlsdetails
from cloudtrail_logs_pp
where eventname = 'GetSecretValue'
and timestamp = '2023/03/28'
and sourceipaddress = '1.2.3.4'
and json_extract_scalar(requestparameters, '$.secretId') = 'vulnerable_lambda_cgid13u1qpdipe-final_flag'

Review the results, with specific focus on the useridentity attribute:

eventtimeuseridentityeventsourceeventnameawsregionsourceipaddressuseragentrequestparameterstlsdetails
2023-03-28T17:18:59Z{type=IAMUser, principalid=AIDAREDACTED, arn=arn:aws:iam::REDACTED:user/cg-bilbo-vulnerable_lambda_cgid13u1qpdipe, accountid=REDACTED, invokedby=null, accesskeyid=AKIAREDACTED, username=cg-bilbo-vulnerable_lambda_cgid13u1qpdipe, sessioncontext=null}secretsmanager.amazonaws.comGetSecretValueus-east-11.2.3.4aws-cli/2.9.1 Python/3.9.11 Linux/5.15.90.1-microsoft-standard-WSL2 exe/x86_64.ubuntu.22 prompt/off command/secretsmanager.get-secret-value{“secretId”:”vulnerable_lambda_cgid13u1qpdipe-final_flag”}{tlsversion=TLSv1.2, ciphersuite=ECDHE-RSA-AES128-GCM-SHA256, clientprovidedhostheader=secretsmanager.us-east-1.amazonaws.com}

We’ve discovered the offending principal is an IAM user prepended with “cg-bilbo”. Let’s determine what else this particular user has done from the provided IP address by pivoting on the corresponding access key:

1
2
3
4
5

select eventtime, sourceipaddress, useridentity.username, eventsource, eventname
from cloudtrail_logs_pp
where useridentity.accesskeyid = 'AKIAREDACTED'
and sourceipaddress = '1.2.3.4'
order by eventtime
eventtimesourceipaddressusernameeventsourceeventname
2023-03-28T17:00:57Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListAccountAliases
2023-03-28T17:01:44Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListGroupsForUser
2023-03-28T17:01:56Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListAttachedUserPolicies
2023-03-28T17:02:05Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListUserPolicies
2023-03-28T17:02:28Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comGetUserPolicy
2023-03-28T17:02:37Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListRoles
2023-03-28T17:02:46Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListRolePolicies
2023-03-28T17:02:56Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comGetRolePolicy
2023-03-28T17:03:39Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipests.amazonaws.comAssumeRole
2023-03-28T17:03:55Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipests.amazonaws.comAssumeRole
2023-03-28T17:05:56Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipests.amazonaws.comAssumeRole
2023-03-28T17:07:23Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipests.amazonaws.comAssumeRole
2023-03-28T17:18:08Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListAttachedUserPolicies
2023-03-28T17:18:25Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipeiam.amazonaws.comListAttachedUserPolicies
2023-03-28T17:18:38Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipesecretsmanager.amazonaws.comListSecrets
2023-03-28T17:18:59Z1.2.3.4cg-bilbo-vulnerable_lambda_cgid13u1qpdipesecretsmanager.amazonaws.comGetSecretValue

It’s entirely possible that the attacker has used this compromised access key from multiple IP addresses to throw us off. However, in order to detangle the attacker activity from legitimate use of the same access key, we would need to determine when the access key was compromised and perform a retroactive analysis of the IAM user’s activity to identify outliers based on known-good IP addresses, user agent string, etc.

Additionally, we should pivot solely on the source IP address to determine if the attacker is present elsewhere in the same AWS account. This could be done to rule out the possibility that the attacker was able to compromise this IAM user account through some other compromised identity or resource in the account.

The initial set of API calls leading up to the call to AssumeRole is interesting, as those calls appear to be potential reconnaissance (series of Get* and List* requests). To confirm this suspicion, let’s dig into the request parameters. It’d be quite strange for a legitimate user to make a bunch of API calls to determine what permissions they had, no?

1
2
3
4
5

select eventtime, eventname, requestparameters, responseelements
from cloudtrail_logs_pp
where useridentity.accesskeyid = 'AKIAREDACTED'
and sourceipaddress = '1.2.3.4'
order by eventtime
eventtimeeventnamerequestparametersresponseelements
2023-03-28T17:00:57ZListAccountAliases  
2023-03-28T17:01:44ZListGroupsForUser{“userName”:”cg-bilbo-vulnerable_lambda_cgid13u1qpdipe”} 
2023-03-28T17:01:56ZListAttachedUserPolicies{“userName”:”cg-bilbo-vulnerable_lambda_cgid13u1qpdipe”} 
2023-03-28T17:02:05ZListUserPolicies{“userName”:”cg-bilbo-vulnerable_lambda_cgid13u1qpdipe”} 
2023-03-28T17:02:28ZGetUserPolicy{“userName”:”cg-bilbo-vulnerable_lambda_cgid13u1qpdipe”,”policyName”:”cg-bilbo-vulnerable_lambda_cgid13u1qpdipe-standard-user-assumer”} 
2023-03-28T17:02:37ZListRoles  
2023-03-28T17:02:46ZListRolePolicies{“roleName”:”cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”} 
2023-03-28T17:02:56ZGetRolePolicy{“policyName”:”lambda-invoker”,”roleName”:”cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”} 
2023-03-28T17:03:39ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023021100268365”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:03:39 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023021100268365”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023021100268365”}}
2023-03-28T17:03:55ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023036636263280”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:03:55 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023036636263280”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023036636263280”}}
2023-03-28T17:05:56ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023157475147111”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:05:56 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023157475147111”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023157475147111”}}
2023-03-28T17:07:23ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023244813312140”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:07:23 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023244813312140”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023244813312140”}}
2023-03-28T17:18:08ZListAttachedUserPolicies{“userName”:”cg-bilbo-“} 
2023-03-28T17:18:25ZListAttachedUserPolicies{“userName”:”cg-bilbo-vulnerable_lambda_cgid13u1qpdipe”} 
2023-03-28T17:18:38ZListSecrets  
2023-03-28T17:18:59ZGetSecretValue{“secretId”:”vulnerable_lambda_cgid13u1qpdipe-final_flag”} 

Shockingly, our suspicion was correct! According to the requestparameters, the user was targeting the very IAM user they used to issue the request. This is a very common form of discovery (more specifically, situational awareness) when a cloud account has been compromised. Ironically, security practitioners often find themselves needing to perform a bit of discovery as well because they typically don’t have deep insight into their organization’s infrastructure, identities, workloads, etc. More specifically, it would be really helpful to understand what permissions this IAM user had at the time of compromise (before the attacker had a chance to make any IAM-specific changes). Unless our organization has AWS Config setup, our only other option is to dig through CloudTrail logs and manually piece together relevant IAM events. Let’s enumerate this IAM user’s current permissions, then we can sweep CloudTrail events for any IAM actions targeting this IAM user. The results won’t be 100% accurate, but it’s at least a reference point.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

# list inline policies attached to user
aws iam list-user-policies --user-name cg-bilbo-vulnerable_lambda_cgid13u1qpdipe
{
    "PolicyNames": [
        "cg-bilbo-vulnerable_lambda_cgid13u1qpdipe-standard-user-assumer"
    ]
}
# get policy document
aws iam get-user-policy --user-name cg-bilbo-vulnerable_lambda_cgid13u1qpdipe --policy-name cg-bilbo-vulnerable_lambda_cgid13u1qpdipe-standard-user-assumer
{
    "UserName": "cg-bilbo-vulnerable_lambda_cgid13u1qpdipe",
    "PolicyName": "cg-bilbo-vulnerable_lambda_cgid13u1qpdipe-standard-user-assumer",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "sts:AssumeRole",
                "Effect": "Allow",
                "Resource": "arn:aws:iam::REDACTED:role/cg-lambda-invoker*",
                "Sid": ""
            },
            {
                "Action": [
                    "iam:Get*",
                    "iam:List*",
                    "iam:SimulateCustomPolicy",
                    "iam:SimulatePrincipalPolicy"
                ],
                "Effect": "Allow",
                "Resource": "*",
                "Sid": ""
            }
        ]
    }
}
# list managed policies attached to user
aws iam list-attached-user-policies --user-name cg-bilbo-vulnerable_lambda_cgid13u1qpdipe
{
    "AttachedPolicies": [
        {
            "PolicyName": "AdministratorAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess"
        }
    ]
}
# list group memberships
aws iam list-groups-for-user --user-name cg-bilbo-vulnerable_lambda_cgid13u1qpdipe
{
    "Groups": []
}
# list inline policies attached to group
# aws iam list-group-policies --group-name <group_name>
# list managed policies attached to group
# aws iam list-attached-group-policies --group-name <group_name>

According to the current state of the affected IAM user, they have full administrative access to the entire account, courtesy of the AdministratorAccess managed policy. Additionally, they’re able to assume any IAM role prepended with “cg-lambda-invoker.” Let’s determine which IAM roles the user was able to assume, and what subsequent actions they were able to perform using that role.

1
2
3
4
5
6

select eventtime, eventname, requestparameters, responseelements
from cloudtrail_logs_pp
where useridentity.accesskeyid = 'AKIAREDACTED'
and sourceipaddress = '1.2.3.4'
and eventname = 'AssumeRole'
order by eventtime
eventtimeeventnamerequestparametersresponseelements
2023-03-28T17:03:39ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023021100268365”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:03:39 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023021100268365”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023021100268365”}}
2023-03-28T17:03:55ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023036636263280”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:03:55 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023036636263280”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023036636263280”}}
2023-03-28T17:05:56ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023157475147111”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:05:56 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023157475147111”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023157475147111”}}
2023-03-28T17:07:23ZAssumeRole{“roleArn”:”arn:aws:iam::REDACTED:role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe”,”roleSessionName”:”1680023244813312140”,”durationSeconds”:3600}{“credentials”:{“accessKeyId”:”ASIAREDACTED”,”sessionToken”:”REDACTED”,”expiration”:”Mar 28, 2023, 6:07:23 PM”},”assumedRoleUser”:{“assumedRoleId”:”AROAREDACTED:1680023244813312140”,”arn”:”arn:aws:sts::REDACTED:assumed-role/cg-lambda-invoker-vulnerable_lambda_cgid13u1qpdipe/1680023244813312140”}}

The AssumeRole operation leverages the STS service to generate a temporary set of credentials with the same permissions as the target role to be assumed. We can further pivot on these access keys (starting with ‘ASIA’) to identify the subsequent activity carried out using these temporary credentials.

1
2
3
4
5

select eventtime, useridentity, eventsource, eventname, requestparameters, responseelements 
from cloudtrail_logs_pp
where useridentity.accesskeyid in ('ASIAREDACTED', 'ASIAREDACTED',  'ASIAREDACTED', 'ASIAREDACTED')
and sourceipaddress = '1.2.3.4'
order by eventtime
eventtimeuseridentityeventsourceeventnamerequestparametersresponseelements
2023-03-28T17:03:55ZASIAREDACTEDlambda.amazonaws.comListFunctions20150331  
2023-03-28T17:05:56ZASIAREDACTEDlambda.amazonaws.comGetFunction20150331v2{“functionName”:”vulnerable_lambda_cgid13u1qpdipe-policy_applier_lambda1”} 
2023-03-28T17:07:25ZASIAREDACTEDlambda.amazonaws.comInvoke  

Investigate Lambda Function Logs Using CloudWatch

It appears the attacker listed the available Lambda functions and used the call to GetFunctions to pull more detailed information about the vulnerable_lambda_cgid13u1qpdipe-policy_applier_lambda1 Lambda function, which includes a direct link to the actual function code. Although CloudTrail doesn’t log the specifics of the function invocation, we can tap into the associated CloudWatch log group (which is automatically created along with the Lambda function) to achieve this.

  1. Navigate to CloudWatch in the console

  2. Choose Logs > Log groups

cloudgoat

  1. Select the appropriate log group (e.g., /aws/lambda/my-function). In this case, /aws/lambda/vulnerable_lambda_cgid13u1qpdipe-policy_applier_lambda1.

  2. Multiple log streams may exist. Be sure to choose the stream that contains logs covering the same time frame as the call to Invoke noted in the CloudTrail events (in this case, 2023-03-28T17:07:25Z)

cloudgoat

timestampmessage
1680023244380“INIT_START Runtime Version: python:3.9.v18 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:edb5a058bfa782cb9cedc6d534ac8b8c193bc28e9a9879d9f5ebaaf619cd0fc0
1680023244888“START RequestId: 4439301b-290c-4ae5-811b-228bb60418c2 Version: $LATEST
1680023244889“target policys are : ["”AdministratorAccess’; –””]
1680023244889“applying AdministratorAccess to cg-bilbo-vulnerable_lambda_cgid13u1qpdipe
1680023245164“result: 200
1680023245183“END RequestId: 4439301b-290c-4ae5-811b-228bb60418c2
1680023245183“REPORT RequestId: 4439301b-290c-4ae5-811b-228bb60418c2 Duration: 294.87 ms Billed Duration: 295 ms Memory Size: 128 MB Max Memory Used: 73 MB Init Duration: 507.83 ms

According to these Lambda function logs, the AdministratorAccess managed policy was attached to the cg-bilbo IAM user in question. We can confirm this by searching for this API call (AttachUserPolicy) in CloudTrail.

1
2
3
4

select eventtime, eventname, sourceipaddress, useridentity.arn, requestparameters, responseelements
from cloudtrail_logs_pp
where eventname = 'AttachUserPolicy'
and timestamp = '2023/03/28'
eventtimeeventnamesourceipaddressarnrequestparametersresponseelements
2023-03-28T17:07:25ZAttachUserPolicy44.200.244.86arn:aws:sts::REDACTED:assumed-role/vulnerable_lambda_cgid13u1qpdipe-policy_applier_lambda1/vulnerable_lambda_cgid13u1qpdipe-policy_applier_lambda1{“userName”:”cg-bilbo-vulnerable_lambda_cgid13u1qpdipe”,”policyArn”:”arn:aws:iam::aws:policy/AdministratorAccess”} 

Two interesting things to note here:

  1. The source IP address differs from the previous requests because this API call originated from a Lambda function, which runs within internal AWS infrastructure
    1
2

    $ dig +short -x 44.200.244.86
ec2-44-200-244-86.compute-1.amazonaws.com.
  2. The identity making this API call is a new IAM role that is attached to the Lambda function as its execution role (vulnerable_lambda_cgid13u1qpdipe-policy_applier_lambda1).

NOTE: If, for some reason, Lambda function logs were not available via CloudWatch or the logs were insufficient, we could proactively hunt for events in CloudTrail where the user identity is the Lambda execution role:

1
2
3
4
5

  select eventtime, useridentity, eventsource, eventname, awsregion, requestparameters, responseelements
  from cloudtrail_logs_pp
  where timestamp >= '2023/03/28'
  and useridentity.sessioncontext.sessionissuer.username = 'vulnerable_lambda_cgid13u1qpdipe-policy_applier_lambda1'
  order by eventtime desc

At this point, we’ve confirmed that the attacker was successfully able to attach the AdministratorAccess managed policy to the compromised IAM user. Given this access, the attacker was likely able to cause a lot more damage, so let’s query for CloudTrail logs involving the compromised IAM user after the eventtime associated with the AttachUserPolicy event.

1
2
3
4
5

select eventtime, eventname, sourceipaddress, useridentity.arn, requestparameters, responseelements
from cloudtrail_logs_pp
where useridentity.principalid = 'AIDAREDACTED'
and sourceipaddress = '1.2.3.4'
and from_iso8601_timestamp(eventtime) >  from_iso8601_timestamp('2023-03-28T17:07:25Z')
eventtimeeventnamesourceipaddressarnrequestparametersresponseelements
2023-03-28T17:18:38ZListSecrets1.2.3.4arn:aws:iam::REDACTED:user/cg-bilbo-vulnerable_lambda_cgid13u1qpdipe  
2023-03-28T17:18:59ZGetSecretValue1.2.3.4arn:aws:iam::REDACTED:user/cg-bilbo-vulnerable_lambda_cgid13u1qpdipe{“secretId”:”vulnerable_lambda_cgid13u1qpdipe-final_flag”} 

And we’ve come full circle. At this point in time, we have confidently established a detailed timeline of events.

Containment

We have to move quickly. The attacker has administrative access to the affected AWS account, and they’ve already extracted one of the company’s secrets from Secrets Manager.

In almost every situation, IAM users should not be used. However, it is worth the effort to identify the owner of the user and confirm whether or not it is being used in production. If it is, this portion of the response playbook will change a bit, as you’ll need to carefully create and distribute a new access key for the IAM user before deactivating the compromised key, so as not to cause any unnecessary downtime.

Quarantine the IAM User

In order to prevent the attacker from causing any further damage by using this IAM user, we must delete the affected access keys (disabling isn’t preferred because disabled access keys can be re-enabled). Note that deleting an access key is an irreversible action, so be sure to properly document the access key ID somewhere (e.g., incident report) for record keeping.

1
2

# delete the access key 
aws iam delete-access-key --access-key-id AKIAREDACTED

If we were able to confidently determine that the compromised IAM user is not currently in use by critical applications, infrastructure, etc., we can also attach a “deny all” IAM policy to ensure the IAM user is fully quarantined, preventing further compromise.

1
2
3

aws iam put-user-policy --user-name cg-bilbo-vulnerable_lambda_cgid13u1qpdipe \ 
    --policy-name 'DenyAllQuarantine' \ 
    --policy-document '{"Version":"2012-10-17","Statement":{"Effect":"Deny","Action":"*","Resource":"*"}}'

Eradication

Now that the user is effectively quarantined, we must comb through the CloudTrail logs in search for any attempts to persist or escalate privileges by creating new IAM resources, modifying existing ones, use of alternate methods of authentication, etc. We’ve already conducted this search and confirmed that the attacker was successful in attaching the managed AdministratorAccess policy to the compromised IAM user. We’ll need to detach this policy to return the IAM user to a “known good” state:

1
2
3

# detach the managed AdministratorAccess policy
aws iam detach-user-policy --user-name cg-bilbo-vulnerable_lambda_cgid13u1qpdipe \ 
    --policy-arn "arn:aws:iam::aws:policy/AdministratorAccess"

Recovery

We’re not out of the woods just yet. The Lambda function is still open to exploitation and offers up a method for trivial privilege escalation by way of the SQL injection vulnerability. To remedy this, we need to work with the owner of the Lambda function to patch the relevant Python code. In this case, we need to remove the string concatenation and replace it with a prepared statement that leverages a parameterized query. This can be achieved using the sqlite_utils library that’s already in use by the Lambda function, specifically by passing parameters to the db.query() method:

1
2
3
4
5

<     statement = f"select policy_name from policies where policy_name='{policy}' and public='True'"
<     for row in db.query(statement):
---
>     prepared_statement = "select policy_name from policies where policy_name=? and public='True'"
>     for row in db.query(prepared_statement, [policy]):

This change can be made from the AWS console via the Lambda function’s “Code” tab (don’t forget to click “Deploy”). Next, navigate to the “Test” tab and create a new event using the following parameters:

1
2
3
4

{
  "user_name": "cg-bilbo-vulnerable_lambda_cgid13u1qpdipe",
  "policy_names": ["AdministratorAccess'; --"]
}

The test response confirms that our fix is working as expected!

1
2
3
4
5
6
7
8
9
10

Test Event Name
myTestEvent

Response
"AdministratorAccess'; -- is not an approved policy, please only choose from approved policies and don't cheat. :) "

Function Logs
START RequestId: 1eb04d3d-8b02-4b7c-9e7c-4602bad751c7 Version: $LATEST
target policys are : ["AdministratorAccess'; --"]
AdministratorAccess'; -- is not an approved policy, please only choose from approved policies and don't cheat. :)

How can we detect this?

Part three of this series will cover how we can deploy detections designed to proactively identify this malicious activity.

Happy (hacking|hunting)!

Cloud, AWS
aws cloud lab walkthrough lambda response
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents