8 Tips for AWS CloudTrail for security monitoring and best practices

1. CloudTrail is turned on by default and will store logs in the CloudTrail portal for 90 days. To extend the retention of the logs past 90 days, you need to configure a trail to send the logs to an S3 bucket. Additionally, you can also send the logs to CloudWatch to monitor your trail logs and notify you when specific activity occurs. Make sure to minimize access to the S3 bucket that CloudTrail writes to prevent destruction of logs. See the HOWTO section below on how to Enable Object Lock on the S3 bucket.

2. By default only Management events (API calls that create/delete/start or stop a service or make a call into an existing service that is running) are recorded. This includes calls made by using the AWS management console, AWS SDKs, command line tools, and higher level AWS services. If you want to obtain Data events (paid service), such as API calls to retrieve/store/query S3 data, calls to Lambda functions, and Database API calls, you must create a Trail and enable optional events. Although there is a cost associated with enabling Data Events, I highly recommend doing so to get detailed insights into your events.

3. Enable CloudTrail Log File Integrity. This gives you a cryptographic signature to ensure the logs you have received are not tampered, and can be verified easily with the AWS CLI. It will also give you a way to prove logs weren’t deleted as you can use the public key to validate the digest file after the log files are delivered.

4. Most resources in AWS are region specific. However, CloudTrail is configured to be global by default i.e. it is enabled in all AWS regions. Ensuring this default will enable more visibility and automatically track data from new Regions as they come online or can be alerted upon if the other regions are not being used but there is an activity. Disabling multi region logging gives a bad actor free reign in every region except for the one the trail was created in.

5. Encrypt CloudTrail logs using server side encryption using KMS (SSE-KMS) instead of the S3 managed server side encryption (SSE-S3) to control key rotation, obtain auditing visibility into key usage and control who can read the CloudTrail log files within your organization.

6. Set up an organization trail to monitor all of the logs generated by the AWS accounts within an AWS Organization. AWS Organizations allows you to centrally manage the access permissions of users in all of the accounts in the organization, and can be set up at no additional cost. Organizations are recommended when your team needs to manage many different AWS accounts by governing your ever-changing environment and enforce configurations on your primary and member accounts.

7. Implement a powerful search capability with CloudTrail, like an ELK cluster, Splunk, or a cloud service like Loggly or Papertrail as these integrations allow you to quickly add and remove filters to explore your data. Segment it from your production environment. If you’re using a Lambda function to process these logs, make sure the Lambda function used to process logs cannot be tampered with by your production users and roles.

8. Minimize access to the CloudTrail API. If a key or role is compromised with write access to any CloudTrail API actions (DeleteTrail, StopLogging, UpdateTrail), bad things could happen and can go undetected for a long time. For example, an attacker can run the following commands with the compromised account to cover their steps

— delete trail
aws cloudtrail delete-trail --name <your-trail-trail>

— disable logging
aws cloudtrail stop-logging --name <your-trail-trail>

- disable multi-region and global services logging
aws cloudtrail update-trail --name <your-trail-trail> --no-is-multi-region-trail --no-include-global-service-events

Hence, grant least privileges to your users and roles. AWS suggests granting AWSCloudTrail_ReadOnlyAccess policy which provides permissions to view the the logs, including recent events and event history. This policy also allows to view existing trails, event data stores, and channels. Roles and users with this policy can download the event history, but they can’t create or update trails, event data stores, or channels. You can further restrict with the policies below

HOWTO: Set up AWS CloudTrail Trails and extract logs on your local machine

Prerequisities

AWS CLI is installed
jq command line utility is installed

Create CloudTrail Trail via AWS management console

From the CloudTrail console, click Trails on the left navigation menu.
Click Create Trail
Under General Details
- Enter Trail name
- Check Enable for all accounts in my organization
- Select Create new S3 bucket
- Enter S3 bucket name under Trail log bucket and folder
- Enter AWS KMS alias
Under Additional settings
- Check Log file validation
Under CloudWatch Logs – optional
- Check Enabled
- Enter name for Log group name
- Enter Role name
Click Next
On the next page, review and Click Create Trail

List Trails via the CLI

aws cloudtrail list-trails

{
    "Trails": [
        {
            "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/management-events-maghilda",
            "Name": "management-events-maghilda",
            "HomeRegion": "us-east-1"
        }
    ]
}

Describe Trails via the CLI

aws cloudtrail describe-trails

{
    "trailList": [
        {
            "Name": "management-events-maghilda",
            "S3BucketName": "aws-cloudtrail-logs-123456789012-14073d57",
            "IncludeGlobalServiceEvents": true,
            "IsMultiRegionTrail": true,
            "HomeRegion": "us-east-1",
            "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/management-events-maghilda",
            "LogFileValidationEnabled": true,
            "CloudWatchLogsLogGroupArn": "arn:aws:logs:us-east-1:123456789012:log-group:aws-cloudtrail-logs-123456789012-decc01f1:*",
            "CloudWatchLogsRoleArn": "arn:aws:iam::123456789012:role/service-role/CloudTrailRoleforCloudWatchLogs_maghilda",
            "HasCustomEventSelectors": true,
            "HasInsightSelectors": false,
            "IsOrganizationTrail": true
        }
    ]
}

aws cloudtrail describe-trails reveals the S3 buckets being logged to.
IncludeGlobalServiceEvents is true, the CloudTrail bucket will include logs for all regions.
Logs land about every 15 minutes.

Extract logs to your local machine

Perform some activities to generate logs. For example, sign out and sign in.
Since we enabled log file validation when creating the Trail, you will see two folders in your S3 bucket

CloudTrail
CloudTrail-Digest

Files under the digest folder are used for log file validation. Each file contains references to the actual log data files written in the past hour, with their digital signature. By comparing the signatures to the actual files, you can determine whether those files were modified after being delivered.

CloudTrail saves your audit logs in gzip’d json files in the S3 bucket that you specified when you created the Trail.

Filename format;
{ACCOUNT_ID}_CloudTrail_{REGION}_{DATETIME}_{UNIQUESTRING}.json.gz

Download the CloudTrail logs from the s3 bucket

On your machine, create a folder cloudtrail-logs and change the directory

$ cd cloudtrail-logs

aws s3 cp s3://aws-cloudtrail-logs-123456789012-14073d57/AWSLogs/o-4yaz7wm5fl/123456789012/CloudTrail/us-east-1/2024/01/09/ . --recursive

You will notice a bunch of logs files downloaded to your folder. Unzip the files

gzip -d *

Display the content of a file in the terminal

cat 123456789012_CloudTrail_us-east-1_20240109T2015Z_DiRi3MgyB7HRzeCu.json

This is not very reader friendly so we will use JQ next.

To easily view and perform analysis on JSON log files, we can use command-line utility like JQ. JQ is a lightweight command-line processor for JSON and is invaluable in reading these logs and doing basic filtering on small sets of logs. For larger logs and cost effective queries, use Athena to query S3 logs.

Display the above file in json format

cat 123456789012_CloudTrail_us-east-1_20240109T2015Z_DiRi3MgyB7HRzeCu.json | jq .

A few jq commands to look up logs.

Filter on the SourceIPAddress. Display all IP addresses logged by CloudTrail:
jq '.Records[] | .sourceIPAddress' *.json

Select only records with the sourceIPAddress=
jq '.Records[] | select(.sourceIPAddress="41.156.31.69")' *.json

Filter on the eventSource.
jq '.Records[] | .eventSource' *.json

Filter on the eventName
jq '.Records[] | .eventName' *.json

Filter on the userAgent
jq '.Records[] | .userAgent' *.json

Filter on errorCode
jq '.Records[] | .errorCode' *.json

Filter where eventName = CreateToek
jq '.Records[] | select(.eventName=="CreateToken")' *.json

cat * | jq '.Records[] | select(.eventName==“CreateToken")'

cat * | jq '.Records[] | select(.eventName=="CreateToken" or .eventName=="PutBucketPolicy" )’

Minimize access to CloudTrail API and S3 bucket containing CloudTrail logs

Ensure that the CloudTrail S3 buckets are not publicly accessible.

aws s3api get-public-access-block --bucket aws-cloudtrail-logs-123456789012-14073d57

{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

Ensure that the Object Lock is enabled.

Ensure that the S3 buckets associated with your Amazon CloudTrail trails are configured to use the Object Lock feature in order to prevent the objects they store (i.e. trail log files) from being deleted and meet regulatory compliance.

aws s3api get-object-lock-configuration --bucket aws-cloudtrail-logs-123456789012-14073d57

An error occurred (ObjectLockConfigurationNotFoundError) when calling the GetObjectLockConfiguration operation: Object Lock configuration does not exist for this bucket

Fix the above

aws s3api put-object-lock-configuration --bucket aws-cloudtrail-logs-123456789012-14073d57  --object-lock-configuration “ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode= GOVERNANCE,Days=30}}"

Run the command again to validate the change

aws s3api get-object-lock-configuration --bucket aws-cloudtrail-logs-123456789012-14073d57

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}
SUCCESS!