In this article we will discuss how you can use Amazon CloudTrail in conjunction with logstash software.

Amazon CloudTrail is a service that can provide a history of the AWS API calls for the AWS account. The API calls can be made through the AWS Management Console, AWS CLI, or SDK.

VMware Training – Resources (Intense)

The user can track which services that support CloudTrail were called and from which IP addresses the calls were made.

CloudTrail captures the AWS API calls and the logs are stored in an S3 bucket. The logs are delivered 15 minutes after the API call, which can happen multiple times in an hour.

One of the tools that can help in analysing large logging data is logstash. We will use this tool to process the logs created by CloudTrail.

To setup the logstash tool, during initial configuration, we will need to provide security credentials from a user that can access the S3 bucket where CloudTrail writes the logs.

So let’s get this started.

I have the S3 bucket where the CloudTrail logs will be stored:

To start CloudTrail, you need to access the “Management Tools” section from the AWS Management Console and then select “CloudTrail”:

You need to specify the S3 bucket where the logs will be kept and then you can start CloudTrail by clicking “Turn On”:

You can see the API activity by selecting the “API Activity History”. As you can see, there were no API calls made so far; hence the list is empty:

Select the “Configuration” menu to see the CloudTrail configuration. As you can see, you can enable CloudWatch so that a notification will be sent in case a specific API happens:

You only need to specify the log group for CloudTrail to enable CloudWatch:

Before that, you need to create a new role that will be assigned to CloudTrail so that the events will be delivered to CloudWatch. You only need to allow this role as the policy assigned to this role is already created by AWS:

From now on, CloudWatch will receive the notifications and as you can see, you can set alarms if you want to:

To install logstash as mentioned earlier, you will need to create a user that will have a policy attached so that it can access the CloudTrail logs.

This is the user I created:

And these are the security credentials which we will use later:

Next you will need to create the policy. Select “Policies” from the IAM console and click on “Create Policy”:

We will create our own policy, so select “Create Your Own Policy”:

Then paste the content of the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1440751066000",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::cloudtrail-012345678901 "
            ]
        },
        {
            "Sid": "Stmt1440751148000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::cloudtrail-012345678901 "
            ]
        }
    ]
}

Just make sure you change the policy and use your AWS account ID. You should have something like this. Click on “Create” to create the policy:

Next select the “Users” menu and then scroll to the “Permissions” section of the user configuration. Click on “Attach Policy”:

Filter the policies so that your own policies are displayed, select the previously created policy and then attach it to the user profile:

Now the policy is attached:

Meanwhile, the logs are generated in the S3 bucket for each event that takes place, as you can see below:

I played around with a few EC2 instances that I launched and then terminated them. You can see API history in the CloudTrail console:

Now, let’s move on to the logstash server. You have two options:

  • Install from scratch a package and then configure it to work with Amazon CloudTrail.
  • Launch an AMI from Amazon Marketplace that is specifically built to use CloudTrail with logstash.

If you don’t want to pay the money for the AMI from Amazon MarketPlace, you can choose the first option with the disadvantage that you will need to build everything.

Because this is a test environment, we will use the AMI from MarketPlace.

The AMI is brought to Amazon MarketPlace by a third party and you need to search in the MarketPlace for “logstash”. What the AMI can do is self-explanatory:

Select the region and the EC2 instance type that you want to use:

Provide the VPC details where the EC2 instance will be launched:

The security group that comes with the AMI allows SSH and HTTP because the user will need to connect to the EC2 instance for the initial configuration. Then HTTP will be used to see the CloudTrail logs:

Once the EC2 instance is up and running, you can connect to it:

Connect to the EC2 instance running the logstash package and move to the /root directory. There you will find the initial configuration utility:

root@ip-172-31-15-212:/root# pwd
/root
root@ip-172-31-15-212:/root# ls -l
total 4
-rwx------ 1 root root 2922 Apr 29 09:47 configure_cloudtrail
root@ip-172-31-15-212:/root#

Now execute it and provide the information required:

root@ip-172-31-15-212:/root# ./configure_cloudtrail


We need to ask a couple of questions before we can begin
collecting logs.  These questions are for setting up the jobs
that will run on this system only!!  All data collected in
this script will remain local to this instance.


What is the name of the S3 bucket you have your CloudTrail log data in? (ex. CloudTrail)

s3-bucket-cloudtrail-logs

What is the Access Key that you would like to use?

AKIAJ2UK45AXY3IZO6WA

What is the Secret Key that you would like to use?

8eRh12jmXX2bCTOkMldq6fl7kSdfCUuH7GuIF+Mq

What is the account number for this account?

012345678901

what name would you like to use for this account?

LOGSTASH-CLOUDTRAIL
Writing out scripts...
ok: run: logstash_indexer: (pid 1797) 0s
Okay, everything should be in place to start collecting logs...

You should be able to goto http://instance:80 and see your data in the next few minutes..
root@ip-172-31-15-212:/root#

Once this is complete, you can connect to the server using the URL: http://<public_IP>. The username will be “cloudtrail” and the password will be the instance ID of the EC2 instance.

The web page of the server can show you different statistics in many forms. The screenshot below shows only how many events were recorded:

But you can show what events took place, which took place most often, and so on.

And this concludes how you can use CloudTrail in conjunction with logstash to analyse the logs reported by CloudTrail and be able to display them in a readable format.

References