In this article we will look at how we can monitor the log files from an EC2 instance and trigger a notification in case a predefined condition, such as when a specific message was recorded in the logs, a specific message was seen multiple times, etc., is met.

Amazon CloudWatch Logs can be used to monitor and access the log files from EC2 instances. For instance, you can monitor if an error occurs and it is written in the logs or you can monitor the number of errors and decide on which threshold you want to take actions.

VMware Training – Resources (Intense)

Let’s discuss the concepts involved:

  • Log event –an activity recorded by the application that is monitored.
  • Log stream –a sequence of log events that have the same source.
  • Log group –a group of log streams that share the same access and monitoring policy.
  • Metric filter –a way to transform data points in a CloudWatch metric from the log events.

We will not only create the metric in this tutorial; we will also create an alarm based on the CloudWatch metric that will inform us when a specific event took place.

So let’s start doing this.

I have one EC2 instance that runs Amazon Linux. Check the instance ID. We will use this later to identify the log stream name:

On this EC2 instance, I have a file that contains some text:

lab@UBUNTU:~/AWS$ ssh -i "eu-central-1.pem" ec2-user@52.28.69.226

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/
33 package(s) needed for security, out of 66 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-30-232 ~]$ cat /home/ec2-user/monitored_file
1. This line does not contain the interesting log message
2. This line does not contain the interesting log message
3. This line does not contain the interesting log message
[ec2-user@ip-172-31-30-232 ~]$

We will monitor this file so that in case the string “ERROR” shows up, a notification will trigger.

Before we jump and configure the CloudWatch Logs Agent on the EC2 instance, we will need the access key ID and the secret access key from the user that has access.

First, we have the policy that will allow access to the logs. This is a custom made policy:

This is the content of the policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:*"
      ],
      "Resource": [
        "arn:aws:logs:eu-central-1:*:*"
      ]
    }
  ]
}

As you can see below:

The policy is then attached to a group that our user whose credentials we will use is part of.

Next, let’s download the latest CloudWatch Logs Agent and install it:

[ec2-user@ip-172-31-30-232 ~]$ wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
--2015-08-10 15:14:35--  https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.32.48
Connecting to s3.amazonaws.com 

===== CUT FOR BREVITY =====

[45459/45459]

[ec2-user@ip-172-31-30-232 ~]$

During installation, the configuration of the Agent is done as well. This is where the credentials will be used and where we will specify what files will be monitored. There are a few configuration options where if you are not sure, you can choose the default values:

[ec2-user@ip-172-31-30-232 ~]$ sudo python ./awslogs-agent-setup.py --region eu-central-1
Launching interactive setup of CloudWatch Logs agent ...

Step 1 of 5: Installing pip ...DONE

Step 2 of 5: Downloading the latest CloudWatch Logs agent bits ... DONE

Step 3 of 5: Configuring AWS CLI ...
AWS Access Key ID [None]: AKIAI7H3CT6BBJD7DNWA
AWS Secret Access Key [None]: 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
Default region name [eu-central-1]:
Default output format [None]:

Step 4 of 5: Configuring the CloudWatch Logs Agent ...
Path of log file to upload [/var/log/messages]: /home/ec2-user/monitored_file
Destination Log Group name [/home/ec2-user/monitored_file]:

Choose Log Stream name:
  1. Use EC2 instance id.
  2. Use hostname.
  3. Custom.
Enter choice [1]: 1

Choose Log Event timestamp format:
  1. %b %d %H:%M:%S    (Dec 31 23:59:59)
  2. %d/%b/%Y:%H:%M:%S (10/Oct/2000:13:55:36)
  3. %Y-%m-%d %H:%M:%S (2008-09-08 11:52:54)
  4. Custom
Enter choice [1]: 1

Choose initial position of upload:
  1. From start of file.
  2. From end of file.
Enter choice [1]: 1
More log files to configure? [Y]: N

Step 5 of 5: Setting up agent as a daemon ...DONE

[ec2-user@ip-172-31-30-232 ~]$

After the installation and configuration is done, it’s time to see the feature in action. Go to CloudWatch console and from the left menu, choose “Logs” and you will see the log group configured during the installation:

If you click on the log group and then on the log stream, you will see the content of the file that we want to monitor:

Now it’s time to create the metric filter. Select the log group and click on “Create Metric Filter”. On the first step, input the pattern that when it’s seen in the file, a notification should be sent. For our test, the pattern is “ERROR” (without quotes):

On the same step, you can check if the pattern selected is already in the file that you want to monitor. Click on “Assign Metric” to move to the next step:

On the next step, you need to provide the name of the filter and the name of the metric. We will use later this metric to send an email based on its value. Finish by clicking on “Create Filter”:

The filter was created and now you can go directly and create an alarm:

The alarm is created in the same way as other alarms. In our case, we want to receive an email whenever we see the word “ERROR” in the file. The check will be done for a period of five minutes:

Once the alarm is created, it will be in INSUFFICIENT_DATA state:

Now, let’s insert two lines that contain the word “ERROR” in the file that is being monitored:

[ec2-user@ip-172-31-30-232 ~]$ echo "ERROR - This line contains the pattern" >> /home/ec2-user/monitored_file
[ec2-user@ip-172-31-30-232 ~]$ echo "ERROR - This line contains the pattern" >> /home/ec2-user/monitored_file

As you can see, the file now has these additional lines:

[ec2-user@ip-172-31-30-232 ~]$ cat /home/ec2-user/monitored_file
1. This line does not contain the interesting log message
2. This line does not contain the interesting log message
3. This line does not contain the interesting log message
ERROR - This line contains the pattern
ERROR - This line contains the pattern
[ec2-user@ip-172-31-30-232 ~]$

Once this is done, the alarm will move to ALARM state:

Because the threshold was crossed:

As you can see from the graph, the pattern was detected two times:

And of course, the email with the notification was sent as well:

If for the next period of five minutes, the pattern will not be seen, then the alarm will move back to INSUFFICIENT_DATA state.

And this is how you can use CloudWatch Logs to monitor the logs from an EC2 instance.

In this article we saw how to install CloudWatch Logs Agent, how to configure it, how to create a metric filter and then use the new metric in an alarm to get notified in case the specific logs are recorded in the file that we are monitoring.

There are few other use cases for CloudWatch Logs that you can read by visiting the links in the references section below.

References