What is CloudWatch logs agent?

The CloudWatch Logs agent provides an automated way to send log data to CloudWatch Logs from Amazon EC2 instances. The agent includes the following components:

  • A plug-in to the AWS CLI that pushes log data to CloudWatch Logs.
  • A script (daemon) that initiates the process to push data to CloudWatch Logs.
  • A cron job that ensures that the daemon is always running.

The AWS CloudWatch Logs Agent can be setup to push logs to the AWS CloudWatch.Its not mandatory to you that you should have ec2 instance.You can setup it on any server like AWS,Linode ,DigitalOcean, Google, Azure, etc.

So here we push Apache2 logs of our wordpress ec2 machine to aws CloudWatch Logs.

Table of Content :

Download / Install the Debian Package 

 
sudo -s

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

dpkg -i -E amazon-cloudwatch-agent.deb
 

Add cwagent User to adm group

Then modify the linux user account that the installer created cwagent and add it to the adm group, which will give it read permission to many of the default Ubuntu system logs. 

 
usermod -aG adm cwagent
 

Setup an IAM User Account and Permissions

Now we need to give your Ubuntu server permission to publish its log data to your AWS account's CloudWatch Logs

Create an IAM user in the AWS Console and take note of the AWS accessID and AWS secretKey. On your Ubuntu server create a file /home/cwagent/.aws/credentials with the following: 

 
[AmazonCloudWatchAgent]
aws_access_key_id = AwsAccessKeyId
aws_secret_access_key = SwsSecretAccessKey
 

You also must specify the AWS Region to send the metrics to, using the region field in the [AmazonCloudWatchAgent] section of the AWS config file, as in the

Create a file /home/cwagent/.aws/config with the following: 

 
[AmazonCloudWatchAgent]
output = text
region = eu-west-1
 

we have to tell CloudWatch Logs Agent to look here for credentials by appending to the /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml file: 

 

echo "[credentials]" >> /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
echo 'shared_credential_file = "/home/cwagent/.aws/credentials"' >> /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

 

Create a Log Group

CloudWatch Logs provides a notion of Log Groups and Log Streams, a log group can hold lots of log streams. Some like to create log groups dynamically, others may like all their server logs in the same group. In this tutorial we are going to create a single log group (eg web-server-log-group) that can hold logs for multiple servers by creating log streams like hostname.example.com/syslog

I like this approach because I can then limit the IAM permissions (setup in the next step) to only give access to publish logs and create streams in a single log group.

Here i have created my-wordpress-machine-log as log group in aws console.



Grant the IAM User / Role Permission to Publish Logs

We have created log group named my-wordpress-machine-log. We will need to attach an IAM policy to the IAM user you created in the previous step. Or if you are running on EC2 you can attach the policy to the assumed role. In AWS console you can the Add inline policy button, 


 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchLogAgentPutForWebServerLogGroup",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:ap-south-1:1234567:log-group:my-wordpress-machine-log",
                "arn:aws:logs:ap-south-1:1234567:log-group:my-wordpress-machine-log:*:*"
            ]
        }
    ]
}
 

Here 1234567 is our aws account id.

If you want to allow the CloudWatch Logs Agent to publish metrics data (CPU, Memory, Network, Disk) then you will need to attach an additional policy to allow cloudwatch:PutMetricDat

Configure CloudWatch Logs Agent to collect Log Files.

create a JSON file and place it here: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json 

 
nano /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
 

Put the foolowing content in the file 

 
{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "cwagent",
		"credentials_path":"/home/cwagent/.aws/credentials"
	},
	"logs": {
		"logs_collected": {
			"files": {
				"collect_list": [
					{
						"file_path": "/var/log/apache2",
						"log_group_name": "my-wordpress-machine-log",
						"log_stream_name": "{hostname}/apache2",
						"timestamp_format" :"%b %d %H:%M:%S"
					}
                                ]
                        }
               }
       }
}
 

The above will publish the contents of /var/log/apache2 from your Ubuntu server to the AWS CloudWatch Logs service.

Similarly you can configure any machine logs like /var/log/syslog or any web server log to AWS CloudWatch Logs service.

Enable and Start the CloudWatch Logs Agent service 

 
systemctl enable amazon-cloudwatch-agent.service

service amazon-cloudwatch-agent start
 

You will see logs are pushed to CloudWatch Logs service.

That's it!! We have seen how to Setup CloudWatch Logs Agent on Ubuntu and Push Ubuntu Apache2 logs to AWS CloudWatch Logs.