This is the second part of the AWS CLI series. The first part discussed the installation and configuration of AWS CLI and how to use parameters and options.

The second part will discuss more advanced topics such as how to use AWS CLI to interact with AWS services, specifically with EC2 and S3.

VMware Training – Resources (Intense)

Using AWS CLI for S3

For S3, AWS CLI provides two tiers of commands:

  • The first tier, s3, deals with frequently used commands: creating, deleting and handling objects and buckets.
  • The second tier, s3api, provides access to more advanced S3 operations. Basically, with s3api you have complete access to S3 resources.

We will discuss only the first tier. Right now, there is one bucket already created using the AWS Console. The bucket contains two files (file1.txt and file2.txt) and one folder with one file (file3.txt) in it.

Let’s check this using AWS CLI, and then remove one file. Afterwards, we will create another S3 bucket.

[ec2-user@EC2-REDHAT-01 ~]$ aws s3 ls
2014-09-05 09:57:31 s3_bucket_01
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 ls s3://s3_bucket_01
                           PRE folder01/
2014-09-05 16:02:00         21 file1.txt
2014-09-05 16:02:01         21 file2.txt
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 ls s3://s3_bucket_01/folder01/
2014-09-05 16:01:22          0
2014-09-05 16:02:14         21 file3.txt
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 rm s3://s3_bucket_01/file1.txt
delete: s3://s3_bucket_01/file1.txt
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 ls s3://s3_bucket_01
                           PRE folder01/
2014-09-05 16:02:01         21 file2.txt
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 mb s3://s3_bucket_test_01
make_bucket: s3://s3_bucket_test_01/
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 ls
2014-09-05 09:57:31 s3_bucket_01
2014-09-05 16:12:37 s3_bucket_test_01
[ec2-user@EC2-REDHAT-01 ~]$

Checking the status of the buckets in AWS console, we will notice that one file is deleted:

And there is another bucket created:

Now let’s copy a local file to one of the S3 buckets. We have this file in the current directory and we will copy it into s3_bucket_test_01 bucket:

[ec2-user@EC2-REDHAT-01 ~]$ ls -l | grep TEST_06.txt
-rw-rw-r--. 1 ec2-user ec2-user      12 Sep  5 16:15 TEST_06.txt
[ec2-user@EC2-REDHAT-01 ~]$ cat TEST_06.txt
TEST_06.txt
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 cp TEST_06.txt s3://s3_bucket_test_01 --grants full=uri=http://acs.amazonaws.com/groups/global/AllUsers
upload: ./TEST_06.txt to s3://s3_bucket_test_01/TEST_06.txt
[ec2-user@EC2-REDHAT-01 ~]$ aws s3 ls s3://s3_bucket_test_01
2014-09-05 16:21:06         12 TEST_06.txt
[ec2-user@EC2-REDHAT-01 ~]$

I used ‘–grant’ to specify the permissions (full), the grantee_type (uri) and the grantee_id (to everyone). As you can see below, the file is accessible by using this link, https://s3.amazonaws.com/s3_bucket_test_01/TEST_06.txt

Let’s now copy a file from an S3 bucket to a local folder. Let’s copy file2.txt from s3_bucket_01 to a local device:

[ec2-user@EC2-REDHAT-01 ~]$ aws s3 cp s3://s3_bucket_01/file2.txt .
download: s3://s3_bucket_01/file2.txt to ./file2.txt
[ec2-user@EC2-REDHAT-01 ~]$ ls -lat | grep file2.txt
-rw-rw-r--. 1 ec2-user ec2-user      21 Sep  5 16:02 file2.txt
[ec2-user@EC2-REDHAT-01 ~]$ 

You can use the command ‘aws s3 sync’ to synchronize the source-target. The possible combinations are:

  • * Local file system to S3 bucket
  • * S3 bucket to local file system
  • * S3 bucket to S3 bucket

Let’s synchronize the S3 bucket s3_bucket_01 to our local file system. Remember that this bucket has one folder and one file:

[ec2-user@EC2-REDHAT-01 test_s3]$ pwd
/home/ec2-user/test_s3
[ec2-user@EC2-REDHAT-01 test_s3]$ aws s3 sync s3://s3_bucket_01 .
download: s3://s3_bucket_01/folder01/file3.txt to folder01/file3.txt
download: s3://s3_bucket_01/file2.txt to ./file2.txt
[ec2-user@EC2-REDHAT-01 test_s3]$ ls -lR
.:
total 4
-rw-rw-r--. 1 ec2-user ec2-user 21 Sep  5 16:02 file2.txt
drwxrwxr-x. 2 ec2-user ec2-user 22 Sep  5 16:29 folder01

./folder01:
total 4
-rw-rw-r--. 1 ec2-user ec2-user 21 Sep  5 16:02 file3.txt
[ec2-user@EC2-REDHAT-01 test_s3]$

The sync operation allows you to use ‘—delete’ option to delete the files or folders from the target not present in the source. Or you can use ‘–include’ or ‘–exclude’ to filter out what files will be synchronized.

Tier two of AWS CLI for S3 provides more commands for fine control of your S3 buckets.

Using AWS CLI for EC2

Now it’s time to use AWS CLI with the EC2 service. As you can see below, I have several EC2 instances running. We will terminate one of them as a test later. But for now, let’s create a new instance.

For this new instance, we will create a new key-pair and use a security group that already allows SSH from any IP address.

Let’s create the key-pair:

[ec2-user@EC2-REDHAT-01 ~]$ aws ec2 create-key-pair --key-name TEST
{
    "KeyMaterial": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAx+LyKwLY6HcddoG4AB9x/ZPbMZS31s2UCFsbYRhJnRln5BXF1X9mBpJVTNCI\nXS6E5z6aaKbm8qOwCXjpK6TUIz0k1uZ+sWt9IRZTML4SPm01q9bUlYU471oq4GmZxvtgpI0fjjBy\nxnKxO9L2Gu5Qr6MaIVmODZyeeQF4o6/UWAZ5xEJaiEVFknpOusoOYyldcdDTdbPygh4oJ2l4yL+E\no8h4P2jcxM1jstytkvT3/FfCX4jwkGhnFuzUwbeoE7yqP+5kKkeiiuo9Xynzyg6PXTke5KkyM9J9\nS5NcIOoT5OdkYDEst75m8BP+lvSVXLpeqq/Ah1NecsAM7pk6RmTfhwIDAQABAoIBADz09skrGx75\nhGCpeoV/EXI1bvkV02KMWwKZViYsE/b20nn6MdTD/0ctxRmh1pQkJZ+ntcsIaxG/3WVmHCWVJKSC\ndwOaUzSkBQrcFg6dSE8FWwLGhxn6ALztkHnTenP2ISjNaLwiBj6Cp44I7ciUuMJ3SPa2qt13z9Qz\nIKk6a9VTybv52RYi5HI2oHsPAFffGv0ZAxMTAQUTKWvHW5fmOejyo5Nt7UXz1pAgI6yBIwUeJt97\nUi3bagiZkrNONLjvK5kUUaCxpWvMICtb4AZYOlUHYA+7RhT9Jodw54uTVrWpcXcsAx1l2HqrnXPn\n0O7sj8aBTLVud48YO0o3LvPlunECgYEA9ss3WOTKVVRgMuA224CyvAO5MYn3utjKY8OkjyKIkTG0\nmK9w5MAcDDue5pOwslHoFUPdj2eual9vcsiwlHKindLFo5fp/QPEBl5M+8XC4QEd4zKQcTh8Qfrf\nRxCPeXWhKVeCGBrnjN3lcl5/Bi/Q95H4cM9jpO2CdHkZH1uGAXMCgYEAz1fIkMQks3ZynCB3pDuY\nYk4LVWyx5Gby5j6c+30mw0tjNI/SQA64ghGhsJlMA8x+OjDTzxieN/8Y0M9wKTx508GfYTwEXdso\nAi+QZ5sNzhwkY/Pxe+MJBoJVJ7uzcU4Y31bm8xYaLWZIx2dzra1KJxbbmLHTBRlXLtSdHYRDFJ0C\ngYEAlwoGBIx5XEG4YkqgqDQar0CkHbJvf9Xf/7CjzMn4xMuj2HI+6HC+eaofdMRctrW0EReoyHRO\n5WElBx3WG/QQnAB73sRBewzL8o3SFaCRxTz2uq6egfxs58CO16JF/LLGAFRDFVeZ8Ld5DkbQ/MtU\n279lSj1SoYSOe5QFkRBlcgsCgYA5r0l8hSVUzHpcV2gRyAbAKshabtAbJE0lo3vXN91l7f5erDip\nW68XeMJMOIn+cJEZrWXwMwApwg3sYhnpTwrxz+eAjMsGNaeu2/7H5Ep18jxpln+zoNls9yurzzvA\nJSMkV7APJzIHOXK8D1sOkayUrXZQyABWZdNskvhXMunavQKBgFOdOaliAgu/XKLKmjynyiM+wakM\n6THMhr3UcDsccvsdzQIOCXN7WwvvIIeVy/qRBjv1aVn9BczqzgWaXdDwDiXwUfumqZ+3jNI7xniZ\n4RGjYvXYL+8shd6+Ra/1NzIp8ktkZ7u8N4/VoJad82y85oQ2Dl0cmnphpBmA/K2qb76L\n-----END RSA PRIVATE KEY-----",
    "KeyName": "TEST",
    "KeyFingerprint": "5b:38:4d:af:4e:21:df:f7:f7:b7:06:35:1b:8f:1f:07:e6:a7:f1:52"
}
[ec2-user@EC2-REDHAT-01 ~]$

Create a new file called TEST.pem and copy the above content. Make sure that you only copy from ‘—–BEGIN RSA PRIVATE KEY—–‘ to ‘—–END RSA PRIVATE KEY—–‘, inclusively. Then add read permissions only to the owner:

[ec2-user@EC2-REDHAT-01 ~]$ vi TEST.pem
[ec2-user@EC2-REDHAT-01 ~]$ chmod 400 TEST.pem
[ec2-user@EC2-REDHAT-01 ~]$

You can view the key-pair:

[ec2-user@EC2-REDHAT-01 ~]$ aws ec2 describe-key-pairs --key-name TEST
{
    "KeyPairs": [
        {
            "KeyName": "TEST",
            "KeyFingerprint": "5b:38:4d:af:4e:21:df:f7:f7:b7:06:35:1b:8f:1f:07:e6:a7:f1:52"
        }
    ]
}
[ec2-user@EC2-REDHAT-01 ~]$

As for security the group, we will use SEC_GRP:

[ec2-user@EC2-REDHAT-01 ~]$ aws ec2 describe-security-groups --group-names SEC_GRP
{
    "SecurityGroups": [
        {
            "IpPermissionsEgress": [
                {
                    "IpProtocol": "-1",
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ],
                    "UserIdGroupPairs": []
                }
            ],
            "Description": "SEC_GRP",
            "IpPermissions": [
                {
                    "ToPort": 22,
                    "IpProtocol": "tcp",
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ],
                    "UserIdGroupPairs": [],
                    "FromPort": 22
                }
            ],
            "GroupName": "SEC_GRP",
            "VpcId": "vpc-6e76a30b",
            "OwnerId": "061167321728",
            "GroupId": "sg-11d78c74"
        }
    ]
}
[ec2-user@EC2-REDHAT-01 ~]$

Now it’s time to create another EC2 instance. On CLI, we need to specify the AMI image used, the instance type, the number of instances, the key-pair and the security group.

  • AMI image: ami-76817c1e
  • Instance type: t2.micro
  • Count: 1
  • Key-pair: TEST
  • Security Group: SEC_GRP

So, the command and the output will be:

[ec2-user@EC2-REDHAT-01 ~]$ aws ec2 run-instances --image-id ami-76817c1e --count 1 --instance-type t2.micro --key-name TEST --security-groups SEC_GRP
{
    "OwnerId": "XXXXXXXXXXXX",
    "ReservationId": "r-a063f18b",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": null,
            "RootDeviceType": "ebs",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2014-09-05T21:32:49.000Z",
            "PrivateIpAddress": "172.31.26.125",
            "ProductCodes": [],
            "VpcId": "vpc-6e76a30b",
            "StateTransitionReason": null,
            "InstanceId": "i-29f726c2",
            "ImageId": "ami-76817c1e",
            "PrivateDnsName": "ip-172-31-26-125.ec2.internal",
            "KeyName": "TEST",
            "SecurityGroups": [
                {
                    "GroupName": "SEC_GRP",
                    "GroupId": "sg-11d78c74"
                }
            ],
            "ClientToken": null,
            "SubnetId": "subnet-e9db339e",
            "InstanceType": "t2.micro",
            "NetworkInterfaces": [
                {
                    "Status": "in-use",
                    "SourceDestCheck": true,
                    "VpcId": "vpc-6e76a30b",
                    "Description": null,
                    "NetworkInterfaceId": "eni-ea62cb9c",
                    "PrivateIpAddresses": [
                        {
                            "PrivateDnsName": "ip-172-31-26-125.ec2.internal",
                            "Primary": true,
                            "PrivateIpAddress": "172.31.26.125"
                        }
                    ],
                    "PrivateDnsName": "ip-172-31-26-125.ec2.internal",
                    "Attachment": {
                        "Status": "attaching",
                        "DeviceIndex": 0,
                        "DeleteOnTermination": true,
                        "AttachmentId": "eni-attach-b0e379d3",
                        "AttachTime": "2014-09-05T21:32:49.000Z"
                    },
                    "Groups": [
                        {
                            "GroupName": "SEC_GRP",
                            "GroupId": "sg-11d78c74"
                        }
                    ],
                    "SubnetId": "subnet-e9db339e",
                    "OwnerId": "XXXXXXXXXXXX",
                    "PrivateIpAddress": "172.31.26.125"
                }
            ],
            "SourceDestCheck": true,
            "Placement": {
                "Tenancy": "default",
                "GroupName": null,
                "AvailabilityZone": "us-east-1a"
            },
            "Hypervisor": "xen",
            "BlockDeviceMappings": [],
            "Architecture": "x86_64",
            "StateReason": {
                "Message": "pending",
                "Code": "pending"
            },
            "RootDeviceName": "/dev/xvda",
            "VirtualizationType": "hvm",
            "AmiLaunchIndex": 0
        }
    ]
}
[ec2-user@EC2-REDHAT-01 ~]$

If I check the EC2 console, I will see that a new EC instance is initializing, the one without a tag as you cannot set a tag from CLI when you start a new instance:

However, you can add a tag to the EC2 instance like so:

[ec2-user@EC2-REDHAT-01 ~]$ aws ec2 create-tags --resources i-29f726c2 --tags Key=Name,Value=EC2_INSTANCE_FROM_AWS_CLI
{
    "return": "true"
}
[ec2-user@EC2-REDHAT-01 ~]$

Now, let’s terminate one EC2 instance. From the screenshot, it’s the one that is selected:

[ec2-user@EC2-REDHAT-01 ~]$ aws ec2 terminate-instances --instance-ids i-da73a431
{
    "TerminatingInstances": [
        {
            "InstanceId": "i-da73a431",
            "CurrentState": {
                "Code": 32,
                "Name": "shutting-down"
            },
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}
[ec2-user@EC2-REDHAT-01 ~]$

Let’s see if this change is reflected on the EC2 console (the bottom EC2 instance is shutting down). Also, note that the tag has been added to the instance that was started from AWS CLI:

We have now reached the end of the article and of the series.

In this article, I showed you how you can use AWS CLI to interact with S3 and EC2 services from Amazon AWS. We learned how to do the following things in AWS CLI:

  • Create an S3 bucket
  • Delete objects from the S3 bucket
  • Copy files between the S3 bucket and local file system
  • Sync the S3 bucket and local file system
  • Create EC2 key-pairs
  • Spawn a new EC2 instance
  • Terminate an EC2 instance

References