In this article we will see how you can setup a FTP server on an EC2 instance that uploads/downloads the data directly from an Amazon S3 bucket. You can find out more about S3 buckets here: Amazon AWS – Understanding EC2 storage – Part III. To be able to do this we will need the s3fs utility. s3fs is a FUSE filesystem that allows us to mount with read/write access an Amazon S3 bucket as a local filesystem. The files are stored in S3 and other applications can access the files. The maximum file size is 64GB.

VMware Training – Resources (Intense)

The steps to enable a FTP server where the files are stored in an Amazon S3 bucket are:

  • Install the software dependencies
  • Download s3fs software
  • Install s3fs software
  • Specify the security credentials
  • Mount the S3 bucket as local filesystem
  • Install the FTP server and configure the users and server settings

So let’s see how taking each of the above steps in detail does this.

First we need to have the S3 bucket created. I have one created. The exact name of the bucket is needed, as it will be later referenced:

Also, I have one EC2 instance to which the S3 bucket will be attached:

There is one thing that should not be forgotten. You must allow inbound traffic for TCP ports 20-21 so that the FTP clients can connect to the FTP server. This security group attached to the EC2 instance should do the job:

Connect to the EC2 instance using the public IP allocated:

lab@UBUNTU:~/AWS$ ssh -i AMAZON_LINUX.pem ec2-user@54.172.169.221
The authenticity of host '54.172.169.221 (54.172.169.221)' can't be established.
ECDSA key fingerprint is 2b:fc:26:c4:be:d0:e3:ed:11:4b:67:e3:95:b4:5b:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.172.169.221' (ECDSA) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/
25 package(s) needed for security, out of 54 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-11-50 ~]$

Login as root as we will execute multiple commands as root:

[ec2-user@ip-172-31-11-50 ~]$ sudo su
[root@ip-172-31-11-50 ec2-user]#

Install the packages that we are going to need to use fuse:

[root@ip-172-31-11-50 ec2-user]# yum install -y gcc libstdc++-devel gcc-c++ fuse fuse-devel curl-devel libxml2-devel mailcap automake openssl-devel
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
===== output cut for brevity =====
Dependency Updated:
  curl.x86_64 0:7.40.0-3.51.amzn1     glibc.x86_64 0:2.17-55.143.amzn1     glibc-common.x86_64 0:2.17-55.143.amzn1     libcurl.x86_64 0:7.40.0-3.51.amzn1

Complete!
[root@ip-172-31-11-50 ec2-user]#

Download the software:

[root@ip-172-31-11-50 ec2-user]# wget http://s3fs.googlecode.com/files/s3fs-1.74.tar.gz
--2015-07-07 10:26:01--  http://s3fs.googlecode.com/files/s3fs-1.74.tar.gz
Resolving s3fs.googlecode.com (s3fs.googlecode.com)... 74.125.22.82, 2607:f8b0:400d:c03::52
Connecting to s3fs.googlecode.com (s3fs.googlecode.com)|74.125.22.82|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 199120 (194K) [application/x-gzip]
Saving to: ‘s3fs-1.74.tar.gz’

s3fs-1.74.tar.gz                        100%[==============================================================================>] 194.45K   230KB/s   in 0.8s

2015-07-07 10:26:02 (230 KB/s) - ‘s3fs-1.74.tar.gz’ saved [199120/199120]

[root@ip-172-31-11-50 ec2-user]#

Decompress the archive to be ready for installation and move to the folder:

[root@ip-172-31-11-50 ec2-user]# tar zxvf s3fs-1.74.tar.gz
s3fs-1.74/
s3fs-1.74/COPYING
===== output cut for brevity =====
s3fs-1.74/missing
s3fs-1.74/configure
[root@ip-172-31-11-50 ec2-user]# cd s3fs-1.74
[root@ip-172-31-11-50 s3fs-1.74]#

Start the installation process by running “./configure”:

[root@ip-172-31-11-50 s3fs-1.74]# ./configure --prefix=/opt
checking build system type... x86_64-unknown-linux-gnu
===== output cut for brevity =====
config.status: creating doc/Makefile
config.status: executing depfiles commands
[root@ip-172-31-11-50 s3fs-1.74]#

Next compile the programs and create the executable by running the “make” command:

[root@ip-172-31-11-50 s3fs-1.74]# make
Making all in src
===== output cut for brevity =====
make[1]: Nothing to be done for `all-am'.
make[1]: Leaving directory `/home/ec2-user/s3fs-1.74'
[root@ip-172-31-11-50 s3fs-1.74]#

Finish the installation where the executable and other required files created in the previous step are copied to the required final directories:

[root@ip-172-31-11-50 s3fs-1.74]# make install
Making install in src
make[1]: Entering directory `/home/ec2-user/s3fs-1.74/src'
===== output cut for brevity =====
make[2]: Leaving directory `/home/ec2-user/s3fs-1.74'
make[1]: Leaving directory `/home/ec2-user/s3fs-1.74'
[root@ip-172-31-11-50 s3fs-1.74]#

Next configure the AWS credentials. We will create the file that will be our s3f3 password file, configure the proper permissions and then put in the password file the AWS credentials. The s3fs password file has this format: accessKeyId:secretAccessKey:

[root@ip-172-31-11-50 s3fs-1.74]# touch /etc/passwd-s3fs && chmod 640 /etc/passwd-s3fs && echo 'AKIAJOORF34SB3HJEVRA:hJC7B2d6s6DehFylyk8JpGFKn62JTNcXz59rvPpH' > /etc/passwd-s3fs
[root@ip-172-31-11-50 s3fs-1.74]#

Next create a folder where the S3 bucket will be mounted:

[root@ip-172-31-11-50 ec2-user]# pwd
/home/ec2-user
[root@ip-172-31-11-50 ec2-user]# mkdir mnt_s3_bucket
[root@ip-172-31-11-50 ec2-user]# cd mnt_s3_bucket/
[root@ip-172-31-11-50 mnt_s3_bucket]# pwd
/home/ec2-user/mnt_s3_bucket
[root@ip-172-31-11-50 mnt_s3_bucket]#

And mount the S3 bucket:

[root@ip-172-31-11-50 mnt_s3_bucket]# /opt/bin/s3fs s3-bucket-test-ftp /home/ec2-user/mnt_s3_bucket -o allow_other
[root@ip-172-31-11-50 mnt_s3_bucket]#

Let’s confirm that the S3 bucket was mounted:

[root@ip-172-31-11-50 mnt_s3_bucket]# mount | grep s3fs
s3fs on /home/ec2-user/mnt_s3_bucket type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
[root@ip-172-31-11-50 mnt_s3_bucket]#

Install the FTP server:

[root@ip-172-31-11-50 mnt_s3_bucket]# yum install vsftpd
Loaded plugins: priorities, update-motd, upgrade-helper
===== output cut for brevity =====
Installed:
  vsftpd.x86_64 0:2.2.2-13.13.amzn1

Complete!
[root@ip-172-31-11-50 mnt_s3_bucket]

Add a user and set the home directory(in this case, the home directory will be the directory where the S3 bucket will be mounted). Disable the login ability for the new user added. The user will still be able to login to FTP, but it will not be able to login to the shell of the EC2 instance where the FTP server is running:

[root@ip-172-31-11-50 mnt_s3_bucket]# useradd -d /home/ec2-user/mnt_s3_bucket -s /sbin/nologin lab
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
[root@ip-172-31-11-50 mnt_s3_bucket]#

Add a password for the user:

[root@ip-172-31-11-50 mnt_s3_bucket]# passwd lab
Changing password for user lab.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@ip-172-31-11-50 mnt_s3_bucket]#

Configure the FTP server by editing the file and adding this content:

[root@ip-172-31-11-50 mnt_s3_bucket]# cat /etc/vsftpd/vsftpd.conf
listen=YES
anonymous_enable=NO
local_enable=YES
write_enable=YES
dirmessage_enable=YES
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
chroot_local_user=YES
secure_chroot_dir=/var/run/vsftpd/empty
[root@ip-172-31-11-50 mnt_s3_bucket]#

Start the FTP server:

[root@ip-172-31-11-50 mnt_s3_bucket]# /etc/init.d/vsftpd start
Starting vsftpd for vsftpd:                                [  OK  ]
[root@ip-172-31-11-50 mnt_s3_bucket]#

Now let’s connect to the FTP server from a server from the Internet and upload a file there:

lab@UBUNTU:~/AWS$ ftp 54.172.169.221
Connected to 54.172.169.221.
220 (vsFTPd 2.2.2)
Name (54.172.169.221:lab): lab
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> put remote_file
local: remote_file remote: remote_file
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 Transfer complete.
35 bytes sent in 0.00 secs (794.9 kB/s)
ftp> quit
221 Goodbye.
lab@UBUNTU:~/AWS$

The file should now show up in the S3 bucket:

Now, let’s unmount the S3 bucket and confirm that no file can be uploaded to the FTP server:

[root@ip-172-31-11-50 ec2-user]# umount mnt_s3_bucket
[root@ip-172-31-11-50 ec2-user]#

The file cannot be uploaded:

lab@UBUNTU:~/AWS$ ftp 54.172.169.221
Connected to 54.172.169.221.
220 (vsFTPd 2.2.2)
Name (54.172.169.221:lab): lab
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> put file1
local: file1 remote: file1
200 PORT command successful. Consider using PASV.
553 Could not create file.
ftp>

So that would be all. Now you should know how to mount an Amazon S3 bucket as a local filesystem to an EC2 instance. The advantages of using S3 buckets as storage are that S3 storage is pretty inexpensive and it is designed for 99.999999999% durability of the objects.

Reference