How to Mount S3 bucket on EC2 Linux Instance

-

Step-1:- If you are using a new centos or ubuntu instance. Update the system.

-> For CentOS or Red Hat

1
yum update all

-> For Ubuntu

1
apt-get update

 

Step-2:- Install the dependencies.

-> In CentOS or Red Hat

1
sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

 

In Ubuntu or Debian

1
sudo apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

 

Step-3:- Clone s3fs source code from git.

 

Step-4:- Now change to source code  directory, and compile and install the code with the following commands:

1
2
3
4
5
cd s3fs-fuse
./autogen.sh
./configure --prefix=/usr --with-openssl
make
sudo make install

 

Step-5:- Use below command to check where s3fs command is placed in O.S. It will also tell you the installation is ok.

1
which s3fs

 

Step-6:- Getting the access key and secret key.

You will need AWS Access key and Secret key with appropriate permissions in order to access your s3 bucket from your EC2 instance. You can easily manage your user permissions from IAM (Identity and Access Management) Service provided by AWS. Create an IAM user with S3 full access(or with a role with sufficient permissions) or use root credentials of your Account. Here we will use the root credentials for simplicity.

Go to AWS Menu -> Your AWS Account Name -> My Security Credentials. Here your IAM console will appear. You have to go to Users > Your Account name and under permissions Tab, check whether you have sufficient access on S3 bucket. If not, you can manually assign an existing  "S3 Full-Access" policy or create a new policy with sufficient permissions.

Now go to Security Credentials Tab and Create Access Key. A new Access Key and Secret Key pair will be generated. Here you can see access key and secret key (secret key is visible when you click on show tab) which you can also download. Copy these both keys separately.

Note that you can always use an existing access and secret key pair. Alternatively, you can also create a new IAM user and assign it sufficient permissions to generate the access and secret key.

 

Step-7 :- Create a new file in /etc with the name passwd-s3fs and Paste the access key and secret key in the below format .

1
2
touch /etc/passwd-s3fs
vim /etc/passwd-s3fs

1
Your_accesskey:Your_secretkey

 

Step-8:- change the permission of file

1
sudo chmod 640 /etc/passwd-s3fs

 

Step-9:- Now create a directory or provide the path of an existing directory and mount S3bucket in it. Here, Provide your S3 bucket name in place of "your_bucketname".

1
2
mkdir /mys3bucket
s3fs your_bucketname -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 /mys3bucket

where, "your_bucketname" = the name of your S3 bucket that you have created on AWS S3, use_cache = to use a directory for its cache purpose, allow_other= to allow other users to write to the mount-point, uid= uid of the user/owner of the mountpoint (can also add "-o gid=1001" for group), mp_umask= to remove other users permission. multireq_max= parameter to send request to s3 bucket, /mys3bucket= mountpoint where the bucket will be mounted.

You can unmount it later by simply using the below command

1
umount /mys3bucket

You can make an entry in /etc/rc.local to automatically remount after reboot.  Find the s3fs binary file by "which" command and make the entry before the "exit 0" line as below.

1
2
which s3fs
/usr/local/bin/s3fs

1
nano /etc/rc.local

1
/usr/local/bin/s3fs your_bucketname -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 /mys3bucket

 

Step-10:- Check mounted s3 bucket. Output will be similar as shown below but Used size may differ.

1
df -Th

"or"

1
df -Th /mys3bucket

1
2
Filesystem Type Size Used Avail Use% Mounted on
s3fs  fuse.s3fs 256T  0   256T   0%  /mys3bucket

If it shows the mounted file system, you have successfully mounted the S3 bucket on your EC2 Instance. You can also test it further by creating a test file.

1
2
3
4
cd /mys3bucket
echo "this is a test file to check s3fs" >> test.txt
ls
 

This change should also reflect on S3 bucket. So Login to your S3 bucket to verify if the test file is present or not.

Popular posts from this blog

Shrink you container size up to 95%.

-
BLAFS is a bloat-aware filesystem for container debloating. The design principles of BLAFS are effective, efficient, and easy to use. It detects the files used by the container, and then debloats the container by removing the unused files. The debloated containers are still functional and can run the same workload as the original containers, but with a much smaller size and faster deployment. https://github.com/negativa-ai/BLAFS?tab=readme-ov-file
Read more

alma linux: dnf Module yaml error: Unexpected key in data

-
While applying recent updates to a server running Rocky Linux 8, an error presented itself. Module yaml error: Unexpected key in data: static_context [line 9 col 3] Module yaml error: Unexpected key in data: static_context [line 9 col 3] The solution turned out to be a simple one. Update the libmodulemd first to correct the problem then perform the dnf updates as usual. dnf update libmodulemd For a detailed explanation as to the cause of this problem.
Read more