Ec2 Download File From S3

Posted on by admin

Choose the Versions tab and then from the Actions menu choose Download or Download as if you want to download the object to a specific folder. Java When you download an object through the AWS SDK for Java, Amazon S3 returns all of the object's metadata and an input stream from which to read the object's contents. The following is a list of the commands you can use to transfer your files. // Download a directory recursive to EC2 $ aws s3 cp s3://yourbucket/dir localdir -recursive // Sync a full directory.

A step-by-step tutorial to create Amazon Elastic Compute Cloud (EC2), linking it with Amazon Simple Storage Service (S3) and connecting to AWS Cloud9 integrated development environment (IDE)

Jun 23, 2020·7 min read

Sometimes, you can be blocked from completing a task just because of your desktop capabilities. As a data scientist for example, most of the time it is almost impossible to load a huge dataset to your personal desktop. Even if you were able to load this data, training your model might take forever. One possible solution is to use Amazon Web Services (AWS). You can read more about AWS from here.

Using Amazon Elastic Compute Cloud (Amazon EC2) will help solve the previously stated problems. It is scalable and eliminates the need to invest in hardware up front. You can read more about Amazon EC2 from here.

There is also Amazon Simple Storage Service (Amazon S3) that can be used to store and retrieve any amount of data at anytime. You can read more about Amazon S3 from here.

To make the communication with the Amazon EC2 easier, you can use AWS Cloud9. It is a cloud-based integrated development environment (IDE) that lets you write, run and debug your code using the browser. You can read more about AWS Cloud9 from here.

In this article, we will:

Access key and secret key are used to authenticate requests you make to AWS.

After creating an AWS account from here, log in to AWS Management Console from here.

First, we need to create an Access Key and Secret Key. In Security, Identity, & Compliance pick IAM (Identity and Access Management).

Press Users then choose your user name. If you cannot find your username, press Add user.

After pressing your user name, inside the Security credentials tab press Create Access Key. Then download .csv file. You will find your Access Key & Secret Key inside. Keep them stored in a safe place as you will need them later.

Go back to AWS Management Console, from compute pick EC2.

Press Launch Instance.

Choose whatever suits you from the options shown. For this tutorial I am going for Ubuntu Server 20.04 LTS. Press Select

Choose the instance you need. For this tutorial I am going for t2.micro. For more information about the Amazon EC2 Instance Types, you can press here.

From the tabs above, choose 4.Add Storage. You can increase the size if needed.

Ec2 download file from s3 python

Choose 6.Configure Security Group. Change Assign a security group to Select an existing security group. Choose the group you want. The policy should be created already by your admin or you can create it. If you do not choose a security group, your machine will be public and can be accessed by anyone. Feel free to play around with any other configurations if needed. Finally, press Review and Launch then Launch.

Change choose an existing key pair to Create a new key pair. Then add a name for this key pair and press download key pair. This will download a pem file. Then press Launch Instances. Finally, Press View Instances.

You should find your machine in there without a name. Rename it to whatever you want then Click on it to choose it. Copy the Public DNS(IPv4) as we will need it in the next step.

For the next step you need to install PuTTY. PuTTY is a terminal emulation software that runs on Windows, Unix and Linux.

Open PuTTYgen. Choose Load. Go to the path where you saved the pem file (Was saved previously in step 2). Make sure to set the type of files to All Files(*.*) to see the pem file. Double click that pem file.

You should get a message that you successfully imported foreign key. Press ok then save private key.

Close PuTTYgen and open PuTTY.

In the Hostname write [email protected]<Paste the Public DNS(IPv4)>

Expand the SSH then press on the AUTH. Click Browse and go to the PPK you saved before then double click on it.

Download

Go back to Session then add a name in the “Saved Sessions” then press Save. This way you saved your configuration to PuTTY. The next time you want to connect to the machine, you will only press load next time.

Ec2 Download File From S3

Finally press “Open

Now you are connected to the machine through the terminal and can write any linux commands you want.

You should start by the following 2 commands to update the environment.

To interact with AWS S3 through the machine, you can install awscli through the following command

aws configure” will ask you to enter the access key and secret key that you generate before in the IAM (From Step 1). For the other 2 prompts, just press enter (No need to fill them).

To copy something from aws to your machine is like using cp command in linux. Example:

The source and destination can be either a local path on the machine, or path on s3.

For s3 path, you just need to add s3:// at the beginning.

Example to copy a file from S3 to local machine:

Keep PuTTY connected to the instance and the terminal running for the next step.

From AWS Management Console, inside the Developer Tools choose Cloud9.

Ec2 Download File From S3

Press Create environment. Add any name for that environment then press next.

Ec2 Download File From S3

In the configure settings, change the Environment type to Connect and run in remote server (SSH). In the user, write ubuntu. Add the instance’s public DNS IPv4. Then press Copy Key to Clipboard.

In PuTTY terminal that is connected to the machine, write the following commands to save the key in the machine:

Cloud9 needs nodejs that is why we installed it.

Press Next Step in Configure Settings then Create Environment.

Right click on “C9 install” and copy link address.

Press Next and leave everything ticked.

Go to PuTTY terminal and run the following commands:

Now you can close PuTTY and continue from Cloud9.

Now you are ready to start development on this IDE. You can upload files or data from your local machine to the EC2 using Cloud9 just by drag and drop. You can also run/debug the code or use terminal below to interact with the machine.

Note: If the instance’s IP address changed (Happens when you stop an Instance then restart its running), you will just simply need to copy the new IPv4 address.

In Cloud9, on the top left corner, press the Cloud9 icon.

Click on “Go to Your Dashboard”. Press the desired environment and from the top right corner then choose Edit. Then Scroll down to the Host and just change the IPv4 address.

Download From S3 To Ec2

This tutorial provided a step-by-step guide to launch and connect AWS EC2 to AWS Cloud9, along with moving data from or to AWS S3. This should help with setting a basic data science environment.

Ec2 Download File From S3 Pc

I recommend AWS Documentation for more information about all the services that amazon provides.