238 lines
6.2 KiB
Markdown
238 lines
6.2 KiB
Markdown
How to access private object storage using S3cmd or boto3 on 3Engines Cloud[🔗](#how-to-access-private-object-storage-using-s3cmd-or-boto3-on-brand-name "Permalink to this headline")
|
||
=======================================================================================================================================================================================
|
||
|
||
LEGACY ARTICLE
|
||
|
||
This article is marked as a legacy document and may not reflect the latest information.
|
||
Please refer to the following articles:
|
||
|
||
[How to access object storage from 3Engines Cloud using boto3](How-to-access-object-storage-from-3Engines-Cloud-using-boto3.html.md)
|
||
|
||
[How to access object storage from 3Engines Cloud using s3cmd](How-to-access-object-storage-from-3Engines-Cloud-using-s3cmd.html.md)
|
||
|
||
**Introduction**
|
||
|
||
Private object storage (buckets within user’s project) can be used in various ways. For example, to access files located in object storage, buckets can be mounted and used as a file system using **s3fs**. Other tools which can be used to achieve better performance are **S3cmd** (command line tool) and **boto3** (AWS SDK for Python).
|
||
|
||
**S3cmd**
|
||
|
||
In order to acquire access to Object Storage buckets via S3cmd, first you have to generate your own EC2 credentials with this tutorial [How to generate and manage EC2 credentials on 3Engines Cloud](../cloud/How-to-generate-ec2-credentials-on-3Engines-Cloud.html.md).
|
||
|
||
Once EC2 credentials are generated, ensure that your instance or local machine is equipped with S3cmd:
|
||
|
||
```
|
||
s3cmd --version
|
||
|
||
```
|
||
|
||
If not, S3cmd can be installed with:
|
||
|
||
```
|
||
apt install s3cmd
|
||
|
||
```
|
||
|
||
Now S3cmd can be configured with the following command:
|
||
|
||
```
|
||
s3cmd --configure
|
||
|
||
```
|
||
|
||
Input and confirm (by pressing Enter) the following values:
|
||
|
||
WAW4-1WAW3-1WAW3-2FRA1-2
|
||
|
||
```
|
||
New settings:
|
||
Access Key: (your EC2 credentials)
|
||
Secret Key: (your EC2 credentials)
|
||
Default Region: default
|
||
S3 Endpoint: s3.waw4-1.3Engines.com
|
||
DNS-style bucket+hostname:port template for accessing a bucket: s3.waw4-1.3Engines.com
|
||
Encryption password: (your password)
|
||
Path to GPG program: /usr/bin/gpg
|
||
Use HTTPS protocol: Yes
|
||
HTTP Proxy server name:
|
||
HTTP Proxy server port: 0
|
||
|
||
```
|
||
|
||
```
|
||
New settings:
|
||
Access Key: (your EC2 credentials)
|
||
Secret Key: (your EC2 credentials)
|
||
Default Region: waw3-1
|
||
S3 Endpoint: s3.waw3-1.3Engines.com
|
||
DNS-style bucket+hostname:port template for accessing a bucket: s3.waw3-1.3Engines.com
|
||
Encryption password: (your password)
|
||
Path to GPG program: /usr/bin/gpg
|
||
Use HTTPS protocol: Yes
|
||
HTTP Proxy server name:
|
||
HTTP Proxy server port: 0
|
||
|
||
```
|
||
|
||
```
|
||
New settings:
|
||
Access Key: (your EC2 credentials)
|
||
Secret Key: (your EC2 credentials)
|
||
Default Region: default
|
||
S3 Endpoint: s3.waw3-2.3Engines.com
|
||
DNS-style bucket+hostname:port template for accessing a bucket: s3.waw3-2.3Engines.com
|
||
Encryption password: (your password)
|
||
Path to GPG program: /usr/bin/gpg
|
||
Use HTTPS protocol: Yes
|
||
HTTP Proxy server name:
|
||
HTTP Proxy server port: 0
|
||
|
||
```
|
||
|
||
```
|
||
New settings:
|
||
Access Key: (your EC2 credentials)
|
||
Secret Key: (your EC2 credentials)
|
||
Default Region: default
|
||
S3 Endpoint: s3.fra1-2.3Engines.com
|
||
DNS-style bucket+hostname:port template for accessing a bucket: s3.fra1-2.3Engines.co
|
||
Encryption password: (your password)
|
||
Path to GPG program: /usr/bin/gpg
|
||
Use HTTPS protocol: Yes
|
||
HTTP Proxy server name:
|
||
HTTP Proxy server port: 0
|
||
|
||
```
|
||
|
||
After this operation, you should be allowed to list and access your Object Storage.
|
||
|
||
List your buckets with:
|
||
|
||
```
|
||
eouser@vm01:$ s3cmd ls
|
||
2022-02-02 22:22 s3://bucket
|
||
|
||
```
|
||
|
||
To see available commands for S3cmd, type the following command:
|
||
|
||
```
|
||
s3cmd -h
|
||
|
||
```
|
||
|
||
**boto3**
|
||
|
||
Warning
|
||
|
||
We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is this: [How to install Python virtualenv or virtualenvwrapper on 3Engines Cloud](../cloud/How-to-install-Python-virtualenv-or-virtualenvwrapper-on-3Engines-Cloud.html.md)
|
||
|
||
If virtualenv is activated:
|
||
|
||
```
|
||
(myvenv) eouser@vm01:~$ pip3 install boto3
|
||
|
||
```
|
||
|
||
Or if we install the package globally:
|
||
|
||
```
|
||
eouser@vm01:~$ sudo pip3 install boto3
|
||
|
||
```
|
||
|
||
**Simple script for accessing your private bucket:**
|
||
|
||
WAW4-1WAW3-1WAW3-2FRA1-2
|
||
|
||
```
|
||
import boto3
|
||
|
||
def boto3connection(access_key,secret_key,bucketname):
|
||
host='https://s3.waw4-1.3Engines.com'
|
||
s3=boto3.resource('s3',aws_access_key_id=access_key,
|
||
aws_secret_access_key=secret_key, endpoint_url=host,)
|
||
|
||
bucket=s3.Bucket(bucketname)
|
||
for obj in bucket.objects.filter():
|
||
print('{0}:{1}'.format(bucket.name, obj.key))
|
||
|
||
#For Python3
|
||
x = input('Enter your access key:')
|
||
y = input('Enter your secret key:')
|
||
z = input('Enter your bucket name:')
|
||
|
||
boto3connection(x,y,z)
|
||
|
||
```
|
||
|
||
```
|
||
import boto3
|
||
|
||
def boto3connection(access_key,secret_key,bucketname):
|
||
host='https://s3.waw3-1.3Engines.com'
|
||
s3=boto3.resource('s3',aws_access_key_id=access_key,
|
||
aws_secret_access_key=secret_key, endpoint_url=host,)
|
||
|
||
bucket=s3.Bucket(bucketname)
|
||
for obj in bucket.objects.filter():
|
||
print('{0}:{1}'.format(bucket.name, obj.key))
|
||
|
||
#For Python3
|
||
x = input('Enter your access key:')
|
||
y = input('Enter your secret key:')
|
||
z = input('Enter your bucket name:')
|
||
|
||
boto3connection(x,y,z)
|
||
|
||
```
|
||
|
||
```
|
||
import boto3
|
||
|
||
def boto3connection(access_key,secret_key,bucketname):
|
||
host='https://s3.waw3-2.3Engines.com'
|
||
s3=boto3.resource('s3',aws_access_key_id=access_key,
|
||
aws_secret_access_key=secret_key, endpoint_url=host,)
|
||
|
||
bucket=s3.Bucket(bucketname)
|
||
for obj in bucket.objects.filter():
|
||
print('{0}:{1}'.format(bucket.name, obj.key))
|
||
|
||
#For Python3
|
||
x = input('Enter your access key:')
|
||
y = input('Enter your secret key:')
|
||
z = input('Enter your bucket name:')
|
||
|
||
boto3connection(x,y,z)
|
||
|
||
```
|
||
|
||
```
|
||
import boto3
|
||
|
||
def boto3connection(access_key,secret_key,bucketname):
|
||
host='https://s3.fra1-2.3Engines.com'
|
||
s3=boto3.resource('s3',aws_access_key_id=access_key,
|
||
aws_secret_access_key=secret_key, endpoint_url=host,)
|
||
|
||
bucket=s3.Bucket(bucketname)
|
||
for obj in bucket.objects.filter():
|
||
print('{0}:{1}'.format(bucket.name, obj.key))
|
||
|
||
#For Python3
|
||
x = input('Enter your access key:')
|
||
y = input('Enter your secret key:')
|
||
z = input('Enter your bucket name:')
|
||
|
||
boto3connection(x,y,z)
|
||
|
||
```
|
||
|
||
Save your file with **.py** extension and execute the following command in the terminal:
|
||
|
||
```
|
||
python3 <filename.py>
|
||
|
||
```
|
||
|
||
Enter the access key, secret key and bucket name. If everything is correct, you should see output in the following format: **<bucket\_name>:<file\_name>**. |