The last individual unit of functionality that we need to implement is storing data in AWS S3. In this lesson, we'll expand our
storage module by adding a way to write a file to S3.
To interface with AWS (S3 specifically), we're going to use the wonderful
boto3 package. We can install this to our virtualenv using
(pgbackup) $ pipenv install boto3
boto3 installed, we'll be able to connect to S3 since it will read the same configuration files that we created in an earlier lesson using the AWS CLI.
storage module that we've already created is the perfect place to put our code that interacts with S3 since it's just another form of storage. We'll create another function for
s3 that will take a few things:
client: An AWS client object that has an
infile: A file object with the data from our PostgreSQL backup.
bucket: The name of the bucket that we'll be storing the backup in.
name: The name of the file we'd like to create in our S3 bucket.
The reason that we are injecting the
client object is that we don't want our
storage module to be in charge of configuring the client, we'll do that when we tie all of the pieces together.
upload_fileobj method works in the exact way that we need it to, so our implementation should be pretty simple:
# ... previous function ommited def s3(client, infile, bucket, name): client.upload_fileobj(infile, bucket, name)
Like we did with our PostgreSQL interaction, let's manually test uploading a file to S3 using our
storage.s3 function. First, we'll create an
example.txt file, and then we'll load into a Python REPL with our code loaded:
(pgbackup) $ echo "UPLOADED" > example.txt (pgbackup) $ PYTHONPATH=./src python >>> import boto3 >>> from pgbackup import storage >>> client = boto3.client('s3') >>> infile = open('example.txt', 'rb') >>> storage.s3(client, infile, 'python-backups', infile.name)
When we check our S3 console, we should see the file there. Lastly, remove the
example.txt file and then commit these changes:
(pgbackup) $ rm example.txt (pgbackup) $ git add . (pgbackup) $ git commit -m 'Implement S3 interactions'