saritasa-s3-tools

Extension for boto3 to ease work with s3

Table of contents

• Installation
• Features
• Django
• Optional dependencies
• Direct upload example
• Pytest

Installation

pip install saritasa-s3-tools

or if you are using poetry

poetry add saritasa-s3-tools

Features

• S3Client and AsyncS3Client for integrations with s3 buckets. This clients are extension to boto3 clients with proper typing, support for async and method to generate signed urls for file upload.
• S3FileTypeConfig for defining configuration parameters for direct upload to s3. Check out more
• S3Key for generating unique keys for s3 upload, used for S3FileTypeConfig
• S3FileField and S3ImageFileField - factory-boy fields for generating files and saving it in s3
• pytest plugin with fixtures for boto3, S3Client and AsyncS3Client and etc
• Django plugin for setting up models and api

Django

saritasa-s3-tools comes with django plugin which helps with models config and api(Django Rest Framework and drf_spectacular). You can try it out in example folder.

Setup model

First you need to add file/image fields to you model, like below

import saritasa_s3_tools.django

from django.db import models

class ModelWithFiles(models.Model):
    """Test model with different files configs."""

    file = saritasa_s3_tools.django.S3FileField(
        blank=True,
        null=True,
        # S3FileTypeConfig is needed so that we could understand where to save
        # file, who can save file, what files can be save and what are size
        # constraints
        s3_config=saritasa_s3_tools.S3FileTypeConfig(
            name="django-files",
            key=saritasa_s3_tools.keys.S3KeyWithPrefix("django-files"),
            allowed=("text/plain",),
            auth=lambda user: bool(user and user.is_authenticated),
            content_length_range=(1000, 20000000),
        ),
    )

    image = saritasa_s3_tools.django.S3ImageField(
        blank=True,
        null=True,
        s3_config=saritasa_s3_tools.S3FileTypeConfig(
            name="django-images",
            key=saritasa_s3_tools.keys.S3KeyWithPrefix("django-images"),
            allowed=("image/png",),
            content_length_range=(5000, 20000000),
        ),
    )

Setup serializers

Then add S3FieldsConfigMixin mixin to your serializer, like this

from rest_framework import serializers

import saritasa_s3_tools.django

from .. import models


class ModelWithFilesSerializer(
    saritasa_s3_tools.django.S3FieldsConfigMixin,
    serializers.ModelSerializer,
):
    """Serializer to show info model with files."""

    class Meta:
        model = models.ModelWithFiles
        fields = "__all__"

Setup view

Then just add S3GetParamsView view to your project urls like that.

from django.urls import path

path(
    "s3/",
    include("saritasa_s3_tools.django.urls"),
    name="saritasa-s3-tools",
),

Setup pytest

Just add this to core conftest.py file

from django.conf import settings
from django.core.files.storage import default_storage
import mypy_boto3_s3
import pytest


@pytest.fixture(scope="session", autouse=True)
def _adjust_s3_bucket(
    s3_bucket: str,
) -> None:
    """Set bucket to a test one."""
    settings.AWS_STORAGE_BUCKET_NAME = s3_bucket


@pytest.fixture(scope="session")
def boto3_resource() -> mypy_boto3_s3.S3ServiceResource:
    """Prepare boto3 resource."""
    return default_storage.connection

Optional dependencies

• [async] - Add this to enable async support
• [factory] - Add this to enable factory-boy field S3FileField and S3ImageFileField from saritasa_s3_tools.factory
• [testing] - Add this to enable testing helping functions from saritasa_s3_tools.testing.shortcuts
• [django] - Add this to enable django support
• [django-openapi] - Add this to enable drf-spectacular support

To install all optional dependencies add [all]

Direct upload example

import saritasa_s3_tools
import pathlib
import xml.etree.ElementTree

s3_client = saritasa_s3_tools.S3Client(
    boto3_client=boto3_client,
    default_bucket=s3_bucket,
)
s3_params = s3_client.generate_params(
    filename=pathlib.Path(__file__).name,
    config=saritasa_s3_tools.S3FileTypeConfig.configs["files"],
    content_type="application/x-python-code",
    extra_metadata={
        "test": "123",
    },
)
with (
    httpx.Client() as client,
    pathlib.Path(__file__).open("rb") as upload_file,
):
    upload_response = client.post(
        url=s3_params.url,
        data={
            key: value
            for key, value in s3_params.params.items()
            if value is not None
        },
        files={"file": upload_file.read()},
    )
parsed_response = xml.etree.ElementTree.fromstring(  # noqa: S314
    upload_response.content.decode(),
)
file_key = parsed_response[2].text
file_url = parsed_response[0].text

pytest plugin

saritasa-s3-tools comes with pytest plugin which can setup boto3 instances and create/clean up buckets for testing. Supports pytest-xdist.

Fixtures

• access_key_getter, s3_endpoint_url_getter, s3_region - are used to configure boto3 session and clients/resources that are used in tests. You can override them or set values in ini file for pytest. Plugin will tell what you are missing.
• aws_session - Returns boto3.Session
• aws_config - Returns botocore.config.Config, override if you need customization, None by default.
• boto3_resource - Returns s3 resource or in typing mypy_boto3_s3.S3ServiceResource
• boto3_client- Returns s3 client or in typing mypy_boto3_s3.S3Client
• s3_bucket_name - Name of bucket for testing, default: saritasa-s3-tools or s3_bucket_name from ini file.
• s3_bucket_cleaner - Returns function which cleans all files from bucket
• s3_bucket_factory - Returns manager which creates bucket, and when it's no longer needed deletes it
• s3_bucket - Creates bucket via s3_bucket_factory and return it's name
• s3_client - Returns saritasa_s3_tools.S3Client
• async_s3_client - Returns saritasa_s3_tools.AsyncS3Client