Configure Storage Destinations

What is a storage destination?

Data subject access requests (DSARs) produce a package of returned data upon completion. This data is then uploaded to a storage destination (e.g. an S3 bucket) before being sent to the user.

Storage destinations are associated with privacy request policy rules, allowing multiple storage destinations to be configured per privacy request policy.

Create a storage destination

Configure your storage method

To configure a Storage destination, first choose a method to store your results. Astralis currently supports the following methods of storage:

• local - This saves upload packages locally, generating a fides_uploads directory at the root of your project. This destination type should only be used for testing purposes, and not to process real-world access requests.
• S3 - Files are uploaded to an S3 bucket of your choosing upon completion of an access request. Use S3 if you need a place to store those files.
• Google Cloud Storage - Files are uploaded to a GCS bucket.

Create your storage destination

Storage destinations are created and managed via the API. To create a new Storage destination, use the following endpoint:

[
  {
    "name": "{{storage_destination_name}}",
    "key": "{{storage_destination_key}}",
    "type": "s3" | "gcs" | "local",
    "format": "json" | "csv" | "html",
    "details": { 
      "naming": "request_id",                       // default value
      "auth_method": "secret_keys" | "automatic",   // for cloud storage destinations
      "bucket": "{{bucket_name}}",                  // for cloud storage destinations
      "enable_streaming": false,                    // GCS only: opt in to streamed uploads (ignored on S3)
      "enable_access_package_redirect": false       // for S3 storage destinations
    }
  }
]

Destination attributes

Additional attributes for s3 buckets

Additional attributes for Google Cloud Storage

Additional attributes for local storage

Upload streaming

When a request completes, Astralis packages the results and uploads them to your storage destination. That upload happens one of two ways:

• Streaming, where the package is written to storage in chunks as it is built. Memory use stays flat regardless of package size, so large packages and large attachments upload reliably.
• Buffered, where the whole package is assembled in memory and then uploaded. A very large package can exhaust memory on this path.

Which one Astralis uses depends on the backend. It is fixed for S3 and local storage, while for Google Cloud Storage it is a setting you control:

• S3 ignores enable_streaming; the format decides. HTML access-package reports stream (a .zip, so large packages and their attachments upload reliably), while JSON and CSV are buffered.
• Google Cloud Storage buffers by default and streams when you set details.enable_streaming to true. Turning it on is the recommended setting for any destination that handles large packages.
• Local storage always buffers. It is intended for quick-start and demo use, so this is not a concern in practice.

The package contents are identical either way: the same files are included and links resolve the same. Streaming only changes how the upload is written, so switching a GCS destination to streaming is safe.

On success, the response from the above endpoint will include a storage_key for each destination, which can be used when defining privacy request policy rules.

{
  "items": [
    {
      "id": "sto_fe4e4dc0-b5d3-4ac1-bfcd-86e60e9891b9",
      "name": "s3 storage 2",
      "type": "s3",
      "details": {
        "auth_method": "secret_keys",
        "bucket": "my-bucket",
        "naming": "request_id",
        "object_name": "requests",
        "enable_streaming": false,
        "enable_access_package_redirect": false
      },
      "key": "s3_storage_2"
    }
  ],
  "total": 1,
  "page": 1,
  "size": 1
}

Authenticate with your destination

Astralis requires authenticated access to update and erase (or mask) data in your storage destination.

Use the key returned during your storage creation to provide access credentials:

{
  "aws_access_key_id": "{{aws_access_key_id}}",
  "aws_secret_access_key": "{{aws_secret_access_key}}"
}

Secrets are not saved if credentials fail authentication with the given storage destination.

Additional attributes for S3 buckets

Additional attributes for Google Cloud Storage

The fields described are extracted from a Service Account JSON file, which contains all the necessary information for authentication with Google Cloud services. This file is downloaded from the Google Cloud Console.

Test your storage connection

To test your storage configuration, send an API request to the upload data endpoint:

{
  "storage_key": "{{storage_key}}",
  "data": {
    // include sample JSON to upload to storage destination
  }
}

Extensibility

Astralis can be extended to support additional storage destinations by:

1. Adding destination-specific enums in src/fides/ops/schemas/storage/storage.py
2. Implementing an authenticator in src/fides/ops/service/storage/storage_authenticator_service.py
3. Implementing the uploader in src/fides/ops/service/storage/storage_uploader_service.py