Fides Workers Reference

Fides uses background workers to process various tasks asynchronously. This guide provides detailed information about worker types, configuration, and resource requirements for all deployment methods.

Worker Configuration by Deployment Method

Worker setup varies significantly based on your deployment method. Be sure to follow the instructions for your specific deployment type.

Automatic Worker Deployment

The Fides Helm chart automatically includes specialized workers by default:

Standard Fides Deployments:

DSR worker (1 instance) - Handles data subject requests
Other worker (1 instance) - Handles general tasks (excludes DSR and privacy preference queues)

Fidesplus Deployments: Includes all standard workers plus:

Classification worker (1 instance) - For data discovery classification tasks
Helios worker (1 instance) - For detection and promotion tasks
Consent worker (1 instance) - For privacy preferences and consent webhooks

Production deployments should customize these defaults based on your specific product usage. See Customizing Workers below.

Production Configuration

Product-Specific Requirements

Lethe Users:

DSR worker

Janus Users:

Consent worker
Consent Webhook worker

Helios Users:

Detection worker
Classification worker
Promotion worker

General Recommendations

Always configure an "Other" worker to handle messaging tasks and new task types released by Ethyca

Use exclude-queues to prevent overlap with your specific workers:

fides worker --exclude-queues=fides.dsr,fides.privacy_preferences

Resource Requirements

Worker Type	vCPU	Memory	Fides	Fidesplus
DSR	0.5-1	scalable up to 16Gi**	✅	✅
Consent	0.5-1	0.75-2Gi	✅	✅
Messaging	0.5-1	1-2Gi	✅	✅
Consent Webhook	0.5-1	0.75-2Gi	❌	✅
Detection*	0.5-1	1-2Gi	❌	✅
Classification*	2-4	scalable up to 16Gi**	❌	✅
Promotion*	0.5-1	1-2Gi	❌	✅

* These tasks are only available when the following environment variable is set: FIDESPLUS__DETECTION_DISCOVERY__MONITOR_CELERY_TASKS_ENABLED=true (enabled in Helm by default).

** Memory usage is heavily dependent on workload. Workers scale up during processing of large tables or DSRs but don't maintain peak memory usage continuously.

Customizing Workers

Overriding Default Workers

By default, the Helm chart includes DSR and Other workers (plus additional workers for Fidesplus). To customize or disable these defaults, you must explicitly define your worker configuration.

To disable a default worker, set its count to 0:

workers:
  - name: dsr
    count: 0  # Disables the default DSR worker
  - name: custom-dsr
    count: 2
    queues:
      - fides.dsr
    image:
      repository: ethyca/fides
      tag: 2.51.0-slim

To modify default workers, explicitly define them with your desired configuration:

workers:
  - name: dsr
    count: 3  # Override default count
    queues:
      - fides.dsr
    resources:
      requests:
        memory: "2Gi"
        cpu: "1000m"

⚠️

If you define any workers in your values.yaml, the default workers will still be created unless you explicitly set their count to 0.

See Deployment Examples below for complete configuration examples.

Worker Types Reference

DSR (Data Subject Requests)

Processes data subject requests and collects data from systems integrated with Lethe.

Image: Any fides or fidesplus image. For fidesplus, use -slim suffixed tags to reduce memory usage.

Startup command:

fides worker --queues=fides.dsr

Use Case: Lethe privacy request execution - retrieving and deleting data across integrated systems

Consent

Processes consent preference saves from users on properties that serve fides.js.

Image: Any fides or fidesplus image. For fidesplus, use -slim suffixed tags to reduce memory usage.

Startup command:

fides worker --queues=fides.privacy_preferences

Use Case: Janus consent management - propagating user consent preferences to integrated systems

Messaging

Processes email tasks and notifications.

Image: Any fides or fidesplus image. For fidesplus, use -slim suffixed tags to reduce memory usage.

Startup command:

fides worker --queues=fidesops.messaging

Use Case: Email notifications for privacy requests and system alerts

Consent Webhook (Fidesplus Only)

Processes consent preference updates from third-party systems via Bidirectional consent.

Image: Any fidesplus image. Recommend using -slim suffixed tags to reduce memory usage.

Startup command:

fides worker --queues=fides.consent_webhooks

Use Case: Receiving consent preference updates from external systems

Detection (Fidesplus Only)

Monitors data sources and detects changes for data discovery workflows.

Image: Any fidesplus image. Recommend using -slim suffixed tags to reduce memory usage.

Startup command:

fides worker --queues=fidesplus.discovery_monitors_detection

Use Case: Helios data discovery - detecting schema and data changes in connected systems

Classification (Fidesplus Only)

Classifies data detected in data detection tasks for data discovery workflows.

Image: Any fidesplus image. Do not use -slim suffixed tags as classification requires full dependencies.

Startup command:

fides worker --queues=fidesplus.discovery_monitors_classification

Use Case: Helios data discovery - automatically classifying detected data fields

Promotion (Fidesplus Only)

Promotes detection and discovery results to the Fides data inventory (triggered when clicking Confirm in the "Data discovery" view).

Image: Any fidesplus image. Recommend using -slim suffixed tags to reduce memory usage.

Startup command:

fides worker --queues=fidesplus.discovery_monitors_promotion

Use Case: Helios data discovery - promoting discovered data into the data map

Deployment Examples

Configure workers in your values.yaml:

workers:
  # DSR worker
  - name: dsr
    count: 2
    queues:
      - fides.dsr
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "16Gi"
        cpu: "1000m"
  
  # Consent worker
  - name: consent
    count: 1
    queues:
      - fides.privacy_preferences
    resources:
      requests:
        memory: "0.75Gi"
        cpu: "500m"
      limits:
        memory: "2Gi"
        cpu: "1000m"
  
  # Other worker (excluding specific queues)
  - name: other
    count: 1
    excludeQueues:
      - fides.dsr
      - fides.privacy_preferences
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
      limits:
        memory: "512Mi"
        cpu: "500m"

Scaling Considerations

DSR Workers: Scale based on privacy request volume and number of integrated systems
Consent Workers: Scale based on consent preference update volume (fides.js traffic)
Classification Workers: Scale based on data source size and discovery frequency
Messaging Workers: Typically 1-2 workers are sufficient unless sending high volumes of emails

Monitoring

Monitor worker health and performance through:

Health endpoint: /health/workers
Redis queue depth monitoring
Worker logs for task processing times
Resource utilization (CPU, memory)

For production monitoring setup, see the Production Kubernetes Deployment guide.

Terraform Managing Domains