Automate Data Store Monitor Activities

Data Store Monitors scan databases to produce Datasets that are labeled with the Fideslang data privacy taxonomy.

Monitor Management

These API calls are for managing the monitor metadata; this is useful for updating scan schedules and classification parameters.

Retrieve Monitors

GET /api/v1/plus/discovery-monitor
GET /api/v1/plus/discovery-monitor/{{monitor_config_key}}

Update a Monitor (Upsert)

{  
 "name": "Monitor Name",  
 "key": "Monitor_Name",  
 "connection_config_key": "integration-key",  
 "classify_params": {  
    "possible_targets": [  
        "user.financial.account_number",  
        "user.payment.financial_account_number",  
        "user.credentials.biometric_credentials"  
    ],  
    "num_samples": 25,  
    "context_weight": 0.95,  
    "content_weight": 0.05,  
    "excluded_categories": [],  
    "context_regex_pattern_mapping": [  
        [ ".*dob", "user.demographic.date_of_birth" ],  
        [ ".*full_name", "user.name" ]  
    ]  
  }  
}

Staged Resources & Results

The /results endpoint is used to retrieve staged resources; this is used to traverse the staged resource hierarchy.

A staged resource is specified by the staged_resource_urn query parameter. If a staged_resource_urn is not provided, then the top-level staged resources are returned.

The API can recursively traverse the staged resource hierarchy; the API response object contains child urn fields, which can then be provided as the staged_resource_urn parameter in a subsequent API call to see that resource's children, and so on.

Retrieve Results

GET /api/v1/plus/discovery-monitor/results
GET /api/v1/plus/discovery-monitor/results?staged_resource_urn={{staged_resource_urn}}&page=1&size=50

Detection Actions

These API calls designate which assets in your data architecture are included in your Fides pipeline.

Confirm

Any confirmed asset will be monitored and listed in Discovery.

When you confirm a resource the endpoint will return a monitor_task_id. You will want to save this for a later step in this guide.

["staged_resource_urn1", "staged_resource_urn2"]

Mute

Any muted asset will not be moved to the Discovery step. Muted assets will no longer be detected in scans. If you wish to include it again unmute it.

["staged_resource_urn1", "staged_resource_urn2"]

Unmute

Unmuted assets will be detected in scans. An unmuted asset will still need to be confirmed in order to be moved to the Discovery step.

["staged_resource_urn1", "staged_resource_urn2"]

Discovery Actions

Update Data Categories for Staged Resources

[  
  {  
    "urn": "staged_resource_urn1",  
    "user_assigned_data_categories": [  
      "user.name"  
    ],  
    "user_assigned_data_uses": [  
      "marketing.third_party"  
    ],  
    "user_assigned_system_key": "system_key"  
  }  
]

Promote Staged Resources to Dataset

["staged_resource_urn1", "staged_resource_urn2"]

Dataset Queries

The resources have now been promoted to a Dataset. To validate the output, retrieve data through the dataset endpoint.

GET /api/v1/dataset/{fides_key}

Task Management

The task management APIs are used to track the Fides worker processing status.

GET /api/v1/plus/discovery-monitor/tasks?status=in_processing&action_type=classification
GET /api/v1/plus/discovery-monitor/tasks/{{monitor_task_id}}