Before
Analysts had to reconcile overlapping entries from separate threat feeds, compare formats, remove allowed values, and track which pages or indicators had already been processed. They also had to maintain the collection index and verify that writes and progress records were complete.
How it works
- Collect recent ThreatFox entries and the configured recent pages from selected OTX pulses, while isolating failed sources or pages.
- Normalize and deduplicate indicators, apply the allowlist, merge source evidence, remove expired entries, and write the results into shared storage.
- Update the hunt index and collection progress only after writes are verified; otherwise preserve the cursor for a later retry.
Workflow details
- Collection: A scheduled or on-demand run retrieves ThreatFox indicators from the previous 24 hours and the configured recent pages from selected OTX pulses. Collection failures are isolated so one source or pulse does not stop the remaining work.
- Validation and normalization: Indicators are grouped by type and value, normalized, deduplicated, and filtered against the allowlist, including private or reserved addresses and broad trusted domains. Conflicting evidence is consolidated using the highest confidence, combined labels and sources, and expanded validity periods.
- Storage: Eligible indicators are merged into shared type-based buckets with a 30-day retention window. Expired indicators are removed during rewrites, while source references and confidence information remain available for later investigation.
- Hunt indexing: Newly collected indicators are added to the hourly hunt index, capped at 500 indicators per run, while the full merged collection remains in storage. The index is updated rather than overwritten when multiple runs occur in the same hour.
- Reliability and continuity: The workflow verifies bucket, index, cursor, and run-log writes before completing. If a write or collection page fails, progress is preserved so the next run can retry the same indicators instead of silently skipping them.
What changed
Each collection run now performs feed reconciliation, eligibility checks, merging, indexing, and write validation automatically. The SOC receives a resumable shared store with source evidence and a capped hunt index, while failed writes leave progress unchanged for the next run.
Work removed
Comparing indicators across ThreatFox and OTX, reconciling feed formats, filtering private or allowlisted values, removing duplicates and expired entries, maintaining page and cursor progress, updating the hunt index, and checking bucket and state writes.
Expected outcome
A continuously refreshed, deduplicated threat intelligence store is available for hunting, with the hunt index limited to 500 newly collected indicators per run and failed collections safely retryable.