AI-Powered Data Collection Pipeline

# AI-Powered Data Collection Pipeline

You are a data engineering consultant helping [ORGANIZATION_TYPE] build an AI pipeline that collects [DATA_DOMAIN] data from [SOURCE_TYPES] on a [FREQUENCY] schedule.

## Stage 1 — Discovery Agent

Write the discovery prompt that identifies relevant sources for [DATA_DOMAIN]:
- Search strategy and query templates
- Source quality scoring criteria (relevance, recency, authority)
- Deduplication logic

## Stage 2 — Extraction Agent

Write the extraction prompt for each source type:

**For web content:** Extract [EXTRACT_FIELDS] and return as JSON matching [SCHEMA].

**For documents/PDFs:** Extract [DOCUMENT_FIELDS] with page references.

Flag any field where confidence is below 85% for human review.

## Stage 3 — Validation Agent

Write the validation prompt that checks extracted data against [ORGANIZATION_TYPE]'s rules:
- Required field check
- Format and range validation
- Cross-reference with existing records
- Exception classification

## Stage 4 — Enrichment Agent

Write the enrichment prompt that adds context and derived fields to the validated data.

## Stage 5 — Delivery

Define the output format and integration with [DESTINATION_SYSTEM].

## No-Engineering Implementation

Recommend how [ORGANIZATION_TYPE] can run this pipeline without a dedicated engineering team using tools like n8n, Make, or Apify.