Prompt Library › ⚙️ Technical › Real-Time Data Pipeline Agent
GPT-4o ⚙️ Technical Advanced

Real-Time Data Pipeline Agent

Design an AI agent that processes and routes streaming data in real time — with transformation logic, AI enrichment, and an architecture diagram.

👁 21 views ⎘ 0 copies ♥ 0 likes

The Prompt

# Real-Time Data Pipeline Agent

You are a data engineering architect designing an AI agent that processes, transforms, and routes streaming data in real time.

## Pipeline Spec

- Data source: [SOURCE — e.g., webhook events, API polling, database change stream, IoT sensor feed]
- Data volume: [APPROXIMATE EVENTS PER MINUTE OR HOUR]
- Transformation required: [WHAT NEEDS TO CHANGE — e.g., enrich, normalize, filter, aggregate]
- Downstream destinations: [WHERE PROCESSED DATA GOES — e.g., database, dashboard, alert system, another API]
- Latency requirement: [ACCEPTABLE PROCESSING DELAY — e.g., under 5 seconds, near real-time, batch OK]

## Pipeline Design

Design the full real-time agent pipeline:

1. Ingestion layer: how events are received and buffered
2. Transformation logic: the step-by-step operations applied to each event
3. AI enrichment step: where the agent applies LLM-based enrichment (e.g., classification, sentiment, entity extraction)
4. Routing rules: which events go where based on content
5. Error handling: what happens to malformed events — dead letter queue, retry, or discard
6. Monitoring: the metrics tracked to confirm the pipeline is healthy

Produce an architecture diagram (ASCII) and a pseudocode implementation for the transformation and routing layer.

📝 Fill in the blanks

Replace these placeholders with your own content:

[SOURCE — e.g., webhook events, API polling, database change stream, IoT sensor feed]
[APPROXIMATE EVENTS PER MINUTE OR HOUR]
[WHAT NEEDS TO CHANGE — e.g., enrich, normalize, filter, aggregate]
[WHERE PROCESSED DATA GOES — e.g., database, dashboard, alert system, another API]
[ACCEPTABLE PROCESSING DELAY — e.g., under 5 seconds, near real-time, batch OK]

How to use this prompt

1
Hit "Run it in your AI"

We open ChatGPT, Claude, Gemini, Perplexity, Grok or Copilot with this prompt in the box, ready for you to press Enter — and copied to your clipboard as a backup. Pick your AI once and we remember it.

2
Replace the placeholders

Swap out anything in [BRACKETS] with your specific details.

3
Send it

This prompt is tuned for GPT-4o, but it works well in any of them. Prefer to do it by hand? Copy to clipboard is always there too.

Get prompts like this every Monday Free weekly digest — hand-picked prompts, new tools, AI news.