GPT-4o
⚙️ Technical
Intermediate
Build an ETL Pipeline with AI
Design and generate an ETL pipeline that extracts [DATA_SOURCE], transforms it to [TARGET_SCHEMA], and loads it into [DESTINATION] on a schedule.
The Prompt
# Build an ETL Pipeline with AI You are a data engineering lead. Design and implement an ETL pipeline for [PIPELINE_NAME] in [LANGUAGE_OR_TOOL]. ## Data Source - **Source:** [DATA_SOURCE] (database, API, file, or stream) - **Format:** [SOURCE_FORMAT] - **Volume:** [RECORD_VOLUME] records per run - **Schedule:** every [INTERVAL] via [SCHEDULER] ## Transformation Rules Apply these transformations: 1. **Filter:** keep only records where [FILTER_CONDITION] 2. **Enrich:** join with [LOOKUP_TABLE] on [JOIN_KEY] to add [NEW_FIELDS] 3. **Transform:** convert [SOURCE_FIELD] to [TARGET_FORMAT] using [TRANSFORM_RULE] 4. **Validate:** reject rows missing [REQUIRED_FIELDS] and log to [DEAD_LETTER_STORE] ## Destination - **Target:** [DESTINATION] (data warehouse, database, or file) - **Load strategy:** [UPSERT_OR_APPEND] on key [PRIMARY_KEY] - **Target schema:** [SCHEMA_DEFINITION] ## Observability Log row counts at each stage. Alert [DATA_TEAM_CHANNEL] if error rate exceeds [THRESHOLD]% or run time exceeds [TIMEOUT_MINS] minutes.
📝 Fill in the blanks
Replace these placeholders with your own content:
[PIPELINE_NAME]
[LANGUAGE_OR_TOOL]
[DATA_SOURCE]
[SOURCE_FORMAT]
[RECORD_VOLUME]
[INTERVAL]
[SCHEDULER]
[FILTER_CONDITION]
[LOOKUP_TABLE]
[JOIN_KEY]
[NEW_FIELDS]
[SOURCE_FIELD]
[TARGET_FORMAT]
[TRANSFORM_RULE]
[REQUIRED_FIELDS]
[DEAD_LETTER_STORE]
[DESTINATION]
[UPSERT_OR_APPEND]
[PRIMARY_KEY]
[SCHEMA_DEFINITION]
[DATA_TEAM_CHANNEL]
[THRESHOLD]
[TIMEOUT_MINS]
How to use this prompt
1
Copy the prompt
Click "Copy Prompt" above to copy the full prompt text to your clipboard.
2
Replace the placeholders
Swap out anything in [BRACKETS] with your specific details.
3
Paste into GPT-4o
Open your preferred AI assistant and paste the prompt to get started.