GPT-4o
⚙️ Technical
Advanced
Build a Web Scraper with AI
Design a robust web scraper for [TARGET_SITE] that handles pagination, rate limiting, anti-bot measures, and stores results in [STORAGE].
The Prompt
# Build a Web Scraper with AI You are a web scraping engineer. Build a production-grade scraper for [TARGET_SITE] using [SCRAPING_LIBRARY] in [LANGUAGE]. ## Target Data Scrape the following data from [TARGET_URL_PATTERN]: - [FIELD_1]: CSS/XPath selector [SELECTOR_1] - [FIELD_2]: CSS/XPath selector [SELECTOR_2] - [FIELD_3]: CSS/XPath selector [SELECTOR_3] ## Pagination Handle pagination by [PAGINATION_METHOD] (next-page button, query param increment, or infinite scroll) up to [MAX_PAGES] pages. ## Anti-Bot Handling - Rotate user agents from [USER_AGENT_LIST] - Add random delays between [MIN_DELAY]s and [MAX_DELAY]s per request - Retry on [RETRY_STATUS_CODES] with exponential backoff up to [MAX_RETRIES] attempts - Handle [CAPTCHA_TYPE] by [CAPTCHA_STRATEGY] ## Storage Save each scraped record to [STORAGE] (database, JSON file, or cloud storage). Deduplicate on [UNIQUE_FIELD]. ## Scheduling Run the scraper every [INTERVAL] using [SCHEDULER]. Alert [CHANNEL] if fewer than [MIN_RECORDS] records are returned.
📝 Fill in the blanks
Replace these placeholders with your own content:
[TARGET_SITE]
[SCRAPING_LIBRARY]
[LANGUAGE]
[TARGET_URL_PATTERN]
[FIELD_1]
[SELECTOR_1]
[FIELD_2]
[SELECTOR_2]
[FIELD_3]
[SELECTOR_3]
[PAGINATION_METHOD]
[MAX_PAGES]
[USER_AGENT_LIST]
[MIN_DELAY]
[MAX_DELAY]
[RETRY_STATUS_CODES]
[MAX_RETRIES]
[CAPTCHA_TYPE]
[CAPTCHA_STRATEGY]
[STORAGE]
[UNIQUE_FIELD]
[INTERVAL]
[SCHEDULER]
[CHANNEL]
[MIN_RECORDS]
How to use this prompt
1
Copy the prompt
Click "Copy Prompt" above to copy the full prompt text to your clipboard.
2
Replace the placeholders
Swap out anything in [BRACKETS] with your specific details.
3
Paste into GPT-4o
Open your preferred AI assistant and paste the prompt to get started.