Web Scrape
Fetch data from web pages, Google Search, Instagram, TikTok, or RSS feeds and emit structured JSON.
Overview
The Web Scrape node retrieves data from external sources using configurable actors (scrapers). Choose from Google Search for keyword results, a content crawler for page or site Markdown, RSS for feed items, or Instagram/TikTok for post metadata. Output is a structured JSON array that you can pipe into Extract Field, JSON Process, or a List node for fan-out.
When to Use
- Pull search results into a content generation pipeline
- Scrape a web page and feed its text to Generate Text for summarization
- Fetch recent Instagram or TikTok posts for analysis or remixing
- Combine with Schedule Trigger to run recurring data ingestion workflows
Configuration
Source (actor)
| Actor |
Label |
Description |
google-search |
Google Search |
Returns up to 10 search-result items |
content-crawler |
Website Content (Markdown) |
Crawls one page or an entire site; emits Markdown |
rss |
RSS Feed |
Directly fetches and parses an RSS/Atom feed (no Apify) |
instagram |
Instagram |
Retrieves posts from a profile or URL |
tiktok |
TikTok |
Retrieves posts from a profile or URL |
Google Search fields
| Field |
Type |
Default |
Description |
| Query |
text |
— |
Search query. Use {} to inject an upstream text value |
| Max results |
number |
5 |
How many results to return (1–10) |
| Country code |
text |
— |
2-letter ISO country code to localise results (e.g., us) |
Website Content (Markdown) fields
| Field |
Type |
Default |
Description |
| Start URL |
text |
— |
URL of the page or site root. Use {} to inject upstream |
| Crawl mode |
select |
page |
Single page (1 CR) — one URL; Site crawl, up to 20 pages (5 CR) |
| Field |
Type |
Default |
Description |
| Feed URL |
text |
— |
URL of the RSS or Atom feed |
| Results limit |
number |
10 |
Maximum items to return (1–50). Emits { title, url, description, pubDate, guid } per item |
Instagram / TikTok fields
| Field |
Type |
Default |
Description |
| Profile or post URL |
text |
— |
Full URL of the profile or post. Use {} to inject upstream |
| Results limit |
number |
10 |
Maximum posts to return (1–20) |
Inputs: Optional upstream connection (used when injecting a value into a field via {}).
Outputs: json — a structured JSON array of result items. Connect to Extract Field or JSON Process to reshape the data.
Pricing
| SKU |
Credits |
| Google Search |
3 CR |
| Content Crawler — single page |
1 CR |
| Content Crawler — site crawl |
5 CR |
| Instagram |
1 CR |
| TikTok |
1 CR |
| RSS |
1 CR |
Common Use Cases
- Search Google for a keyword, extract the top titles, and feed each into a Generate Text node for article drafts
- Crawl a product page and pipe its Markdown to an LLM for structured data extraction
- Pull the latest Instagram posts from a brand account and batch-generate captions for remixes
- Use RSS + Schedule Trigger to build a fully automated daily newsletter pipeline
Tips
- All actor fields support FieldMapping injection via
{} so you can drive the query or URL from an upstream Text node.
- For site crawls, the crawler follows internal links up to 20 pages; use
Single page when you only need one URL’s content.
- RSS items contain
title, url, description, pubDate, and guid. Use Extract Field with title to pull just the headlines.
- Chain an Extract Field node after Web Scrape to pull a specific property (e.g.,
caption from TikTok, title from Google results) before feeding a List node.