Scrapfly is a web scraping API that enables developers to extract data from websites efficiently, offering features like JavaScript rendering, anti-bot protection bypass, and proxy rotation.
Scrapfly is a web scraping API that enables developers to extract data from websites efficiently, offering features like JavaScript rendering, anti-bot protection bypass, and proxy rotation. On Nagent, Scrapfly is exposed as a fully-configurable ai web scraping integration that any agent can call — 12 actions, and API key authentication. No code is required to wire Scrapfly into your workflow — connect it once via the External Integrations panel and reuse it across every agent you build.
Agent builders use Scrapfly to automate the kinds of tasks ai web scraping teams previously handled manually. Concrete examples — each one is a single agent step in Nagent — include:
Every action and trigger is paired with a structured input/output schema (visible in the sections below), so when you wire Scrapfly into Helix — our agentic agent builder — the editor knows exactly what each step expects and produces. Configure once, deploy anywhere across your Nagent agents.
Every operation an agent can call against Scrapfly, with input parameters and output schema. Drop these into any step of an agent built in Helix.
SCRAPFLY_CAPTURE_SCREENSHOTTool to capture a full-page or viewport screenshot of a website. Use when you need to take a screenshot with options like JS rendering, custom resolution, or accessibility testing. Returns the screenshot image directly. Supports vision deficiency simulations and dark mode.
Input parameters
JavaScript code to execute on the page before capturing screenshot.
Target URL to capture a screenshot of.
Enable caching of the screenshot result.
Screenshot image format (jpg, png, webp, or gif). Default is jpg.
Area to capture: 'viewport' for visible area, 'fullpage' for entire page, or a CSS/XPath selector for a specific element. Default is viewport.
Proxy geolocation country code (e.g., 'US', 'FR', 'DE') to capture screenshot from that region.
Comma-separated screenshot options: 'dark_mode' for dark theme, 'block_banners' to hide cookie/privacy banners, 'print_media_format' for print styles.
Request timeout in milliseconds. Default is 30000 (30 seconds).
Cache time-to-live in seconds. Only applies when cache is enabled.
Screen resolution in WIDTHxHEIGHT format (e.g., '1920x1080'). Default is 1920x1080.
Automatically scroll down the page before capturing. Useful for lazy-loaded content.
Clear and refresh cache for this request.
Time to wait after page load before capturing screenshot, in milliseconds. Useful for waiting for dynamic content.
CSS selector or XPath to wait for before capturing. Screenshot is taken after this element appears.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_CAPTURE_SCREENSHOT_HEADTool to capture screenshot metadata without downloading the image body. Use this for async screenshot workflows where you need the URL to retrieve the image later. Returns the screenshot URL in response, saving bandwidth compared to full screenshot retrieval.
Input parameters
Base64-encoded JavaScript to execute on the page before screenshot (max 16KB).
Target URL to capture screenshot of.
Enable caching for repeated requests. Default is false.
Image format options for screenshot.
Capture area: 'viewport' for visible area, 'fullpage' for entire page, or a CSS selector/XPath for specific element.
Proxy location using ISO 3166 alpha-2 country code (e.g., 'US', 'FR', 'DE'). Default is 'us'.
Comma-separated flags: 'dark_mode' for dark theme, 'block_banners' to hide ads/popups, 'print_media_format' for print styles.
Maximum request time in milliseconds (range: 60000-120000). Default is 60000.
Cache duration in seconds. Default is 86400 (24 hours).
Screen dimensions as widthxheight (e.g., '1920x1080'). Default is 1920x1080.
Auto-scroll to bottom of page to trigger lazy-loaded content. Default is false.
Force cache refresh by clearing existing cached result. Default is false.
Delay after page load before capturing screenshot, in milliseconds. Default is 1000.
Accessibility simulation options for vision deficiency.
CSS selector or XPath to wait for before capturing screenshot.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_CREATE_CRAWLERTool to create a new web crawler to recursively crawl an entire website. Returns a crawler UUID for tracking progress. Use when you need to crawl multiple pages from a website with configurable limits and extraction rules.
Input parameters
Enable Anti Scraping Protection bypass
Starting URL for the crawl. Must be a valid HTTP/HTTPS URL
Delay between requests in milliseconds. Range: 0-15000ms
Maximum crawl depth - controls how many levels deep the crawler will follow links
Enable JavaScript rendering
Maximum number of pages to crawl. Set to 0 for unlimited (subject to subscription limits)
Maximum concurrent scrape requests. Set 0 for account default
Automatically stop the crawl when reaching a credit limit
Exclude URLs matching these patterns. Mutually exclusive with include_only_paths
Array of desired output formats (markdown, extracted_data, page_metadata, etc.)
Only crawl URLs matching these patterns. Supports wildcards (*). Max 100 paths
Allow crawler to follow links to external domains
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_EXTRACT_DATATool to extract structured data from HTML or other content using AI models, LLM prompts, or custom templates. Use when you need to parse web pages or documents into structured JSON data. Supports predefined extraction models for common types (articles, products, events) or custom extraction via prompts/templates.
Input parameters
Base URL for converting relative URLs to absolute URLs in the extracted data. Useful when extracting links and images.
Document charset encoding. Use 'auto' for automatic detection or specify a charset like 'utf-8', 'iso-8859-1', etc.
HTML, text, or structured content to extract data from. This will be sent as the request body.
Content type of the document body. Required to properly parse the input content.
Webhook name for asynchronous processing. If provided, the extraction will be processed asynchronously and results sent to the webhook.
AI extraction models for structured data extraction.
Custom LLM prompt for extraction. Use this when you need custom extraction logic beyond predefined models. Cannot be used together with extraction_model or extraction_template.
JSON extraction template defining custom extraction rules. Cannot be used together with extraction_model or extraction_prompt.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_GET_ACCOUNT_INFOTool to retrieve Scrapfly account information. Use after authenticating to get API credit balance and usage stats. Returns comprehensive account data including subscription plan, usage statistics, billing info, and project settings.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_GET_CRAWLER_ARTIFACTTool to download crawler artifact files in WARC or HAR format. Use when you need to retrieve the complete crawl results as an archive file. WARC format is recommended for large crawls as it includes gzip compression.
Input parameters
The artifact format type. WARC format is recommended for large crawls with gzip compression.
The unique identifier (UUID) of the crawler whose artifact you want to download.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_GET_CRAWLER_CONTENTSTool to retrieve extracted content from crawled pages. Supports multiple output formats including markdown, text, HTML, and JSON. Use when you need to access the actual content extracted during a crawl, with optional filtering by URL and format selection.
Input parameters
Specific URL to query for single-URL retrieval. When specified, returns content only for this URL.
Number of items per page (default: 10, max: 50).
Return raw content without JSON wrapper. Only works with single URL queries (when 'url' parameter is specified).
Number of results to skip for pagination (default: 0).
Comma-separated list of content formats to retrieve. Available formats: html (raw HTML), clean_html (HTML with boilerplate removed), markdown (LLM-optimized markdown), text (plain text only), json (structured JSON), extracted_data (AI-extracted structured data), page_metadata (page metadata like title, description).
The unique identifier (UUID) of the crawler whose contents you want to retrieve.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_GET_CRAWLER_STATUSTool to get the current status of a crawler including progress, pages crawled, and completion state. Use for polling workflow to monitor crawl progress.
Input parameters
The unique identifier (UUID) of the crawler job to check status for.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_GET_CRAWLER_URLSTool to retrieve the list of discovered and crawled URLs from a crawler. Use when you need to get all URLs found during a crawl or filter by status to analyze failed URLs with error codes. Supports pagination for large result sets.
Input parameters
Page number for pagination (default: 1).
Filter results by status. Use 'visited' for successfully crawled URLs or 'failed' to get URLs that failed with error codes.
Number of URLs to return per page (default: 100, max: 1000).
The unique identifier (UUID) of the crawler whose URLs you want to retrieve.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_SCRAPETool to perform a web scraping request. Use when you need to fetch a page with custom configuration like JS rendering, proxies, and extraction.
Input parameters
Enable anti-scraping protection bypass.
The URL to scrape.
Request body payload for POST/PUT requests.
Custom labels for analytics.
Enable result caching.
Number of retries on failure (default is 2).
HTTP method to use for request, default is GET.
Target country for the proxy (e.g., 'US', 'FR', 'DE'). Matching the target site's region can improve success rates against geo-restricted or anti-bot measures.
Custom HTTP request headers to send.
Proxy session ID for sticky sessions.
Timeout for the scrape request in seconds (default is 60).
Enable JavaScript rendering for sites requiring it. Default returns static HTML only; dynamic/SPA content requires this set to true.
Extraction rules (JMESPath, CSS selector, or JSONPath).
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_SCRAPE_POSTTool to scrape web pages using POST method to send data in the request body. Use when you need to scrape endpoints that require POST requests, such as form submissions or APIs that expect data payload.
Input parameters
Enable anti-scraping protection bypass.
The target URL to scrape with POST request.
POST request body payload as JSON object. This data will be sent in the request body to the target URL.
Custom labels for analytics and monitoring.
Enable result caching to avoid repeated requests.
Number of retries on failure (default is 2).
Target country for the proxy using ISO 3166-1 alpha-2 codes (e.g., 'US', 'FR', 'DE').
Custom HTTP request headers to send to the target URL. Note: Content-Type defaults to application/json for POST requests.
Proxy session ID for sticky sessions and persistent cookies.
Timeout for the scrape request in seconds (default is 60).
Enable JavaScript rendering for sites requiring browser execution.
Extraction rules using JMESPath, CSS selector, or JSONPath for structured data extraction.
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
SCRAPFLY_SCRAPE_WITH_PUTTool to scrape web pages using PUT method with body payload. Use when the target API requires PUT requests with data in the request body. Forwards PUT request with custom body to the target URL. If not specified, content-type defaults to application/x-www-form-urlencoded.
Input parameters
Enable anti-scraping protection bypass.
The target URL to scrape with PUT request.
Request body payload to send with the PUT request. Should be JSON string for JSON payloads.
Custom labels for analytics.
Enable result caching.
Number of retries on failure (default is 2).
Target country for the proxy (e.g., 'US', 'FR', 'DE').
Custom HTTP request headers to send. For JSON body, include 'content-type': 'application/json'.
Proxy session ID for sticky sessions.
Timeout for the scrape request in seconds (default is 60).
Enable JavaScript rendering for sites requiring it.
Extraction rules (JMESPath, CSS selector, or JSONPath).
Output
Data from the action execution
Error if any occurred during the execution of the action
Whether or not the action execution was successful or not
No publicly available marketplace agent is found using this tool yet. There are 80 agents privately built on Nagent that already use Scrapfly.
Build on Nagent
Connect Scrapfly to any Nagent agent in minutes — no API key management, no boilerplate. Just configure and deploy.
The five questions agent builders ask before adopting a new integration.
Open the External Integrations panel inside Nagent (app.nagent.ai/externalIntegration), find Scrapfly, and click "Connect Now." You'll authenticate with an API key — Nagent handles credential storage and refresh automatically. Once connected, Scrapfly is available to any agent in your workspace.
No. Nagent provides no-code integration for every tool. Once Scrapfly is connected, you configure its 12 actions directly in the agent builder UI — no API calls, no boilerplate, no schema management.
Helix — Nagent's agentic agent builder — lets you drop Scrapfly steps into any workflow visually. Pick an action (e.g., one of those listed above), fill in the inputs (Helix knows the required vs. optional schema for each parameter), and connect it to upstream/downstream steps. Triggers run as the entry point of an agent, so when a Scrapfly event fires, the agent kicks off automatically.
Every Scrapfly action and trigger ships with a fully-typed schema — input parameters with name, type, required flag, and description, plus the output payload shape. The schemas are documented in the sections above. Helix uses these schemas to validate your configuration at build time and to type-check the data flowing between steps.
Yes. While Scrapfly ships with 12 pre-built ai web scraping actions, you can layer custom logic around them inside Helix — pre/post-processing steps, conditional branches, retries, or stitching Scrapfly together with other connected tools. For deeper customization, talk to our team about Nagent's Agentic AI Lab — forward-deployed engineers who build Scrapfly-based workflows tailored to your business.