Docs/Knowledge Base/Add Websites & URLs

Add Websites & URLs

The website crawler lets you point the agent at any public URL — your main site, help centre, blog, or documentation — and it will automatically scrape, chunk, and index the content. No manual copy-paste required. You can also set up automatic recrawling so the agent stays up to date when your website changes.

Adding a website source

Open Knowledge Base → Websites

In your agent's left menu, click Knowledge Base and select the Websites & URLs tab.

Enter the starting URL

Paste the URL you want to crawl — for example https://yoursite.com/help or the root of your site. You can also paste a sitemap URL directly.

Configure crawl scope

Choose how many pages to crawl (see crawl scope options below).

Set advanced options (optional)

Add URL patterns to include or exclude, or configure recrawl frequency.

Click Add

Crawling starts immediately. The status panel shows pages crawled and pages indexed as the job runs.

Crawl scope options

Mode	What it crawls
Single Page	Only the exact URL you entered. No links are followed. Use for a specific help article or landing page.
All Pages on Domain	Follows every internal link found on the starting page and recursively crawls the entire domain. Best for indexing a full website or help centre.
Custom Depth	Follows links up to N levels deep from the starting URL. Level 1 = just the starting page and its direct links; Level 2 = those pages and their links, and so on. Use when you want to crawl a section of a site without crawling the whole domain.

URL filtering (include and exclude patterns)

When crawling a large site you may want to limit which pages are indexed. Use patterns to control this:

•Include patterns — only crawl URLs that match these patterns. For example, /help/ to index only help-centre pages.
•Exclude patterns — skip URLs that match these patterns. For example, /blog/ to avoid indexing blog articles, or /author/ to skip author profile pages.
•Patterns are matched against the URL path. Multiple patterns are supported.

💡

Use include patterns when you want to index a specific section of a site (e.g. only /docs/ or /support/). Use exclude patterns when you want everything except certain sections.

Sitemap support

Sitemaps tell the crawler exactly which pages exist on a site, making crawling faster and more complete:

•Auto-detect sitemap — when enabled, the crawler automatically looks for sitemap.xml at the domain root (e.g. https://yoursite.com/sitemap.xml) and uses it as the page list.
•Manual sitemap URL — if your sitemap is at a non-standard path, you can enter its URL directly.
•Using a sitemap is recommended for large sites — it ensures the crawler finds every page rather than relying on link-following.

Automatic recrawling

Enable recrawling to keep your agent up to date when your website content changes. Choose a frequency that matches how often your site is updated:

Frequency	When to use
Manual	Crawl only when you click Recrawl. Use for sites that rarely change.
Daily	Recrawls every 24 hours. Use for actively maintained blogs, news sections, or frequently updated help centres.
Weekly	Recrawls every 7 days. Suitable for most business websites with regular but not daily updates.
Monthly	Recrawls every 30 days. Use for stable documentation or landing pages that change infrequently.

When a recrawl runs, pages are re-fetched and re-indexed. The old vectors for that source are replaced with fresh ones, so outdated content is automatically removed.

Crawl statistics

Each website source shows two numbers:

•Pages crawled — the total number of URLs the crawler attempted to fetch.
•Pages indexed — the number of pages that were successfully scraped, chunked, and added to the vector store. Pages that returned errors or had no text content are counted as crawled but not indexed.

How the scraper handles protected sites

Many modern websites use anti-bot protection. The crawler uses a tiered approach to handle them automatically, starting with the simplest method and escalating as needed:

Tier	Method	Handles
1	Standard HTTP request	Most public websites with no protection.
2	TLS fingerprint impersonation	Sites that block simple bots but allow real browsers — handles most Cloudflare-protected sites at no extra cost.
3	Residential proxy (Scrape.do)	Sites that block even TLS impersonation. Uses a residential IP network. Limited to 50 pages per source.
4	Advanced proxy (ScrapingAnt)	Last resort for the most heavily protected sites.

This escalation happens automatically — you do not need to configure it. The agent selects the lowest-cost tier that successfully retrieves the page.

What can and cannot be crawled

Can be crawled	Cannot be crawled
Publicly accessible web pages	Pages behind login or a paywall
Cloudflare-protected sites (handled automatically)	Content loaded entirely by JavaScript with no server-rendered HTML
Sitemaps and linked pages	PDFs linked on web pages (add them as file sources instead)
Help centres and documentation sites	Pages that require CAPTCHA completion
Blog articles and landing pages	Private or password-protected pages

⚠️

If a website source shows 0 pages indexed after crawling, the site may be blocking crawlers entirely. Try adding the content as a file upload instead, or contact support.

Upload Files

Manual Q&A

Was this page helpful?