Sitemaps

Add entire websites using XML sitemap files. Sitemaps efficiently index large amounts of content from a single domain.

When to use sitemaps

Use sitemaps to:

• Add complete documentation sites or blogs
• Index hundreds or thousands of pages efficiently
• Automatically discover new pages as they're added
• Handle structured website content

Requirements

Sitemaps must:

• End with .xml
• Be publicly accessible
• Follow the XML Sitemap Protocol

Supported types:

• Standard sitemaps (single XML files)
• Sitemap indexes (reference multiple sitemaps)
• Nested sitemaps (linked sitemaps)

Add sitemaps

1. In the Biel.ai dashboard, select your project.

2. Go to Settings > Sources > Sitemaps.

   

3. Enter the sitemap URL (must end in .xml).

4. Click Save.

Example sitemap URLs:

https://docs.example.com/sitemap.xml
https://blog.example.com/sitemap.xml
https://help.example.com/sitemap_index.xml

Find sitemaps

Common locations:

https://yoursite.com/sitemap.xml
https://yoursite.com/sitemap_index.xml
https://yoursite.com/sitemaps/sitemap.xml

Platform-specific patterns:

Platform	Sitemap URL
Docusaurus	https://yoursite.com/sitemap.xml
GitBook	https://yourorg.gitbook.io/sitemap.xml
WordPress	https://yoursite.com/wp-sitemap.xml

For other platforms, check robots.txt at https://yoursite.com/robots.txt for sitemap declarations.

Filter URLs

Use URL filters to decide which pages from your sitemaps Biel indexes. Each filter has a regular expression pattern and an action: Include or Exclude. Biel compares the pattern with the complete URL, such as https://docs.example.com/guides/install/.

1. Go to Projects > your project > Settings > Sources > Sitemaps.

2. Add one or more filter rules under Filter URLs.

   

Choose an action

• If no include rules exist, all URLs are included by default.
• If include rules exist, a URL is kept when it matches any include pattern.
• An exclude rule always takes precedence: a matching URL is skipped even when it also matches an include rule.

Filters apply to URLs from every sitemap configured in the project.

Write a URL pattern

Start with the part of the URL you want to match. For example, this include pattern keeps documentation paths on any configured domain:

/docs(?:/|$)

Use regular expression syntax rather than wildcard syntax: write /docs(?:/|$), not **/docs/**. Patterns are case-sensitive.

To restrict a rule to one domain, match from the beginning of the complete URL:

^https://docs\.example\.com/guides(?:/|$)

In this pattern:

• ^ means the match must start at the beginning of the URL.
• docs\.example\.com matches the domain; \. represents a literal dot.
• /guides(?:/|$) matches /guides, everything below it, and no similarly named path such as /guides-old.

You can omit ^https:// when a path pattern should apply to every configured domain.

Examples

Pattern	Action	Effect
`^https://docs\.example\.com/docs(?:/	$)`	Include
`/api(?:/	$)`	Exclude
`/deprecated(?:/	$)`	Exclude
`^https://docs\.example\.com/(?:en	fr)(?:/	$)`
`.pdf(?:$	[?#])`	Exclude
`/[0-9]+.[0-9]+.[0-9]+(?:/	$)`	Exclude

To include selected sections from different domains, create one include rule per section. For example:

Pattern	Action
`^https://docs\.example\.com(?:/	$)`
`^https://example\.com/extensions/product(?:[-/]	$)`
`^https://example\.com/api/Product(?:[./]	$)`

All other sitemap URLs are omitted because they do not match an include rule. You do not need to create exclude rules for every unrelated path.

Avoid broad catch-all rules unless they are intentional:

• An include rule containing .* includes every URL.
• An exclude rule containing .* excludes every URL.

Sitemap index files

Large sites often use index files that reference multiple sitemaps:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/docs-sitemap.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/blog-sitemap.xml</loc>
  </sitemap>
</sitemapindex>

Biel.ai automatically follows sitemap indexes and processes all referenced sitemaps.

Troubleshooting

Sitemap not found

• Check URL format (must end with .xml).
• Try common sitemap locations listed in Find sitemaps.
• Check robots.txt for sitemap declarations.

Partial indexing

• Verify pages in the sitemap are publicly accessible.
• Review URL filters for over-exclusion.
• JavaScript-heavy pages may not index properly.