Skip to main content

URLs

Add individual web pages to your project. Use URLs when you need specific pages rather than entire websites.

tip

For multiple pages from the same website, use sitemaps instead.

When to use URLs​

Use URLs to:

  • Add specific pages from different websites
  • Test new content before adding full sites
  • Include only high-priority pages from larger sites
  • Add pages that aren't in sitemaps

Requirements​

URLs must be:

  • Publicly accessible (not behind login)
  • Include the protocol (https://)
  • Contain searchable text content

Add URLs​

  1. Go to app.biel.ai
  2. Select Projects > your project > Settings > Sources
  3. Select URLs
  4. Enter the complete URL (including https://)
  5. Click Save

Example URLs:

https://example.com/blog/getting-started
https://docs.example.com/api-reference
https://support.example.com/troubleshooting

Filter URLs​

Control which URLs get indexed using include and exclude patterns:

  1. Go to Projects > your project > Settings > Sources > Restrictions > Filter URLs
  2. Add patterns:

Exclude patterns - Skip URLs that match:

  • **/es/** - Skip Spanish content
  • **/old/** - Skip archived content
  • **/_internal/** - Skip internal pages

Include patterns - Only index URLs that match:

  • **/docs/** - Only documentation
  • **/blog/** - Only blog posts
  • **/help/** - Only help content

Regex patterns - Advanced filtering:

  • ^https://example\.com/old/.*$ - Skip /old/ path
  • ^https://[^/]+/(en|fr)/.*$ - English and French only
  1. Click Save

Best practices​

  • Use canonical URLs (primary version of each page)
  • Start with 10-20 key pages
  • Test one URL at a time initially
  • Use sitemaps for multiple related pages

Troubleshooting​

URL not indexing

  • Check that the URL is publicly accessible
  • Verify the URL format includes https://
  • Check if the site blocks crawlers in robots.txt

Content not appearing

  • Wait 5-10 minutes for processing
  • Test with specific questions about page content
  • Check if the page uses dynamic JavaScript content

Need help? Contact support at support@biel.ai if you experience issues with content processing or indexing takes too long.