Skip to main content

Sources

Configure sources to provide content for your chatbot and search widget. Sources determine what information your AI can access to answer questions.

important

Only Administrator and Maintainer roles can manage sources. See Manage roles.

Choose a source type​

URLs​

Index individual web pages like blog posts or landing pages.

For multiple pages, use sitemaps instead.

Sitemaps​

Index entire websites using XML sitemap files. Sitemaps must end with .xml.

Supports nested sitemaps for large sites.

GitHub​

Index GitHub repositories directly.

  • Public repositories: No authentication required
  • Private repositories: Requires GitHub token
  • File filtering: Control which files to include or exclude

Files remain private—widgets won't expose file links as sources.

Files​

Upload documents in these formats: PDF, CSV, Excel, Word, TXT, MD.

Files remain private—widgets won't expose file links as sources.

Add a source​

  1. Go to app.biel.ai

  2. Select Projects > your project > Settings

  3. In Sources, choose your source type:

    • URLs: Enter the complete web page URL

    • Sitemaps: Enter the sitemap URL ending in .xml

    • GitHub: Enter the repository URL: https://github.com/owner/repo

      For private repositories:

      1. Generate a GitHub personal access token with repo permissions
      2. In Advanced options, enter your GitHub token

      Filter files (optional):

      • Include patterns: *.md, docs/, README.md
      • Exclude patterns: *.log, node_modules/, .git/

    • Files: Upload documents directly

  4. Click Save

Your content will be indexed automatically. Test by asking your chatbot about the indexed content.

Filter URLs​

Restrict which URLs your chatbot can access using include/exclude patterns.

  1. Go to Settings > Sources > Restrictions > Filter URLs
  2. Add patterns:
    • Exclude /es: Skip URLs containing /es
    • Include /docs: Only index URLs containing /docs
    • Exclude ^https://example.com/old/.*$: Skip the /old/ path (regex)
  3. Click Save

Troubleshoot​

First crawl issues: Initial crawls may need manual configuration. Contact support for optimization help.