Skip to main content

URLs

Add individual web pages to your project. Use URLs when you need specific pages rather than entire websites.

tip

For multiple pages from the same website, use sitemaps instead.

When to use URLs​

Use URLs to:

  • Add specific pages from different websites
  • Test new content before adding full sites
  • Include only high-priority pages from larger sites
  • Add pages that aren't in sitemaps

Requirements​

  • Publicly accessible, or accessible with saved Web login credentials.
  • Full URL with protocol (e.g., https://example.com/page).
  • Page contains text content (not just images or videos).

Add URLs​

  1. In the Biel.ai dashboard, select your project.

  2. Go to Settings > Sources > URLs.

    The URLs tab of the Sources settings page

  3. Enter the full URL (including https://).

  4. Click Save.

  5. Click Refresh sources to index the pages. Check the loading status and Logs in the project dashboard.

Example URLs:

https://example.com/blog/getting-started
https://docs.example.com/api-reference
https://support.example.com/troubleshooting

For a source behind a login, open its Advanced options and choose Credentials (optional) before saving. See Private content to set up Web login credentials.

Troubleshooting​

URL not indexing​

  • Verify the URL is public, or check the selected credentials’ access.
  • Check the URL includes https://.
  • Check if the site blocks crawlers in robots.txt.

Content not appearing​

  • Wait for source processing to finish. Check the project’s Logs if it fails.
  • Test with specific questions about the page content.
  • Check if the page relies on JavaScript to render content.