Skip to main content

Website

Add a website or section when it doesn't have a sitemap. Biel.ai follows links from your starting page to find pages to index.

Add a website​

  1. In the Biel.ai dashboard, select your project.
  2. Go to Settings > Sources > Website.
  3. Enter the starting URL, such as https://docs.example.com/user/. Click Add website to add another site or section.
  4. If the site requires sign-in, open Website credentials and select saved Credentials (optional). See Sites that require sign-in.
  5. Click Save, then Refresh sources.
  6. Check the loading status and Logs in the project dashboard. When processing finishes, ask your assistant a question about an indexed page.

Choose a starting URL​

Use the URL of the website or section you want to index:

Starting URLPages included
https://docs.example.com/The homepage and linked pages on this website
https://docs.example.com/user/The starting page and linked pages under /user/, such as /user/topic/

The second example excludes /users/, /technical/, and other sections. Links to other websites are not followed. Use the address shown in your browser after any redirects.

Pages must be reachable through links from the starting page. Content that requires clicking a button or scrolling to load more results may not be found. To index downloads such as PDFs, use a file source.

Filter URLs​

In the Website tab, use Filter URLs to include or exclude pages. These rules apply to all Website and sitemap sources in the project.

Allow the starting page and any pages needed to reach your content. For example, excluding a section's overview page can prevent Biel.ai from finding its linked articles. See URL filter rules and examples.

Sites that require sign-in​

Private sources require Professional or higher. Create Web login credentials, then select them under the source's Website credentials. Biel.ai uses those credentials for the starting page and linked pages.

If the homepage is public but your documentation requires sign-in, use a protected page as the credentials' Login URL.

Saved credentials let Biel.ai read your content. To restrict who can query it, configure access to your assistant.

Troubleshooting​

Pages are missing​

  • Check that the starting page links to the missing pages, directly or through other pages.
  • Check that the pages are inside the website or section you selected.
  • Review Filter URLs for rules that exclude the pages or the links leading to them.
  • For a site that requires sign-in, check that the selected credentials can open the pages.

The crawl failed​

Open Logs to find the page that could not be read. Fix its URL or credentials, or exclude the unavailable page with Filter URLs, then click Save and Refresh sources.

If a crawl cannot finish, previously indexed content stays available. A project has no indexed content until its first crawl finishes successfully.

Use the Sources API​

Add Website sources with the Add source endpoint using JSON:

{
"websites": ["https://docs.example.com/user/"]
}

For a site that requires sign-in, pass an object with url and the saved credentials' connector ID. See Use credentials with the Sources API.

After adding sources, sync the project to index their pages. To update one page later, use the page recrawl endpoint. The page must be inside the Website or section you added.