Website
Add a website or section when it doesn't have a sitemap. Biel.ai follows links from your starting page to find pages to index.
Add a website
- In the Biel.ai dashboard, select your project.
- Go to Settings > Sources > Website.
- Enter the starting URL, such as
https://docs.example.com/user/. Click Add website to add another site or section. - If the site requires sign-in, open Website credentials and select saved Credentials (optional). See Sites that require sign-in.
- Click Save, then Refresh sources.
- Check the loading status and Logs in the project dashboard. When processing finishes, ask your assistant a question about an indexed page.
Choose a starting URL
Use the URL of the website or section you want to index:
| Starting URL | Pages included |
|---|---|
https://docs.example.com/ | The homepage and linked pages on this website |
https://docs.example.com/user/ | The starting page and linked pages under /user/, such as /user/topic/ |
The second example excludes /users/, /technical/, and other sections. Links to other websites are not followed. Use the address shown in your browser after any redirects.
Pages must be reachable through links from the starting page. Content that requires clicking a button or scrolling to load more results may not be found. To index downloads such as PDFs, use a file source.
Filter URLs
In the Website tab, use Filter URLs to include or exclude pages. These rules apply to all Website and sitemap sources in the project.
Allow the starting page and any pages needed to reach your content. For example, excluding a section's overview page can prevent Biel.ai from finding its linked articles. See URL filter rules and examples.
Sites that require sign-in
Private sources require Professional or higher. Create Web login credentials, then select them under the source's Website credentials. Biel.ai uses those credentials for the starting page and linked pages.
If the homepage is public but your documentation requires sign-in, use a protected page as the credentials' Login URL.
Saved credentials let Biel.ai read your content. To restrict who can query it, configure access to your assistant.
Troubleshooting
Pages are missing
- Check that the starting page links to the missing pages, directly or through other pages.
- Check that the pages are inside the website or section you selected.
- Review Filter URLs for rules that exclude the pages or the links leading to them.
- For a site that requires sign-in, check that the selected credentials can open the pages.
The crawl failed
Open Logs to find the page that could not be read. Fix its URL or credentials, or exclude the unavailable page with Filter URLs, then click Save and Refresh sources.
If a crawl cannot finish, previously indexed content stays available. A project has no indexed content until its first crawl finishes successfully.
Use the Sources API
Add Website sources with the Add source endpoint using JSON:
{
"websites": ["https://docs.example.com/user/"]
}
For a site that requires sign-in, pass an object with url and the saved credentials' connector ID. See Use credentials with the Sources API.
After adding sources, sync the project to index their pages. To update one page later, use the page recrawl endpoint. The page must be inside the Website or section you added.