Skip to main content

Websites Datasources

Beak AI Studio enables agents to use websites as data sources, supporting integration with public and private sites, web portals and SharePoint sites.

Supported Website Integrations​

  • Public Websites:
    • Extract text, tables, and structured data from publicly accessible web pages.
  • Private Sites & Web Applications:
    • Connect to internal portals, dashboards, and authenticated web apps.
  • SharePoint Sites:
    • Retrieve documents and data from SharePoint web portals (requires authentication).

Integration Methods​

  • Web Scraping:
    • Define extraction rules using CSS selectors or XPath.
    • Supports both static and dynamic (JavaScript-rendered) content where permitted.
  • SharePoint Integration:
    • Authenticate and access SharePoint sites using OAuth2 or API tokens.
  • Authentication:
    • OAuth2, Basic Auth, or session cookies for private sites.
    • For web apps requiring login, provide credentials or use SSO where supported.

Supported Protocols​

  • HTTP and HTTPS (secure connections recommended)
  • WebSocket and other protocols are not currently supported for website datasources.

Configuration Requirements​

  • URL or Endpoint:
    • Enter the full URL of the website, web app, API, or SharePoint site.
  • Authentication Details:
    • Provide required credentials (OAuth2, username/password, etc.).
  • Permissions:
    • Ensure the AI Studio has access to private/internal sites (may require VPN or network configuration).

Limitations & Considerations​

  • robots.txt Compliance:
    • Scraping respects robots.txt rules; some sites may block automated access.
  • CAPTCHAs & Anti-Bot Measures:
    • Sites with CAPTCHAs or advanced anti-bot protections may not be fully accessible.
  • Authentication Complexity:
    • Some web apps require multi-factor or SSO authentication, which may need additional setup.
  • Rate Limits:
    • APIs and websites may enforce rate limits; configure agents to avoid excessive requests.
  • Legal & Ethical Use:
    • Only scrape or access data you are authorized to use.

How to Add a Website Datasource​

  1. Navigate to Data Sources

    • In Studio, go to the "Data Sources" tab.

  2. Add New Website Source

    • Click "Add Data Source" and select "Website".

  3. Configure Website Access

    • Enter the URL.
    • Provide authentication details if required.

Example Use Cases​

  • E-commerce Monitoring: Scrape product prices and availability from public retail sites.
  • Internal Dashboard Integration: Extract KPIs from a private business dashboard (with authentication).
  • News Aggregation: Ingest articles from multiple news sites via RSS feeds.

Best Practices​

  • Use authenticated access for private or sensitive data.
  • Respect site terms of service and legal requirements.
  • Monitor for changes in site structure that may affect extraction rules.

For advanced integration or troubleshooting, refer to your organization's IT policies or contact your administrator.