XnoidBot

XnoidBot is the web crawler behind XnoidBrowse, an independent search engine. It continuously crawls the public web to discover, fetch, and index pages so they appear in search results.

The bot exists because search engines need fresh data. Every time someone searches on XnoidBrowse, the results come from pages that XnoidBot has crawled and indexed. Without it, the search engine would have nothing to show.

XnoidBot follows links from page to page, starting from known URLs and discovering new ones as it goes. For each page it visits, it extracts the title, headings, body text, and links — then stores a lightweight index entry in its database. Raw HTML is discarded after extraction.

The crawler respects robots.txt directives on every domain it visits. If a site specifies a Crawl-delay, XnoidBot honors it. Sites can also block XnoidBot entirely via robots.txt.

For pages that rely on JavaScript to render content (detected via <noscript> tags or very thin static HTML), XnoidBot uses a headless browser to render the page before extracting content.