Skip to content
Log inJoin waitlistJoin the waitlist

The VarlyReview bot

Varly helps online businesses improve their own websites. When a Varly customer asks how their site compares with a few competitors, our bot visits those sites once, like a polite visitor, and reads a handful of public pages, and on a shop it looks at the cart page.

How to recognize it

The bot adds this to the user agent of a normal browser:

VarlyReview/1.0 (+https://varly.ai/bot)

It does not hide behind other user agents, rotate IP addresses or use tricks to get past bot protection.

What it does

  • It visits only when a Varly customer starts a comparison, or asks for competitor suggestions (then it reads only robots.txt and the homepage). There is no regular crawl of the web.
  • It opens at most 12 public pages of your site per comparison, such as the homepage, a category page, one product page and your shipping, returns and contact pages. Each page is opened on a desktop and a mobile screen size.
  • It asks for at most one page per second, or slower if your robots.txt sets a crawl delay.
  • It visits your site at most once a day. If another comparison includes your site within 24 hours, Varly uses the pages it already read instead of visiting again. If your site asked it to stay away, it does not try again within that time either.
  • It reads the visible text and takes screenshots, to show what your site offers: for example your free-shipping amount, return period, payment methods or review score.
  • On an online shop, it may add one product to a cart to see the cart page, with the first available option and a quantity of one. It does this in a separate browser session that it throws away right after, so nothing stays in a cart.
  • While it does that, it blocks known analytics and ad tracking requests, so its visit does not count as a shopper in your add-to-cart numbers.
  • The result is private to the Varly customer who asked for it. It is never shown to other customers or published, and Varly deletes old comparisons automatically.

What it never does

  • Log in, create an account or open account pages.
  • Type anything, search, or submit forms. The only button it presses is the add-to-cart button on one product page.
  • Go to checkout or pay. It never opens a checkout or payment page, also not when a site tries to send it there.
  • Add more than one product, or open the cart when your robots.txt asks bots not to.
  • Solve or get around a CAPTCHA or other bot protection. If your site blocks it, it stops.
  • Collect personal data.

How to block it

The bot follows robots.txt. It reads the rules for VarlyReview first, and the rules for all bots (*) if there are none for it. To keep it off your whole site, add:

User-agent: VarlyReview
Disallow: /

You can also block only some parts, for example Disallow: /blog/. If your robots.txt cannot be read because your server returns an error, the bot does not visit at all.

The cart follows the same rules. To keep the bot out of your cart only, add Disallow: /cart (or the path of your cart page) for VarlyReview or for all bots. Many shop platforms, such as Shopify, already do this by default; then the bot does not open the cart at all. It also skips the cart when the add-to-cart button sends to a path that robots.txt does not allow.

It also respects an opt-out from text and data mining, as meant in the EU copyright rules. Any one of these is enough:

  • the HTTP header tdm-reservation: 1
  • the tag <meta name="tdm-reservation" content="1"> on your pages
  • a file at /.well-known/tdmrep.json that reserves your pages

When a site opts out, the Varly customer sees that the site was skipped, and the bot does not try again in another way.