RTF-Research: TransactionTree's research crawler
RTF-Research is the web crawler for TransactionTree's Retailer Technology Footprint research. It reads a small number of public business pages, such as company websites, newsrooms, customer stories and job posts, to learn which retail technology systems companies use.
How to recognize it
- User-Agent:
RTF-Research/0.1 (+https://rtf-research.transactiontree.io) - robots.txt token:
RTF-Research - Where requests come from: TransactionTree's Cloudflare account. The addresses belong to Cloudflare's network and may be shared with other Cloudflare customers, so please identify RTF-Research by its User-Agent rather than by IP address.
How it behaves
- It reads your robots.txt before fetching and re-reads it at least every 24 hours. It
follows rules for
RTF-Research, for all user agents (*), and for AI crawlers such as GPTBot, ClaudeBot and CCBot. If your robots.txt blocks AI crawlers from a page, RTF-Research does not fetch that page either. - It fetches public pages only. It does not log in, and it does not get around paywalls, CAPTCHAs or bot challenges.
- It sends at most one request per second to a site, and goes slower if your site asks for that.
- It stops fetching from your site if it receives a 401, 402, 403 or 429 response, or a challenge page.
- It always uses the User-Agent above and does not change user agents or IP addresses to get around limits.
What it keeps
- Payment card numbers, credentials and personal contact details are removed before anything is stored.
- The full text of a page is deleted within 30 days.
- After that, TransactionTree keeps only the business fact, the page address, a fingerprint of the page content and a short excerpt of at most 300 characters.
How to opt out
Add this to your robots.txt:
User-agent: RTF-Research Disallow: /
Or email us at rtf-research@transactiontree.io. We will:
- acknowledge your request within 10 business days;
- stop fetching your site within 48 hours;
- delete what we hold from it within 30 days.