TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)
R6_FeedFetcher(www.radian6.com/crawler)
CheckHost (http://check-host.net/)
YandeG 1.03
yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
SEMrushBot
Protopage/3.0 (http://www.protopage.com)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
StatoolsBot (+http://www.statools.com/bot.html)
adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
Scrapy/1.0.5 (+http://scrapy.org)
yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
XmlSitemapGenerator - http://xmlsitemapgenerator.org
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
WatchMouse/18990 (http://watchmouse.com/ ; gab)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
PercolateCrawler/4 (ops@percolate.com)
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
UnwindFetchor/1.0 (+http://www.gnip.com/)
nrsbot/5.0(loopip.com/robot.html)
SemrushBot/0.9
Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
Netvibes (http://www.netvibes.com)
Acoon v4.1.0 (www.acoon.de)
msnbot/2.0b (+http://search.msn.com/msnbot.htm).
Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
HubSpot Links Crawler 2.0 http://www.hubspot.com/
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
WebDoc (abuse-webdoc at service.moquadv.com)
coccoc
Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty.  ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
SpeedySpider - http://www.entireweb.com
Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
crawler for netopian (http://www.netopian.co.uk/)
L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor
Scrapy/1.1.1 (+http://scrapy.org)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
eBot / v.1.0a (http://alfa.elchron.cz)
Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Crowsnest/0.5 (+http://www.crowsnest.tv/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
Experibot_v1
Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
CopperEgg/RevealUptime/DallasTX(linode)
Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
Curious George - www.analyticsseo.com
GozaikBot (www.gozaik.com;webmaster@gozaik.com;www.gozaik.com/gozaikbot.html)
Pu_iN Crawler (+http://semanticjuice.com/)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
FAST Enteprise Crawler/6 (www dot fastsearch dot com)
Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
Inspingbot/1.0 (+https://www.insping.com/)
OrgProbe/0.9.4 (+http://www.blocked.org.uk)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
Feedspot http://www.feedspot.com
Zookabot/2.0;++http://zookabot.com
drupact/0.7; http://www.arocom.de/drupact
RobotsChecker/0.6 (+http://www.blocked.org.uk)
yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
Scrapy/1.0.3 (+http://scrapy.org)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
NalezenCzBot/1.0 (http://www.nalezen.cz)
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
Crawler powered by contentDetection (www.mindup.de)
PagePeeker.com (info: http://pagepeeker.com/robots)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
coccoc/1.0 (http://help.coccoc.com/)
Favicon downloader (+https://favico.be/bot.html)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
Mozilla/5.0 (compatible; theoldreader.com)
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
larbin_2.6.3 gqnmgsp@ruc.edu.cn
yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
LapozzBot/1.4 (+http://robot.lapozz.com)
Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
Mozilla/5.0 (compatible; ExaleadCloudview/6;)
ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 LinkTiger 2.0
Pixray-Seeker/1.1 (Pixray-Seeker; crawler@pixray.com)
mbot v.1.16
MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
polybot 1.0 (http://cis.poly.edu/polybot/)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
alexa v0.1.4 (http://www.openwebspider.org/)
http://arachnode.net 1.4
Photon/1.0
NetpeakCheckerBot
GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
Yandex/1.01.001 (compatible; Win16; P)
w3dt.net httphr/2.0
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
iqdb/0.1 (+http://iqdb.org/)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon (lab@evidon.com)
Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
mindUpBot (datenbutler.de)
Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
Zookabot/2.1;++http://zookabot.com
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
focusbot/1.1
Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
ImplisenseBot 1.1
Promotion_Tools_www.searchenginepromotionhelp.com
Aboundex/0.2 (http://www.aboundex.com/crawler/)
CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
LumpImageSearch/0.1 (+http://lump.co/about/bot)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
GAChecker (+http://www.gachecker.com)
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
DNSPod-reporting(http://www.dnspod.cn/reporting)
yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
PostPost/1.0 (+http://postpo.st/crawlers)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
Snapbot/1.0 (Snap Shots, +http://www.snap.com)
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
Jyxobot/1
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
Priceonomics Analysis Engine - Fetch/1.0
Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
Baiduspider+(+http://help.baidu.jp/system/05.html)
Openstat/0.1
Yandex/1.01.001 (compatible; Win16; m)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
gonzo/1[P] (+http://www.suchen.de/faq.html)
MixBot (+http://t.co/GSRLLKex24)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
ThumbSniper (http://thumbsniper.com)
Shelob (shelob@gmx.net)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
Yandex/1.01.001 (compatible; Win16; I)
FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
Scrapy/0.24.4 (+http://scrapy.org)
ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - support@admantx.com
Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
http://arachnode.net 1.2
Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
Mozilla/5.0 (compatible; evc-batch/2.0)
Orbiter/1.2 (http://dailyorbit.com/)
crawler4j (https://github.com/yasserg/crawler4j/)
Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
Scopia crawler 1.0 (+http://www.scopia.co)
yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,support@lxrmarketplace.com)
yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
gonzo1[P] +http://www.suchen.de/faq.html
Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
Sogou web spider/4.02525A
Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;bot@visvo.com)
Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34 ShoppimonAgent/1.0 (feedback+agent@shoppimon.com)
webinatorbot 1.1; +http://www.webinator.de
findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
WikiDo/1.1 (http://wikido.com; crawler@wikido.com)
yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
Lijit Crawler (+http://www.lijit.com/robot/crawler)
baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com)
Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Pompos/1.3 http://dir.com/pompos.html
Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
al_viewer (larbin2.6.3@unspecified.mail)
LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
Norton-Safeweb
WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)
Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-wherecat@moz.com)
ROR Sitemap Generator (http://www.rorweb.com)
http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
PagePeeker.com
CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
www.adressendeutschland.de
DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
librabot/2.0 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
SalesIntelligent/v1.0
larbin_2.6.2 pierre@micro-fun.ch
Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
Linguee Bot (http://www.linguee.com/bot)
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Goodzer/2.0; crawler@goodzer.com)
Acoon v4.10.5 (www.acoon.de)
CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
EasyBib AutoCite (http://autocite-info.citation-api.com/)
Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
Gigabot/2.0
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Morning Paper 1.0 (robots.txt compliant!)
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
SuperPagesUrlVerifyBot/1.0
CopperEgg/RevealUptime/LondonUK(linode)
Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; )  Firefox/1.5.0.11; 360Spider
wangling
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
RSSMicro.com RSS/Atom Feed Robot
GarlikCrawler/1.1 (http://garlik.com/, crawler@garik.com)
al_org_viewer (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
dj-research/Nutch-1.11 (analytics@@demandjump.com)
Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
SauceNAO/1.0 (+http://saucenao.com/)
Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
SSL Labs (https://www.ssllabs.com/about/assessment.html)
Mozilla/5.0 (compatible; http://alyze.info)
GigablastOpenSource/1
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1  (http://www.voila.com/)
Vorboss Web Crawler [crawl@vorboss.net]/Nutch-2.3
Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
YahooCacheSystem
search.KumKie.com
yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
Embedly +support@embed.ly
stq_bot (+http://www.searchteq.de)
MSRBOT
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
NG-Search/0.86 (+http://www.ng-search.com)
ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
TwengaBot/1.1 (+http://www.twenga.com/bot.html)
WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
Mozilla/5.0 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Page Analyzer v4.0 ( http://www.ranks.nl/ )
web_bh (larbin2.6.3@unspecified.mail)
findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
PagesInventory (robot http://www.pagesinventory.com)
Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453 Safari/537.36
404 Checker [http://www.404checker.com/user-agent]
CopperEgg/RevealUptime/
Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
CopperEgg/RevealUptime/TokyoJapan
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
GoSquared-Status-Checker/0.2
Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; adbeat_bot; +support@adbeat.com; support@adbeat.com)
linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
LSSRocketCrawler/1.0 LightspeedSystems
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
woobot/2.0
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
SEO Consulting; Redirect Checker Tool V.02; IP:
Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru)
Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.com
GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
Pinterest/0.2 (+http://www.pinterest.com/)
CopperEgg/RevealUptime/AtlantaGA(linode)
OdklBot/1.0 (klass@odnoklassniki.ru)
Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
Sogou web spider/4.05252A
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
NG/2.0
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
StackRambler/2.0 (MSIE incompatible)
Baiduspider+(+http://www.baidu.jp/spider/)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
Woko robot 3.0
Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - support@admantx.com
Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
Acoon v4.10.4 (www.acoon.de)
Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
Readability/740ec9 - http://readability.com/about/
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
eCommerceBot (http://www.ehandel.se/botinfo.html)
Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
Nymesis/2.0 (http://nymesis.com)
CopperEgg/RevealUptime/OregonUSA
uclassify.com/1.0
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
AdnormCrawler www.adnorm.com/crawler
Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
Mozilla/4.0 (xcm@huaweisymantec.com)
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
Sogou web spider/4.025251
SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin@unspecified.mail)
Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
mozilla/5.0 (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
seebot/2.0 (+http://www.seegnify.com/bot)
bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
hledejLevne.cz/2.0
page_verifier (http://www.securecomputing.com/goto/pv)
url_test (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Is is up? (+http://isitup.org)
Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)
TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
LivelapBot/0.2 (http://site.livelap.com/crawler)
HubSpot Crawler 1.0 http://www.hubspot.com/
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
Scrapy/0.24.6 (+http://scrapy.org)
FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Mozilla/5.0 (compatible; seplinkbot/1.0 )
Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
MaxPoint Bot (+http://www.maxpoint.com)
Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
cg-eye interactive
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-05@moz.com)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
HolmesBot (http://holmes.ge)
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; no-reply@alexa.com)
Mediapartners-Google
Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
larbin_2.6.2 kalou@kalou.net
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
BlogPulseLive (support@blogpulse.com)
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
ImplisenseBot 1.0
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
gonzo2[P] +http://www.suchen.de/faq.html
Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, support@lxrseo.com)
Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
gonzo2[p] (+http://www.suchen.de/faq.html)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
WatchMouse/18990 (http://watchmouse.com/ ; uk)
yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
VeBot (+http://www.veinteractive.com/vebot)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)
wscheck.com/1.0.0 (+http://wscheck.com/)
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0
Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
Kyoto-Crawler/n1.0 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
seo-nastroj.cz
LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
CSS Certificate Spider (http://www.css-security.com/certificatespider/)
MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
BLEXBot
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
HeartRails Robot/0.1 (http://www.heartrails.com)
Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
Castabot/0.1 (+http://topixtream.com/)
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
istellabot-nutch/Nutch-1.10
Mail.RU/2.0
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
LexxeBot/1.0 (lexxebot@lexxe.com)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
it2media-domain-crawler/1.0 on crawler-prod.it2media.de
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
checkgzipcompression.com robot
Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
RankurBot/3.3 (+http://rankur.com)
L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
oBot
Snappy/2.0 ( http://www.urltrends.com/ )
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)
istellabot/Nutch-1.11
GetintentCrawler getintent.com
Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
Microsearch.ru/1.0; http://microsearch.ru/webmasters
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
FeedlyBot/1.0 (http://feedly.com)
Clickagy Intelligence Bot v2
GetURLInfo/1.0
DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
Scrapy/1.1.0 (+http://scrapy.org)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; VSAgent/1.2)
Experibot_v1 [goo.gl/n6zrAf]
Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
boitho.com-robot/1.0
yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
RelateIQ Crawler www.relateiq.com
Mozilla/4.0 (compatible; Netcraft Web Server Survey)
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
Scrubby/2.2 (http://www.scrubtheweb.com/)
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
bitlybot/3.0 (+http://bit.ly/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
agentslug.com - website monitoring tool
SafeDNSBot (https://www.safedns.com/searchbot)
Feedbin
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
baypup/colbert (Baypup; http://sf.baypup.com/webmasters; jason@baypup.com)
MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
CatchBot/2.0; +http://www.catchbot.com
Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
Quora Link Preview/1.0 (http://www.quora.com)
radian6_default_(www.radian6.com/crawler)
DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
sogou spider
Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Scooter/3.3
Zookabot/2.5;++http://zookabot.com
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
es_com_viewer (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
KD Bot
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
Superarama.com-Tarama-Botu-v.01
Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
MnoGoSearch/3.3.9
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
JyxobotRSS/0.06
Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
SentiBot www.sentibot.eu (compatible with Googlebot)
Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
IDG/IT (http://spaziodati.eu/)
LinkAider (http://linkaider.com/crawler/)
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
bitlybot
Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
WWW::LayeredExtractor::Handler::Feed/0.01
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
Mozilla/5.0 (compatible; Goodzer/1.0)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)
Facebot/1.0
SafeAds.xyz bot
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
scrapyproject (+http://www.profound.net)
Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de)
Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/  ; Orgby.com Search Engine)
Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
datagnionbot (+http://www.datagnion.com/bot.html)
CopperEgg/RevealUptime/TokyoJP(linode)
MB-SiteCrawler
WatchMouse/18990 (http://watchmouse.com/ ; liz)
HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: m.lyashkov@targetix.net)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
Nusearch Spider (www.nusearch.com)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
Orbiter/1.3 (http://dailyorbit.com/)
CCBot/2.0
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Dex Social Bot
yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4  Firefox/3.5.3
SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)
Zemanta Aggregator/0.9 +http://www.zemanta.com
Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
holmes/3.12.4 (http://morfeo.centrum.cz/bot)
WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
ZoomInformation Bot
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)
Mozilla/5.0 (compatible; HomeTags/1.0;  http://www.hometags.nl/bot)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
WatchMouse/18990 (http://watchmouse.com/ ; it)
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
Readability/1900e6 - http://readability.com/about/
Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
FeedBucket/1.0 (+http://www.feedbucket.com)
topster.de HTTP-Header 1.0
MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
NETCRAFT
Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
WebCookies/1.0 (+http://webcookies.info/faq/#agent)
netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
CopperEgg/RevealUptime/DublinIEUSA
StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
WebWatch/Robot_txtChecker
Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
Toweyabot: toweya.com
Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
FeedCatBot/3.0 (+http://www.feedcat.net/)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; libot@linkedin.com)
R6_CommentReader(www.radian6.com/crawler)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
Riddler (http://riddler.io/about.html)
Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
Favicon downloader (http://favicon.netk6.com/)
findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
GetFoundBot (+http://www.getfound.cz/getfoundbot/)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate
Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
COMODOSpider/Nutch-1.2
Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
psbot-image (+http://www.picsearch.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
larbin_2.6.4 (atyzos@yahoo.com)
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
BacklinkCrawler (http://www.backlinktest.com/crawler.html)
Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
Tools4noobs.com/1.0 Spider
yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
istellabot/t.1
Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
SemrushBot/Nutch-1.5-SNAPSHOT
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
CopperEgg/RevealUptime/Oregon(aws)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
dubaiindex (adressendeutschland.de)
Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact info@netcraft.com)
Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
CopperEgg/RevealUptime/FrankfurtGermany
Scrapy/1.1.2 (+http://scrapy.org)
yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
Surphace Scout&v4.0 - scout at surphace dot com
yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
MetaGeneratorCrawler/1.1 (www.metagenerator.info)
Porkbun/Mustache (Website Analysis; http://porkbun.com; tech@porkbun.com)
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Ruby, link_thumbnailer
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com
Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
DataparkSearch/4.53 (+http://dataparksearch.org/bot)
Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; collection@infegy.com)
Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
domainsbot (+http://www.domainsbot.com)
WEPA/3.1 (http://www.wepa.com/; webmaster@wepa.com)
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
192.comAgent
JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
Scrapy/1.0.1 (+http://scrapy.org)
seegnifybot/1.0.0 (http://www.seegnify.com/bot)
omgili/0.5 +http://omgili.com
Slack-ImgProxy 149 (+https://api.slack.com/robots)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
www.integromedb.org/Crawler
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 collection@infegy.com
Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
Gigabot/1.0
page_test (larbin2.6.3@unspecified.mail)
istellabot/Nutch-1.10
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
ldspider (http://code.google.com/p/ldspider/wiki/Robots)
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
facebookplatform/1.0 (+http://developers.facebook.com)
Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
MnoGoSearch/3.3.15
Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
Web-sniffer/1.1.0 (+http://web-sniffer.net/)
Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
50.nu/0.01 ( +http://50.nu/bot.html )
Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1
Comodo-Certificates-Spider
Iframely/0.9.8 (+https://iframely.com/;)
MnoGoSearch/3.3.12
CopperEgg/RevealUptime/FrankfurtEU
Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
Mozilla/5.0 (compatible; JobKereso;  +http://www.kozvetlen-allasok.hu/robot.jsp  info@kozvetlen-allasok.hu)
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;)
AppleNewsBot
classbot (+http://allclasses.com)
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
CopperEgg/RevealUptime/TokyoJP
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
CJNetworkQuality; http://www.cj.com/networkquality
psbot-page (+http://www.picsearch.com/bot.html)
DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
Pingdom GIGRIB (http://www.pingdom.com)
Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)
yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html
The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/; admin@lemurproject.org)
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)
copyright sheriff (+http://www.copyrightsheriff.com/)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
Touche (+http://www.touche.com.ve)
Panopta v1.1
Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
Zookabot/2.2;++http://zookabot.com
yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
it2media-domain-crawler/2.0
yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.386)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
CopperEgg/RevealUptime/AtlantaGAUSA
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html
Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua
Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
Google-Calendar-Importer
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Websnapr/3.0 Safari/534.34
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 PTST/276
WeSEE
librabot/2.0 (+http://academic.research.microsoft.com/)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276
Cliqzbot/0.1 (+http://cliqz.com +cliqzbot@cliqz.com)
Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
Pinterest/0.1 +http://pinterest.com/
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)
Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
Iframely/0.6.0 (+http://iframely.com/;)
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
Mozilla/5.0 (compatible; U; Hootsuite-WebFeed/1.0; mailto:abuse@hootsuite.com)
RankFlex.com Webspider
TipTop http://feeltiptop.com
vebidoobot
Mozilla/5.0 (compatible; Dow Jones Searchbot)
Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Ara.com.tr AraBot 1.0
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
downnotifier.com monitoring
A6-Indexer
Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)
MetaURI API/2.0 +metauri.com
Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
Mozilla/5.0 Project 25499 (project25499.com)
COMODOspider/Nutch-1.0
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
Uptime.com_(http://uptime.com/)
AcoonBot/4.10.5 (+http://www.acoon.de)
BDFetch
20090717_aa_page (larbin2.6.3@unspecified.mail)
Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; support@meta-spinner.de)
Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)
Putin (+http://semanticjuice.com/)
Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)
WatchMouse/18990 (http://watchmouse.com/ ; ny)
bitlybot/2.0
yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
Webshot/0.9 ( http://www.shrinktheweb.com/ )
MetaURI API +metauri.com
Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)
Google-Adwords-Instant (+http://www.google.com/adsbot.html)
yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/277
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
HyperZbozi.cz Feeder/3.1
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Google-AdWords-Express
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
SMTBot (similartech.com/smtbot)
Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)
Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) web-capture.net/0.9 Safari/537.21
Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html find@mtld.mobi)
Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
iZSearch.com
Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)
ruijie (larbin@unspecified.mail)
CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)
ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
WordPress.com; https://botd.wordpress.com
Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)
ShowyouBot (http://showyou.com/crawler)
Yoleo Consumer v0.2
Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 YaBrowser/15.12.0.6151 Safari/537.36 Yandex.Translate
Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
CopperEgg/RevealUptime/DublinIE(aws)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html
RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)
Virusdie crawler/2.1
findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
Google_Analytics_Snippet_Validator
SurcentroBot
Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
panscient.com
Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
GOFORITBOT ( http://www.goforit.com/about/ )
CopperEgg/RevealUptime/SydneyAustralia
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
MetaCommentBot; http://metacomment.io/about
Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
Twikle/1.0 , http://twikle.com , contact@twikle.com
yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
Mozilla/4.0 (compatible;  Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Comodo-Webinspector-Crawler 2.1
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 KimonoLabs/0.2
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
Scrapy/0.24.5 (+http://scrapy.org)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
Port Monitor check service 1.0 | http://www.port-monitor.com
Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
SolomonoBot/1.04 (http://www.solomono.ru)
Easy-Thumb (https://www.easy-thumb.net/)
Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
nic.it crawler v.1.221115 (7.35.0)
Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)
Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)
Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/43.0
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)
Web-sniffer/1.0.31 (+http://web-sniffer.net/)
woobot
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0
AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)
adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)
Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
EasyBib AutoCite (http://content.easybib.com/autocite/)
yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; woriobot +http://worio.com)
Mozilla/4.0 (compatible;  Vagabondo/4.0; http://webagent.wise-guys.nl/)
SniffRSS/0.5beta (+http://www.blogator.com/)
woobot/1.1
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) WordPress.com mShots Safari/537.36
ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com - support@admantx.com
Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)
Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
UptimeDog Robot (www.uptimedog.com)
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact abuse@inpwrd.com for information)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276
ServiceUptime.robot
Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
SEO Browser
Leikibot/1.0 (+http://www.leiki.com)
yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) http://yacy.net/bot.html
AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1
Seobility (SEO-Check; http://bit.ly/1dJuuzs)
Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
Site24x7
websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)
Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)
HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)
lmspider (lmspider@scansoft.com)
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage) Gecko/2009021910 Firefox/3.0.7
Mozilla/5.0 (compatible; Najdi.si/3.1)
larbin_2.6.3 (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)
RED/1 (https://redbot.org/)
Twitterbot/1.0
Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)
Open Web Analytics Bot 1.5.4
JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; webmaster@elefent.eu)
Genderanalyzer/1.0
elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)
Yandex/1.01.001 (compatible; Win16; H)
crawler4j for XQuery
Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)
Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
AskQuickly v2 (http://askquickly.org/)
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
Miniflux (http://miniflux.net)
Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
Google favicon
Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-marktplatz.de/spider; spider@mia-marktplatz.de)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com
Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
wp.com feedbot/1.0 (+https://wp.com)
Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
Slack-ImgProxy (+https://api.slack.com/robots)
Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-headers.online-domain-tools.com)
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
WGETbot/1.0 (+http://wget.alanreed.org)
yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0
yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
Link Valet Online 1.2
Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
NeutrinoAPI/2.0.1
thumbshots-de-bot (+http://www.thumbshots.de/)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
argenfybot
MergadoBot
Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
CopperEgg/RevealUptime/SaoPauloBrazil
Mozilla/4.0 (compatible;  Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)
all_web1 (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)
Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)
Mediatoolkitbot (info@mediatoolkit.com)
OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0
fastbot crawler beta 2.0 (+http://www.fastbot.de)
Sogou Web Spider
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
XoviBot/1.0
Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-link-checker.online-domain-tools.com)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)
yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
ZnajdzFoto/ImageBot 2.0b
Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )
Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2014)
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36 Yandex.Translate
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)
Kemvibot/1.0 (http://kemvi.com)
Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)
yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
infohelfer/1.1.0 (http://www.infohelfer.de/)
yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html
SeoCheckBot (FischerNetzDesign Seo Checker, info@fischernetzdesign.de)
netEstate NE Crawler (+http://www.website-datenbank.de/)
Mail.Ru/1.0
binlar_2.6.3 (larbin2.6.3@unspecified.mail)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
bot.wsowner.com/1.0.0 (+http://wsowner.com/)
Ronzoobot/1.5 (http://www.ronzoo.com/about/)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)
Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/4.0 (compatible;  MSIE 5.01; GomezAgent 2.0; Windows NT)
Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)
yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
Online Virus Scanner: http://tools.geek-tools.org
DwnldBot (+http://dwnld.me)
Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)
taptubot *** please read http://www.taptu.com/corp/taptubot ***
yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en) http://yacy.net/bot.html
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta10
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)
Cloud mapping experiment. Contact research@pdrlabs.net
emefgebot/beta (+http://emefge.de/bot.html)
DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)
WPScan v2.9 (http://wpscan.org)
Ronzoobot/1.6 (http://www.ronzoo.com/about/)
livedoor ScreenShot/0.10
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)
yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)
Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)
CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
linkdex.com/v2.0
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)
Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)
Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)
Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)
DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)
yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
CopperEgg/RevealUptime/SydneyAU(aws)
OpenAcoon v4.1.0 (www.openacoon.de)
semanticbot (info@semanticaudience.com)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
R6_FeedFetcher_(www.radian6.com/crawler)
sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)
Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/39.0
yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
FFbot/1.0; +http://feedfury.com
Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
LinqiaRSSBot/1.0 (eng@linqia.com)
FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)
Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)
Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )
yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
CatchBot/5.0; +http://www.catchbot.com
elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; jason@elefent.eu)
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
Googlebot-Video/1.0
Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; info@xmarks.com)
XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36 PTST/276
CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html; systemsATopenindexDOTio)
Ronzoobot/1.3 (http://www.ronzoo.com/about.php)
MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)
Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)
yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (Anturis Agent)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
webnumbrFetcher/1.0 (http://webnumbr.com/)
Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
NIF/1.1  (http://www.newsisfree.com/robot.php  users:)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
Iframely/0.8.8 (+https://iframely.com/;)
yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Mozilla/5.0 (compatible; OrangeBot/2.0; support.voilabot@orange.com)
Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html
pr-cy.ru Screenshot Bot
GentleSource Short URL Checker ( http://www.gentlesource.com/ )
crazywebcrawler 0.9.0, http://www.crazywebcrawler.com
a.pr-cy.ru
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html
Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
Technoratibot/8.0
Setooz/Nutch-1.0 (http://www.setooz.com)
Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
semanticbot
Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
MetaHeadersBot (+http://www.metaheaders.com/bot.html)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
voyager/2.0 (http://www.kosmix.com/crawler.html)
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; info@yooz.ir)
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)
larbin_2.5.0 (larbin2.5.0@unspecified.mail)
Scrapy/0.22.2 (+http://scrapy.org)
Ronzoobot/1.2 (http://www.ronzoo.com/about.php)
gooblog/2.0 (http://help.goo.ne.jp/contact/)
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/
WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+; (+http://immediatenet.com/thumbnail_api.html; free thumbnails)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)
SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
Scrapy/1.0.4 (+http://scrapy.org)
SanszBot/1.7(http://www.sansz.org/sanszbot, spider@sansz.org) (spider@sansz.org)
Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36 collection@infegy.com
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
Curious George - www.analyticsseo.com/crawler
Grammarly/1.0 (http://www.grammarly.com)
OpenCalaisSemanticProxy
yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
netEstate RSS crawler (+http://www.rss-directory.info/)
ipv6-test.com validator
Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)
ichiro/2.0 (ichiro@nttr.co.jp)
yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
cg-eye live
Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)
Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru)
Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
HubSpot Connect 1.0 (http://dev.hubspot.com/)
Page Valet/4.1pre5
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831
SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
EdisterBot (http://www.edister.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)                 Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com; tarantula@enlle.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)
CatchBot/3.0; +http://www.catchbot.com
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)
Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
adbeat_bot
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)
NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Statastico/4.0
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
EuripBot/2.0 (+http://www.eurip.com) GetRobots
Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
Braintree-Webhooks
Nutch Checksem/Nutch-1.10
Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
Whibse
Yahoo! Site Explorer Feed Validator http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
Siteshooter B0t 2.1
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
eBot / v.0.1e (http://alfa.elchron.cz)
Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)
yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html
YandexSomething/1.0
Scoop
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/5.0 (compatible; CrawlForMe/3.0.2; +info@crawlforme.com)
Seologies/0.1 Bot
www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
dubaiindex (addressendeutschland.de)
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html
Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
LinkStats Bot
Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; +crawler@scaleunlimited.com; crawler@scaleunlimited.com)
CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)
librabot/1.0 (+http://search.msn.com/msnbot.htm)
SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)
yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)
yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
BrainbruBot/1.0 (+http://www.brainbru.com/)
Docoloc
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Prerender (+https://github.com/prerender/prerender)
dlvr.it/1.0 (+http://dlvr.it/)
Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; +crawler@pixray.com)
Abrave v5.5 (http://robot.abrave.co.uk)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429 HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20
yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
MLBot (www.metadatalabs.com/mlbot)
yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
ZnajdzFoto/Image 2.0
Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
Eurobot/1.2 (http://eurobot.ayell.eu)
Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)
yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
CopperEgg/RevealUptime/Singapore(aws)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview) Chrome/11.0.696 Safari/534.24
yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
Server Density Service Monitoring v2
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta
TinEye/1.0; +http://www.tineye.com/
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
GigablastOpenSource/1.0
Googlebot (gocrawl v0.4)
CSmartCrawler (email: abuse[at]adsadv[dot]com)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta9
KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
PingSpot/1.0 (http://www.pingspot.org/)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)
AddThis.com (http://support.addthis.com/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
ExactSeekCrawler/1.0
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)
g2reader-bot/1.0 (+http://www.g2reader.com/)
MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)
QuerySeekerSpider ( http://queryseeker.com/bot.html )
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0;  WOW64;  Trident/6.0;  BingPreview/1.0b)
Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
crawler4j (http://code.google.com/p/crawler4j/)
CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)
Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
nWormFeedFinder (http://www.nworm.com)
netEstate NE Crawler (+http://www.sengine.info/)
Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)
wikiwix-bot-3.0
SEOCentro Page Keyword Analyzer v1.2
yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)
Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/
SemrushBot/0.92
http://www.uni-koblenz.de/~flocke/robot-info.txt
Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/25.0
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
Stroke.cz (http://stroke.cz)
Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-collection/internet_archive/Pages/default.aspx)
audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)
http://arachnode.net 2.5
SurveyBot/2.3 (Whois Source)
ant.com/Nutch-1.7 (http://ant.com)
CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)
Seo Servis - Analyza zdrojoveho kodu
SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)
HubSpot Marketing Grader
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
nrsbot/6.0(loopip.com/robot.html)
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)
Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Kemvibot/1.0 (http://kemvi.com, marco@kemvi.com)
CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)
dotbot
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
page_viewer (larbin2.6.3@unspecified.mail)
HootSuite Image proxy (hosting@hootsuite.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)
Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
VSE/1.0 (rabraham@multiview.com)
Scrapy/0.24.0 (+http://scrapy.org)
Speedy Spider (http://www.entireweb.com)
Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
WatchMouse/18990 (http://watchmouse.com/ ; hk)
Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)
Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de) http://yacy.net/bot.html
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta6
yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
YisouSpider
Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)
Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)
Site24x7 Tools
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
Aboundex/0.3 (http://www.aboundex.com/crawler/)
yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)
Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)
Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Factbot 1.09 (see http://www.factbites.com/webmasters.php)
CopperEgg/RevealUptime/SaoPauloBR(aws)
CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)
yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
URLitor.com
RavenCrawler
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)
Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)
MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)
LongURL API
Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; +http://flipboard.com/browserproxy)
http://tools.geek-tools.org/link-counter/
SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; http://www.stolencamerafinder.com/; crawler@stolencamerafinder.com)
Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)
yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
PagesInventory (robot +http://www.pagesinventory.com)
Sosospider+(+http://help.soso.com/webspider.htm)
Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
PagesInventory (robot http://www.pagesinvenotry.com)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;  http://www.changedetection.com/bot.html )
Wotbox/2.01 (+http://www.wotbox.com/bot/)
Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com)
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
CopperEgg/RevealUptime/DallasTXUSA
yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
Cityreview Robot (+http://www.cityreview.org/crawler/)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
Dlvr.it/1.0 (http://dlvr.it/)
Zookabot/2.4;++http://zookabot.com
AddThis.com robot tech.support@clearspring.com
Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)
TwengaBot
Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)
Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
gosquared-thumbnailer/1.0
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html
BPImageWalker/2.0 (www.bdbrandprotect.com)
Acoon v4.10.3 (www.acoon.de)
Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
WP Engine Install Performance API
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277
CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
CopperEgg/RevealUptime/N.Virginia(aws)
Mozilla/5.0, Google-AdWords-Express
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) http://yacy.net/bot.html
Acoon v4.10.1 (www.acoon.de)
Mediatoolkitbot (complaints@mediatoolkit.com)
Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) SPEng
2Bone_LinkChkr/1.0 libwww-perl/6.03
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
Comodo SSL Checker
Comodo Spider 1.2
Who.is Bot
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
psbot/0.1 (+http://www.picsearch.com/bot.html)
IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)
Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/  ; Orgby.com Search Engine)
Browsershots
AdsBot-Google (+http://www.google.com/adsbot.html)
Qseero v1.0.0
Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
visionutils/0.2
Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Chrome/22.0.1229 Safari/537.4
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)
Mozilla/4.0 (compatible; LinkMarket-Bot v1.2)
Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
WordPress.com; https://public-api.wordpress.com
Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
CopperEgg/RevealUptime/FremontCAUSA
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (90.177.52.133)
Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com)
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
GetIntent Crawler (http://getintent.com/bot.html)
Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)
Cliqzbot
AboutUsBot
yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)
ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)
Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )
Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
Experibot_v1 [bit.ly/2bLMJdw]
WikioFeedBot 1.0 (http://www.wikio.com)
SalesIntelligent/v1.1 (https://www.salesintelligent.com)
Mozilla/5.0 (compatible; ExaleadCloudView/5;)
Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
FollowSite Bot ( http://www.followsite.com/bot.html )
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - web-crawlers@hubspot.com
Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*
findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)
Iframely/0.8.8 (+http://iframely.com/;)
Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);
RSSingBot (http://www.rssing.com)
RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)
Ocelli/1.4 (http://www.globalspec.com/Ocelli)
Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)
Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)
omgilibot/0.4 +http://omgili.com
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
Symfony Spider (http://symfony.com/spider)
Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com - support@admantx.com
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
cuwhois/1.0 (+http://www.cuwhois.com/)
Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)
Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
EuripBot/1.1 (+http://www.eurip.com) GetRobots
yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
socialbm_bot http://spider.socialbm.net
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
webinatorbot 1.0; +http://www.webinator.de
WorldBrewBot/2.1 (+http://www.marketbrew.com/)
BotOnParade, http://www.bots-on-para.de/bot.html
ltx71 - (http://ltx71.com/)
VSB-TUO/1.0 (+http://www.vsb.cz)
yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
CopperEgg/RevealUptime/TokyoJP(aws)
Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com
Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
Autonomy
yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0
CatchBot/1.0; +http://www.catchbot.com
Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)
Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
SiteTruth.com site rating system
SiteUptime.com
yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
UrlTrends Ranking Updater/2.0
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com)
Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; crawler@parsijoo.ir)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Willow Internet Crawler by Twotrees V2.1
KrOWLer/0.1.0, matentzn at cs dot man dot ac dot uk
SimplyFast.info Headers
yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/278
gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)
Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
PayPal IPN ( https://www.paypal.com/ipn )
Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15
ImageEngine/1.0
DER-bot (+https://goo.gl/9ZibHG)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
TelegramBot (like TwitterBot)
Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)
calculon spider/0.1.0 mailto:abuse@yieldbot.com
yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
montastic-monitor http://www.montastic.com
Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Yaanb/1.5.001 (compatible; Win64;)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)
Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)
CopperEgg/RevealUptime/N.VirginiaUSA
Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com)
OrangeCrawler/Nutch-1.0 (ldorange.crawler@orange-ftgroup.com)
CopperEgg/RevealUptime/AmsterdamHolland
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google; (+http://code.google.com/appengine; appid: s~natmacdevice)
yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
SmabblerBot/1.0 (http://www.smabbler.com/)
Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk
Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)
oku-taka-lab-bot (Mozilla-compatible; htakimoto@lr.pi.titech.ac.jp; http://lr-www.pi.titech.ac.jp/wp/)
Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)
yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11)  Firefox/1.5.0.11; 360Spider
Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
mfibot/1.1 (http://www.mfisoft.ru/analyst/; <admin@mfisoft.ru>; en-RU)
Visited by http://tools.geek-tools.org
peerindex
Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
http://www.amagit.com/
yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;  WOW64;  Trident/5.0;  BingPreview/1.0b)
Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)
Mozilla/5.0 (compatible; Cloudinary/1.0)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
Iframely/0.9.8 (+http://iframely.com/;)
yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
Interdose AntiSpamBot/2.0 (+http://www.piep.net)
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)
findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277
yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
NeutrinoAPI/2.0.3
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
Site24x7 Agent
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.7.704.0
Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
Super Monitoring
Seznam-Zbozi-robot/3.0
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)
Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)
Netcraft SSL Server Survey - contact info@netcraft.com
Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)
Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
seebot/1.0.0 (http://www.seegnify.com/bot)
Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Eurobot/1.1 (http://eurobot.ayell.eu)
Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)
Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
UXCrawlerBot
Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; crawler@pixray.com)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 PTST/276
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
Technoratibot/7.0
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/)
Abrave v6.0 (http://robot.abrave.com)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta11
DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)
Comodo Spider 1.1
Link Valet Online 1.1
yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en) http://yacy.net/bot.html
li_viewer (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; Mail.RU/2.0c)
Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)
Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)
NextGenSearchBot 69 (for information visit http://www.zoominfo.com/business/nextgensearchbot.html)
Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en) http://yacy.net/bot.html
peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)
iqdb/0.2 (+http://iqdb.harry.lu/)
Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21
AntBot/1.0 (http://www.ant.com)
Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3
Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)
Digg Deeper/v1 (http://digg.com/about)
Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; http://spyonweb.com spyonweb@gmail.com
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
TinEye/1.1 (http://tineye.com/crawler.html)
Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)
aboutthedomain
Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)
LinqiaMetadataDownloaderBot/1.0 (eng@linqia.com)
yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
NeumobBot/0.2.16 (+http://www.neumob.com/bot)
Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
Miniflux (https://miniflux.net)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)
NerdyBot
Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
Mozilla/4.0 (compatible; LinkMarket-Bot)
SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)
Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)
IdeelaborPlagiaat/1
x28-job-bot; +http://x28.ch/bot.html
Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; info@relateiq.com)
WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
LinkWalker/3.0 (http://www.brandprotect.com)
Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
ia_archiver-web.archive.org
Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)
Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)
Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)
yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; jason@elefent.eu)
HubSpot Links Crawler 1.0 http://www.hubspot.com/
Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)
CopperEgg/RevealUptime/AmsterdamNL(softlayer)
Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.18
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3
KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.21.754.0
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
ownCloud Server Crawler
Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*
LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; someone@company.com)
Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)
findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)
YottaBot Spider/Nutch-2.3
Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
ThumbShots-Bot (+http://thumbshots.in/bot.html)
Mnogosearch-3.1.21
Favicon downloader (+https://favico.be/)
Mozilla/5.0 (compatible;  Page2RSS/0.7; +http://page2rss.com/)
ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)
PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
nodestackbot/0.1 (bot@nodestack.com http://nodestack.com/bot.html)
iskanie (+http://www.iskanie.com)
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)
Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284
Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)
www.deadlinkchecker.com XMLHTTP/1.0
Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.11.740.0
XING-contenttabreceiver/2.0
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
Pingoscope
Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com)
Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)
HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)
Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);
Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en) http://yacy.net/bot.html
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)
Mozilla/5.0 (compatible; +http://servernfo.com/)
Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
CopperEgg/RevealUptime/NewarkNJUSA
Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; musobot/1.0; info@muso.com; +http://www.muso.com)
yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb) http://yacy.net/bot.html
ResponseCodeTest/1.1
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
Acoon v4.9.5 (www.acoon.de)
Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
SiteGuardian/2.0 (Internet Monitoring)
Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)
BoardReader Favicon Fetcher /1.0 info@boardreader.com
baypup/colbert (Baypup; http://www.baypup.com/webmasters; jason@baypup.com)
Mozilla/5.0 (compatible; MonTools.com)
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) http://yacy.net/bot.html
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
linkapediabot (+http://www.linkapedia.com)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
BCKLINKS 1.0
CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
BlogSearch/2 +http://www.icerocket.com/
Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (BrowserSpyBot/1.0) (http://browserspy.dk/)
Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
CopperEgg/RevealUptime/NewarkNJ(linode)
Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/
larbin_2.6.3 capveg@cs.umd.edu
HatenaScreenshot
CopperEgg/RevealUptime/LondonUK
Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)
yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
WordPress.com;
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)
WebTarantula.com Crawler
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)
LinqiaScrapeBot/1.0 (eng@linqia.com)
Orbiter (+http://www.dailyorbit.com/bot.htm)
Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)
InternetSeer.com
Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)
Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; helpdesk@web.nl)
coccoc/1.0 (http://help.coccoc.vn/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)
Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)
Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)
Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)
FAST-WebCrawler/3.8
Googlebot-Image/1.0
enlle punto com/Nutch-1.9
OpenAcoon v4.10.5 (www.openacoon.de)
yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters
Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-11@moz.com)
HubSpot Connect 2.0 (http://dev.hubspot.com/)
Robozilla/1.0
Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
facebookexternalhit/1.1
PercolateCrawler/3.1.30 (ops@percolate.com)
Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)
fastbot crawler beta 4.0 (+http://www.fastbot.de)
SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
coccoc/1.0 ()
larbin_2.6.3 (atyzos@yahoo.com)
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
Scrapy/0.25.1 (+http://scrapy.org)
Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)
cn_ab_viewer (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)
Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en) http://yacy.net/bot.html
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.9.726.0
Scrubby/2.1 (http://www.scrubtheweb.com/)
Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; info@kozvetlen-allasok.hu; 5.0
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
Slackbot-LinkExpanding (+https://api.slack.com/robots)
yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)
Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
WeSEE:Search
Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en) http://yacy.net/bot.html
WordPress.com mShots; http://support.wordpress.com/contact/
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
https_page_test (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko) Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-web.princeton.edu/~fy/bot.html)
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)
findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
CopperEgg/RevealUptime/FremontCA(linode)
Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
link_thumbnailer
KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
Mozilla/5.0 (compatible; Mail.RU/2.0)
CopperEgg/RevealUptime/N.California(aws)
Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
^Nail (http://CaretNail.com)
MarketBrewBot/2.1 (+http://www.marketbrew.com/)
Updownerbot (+http://www.updowner.com/bot)
yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@mail.bixolabs.com)
Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)
asafaweb.com
KrOWLer/0.0.1, matentzn at cs dot man dot ac dot uk
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; siteaudit@alexa.com)
findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
SygolBot http://www.sygol.com
voltron
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)
Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
Superarama.com-Tarama-Botu-v.0.1
yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; SSLBot/1.0;  http://www.sslstats.com/sslbot)
PagesInventory.com (robot http://www.pagesinvenotry.com)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
WordPress.com Reader
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Woko 3.0
Gigabot/3.0 (http://www.gigablast.com/spider.html)
ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
bl.uk_lddc_bot/3.1.1 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
Urlstat (http://bit.ly/1dJuuzs)
yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)
Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)
NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)
WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html
studylib download bot, see http://studylib.es/public/bot for additional info
SemrushBot/0.91
Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
PostPost/1.0 (+http://postpost.com/crawlers)
Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)
Riddler (http://riddler.io/about)
Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
internetVista monitor (Mozilla compatible)
2Bone_LinkChecker/1.0 libwww-perl/6.03
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
DomainStatsBot/1.0 (http://domainstats.io/our-bot)
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
Mergadobot/3.0.2 (+http://mergado.cz)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
SalesIntelligent/v1.1
SiteAnalyzerBot
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
Cliqz Bot (+http://www.cliqz.com)
SalesIntelligent (+https://www.salesintelligent.com)
Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)
gimme60 (Gimme60 Store ID Bot; gimme60.com)
Orbiter/1.0 (http://dailyorbit.com/)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)
CCBot/2.0 (http://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; openstat.ru/Bot)
Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )
Readability/6a54d4 - http://readability.com/about/
percbotspider <ops@percolate.com>
Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
HatenaScreenshot/1.0 (checker)
Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com
Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-generator.online-domain-tools.com)
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36 Yandex.Translate
Sogou web spider/4.0
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)
LinkWalker/2.0
SeoCheck (FischerNetzDesign Seo Checker, info@fischernetzdesign.de)
yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
SiteCheck - http://downforeveryoneorjustme.com
Robots_Tester_http_www.searchenginepromotionhelp.com
CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html
Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html
DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)
www.probethenet.com scanner
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)
LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Imagga Downloader/v0.14.3
SeopultContentAnalyzer/1.0
Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)
Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
Mozilla/5.0 (compatible;acapbot/0.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;Alibaba.Security.Heimdall)
HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))
Jetty/9.3.z-SNAPSHOT
Jetty/9.3.z-SNAPSHOT, SMUrlExpander
Jetty/9.3.z-SNAPSHOT, SMViewer
Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
SynHttpClient-Built/5825
asynchttp
SMUrlExpander
KickFire
HAAARTLAND http client
PEAR HTTPRequest class ( http://pear.php.net/ )
Mozilla/4.0 (vBSEO; http://www.vbseo.com)
HAARTLAND http client
JS-Kit URL Resolver http://js-kit.com/
http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
Kml-Google; (+http://code.google.com/apis/kml) gzip
gvfs/1.20.3
request.js
Microsoft URL Control - 6.01.9782
git/2.1.4
IPS/1.0
OxfordCloudService/1.0
Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24; informatique at mixdata dot com)
Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)
Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)
Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-research/keywords-adwords.php)
CAAM/0.1 (CAAM; caam dot crwlr at gmail dot com)
i2kconnect/"1.7.0"
resolver/0.1.12 (+http://github.com/andris9/resolver)
DangDang
Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)
PritTorrent/0.1
FeedBunch/1.0 (+http://www.feedbunch.com)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) www.HomepageScreenshot.de - Free ScreenShotService/1.0 Safari/537.21
Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)
Pinterest/0.2 (+https://www.pinterest.com/bot.html)
Pinterest/0.1 (+https://www.pinterest.com/bot.html)
Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
Mozilla/5.0 zgrab/0.x
Zend_Http_Client
Zend\Http\Client
Zao/0.1 (http://www.kototoi.org/zao/)
YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)
yoogliFetchAgent/0.1
Yo-yo
Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
xpymep.exe
xpymep2.exe
xpymep3.exe
Xenu Link Sleuth 1.2b
Xenu Link Sleuth 1.2c
Xenu Link Sleuth 1.2d
Xenu Link Sleuth 1.2e
Xenu Link Sleuth 1.2f
Xenu Link Sleuth 1.2g
Xenu Link Sleuth 1.2h
Xenu Link Sleuth 1.2i
Xenu Link Sleuth/1.3.7
Xenu Link Sleuth/1.3.8
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com
WWW-Mechanize/1.73
WordPress/4.0.1; http://bestcomp.hol.es
WordPress/4.3.3; http://homebusinessreviews.online
WordPress/4.3.3; http://legithomejobs.club
WordPress/4.4.2; http://cheapautoinsuranceinva.com
WordPress/4.4.2; http://citycollegeinmiami.com
WordPress/4.4.2; http://cookingideabook.com
WordPress/4.4.2; http://oldmanhealth.com
WordPress/4.4.2; http://sapcrmarchitecture.com
WordPress/4.4.2; http://www.spar3d.com
WordPress/4.5.2; http://1.we-careu.xyz
WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.0.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.25.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.47.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.48.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.50.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.56.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.61.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.62.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.63.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Word/14.64.0
WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)
WhatsApp
WhatsApp/2.12.15/i
WhatsApp/2.12.16/i
WhatsApp/2.12.17/i
WhatsApp/2.12.449 A
WhatsApp/2.12.453 A
WhatsApp/2.12.510 A
WhatsApp/2.12.540 A
WhatsApp/2.12.548 A
WhatsApp/2.12.555 A
WhatsApp/2.12.556 A
WhatsApp/2.16.1/i
WhatsApp/2.16.13 A
WhatsApp/2.16.2/i
WhatsApp/2.16.42 A
WhatsApp/2.16.57 A
WhatWeb/0.4.8-dev
http://www.almaden.ibm.com/cs/crawler [wf84]
WeLikeLinks - WeAreNotEvil
Websquash.com (Add url robot)
WeCrawlForThePeace - We are not Evil
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) webscreenie Safari/537.21
Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)
Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) webkit2png Safari/534.34
WebFetch
WebIndex
webcollage/1.114
webcollage/1.117
webcollage/1.125
webcollage/1.129
webcollage/1.93
Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)
W3C_Unicorn/1.0
W3C_I18n-Checker/1.0
W3C-mobileOK/DDC-1.0
W3C-checklink
W3C-checklink/2.90 libwww-perl/5.64
W3C-checklink/3.6.2.3 libwww-perl/5.64
W3C-checklink/4.2 [4.20] libwww-perl/5.803
W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
W3C-checklink/4.3 [4.42] libwww-perl/5.805
W3C-checklink/4.3 [4.42] libwww-perl/5.808
W3C-checklink/4.3 [4.42] libwww-perl/5.820
W3C-checklink/4.5 [4.154] libwww-perl/5.823
W3C-checklink/4.5 [4.160] libwww-perl/5.823
w3af.org
VYU2 (GNU; OpenRISC)
VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
URLChecker
urlresolver
Upflow/1.0
Typhoeus - https://github.com/typhoeus/typhoeus
UdmSearch/3.1.x
Mozilla/5.0 Tweetminster
Mozilla/5.0 (compatible; +http://tweetedtimes.com)
truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
Traackr.com
More-Testing (+http://www.theinternetrules.com)
The Expert HTML Source Viewer (http://www.expert-html.com)
htmlayout 3.3; above-Windows-7; www.terrainformatica.com )
WebFuck V2.1 T0PHackTeam www.t0p.xyz
Symfony2 BrowserKit
Mozilla/5.0 (compatible; SwiteScraper)
Summify (Summify/1.0.1; +http://summify.com)
Stratagems Kumo
Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
Sprinklr 2.0
spray-can/1.3.3
SortSite/4.4.500.0
Snoopy v2.0.0
SMRF URL Expander
silk/1.0 (+http://www.slider.com/silk.htm)/3.7
SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)
SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242
ShortLinkTranslate
Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats
semanticdiscovery/0.1
SearchSight/2.0 (http://SearchSight.com/)
ScoutURLMonitor/5.9.8
Robosourcer/1.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
RestSharp/105.2.3.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.1(2.5.3) Safari(ReederForMac)
redback/v0-570-g26f8c96
RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1
RealPlayer%20Downloader/1569 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
RealPlayer%20Downloader/1701 CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
RealPlayer%20Downloader/1725 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
RealPlayer%20Downloader/1750 CFNetwork/720.5.7 Darwin/14.5.0 (x86_64)
RealPlayer%20Downloader/1750 CFNetwork/760.0.5 Darwin/15.0.0 (x86_64)
RealPlayer%20Downloader/1750 CFNetwork/760.4.2 Darwin/15.4.0 (x86_64)
Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic
python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual
python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64
python-requests/2.10.0
python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic
python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic
python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic
python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64
python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64
python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64
python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic
python-requests/2.8.0
python-requests/2.8.1
python-requests/2.9.1
Python-httplib2/0.7.4 (gzip)
Pulsepoint XT3 web scraper
Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)
Mozilla/5.0 (Macintosh; Intel Mac OS X) PowerPoint/14.39.0
PostRank/2.0 (postrank.com; 1 subscribers)
PostRank/2.0 (postrank.com)
postano
Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 (support@p-und-z.de) for ARIS Business Architect
Pizilla++ ver 2.45
phpcrawl
Pearltrees
Pcore-HTTP/v0.23.20
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.30729)
Mozilla/5.0 (compatible; Optimizer)
Mozilla/5.0 [en] (X11, U; OpenVAS 7.0.10)
JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
okhttp/2.5.0
okhttp/4.9.0
oegp v. 1.3.0
Nuzzel
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
Notifixious/LinkChecker (http://notifixio.us)
http://www.nominet.org.uk/privacypolicy
node-superagent/1.8.3
node.io
Kimengi/nineconnections.com
newspaper/0.0.9.8
newspaper/0.1.7
newsme/1.0; feedback@news.me
NewsGator/2.0 Bot (http://www.newsgator.com)
NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
NetTrack Anonymous Web Statistics https://nettrack.info/support.php
NetTrack.info Crawler
NetResearchServer(http://www.look.com)
NetResearchServer/2.5(loopimprovements.com/robot.html)
NetResearchServer/2.7(loopimprovements.com/robot.html)
NetResearchServer/2.8(loopimprovements.com/robot.html)
NetResearchServer/3.5(loopimprovements.com/robot.html)
NetResearchServer/4.0(loopimprovements.com/robot.html)
NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)
check_http/v2.0.3 (nagios-plugins 2.0.3)
MVAClient
Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)
Mojolicious (Perl)
mogimogi/1.0
Mindjet
Mindjet MindManager
Microsoft Windows Network Diagnostics
Microsoft Office Excel 2013
Microsoft Office Excel 2014
Microsoft Office Excel 2014 (16.0.6528) Windows NT 10.0
Microsoft Office Existence Discovery
Microsoft Office Mobile/15.0
Microsoft Office OneNote 2013
Microsoft Office PowerPoint 2013
Microsoft Office PowerPoint 2013 (15.0.4420) Windows NT 6.2
Microsoft Office PowerPoint 2013 (15.0.4641) Windows NT 6.2
Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 10.0
Microsoft Office PowerPoint 2013 (15.0.4797) Windows NT 6.2
Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.1
Microsoft Office PowerPoint 2013 (15.0.4805) Windows NT 6.2
Microsoft Office PowerPoint 2013 (15.0.4815) Windows NT 10.0
Microsoft Office PowerPoint 2014
Microsoft Office Word 2013
Microsoft Office Word 2013 (15.0.4623) Windows NT 6.2
Microsoft Office Word 2013 (15.0.4701) Windows NT 6.1
Microsoft Office Word 2013 (15.0.4797) Windows NT 10.0
Microsoft Office Word 2013 (15.0.4805) Windows NT 10.0
Microsoft Office Word 2013 (15.0.4805) Windows NT 6.1
Microsoft Office Word 2013 (15.0.4823) Windows NT 10.0
Microsoft Office Word 2014
Microsoft Office/16.0 (Windows NT 10.0; Microsoft Excel 16.0.13231; Pro)
Melvil Rawi/1.0
marketinggrader.com
Mandrill-Webhook/1.0
MailChimp.com
MagpieRSS/0.72 (+http://magpierss.sf.net)
http://www.mabontland.com
lwp-trivial/1.33
lwp-trivial/1.35
lwp-trivial/1.36
lwp-trivial/1.38
lwp-trivial/1.41
looksystems.net
LinkExaminer/1.01 (Windows)
VLC/2.2.4 LibVLC/2.2.4
hello@knows.is
Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)
JoBo/1.x (http://www.matuschek.net/jobo.html)
Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)
Integrity/6
InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)
InDesign%20CC/11300 CFNetwork/720.4.4 Darwin/14.4.0 (x86_64)
inbound.li parser
InAGist URL Resolver (http://inagist.com)
igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
IDG Twitter Links Resolver
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
Jersey/2.19 (HttpUrlConnection 1.7.0_85)
httpunit/1.x
httpssites_power/1 CFNetwork/758.3.15 Darwin/15.4.0
httpscheck (unknown version) CFNetwork/760.5.1 Darwin/15.5.0 (x86_64)
HttpComponents/1.1
http_requester/0.1
HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-1ubuntu4.14
HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29
HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33
HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35
HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
HTTP-Tiny/0.054
http-kit/2.0
HTMLParser/2.0
htdig/3.1.5
htdig/3.1.5 (infosys@storm.rmi.org)
htdig/3.1.5 (root@localhost)
htdig/3.1.5 (webmaster@online-medien.de)
htdig/3.1.6 (mathieu.peltier@inrialpes.fr)
htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)
ht://check/2.0.0-rc1 (Linux 2.6.32-573.18.1.el6.x86_64 x86_64)
HEADMasterSEO
Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-2.3)
Mozilla/4.0 (compatible; grub-client-2.6.0)
Mozilla/4.0 (compatible; grub-client-2.6.1)
GoSpotCheck 1.0.6 (iPhone; iPhone OS 5.1.1; en_US)
GoScraper
GoogleProducer; (+http://goo.gl/7y4SX)
GoogleHC/
Google-HTTP-Java-Client/1.17.0-rc (gzip)
Google-HTTP-Java-Client/1.9.0-alpha
Mozilla/5.0 (compatible; Google-Apps-Script)
Go-http-client/1.1
Go 1.1 package http
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)
Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)
Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)
FeedBurner/1.0 (http://www.FeedBurner.com)
FavOrg
Faraday v0.9.1
Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.10.0
Mozilla/5.0 (Macintosh; Intel Mac OS X) Excel/14.62.0
ExactSearch
EventMachine HttpClient
Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
Embed PHP Library
ECCP/1.2.1+(productlists@eniro.com)
ec2linkfinder
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
Drupal (+http://drupal.org/)
Dragonfly File Reader
DNS-Tools Header-Analyzer
Dispatch/0.11.1-SNAPSHOT
luis@cybo.com
CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
curb
Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)
cosmos/0.9_(robot@xyleme.com)
Mozilla/5.0/Firefox/42.0 (contactbigdatafr at gmail.com)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MAXTHON 2.0); Connect Us: webauth@cmcm.com
Pattern/2.6 +http://www.clips.ua.ac.be/pattern
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
cb crawl (+http://www.companybooknetworking.com)
CapsuleChecker (http://www.capsulink.com/)
CakePHP
Branch-Passthrough
Branch Metrics API
boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
boitho.com-robot/1.1
Blackboard Safeassign
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch)
BIGLOTRON (Beta 2;GNU/Linux)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
Astute SRM
aria2/1.18.8
Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)
Mozilla/4.0 (compatible; Arachmo)
Apache-HttpClient/4.2.3 (java 1.5)
Apache-HttpClient/4.2.5 (java 1.5)
Apache-HttpClient/4.3.1 (java 1.5)
Apache-HttpClient/4.3.3 (java 1.5)
Apache-HttpClient/4.3.5 (java 1.5)
Apache-HttpClient/4.5 (Java/1.8.0_51)
Apache-HttpClient/4.5 (Java/1.8.0_72)
Apache-HttpClient/4.5 (Java/1.8.0_74)
Apache-HttpClient/4.5.1 (Java/1.7.0_79)
Apache-HttpClient/4.5.1 (Java/1.8.0_40)
Apache-HttpClient/4.5.2 (Java/1.8.0_72)
Apache-HttpClient/UNAVAILABLE (java 1.4)
Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)
Anemone/0.7.2
AndroidDownloadManager/6.0.1 (Linux; U; Android 6.0.1; SM-G900F Build/MMB29M)
AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)
AndroidDownloadManager/4.4.2 (Linux; U; Android 4.4.2; SM-T210 Build/KOT49H)
Airmail 1.3.3 rv:237 (Macintosh; Mac OS X 10.9.3; sk_SK)
Python/3.5 aiohttp/0.21.6
AHC/1.0
Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)
Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
Accoona-AI-Agent/1.1.2
Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) shipra.goel@a3logics.in
scrutiny/6
RMA/1.0 (compatible; RealMedia)
PHP/5.5
NING/1.0
Mget/0.1.9
Java/1.4.1_04
Java/1.6.0
Java/1.6.0_04
Java/1.6.0_22
Java/1.6.0_24
Java/1.6.0_26
Java/1.6.0_31
Java/1.6.0_32
Java/1.6.0_34
Java/1.6.0_37
Java/1.7.0_101
Java/1.7.0_45
Java/1.7.0_51
Java/1.7.0_65
Java/1.7.0_80
Java/1.8.0_05
Java/1.8.0_20
Java/1.8.0_31
Java/1.8.0_40
Java/1.8.0_45
Java/1.8.0_51
Java/1.8.0_60
Java/1.8.0_65
Java/1.8.0_66
Java/1.8.0_73
Java/1.8.0_77
Java/1.8.0_91
Java/1.8.0_92
Goose/1.0.25
FDM 3.x
bluefish 0.6 HTML editor
2ip.ru CMS Detector (http://2ip.ru/cms/)
masscan/1.0 (https://github.com/robertdavidgraham/masscan)
VidibleScraper/1.0
WhereGoes? Redirect Tracer/1.0
unfurlist (https://github.com/Doist/unfurlist)
pshtt, https scanning
gofetch
COMODO DCV
CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
ZmEu
RSSOwl/2.2.1.201312301314 (Windows; U; en)
WebClient/1.0
Microsoft-WebDAV-MiniRedir/10.0.10586
Microsoft-WebDAV-MiniRedir/10.0.14393
Microsoft-WebDAV-MiniRedir/5.1.2600
Microsoft-WebDAV-MiniRedir/6.0.6001
Microsoft-WebDAV-MiniRedir/6.1.7601
Microsoft-WebDAV-MiniRedir/6.3.9600
Microsoft Data Access Internet Publishing Provider Protocol Discovery
Test Certificate Info
Needle/1.4.2 (Node.js v6.9.2; linux x64)
DavClnt
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 mixed-content-scan
Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)
Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)
/n software IPWorks HTTP/S Component - www.nsoftware.com
safe-agent-scanner
Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
ANAM (github.com/dutchcoders/anam)
Dolphin http client/11.2.8(385) (Android)
Dolphin http client/11.3.4(405) (Android)
Calypso v/0.01
-Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)
Jeode/1.x.x
KeyCDN Perf Test
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/27.0.1453.110 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV
Taringa UGC www.taringa.net/shared-link.php
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34
ganarvisitas/1.0 (+http://www.ganarvisitas.com/)
FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
Seznam-Zbozi-robot/3.2.1
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
Seznam-Zbozi-robot/3.2.2
Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)
Seznam-Zbozi-robot/3.3
SklikBot/2.0 (sklik@firma.seznam.cz;+http://napoveda.sklik.cz/)
SeznamBot/3.0 (+http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)
Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I
SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)
SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I
SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
SeznamBot/2.0 (+http://fulltext.seznam.cz/)
SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)
Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)
Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Daum Raam Edition b20140820 Safari/534.34
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; special_archiver/3.3.0 bot@archive.org +https://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
Qwantify/1.0
Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Scrapy/1.3.0 (+http://scrapy.org)
Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
Scrapy/1.2.1 (+http://scrapy.org)
Scrapy/0.16.5 (+http://scrapy.org)
Mozilla/5.0 (compatible; Dataprovider.com;)
Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)
Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1
Mozilla/5.0 (Windows NT 6.1; WOW64; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X; Neustar WPM) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
Mozilla/5.0 (Windows NT 6.0; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36
Mozilla/5.0 (Windows NT 6.0; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
Mozilla/5.0 (Windows NT 6.1; rv:38.0; Neustar WPM) Gecko/20100101 Firefox/38.0
Mozilla/5.0 (Windows NT 6.1; Neustar WPM) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
Mozilla/5.0 (Linux; Android 4.1.1; HTC6435LVW Build/JRO03C; Neustar WPM) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.169 Mobile Safari/537.22
Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost
Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
HubSpot Website Grader (web-crawlers@hubspot.com)
HubSpot Marketing Grader, HubSpot Marketing Grader
com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (http://www.shrinktheweb.com)
scrapy-redis (+https://github.com/rolando/scrapy-redis)
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 PTST/348
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 PTST/345
TeamSpeak3-ImageFetcher-1.0
wf_crawler (http://www.websitefigures.com)
wf_crawler (http://websitefigures.com)
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)
Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
Googlebot deepcrawl
Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)
Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)
Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)
Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)
Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/49.0
Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)
Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)
YOOBI.de - WebBot 1.0
YOOBI - WebBot 1.0
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)
Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp; info@kozvetlen-allasok.hu; 5.0
Mozilla/5.0 (compatible; JobKereso;  +https://www.kozvetlen-allasok.hu/robot.jsp  info@kozvetlen-allasok.hu)
pyspider/0.3.9 (+http://pyspider.org/)
Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.24.776.0
Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
TwengaBot-2.0
Iframely/1.0.0 (+https://iframely.com/;)
Iframely/0.8.5 (+http://iframely.com/;)
rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+phaser-testing-crawler-01@moz.com)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com
Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
mindUpBot
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Contacts Crawler (+http://www.scrapinghub.com)
TA SEO Crawler v0.1 URLData.pm (smcquaker@tripadvisor.com)
Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)
ArielisBot/1.1 (+http://arielis.com/
RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)
Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)
botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; botrobin@smarter.codes)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)
GloomarBot/1.1 (https://www.gloomar.com/bot)
Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)
Opendi Screenshot Bot bot@opendi.com
Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)
Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)
BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
uipbot/1.0 (uipbot@semasio.net)
gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-Google; (+http://code.google.com/appengine)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) MapperCmd/5.24.780.0
Seeker v.1 http://LookSeek.com/seeker/
AppManager RPT-HTTPClient/0.3-3E
LinkAlarm/3.1
Mozilla/5.0 (X11; U; Linux i686) Firefox/2.0.0.11 wprecon.com survey
Jeffrey's Exif Viewer (http://regex.info/exif)
Untiny
Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
biNu image cacher (info@binu-inc.com)
Mozilla/5.0 (compatible; Google-Youtube-Links)
ping.blo.gs/2.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
Mozilla/5.0 (X11; Linux x86_64; rv:52.0.2) Gecko/20100101 Firefox/52.0.2 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:50.1.0) Gecko/20100101 Firefox/50.1.0 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QtWebEngine/5.3.0 Safari/538.1 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:45.0.1) Gecko/20100101 Firefox/45.0.1 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:46.0.1) Gecko/20100101 Firefox/46.0.1 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:47.0.2) Gecko/20100101 Firefox/47.0.2 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:48.0.2) Gecko/20100101 Firefox/48.0.2 Ghost Inspector
Mozilla/5.0 (X11; Linux x86_64; rv:49.0.2) Gecko/20100101 Firefox/49.0.2 Ghost Inspector
Netpursual/1.0
Kaspersky Lab CFR link resolver cfradmins@kaspersky.com
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0)
b0t
"echocrawl 2.0"
masscan/1.0
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)
Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)
CloudEndure Scanner (ops@cloudendure.com)
AFB/3.0 (+http://allloadin.com)
TLSProbe/1.0 (+https://scan.trustnet.venafi.com/)
Sens.ai http://sovereign.ai/contact
eContext/1.0 (eContext Classification Engine)
Owler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Slack/1.2.6 Chrome/45.0.2454.85 AtomShell/0.34.3 Safari/537.36 Slack_SSB/1.2.6
Mozilla/5.0 (compatible; ChangesMeter/1.9.1; http://intuiware.com/apps/changes-meter)
Mozilla/5.0 (compatible; Gookey.co/1.0; +http://gookey.co/)
TrapitAgent/0.1 (feed processor; +http://trapit.com/about)
sqlmap/1.0-dev-nongit-201612050a8c (http://sqlmap.org)
UniversalFeedParser/3.3 +http://feedparser.org/
TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
scalaj-http/1.0
HggH PhantomJS Screenshoter
fasthttp
fasthttp, Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36
lua-resty-http/0.10 (Lua) ngx_lua/10000
GIS-LABS:CertMon
GuzzleHttp/6.2.1 PHP/7.0.15-0ubuntu0.16.04.4
http-request/v0.7.0 (http://git.io/tl_S2w) node.js/v0.10.29
intraVnews/1.x
MailChimp
G-i-g-a-b-o-t
Symfony BrowserKit
VCPP71 http-get 1.0a
Jaunt/1.2
Mozilla/5.0 (compatible; Owler/0.4; +; )
MovableType/x.x
OMSC
websitepulse agent/0.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
Mozilla/5.0 (compatible; Uptimia; www.uptimia.com)
ssl-tools.net
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) DMBrowser/2.1
keycdn-tools/perf
keycdn-tools/br
keycdn-tools/h2
keycdn-tools/perf
keycdn-tools/br
keycdn-tools/h2
KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)
Donuts Content Explorer (www.donuts.domains)
Netsparker
Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)
Apache-HttpAsyncClient/4.1.3 (Java/1.8.0_121)
Tenon.io Automated Web Accessibility Tester
ApacheBench/2.3
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Clarsentia)
Mozilla/5.0 (compatible; historious/2.0; +http://historio.us/)
Mozilla/5.0 (compatible; uCrawlr/1.0 ; +https://blog.ucoz.ru/upolicy)
Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
Brodie/1.0 (+http://www.15miles.com/)
Jobg8 URL Monitor
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) wpif Safari/537.21
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) imgsizer Safari/537.21
Mozilla/5.0 (compatible; AppBeat/1.0; +http://www.appbeat.io/)
Mozilla/5.0 (compatible; Chirp/1.0; +http://www.binarycanary.com)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SiteMonitor)
Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored)
Pingability.com Monitoring Service
SiteSucker for macOS/2.10.6
Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)
Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Sideqik +http://www.sideqik.com
SiteSucker/2.4.6
Vulnbusters Meter (see http://vulnbusters.com for details)
Express WebPictures (www.express-soft.com)
gobyus
Search37/1.0 (http://www.search37.com; info@search37.com)
Search37/1.2 (http://www.search37.com; info@search37.com)
Mozilla/5.0 (compatible; SiteIndexed/1.2; +http://www.siteindexed.com)
Jersey/2.25.1 (Apache HttpClient 4.5)
URL Verifier by Daanav.com
trendspottr.com
Kerrigan/2.0 (kerrigan@mashable.com)
Keyword Extractor 5000; lucas@hyscore.io
Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)
grokkit (pdsupport@purediscovery.com)
akka-http/10.0.10
POE-Component-Client-HTTP/0.943 (perl; N; POE; en; rv:0.943000)
RecurPost/1.0 https://recurpost.com (KHTML, like Gecko)
Mozilla/5.0 Firefox Windows WhoRunsCoinHive.com
Mozilla/5.0 (compatible; Kelny/2.0; +http://www.kelny.com/)
colly - https://github.com/gocolly/colly
WebCopier vx.x
gSOAP/2.8
http.rb/2.2.2
Grabber -  https://sites.google.com/3ziele.de/grabber/
JobFeed discovery script (http://www.jobfeed.com)
JoinVision Generic Site Loader 1.0
SSI-LinkPreviewGenerator/1.0
SnowHaze Search/1.0 support@snowhaze.com
Saleslift
Perlu - HTML Analyzer and Relationship Builder 1.1
Offline Explorer 1.*
Mozilla/5.0 (compatible; Wappalyzer; https://www.wappalyzer.com)
AngleSharp/0.9.9
Email%20Extractor%20Lite/26 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
HubSpot Page Screenshot Service
NewRelicPinger/1.0 (543245)
APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
updown.io daemon 2.2
updown.io daemon 2.3
updown.io daemon 2.2
updown.io daemon 2.3
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36 PrintFriendly.com
Unshorten.It!/1.0 (https://unshorten.it/)
github-camo (65c9ebe0)
github-camo (abb93c82)
github-camo (bacd3d5b)
github-camo (df02412a)
Jobsearch1.5
HubSpot Content Strategy link check
Twisted PageGetter
node-urllib/2.25.3 Node.js/6.12.2 (Linux 4.4; x64)
sentry/8.18.0 (https://sentry.io)
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 (compatible; Sitebulb/1.1; +https://sitebulb.com)
Thinklab (thinklab.com)
HubSpot inbound link reporting check
SetCronJob/1.0 (+http://www.setcronjob.com/)
Burf Search Engine
Apache Droid
Mozilla/5.0 (compatible; adscanner/)
hackney/1.11.0
Mozilla/5.0 (compatible; MuckRack/1.0; +http://muckrack.com)
Mozilla/5.0 (compatible; dotMailer content retrieval service)
Nexgate Ruby Client
Miniature.io/2.0 (Linux; en-us)
help@dataminr.com
Mozilla/5.0 (compatible; Your-Website-Sucks/2.0; +http://www.Your-Website-Sucks.net/WWW.SOMEWEBSITE.COM)
axios/0.17.1
Wtrace
PostmanRuntime/7.1.1
hackney/1.11.0
Lavf/57.25.100
makecontact/1.0 (+https://makecontact.io)
QrafterPro/4600 CFNetwork/889.9 Darwin/17.2.0
Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
Go http package
HonesoSearchEngine/1.0
orion-semantics.com 0.1
ExperianCrawlUK (andrew dot swanton at phgroup dot com)
Trendsmap Resolver/0.0.1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; PeoplePal 6.2)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent
Mozilla/3.0 (compatible; Indy Library)
VB Project
webster-internet.de pad browser
LWP::Simple/5.805
Microsoft Data Access Internet Publishing Provider DAV
MSFrontPage/4.0
Microsoft Data Access Internet Publishing Provider Cache Manager
VB Project
eCatch/3.0
MSFrontPage/12.0
Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4
Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2
webster-internet.de pad browser
Trendsmap Resolver/0.0.1
Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.17929)
Arukereso/1.0
Trendsmap Resolver/0.0.1
Flock/14.15 (Android 2.9; fr_BE;)
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.0.16) Gecko/2010021003 Firefox/3.0.16 Flock/2.5.6
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.14) Gecko/20080414 Firefox/2.0.0.14 Flock/1.1.2
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070801 Firefox/2.0.0.6 Flock/0.9.0.2
AppStoreScraperZ/1 CFNetwork/889.9 Darwin/17.2.0 (x86_64)
Everwall Link Expander ( http://everwall.com )
Wfuzz/2.2.2
The Knowledge AI the.knowledge.ai@gmail.com
The Knowledge AI
Mozilla/5.0 (compatible; DataXu/1.0; +http://dataxu.com)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)
AdAuth/1.0; +https://www.adauth.com
twibble.io
JolokiaPwn, https://matmannion.com/jolokiapwn/, mat.mannion@gmail.com
CookieReports.com/0.1
GoogleCloudMonitoring
kube-probe/1.9+
Mozilla/5.0 Chrome/10.0.613.0 Safari/534.15 Zombie.js/5.0.7
Microsoft Office/16.0 (Microsoft Outlook Mail 16.0.7167; Pro)Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/7.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; Tablet PC 2.0; InfoPath.3; Microsoft Outlook 14.0.6025; ms-office; MSOf
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/7.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E; Microsoft Outlook 14.0.7167; ms-office; MSOffice 14)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; Microsoft Outlook 15.0.4805; ms-office; MSOffice 15)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; Tablet PC 2.0; MALNJS; Microsoft Outlook 16.0.6001; ms-office; MSOffice 16)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; Microsoft Outlook 14.0.6025)
Outlook-Express/7.0 (MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; McAfee; .NET4.0C; .NET4.0E; GWX:QUALIFIED; TmstmpExt)
Mail/3445.4.7 CFNetwork/889.9 Darwin/17.2.0 (x86_64)
SPDYCheck SPDY Protocol Tester, see http://spdycheck.org/about.html
SpamExperts/128741 (Virus Scan)
Unshorten.It! /1.0 (unshorten.it)
Monit/5.25.2
Mozilla/5.0 (compatible; PingAdmin.Ru/1.1; +http://pingadmin.ru/)
Mozilla/5.0 (compatible; localsearch-web/2.0; +http://www.localsearch.ch/en/legal-notice)
Mozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
annotate_google; http://ponderer.org/download/annotate_google.user.js
Apache OpenOffice/4.1.2
Bidtellect/0.0.346.0
BlockNote.Net
Bloglines/3.1 (http://www.bloglines.com)
DemandbasePublisherAnalyzer/0.1 (http://www.demandbase.com; info@demandbase.com)
endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo)
Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1)
Feedreader 3.xx (Powered by Newsbrain)
FeedshowOnline (http://www.feedshow.com)
FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector
ImageVisu/v4.x.x
InetURL:/1.0
Mixmax-LinkPreview 1.0 (+https://sdk.mixmax.com)
Mozilla/5.0 (compatible; DNS SSL/TLS HTTP and HTML scanner/0.1; +https://www.htmlyse.com/)
Mozilla/5.0 (compatible; DNS SSL/TLS HTTP HTML Website Security Scanner/0.2 beta; +https://www.htmlyse.com/)
Mozilla/5.0 (compatible; Scrutiny/8; +https://peacockmedia.software/mac/scrutiny/
NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/)
og-scraper/1.0.0
Plagger/0.x.xx (http://plagger.org/)
raynette_httprequest/1.0
RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0
ScrapeBox Page Scanner
SearchExpress
Snarfer/0.x.x (http://www.snarfware.com/)
unirest-java/1.3.11
WinPodder (http://winpodder.com)
WordupinfoSearch/1.0
13TABS/13TABS-1.14 (https://www.13tabs.com/aboutUs.jsp)
777Siters/1.05 (+http://777s.ru/siters/)
7Siters/1.07 (+http://7ooo.ru/siters/)
Aranea Web Corpora Project (Sk Crawl) (+http://unesco.uniba.sk/guest)
censys.io internet census
Chromaxa/1.0 Sonar/1.0
compatible; Licorne Uri Checker 1.0; +http://www.jednorozec.cz)
CronDroid v0.80a http://www.crondroid.com
Cronjob.de
Cronless/2.0 (+http://cronless.com)
EasyCron/1.0 (https://www.easycron.com/)
EDS EXIF/1.0 (https://www.exif-search.com)
EDS EXIF/1.0 (https:/exif-search.com)
faviconkit/1.0 (https://faviconkit.com)
HomePage Rss Reader 1.0
Hong Kong Education City/1.0 (built with Hong Kong Education City 1.10; http://www.hkedcity.net/; search@hkedcity.net)
https://getroot.sh survey
isUp.li Website Monitoring 2
JAHHO check
MB-SitemapGenerator
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://basicstate.com/)
Mozilla/5.0 (compatible; Aprc/2.9.15-24; +https://aprc.it/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3528.4 Safari/537.36
Mozilla/5.0 (compatible; Google AppsViewer; http://drive.google.com)
Mozilla/5.0 (compatible; GoogleSites)
Mozilla/5.0 (compatible; iThemes Sync/1.0; +http://ithemes.com/sync)
Mozilla/5.0 (compatible; iThemes Sync/1.0; ITX WP-Admin API/1.0; +http://ithemes.com/sync)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; CHKD 1.2; Akamai_Site_Analyzer)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WAC-OFU)
Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)
Mozilla/5.0 (Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36 Nodemeter 0.7
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)
Mozilla/5.0 (Windows NT 6.1; rv:12.0; StumbleUpon; noc@stumbleupon.com) Gecko/20100101 Firefox/12.0
Mozilla/5.0 (Windows NT 6.1; rv:40.0; StumbleUpon; noc@stumbleupon.com) Gecko/20100101 Firefox/40.1
Mozilla/5.0 (Windows NT 6.1; stumbleupon.com) Chrome/58.0.3029.110 Safari/537.36
Mozilla/5.0 AppleWebKit (compatible; s4a-probe/1.0; +https://www.seo4ajax.com/webscraper)
Mozilla/5.0 AppleWebKit (compatible; s4a/1.0; +https://www.seo4ajax.com/webscraper)
Mozilla/5.0 Gecko/20100101 Firefox/51.0.1 SnapSearch
NodePing
PocketImageCache/1.0
queuedriver/1.0 (eng@linqia.com)
Re-re Studio (+http://re-re.ru/)
Tarmot Gezgin/1.0 (compatible; TarmotGezgin/1.1; +http://www.tarmot.com/gezgin)
WDT.io
webeval.researchready.com - support@imagineeasy.com for info.
WWW::Document/0.02
Y!J-BRW/1.0 (https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716)
YoYs.net
YP.PL
Mozilla/5.0 (Windows NT 6.1; rv:40.0; StumbleUpon; noc@stumbleupon.com) Gecko/20100101 Firefox/40.1
Mozilla/5.0 (Windows NT 6.1; StumbleUpon; noc@stumbleupon.com) Chrome/58.0.3029.110 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; stumbleupon.com) Chrome/58.0.3029.110 Safari/537.36
Mozilla/5.0 (compatible; tracemyfile/1.0)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/66.0.3336.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36 Pix/1.0 (Criteo; x64-criteo)
CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0
Email%20Extractor/1 CFNetwork/760.6.3 Darwin/15.6.0 (x86_64)
Email%20Extractor/29 CFNetwork/897.15 Darwin/17.5.0 (x86_64)
HTTP Banner Detection (https://security.ipip.net)
HTTP Banner Detection(security.ipip.net)
Mozilla/5.0 (compatible; SimpleScraper)
http_get
MeltwaterNews www.meltwater.com
Disqus/1.0
Datafeedwatch/2.1.x
Zabbix
Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)
httpRequest
Google-Transparency-Report
Mozilla/5.0 (compatible; TrendsmapResolver/0.1)
Embarcadero URI Client/1.0
GetCode
Gregarius/0.5.2 ( http://devlog.gregarius.net/docs/ua)
GRequests/0.10
DAP/NetHTTP
Hadi Agent
Canary%20Mail/397 CFNetwork/893.13.1 Darwin/17.4.0 (x86_64)
Sendsay.Ru/1.0; https://Sendsay.Ru/; ask@sendsay.ru
Mozilla/5.0 (Zoom.Mac 10.8.5 x86)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36 TryJsoup/1.0 (+http://try.jsoup.org/)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; BacklinkHttpStatus)
SilverReader/1.0; http://silverreader.com
ExtractorPro
WebsiteExtractor
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1 google_partner_monitoring FWSzVTDDBz14547302713138T
Outlook-iOS/696.1188109.prod.iphone (2.102.0)
Polymail/1.47 (ctrlla.Polymail; build:64; OS X 10.12.6) Alamofire/4.5.1
nyawc/1.8.1 CPython/3.6.5 Linux/3.10.0-862.9.1.el7.x86_64
commonscan.org cralwer v1.01
AdminLabs
WebGazer/1.0 (+https://www.webgazer.io)
HappyApps-WebCheck/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6; +feeder.co) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Mozilla/5.0 (compatible; TestURI; +http://testuri.org/)
WebSniffer/1.0 (+http://websniffer.cc/)
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 SeoSiteCheckup (https://seositecheckup.co
Adstxtaggregator.com/1.0
Discourse Forum Onebox v2.0.0.beta10
faviconarchive.org
GreatNews/1.0
MemGator:1.0-rc2 <@WebSciDL>
URLTester/1 CFNetwork/974.1 Darwin/18.0.0 (x86_64)
URLTester/1 CFNetwork/975.0.3 Darwin/18.2.0 (x86_64)
Urlcheckr/2.0
SimpleChecker
reqwest/0.9.9
ReactorNetty/0.7.10.RELEASE
Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/)
https://www.nominet.uk/privacy-notice
<a href='http://www.unchaos.com/'> UnChaos </a> From Chaos To Order Hybrid Web Search Engine.(vadim_gonchar@unchaos.com)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/61.0.3116.0 Mobile Safari/537.36 Chrome-Lighthouse
Trackuity (+http://trackuity.com)
deeris/1.0 (+http://www.deeris.com)
Mediametric
TextRazor Downloader (https://www.textrazor.com)
ddline.cn rank history
Web spyder
AppleSyndication/52.8
AutomaticWPTester/1 CFNetwork/975.0.3 Darwin/17.7.0
CSSCheck/1.2.2
dBpoweramp
Thumbor/6.4.1
Reeder/1000.02.01 CFNetwork/454.11.5 Darwin/10.5.0 (i386) (Macmini2)
worldping-api
Geziyor 1.0
adstxt-worker (www.column6.com)
adstxt.com/1.2
Mozilla/5.0 (compatible; ZnHTTP/7.80)
khttp/1.0.0-SNAPSHOT
IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)
HTTPie/1.0.2
Craw/1.0
internetwache.org v3.4
looid.com Search Engine/0.1
OgScrper/1.0.0
NotionEmbedder
facebookexternalua
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)
FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
Amazon Simple Notification Service Agent
cortex/1.0
AWS Security Scanner
HaxerMen
kubectl/v1.12.0 (linux/amd64) kubernetes/0ed3388
NetSystemsResearch studies the availability of various services across the internet. Our website is netsystemsresearch.com
Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36 LieBaoFast/4.51.3
sc-downloader/2.0 (http://www.searchconcepts.io)
[object Promise] delve.ai website analyse
attohttpc/0.9.0
Bewica-security-scan
BlogBridge 2.13 (http://www.blogbridge.com/)
Boost.Beast/248
DomainLabz
ExoRank
Fuzz Faster U Fool v1.0-rc1
Hexometer
longurl-r-package
looid.com Search Engine/0.1
Mozilla/5.0 (compatible; RyowlEngine/1.0; +https://ryowl.org)
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0; https://app.limber.io) Gecko/20100101 Firefox/25.0
OSPScraper (+https://www.opensyllabusproject.org)
python-httpx/0.11.1
shortURL lengthener/0.1 +http://kevindriscoll.info/
Scamadviser-Frontend
Wolfram HTTPClient 12.
SEOlizer/1.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13
Mozilla/5.0 (pc-x86_64-linux-gnu) Siege/4.0.4
Filestack
Google-Ads-Overview Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 6.0.1; generic) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Version/4.0 Mobile Safari/537.36
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Version/4.0 Mobile Safari/537.36
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36
adreview/1.0
Google-speakr
Google-speakr,gzip(gfe)
PageFreezer
Prebid.js Scraper
Mr.4x3 Powered
Goldfire Server
OhDear.app (+https://ohdear.app/docs/checks/uptime)
Mozilla/5.0 (compatible;Cloudflare-Healthchecks/1.0;+https://www.cloudflare.com/; healthcheck-id: 755842180c7ae904)
WinHttp-Autoproxy-Service/5.1
DynatraceSynthetic/
acebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
Mozilla/5.0 (compatible; Google-Podcast)
Project25499 Scanner (opt-out@project25499.com)
GitHub-Hookshot/a6f2714
Fiery%20Feeds/272 CFNetwork/1125.2 Darwin/19.4.0 (x86_64)
FreshRSS/1.13.0 (Linux; https://freshrss.org)
Gofeed/1.0
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/538.1 (KHTML, like Gecko) QuiteRSS/0.18.12 Safari/538.1
php-requests/1.7
nghttp2/1.40.0
WinHTTP/1.1
Mozilla/5.0 (compatible; Javelin; +https://about.javelin.io/)
swcd (unknown version) CFNetwork/1125.2 Darwin/19.4.0
YahooMailProxy; https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html
Mozilla/5.0 (Java) outbrain
DHSH
DHSH Virus Check
DHSH Ads Check
DHSH Statistik Chec
DHSH Law Check
DHSH User Check
Mozilla/5.0+(compatible; Cula/2.0; https://cula.io/)
Winds: Open Source RSS & Podcast app: https://getstream.io/winds/
XenForo/2.x (https://www.recreationalflying.com)
TECHIAITH_CY (+https://techiaith.cymru)
ScopeContentAG-HTTP-Client www.thescope.com/0.1
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Lush Http Client
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36 The National Archives UK Government Web Archive: http://www.nationalarchives.gov.uk/webarchive/; webarchive@nationalarchives.gov.uk
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 (compatible; NameFo; https://namefo.com/)
Mozilla/5.0 (Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36 (compatible; Statically-Screenshot; +https://statically.io/screenshot)
Mozilla/5.0 (compatible; Domains Project/1.0.8; +https://domainsproject.org)
Mozilla/5.0 (compatible; AffiliateLabz/1.0; +https://affiliatelabz.com/)
Mozilla/5.0 (compatible; 2GDPR/1.2; https://2gdpr.com)
Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1
Microsoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)
LinkPreview/1.5 (http://linkpreview.net)
https://gdnplus.com:Gather Analyze Provide.
Feedshow/x.0 (http://www.feedshow.com; 1 subscriber)
apimon.de/http-apis#redirect
ZoteroTranslationServer/WMF (mailto:services@lists.wikimedia.org)
yomins.com/0.3
WakeletLinkExpander/0.1
ultimate_sitemap_parser/0.5
Swurl Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36
postplanner.com/site-scraping
Nuclei (@pdiscoveryio)
myseosnapshot/1.0
Contextual Code Sites Explorer
Corax - support@coraxcyber.com
Florienzh/1.0 
BKCTwitterUnshortener/0.1 (Contact: systems@cyber.harvard.edu)
Syndirella/0.91pre 
PodcastAddict/v2 - Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-G920F Build/MMB29K) 
PodcastAddict/v5 (+https://podcastaddict.com/; Android podcast app)
PDF24 URL To PDF
MacOutlook/16.41.20091302 (Intelx64 Mac OS X 10.14.4 (Build 18E226))
HubSpot-Link-Resolver
Railgun/5.3.3
RSSMix/3.0
SearchWP
Stripe/1.0 (+https://stripe.com/docs/webhooks)
ZoominfoBot (zoominfobot at zoominfo dot com)
Asana/1.4.0 WebsiteMetadataRetriever
mattermost-5.19.0
WP Rocket/Homepage Preload
Anthill
cf-facebook/6.8.48 Node.js/v6.11.1
LinkAnalyser/2.0
PHP/{5|6|7}.{3|2}.{1|2|3|4|5|6|7|8|9|0}{1|2|3|4|5|6|7|8|9|0}
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0 Observatory/82.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) Project-Resonance (http://project-resonance.com/) (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Mozilla/5.0 (compatible; DNSResearchBot/2.1; +http://195.37.190.77)
Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)
Mozilla/5.0 (compatible; tchelebi/1.0; +http://tchelebi.io)
OnalyticaBot
Pleroma 2.1.50-495-gf9ece1a7-develop+dev; https://search.fedi.app <pleroma@search.fedi.app>; Bot
SmerBot/0.1 (smerity@smerity.com) Language Model Dataset Fetcher
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/71.0.3578.98 Safari/537.36 Prerender (+https://github.com/prerender/prerender)
Bitrix link preview
Mikrotik/6.x Fetch
Mikrotik/7.x Fetch
HTTPing v2.5
Mozilla/5.0 (compatible; JungleKeyThumbnail/1.1; +http://www.junglekey.fr/)
postfix-mta-sts-resolver
go-mtasts/1.0
Mail::STS
Google-SMTP-STS
Netlab360
TLS tester from https://testssl.sh/dev/
TBI-HttpOpenPlugin/0.1.0 (+https://leakix.net/)
TBI-WebScanner/0.0.1 (+https://leakix.net/)
quic-go-HTTP/3
CISPA Web Analyser (https://notify.cispa.saarland)
webtech/1.2.11
Hardenize (https://www.hardenize.com)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0 | Hardenize (https://www.hardenize.com)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0 | Hardenize/v1.1196.1 (https://www.hardenize.com)
Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com
Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers&#39; presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com
PageThing http://pagething.com
PageThing.com
Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; PageThing http://pagething.com); rv:1.9; Gecko/2008052906 Firefox/3.0
ureq
ureq/2.0.0-rc1
ureq/2.0.1
URL/Emacs Emacs/26.1 (X11; x86_64-pc-linux-gnu)
unshortenit 0.4.0
Statically-Images/2.0
Statically-Icon/1.4.1
RadioPublicImageResizer/2.0
nettle (+https://www.nettle.sk)
NihilScio - Educational search engine - +https://www.nihilscio.it/NihilScio.htm
Miro-HttpClient/1.0
mio_httpc 0.9.4
Amazon-Route53-Health-Check-Service (ref c96ad03c-acdf-48c6-8ec7-45806bf08123; report http://amzn.to/1vsZADi)
pnpm/6.18.0 npm/? node/v16.11.1 linux x64
Gradle/7.1 (Linux;4.15.0-162-generic;amd64) (Oracle Corporation;11.0.10;11.0.10+9)
npm/6.14.12 node/v10.24.1 linux x64 ci/drone
pip/21.3.1 {"ci":null,"cpu":"x86_64","distro":{"name":"macOS","version":"11.6"},"implementation":{"name":"CPython","version":"3.9.7"},"installer":{"name":"pip","version":"21.3.1"},"openssl_version":"OpenSSL 1.1.1l  24 Aug 2021","python":"3.9.7","setuptools_version":"59.2.0","system":{"name":"Darwin","release":"20.6.0"}}
pip/21.0.1 {"ci":null,"cpu":"x86_64","distro":{"id":"stretch","libc":{"lib":"glibc","version":"2.24"},"name":"Debian GNU/Linux","version":"9"},"implementation":{"name":"CPython","version":"3.7.9"},"installer":{"name":"pip","version":"21.0.1"},"openssl_version":"OpenSSL 1.1.0l  10 Sep 2019","python":"3.7.9","setuptools_version":"53.0.0","system":{"name":"Linux","release":"4.19.123-coreos"}}
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
Aether
PHP-AYMAPI/1.0
Adyen HttpClient 1.0
affilimate-puppeteer
docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.19.123-coreos os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))
Apache Ant/1.10.9
Synapse/1.36.0
GoogleProber
Lucee (CFML Engine)
RapidLoad/1.0
SAP NetWeaver Application Server (1.0;750)
Y!J-DLC/1.0 (https://support.yahoo-net.jp/PccSearch/s/article/H000007955)
ColdFusion
node-fetch
awin.com - site screen shotter
chatterino-api-cache/1.0 link-resolver