-
Notifications
You must be signed in to change notification settings - Fork 0
/
bots.txt
306 lines (306 loc) · 3.22 KB
/
bots.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
360Spider
80legs.com
Abonti
AcoonBot
Acunetix
adbeat_bot
AddThis.com
adidxbot
ADmantX
AhrefsBot
AngloINFO
Antelope
Applebot
BaiduSpider
BeetleBot
billigerbot
binlar
bitlybot
BlackWidow
BLP_bbot
BoardReader
Bolt\ 0
BOT\ for\ JCE
Bot\ mailto\:craftbot@yahoo\.com
casper
CazoodleBot
CCBot
checkprivacy
ChinaClaw
chromeframe
Clerkbot
Cliqzbot
clshttp
CommonCrawler
comodo
CPython
crawler4j
Crawlera
CRAZYWEBCRAWLER
Curious
Custo
Default\ Browser\ 0
diavol
DigExt
Digincore
DIIbot
DISCo
discobot
DoCoMo
DotBot
Download\ Demon
DTS.Agent
EasouSpider
eCatch
ecxi
EirGrabber
Elmer
EmailCollector
EmailSiphon
EmailWolf
Exabot
ExaleadCloudView
ExpertSearch
ExpertSearchSpider
Express\ WebPictures
extract
ExtractorPro
EyeNetIE
Ezooms
F2S
FastSeek
feedfinder
FeedlyBot
FHscan
finbot
Flamingo_SearchEngine
FlappyBot
FlashGet
flicky
Flipboard
g00g1e
genieo
Genieo
GetRight
GetWeb\!
GigablastOpenSource
Go\-Ahead\-Got\-It
Go\!Zilla
GozaikBot
grab
GrabNet
Grafula
GrapeshotCrawler
GT\:\:WWW
GTB5
Guzzle
harvest
heritrix
HMView
HomePageBot
HTTP\:\:Lite
HTTrack
HubSpot
ia_archiver
icarus6
id\-search
IDBot
IlseBot
Image\ Stripper
Image\ Sucker
Indigonet
Indy\ Library
integromedb
InterGET
Internet\ Ninja
InternetSeer\.com
IRLbot
ISC\ Systems\ iRc\ Search\ 2\.1
jakarta
Java
JetCar
JobdiggerSpider
JOC\ Web\ Spider
Jooblebot
Jorgee
kanagawa
KINGSpider
kmccrew
larbin
LeechFTP
libwww
Lingewoud
LinkChecker
linkdexbot
LinksCrawler
LinksManager\.com_bot
linkwalker
LinqiaRSSBot
LivelapBot
ltx71
LubbersBot
lwp\-trivial
Mail.RU_Bot
Mass\ Downloader
masscan
maverick
Maxthon$
Mediatoolkitbot
megaindex
MegaIndex
MegaIndex
MFC_Tear_Sample
Microsoft\ URL\ Control
microsoft\.url
MIDown\ tool
miner
Missigua\ Locator
Mister\ PiX
mj12bot
Mozilla.*Indy
Mozilla.*NEWT
MSFrontPage
msnbot
Navroad
NearSite
Net\ Vampire
NetAnts
netEstate
NetSpider
NetZIP
NextGenSearchBot
nutch
Octopus
Offline\ Explorer
Offline\ Navigator
OpenindexSpider
OpenWebSpider
OrangeBot
Owlin
PageGrabber
PagesInventory
panopta
panscient\.com
Papa\ Foto
pavuk
pcBrowser
PECL\:\:HTTP
PeoplePal
Photon
PHPCrawl
planetwork
PleaseCrawl
PNAMAIN.EXE
PodcastPartyBot
prijsbest
proximic
psbot
purebot
pycurl
QuerySeekerSpider
R6_CommentReader
R6_FeedFetcher
RealDownload
ReGet
Riddler
Rippers\ 0
rogerbot
RSSingBot
rv\:1.9.1
RyzeCrawler
SafeSearch
SBIder
Scrapy
Scrapy
Screaming
search_robot
search.goo.ne.jp
SearchmetricsBot
Semrush
SemrushBot
SentiBot
SEOkicks
SeznamBot
ShowyouBot
SightupBot
SISTRIX
sitecheck\.internetseer\.com
siteexplorer.info
SiteSnagger
skygrid
Slackbot
Slurp
SmartDownload
Snoopy
Sogou
Sosospider
spaumbot
Steeler
sucker
SuperBot
Superfeedr
SuperHTTP
SurdotlyBot
Surfbot
tAkeOut
Teleport\ Pro
TinEye
TinEye-bot
Toata\ dragostea\ mea\ pentru\ diavola
Toplistbot
trendictionbot
turnit
TurnitinBot
Twenga
TwengaBot
Twitterbot
URI\:\:Fetch
urllib
Vagabondo
Vagabondo
vikspider
VoidEYE
VoilaBot
WBSearchBot
Web\ Image\ Collector
Web\ Sucker
webalta
WebAuto
WebBandit
WebCollage
WebCopier
WebFetch
WebGo\ IS
WebLeacher
WebReaper
WebSauger
Website\ eXtractor
Website\ Quester
WebStripper
WebWhacker
WebZIP
Wells\ Search\ II
WEP\ Search
WeSEE
Wget
Widow
WinInet
woobot
woopingbot
worldwebheritage.org
Wotbox
WPScan
WWW\-Mechanize
WWWOFFLE
Xaldon\ WebSpider
XoviBot
yacybot
Yandex
YandexBot
YisouSpider
zermelo
Zeus
zh-CN
ZmEu
ZumBot
ZyBorg