Data Mining

Top 50 open source web crawlers for data mining

A web crawler (also known in other terms like ants, automatic indexers, bots, web spiders, web robots or web scutters) is an automated program, or script, that methodically scans or “crawls” through web pages to create an index of the data it is set to look for. This process is called Web crawling or spidering.

There are various uses for web crawlers, but essentially a web crawler is used to collect/mine data from the Internet. Most search engines use it as a means of providing up-to-date data and to find what’s new on the Internet. Analytics companies and market researchers use web crawlers to determine customer and market trends in a given geography. In this article, we present top 50 open source web crawlers available on the web for data mining.

Top 50 open source web crawlers

NameLanguagePlatform
HeritrixJavaLinux
NutchJavaCross-platform
ScrapyPythonCross-platform
DataparkSearchC++Cross-platform
GNU WgetCLinux
GRUBC#, C, Python, PerlCross-platform
ht://DigC++Unix
HTTrackC/C++Cross-platform
ICDL CrawlerC++Cross-platform
mnoGoSearchCWindows
Norconex HTTP CollectorJavaCross-platform
Open Source ServerC/C++, Java PHPCross-platform
PHP-CrawlerPHPCross-platform
YaCyJavaCross-platform
WebSPHINXJavaCross-platform
WebLechJavaCross-platform
AraleJavaCross-platform
JSpiderJavaCross-platform
HyperSpiderJavaCross-platform
ArachnidJavaCross-platform
SpindleJavaCross-platform
SpiderJavaCross-platform
LARMJavaCross-platform
MetisJavaCross-platform
SimpleSpider>JavaCross-platform
GrunkJavaCross-platform
CAPEKJavaCross-platform
ApertureJavaCross-platform
Smart and Simple Web CrawlerJavaCross-platform
Web HarvestJavaCross-platform
AspseekC++Linux
BixoJavaCross-platform
crawler4jJavaCross-platform
EbotErlandLinux
HounderJavaCross-platform
Hyper EstraierC/C++Cross-platform
OpenWebSpiderC#, PHPCross-platform
PavukCLunix
SphiderPHPCross-platform
XapianC++Cross-platform
Arachnode.netC#Windows
CrawwwlerC++Java
Distributed Web CrawlerC, Java, PythonCross-platform
iCrawlerJavaCross-platform
pycreepJavaCross-platform
OpeseC++Linux
AndjingJava
CcrawlerC#Windows
WebEaterJavaCross-platform
JoBoJavaCross-platform
1,255 Comments
  1. dreamproxies.com 5 months ago
    Reply

    Attractive section of content. I just stumbled upon your web site and in accession capital to
    assert that I acquire in fact enjoyed account
    your blog posts. Anyway I’ll be subscribing to your feeds and even I achievement you access consistently rapidly.

  2. read article 5 months ago
    Reply

    Excellent read, I just passed this onto a friend who was doing a little research on that. And he actually bought me lunch since I found it for him smile So let me rephrase that: Thanks for lunch!

  3. sneaker clearance 5 months ago
    Reply

    I am not sure where you’re getting your info, but good topic.
    I needs to spend some time learning much more or understanding more.
    Thanks for magnificent info I was looking for this information for my mission.

  4. Excellent article. I definitely love this site. Keep writing!

  5. Right here is the perfect blog for everyone who really wants to find out about this topic.
    You understand so much its almost hard to argue with you (not that I really will need to?HaHa).
    You certainly put a brand new spin on a topic which has been discussed for
    a long time. Great stuff, just wonderful!

  6. Greetings! Very helpful advice within this article!

    It is the little changes that make the most significant changes.
    Thanks for sharing!

  7. Hi there! This post couldn’t be written any better!

    Reading through this post reminds me of my good old room mate!
    He always kept chatting aabout this. I will forward this post tto him.

    Fairly certain he will have a good read.
    Thank you for sharing!

  8. 10.0.0.1 4 months ago
    Reply

    Way cool! Some extremely valid points! I appreciate you writing this article plus the rest of the site is also
    really good.

  9. Hi, its pleasant post on the topic of media print, we
    all know media is a enormous source of information.

  10. That is an instance of a small backyard bridge in wood.

  11. Bridges may be built of wooden, bamboo, earth or stone.

  12. What’s up, I log on to your blog regularly. Your writing style is witty,
    keep it up!

  13. adelaide forecast 3 months ago
    Reply

    It’s remarkable designed for me to have a web site, which is valuable designed for my experience.
    thanks admin

  14. Wikipedia 3 months ago
    Reply

    Well I truly liked reading it. This subject procured by you is very useful for proper planning. https://www.inc.com/justin-bariso/amazon-wikimedia-wikipedia-donation-1-million-emotional-intelligence.html

  15. Hugo Carvasi 3 months ago
    Reply

    Good chart! Nice article! thank you for this comprehensive list!

  16. Nice article man. Great fun . X back teacher more info – website please visit

  17. Do you mind if I quote a couple of your posts as long
    as I provide credit and sources back too your webpage?
    My website is in the exact same niche as yours and my visitors would certainly benefit from some of
    the information you present here.Please let me know if this okay with you.
    Many thanks!

  18. 0c.be 3 months ago
    Reply

    Hier vind jе meer informatie οѵer elektrische fietsen.

  19. Temp Agencies 3 months ago
    Reply

    I visited various sites but the audio quality for audio
    songs current at this website is actually superb.

  20. It’s impressive that you are getting ideas from this piece of
    writing as well as from our dialogue made at this time.

  21. Tehran rent car 2 months ago
    Reply

    It’s the be completely web craweler list but some link Forbidden or … .
    Is usefully list for me.
    Thanks.

  22. AaronRhync 2 months ago
    Reply
  23. AaronRhync 2 months ago
    Reply

    http://hydrochlorothiazide.company/ – hydrochlorothiazide online

  24. Buch 2 months ago
    Reply

    Yeni Çıkan Kitap Serileri 2016 Okunan Kitaplar

  25. Keep up the good work! Thanks.

  26. iptv test 2 months ago
    Reply

    Facebook Üzerinden TEST Talep Edebilirsiniz.

  27. BennyGuake 2 months ago
    Reply
  28. AaronRhync 2 months ago
    Reply

    http://hydrochlorothiazide.icu/ – hydrochlorothiazide

  29. AaronRhync 2 months ago
    Reply
  30. BennyGuake 2 months ago
    Reply
  31. AaronRhync 2 months ago
    Reply

    http://trazodone.company/ – trazodone hcl

  32. AaronRhync 2 months ago
    Reply

    http://tadalafil.icu/ – tadalafil

  33. AaronRhync 2 months ago
    Reply
  34. AaronRhync 2 months ago
    Reply

    http://buyadvair.us.com/ – advair diskus

  35. BennyGuake 2 months ago
    Reply
  36. AaronRhync 2 months ago
    Reply
  37. BennyGuake 2 months ago
    Reply
  38. Charlesraics 2 months ago
    Reply
  39. MichaelCoift 2 months ago
    Reply
  40. AaronRhync 2 months ago
    Reply

    http://triamterene.us.com/ – triamterene 37.5 mg

  41. AaronRhync 2 months ago
    Reply

    http://celebrex.us.com/ – celebrex online

  42. Charlesraics 2 months ago
    Reply
  43. AaronRhync 2 months ago
    Reply

    http://buynolvadex.us.com/ – nolvadex drug

  44. AaronRhync 2 months ago
    Reply
  45. AaronRhync 2 months ago
    Reply

    http://albuterolonline.us.com/ – albuterol online

  46. AaronRhync 2 months ago
    Reply

    http://buyanafranil.us.org/ – buy anafranil

  47. AaronRhync 2 months ago
    Reply
  48. AaronRhync 2 months ago
    Reply

    http://buyadvair.us.com/ – advair medication

  49. AaronRhync 2 months ago
    Reply

    http://metforminnorx.us.com/ – Generic Metformin

  50. BennyGuake 2 months ago
    Reply
  51. BennyGuake 2 months ago
    Reply
  52. AaronRhync 2 months ago
    Reply
  53. AaronRhync 2 months ago
    Reply
  54. KevinFlids 2 months ago
    Reply

    Natural Stress Solutions CBD Capsules Daytime (Standard): https://arill.us/aturaltressolutions95336

  55. AaronRhync 2 months ago
    Reply
  56. AaronRhync 2 months ago
    Reply

    http://buycelebrex.us.com/ – celebrex for sale

  57. AaronRhync 2 months ago
    Reply
  58. AaronRhync 2 months ago
    Reply

    http://metforminnorx.us.com/ – metformin 850mg

  59. BennyGuake 2 months ago
    Reply
  60. Evaraics 2 months ago
    Reply
  61. AaronRhync 2 months ago
    Reply

    http://toradol.us.com/ – toradol 60 mg

  62. Kiaraics 2 months ago
    Reply
  63. AaronRhync 2 months ago
    Reply
  64. PeorgeSnibe 2 months ago
    Reply
  65. Kiaraics 2 months ago
    Reply
  66. MichaelCoift 2 months ago
    Reply
  67. Kiaraics 2 months ago
    Reply
  68. Charlesraics 2 months ago
    Reply
  69. BennyGuake 2 months ago
    Reply
  70. Judyraics 2 months ago
    Reply
  71. AaronRhync 2 months ago
    Reply

    http://zithromax.us.com/ – buy zithromax

  72. Evaraics 2 months ago
    Reply
  73. Kiaraics 2 months ago
    Reply
  74. AaronRhync 2 months ago
    Reply

    http://meloxicam.club/ – buy meloxicam

  75. AaronRhync 2 months ago
    Reply

    http://triamterene.us.com/ – buy triamterene

  76. Brettstesk 2 months ago
    Reply
  77. Amyraics 2 months ago
    Reply
  78. AaronRhync 2 months ago
    Reply

    http://celebrex.us.com/ – celebrex 200mg no perscription needed

  79. Kiaraics 2 months ago
    Reply
  80. Kiaraics 2 months ago
    Reply
  81. Evaraics 2 months ago
    Reply
  82. BennyGuake 2 months ago
    Reply
  83. Kimraics 2 months ago
    Reply
  84. Evaraics 2 months ago
    Reply
  85. Janeraics 2 months ago
    Reply
  86. Kiaraics 2 months ago
    Reply
  87. Kimraics 2 months ago
    Reply
  88. Lisaraics 2 months ago
    Reply
  89. BennyGuake 2 months ago
    Reply
  90. purchase zoloft 2 months ago
    Reply

    price of zoloft Superb site. zoloft sertraline

  91. SiteElaws 2 months ago
    Reply

    https://zoloftxgeneric50.com – cheap zoloft online order sertraline generic zoloft

  92. MichaelCoift 2 months ago
    Reply
  93. Danieljeale 2 months ago
    Reply

    Ich bin 23. Ich habe € 3000. Wie kann ich es am besten nutzen, um mehr Geld zu verdienen: http://rih.co/cryptoinvestbitcoin57206

  94. Evaraics 2 months ago
    Reply
  95. KevinFlids 2 months ago
    Reply
  96. BennyGuake 1 month ago
    Reply
  97. Kimraics 1 month ago
    Reply
  98. KevinFlids 1 month ago
    Reply

    SEnuke TNG 2019 Version Reviewed: DISCOUNT & HUGE BONUS: http://www.abcagency.se/bestseotools93644

  99. Amyraics 1 month ago
    Reply
  100. Lisaraics 1 month ago
    Reply
  101. MichaelCoift 1 month ago
    Reply
  102. Janeraics 1 month ago
    Reply
  103. MichaelCoift 1 month ago
    Reply
  104. Amyraics 1 month ago
    Reply
  105. Kimraics 1 month ago
    Reply
  106. Kimraics 1 month ago
    Reply
  107. Kiaraics 1 month ago
    Reply
  108. KevinFlids 1 month ago
    Reply

    SEnuke TNG 2019 Version Reviewed: DISCOUNT & HUGE BONUS: http://www.lookweb.it/bestseotools83962

  109. BennyGuake 1 month ago
    Reply
  110. MichaelCoift 1 month ago
    Reply
  111. Kimraics 1 month ago
    Reply
  112. Janeraics 1 month ago
    Reply
  113. Proses 1 month ago
    Reply

    great list thanks

  114. Amyraics 1 month ago
    Reply
  115. Amyraics 1 month ago
    Reply
  116. Lisaraics 1 month ago
    Reply
  117. BennyGuake 1 month ago
    Reply
  118. Miguelmossy 1 month ago
    Reply

    $10000 per day Bitcoin Brokers | Top Bitcoin Brokers Reviews | Bitcoin Binary Options: https://lil.ink/15000investbinarycrypto91098

  119. BennyGuake 1 month ago
    Reply
  120. Charlesraics 1 month ago
    Reply
  121. Evaraics 1 month ago
    Reply
  122. Evaraics 1 month ago
    Reply
  123. BennyGuake 1 month ago
    Reply
  124. AaronRhync 1 month ago
    Reply
  125. AaronRhync 1 month ago
    Reply
  126. AaronRhync 1 month ago
    Reply
  127. AaronRhync 1 month ago
    Reply
  128. Kimraics 1 month ago
    Reply
  129. BennyGuake 1 month ago
    Reply
  130. AaronRhync 1 month ago
    Reply
  131. AaronRhync 1 month ago
    Reply

    http://medrol.international/ – medrol generic

  132. Douglasprabs 1 month ago
    Reply
  133. Kiaraics 1 month ago
    Reply
  134. AaronRhync 1 month ago
    Reply

    http://pyridium.network/ – pyridium brand name

  135. AaronRhync 1 month ago
    Reply
  136. AaronRhync 1 month ago
    Reply

    http://azithromycin.run/ – azithromycin

  137. AaronRhync 1 month ago
    Reply
  138. AaronRhync 1 month ago
    Reply
  139. AaronRhync 1 month ago
    Reply

    http://metformin.capetown/ – metformin 500mg with out presciption

  140. AaronRhync 1 month ago
    Reply
  141. BennyGuake 1 month ago
    Reply
  142. AaronRhync 1 month ago
    Reply
  143. AaronRhync 1 month ago
    Reply
  144. BennyGuake 1 month ago
    Reply
  145. BennyGuake 1 month ago
    Reply
  146. Danieljeale 1 month ago
    Reply

    Comment gagner 3000 € par jour: http://www.lookweb.it/bestinvestcrepto68006

  147. KevinFlids 1 month ago
    Reply

    Ich bin 23. Ich habe € 3000. Wie kann ich es am besten nutzen, um mehr Geld zu verdienen: https://tinyurl.com/bestinvestcrepto26995

  148. AaronRhync 1 month ago
    Reply

    http://tamoxifen.golf/ – buy tamoxifen

  149. Miguelmossy 1 month ago
    Reply

    The Top 5 Best Cryptocurrencies 2019: http://jnl.io/bestinvestcrepto16638

  150. AaronRhync 1 month ago
    Reply

    http://dapoxetine.network/ – dapoxetine approval

  151. Judyraics 1 month ago
    Reply
  152. AaronRhync 1 month ago
    Reply

    http://albenza.network/ – albenza over the counter

  153. AaronRhync 1 month ago
    Reply
  154. AaronRhync 1 month ago
    Reply

    http://bupropion.run/ – bupropion

  155. AaronRhync 1 month ago
    Reply
  156. AaronRhync 1 month ago
    Reply

    http://flagyl.international/ – metronidazole 500 mg

  157. Kiaraics 1 month ago
    Reply
  158. BennyGuake 4 weeks ago
    Reply
  159. Kiaraics 4 weeks ago
    Reply
  160. AaronRhync 4 weeks ago
    Reply
  161. AaronRhync 4 weeks ago
    Reply
  162. AaronRhync 4 weeks ago
    Reply
  163. Judyraics 4 weeks ago
    Reply
  164. BennyGuake 4 weeks ago
    Reply
  165. AaronRhync 4 weeks ago
    Reply

    http://zithromax.durban/ – buy cheap zithromax online

  166. Amyraics 4 weeks ago
    Reply
  167. BennyGuake 4 weeks ago
    Reply
  168. BennyGuake 4 weeks ago
    Reply
  169. AaronRhync 4 weeks ago
    Reply
  170. BennyGuake 4 weeks ago
    Reply
  171. AaronRhync 4 weeks ago
    Reply
  172. BennyGuake 4 weeks ago
    Reply
  173. AaronRhync 4 weeks ago
    Reply
  174. AaronRhync 4 weeks ago
    Reply

    http://atarax.network/ – buy atarax online

  175. AaronRhync 4 weeks ago
    Reply

    http://propranolol.srl/ – propranolol

  176. AaronRhync 4 weeks ago
    Reply
  177. BennyGuake 4 weeks ago
    Reply
  178. AaronRhync 4 weeks ago
    Reply

    http://furosemide2020.com/ – furosemide 40

  179. AaronRhync 4 weeks ago
    Reply
  180. Kimraics 4 weeks ago
    Reply
  181. AaronRhync 4 weeks ago
    Reply
  182. MichaelCoift 4 weeks ago
    Reply
  183. AaronRhync 4 weeks ago
    Reply
  184. Brettstesk 4 weeks ago
    Reply
  185. AaronRhync 4 weeks ago
    Reply

    http://buyprednisolone.network/ – buy prednisolone

  186. AaronRhync 4 weeks ago
    Reply
  187. AaronRhync 4 weeks ago
    Reply
  188. Sueraics 4 weeks ago
    Reply
  189. BennyGuake 4 weeks ago
    Reply
  190. AaronRhync 4 weeks ago
    Reply
  191. BennyGuake 4 weeks ago
    Reply
  192. AaronRhync 4 weeks ago
    Reply
  193. AaronRhync 4 weeks ago
    Reply

    http://provera.network/ – provera cost

  194. AaronRhync 4 weeks ago
    Reply
  195. AaronRhync 4 weeks ago
    Reply
  196. KennethCaw 4 weeks ago
    Reply
  197. AaronRhync 4 weeks ago
    Reply
  198. AaronRhync 4 weeks ago
    Reply
  199. Pingback: sildenafil 100mg
  200. AaronRhync 4 weeks ago
    Reply

    http://albendazole.run/ – albenza cost

  201. AaronRhync 4 weeks ago
    Reply

    http://buyhydrochlorothiazideonlinenorx.us.com/ – losartan hydrochlorothiazide

  202. AaronRhync 4 weeks ago
    Reply
  203. BennyGuake 4 weeks ago
    Reply
  204. AaronRhync 4 weeks ago
    Reply

    http://abilify.ltda/ – abilify medication purchase

  205. AaronRhync 4 weeks ago
    Reply

    http://hydrochlorothiazide.joburg/ – hydrochlorothiazide

  206. AaronRhync 4 weeks ago
    Reply
  207. AaronRhync 4 weeks ago
    Reply
  208. Judyraics 4 weeks ago
    Reply
  209. BennyGuake 4 weeks ago
    Reply
  210. AaronRhync 4 weeks ago
    Reply
  211. AaronRhync 4 weeks ago
    Reply
  212. AaronRhync 4 weeks ago
    Reply

    http://tretinoin2020.com/ – tretinoin cream

  213. Kiaraics 4 weeks ago
    Reply
  214. BennyGuake 3 weeks ago
    Reply
  215. AaronRhync 3 weeks ago
    Reply

    http://stromectol.run/ – where to buy stromectol

  216. Amyraics 3 weeks ago
    Reply
  217. AaronRhync 3 weeks ago
    Reply
  218. BennyGuake 3 weeks ago
    Reply
  219. MichaelCoift 3 weeks ago
    Reply
  220. AaronRhync 3 weeks ago
    Reply
  221. BennyGuake 3 weeks ago
    Reply
  222. AaronRhync 3 weeks ago
    Reply

    http://buymotrin.recipes/ – motrin generic

  223. hbr 3 weeks ago
    Reply

    If some one wants expert view regarding
    blogging and site-building after that i advise him/her to pay a quick
    visit this weblog, Keep up the fastidious work.

  224. AaronRhync 3 weeks ago
    Reply

    http://buyprednisolone.network/ – buy prednisolone

  225. AaronRhync 3 weeks ago
    Reply
  226. Sueraics 3 weeks ago
    Reply
  227. AaronRhync 3 weeks ago
    Reply
  228. AaronRhync 3 weeks ago
    Reply
  229. Janeraics 3 weeks ago
    Reply
  230. Amyraics 3 weeks ago
    Reply
  231. BennyGuake 3 weeks ago
    Reply
  232. AaronRhync 3 weeks ago
    Reply

    http://cephalexin2020.com/ – online keflex

  233. AaronRhync 3 weeks ago
    Reply

    http://ordersildenafilonline.us.com/ – where to buy sildenafil citrate online

  234. Kimraics 3 weeks ago
    Reply
  235. Kimraics 2 weeks ago
    Reply
  236. KennethCaw 2 weeks ago
    Reply
  237. AaronRhync 2 weeks ago
    Reply
  238. Amyraics 2 weeks ago
    Reply
  239. BennyGuake 2 weeks ago
    Reply
  240. AaronRhync 2 weeks ago
    Reply

    http://biaxin.network/ – biaxin cost

  241. Evaraics 2 weeks ago
    Reply
  242. Judyraics 2 weeks ago
    Reply
  243. AaronRhync 2 weeks ago
    Reply
  244. AaronRhync 2 weeks ago
    Reply
  245. Sueraics 2 weeks ago
    Reply
  246. AaronRhync 2 weeks ago
    Reply

    http://prozac.capetown/ – prozac drug

  247. Janeraics 2 weeks ago
    Reply
  248. BennyGuake 2 weeks ago
    Reply
  249. AaronRhync 2 weeks ago
    Reply

    http://buyzithromaxonlinenorx.us.com/ – buy cheap zithromax online

  250. AaronRhync 2 weeks ago
    Reply

    http://buydiclofenaconlinenorx.us.com/ – diclofenac sodium 75mg ec tablets

  251. AaronRhync 2 weeks ago
    Reply
  252. AaronRhync 2 weeks ago
    Reply
  253. AaronRhync 2 weeks ago
    Reply
  254. BennyGuake 2 weeks ago
    Reply
  255. AaronRhync 2 weeks ago
    Reply