Data Mining

Top 50 open source web crawlers for data mining

A web crawler (also known in other terms like ants, automatic indexers, bots, web spiders, web robots or web scutters) is an automated program, or script, that methodically scans or “crawls” through web pages to create an index of the data it is set to look for. This process is called Web crawling or spidering.

There are various uses for web crawlers, but essentially a web crawler is used to collect/mine data from the Internet. Most search engines use it as a means of providing up-to-date data and to find what’s new on the Internet. Analytics companies and market researchers use web crawlers to determine customer and market trends in a given geography. In this article, we present top 50 open source web crawlers available on the web for data mining.

Top 50 open source web crawlers

NameLanguagePlatform
HeritrixJavaLinux
NutchJavaCross-platform
ScrapyPythonCross-platform
DataparkSearchC++Cross-platform
GNU WgetCLinux
GRUBC#, C, Python, PerlCross-platform
ht://DigC++Unix
HTTrackC/C++Cross-platform
ICDL CrawlerC++Cross-platform
mnoGoSearchCWindows
Norconex HTTP CollectorJavaCross-platform
Open Source ServerC/C++, Java PHPCross-platform
PHP-CrawlerPHPCross-platform
YaCyJavaCross-platform
WebSPHINXJavaCross-platform
WebLechJavaCross-platform
AraleJavaCross-platform
JSpiderJavaCross-platform
HyperSpiderJavaCross-platform
ArachnidJavaCross-platform
SpindleJavaCross-platform
SpiderJavaCross-platform
LARMJavaCross-platform
MetisJavaCross-platform
SimpleSpider>JavaCross-platform
GrunkJavaCross-platform
CAPEKJavaCross-platform
ApertureJavaCross-platform
Smart and Simple Web CrawlerJavaCross-platform
Web HarvestJavaCross-platform
AspseekC++Linux
BixoJavaCross-platform
crawler4jJavaCross-platform
EbotErlandLinux
HounderJavaCross-platform
Hyper EstraierC/C++Cross-platform
OpenWebSpiderC#, PHPCross-platform
PavukCLunix
SphiderPHPCross-platform
XapianC++Cross-platform
Arachnode.netC#Windows
CrawwwlerC++Java
Distributed Web CrawlerC, Java, PythonCross-platform
iCrawlerJavaCross-platform
pycreepJavaCross-platform
OpeseC++Linux
AndjingJava
CcrawlerC#Windows
WebEaterJavaCross-platform
JoBoJavaCross-platform
2,948 Comments
  1. dreamproxies.com 2 years ago
    Reply

    Attractive section of content. I just stumbled upon your web site and in accession capital to
    assert that I acquire in fact enjoyed account
    your blog posts. Anyway I’ll be subscribing to your feeds and even I achievement you access consistently rapidly.

  2. read article 2 years ago
    Reply

    Excellent read, I just passed this onto a friend who was doing a little research on that. And he actually bought me lunch since I found it for him smile So let me rephrase that: Thanks for lunch!

  3. sneaker clearance 2 years ago
    Reply

    I am not sure where you’re getting your info, but good topic.
    I needs to spend some time learning much more or understanding more.
    Thanks for magnificent info I was looking for this information for my mission.

  4. Excellent article. I definitely love this site. Keep writing!

  5. Right here is the perfect blog for everyone who really wants to find out about this topic.
    You understand so much its almost hard to argue with you (not that I really will need to?HaHa).
    You certainly put a brand new spin on a topic which has been discussed for
    a long time. Great stuff, just wonderful!

  6. Greetings! Very helpful advice within this article!

    It is the little changes that make the most significant changes.
    Thanks for sharing!

  7. Hi there! This post couldn’t be written any better!

    Reading through this post reminds me of my good old room mate!
    He always kept chatting aabout this. I will forward this post tto him.

    Fairly certain he will have a good read.
    Thank you for sharing!

  8. 10.0.0.1 2 years ago
    Reply

    Way cool! Some extremely valid points! I appreciate you writing this article plus the rest of the site is also
    really good.

  9. Hi, its pleasant post on the topic of media print, we
    all know media is a enormous source of information.

  10. That is an instance of a small backyard bridge in wood.

  11. Bridges may be built of wooden, bamboo, earth or stone.

  12. What’s up, I log on to your blog regularly. Your writing style is witty,
    keep it up!

  13. adelaide forecast 2 years ago
    Reply

    It’s remarkable designed for me to have a web site, which is valuable designed for my experience.
    thanks admin

  14. Wikipedia 2 years ago
    Reply

    Well I truly liked reading it. This subject procured by you is very useful for proper planning. https://www.inc.com/justin-bariso/amazon-wikimedia-wikipedia-donation-1-million-emotional-intelligence.html

  15. Hugo Carvasi 1 year ago
    Reply

    Good chart! Nice article! thank you for this comprehensive list!

  16. Nice article man. Great fun . X back teacher more info – website please visit

  17. Do you mind if I quote a couple of your posts as long
    as I provide credit and sources back too your webpage?
    My website is in the exact same niche as yours and my visitors would certainly benefit from some of
    the information you present here.Please let me know if this okay with you.
    Many thanks!

  18. 0c.be 1 year ago
    Reply

    Hier vind jе meer informatie οѵer elektrische fietsen.

  19. Temp Agencies 1 year ago
    Reply

    I visited various sites but the audio quality for audio
    songs current at this website is actually superb.

  20. It’s impressive that you are getting ideas from this piece of
    writing as well as from our dialogue made at this time.

  21. Tehran rent car 1 year ago
    Reply

    It’s the be completely web craweler list but some link Forbidden or … .
    Is usefully list for me.
    Thanks.

  22. AaronRhync 1 year ago
    Reply
  23. AaronRhync 1 year ago
    Reply

    http://hydrochlorothiazide.company/ – hydrochlorothiazide online

  24. Buch 1 year ago
    Reply

    Yeni Çıkan Kitap Serileri 2016 Okunan Kitaplar

  25. Keep up the good work! Thanks.

  26. iptv test 1 year ago
    Reply

    Facebook Üzerinden TEST Talep Edebilirsiniz.

  27. AaronRhync 1 year ago
    Reply

    http://hydrochlorothiazide.icu/ – hydrochlorothiazide

  28. AaronRhync 1 year ago
    Reply
  29. AaronRhync 1 year ago
    Reply

    http://trazodone.company/ – trazodone hcl

  30. AaronRhync 1 year ago
    Reply

    http://tadalafil.icu/ – tadalafil

  31. AaronRhync 1 year ago
    Reply

    http://buyadvair.us.com/ – advair diskus

  32. AaronRhync 1 year ago
    Reply

    http://triamterene.us.com/ – triamterene 37.5 mg

  33. AaronRhync 1 year ago
    Reply

    http://celebrex.us.com/ – celebrex online

  34. AaronRhync 1 year ago
    Reply

    http://buynolvadex.us.com/ – nolvadex drug

  35. AaronRhync 1 year ago
    Reply
  36. AaronRhync 1 year ago
    Reply

    http://albuterolonline.us.com/ – albuterol online

  37. AaronRhync 1 year ago
    Reply

    http://buyanafranil.us.org/ – buy anafranil

  38. AaronRhync 1 year ago
    Reply
  39. AaronRhync 1 year ago
    Reply

    http://buyadvair.us.com/ – advair medication

  40. AaronRhync 1 year ago
    Reply

    http://metforminnorx.us.com/ – Generic Metformin

  41. AaronRhync 1 year ago
    Reply
  42. KevinFlids 1 year ago
    Reply

    Natural Stress Solutions CBD Capsules Daytime (Standard): https://arill.us/aturaltressolutions95336

  43. AaronRhync 1 year ago
    Reply
  44. AaronRhync 1 year ago
    Reply

    http://buycelebrex.us.com/ – celebrex for sale

  45. AaronRhync 1 year ago
    Reply
  46. AaronRhync 1 year ago
    Reply

    http://metforminnorx.us.com/ – metformin 850mg

  47. BennyGuake 1 year ago
    Reply
  48. Evaraics 1 year ago
    Reply
  49. AaronRhync 1 year ago
    Reply

    http://toradol.us.com/ – toradol 60 mg

  50. Kiaraics 1 year ago
    Reply
  51. AaronRhync 1 year ago
    Reply
  52. PeorgeSnibe 1 year ago
    Reply
  53. Kiaraics 1 year ago
    Reply
  54. Kiaraics 1 year ago
    Reply
  55. BennyGuake 1 year ago
    Reply
  56. AaronRhync 1 year ago
    Reply

    http://zithromax.us.com/ – buy zithromax

  57. Kiaraics 1 year ago
    Reply
  58. AaronRhync 1 year ago
    Reply

    http://meloxicam.club/ – buy meloxicam

  59. AaronRhync 1 year ago
    Reply

    http://triamterene.us.com/ – buy triamterene

  60. AaronRhync 1 year ago
    Reply

    http://celebrex.us.com/ – celebrex 200mg no perscription needed

  61. Kiaraics 1 year ago
    Reply
  62. Kiaraics 1 year ago
    Reply
  63. Evaraics 1 year ago
    Reply
  64. Kimraics 1 year ago
    Reply
  65. Janeraics 1 year ago
    Reply
  66. Kiaraics 1 year ago
    Reply
  67. Kimraics 1 year ago
    Reply
  68. BennyGuake 1 year ago
    Reply
  69. purchase zoloft 1 year ago
    Reply

    price of zoloft Superb site. zoloft sertraline

  70. SiteElaws 1 year ago
    Reply

    https://zoloftxgeneric50.com – cheap zoloft online order sertraline generic zoloft

  71. Danieljeale 1 year ago
    Reply

    Ich bin 23. Ich habe € 3000. Wie kann ich es am besten nutzen, um mehr Geld zu verdienen: http://rih.co/cryptoinvestbitcoin57206

  72. KevinFlids 1 year ago
    Reply
  73. Kimraics 1 year ago
    Reply
  74. KevinFlids 1 year ago
    Reply

    SEnuke TNG 2019 Version Reviewed: DISCOUNT & HUGE BONUS: http://www.abcagency.se/bestseotools93644

  75. Lisaraics 1 year ago
    Reply
  76. MichaelCoift 1 year ago
    Reply
  77. Kimraics 1 year ago
    Reply
  78. KevinFlids 1 year ago
    Reply

    SEnuke TNG 2019 Version Reviewed: DISCOUNT & HUGE BONUS: http://www.lookweb.it/bestseotools83962

  79. MichaelCoift 1 year ago
    Reply
  80. Kimraics 1 year ago
    Reply
  81. Proses 1 year ago
    Reply

    great list thanks

  82. BennyGuake 1 year ago
    Reply
  83. Miguelmossy 1 year ago
    Reply

    $10000 per day Bitcoin Brokers | Top Bitcoin Brokers Reviews | Bitcoin Binary Options: https://lil.ink/15000investbinarycrypto91098

  84. BennyGuake 1 year ago
    Reply
  85. Evaraics 1 year ago
    Reply
  86. BennyGuake 1 year ago
    Reply
  87. AaronRhync 1 year ago
    Reply
  88. AaronRhync 1 year ago
    Reply
  89. AaronRhync 1 year ago
    Reply
  90. BennyGuake 1 year ago
    Reply
  91. AaronRhync 1 year ago
    Reply
  92. AaronRhync 1 year ago
    Reply

    http://medrol.international/ – medrol generic

  93. AaronRhync 1 year ago
    Reply

    http://pyridium.network/ – pyridium brand name

  94. AaronRhync 1 year ago
    Reply
  95. AaronRhync 1 year ago
    Reply

    http://azithromycin.run/ – azithromycin

  96. AaronRhync 1 year ago
    Reply
  97. AaronRhync 1 year ago
    Reply
  98. AaronRhync 1 year ago
    Reply

    http://metformin.capetown/ – metformin 500mg with out presciption

  99. AaronRhync 1 year ago
    Reply
  100. AaronRhync 1 year ago
    Reply
  101. AaronRhync 1 year ago
    Reply
  102. BennyGuake 1 year ago
    Reply
  103. BennyGuake 1 year ago
    Reply
  104. Danieljeale 1 year ago
    Reply

    Comment gagner 3000 € par jour: http://www.lookweb.it/bestinvestcrepto68006

  105. KevinFlids 1 year ago
    Reply

    Ich bin 23. Ich habe € 3000. Wie kann ich es am besten nutzen, um mehr Geld zu verdienen: https://tinyurl.com/bestinvestcrepto26995

  106. AaronRhync 1 year ago
    Reply

    http://tamoxifen.golf/ – buy tamoxifen

  107. Miguelmossy 1 year ago
    Reply

    The Top 5 Best Cryptocurrencies 2019: http://jnl.io/bestinvestcrepto16638

  108. AaronRhync 1 year ago
    Reply

    http://dapoxetine.network/ – dapoxetine approval

  109. AaronRhync 1 year ago
    Reply

    http://albenza.network/ – albenza over the counter

  110. AaronRhync 1 year ago
    Reply
  111. AaronRhync 1 year ago
    Reply

    http://bupropion.run/ – bupropion

  112. AaronRhync 1 year ago
    Reply
  113. AaronRhync 1 year ago
    Reply

    http://flagyl.international/ – metronidazole 500 mg

  114. Kiaraics 1 year ago
    Reply
  115. AaronRhync 1 year ago
    Reply
  116. AaronRhync 1 year ago
    Reply
  117. AaronRhync 1 year ago
    Reply
  118. BennyGuake 1 year ago
    Reply
  119. AaronRhync 1 year ago
    Reply

    http://zithromax.durban/ – buy cheap zithromax online

  120. BennyGuake 1 year ago
    Reply
  121. BennyGuake 1 year ago
    Reply
  122. AaronRhync 1 year ago
    Reply
  123. AaronRhync 1 year ago
    Reply
  124. AaronRhync 1 year ago
    Reply
  125. AaronRhync 1 year ago
    Reply

    http://atarax.network/ – buy atarax online

  126. AaronRhync 1 year ago
    Reply

    http://propranolol.srl/ – propranolol

  127. AaronRhync 1 year ago
    Reply
  128. BennyGuake 1 year ago
    Reply
  129. AaronRhync 1 year ago
    Reply

    http://furosemide2020.com/ – furosemide 40

  130. AaronRhync 1 year ago
    Reply
  131. AaronRhync 1 year ago
    Reply
  132. AaronRhync 1 year ago
    Reply

    http://buyprednisolone.network/ – buy prednisolone

  133. AaronRhync 1 year ago
    Reply
  134. AaronRhync 1 year ago
    Reply
  135. Sueraics 1 year ago
    Reply
  136. AaronRhync 1 year ago
    Reply
  137. AaronRhync 1 year ago
    Reply
  138. AaronRhync 1 year ago
    Reply

    http://provera.network/ – provera cost

  139. AaronRhync 1 year ago
    Reply
  140. AaronRhync 1 year ago
    Reply
  141. KennethCaw 1 year ago
    Reply
  142. AaronRhync 1 year ago
    Reply
  143. Pingback: sildenafil 100mg
  144. AaronRhync 1 year ago
    Reply

    http://albendazole.run/ – albenza cost

  145. AaronRhync 1 year ago
    Reply

    http://buyhydrochlorothiazideonlinenorx.us.com/ – losartan hydrochlorothiazide

  146. AaronRhync 1 year ago
    Reply
  147. AaronRhync 1 year ago
    Reply

    http://abilify.ltda/ – abilify medication purchase

  148. AaronRhync 1 year ago
    Reply

    http://hydrochlorothiazide.joburg/ – hydrochlorothiazide

  149. AaronRhync 1 year ago
    Reply
  150. AaronRhync 1 year ago
    Reply
  151. BennyGuake 1 year ago
    Reply
  152. AaronRhync 1 year ago
    Reply
  153. AaronRhync 1 year ago
    Reply
  154. AaronRhync 1 year ago
    Reply

    http://tretinoin2020.com/ – tretinoin cream

  155. AaronRhync 1 year ago
    Reply

    http://stromectol.run/ – where to buy stromectol

  156. Amyraics 1 year ago
    Reply
  157. AaronRhync 1 year ago
    Reply
  158. BennyGuake 1 year ago
    Reply
  159. MichaelCoift 1 year ago
    Reply
  160. AaronRhync 1 year ago
    Reply

    http://buymotrin.recipes/ – motrin generic

  161. hbr 1 year ago
    Reply

    If some one wants expert view regarding
    blogging and site-building after that i advise him/her to pay a quick
    visit this weblog, Keep up the fastidious work.

  162. AaronRhync 1 year ago
    Reply

    http://buyprednisolone.network/ – buy prednisolone

  163. AaronRhync 1 year ago
    Reply
  164. Sueraics 1 year ago
    Reply
  165. AaronRhync 1 year ago
    Reply
  166. AaronRhync 1 year ago
    Reply
  167. Janeraics 1 year ago
    Reply
  168. AaronRhync 1 year ago
    Reply

    http://cephalexin2020.com/ – online keflex

  169. AaronRhync 1 year ago
    Reply

    http://ordersildenafilonline.us.com/ – where to buy sildenafil citrate online

  170. Kimraics 1 year ago
    Reply
  171. KennethCaw 1 year ago
    Reply
  172. AaronRhync 1 year ago
    Reply
  173. Amyraics 1 year ago
    Reply
  174. BennyGuake 1 year ago
    Reply
  175. AaronRhync 1 year ago
    Reply

    http://biaxin.network/ – biaxin cost

  176. AaronRhync 1 year ago
    Reply
  177. AaronRhync 1 year ago
    Reply

    http://prozac.capetown/ – prozac drug

  178. AaronRhync 1 year ago
    Reply

    http://buyzithromaxonlinenorx.us.com/ – buy cheap zithromax online

  179. AaronRhync 1 year ago
    Reply

    http://buydiclofenaconlinenorx.us.com/ – diclofenac sodium 75mg ec tablets

  180. AaronRhync 1 year ago
    Reply
  181. AaronRhync 1 year ago
    Reply
  182. AaronRhync 1 year ago
    Reply
  183. BennyGuake 1 year ago
    Reply
  184. AaronRhync 1 year ago
    Reply
  185. AaronRhync 1 year ago
    Reply

    http://acyclovir2020.com/ – acyclovir buy online

  186. BennyGuake 1 yea