Data Mining

Top 50 open source web crawlers for data mining

A web crawler (also known in other terms like ants, automatic indexers, bots, web spiders, web robots or web scutters) is an automated program, or script, that methodically scans or “crawls” through web pages to create an index of the data it is set to look for. This process is called Web crawling or spidering.

There are various uses for web crawlers, but essentially a web crawler is used to collect/mine data from the Internet. Most search engines use it as a means of providing up-to-date data and to find what’s new on the Internet. Analytics companies and market researchers use web crawlers to determine customer and market trends in a given geography. In this article, we present top 50 open source web crawlers available on the web for data mining.

Top 50 open source web crawlers

NameLanguagePlatform
HeritrixJavaLinux
NutchJavaCross-platform
ScrapyPythonCross-platform
DataparkSearchC++Cross-platform
GNU WgetCLinux
GRUBC#, C, Python, PerlCross-platform
ht://DigC++Unix
HTTrackC/C++Cross-platform
ICDL CrawlerC++Cross-platform
mnoGoSearchCWindows
Norconex HTTP CollectorJavaCross-platform
Open Source ServerC/C++, Java PHPCross-platform
PHP-CrawlerPHPCross-platform
YaCyJavaCross-platform
WebSPHINXJavaCross-platform
WebLechJavaCross-platform
AraleJavaCross-platform
JSpiderJavaCross-platform
HyperSpiderJavaCross-platform
ArachnidJavaCross-platform
SpindleJavaCross-platform
SpiderJavaCross-platform
LARMJavaCross-platform
MetisJavaCross-platform
SimpleSpider>JavaCross-platform
GrunkJavaCross-platform
CAPEKJavaCross-platform
ApertureJavaCross-platform
Smart and Simple Web CrawlerJavaCross-platform
Web HarvestJavaCross-platform
AspseekC++Linux
BixoJavaCross-platform
crawler4jJavaCross-platform
EbotErlandLinux
HounderJavaCross-platform
Hyper EstraierC/C++Cross-platform
OpenWebSpiderC#, PHPCross-platform
PavukCLunix
SphiderPHPCross-platform
XapianC++Cross-platform
Arachnode.netC#Windows
CrawwwlerC++Java
Distributed Web CrawlerC, Java, PythonCross-platform
iCrawlerJavaCross-platform
pycreepJavaCross-platform
OpeseC++Linux
AndjingJava
CcrawlerC#Windows
WebEaterJavaCross-platform
JoBoJavaCross-platform
29 Comments
  1. dreamproxies.com 3 months ago
    Reply

    Attractive section of content. I just stumbled upon your web site and in accession capital to
    assert that I acquire in fact enjoyed account
    your blog posts. Anyway I’ll be subscribing to your feeds and even I achievement you access consistently rapidly.

  2. read article 3 months ago
    Reply

    Excellent read, I just passed this onto a friend who was doing a little research on that. And he actually bought me lunch since I found it for him smile So let me rephrase that: Thanks for lunch!

  3. sneaker clearance 3 months ago
    Reply

    I am not sure where you’re getting your info, but good topic.
    I needs to spend some time learning much more or understanding more.
    Thanks for magnificent info I was looking for this information for my mission.

  4. Excellent article. I definitely love this site. Keep writing!

  5. Right here is the perfect blog for everyone who really wants to find out about this topic.
    You understand so much its almost hard to argue with you (not that I really will need to?HaHa).
    You certainly put a brand new spin on a topic which has been discussed for
    a long time. Great stuff, just wonderful!

  6. Greetings! Very helpful advice within this article!

    It is the little changes that make the most significant changes.
    Thanks for sharing!

  7. Hi there! This post couldn’t be written any better!

    Reading through this post reminds me of my good old room mate!
    He always kept chatting aabout this. I will forward this post tto him.

    Fairly certain he will have a good read.
    Thank you for sharing!

  8. 10.0.0.1 2 months ago
    Reply

    Way cool! Some extremely valid points! I appreciate you writing this article plus the rest of the site is also
    really good.

  9. Hi, its pleasant post on the topic of media print, we
    all know media is a enormous source of information.

  10. That is an instance of a small backyard bridge in wood.

  11. Bridges may be built of wooden, bamboo, earth or stone.

  12. What’s up, I log on to your blog regularly. Your writing style is witty,
    keep it up!

  13. adelaide forecast 1 month ago
    Reply

    It’s remarkable designed for me to have a web site, which is valuable designed for my experience.
    thanks admin

  14. Wikipedia 4 weeks ago
    Reply

    Well I truly liked reading it. This subject procured by you is very useful for proper planning. https://www.inc.com/justin-bariso/amazon-wikimedia-wikipedia-donation-1-million-emotional-intelligence.html

  15. Hugo Carvasi 2 weeks ago
    Reply

    Good chart! Nice article! thank you for this comprehensive list!

  16. Nice article man. Great fun . X back teacher more info – website please visit

  17. Do you mind if I quote a couple of your posts as long
    as I provide credit and sources back too your webpage?
    My website is in the exact same niche as yours and my visitors would certainly benefit from some of
    the information you present here.Please let me know if this okay with you.
    Many thanks!

  18. 0c.be 1 week ago
    Reply

    Hier vind jе meer informatie οѵer elektrische fietsen.

  19. Temp Agencies 1 week ago
    Reply

    I visited various sites but the audio quality for audio
    songs current at this website is actually superb.

  20. It’s impressive that you are getting ideas from this piece of
    writing as well as from our dialogue made at this time.

  21. Tehran rent car 7 days ago
    Reply

    It’s the be completely web craweler list but some link Forbidden or … .
    Is usefully list for me.
    Thanks.

  22. AaronRhync 3 days ago
    Reply
  23. AaronRhync 2 days ago
    Reply

    http://hydrochlorothiazide.company/ – hydrochlorothiazide online

  24. Buch 1 day ago
    Reply

    Yeni Çıkan Kitap Serileri 2016 Okunan Kitaplar

Leave a Comment

Your email address will not be published.

You may also like

Pin It on Pinterest