A lot of bot traffic has a negative impact on the performance of your webshop. Some bots are very useful (e.g. Google), others have no value at all and even have a negative influence on the performance of your shop. We offer you the possibility to block bad user agents with a few mouse clicks, via your Control Panel.
TABLE OF CONTENTS
Add Bad Bots to the Blacklist
If blocking (a part of) the recommended list of bad bots does not have enough positive impact on your webshop, please add unwanted bots to the list yourself.
- Log in to the Control Panel.
- Select Hypernodes in the sidebar on the left.
- Select a Hypernode and then click on Bot Blacklist in the sidebar.
- Click the Analyse logs button to see which user agens visited your webshop today and how many requests they performed.
- To add a bot, fill in the user agent, add a description and click Add bot.
You do not have to copy the entire user agent, using the unique part (the name) is sufficient. Please pay attention: if you use a generic name or abbreviation, you can unintentionally block more than desired. Therefore you must enter at least 4 characters.
Also make sure you do not make any spelling errors!
User agents are not case sensitive, there is no need to use uppercase letters.
Mozilla / 5.0 (compatible; DotBot / 1.1; http://www.opensiteexplorer.org/dotbot, firstname.lastname@example.org)
Please do not block Mozilla, because the results will be disastrous. Block
We also recommend you to specify the entire bot name, in this case,
DotBot. Do not use an abbreviation like Bot, because it is very likely you will block too much.
Overview of the Top 20 Bad Bots Identified by Hypernode
We have already added 20 bots as a suggestion to the Bots blacklist. Please see below for an explanation of these bots. You can tick the box in the column Blocked to either enable or disable blocking per bot. Don't forget to click the Save button when you change something!
Data collected by SEMrushBot is used in: AdSense (Display Advertising) reports, public backlink search engine index maintained as a dedicated tool called “SEMrush backlinks”, Site audit tool that analyzes on-page SEO, technical and usability issues.
Yandex is the largest search engine in Russia. It is ranked as the fifth largest search engine worldwide. You may want to consider blocking Yandex bot if you are not interested in doing business in the Russian market.
OpenLinkProfiler.org ist a link research tool that enables you to check the backlinks of any website.
BLEXBot visits pages to examine and analyse the content. BLEXBot collects information on the link structure of sites and their interlinking on the web. This information is used to avoid any technical and possible legal issues and improve overall online experience.
Ltx71 continuously scans the internet for security research purposes. They claim their crawling is not malicious and only notes summary information for a page.
Rogerbot accesses the code of a site. It is used for link intelligence and other SEO-related information.
Moz Open Site Explorer collects information to create analytics software for marketing purposes.
- Leipzig Corpora Collection
The Leipzig Corpora Collection (LCC) is a project of the University of Leipzig. The LCC offers access to monolingual dictionaries in more than 200 languages. The crawler that visited your website is collecting data for this project. The crawled data are used for language documentation and language statistics which are freely available on their website.
The Linguee bot is a web scraping bot that will scan the content of any website it encounters to search for multilingual text. It does not harvest email addresses, and it won’t index content that is not multilingual.
HubSpot bot is a web scraping bot that scrapes contact information from various lists for companies to use to generate sales leads.
Qwantify is a search engine from France.
Chinese search engine Baidu scrapes webpages and returns updates to the Baidu index. If your business is not interested in marketing to the Chinese market, you might want to consider blocking this bot.
Offers SEO reports about crawling, head elements, content relevance, links, domain, url structure, user experience, etc.
Provides search technology as well as data mining solutions on internet data.
AlphaSeoBot collects new and updated web data. Data is used in AdSense (Display Advertising) reports, public backlink search engine index maintained as a dedicated tool called “AlphaSeo backlinks”and Site audit tool that analyzes on-page SEO, technical and usability issues
Mail.Ru Group is a leading internet company in Russia. Search Mail.Ru is the third largest search engine in Russia. You might want to block this bot if you are not interested in doing business in Russia.
Magebee is an online web service helping Magento merchants and developers to track their store availability and other metrics including security posture. Consider blocking this bot is you do not make use of this tool.
Seznam is the most visited search engine in the Czech Republic. You may want to consider blocking this bot if you're not interested in doing business in the Czech Republic.
- Domain ReAnimator
Domain ReAnimator is a web scraping bot that scrapes domains for various purposes, allowing users to research, register, and rebuild unlimited expired domains.
This robot collects content from the Internet for the purpose of helping educational institutions prevent plagiarism.