In accordance with the European law, we inform you that this shop uses cookies to secure the ordering process, observe the customer traffic to improve our website/offer. For more information, You can consult our privacy policy in our TOS.
We have put a protection server wide with a list of not desired bots. Indeed we remarked that most of bots don't respect anymore robots.txt. We analyze traffic logs regularly and reinforce filters. We now close the door to undesired bots by force !You can complement if you wish by setting your own robots.txt for several other reasons... On 09/30/2016, the persistence of Chinese, Russian spam traffic and attacks, we have implemented IP blocking (not perfect as they are also servers elsewhere), see : https://support.yoorshop.hosting/knowledgebase/2464/IP-block-by-country.htmlFor Prestashop, block by countries :https://mypresta.eu/modules/administration-tools/block-ip-free Wordpress :https://fr.wordpress.org/plugins/iq-block-country/ If your site is under attack or not, we propose various tools to use temporarily or permanently :https://support.yoorshop.hosting/knowledgebase/3077/you-are-under-ddos-attack.html Results of our system in awstats of cpanel : 444 Unknown error 4 751 2.9 % 0 See your visits in real time in cpanel (with theme :paper_lantern), section visitors, click all, and in the settings button on the right to see the status/error codes : click status, unclick URL and URL referrer if necessary so that to see all status codes, and if you see 444, 429, 405, 410 error on legitimate pages/URL, contact us so that we look at what we can do... It is normal to see a list of errors in your AWSTATS of cPanel, a lot of traffic can blocked for precise reasons.To know, to limitate the crawling rate of bots, you can add this in your robots.txt after the line containing User-agent: *, this will indicate a delay of 5 seconds crawling frequency between each page : User-agent: * Crawl-delay: 5 We use these error codes to monitor the fight :- 206 : partial content displayed, is a consequence of an issue with our code 410 below, contact us. Beware that it is absolutely normal to have a certain inherent number of 206 in your logs, this is due to load of external resources.- 405 : requests which looks abnormal, types : frame/xss/injection SQL/http.- 410 : URL referrers (russian, chinese websites and others) or keywords blocked. If some pages. on your website don't display correctly, contact us, an issue can be confirmed by presence of error code 206 which would increase.- 429 : Number of requests has been limited by our system on delicate files like wp-login.php, but not only...- 444 : To stop bots, and attempts to delicate files like config.php files.- 499 : Visitor or client has not answered the server request in time, conenction is closed simply. It can be re-opened the next second in case it was a network issue.- 499 : Visitor or client has not answered the server request in time, connection is closed simply. It can be re-opened the next second in case it was a network issue. Excessive connections from one IP can cause this too...- 503 : Too many requests, can be used by anti ddos plugins, but also by our server security... Depending if your website has a lot of normal traffic or not, you can see either 10-20% of traffic blocked, but also triple than your legit traffic you want, that's life ! To read :https://support.yoorshop.hosting/knowledgebase/2464/IP-block-by-country.htmlYou must always set up a robots.txt file that allows to limit the bot traffic on the network, and part of them does not have good intentions: they seek loopholes on your website and/or squatting your website, and therefore its resources. This can also prevent small Ddos attack types...Contact the documentation for your CMS, or other sources on the internet to learn generate a good robot txt file.Also : https://support.google.com/webmasters/answer/6062598?hl=en After, declare this file in your webmaster tools account. We took an example of a site on our server using the CMS spip where we noticed a high level of resources used, so we just analyze the traffic by awstats from cpanel to see if it was normal/legitimate. The problem with this site is obvious as shown in the screenshot, there is a range of IP responsible for all this coming from Russia : though this reveals a robot from the same publisher is engaged in a zombie traffic on this site particular.We check IP identity by prudence to avoid to block an IP we would love (if hosting company, this means bots) :https://apps.db.ripe.net/db-web-ui/#/queryTo block all traffic coming from the IP range in cpanel (because is the case of multiple IPs in this example), go to blockers of IPs and input the broadest IP range than that found in the stats. (Never put big ranges, this will cause slow down to your website, contact us instead, and we'll see at nginx level what we can do)Don't overuse IP ranges, and IP blocking, this can lower your website performances.Once done, you will see in the cPanel Errors section that traffic is now blocked.188.143.232.0-188.143.232.255 Here is the most important primary part of the robots.txt file you can use only, the robot list you can forbid to visit your website : User-agent: 008User-agent: AlexibotUser-agent: AlvinetSpiderUser-agent: Antenne HatenaUser-agent: ApocalXExplorerBotUser-agent: asteriasUser-agent: BackDoorBot/1.0User-agent: BizInformationUser-agent: Black HoleUser-agent: BlowFish/1.0User-agent: BotALotUser-agent: BuiltBotToughUser-agent: Bullseye/1.0User-agent: BunnySlippersUser-agent: CegbfeiehUser-agent: CheeseBotUser-agent: CherryPickerUser-agent: CherryPickerElite/1.0User-agent: CherryPickerSE/1.0User-agent: CopyRightCheckUser-agent: cosmosUser-agent: CrescentUser-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0User-agent: DISCo Pump 3.1User-agent: DittoSpyderUser-agent: dotbotUser-agent: EmailCollectorUser-agent: EmailSiphonUser-agent: EmailWolfUser-agent: EroCrawlerUser-agent: ExtractorProUser-agent: Flamingo_SearchEngineUser-agent: FoobotUser-agent: Harvest/1.5User-agent: hloaderUser-agent: httplibUser-agent: HTTrackUser-agent: HTTrack 3.0User-agent: humanlinksUser-agent: IgentiaUser-agent: InfoNaviRobotUser-agent: JennyBotUser-agent: JikeSpiderUser-agent: Kenjin SpiderUser-agent: LexiBotUser-agent: libWeb/clsHTTPUser-agent: LinkextractorProUser-agent: LinkScan/8.1a UnixUser-agent: LinkWalkerUser-agent: lwp-trivialUser-agent: lwp-trivial/1.34User-agent: Mata HariUser-agent: Microsoft URL Control - 5.01.4511User-agent: Microsoft URL Control - 6.00.8169User-agent: MIIxpcUser-agent: MIIxpc/4.2User-agent: Mister PiXUser-agent: MLBotUser-agent: mogetUser-agent: moget/2.1User-agent: MS Search 4.0 RobotUser-agent: MS Search 5.0 RobotUser-agent: NaverbotUser-agent: NetAntsUser-agent: NetAttacheUser-agent: NetMechanicUser-agent: NICErsPROUser-agent: Offline ExplorerUser-agent: OpenfindUser-agent: OpenindexSpiderUser-agent: ProPowerBot/2.14User-agent: ProWebWalkerUser-agent: psbotUser-agent: QuepasaCreepUser-agent: QueryN MetasearchUser-agent: RepoMonkeyUser-agent: RMAUser-agent: SemrushBotUser-agent: SightupBotUser-agent: SiteBotUser-agent: SiteSnaggerUser-agent: SiteSuckerUser-agent: Sogou web spiderUser-agent: sosospiderUser-agent: SpankBotUser-agent: spannerUser-agent: SpeedyUser-agent: suggybotUser-agent: SuperBotUser-agent: SuperBot/2.6User-agent: suzuranUser-agent: Szukacz/1.4User-agent: TeleportUser-agent: TelesoftUser-agent: The IntraformantUser-agent: TheNomadUser-agent: TightTwatBotUser-agent: TitanUser-agent: toCrawl/UrlDispatcherUser-agent: TosCrawlerUser-agent: True_RobotUser-agent: True_Robot/1.0User-agent: turingosUser-agent: TurnitinBotUser-agent: UrlPoulsUser-agent: URLy WarningUser-agent: VCIUser-agent: Web Image CollectorUser-agent: WebAutoUser-agent: WebBanditUser-agent: WebBandit/3.50User-agent: WebCopierUser-agent: webcopyUser-agent: WebEnhancerUser-agent: WebmasterWorldForumBotUser-agent: webmirrorUser-agent: WebReaperUser-agent: WebSaugerUser-agent: website extractorUser-agent: Website QuesterUser-agent: Webster ProUser-agent: WebStripperUser-agent: WebStripper/2.02User-agent: WebZipUser-agent: wgetUser-agent: WikioFeedBotUser-agent: WinHTTrackUser-agent: WWW-Collector-EUser-agent: Xenu Link Sleuth/1.3.8User-agent: yacyUser-agent: yandexUser-agent: YRSPiderUser-agent: ZeusUser-agent: ZookabotDisallow: /Here is second part as an example, that you must personalize according to your website :User-agent: *Disallow: /cgi-bin/Disallow: /temp/ Disallow: /client.phpSitemap: http://www.yoursite.com/sitemap.xml
We have put a protection server wide with a list of not desired bots. Indeed we remarked that most of bots don't respect anymore robots.txt. We analyze traffic logs regularly and reinforce filters. We now close the door to undesired bots by force !You can complement if you wish by setting your own robots.txt for several other reasons...
On 09/30/2016, the persistence of Chinese, Russian spam traffic and attacks, we have implemented IP blocking (not perfect as they are also servers elsewhere), see :
https://support.yoorshop.hosting/knowledgebase/2464/IP-block-by-country.htmlFor Prestashop, block by countries :https://mypresta.eu/modules/administration-tools/block-ip-free Wordpress :https://fr.wordpress.org/plugins/iq-block-country/
If your site is under attack or not, we propose various tools to use temporarily or permanently :https://support.yoorshop.hosting/knowledgebase/3077/you-are-under-ddos-attack.html
Results of our system in awstats of cpanel :
See your visits in real time in cpanel (with theme :paper_lantern), section visitors, click all, and in the settings button on the right to see the status/error codes : click status, unclick URL and URL referrer if necessary so that to see all status codes, and if you see 444, 429, 405, 410 error on legitimate pages/URL, contact us so that we look at what we can do... It is normal to see a list of errors in your AWSTATS of cPanel, a lot of traffic can blocked for precise reasons.To know, to limitate the crawling rate of bots, you can add this in your robots.txt after the line containing User-agent: *, this will indicate a delay of 5 seconds crawling frequency between each page :
User-agent: *
Crawl-delay: 5
We use these error codes to monitor the fight :- 206 : partial content displayed, is a consequence of an issue with our code 410 below, contact us. Beware that it is absolutely normal to have a certain inherent number of 206 in your logs, this is due to load of external resources.- 405 : requests which looks abnormal, types : frame/xss/injection SQL/http.- 410 : URL referrers (russian, chinese websites and others) or keywords blocked. If some pages. on your website don't display correctly, contact us, an issue can be confirmed by presence of error code 206 which would increase.- 429 : Number of requests has been limited by our system on delicate files like wp-login.php, but not only...- 444 : To stop bots, and attempts to delicate files like config.php files.- 499 : Visitor or client has not answered the server request in time, conenction is closed simply. It can be re-opened the next second in case it was a network issue.- 499 : Visitor or client has not answered the server request in time, connection is closed simply. It can be re-opened the next second in case it was a network issue. Excessive connections from one IP can cause this too...- 503 : Too many requests, can be used by anti ddos plugins, but also by our server security...
Depending if your website has a lot of normal traffic or not, you can see either 10-20% of traffic blocked, but also triple than your legit traffic you want, that's life ! To read :https://support.yoorshop.hosting/knowledgebase/2464/IP-block-by-country.htmlYou must always set up a robots.txt file that allows to limit the bot traffic on the network, and part of them does not have good intentions: they seek loopholes on your website and/or squatting your website, and therefore its resources. This can also prevent small Ddos attack types...Contact the documentation for your CMS, or other sources on the internet to learn generate a good robot txt file.Also : https://support.google.com/webmasters/answer/6062598?hl=en After, declare this file in your webmaster tools account.
We took an example of a site on our server using the CMS spip where we noticed a high level of resources used, so we just analyze the traffic by awstats from cpanel to see if it was normal/legitimate. The problem with this site is obvious as shown in the screenshot, there is a range of IP responsible for all this coming from Russia : though this reveals a robot from the same publisher is engaged in a zombie traffic on this site particular.We check IP identity by prudence to avoid to block an IP we would love (if hosting company, this means bots) :https://apps.db.ripe.net/db-web-ui/#/queryTo block all traffic coming from the IP range in cpanel (because is the case of multiple IPs in this example), go to blockers of IPs and input the broadest IP range than that found in the stats. (Never put big ranges, this will cause slow down to your website, contact us instead, and we'll see at nginx level what we can do)Don't overuse IP ranges, and IP blocking, this can lower your website performances.Once done, you will see in the cPanel Errors section that traffic is now blocked.188.143.232.0-188.143.232.255
User-agent: 008User-agent: AlexibotUser-agent: AlvinetSpiderUser-agent: Antenne HatenaUser-agent: ApocalXExplorerBotUser-agent: asteriasUser-agent: BackDoorBot/1.0User-agent: BizInformationUser-agent: Black HoleUser-agent: BlowFish/1.0User-agent: BotALotUser-agent: BuiltBotToughUser-agent: Bullseye/1.0User-agent: BunnySlippersUser-agent: CegbfeiehUser-agent: CheeseBotUser-agent: CherryPickerUser-agent: CherryPickerElite/1.0User-agent: CherryPickerSE/1.0User-agent: CopyRightCheckUser-agent: cosmosUser-agent: CrescentUser-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0User-agent: DISCo Pump 3.1User-agent: DittoSpyderUser-agent: dotbotUser-agent: EmailCollectorUser-agent: EmailSiphonUser-agent: EmailWolfUser-agent: EroCrawlerUser-agent: ExtractorProUser-agent: Flamingo_SearchEngineUser-agent: FoobotUser-agent: Harvest/1.5User-agent: hloaderUser-agent: httplibUser-agent: HTTrackUser-agent: HTTrack 3.0User-agent: humanlinksUser-agent: IgentiaUser-agent: InfoNaviRobotUser-agent: JennyBotUser-agent: JikeSpiderUser-agent: Kenjin SpiderUser-agent: LexiBotUser-agent: libWeb/clsHTTPUser-agent: LinkextractorProUser-agent: LinkScan/8.1a UnixUser-agent: LinkWalkerUser-agent: lwp-trivialUser-agent: lwp-trivial/1.34User-agent: Mata HariUser-agent: Microsoft URL Control - 5.01.4511User-agent: Microsoft URL Control - 6.00.8169User-agent: MIIxpcUser-agent: MIIxpc/4.2User-agent: Mister PiXUser-agent: MLBotUser-agent: mogetUser-agent: moget/2.1User-agent: MS Search 4.0 RobotUser-agent: MS Search 5.0 RobotUser-agent: NaverbotUser-agent: NetAntsUser-agent: NetAttacheUser-agent: NetMechanicUser-agent: NICErsPROUser-agent: Offline ExplorerUser-agent: OpenfindUser-agent: OpenindexSpiderUser-agent: ProPowerBot/2.14User-agent: ProWebWalkerUser-agent: psbotUser-agent: QuepasaCreepUser-agent: QueryN MetasearchUser-agent: RepoMonkeyUser-agent: RMAUser-agent: SemrushBotUser-agent: SightupBotUser-agent: SiteBotUser-agent: SiteSnaggerUser-agent: SiteSuckerUser-agent: Sogou web spiderUser-agent: sosospiderUser-agent: SpankBotUser-agent: spannerUser-agent: SpeedyUser-agent: suggybotUser-agent: SuperBotUser-agent: SuperBot/2.6User-agent: suzuranUser-agent: Szukacz/1.4User-agent: TeleportUser-agent: TelesoftUser-agent: The IntraformantUser-agent: TheNomadUser-agent: TightTwatBotUser-agent: TitanUser-agent: toCrawl/UrlDispatcherUser-agent: TosCrawlerUser-agent: True_RobotUser-agent: True_Robot/1.0User-agent: turingosUser-agent: TurnitinBotUser-agent: UrlPoulsUser-agent: URLy WarningUser-agent: VCIUser-agent: Web Image CollectorUser-agent: WebAutoUser-agent: WebBanditUser-agent: WebBandit/3.50User-agent: WebCopierUser-agent: webcopyUser-agent: WebEnhancerUser-agent: WebmasterWorldForumBotUser-agent: webmirrorUser-agent: WebReaperUser-agent: WebSaugerUser-agent: website extractorUser-agent: Website QuesterUser-agent: Webster ProUser-agent: WebStripperUser-agent: WebStripper/2.02User-agent: WebZipUser-agent: wgetUser-agent: WikioFeedBotUser-agent: WinHTTrackUser-agent: WWW-Collector-EUser-agent: Xenu Link Sleuth/1.3.8User-agent: yacyUser-agent: yandexUser-agent: YRSPiderUser-agent: ZeusUser-agent: ZookabotDisallow: /Here is second part as an example, that you must personalize according to your website :User-agent: *Disallow: /cgi-bin/Disallow: /temp/ Disallow: /client.phpSitemap: http://www.yoursite.com/sitemap.xml
YOORshop SAS © 2024 All rights reserved
