Search Engine Block Getting Better, but....

Joe Rebele
Joe Rebele
Offline
0
Germi,

The search engine robot block you put in for me is working great. But, I still have a few that are getting through. The following are the ones that are getting through:

1. Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
2. msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
3. Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
4. Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

The following is a copy of my "Agent Block" from Content Statistics:

Googlebot,Googlebot/2.1,Googlebot-Image/1.0,msnbot,bingbot,FeedBurner,Feedfetcher-Google,discobot,ScoutJet,YoadoBot,MJ12bot,YandexBot,Yandexbot/3.0,YandexMedia,SBlder,Yahoo,FAST Enterprise Crawler,Yahoo! Slurp,Baiduspider/2.0,Baiduspider

Any thoughts on why these are coming through in my content statistics results? One thought would be to add a field in the jos_content_statistics table that puts the agent description and then I could filter based on text rather than an IP address which is difficult. Thanks in advance for any help you could provide!

Thanks,
Joe
Responses (0)
  • There are no replies here yet.
Your Reply