Commits

Hadi Moshayedi committed 139f4af

updated spider_bots list

Comments (0)

Files changed (1)

tracking/middleware.py

 
 title_re = re.compile('<title>(.*?)</title>')
 log = logging.getLogger('tracking.middleware')
-spider_bots = [".*bot.*", ".*AltaVista.*", ".*[Ss]pider.*", ".*Disqus.*", ".*Loadimpact.*"]
+spider_bots = [".*bot.*", ".*AltaVista.*", ".*[Ss]pider.*",
+               ".*Disqus.*", ".*Loadimpact.*", ".*UnwindFetchor.*",
+               ".*PostRank.*", ".*Butterfly.*", ".*JS-Kit.*",
+               ".*Slurp.*", ".*facebookexternalhit.*", ".*Summify.*",
+               ".*Python.*", ]
 
 class VisitorTrackingMiddleware:
     """