After writing a not so nice email to these guys asking them their intent, they did respond and the good thing is their email did not bounce off any Russian or Asian servers. This does not prove though that their site is not spoofed - but at least a response was comforting. Most hackers do not respond as they unleash self running code out onto the net.
I made complaint their bot was hammering my database driven site at 4 times the rate of any other bot that visits me.
They explained their bot was experimental and in beta status. They asked for my logs to show that their bot was causing issue. Of course I nicely responded that I did not want to be a beta tester for their bad acting bot and I know how to easily block their bot, but I do hate it for those unsuspecting folks whose bandwidth is getting sucked dry. No response to that of course as I didn't expect any
If these folks are legit, they are not making very many friends unleashing this bandwidth sucker out on the net. This is again why I think they are suspect or they are just full of themselves. At least they do have some phone numbers listed on the website and I just may get bored one day and call or call Stamford U. since the website owner is using a Stamford email address.
I was told to add a timer into my robots file and asked to report back to them. I have a problem with this mentality, in that, hey I'll screw with your site and its up to you to keep our bad bot at bay.
User Agent: Twiceler
This supposedly will tell the bot to hit your site every 300 seconds (5 minutes), that's if the bot will listen to this request.
The jury is still out on that though as the bot has not reacted to this directive in the past 12 hours. I will give it a couple of days to see if the bot will listen. Right now it is still hammering my site, as it has for 2 weeks now, at 1 second to 2 minutes per hit. It's hit over 25,000 page in my stats. I think I only have maybe 6-10 thousand pages that it could possible read.
The thing that worries me is that I blocked the bot at a higher level for two days and instead of the bot going somewhere else it kept trying to hammer my site. This was deduced because as soon as I took the block off, the Twiceler bot immediately began crawling my site again.
To block the bot from your site totally...add the following code to your .htaccess file:
SetEnvIfNoCase User-Agent .*Twiceler.* bad_bot
deny from env=bad_bot
allow from all
After you do that and if they want to keep hammering your site that's their problem and their bandwidth.
ps. If anyone is so inclined to build a bot trap for bad bots check out this link:http://www.kloth.net/internet/bottrap.php