Hi patch,
I was actually ignoring them till they don't create a high load problem. They are still less on FE. Actually that 4 days of downtime has decreased the hits we were getting, will probably take a month now to get back on track.
Yahoo doen't contribute much to us though, so once it becomes too hot to handle i will set a crawl delay. (I did it for TE)
Ok I got what you were trying to say It's a "bot" index tool for search engines but I don't know if its really harmful. I will talk to safin about it. Thanks for noticing