Que tienen de malo estas arañas y bots en el sitemap?

User-agent: SensikaBot
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: Toplistbot
Disallow: /

User-Agent: trendictionbot
Disallow: /

User-agent: ltx71
disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: URLMetriken
Disallow: /

User-agent: urlfan-bot
Disallow: /

User-agent: WebAlta Crawler
Disallow: /

User-agent: WebmasterCoffee
Disallow: /

User-agent: Riddler
Disallow: /

User-Agent: Webwiki
Disallow: /

User-agent: WebsiteWiki
Disallow: /

User-agent: Whitevector Crawler
Disallow: /

User-agent: Psbot
Disallow: /

User-agent: Superfeedr
Disallow: /

User-agent: Whitevector
Disallow: /

User-agent: ConveraCrawler/0.9d
Disallow: /

User-agent: ConveraCrawler/0.9e
Disallow: /

Hacer clic para expandir...