User-agent: Scrapy Allow: / User-agent: MJ12bot Disallow: / User-agent: SemrushBot Crawl-delay: 600 User-agent: Yandex Crawl-delay: 30 User-agent: * Disallow: /en/category/senza-categoria-en/ Disallow: /category/senza-categoria/ Crawl-delay: 30 Sitemap: https://www.nuovaelva.it/sitemaps/sitemap.xml Sitemap: https://www.nuovaelva.it/wpms-sitemap.xml Sitemap: https://www.nuovaelva.it/sitemap-files.xml