User-agent: Scrapy Allow: / Sitemap: # global User-agent: * Disallow: /xmlrpc.php Disallow: /map/