User-agent: * Disallow: /media # Certain social media sites are whitelisted to allow crawlers to access page markup when links to necgroup.co.uk/media* are shared. User-agent: Twitterbot Allow: /media User-agent: facebookexternalhit Allow: /media Sitemap: https://www.theicc.co.uk/sitemap.xml