Robots.txt file from http://www.taizhou.com.cn All robots will spider the domain User-agent: * Sitemap: http://www.taizhou.com.cn/news.xml Sitemap: http://www.taizhou.com.cn/robots.txt Sitemap: http://news.taizhou.com.cn/robots.txt #Sitemap: http://www.taizhou.com.cn/sitemap/google/2/google.xml #Disallow: