Página inicial Explorar Ajuda
Entrar
Shellmiao
/
WeiBoCrawlerFrontEnd
1
0
Fork 0
Arquivos Issues 0 Pull Requests 0 Wiki
Tree: 4e447a3864
Branches Tags
WeiBoCrawler...
/
public
/
robots.txt

robots.txt 67 B
Histórico Raw

123 
  1. # https://www.robotstxt.org/robotstxt.html
    2. 

  2. User-agent: *
    3. 

  3. Disallow:
    4.