Skip to main content

uncategorized

Facebook's Fascination with My Robots.txt

By Mikko Ahlroth Source: Nytsoi.net
Facebook's Fascination with My Robots.txt

Facebook has been repeatedly accessing a self-hosted Forgejo instance's robots.txt file multiple times per second for at least four days, raising questions about the purpose and necessity of such frequent requests. The excessive traffic to the robots.txt file, which typically only needs to be checked occasionally, suggests either a technical issue with Facebook's crawler or unusual behavior that warrants investigation.

Read Full Article at Nytsoi.net

Related Articles