I just discovered our image system domain is not being crawled by Google for a long time.
The reason is that all the URLs seem to be blocked by
robots.txt -- but I don't even have one.
Disclaimer: Due to some config testing, I now have a generic allow-everything robots file at the website root. I didn't have one prior to this hour.
We run an image resizing system at a subdomain of our website.
I'm getting a very weird behaviour as Search Console claims to be blocked by
robots.txt, when in fact I don't even have one in the first place.
All URLs at this subdomain give me this result when live testing them:
Trying to debug the issue, I created a robots.txt at the root:
The robots file is even already visible at search results:
The response headers also seem to be ok:
HTTP/2 200 date: Sun, 27 Oct 2019 02:22:49 GMT content-type: image/jpeg set-cookie: __cfduid=d348a8xxxx; expires=Mon, 26-Oct-20 02:22:49 GMT; path=/; domain=.legiaodosherois.com.br; HttpOnly; Secure access-control-allow-origin: * cache-control: public, max-age=31536000 via: 1.1 vegur cf-cache-status: HIT age: 1233 expires: Mon, 26 Oct 2020 02:22:49 GMT alt-svc: h3-23=":443"; ma=86400 expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" server: cloudflare cf-ray: 52c134xxx-IAD
Here are some sample URLs for testing:
https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/10/legiao_zg1YXWVbJwFkxT_ZQR534L90lnm8d2IsjPUGruhqAe.png.jpeg https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/10/legiao_FPutcVi19O8wWo70IZEAkrY3HJfK562panvxblm4SL.png.jpeg https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/09/legiao_gTnwjab0Cz4tp5X8NOmLiWSGEMH29Bq7ZdhVPlUcFu.png.jpeg
What should I do?