3

My agency uses SEMRush, and we especially find the Site Audit function helpful for spotting potential issues with our SEO and UX.

However of late, SEMRush has been flagging PDFs quite frequently, stating that they can't be crawled.

It's not consistent though. Some can be crawled without problem, but some can't. As far as I can tell it isn't related to the size of the PDF file, or how they've been created (we have a mix of Word documents saved to PDF, brochures created in Illustrator, and some datasheets which are programmatically generated).

Google has no problem crawling these pages. Checking their indexing status in Search Console shows that every file SEMRush can't crawl has been crawled by Google - although, often not indexed as the content is typically too similar to what is already on the website elsewhere, we are working on sending canonical data in relation to PDFs.

SEMRush's documentation doesn't offer a solution. Requests for support from SEMRush have so far gone unanswered.

My question is: Is this an issue with my server (or my PDFs)?

Improve this question
3
  • 1
    Can you share an example of a PDF that SEMRush complains about? I"d start to investigate by downloading it using wget or curl and then running it through converters that extract the text from it. I'm not sure exactly how SEMRush tries to handle PDFs, but I'd think that if standard command line tools can download them and extract text from them, they would be fine.
    – Stephen Ostermiller ♦
    Commented Sep 4, 2023 at 16:43
  • 1
    Fetching it with wget worked flawlessly and converting it to PGIGuideToPersonalData.txt with pdftotext found all of the words in it.
    – Stephen Ostermiller ♦
    Commented Sep 5, 2023 at 10:20
  • Which would suggest that the problem lies not with my server, my PDFs, or me, but with SEMRush. Which is, regrettably, becoming a more common occurrence. Thanks for your assistance.
    – GeoffAtkins
    Commented Sep 5, 2023 at 10:54

1 Answer 1

Reset to default
6

I had the same issue. I tested the same links from Semrush in my GSC and Google said there was no problem, so I reported it to Semrush.

They said they have a bug which reports PDF crawl issue when JS Rendering is enabled in the audit settings, so the issue is purely a Semrush issue and can be ignored.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.