5

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx where xxx can be different JSON files or images.

My question is whether I should block the /_next directory so that I don't run into duplicate content issues or stale content (as these JSON files change for each deployment).

1 Answer 1

4

I wrote an entire blog post on the topic:

Robots.txt - Block _Next Folder in Next.js Application

I tried to block the entire /_next/ directory in my robots.txt file to test out the idea.

I then ran into issues with the CSS not being able to be rendered by Google bot.

So I did the next best thing and disallowed files that were only used for User Experience (Metadata files):

User-agent: *

# Next.JS Crawl Budget Performance Updates
# Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
# The asterisks allows any file name
# The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html) 
Disallow: /*.json$
Disallow: /*_buildManifest.js$
Disallow: /*_middlewareManifest.js$
Disallow: /*_ssgManifest.js$
Disallow: /*.js$

We now save some additional Google Crawl Budget for the files we actually want Google to crawl :)

PS - If you are looking for a great explanation on blocking specific files in your robots.txt file, use this reference:

https://martijnoud.com/robots-txt/#Disallow_a_certain_file

EDIT: 3/13/22 - I decided to block all JS files, I originally was only blocking buildManifest, middlewareManifest, and ssgManifest.. But Google just really likes to check to make sure expired JS is still a 404. So just blocking all JS for now!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.