Should I block the '_next' directory in a site using Next.js with robots.txt?

Question

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx where xxx can be different JSON files or images.

My question is whether I should block the /_next directory so that I don't run into duplicate content issues or stale content (as these JSON files change for each deployment).

Answer 1

I wrote an entire blog post on the topic:

Robots.txt - Block _Next Folder in Next.js Application

I tried to block the entire /_next/ directory in my robots.txt file to test out the idea.

I then ran into issues with the CSS not being able to be rendered by Google bot.

So I did the next best thing and disallowed files that were only used for User Experience (Metadata files):

User-agent: *

# Next.JS Crawl Budget Performance Updates
# Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
# The asterisks allows any file name
# The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html) 
Disallow: /*.json$
Disallow: /*_buildManifest.js$
Disallow: /*_middlewareManifest.js$
Disallow: /*_ssgManifest.js$
Disallow: /*.js$

We now save some additional Google Crawl Budget for the files we actually want Google to crawl :)

PS - If you are looking for a great explanation on blocking specific files in your robots.txt file, use this reference:

https://martijnoud.com/robots-txt/#Disallow_a_certain_file

EDIT: 3/13/22 - I decided to block all JS files, I originally was only blocking buildManifest, middlewareManifest, and ssgManifest.. But Google just really likes to check to make sure expired JS is still a 404. So just blocking all JS for now!

EDIT: 8/10/2022 - This is still working like a charm after 5 months of implementing this change. No crawl/indexing issues from Google and my crawl budget is being used much more efficiently.