I wrote an entire blog post on the topic:
Robots.txt - Block _Next Folder in Next.js Application
I tried to block the entire
/_next/ directory in my
robots.txt file to test out the idea.
I then ran into issues with the CSS not being able to be rendered by Google bot.
So I did the next best thing and disallowed files that were only used for User Experience (Metadata files):
# Next.JS Crawl Budget Performance Updates
# Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
# The asterisks allows any file name
# The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
We now save some additional Google Crawl Budget for the files we actually want Google to crawl :)
PS - If you are looking for a great explanation on blocking specific files in your
robots.txt file, use this reference:
EDIT: 3/13/22 - I decided to block all JS files, I originally was only blocking buildManifest, middlewareManifest, and ssgManifest.. But Google just really likes to check to make sure expired JS is still a 404. So just blocking all JS for now!