So I have a weird issue, and I'm not sure if its caused due to my error or what, but when I upload a robots.txt like below:

User-Agent: *
Disallow: /dev/

It shows up in Google Webmaster like below, and also if I redownload said file, it will also appear this way

User-Agent: *Disallow: /dev/Disallow:/wp-admin/Disallow:/wp-includes/Disallow:/wp-login.php

How can I correct this? Is this telling me I'm doing something wrong? Is this a whitespace issue?

  • Are you uploading it as text or binary?? Use text. – closetnoc Aug 18 '16 at 15:58
  • Editing in np++ and uploading via FTP as text, robots.txt – semaj0 Aug 18 '16 at 15:59
  • 1
    Try opening up the file locally both original and the uploaded copy in Windows Notepad and see what you get. This looks like a typical CR LF issue between Windows and Linux. I would assume that NP++ should handle this correctly. It could be the FTP client or server not making the conversion. – closetnoc Aug 18 '16 at 16:04
  • It looks the same in either program.. so i tried to do it through file manager in my cpanel, and it seems to upload fine that way. I guess for robots.txt I will just edit it that way. So I guess my FTP is uploading it wrong.. – semaj0 Aug 18 '16 at 16:42
  • That is what it looks like. There my be a setting in your FTP (and other tools) to preserve [EOL] (end of line) markers or modify them. Some tools have this feature and have a default set. Some look at file extensions for this too, however, being a .txt file, you think this would get handled. It sounds like you have a work-around. Also, not sure if you were using FTP in NP++ or an FTP client. It maybe that one or the other should work better. Cheers!! – closetnoc Aug 18 '16 at 16:50

Much of the time, there are two things to look at.

One is, if you are editing the file on Windows, the CR (carriage control) LF (line feed) [EOL] (end of line) marker needs to be changed to just LF for Linux. Some editors and tools allow for this with an option while others do not. Also keep in mind that there are 3rd. party tools that get in the way of the process such as management tools.

The second is how the file is transferred. Most of the time, an FTP client is used and text is not the default transfer method, however, most FTP clients will use text for a .txt file extension at least. I have had to specifically select text over binary even for obvious text files before. Most FTP clients will handle file extensions correctly. Still, sometimes the file is not converted correctly even with the best of intentions. As well, I have seen FTP servers at fault here though rarely.

Whenever you see a problem like this, check to see if the uploaded file matches the original with something like WordPad where the code can be examined. This can tell you if the file is being uploaded correctly. Many control panels will allow you to open a file transparently to properly compare the two. If the upload is not handling the [EOL] marker properly, then it is likely the transfer method. For FTP, you can always force text transfers over binary. If this still does not work, then in any control panel, you can simply upload the file. Some will make the change for you. If that still does not work use SSH (shell) and edit the file, remove the contents, then cut and paste from the original. This should always work. For smaller files, this works a treat. For larger files, it can be frustratingly slow but should work fine as well.

| improve this answer | |
  • Okay, thanks. I had checked for for binary over text, and text is selected, but I will have to look more in-depth I think. Thanks a lot for some insight! – semaj0 Aug 22 '16 at 12:42

Edit the file again with the editor in your Cpanel. Use Code editor or Simple editor both works.

For this - Select the Robot.txt file and select editor or code editor from the toolbar in file manager (cpanel).

Save changes and test again.

| improve this answer | |
  • This is what I had done, I mention it in the comments below mypost, thanks! – semaj0 Aug 22 '16 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.