I'm currently performing SEO for my client's project. I'm kind of new to this, so please bear with me.
I've read a lot mixed reviews on inclusion of robots.txt (some say it's good to include even if you have no URLs to block, some say we should not even have it).
Also, a lot of online tools kept mentioning that my client's site did not have a robots.txt, which I why I decided to include the robots.txt into my site.
However, my developers deployed the robots.txt containing these items:
User-agent: * Disallow: /
I understand that by adding that backslash for disallow, it will tell Google to not crawl everything on my site
31st Jan: Wrong robots.txt was deployed
6th Feb: I realized that I couldn't find my website in SERPs, and found the robots.txt error, which I told my developers to change it immediately.
14th Feb: Correct robots.txt was deployed
User-agent: * Disallow:
9th March: Till date, all my pages (except homepge) can't be found in Google
I just can't seem to figure out what the problem is. My only best guess is that because of the disallow backslash, Google kind of "blacklisted" all my webpages. After I changed the robots.txt to the right one, Google have yet to crawl my site, and hence my webpages are still in their "blacklist".
What should I do now?
I thought it could be because of the shift in HTTP to HTTPS because Google webmaster tool sees http and https as separate sites. I've read from here (https://webmasters.stackexchange.com/questions/68435/moving-from-http-to-https-google-search-console) stating that we need to have both old and new sitemaps in GWT.
In my GWT I only had http, so I included https recently. However, the sitemap.xml for both my http and https console are linking to the same thing. Could that be a problem?