Skip to content

Help regarding Robot.txt (google search)

Post this topic on X

In Blogging Ethics

The IndiBlogger forum is now a read-only archive. Every public discussion is still here to read, but new topics and replies are closed.

Uma Maheswari Anandane
from Karaikal |

Hi,

Recently I noticed that one of my blog is not being indexed by Google . And the reason I found was that robot.txt keeps the search engine from crawling my blog

This is the code

----------

User-agent: Mediapartners-Google

Disallow: 

User-agent: *

Disallow: /search

Allow: /

Sitemap: http://perpetualthinkings.blogspot.com/feeds/posts/default?orderby=UPDATED

---------

Can anyone help me what to do next ? Is the sitemap with my feeds means all my feeds is being blocked ? 

Replies 1 to 8 of 8

Ranjith
from hyderabad |

Your robots.txt is perfect. That is the default for all blogs and there is nothing that you need to change.

User-agent: Mediapartners-Google  // for the user agent Mediapartners-Google

Disallow:  // you are not blocking anything

User-agent: *  // for user agents other than  Mediapartners-Google  

Disallow: /search // you are blocking search pages on your blog and they SHOULD be blocked

Allow: / // allow all other pages

Sitemap: http://perpetualthinkings.blogspot.com/feeds/posts/default?orderby=UPDATED // you are simply stating the URL of your sitemap, you are not blocking anything

Uma Maheswari Anandane
from Karaikal |

@ranjith

Thanks ! But why is my blog not indexed by search engine. I have created a sitemap also. One of my other blog is working quite well than this one. Any reasons as to why?

And even if I type the full title of my post along with my blog name, I find nothing in google search instead it shows the same title which I submitted as links to bloggers.com and indiblogger.com.

Aditya Dey
from Kolkata |

I hope you haven't done anything silly with your search description tab under blogger settings.......Try to disable your robot.txt file and check the results.....resubmit your sitemap....let me know what happens

Uma Maheswari Anandane
from Karaikal |

@aditya

I hope I did not

these r my settings for meta tags and crawler

-------------

Description ? [Disabled]  Custom robots.txt ? [Disabled]  Custom robots header tags ? [Disabled] 

-----------

I don't know how to disable robot.txt in blogger blogspot , tried but in vain 

can u guide on how to do?

Uma Maheswari Anandane
from Karaikal |

also i don't think disabling robot.txt file will work

just found out all web domain and sub domain has the same robot txt  http://yoururl.blogspot.co.uk/robots.txt

so what might be the reason for google not indexing or even showing up my blog post? 

Aditya Dey
from Kolkata |

Try one thing, sign in to your Webmaster tools account, click on your blog/site. Then click on "Health" (It will be on the left)....Check whaether there is any crawl error or not.....After that click on "Fetch as Google".....and Fetch your Home page, and all other linked page.....Sometimes it works....

Uma Maheswari Anandane
from Karaikal |

Yes finally found the problem - solved ! - after deleting and adding my blog again in webmaster :)

Ranjith
from hyderabad |

So, is your blog being indexed now?

Aditya Dey
from Kolkata |

Happy to hear that......Happy blogging.... Smile

Uma Maheswari Anandane
from Karaikal |

@ranjith @aditya - Haha not all the urls , I do find some still missing .I have to wait for a couple of days to see the changes though.  And it seems google uses relevent urls from g+, digg and all that pings to consider the urls to be highlighted in search result, I am waiting for the actual outcome .

Harsh Agrawal
from New Delhi |

@Uma

Here are few things you could do for better indexing:

  • Ping your blog manually using Pingler: https://pingler.com
  • Comment on few active WordPress blog, as it will help search engine bots to follow link and crawl your blog.
  • Follow a posting schedule, as this way search engine bots will fix a crawling rate for your blog
Aditya Dey
from Kolkata |

@Harsh-Nice to have you back here again.......Smile

Uma Maheswari Anandane
from Karaikal |

@harsh 

Thank you ! But I think the problem is not with indexing. I did not find any crawl errors on my webmaster too and my specific url seems to be cached 

It is with search result. Instead of showing my blog post title (with the respective page) , it shows my blog title ( my home page with description matching my search result ) and other linked in pages with my title (like the one I submitted the link for indiblogger, bloggers, etc) but not my original page

I tried changing the  code in the <title> html but still i dont see any change