Hi
I am indexing 700 URLs and on some of them there are calender scripts which generate a very large number of pages. Is there a smart way to avoid indexing these pages? I can skip folders like /cgi-bin/ but then perhaps miss some other pages under folders with that name.
I am indexing 700 URLs and on some of them there are calender scripts which generate a very large number of pages. Is there a smart way to avoid indexing these pages? I can skip folders like /cgi-bin/ but then perhaps miss some other pages under folders with that name.
Comment