Newbie trying to make site map

[SnowTao]

I have Zoom 6, build 1020, on a Win 7 machine. I have been very happily using Zoom for a couple years on my site of under 300 pages. Now, I am trying to create a site map for the site. I (re-)ran the indexing and Zoom created both the URLIST.txt and the two (Google) xml files. So that worked.

But alas -- those files only list the URLs and nothing more? The site map function doesn't pull the title of each file also (from the meta data)? Is there some way to get it to do so, or do I need to pull all 289 page titles by hand, one by one? {wince}

Thanks,
Elenor

[Ray]

Both sitemap formats are compliant with Google (XML) and Yahoo (urllist.txt), and neither of those have page titles.

What do you intend to use the sitemap file for? If it is for submitting to these search engines to allow them better coverage of your website, then the page titles would not be of use. I presume you are have some other use in mind?

[SnowTao]

Hi Ray,
Thanks for the quick response!

I want to create a sitemap that is a list of all the files on the site so if someone cannot find the article or column they are looking for in the couple of indexes, they can look on one (sitemap) page and search within that page for whatever.

I'm not quite sure WHY I need a sitemap on a site that has a Zoom search engine -- but apparently it's de rigeur for 'library' sites. (The site is www.psandman.com: it has three date-ordered indexes based on the type of ... text object... columns, articles, mentions 'in the news.' (Those three list every text object on the site.) Some of those text objects are additionally indexed in five topical indexes... There is no single place to see (or search through) every title on the site. (But then, there IS the Zoom search...)

My impression is the sitemap is a fall-back for folks who cannot find what they're looking for on a site; or a page to go to so you can see what-all is on the site. I currently have a sitemap link (inactive) on every page... Am I missing the point of sitemaps? (We have high Google ranking already, but it wouldn't hurt to give Google (and Yahoo!) what they want...
Elenor

[Ray]

"Sitemaps" can be an ambiguous term as it is used interchangeably to mean quite different things.

Google, Yahoo, and more recent usage in web development tend to use this term for files which offer a comprehensive list of links to web crawlers. These files are not designed to be accessible for users, and as such, the two formats provided (a simple text file and a XML file) are like so, and not HTML pages.

An older idea of a "sitemap" is one which is more like a directory of links, an overview of the site, designed for the end user who wishes to bypass the main navigation method. It is rather unfortunate that Google and developers of recent times have adopted (or rather, stolen) the name from this as it's just plain confusing now. In some respect, this is because many developers consider this type of sitemap to be obsolete - with the invention of in-site search engines (such as Zoom itself), and the average site size having grown dramatically such that a comprehensive overview would really be impossible to navigate.

Indeed, if you have an in-site search engine such as Zoom running on your website, it's probably fair to say that a sitemap for the end-user is not necessarily all that helpful. Your site would have to be quite small (by today's standards) for that to be usable.

[SnowTao]

Thanks Raymond. Very helpful. Maybe I can convince my client we don't need one.

Elenor

[SnowTao]

Oh, does having the Google and Yahoo! sitemap files on the site change how bots/spiders act on the site? We got a drastic drop in "visits" (apparently after the posting of the sitemaps), as recorded by Awstats at Hostgator.

[Ray]

Does your sitemap contain less pages than your full site? For example, are you indexing just a sub-section of the full site for searching purposes, but excluding other areas of the site which would normally be picked up by GoogleBot?

If so, it is possible (because you basically told them to follow your rules and index the smaller portion of your site).

If your sitemap contains all the pages of your site, then no, it would not have a negative effect, if anything, it should help them find more pages and better sort them.

These things can be hard to gauge. There are various other factors. Did you upload/modify your robots.txt file recently? Was there a change in your ranking due to an algorithm change at Google (which happens regularly)?

[SnowTao]

Nope, the sitemap (xml and txt) both have 289 URLs, which is about how many files there should be. (I've never actually counted! <G>)

I hate to admit (although I'm not sure quite what I'm admitting TO) that I don't have a robots.txt file. (The implication being, I should go learn about them and make one?) Essentially,there are no pages on the site that shouldn't be indexed. I mean, maybe the guestbook entry (php) page could be skipped.

I have (alas for a long-ago mistake) both an index.htm page and an index.html page (identical but for suffix) and I daren't remove one or the other in case it's linked to from the CDC or some important overseas agency/company... (I suppose I should get around to putting up a forward page in place of one or the other -- but that again requires research to decide which to keep...). But that shouldn't affect robot indexing, should it?

We've always had quite high Google rankings under most of the keywords we'd like to be recognized for (my client/friend has lots of high-quality incoming links, both domestically and from overseas). And we add stuff often enough that we don't seem to lose rankings.

I guess I need to call Hostgator to see if something is broken with Awstats (or with Hostgator's site tracking). It's odd that it seems not to be recording visits (or not displaying them) overnight, but picking back up in the morning.

Thanks Ray for your help,
Elenor

[David]

If you are now only seeing web stats for half the day, then this might explain the reduced count.

[j_uu]

complete info here:

http://www.sitemaps.org/