Kryptronic Software Support Forum

I have had this robots.txt file and had no issues like you describe

User-agent: Googlebot
Disallow: /index.php?app=

User-agent: Slurp
Disallow: /index.php?app=


User-agent: Msnbot
Disallow: /index.php?app=


User-agent: Teoma
Disallow: /index.php?app=

User-agent: *
Disallow: /index.php?app=

You're good with that. I actually had the email a friend line as well... but didn't post it because I didn't think thats what you were looking for.

I just realized - when a category has more products than can display on one page, it goes to a that count=20&offset=20 url reference...which begins with index.php?app=....so basically those pages won't get indexed with the disallow above.

Any ideas how to get around this? I don't want to change my # of products per page because it'll load slower.  Here's how the URL is written:

"/index.php?app=ccp0&ns=catshow&ref=christmas-gift-baskets&count=20&offset=20"

The category pages look like "/category/christmas-gift-baskets.php".

Is there a way to use mod-rewrite to make pages like above look like "/category/christmas-gift-baskets-page-2.php" or 3 or 4, etc.. - something along those lines? maybe even "christmas-gift-baskets.php?page=2" ?

Regards,

James...

1. The duplicated _page will be demoted, _not the entire site.
The original/first page will be just fine.

2. You can avoid duplicate penalty by restricting access to your cgi-bin or wherever your "private" scripts reside. This is similar to theblade24's suggestion.
Restricting access to cgi-bin will also restrict access to carts and the shopper's cart won't be indexed.

The only thing I'd be concerned with is if the SE gets to a page with the long URLs, it'll index that first and consider your nice URL pages as the duplicate.

Last edited by Blitzen (11-11-2008 20:31:10)

If you look at the logic in CORE_Display.php you'll see how CCP handles this.

wyattea is long gone but what he posted should work though I'll point out that it should not matter in the end. With the User-agent: * line I don't think you need the individual agent lines such as User-agent: Teoma also. Not all bots respect a robots file anyway.

Should be. If you have SEO enabled you shouldn't be having any duplicate content issues. If your site got indexed before you enabled SEO simply tell Google to remove the non-SEO format links from their index if you really believe it makes a difference.

I have seen/heard a lot of talk about "duplicate content issues" but have never seen anything definitive that says it is really a problem or issue. I do know it is one of the "sellilng points" SEO companies tout.

I would treat Google reports with caution, as I get many warnings relating to things that are not really a problem at all. - Image size being a common one, even with correct implementation of Apache mod_deflate.

The other big issue with Google at the moment is site speed, yet their own tool for analyzing the performance of your website comes with a warning to the effect that the speed checks are all based on estimates ...even though Google is now penalizing slow sites in terms of ranking and visibility.

Finally, if you have a sitemap, this is something else that may help if you don't want to waste a load of time trying to hunt down each search engine's site submission process:

SITEMAP: http://litez.co.uk/sitemap.xml

- just add it to the  file (modified according to your own site address, of course).

Last edited by Design_Wholesale (09-27-2010 10:31:44)