Kryptronic Software Support Forum

DarrinWard · 04-06-2003 18:28:21

Hello All..

I am a professional SEO (Search Engine Optimizer). This means that I optimize websites for search engines.

One of my clients uses CCP and I have found some issues which will stop robots (such as Google & Inktomi) from successully spidering the site.

There are too may parameters in the URL such as the IP of the client, the "pg", "usr", "rnd" & "cip" variables.

I have tried to remove these and have successfully done so for catalog pages but not for product pages. My goal is to reduce the parameters in the URL to a maximum of 2.

Does anybody know of a fix which can be implemented to solve this problem. Having more than 2 parameters in the URL will stop search engines from spidering the site, particularly when more than one are dynamic (such as the cip and rnd variables).

Lisaweb · 04-07-2003 08:26:35

<font size="1" color="#FF0000">LAST EDITED ON 04-07-03 AT 09:28 AM (EST)</font><p>Welcome, fellow SEO.

To combat this problem, I have been turning some of my main CCP dymanic pages into statics, but I like this idea even better. Can you tell me how you stripped the links down?

If there is no way to strip them down on the product URLs, statics may be the only answer for these. But that's a headache to code into CCP! I wouldn't even attempt it. I'll just create & link to some duplicate product static pages enough to get them spidered, but let my clients mainly find/use the dynamics.

Lisa

Guest · 04-07-2003 14:32:18

I'll take a look at the scripts when I get a chance and see how it can be done. The only thing is the stats (logs) may not work as intended when we do it. There may be a way to stop it from writing to the url string and leave the stats alone.
I should be able to look at this later this week.
Maybe Nick will have answered by then!

webmaster · 04-08-2003 08:56:49

These can easilly be stripped out of the URLs and hidden form fields in the first routine in ./cgi-bin/library/common/vars.pl. A note about the safety of removing each:

pg -> Don't touch this one - the progrmam will have no idea what to do.

usr -> Safe to remove if you can expect all of your users to have cookies working.

rnd -> Safe to remove if you do not expect any users on Linux systems using Konqueror or Mozilla for a browser (they cache cgi responses).

cip -> Safe to remove. It's just there to fool with paranoid hackers and would-be theives.

___________________________

Nick Hendler
Webmaster, Kryptronic, Inc.

Guest · 04-09-2003 04:25:52

Hi Nick,

I understand that rrc has to do with the DB.

What would you say about removing rrc from the hidden fields and URL?

Is this also dangerous or not?

Thanks.

Lisaweb · 04-09-2003 10:04:13

Nick,

So is it better to code the URLs as:
(example)

cgi-bin/cp-app.cgi?pg=track

or

cgi-bin/cp-app.cgi?&pg=track

Both work, but which do most browsers prefer?

Thanks,
Lisa

webmaster · 04-09-2003 22:35:56

Removing 'rrc', which stands for 'read response code' if anyone cares, will not hose up the program, it will just check for a response code from a processor with each page load. If that field is passed as an 'N', the response code logic is not hit because we know we do not need to hit it. This just saves a little CPU and RAM from each request.

As far as links, 'cgi-bin/cp-app.cgi?pg=track' is preferable as that's the standard. I'm not sure if any browsers will choke on the other format, but I'm not one to take risks like that.

Physical damage to myself is one thing, message with a URL is something completely different

Speaking of that, we have a NorEaster hitting the Outer Banks this weekend and I haven't been in the water since October. The surf is going to be overhead to double overhead here on the banks and I've got a new 8' stick to try out. The water should be a refreshing 55 degrees. I should be back on Monday... Just kidding - I'll be here.

___________________________

Nick Hendler
Webmaster, Kryptronic, Inc.

Lisaweb · 04-09-2003 23:28:12

Nick,

You sure know how to give an old lady heart failure!

I'll be more than happy to let you go wave-riding as soon as my update's done.

Thanks for the tip,
Lisa

webmaster · 04-09-2003 23:39:42

That would be "messing with a URL" not "message with a URL". I just sent you an email about that update - it's not going to be delayed because of the impending safari - it just needs to be tested bigtime. When you get it it will be perfect.

___________________________

Nick Hendler
Webmaster, Kryptronic, Inc.

Lisaweb · 12-02-2003 00:13:49

[size=14]

DO NOT remove the "usr" variable from the vars.pl file. It's required. Otherwise your checkout process will fail.

Wish I'd known.
Lisa

webmaster · 12-04-2003 08:30:08

The usr variables should not be removed whether using SSL for checkout or not. If the user does not have cookies enabled and those variables are removed, the store will become un-functional for users because we'd be relying 100% on cookies.

theblade24 · 12-15-2003 07:43:54

Reading through this then it would be safe to say that removing rrc rnd and cip are really the only ones "safe or worth" removing?

That said, if those three are removed, will the site be spidered completely or are there still issues?

scoutch · 12-15-2003 09:16:52

There will still be issues. All links are needed from the common side.

Lisaweb · 12-15-2003 10:18:13

Hi all,

Just got back from the Search Engine Strategies Conference in Chicago, and they discussed this problem of dynamic pages being overlooked by spiders.

They said the way to fix this is to do a "mod_rewrite", which changes the way the pages are appear to the robots and end users. Example, it can make a php file look like a html file, and so on.

As I've had very little exposure in the past to this whole mod_rewrite business, I'm going to do further research and get back to you all.

If anyone has more info on this, please post it.

Thanks,
Lisa

scoutch · 12-15-2003 13:44:31

For new MODs, e-mail Nick at ec@kryptronic.com . He will give you additional sub routines to apply your request.

Thanks.

theblade24 · 12-16-2003 07:36:03

I have spoken to my hosting company about this. They state all their servers are enabled with mod-rewrite and they will handle this for me at no charge. They can somehow shorten the url and change it to an html page that the robots will see using this method. I'm going to do this immediately.

Lisa, I would check with your hosting company as well. I'll bet it may be easier than we think to do this.

Kryptronic Software Support Forum

#1 04-06-2003 18:28:21

CCP & Robots

#2 04-07-2003 08:26:35

Re: CCP & Robots

#3 04-07-2003 14:32:18

Re: CCP & Robots

#4 04-08-2003 08:56:49

Re: CCP & Robots

#5 04-09-2003 04:25:52

Re: CCP & Robots

#6 04-09-2003 10:04:13

Re: CCP & Robots

#7 04-09-2003 22:35:56

Re: CCP & Robots

#8 04-09-2003 23:28:12

Re: CCP & Robots

#9 04-09-2003 23:39:42

Re: CCP & Robots

#10 12-02-2003 00:13:49

Re: CCP & Robots

#11 12-04-2003 08:30:08

Re: CCP & Robots

#12 12-15-2003 07:43:54

Re: CCP & Robots

#13 12-15-2003 09:16:52

Re: CCP & Robots

#14 12-15-2003 10:18:13

Re: CCP & Robots

#15 12-15-2003 13:44:31

Re: CCP & Robots

#16 12-16-2003 07:36:03

Re: CCP & Robots

Board footer