Яндекс в поиске выдает одинаковых главных страниц, разница только в параметрах.
пример.
XXXXXXXX.ru/?PAGEN_2=6&PAGEN_1=212
XXXXXXXX.ru/?PAGEN_3=57&PAGEN_2=47
XXXXXXXX.ru/?PAGEN_1=17&PAGEN_2=2
XXXXXXXX.ru/?PAGEN_2=57&PAGEN_1=78
XXXXXXXX.ru/?PAGEN_1=38&PAGEN_2=73
XXXXXXXX.ru/?PAGEN_3=45&PAGEN_1=65
И таких дубликатов страниц, причем только главной, больше 1000 в поисковике. Понятно, что они они одинаковы. Не понятно, где он нашел на них ссылки.
файл htaccess
Цитата Options +FollowSymLinks RewriteEngine On RewriteRule ^([^.]+)(?<!/)$ /$1/ [R=301,L] RewriteCond %{HTTP_HOST} ^www.XXXXXXX\.ru$ [NC] RewriteRule ^(.*)$ http://XXXXXXX.ru/$1 [R=301,L] RewriteRule ^news/detail/(.*)$ /news/$1 [L,R=301] RewriteRule ^news/detail/industry_news/(.*)$ /news/$1 [L,R=301] RewriteRule ^news/industry_news/(.*)$ /news/$1 [L,R=301] RewriteRule ^news/detail/news_foreign_archives/(.*)$ /news/$1 [L,R=301] RewriteRule ^news/news_foreign_archives/(.*)$ /news/$1 [L,R=301] RewriteRule ^articles/detail/(.*)$ /articles/$1 [L,R=301] RewriteRule ^articles/detail/all_articles/(.*)$ /articles/$1 [L,R=301] RewriteRule ^articles/all_articles/(.*)$ /articles/$1 [L,R=301] RewriteRule ^articles/detail/popular/(.*)$ /articles/$1 [L,R=301] RewriteRule ^articles/popular/(.*)$ /articles/$1 [L,R=301] RewriteRule ^dictionary/dictionary_(.*).php$ /dictionary/$1/ [L,R=301] RewriteRule ^blanks/show/(.*)$ /blanks/$1 [L,R=301] RewriteRule ^acts/show/(.*)$ /acts/$1 [L,R=301] Options -Indexes ErrorDocument 404 /404.php <IfModule mod_php5.c> php_flag allow_call_time_pass_reference 1 php_flag session.use_trans_sid off php_value display_errors 1 php_value mbstring.func_overload 2 php_value mbstring.internal_encoding UTF-8 </IfModule> <IfModule mod_rewrite.c> Options +FollowSymLinks RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-l RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !/bitrix/urlrewrite.php$ RewriteRule ^(.*)$ /bitrix/urlrewrite.php [L] RewriteRule .* - [E=REMOTE_USER:%{HTTP:Authorization}] </IfModule> <IfModule mod_dir.c> DirectoryIndex index.php index.html </IfModule> <IfModule mod_expires.c> ExpiresActive on ExpiresByType image/jpeg "access plus 3 day" ExpiresByType image/gif "access plus 3 day" </IfModule> AddDefaultCharset utf-8
в логах сейчас бот яндекса шагает вот так
Код [/CODE]
[SIZE=3]
[CODE]95.108.156.251 - - [17/Sep/2014:18:48:00 +0400 - 0.198] "GET /news/3924/www.pravitelstvorb.ru?PAGEN_1=10&SHOWALL_2=1&SHOWALL_3=1 HTTP/1.1" 200 12034 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965280.340
95.108.158.144 - - [17/Sep/2014:18:48:03 +0400 - 0.209] "GET /news/?PAGEN_1=93&PAGEN_2=250&PAGEN_3=16 HTTP/1.1" 200 13042 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965283.088
95.108.156.251 - - [17/Sep/2014:18:48:03 +0400 - 0.229] "GET /news/?PAGEN_1=72&PAGEN_2=41&SHOWALL_3=1 HTTP/1.1" 200 12747 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965283.219
95.108.156.251 - - [17/Sep/2014:18:48:06 +0400 - 0.348] "GET /?PAGEN_1=274&PAGEN_2=55&PAGEN_3=35 HTTP/1.1" 200 11490 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965286.220
95.108.158.144 - - [17/Sep/2014:18:48:07 +0400 - 0.209] "GET /news/?PAGEN_1=25&PAGEN_2=45&PAGEN_3=3 HTTP/1.1" 200 12643 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965287.877
95.108.156.251 - - [17/Sep/2014:18:48:09 +0400 - 0.176] "GET /?PAGEN_1=274&PAGEN_2=55&PAGEN_3=53 HTTP/1.1" 200 11581 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965289.050
95.108.156.251 - - [17/Sep/2014:18:48:11 +0400 - 0.163] "GET /?PAGEN_1=332&PAGEN_3=37 HTTP/1.1" 200 11586 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965291.918
95.108.156.251 - - [17/Sep/2014:18:48:17 +0400 - 0.157] "GET /?PAGEN_1=332&PAGEN_3=46 HTTP/1.1" 200 11564 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965297.571
95.108.156.251 - - [17/Sep/2014:18:48:20 +0400 - 0.162] "GET /?PAGEN_1=332&PAGEN_3=57 HTTP/1.1" 200 11566 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965300.407
95.108.156.251 - - [17/Sep/2014:18:48:23 +0400 - 0.133] "GET /?PAGEN_1=332&PAGEN_3=58 HTTP/1.1" 200 11569 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965303.190
95.108.156.251 - - [17/Sep/2014:18:48:28 +0400 - 0.166] "GET /?PAGEN_1=284&PAGEN_2=70&PAGEN_3=25 HTTP/1.1" 200 11726 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965308.752
95.108.156.251 - - [17/Sep/2014:18:48:34 +0400 - 0.205] "GET /news/4050/'/?PAGEN_1=83&PAGEN_2=2&PAGEN_3=16 HTTP/1.1" 200 11763 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965314.532
95.108.156.251 - - [17/Sep/2014:18:48:37 +0400 - 0.150] "GET /news/?PAGEN_1=20&PAGEN_2=247&PAGEN_3=19 HTTP/1.1" 200 12689 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965317.347
95.108.156.251 - - [17/Sep/2014:18:48:40 +0400 - 0.249] "GET /news/?PAGEN_1=20&PAGEN_2=247&PAGEN_3=10 HTTP/1.1" 200 12791 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965320.245
95.108.156.251 - - [17/Sep/2014:18:48:43 +0400 - 0.212] "GET /news/?PAGEN_1=20&PAGEN_2=247&PAGEN_3=13 HTTP/1.1" 200 12689 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965323.106
95.108.158.144 - - [17/Sep/2014:18:48:43 +0400 - 0.218] "GET /news/?PAGEN_2=37&SHOWALL_1=0&SHOWALL_3=1 HTTP/1.1" 200 13223 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965323.787
95.108.156.251 - - [17/Sep/2014:18:48:45 +0400 - 0.185] "GET /news/?PAGEN_1=34&PAGEN_2=238&SHOWALL_3=1 HTTP/1.1" 200 12742 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965325.939
95.108.158.144 - - [17/Sep/2014:18:48:47 +0400 - 0.190] "GET /news/15397/www.familyspace.ru?PAGEN_1=184&PAGEN_3=17&SHOWALL_2=1 HTTP/1.1" 200 11895 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965327.916
95.108.156.251 - - [17/Sep/2014:18:48:51 +0400 - 0.218] "GET /news/?PAGEN_1=133&PAGEN_2=7 HTTP/1.1" 200 12710 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965331.474
95.108.156.251 - - [17/Sep/2014:18:48:54 +0400 - 0.199] "GET /news/?PAGEN_1=71&PAGEN_2=109&SHOWALL_3=1 HTTP/1.1" 200 13090 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965334.345
95.108.158.144 - - [17/Sep/2014:18:48:56 +0400 - 0.233] "GET /news/?PAGEN_1=78&PAGEN_2=251&PAGEN_3=9 HTTP/1.1" 200 12981 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965336.013
95.108.156.251 - - [17/Sep/2014:18:48:57 +0400 - 0.192] "GET /?PAGEN_1=238&PAGEN_2=2&PAGEN_3=49 HTTP/1.1" 200 11576 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965337.185
95.108.158.144 - - [17/Sep/2014:18:48:59 +0400 - 0.303] "GET /news/?PAGEN_1=90&PAGEN_2=37&PAGEN_3=2 HTTP/1.1" 200 13128 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965339.976
95.108.156.251 - - [17/Sep/2014:18:49:00 +0400 - 0.317] "GET /news/?PAGEN_2=124&PAGEN_3=3&PAGEN_1=2 HTTP/1.1" 200 13123 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965340.150
95.108.156.251 - - [17/Sep/2014:18:49:02 +0400 - 0.129] "GET /blanks/download.php?doc=791&id=15209 HTTP/1.1" 200 7374 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965342.928
95.108.156.251 - - [17/Sep/2014:18:49:05 +0400 - 0.229] "GET /news/?PAGEN_1=102&PAGEN_2=7&PAGEN_3=27 HTTP/1.1" 200 12709 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965345.810
95.108.156.251 - - [17/Sep/2014:18:49:08 +0400 - 0.195] "GET /news/?PAGEN_1=100&PAGEN_2=250&PAGEN_3=13 HTTP/1.1" 200 13045 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965348.654
95.108.156.251 - - [17/Sep/2014:18:49:14 +0400 - 0.169] "GET /news/?PAGEN_1=100&PAGEN_2=250&PAGEN_3=18 HTTP/1.1" 200 12949 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965354.258
95.108.156.251 - - [17/Sep/2014:18:49:17 +0400 - 0.216] "GET /news/?PAGEN_1=100&PAGEN_2=250&PAGEN_3=19 HTTP/1.1" 200 12953 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965357.140
95.108.158.144 - - [17/Sep/2014:18:49:18 +0400 - 0.280] "GET /news/?PAGEN_1=77&PAGEN_2=8&PAGEN_3=16 HTTP/1.1" 200 12650 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965358.634
95.108.158.144 - - [17/Sep/2014:18:49:22 +0400 - 0.126] "GET /?PAGEN_1=77&PAGEN_2=69 HTTP/1.1" 200 11563 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965362.952
95.108.156.251 - - [17/Sep/2014:18:49:23 +0400 - 0.205] "GET /news/?PAGEN_1=100&PAGEN_2=250&PAGEN_3=23 HTTP/1.1" 200 12950 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965363.062
95.108.156.251 - - [17/Sep/2014:18:49:25 +0400 - 0.187] "GET /news/?PAGEN_1=80&PAGEN_2=57&PAGEN_3=13 HTTP/1.1" 200 12661 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965365.904
95.108.158.144 - - [17/Sep/2014:18:49:27 +0400 - 0.424] "GET /news/?PAGEN_1=23&PAGEN_2=110&PAGEN_3=3 HTTP/1.1" 200 12864 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965367.142
95.108.156.251 - - [17/Sep/2014:18:49:28 +0400 - 0.227] "GET /?PAGEN_1=56&PAGEN_2=53&PAGEN_3=4 HTTP/1.1" 200 11621 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965368.780
95.108.156.251 - - [17/Sep/2014:18:49:31 +0400 - 0.160] "GET /news/?PAGEN_1=80&PAGEN_2=57&PAGEN_3=15 HTTP/1.1" 200 12579 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965371.665
95.108.156.251 - - [17/Sep/2014:18:49:34 +0400 - 0.169] "GET /?PAGEN_1=99&PAGEN_2=69&PAGEN_3=45 HTTP/1.1" 200 11578 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965374.505
95.108.156.251 - - [17/Sep/2014:18:49:37 +0400 - 0.181] "GET /news/?PAGEN_1=73&PAGEN_2=70 HTTP/1.1" 200 12854 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965377.614
95.108.156.251 - - [17/Sep/2014:18:49:40 +0400 - 0.215] "GET /news/?PAGEN_1=77&PAGEN_2=237&PAGEN_3=15 HTTP/1.1" 200 12870 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965380.494
95.108.156.251 - - [17/Sep/2014:18:49:45 +0400 - 0.111] "GET /?PAGEN_1=285&PAGEN_2=28&PAGEN_3=51 HTTP/1.1" 200 11575 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 1410965385.978
[/SIZE]
на странице /news/..... нет проблемных ссылок, которые бы создавали дубликаты.
По этому у меня большой вопрос, откуда яндекс берет такие ссылки
Код ?PAGEN_1=56&PAGEN_2=53&PAGEN_3=4