Duba robots.txt akan layi: yadda ake nemo kurakurai da ke ɓoye shafinku daga injunan bincike

Amsar a takaice: don duba robots.txt akan layi, shigar da yankinku a cikin kayan aikin bincike na robots.txt na PromoPilot sannan danna "Bincike." Kayan aikin zai sauke fayil ɗin ta atomatik, ya bincika duk umarnin (Bada izini, Hana, Rarraba-jinkiri, Taswirar Yanar Gizo), yana nuna tubalan Wakilin Mai Amfani da robot ɗin da suka yi niyya, ya nemo kurakuran rubutu da ƙa'idodi masu haɗari, duba samuwar URLs masu mahimmanci, kuma ya ba da shawarwari. Kyauta ne kuma babu buƙatar rajista.

Layin da ya fi tsada da na taɓa gani a cikin fayil ɗin robots.txt na wani shine " Disallow: /," wanda ya rage daga haɓakawa. An ƙaddamar da shafin, an biya kuɗin talla, amma zirga-zirgar ababen hawa ta kasance sifili na tsawon makonni uku saboda an toshe robots daga ziyartar duk shafuka.

Menene binciken robots.txt yake dubawa?

Robots.txt gajeriyar fayil ce ta rubutu a tushen gidan yanar gizo, amma yana da sauƙi a yi kuskure: ƙarin slatch ɗaya yana canza ma'anar doka. Kayan aikin yana karanta fayil ɗin kamar yadda robots na bincike suke yi.

Misalin sakamako daga bayanin kayan aiki: "An sami dokoki 2 masu haɗari da ke hana a yi amfani da su da kuma umarnin Taswirar Yanar Gizo 1 da ya ɓace." Waɗannan su ne nau'ikan binciken da suka cancanci kulawa ta musamman.

Yadda ake duba robots.txt akan layi?

1. Buɗe kayan aikin. Binciken Robots.txt yana aiki a cikin mai bincike.
2. Shigar da yankin: https://example.com. Za a sauke fayil ɗin ta atomatik zuwa /robots.txt.
3. Danna "Nazari".
4. Duba jerin umarnin don toshewar wakilin Mai amfani.
5. Yi nazarin kurakurai da ƙa'idodi masu haɗari - wannan shine babban abin da ke cikin rahoton.
6. Gyara fayil ɗin akan sabar ko a cikin saitunan CMS kuma sake gudanar da binciken.

Shafin Kayan Aiki: filin adireshi, maɓallin "Nazari", da bayanin—abin da aka duba, yanayi, da ƙuntatawa.

Duba robots.txt a cikin yanayi biyu masu mahimmanci: kafin ƙaddamar da sabon sigar shafin yanar gizon ku da kuma bayan kowane motsi. Wannan shine lokacin da ƙuntatawa na gwaji za su iya ƙaura zuwa sigar samarwa. Hakanan yana da amfani a duba fayil ɗin robots.txt na mai fafatawa da ku: sassan da aka takaita suna nuna waɗanne shafuka ne suke la'akari da shafukan sabis da kuma yadda aka tsara kundin adireshin su.

Yadda ake karanta sakamakon da kuma abin da za a gyara da farko?

Daga ƙa'idodi da suka shafi dukkan shafin zuwa kayan kwalliya.

1. Disallow: / a cikin User-agent: * toshe. An rufe dukkan shafin. Idan wannan ba sigar gwaji ba ce, cire wannan layin nan da nan.
2. An rufe muhimman sassa. Disallow: /catalog ba wai kawai yana rufe kundin adireshi ba har ma da duk wani abu da ya fara da waɗannan haruffa, kamar /catalog-sale/. Dokokin sun dogara ne akan farkon hanyar, kuma shari'ar tana da mahimmanci.
3. An toshe CSS, JS, da hotuna. Dokoki kamar Disallow: /wp-content/ suna hana Google duba shafin kamar yadda ɗan adam zai yi. Mai rarrafe ya kamata ya loda salo da rubutun.
4. Babu layin taswirar shafin yanar gizo ko kuma yana kaiwa ga 404. Da fatan za a bayar da cikakken adireshin taswirar shafin tare da yarjejeniyar.
5. Tsarin rubutu. Ana amfani da kalmomi marasa tushe a cikin sunayen umarni, dokoki a wajen toshewar Mai Amfani da Wakili, da layuka marasa komai a cikin toshewar—robots suna yin watsi da layuka marasa inganci, kuma dokar ta gaza a hankali.
6. Umarnin da Google ba ya goyon baya. Google ya yi watsi da Crawl-delay, kuma Google ba ya goyon bayan Noindex a cikin robots.txt. Idan kuna dogara da waɗannan, wannan doka ba ta aiki.

Misalin fayil ɗin da ya dace don shafin WordPress:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

An rufe kwamitin gudanarwa, binciken ciki, da shafukan amfani na shago, an kunna mai kula da AJAX (ba tare da waɗanne sassan rukunin yanar gizon ba) kuma an ƙayyade taswirar shafin. Duk sauran abubuwan an kunna su ta hanyar tsoho.

Ta yaya za ku san ko takamaiman URL yana buɗe idan akwai ƙa'idodi da yawa? Google yana amfani da ƙa'idar da ta fi dacewa—wadda take da mafi tsayin hanya. Idan fayil ya ƙunshi Disallow: /blog/ da Allow: /blog/seo/, to shafin /blog/seo/proverka/ a buɗe yake, kuma /blog/news/ a rufe yake. Idan hanyoyin suna da tsayi iri ɗaya, ƙa'idar izinin ta yi nasara. Alamar * tana wakiltar kowane jerin haruffa, kuma $ tana nuna ƙarshen URL ɗin: Disallow: /*.pdf$ yana rufe duk fayilolin PDF, amma ba shafin /pdf-guide/ ba.

Wani abin da aka fi sani da shi shine tubalan da yawa don robot ɗaya, ko toshe na gama gari wanda ya haɗu da wani daban. Idan fayil ya ƙunshi User-agent: Google bot, bot Google yana karanta wannan toshe ne kawai kuma ya yi watsi da ƙa'idodin da ke cikin User-agent: *. Saboda haka, dole ne a kwafi duk ƙuntatawa da suka wajaba a cikin toshewarsa.

Wani batu daban na 2026 shine masu rarrafe na AI. User-agent: GPTBot toshe tare da Disallow: / yana hana OpenAI rarrafe shafin. Wannan shine shawarar ku, amma ku tuna: idan wani rukunin yanar gizo ya rufe ga bots na AI, yana da wahala ga mataimaka su ba da shawarar hakan.

Kwatanta: Binciken robots.txt na PromoPilot da madadinsa

Kammalawa: Search Console yana nuna yadda Google ke ganin fayil ɗin, amma don shafinka ne kawai. Nazarin kan layi yana aiki tare da kowane yanki kuma yana bayyana abin da ke faruwa.

Binciken Robots.txt yana ɗaya daga cikin kayan aikin SEO 87 kyauta na PromoPilot. Idan ba ku da fayil ɗin robots.txt ko kuma ya fi sauƙi a sake rubuta shi daga farko, duba labarin da ke da alaƙa a cikin sashin samar da robots.txt - a nan za ku sami mai ginawa na gani don ƙirƙirar fayil ɗin. Kuma don duba taswirar shafin da kanta, akwai kayan aikin nazarin sitemap.xml daban.

Iyakoki: Wane duba robots.txt ba zai maye gurbinsa ba?

Wannan kayan aikin yana duba fayil ɗin robots.txt ne kawai kuma baya maye gurbin binciken meta na robots matakin shafi. Shafin na iya buɗewa a cikin robots.txt amma an toshe shi da alamar noindex meta ko kanun X-Robots-Tag. Ya kamata a gano irin waɗannan ƙuntatawa ta hanyar yin binciken shafi ko cikakken bincike.

Injunan bincike daban-daban suna fassara umarnin da ba na yau da kullun ba ta hanyoyi daban-daban. Abin da injin bincike ɗaya ke goyan baya, wani kuma zai yi watsi da shi a hankali. Dogara da Sitemap User-agent, Allow, Disallow da taswirar Yanar Gizo - duk manyan injunan bincike suna fahimtar su.

Kuma ku tuna: robots.txt ba tsaro bane. Yana roƙon robots da kada su shiga cikin ladabi, amma baya toshe hanyar shiga. Ana kare kwamitin gudanarwa da bayanan sirri da kalmar sirri, ba layin "Disallow" ba. Bugu da ƙari, robots.txt a buɗe yake ga kowa: ta hanyar lissafa sassan "sirri" a ciki, kuna fallasa su ga mutanen da ke son sani.

Yaushe kayan aiki kyauta bai isa ba?

Nemo ƙa'ida mai haɗari rabin yaƙi ne. Shawarar waɗanne sassa za a rufe dabara ce. Kuna buƙatar ƙwararre idan:

• Shagon yana da dubban shafuka na matattara da zaɓuɓɓukan rarrabawa, kuma ba a san wanne daga cikinsu za a rufe ba;
• Shafin ya daina amfani da sakamakon bincike, amma robots.txt "ya yi kama da yana da kyau";
• robots.txt ana samar da shi ta hanyar CMS ko plugin, kuma an sake rubuta gyare-gyarenku;
• Muna buƙatar yanke shawara ko za mu kunna masu rarrafe na AI da kuma yadda wannan zai shafi gani a cikin mataimaka.

Me zan iya yi wa gidan yanar gizonku?

Sunana Valeriy Danyuk, kuma ni ƙwararren mai haɓakawa ne kuma ƙwararre a fannin SEO, ina da ƙwarewa ta shekaru 12 da kuma abokan ciniki sama da 200. Robots.txt, meta tags, da sitemaps su ne abubuwan farko da nake dubawa a kowace bincike.

SEO da haɓakawa

• Binciken SEO na Turnkey: robots.txt, noindex, canonical, taswirar shafin yanar gizo, da ainihin ƙididdiga—tare da gyare-gyare.
• Autoblog tare da wakilan AI ƙungiya ce ta edita wacce ba ta da ɗan adam wacce ke buga labarai kuma nan take ta buɗe su don yin rijista.
• GEO: Yadda Otal Ya Samu Fitowa a ChatGPT, Gemini, da Perplexity — Nazarin Lamarin Ganuwa a cikin Amsoshin Mataimakan AI.

Shafukan yanar gizo da shagunan kan layi

• Shafin yanar gizo WordPress tare da mai taimakawa AI akan €299 — tare da ingantaccen fayil ɗin robots.txt da taswirar shafin tun daga rana ta farko.
• Shafin yanar gizo akan 1001site: taswirar shafin, bayanai masu tsari, da kuma AI-bot ana saita su ta atomatik.

CRM, lissafi da sarrafa kansa

• ERP, gidan yanar gizo, da kuma bot ɗin Telegram mai amfani da AI don wargaza ta atomatik cikin kwanaki 6.
• Dandalin jigilar kaya tare da shagunan abokan hulɗa - inda kowane shago ke buƙatar fayil ɗin robots.txt mai inganci.

Duk shari'o'in suna cikin fayil ɗin. Aika min adireshinka a Telegram, zan karanta robots.txt ɗinka kuma in sanar da kai idan na rufe wasu sassan da ba dole ba.

Sakamako

1. Duba robots.txt akan layi cikin minti ɗaya ya bayyana ƙa'idodi da ke ɓoye wani shafi ko sassansa daga robots.
2. Gyara tsari: Disallow: / → sassan da aka rufe → CSS da JS → Taswirar Yanar Gizo → Tsarin rubutu.
3. Disallow ba ya cire shafi daga cikin fihirisa; wannan yana buƙatar noindex a kan shafin da aka buɗe.
4. Idan CMS ne ya samar da robots.txt kuma gyare-gyaren ba su tsaya ba, zai fi sauƙi a amince da saitin ga ƙwararre ko a canza zuwa 1001site, inda aka tsara indexing daga cikin akwatin.

Duba fayil ɗinka yanzu: gudanar da bincike na robots.txt kyauta, kuma idan kun sami wasu ƙa'idodi masu haɗari, rubuta mini a Telegram.

Majiyoyi

• Shafin kayan aikin bincike na robots.txt na PromoPilot - duba, sakamakon samfurin, da tambayoyin shafi.
• Cibiyar Bincike Google - Gabatarwa ga robots.txt da umarnin da aka tallafa.
• Taimakon Google Search Console - Rahoton Robots.txt.