Mai samar da robots.txt akan layi: yadda ake ƙirƙirar fayil ɗin da ya dace cikin mintuna 5 ba tare da kurakurai ba

Amsa a takaice: don ƙirƙirar fayil ɗin robots.txt akan layi, buɗe janareta na PromoPilot robots.txt, zaɓi saitin da aka saita (Custom, Izini duka, Hana duk, WordPress, Laravel, ko Next.js), shigar da hanyoyin Disallow da Allow, shigar da adireshin gidan yanar gizon layin Taswirar Yanar Gizo, kuma zaɓi duba Crawl-delay ko toshe masu rarrafe na AI. Fayil ɗin da aka kammala zai bayyana a dama - zaku iya kwafa shi ko sauke shi. Kyauta ne kuma babu buƙatar rajista.

Ina rubuta robots.txt ga kowane sabon gidan yanar gizo kuma na san yadda yake da sauƙi a yi kurakurai da hannu: ƙarin sarari, slash da aka manta, da ƙa'ida a wajen toshe. Mai gina robots.txt yana kawar da wannan matsalar—kuna tunanin abin da za ku rufe, ba tsarin rubutu ba. A ƙasa, za mu yi bayani kan yadda ake amfani da shi, abin da za a rufe, da kuma inda za a sanya fayil ɗin da aka gama.

Me injin samar da robots.txt ke yi?

Janareta tsari ne maimakon editan rubutu. Za ka zaɓi saitunan, kuma kayan aikin zai haɗa su cikin fayil mai inganci kuma ya nuna su a cikin samfoti kai tsaye.

Yana da mahimmanci a fahimci iyakancewar: janareta yana ƙirƙirar fayil, amma baya duba wanda ke akwai. Idan kuna da fayil ɗin robots.txt kuma kuna son nemo kurakurai a ciki, kuna buƙatar bincika fayil ɗin robots.txt - labarin da ke cikin sashe na gaba ya keɓe ga wannan.

Yadda ake ƙirƙirar robots.txt akan layi?

1. Buɗe janareta na robots.txt.
2. Zaɓi saitin da aka saita don injin ku. Idan injin ku bai jera ba, zaɓi "Na musamman."
3. Shigar da adireshin gidan yanar gizon — https://example.com. Wannan zai ƙirƙiri layin Sitemap.
4. Ƙara hanyoyin da ba a yarda da su ba, ɗaya a kowace layi: /admin/, /tmp/, /search/.
5. Idan ya cancanta, ƙara Allow exceptions a cikin fayilolin da aka takaita, kamar /uploads/images/.
6. Duba akwatunan: Taswirar Yanar Gizo, Jinkirin Rarrafe, toshe masu rarrafe na AI.
7. Kwafi ko sauke fayil ɗin sannan a loda shi zuwa tushen shafin.

A gefen hagu na shafin akwai saitunan, kuma a gefen dama akwai samfoti kai tsaye na fayil ɗin tare da maɓallan "Kwafi" da "Zazzage robots.txt".

Dole ne fayil ɗin ya kasance a tushen yankin kawai: https://example.com/robots.txt. Robots ba za su same shi a cikin babban fayil ɗin /blog/robots.txt ko a wani ƙaramin yanki ba - kowane ƙaramin yanki yana da nasa fayil ɗin robots.txt.

Inda ainihin fayil ɗin da aka sauke zai dogara ne akan dandamalin:

• Bayar da masauki na yau da kullun - ta hanyar mai sarrafa fayil ɗin kwamitin ko FTP zuwa babban fayil ɗin da babban shafin yake (sau da yawa public_html ko www).
• WordPress — idan babu fayil na zahiri, injin yana amfani da fayil ɗin robots.txt na kama-da-wane. Loda fayil ɗin zuwa tushen zai maye gurbinsa; zaɓi na biyu shine amfani da editan fayil a cikin plugin ɗin SEO.
• Next.js da sauran tsare-tsare — ana sanya fayil ɗin a cikin babban fayil ɗin public ko kuma an samar da shi daga lambar aikin don kada ya ɓace yayin aiwatarwa na gaba.
• Masu gina gidan yanar gizo - galibi ana gyara fayil ɗin a cikin saitunan SEO maimakon loda shi da hannu.

Dole ne a sanya wa fayil ɗin suna robots.txt a ƙananan haruffa kuma a adana shi a cikin UTF-8 wanda ke ɓoye a matsayin rubutu mara tsari. Bayan saukewa, buɗe shi a cikin burauzarka: idan ka ga rubutunka, komai yana wurin.

Me ya kamata a rufe da kuma me bai kamata a rufe a cikin robots.txt ba?

Babban ƙa'ida: rufe duk wani abu da ba ya ƙara wa bincike muhimmanci kuma yana ɓata lokacin robot ɗin. Kada a rufe duk wani abu da yake da mahimmanci ga robot ɗin ya fahimci shafin.

Yawanci a rufe:

• kwamitin gudanarwa da asusun sirri: /admin/, /wp-admin/, /account/;
• keken siyayya da wurin biya: /cart/, /checkout/;
• binciken ciki: /search/, /?s=;
• manyan fayiloli na sabis da na ɗan lokaci: /tmp/, /cgi-bin/;
• Haɗuwa mara iyaka na matattara da zaɓuɓɓukan rarrabuwa - mai kyau, kawai idan akwai shafukan sauka daban-daban don matattara da ake buƙata.

Kada a taɓa rufewa:

• CSS, JavaScript, da hotuna—ba tare da su ba, Google ba zai ga shafin kamar yadda ɗan adam zai gani ba;
• kasida, shafin yanar gizo, katunan samfura da shafukan saukarwa;
• shafukan da kake son cirewa daga cikin fihirisa - don wannan kana buƙatar noindex, kuma robot ɗin dole ne ya iya karanta shi.

Ga fayil ɗin da zaku samu don shagon WordPress na yau da kullun tare da WooCommerce:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /?s=

Sitemap: https://example.com/sitemap.xml

Dangane da tutoci. Google ba ya la'akari da jinkirin crawl-delay —yana kunna shi ne kawai idan sabar ku tana fama da wasu bots. Toshe masu rarrafe na AI shawara ce ta kasuwanci: idan kuna son ChatGPT, Gemini, da Perplexity su san game da ku kuma su ba ku shawarar, kada ku toshe damar shiga su.

Wani abu mai amfani kuma shine sigar gwaji ta shafin. Don kwafin tsari, yi amfani da saitin "Hana Duk", amma tabbatar da maye gurbin fayil ɗin bayan an fara aiki: wanda aka manta da shi Disallow: / shine abin da galibi ke rage zirga-zirgar zuwa sabbin shafuka.

Kwatanta: Manhajar PromoPilot da Sauran Hanyoyin Ƙirƙirar Robots.txt

Kammalawa: Janareta ita ce hanya mafi sauri don samun ingantaccen fayil ba tare da sanin tsarin rubutu ba. Ƙarin plugin da lambar aikin suna da amfani lokacin da fayil ɗin ke buƙatar canzawa tare da gidan yanar gizon.

Janareta ɗaya ce daga cikin kayan aikin SEO 87 kyauta na PromoPilot. Bayan saukar da fayil ɗin, duba shi da robots.txt analyzer, kuma don layin taswirar shafin, yi amfani da janareta sitemap.xml - duka an rufe su a cikin labaran da ke kusa da wannan sashe.

Iyakokin Janareta: Abin da Ba Zai Yi Maka Ba

Janareta bai san tsarin shafin yanar gizonku ba. Saitin WordPress zai rufe hanyoyin sabis na yau da kullun, amma ba zai yi tsammanin matatun shagonku suna rayuwa a cikin /filter/ da kuma binciken ciki a cikin /find/ ba. Za ku buƙaci shigar da waɗannan hanyoyin da kanku.

Ba ya loda fayil ɗin zuwa sabar ko kuma tabbatar da cewa fayil ɗin yana samuwa a /robots.txt tare da lambar matsayi 200. Bayan lodawa, buɗe adireshin a cikin burauza ko gudanar da shi ta hanyar mai bincike.

Kuma baya maye gurbin noindex: idan manufar ita ce a cire shafuka daga sakamakon bincike, robots.txt bai dace ba. A ƙarshe, fayil ɗin baya kare bayananka: yana da jama'a, kuma kowa zai iya karanta waɗanne sassa ka kulle. Sirri yana ɓoye a bayan kalmar sirri.

Yaushe kayan aiki kyauta bai isa ba?

Samar da fayil yana ɗaukar mintuna biyar. Shawarar abin da za a kulle a shafin yanar gizo mai dubban shafuka aiki ne mai yawa. Kuna buƙatar ƙwararre idan:

• Shagon yana da ɗaruruwan haɗakar matattara, kuma wasu daga cikinsu suna haifar da cunkoso;
• Shafin yana da harsuna da yawa, a kan ƙananan yankuna ko kuma tare da manyan fayiloli daban-daban don harsuna;
• Bayan gyara robots.txt, zirga-zirgar ababen hawa ta ragu, kuma ba a san wace doka ce za a ɗora wa laifi ba;
• Kana buƙatar daidaita robots.txt, meta tags, canonical, da sitemap zuwa tsarin guda ɗaya.

Me zan iya yi wa gidan yanar gizonku?

Sunana Valeriy Danyuk, ni ƙwararren mai haɓakawa ne kuma ƙwararre a fannin SEO, ina da ƙwarewa ta shekaru 12 da kuma abokan ciniki sama da 200. Fayil ɗin robots.txt mai kyau yana cikin kowane gidan yanar gizo da nake ginawa.

SEO da haɓakawa

• Binciken SEO na Turnkey: robots.txt, noindex, canonical, da kuma taswirar shafin yanar gizo a matsayin tsarin haɗin kai, ba jerin ƙa'idodi na bazata ba.
• Shafin yanar gizo na SEO wanda ke amfani da fasahar AI wanda ke kawo jagororin halitta.
• Shafin yanar gizo mai harsuna da yawa ba tare da mai fassara ba - inda robots.txt da taswirar shafin dole ne su yi la'akari da dukkan harsuna.

Shafukan yanar gizo da shagunan kan layi

• Shagon WooCommerce na kan layi: samfura sama da 40, €299 — tare da robots.txt don keken siyayya, matattara, da biyan kuɗi.
• Shafin yanar gizo a shafin yanar gizo 1001site, inda aka tsara taswirar shafin da kuma tsarin ƙididdiga ta atomatik. Ta yaya wannan ya bambanta da mai gini mai kuɗin biyan kuɗi? Nazarin gidan yanar gizo mai amfani da fasahar AI da Tilda.

CRM, lissafi da sarrafa kansa

• Tsarin lissafin kuɗi na wata-wata don shagon kan layi - wani aiki na kamfani daga Jamhuriyar Czech.
• QuestFlow wani mai gina SaaS ne Telegram tare da AI.

Karin bayani a cikin fayil ɗin. Ba ku da tabbas game da abin da za ku rufe? Aika min adireshin gidan yanar gizon ku a Telegram - Zan ba da shawarar ƙa'idodi don tsarin ku.

Sakamako

1. Manhajar robots.txt ta yanar gizo tana ƙirƙirar fayil ɗin da ya dace cikin mintuna biyar: an saita shi, hanyoyi, taswirar shafin yanar gizo, tutoci.
2. Rufe kwamitin gudanarwa, sake yin amfani da akwatin ajiya, manyan fayilolin bincike, da sabis; kar a rufe CSS, JS, kundin adireshi, ko shafin yanar gizo.
3. robots.txt ba ya cire shafuka daga cikin fihirisa - wannan shine abin da noindex yake nufi.
4. Tsarin tsari mai rikitarwa, matattara, da harsuna dalili ne na ɗaura saitin ga ƙwararre ko gina shafi a 1001site, inda aka tsara indexing daga cikin akwati.

Ƙirƙiri fayil ɗin robots.txt ɗinka yanzu ta amfani da injin samar da robots.txt kyauta, kuma idan kuna da wasu tambayoyi game da ƙa'idodi, rubuto min a Telegram.

Majiyoyi

• Shafin kayan aikin samar da wutar lantarki na robots.txt na PromoPilot — saituna, saituna, da tambayoyin shafi.
• Cibiyar Bincike Google - Yadda ake Ƙirƙiri da Aika Fayil ɗin Robots.txt.
• Takardun Next.js - fayil ɗin metadata na robots.