{"id":17104,"date":"2025-10-25T22:43:31","date_gmt":"2025-10-25T15:43:31","guid":{"rendered":"https:\/\/www.cipher.co.th\/blogs\/robots-txt\/"},"modified":"2026-02-13T10:25:20","modified_gmt":"2026-02-13T03:25:20","slug":"robots-txt","status":"publish","type":"post","link":"https:\/\/www.cipher.co.th\/en\/blogs\/robots-txt\/","title":{"rendered":"\u200b\u200bRobots.txt: What Is It and Why Is It Important for Website SEO?<br>"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"17104\" class=\"elementor elementor-17104 elementor-15378\" data-elementor-post-type=\"post\">\n\t\t\t\t<div class=\"elementor-element elementor-element-53f1852 e-flex e-con-boxed e-con e-parent\" data-id=\"53f1852\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-ca74e06 elementor-widget elementor-widget-text-editor\" data-id=\"ca74e06\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>If you&#8217;re developing a website and want to improve SEO performance, one component you shouldn&#8217;t overlook is the Robots.txt file. This small file plays a crucial role in determining which parts of your website search engine bots can access. <a href=\"https:\/\/www.cipher.co.th\/en\"><strong>CIPHER <\/strong><\/a>will guide you through understanding and properly configuring this file to help your website be indexed more efficiently.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ae749e9 elementor-toc--minimized-on-tablet elementor-widget elementor-widget-table-of-contents\" data-id=\"ae749e9\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;exclude_headings_by_selector&quot;:[],&quot;marker_view&quot;:&quot;bullets&quot;,&quot;icon&quot;:{&quot;value&quot;:&quot;&quot;,&quot;library&quot;:&quot;&quot;},&quot;no_headings_message&quot;:&quot;No headings were found on this page.&quot;,&quot;headings_by_tags&quot;:[&quot;h2&quot;,&quot;h3&quot;,&quot;h4&quot;,&quot;h5&quot;,&quot;h6&quot;],&quot;minimize_box&quot;:&quot;yes&quot;,&quot;minimized_on&quot;:&quot;tablet&quot;,&quot;hierarchical_view&quot;:&quot;yes&quot;,&quot;min_height&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;min_height_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;min_height_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]}}\" data-widget_type=\"table-of-contents.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div class=\"elementor-toc__header\">\n\t\t\t\t\t\t<h2 class=\"elementor-toc__header-title\">\n\t\t\t\tTable of Contents\t\t\t<\/h2>\n\t\t\t\t\t\t\t\t\t\t<div class=\"elementor-toc__toggle-button elementor-toc__toggle-button--expand\" role=\"button\" tabindex=\"0\" aria-controls=\"elementor-toc__ae749e9\" aria-expanded=\"true\" aria-label=\"Open table of contents\"><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-chevron-down\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M207.029 381.476L12.686 187.132c-9.373-9.373-9.373-24.569 0-33.941l22.667-22.667c9.357-9.357 24.522-9.375 33.901-.04L224 284.505l154.745-154.021c9.379-9.335 24.544-9.317 33.901.04l22.667 22.667c9.373 9.373 9.373 24.569 0 33.941L240.971 381.476c-9.373 9.372-24.569 9.372-33.942 0z\"><\/path><\/svg><\/div>\n\t\t\t\t<div class=\"elementor-toc__toggle-button elementor-toc__toggle-button--collapse\" role=\"button\" tabindex=\"0\" aria-controls=\"elementor-toc__ae749e9\" aria-expanded=\"true\" aria-label=\"Close table of contents\"><svg aria-hidden=\"true\" class=\"e-font-icon-svg e-fas-chevron-up\" viewBox=\"0 0 448 512\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M240.971 130.524l194.343 194.343c9.373 9.373 9.373 24.569 0 33.941l-22.667 22.667c-9.357 9.357-24.522 9.375-33.901.04L224 227.495 69.255 381.516c-9.379 9.335-24.544 9.317-33.901-.04l-22.667-22.667c-9.373-9.373-9.373-24.569 0-33.941L207.03 130.525c9.372-9.373 24.568-9.373 33.941-.001z\"><\/path><\/svg><\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<div id=\"elementor-toc__ae749e9\" class=\"elementor-toc__body\">\n\t\t\t<div class=\"elementor-toc__spinner-container\">\n\t\t\t\t<svg class=\"elementor-toc__spinner eicon-animation-spin e-font-icon-svg e-eicon-loading\" aria-hidden=\"true\" viewBox=\"0 0 1000 1000\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M500 975V858C696 858 858 696 858 500S696 142 500 142 142 304 142 500H25C25 237 238 25 500 25S975 237 975 500 763 975 500 975Z\"><\/path><\/svg>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-8845142 e-flex e-con-boxed e-con e-parent\" data-id=\"8845142\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-1ffc6f8 elementor-widget elementor-widget-heading\" data-id=\"1ffc6f8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">What is Robots.txt?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5762212 elementor-widget elementor-widget-image\" data-id=\"5762212\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"611\" src=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-3-1024x611.webp\" class=\"attachment-large size-large wp-image-15384\" alt=\"Robots.txt\" srcset=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-3-1024x611.webp 1024w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-3-300x179.webp 300w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-3-768x459.webp 768w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-3-500x299.webp 500w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-3-700x418.webp 700w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-3.webp 1340w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3938cf5 elementor-widget elementor-widget-text-editor\" data-id=\"3938cf5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Robots.txt\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Robots.txt <\/strong><\/a>is a plain text file located in the root directory of your website. It functions as a &#8220;navigation guide&#8221; for web crawlers or search engine bots (<a href=\"https:\/\/www.cipher.co.th\/en\/blogs\/search-engine\/\"><strong>Search Engine<\/strong><\/a>). When bots visit your website, the first thing they do is look for the Robots.txt file to check which parts of your website they are allowed or not allowed to access.<\/p><p>Think of Robots.txt as a security guard that tells bots &#8220;You may enter this section&#8221; or &#8220;Sorry, this area is restricted.&#8221; This control is extremely important for <a href=\"https:\/\/www.cipher.co.th\/en\/blogs\/seo-guide\/\"><strong>website SEO.<\/strong><\/a><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-45c514a elementor-widget elementor-widget-image\" data-id=\"45c514a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.cipher.co.th\/en\/e-book-seo-and-search-intent-optimizing-for-2025\/\">\n\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1024\" height=\"341\" src=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/01\/cta-e-book-15-1024x341.webp\" class=\"elementor-animation-shrink attachment-large size-large wp-image-8690\" alt=\"\u0e14\u0e32\u0e27\u0e19\u0e4c\u0e42\u0e2b\u0e25\u0e14 E-Book\" srcset=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/01\/cta-e-book-15-1024x341.webp 1024w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/01\/cta-e-book-15-300x100.webp 300w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/01\/cta-e-book-15-768x256.webp 768w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/01\/cta-e-book-15-500x167.webp 500w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/01\/cta-e-book-15-700x233.webp 700w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/01\/cta-e-book-15.webp 1200w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b582e67 elementor-widget elementor-widget-heading\" data-id=\"b582e67\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Why is Robots.txt Important for Website SEO?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ba00037 elementor-widget elementor-widget-text-editor\" data-id=\"ba00037\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Many people wonder how such a small file can affect SEO. In reality, Robots.txt plays a very significant role in <a href=\"https:\/\/www.cipher.co.th\/en\/blogs\/seo-optimization\/\"><strong>optimizing SEO performance<\/strong><\/a> as follows:<\/p><ol><li><strong>Helps Google better understand your website structure<\/strong> &#8211; When you guide web crawlers on which sections to view and which to skip, search engines better understand which content you want users to see, helping Google prioritize crawling and indexing your content more efficiently.<\/li><li>Saves crawl budget &#8211; Search engines have limitations on crawling each website. Having a properly configured Robots.txt helps ensure this quota is used on truly important pages, not wasting time on pages that don&#8217;t need to appear in search results.<\/li><li>Prevents duplicate content &#8211; You can use Robots.txt to tell bots not to crawl pages with duplicate content, helping prevent Duplicate Content issues that negatively impact SEO and may lower your website&#8217;s ranking.<\/li><li><strong>Helps manage server resources<\/strong> &#8211; Limiting bot access can reduce server load, especially for large websites, making your website load faster and providing a better user experience, which is an important factor in Google&#8217;s ranking.<\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a6886cb elementor-widget elementor-widget-heading\" data-id=\"a6886cb\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Benefits of Having a Robots.txt File<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6300500 elementor-widget elementor-widget-image\" data-id=\"6300500\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"1024\" height=\"611\" src=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-4-1024x611.webp\" class=\"attachment-large size-large wp-image-15386\" alt=\"Robots.txt\" srcset=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-4-1024x611.webp 1024w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-4-300x179.webp 300w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-4-768x459.webp 768w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-4-500x299.webp 500w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-4-700x418.webp 700w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2025\/10\/Robots.txt-4.webp 1340w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8d0750d elementor-widget elementor-widget-text-editor\" data-id=\"8d0750d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>A properly configured Robots.txt file offers many benefits to your website, including:<br><\/p><ul><li>Improves web crawler efficiency &#8211; Bots will spend time and resources only on crawling important pages, allowing your important pages to be discovered and indexed faster. Your new pages will appear in search results more quickly.<\/li><li>Helps protect private information &#8211; You can instruct bots not to access pages with sensitive information, such as registration pages, payment pages, or pages with users&#8217; personal information, keeping important data from appearing in Google search results.<\/li><li>Helps prevent SEO problems &#8211; Such as preventing search engines from displaying test pages or draft pages in search results, which might confuse users or give them a poor experience from your website.<\/li><li>Assists in website improvements &#8211; You can prevent bots from accessing pages under construction, allowing you to develop your website without affecting pages already in search results. Users won&#8217;t encounter incomplete pages.<\/li><\/ul><p>For WordPress websites, creating and properly customizing WordPress Robots.txt is even more important because this platform often has many pages and folders that you may not want to appear in search results.<br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8d2646a elementor-widget elementor-widget-heading\" data-id=\"8d2646a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Cautions When Creating Robots.txt<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ddddc20 elementor-widget elementor-widget-text-editor\" data-id=\"ddddc20\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Although Robots.txt is very useful, incorrect usage can cause serious problems with your SEO. Here are things you should be cautious about:<\/p><ol><li><strong>Don&#8217;t block all web crawlers<\/strong> &#8211; Using the command User-agent: * Disallow: \/ will prevent all bots from accessing your website, meaning your website won&#8217;t appear in search results at all! Many people make this mistake unknowingly and wonder why their website disappeared from Google. <\/li><li><strong>Not a complete security measure <\/strong>&#8211; Robots.txt is merely a recommendation, not enforcement. Malicious bots can ignore these commands. Therefore, don&#8217;t use it to hide important information that needs security, as anyone can view your Robots.txt.<\/li><li>Should not be used instead of hiding content &#8211; If you want to truly protect data, you should use authentication or other methods instead, such as using .htaccess or other more appropriate security systems.<\/li><li><strong>In WordPress, incorrectly editing Robots.txt can be harmful<\/strong> &#8211; WordPress Robots.txt users should understand the impact of modifications before making changes. Always back up data first and test that the changes don&#8217;t affect website functionality.<\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bcc848f elementor-widget elementor-widget-heading\" data-id=\"bcc848f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Scripts and Commands in Robots.txt Files You Should Know<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8998ca5 elementor-widget elementor-widget-text-editor\" data-id=\"8998ca5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tIn Robots.txt files, there are important commands you should understand for effective configuration. Each command has a specific function in controlling web crawler behavior.<br>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-df3d13f elementor-widget elementor-widget-heading\" data-id=\"df3d13f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">User-agent<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ba8c38f elementor-widget elementor-widget-text-editor\" data-id=\"ba8c38f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>User-agent is the first command that must be specified in a robots.txt file, which defines which search engine bots the rules will apply to.<br><\/p><ul><li>If you want rules to apply to all search engines, use *<\/li><li>You can specify particular search engines like Googlebot, Bingbot, or Yahoobot<\/li><\/ul><p><strong>Examples<br><\/strong><\/p><p>Example 1: Apply rules only to Google<br>User-agent: Googlebot<br>Disallow: \/<br><\/p><p>Example 2: Apply rules to multiple search engines<br>User-agent: Googlebot<br>User-agent: AdsBot-Google<br>Disallow: \/<br><\/p><p>Example 3: Apply rules to all search engines<br>User-agent: *<br>Allow: \/<br><\/p><p>Defining different User-agents allows you to control access for each bot separately. For example, you might allow Google to access all sections but restrict access for other bots.<br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-70e6ac1 elementor-widget elementor-widget-heading\" data-id=\"70e6ac1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Disallow<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-71271b8 elementor-widget elementor-widget-text-editor\" data-id=\"71271b8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The Disallow command tells which parts of the website crawlers cannot access. You can specify folders or specific files. The more specific you can be, the better control you have.<br><\/p><p><strong>Examples<br><\/strong><\/p><p>Disallow: \/admin\/ # Prohibit access to all admin folders<br>Disallow: \/wp-admin\/ # For WordPress, prohibit access to admin pages<br>Disallow: \/private-file.html # Prohibit access to specific files<br><\/p><p>The Disallow command is very important for protecting areas you don&#8217;t want to appear in search results, such as admin pages, which often contain important information that shouldn&#8217;t be disclosed.<br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cb57d1f elementor-widget elementor-widget-heading\" data-id=\"cb57d1f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Allow<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1747fb3 elementor-widget elementor-widget-text-editor\" data-id=\"1747fb3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The Allow command permits search engines to access specified webpages or folders, even if there&#8217;s a Disallow command previously. It creates exceptions to Disallow commands. The writing format is the same as Disallow.<br><\/p><p><strong>Examples<\/strong><\/p><p>Example: Prohibit access to everything but allow only the public folder<br>User-agent: *<br>Disallow: \/<br>Allow: \/public\/<br><\/p><p>Example: Allow the entire website<br>User-agent: *<br>Allow: \/<br><\/p><p>This specificity helps you control access in detail without having to open or close everything.<br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-17c1f5f elementor-widget elementor-widget-heading\" data-id=\"17c1f5f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Sitemap<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-12d0c7f elementor-widget elementor-widget-text-editor\" data-id=\"12d0c7f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The Sitemap command helps web crawlers know where your sitemap is located. This is a file that lists all pages you want to appear in search results, making it easier for search engines to find and index your webpages.<br><\/p><p><strong>Examples<\/strong><\/p><p>Sitemap: https:\/\/www.yourwebsite.com\/sitemap.xml<br><\/p><p>Specifying a Sitemap in your Robots.txt file helps search engines discover your new pages faster and better understand your website structure, which is beneficial for SEO.<br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-842ed02 elementor-widget elementor-widget-heading\" data-id=\"842ed02\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How to Create a Robots.txt File<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-19ab723 elementor-widget elementor-widget-text-editor\" data-id=\"19ab723\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tThere are 2 main methods to create a Robots.txt file:<br>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dc4ea80 elementor-widget elementor-widget-heading\" data-id=\"dc4ea80\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">1. Create a Robots.txt File Manually and Upload to Your Website<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4ebc76a elementor-widget elementor-widget-text-editor\" data-id=\"4ebc76a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Write the Robots.txt file in Notepad (Windows) or TextEdit (Mac)<\/p><ol><li>Open a Text Editor like Notepad, TextEdit, or VS Code<\/li><li>Write the desired commands<\/li><li>Save the file as &#8220;robots.txt&#8221;<\/li><li>Upload the file to the root directory of your website via FTP or File Manager in your hosting<\/li><li>Check that the file is accessible at https:\/\/www.yourdomain.com\/robots.txt<\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bf21e28 elementor-widget elementor-widget-heading\" data-id=\"bf21e28\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">2. Create a Robots.txt File Using Plugins<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-24502c5 elementor-widget elementor-widget-text-editor\" data-id=\"24502c5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>For WordPress users, creating WordPress Robots.txt is easy with plugins<\/p><ol><li>Yoast SEO &#8211; This popular plugin has a Robots.txt management feature in the &#8220;SEO&#8221; &gt; &#8220;&gt;Tools&#8221; &gt;&gt; &#8220;Edit Files&#8221; menu<\/li><li>All in One SEO &#8211; Go to &#8220;All in One SEO&#8221; &gt; &gt;&#8221;Tools&#8221; &gt; &gt;&#8221;Robots.txt Editor&#8221;<\/li><li>Rank Math &#8211; Go to &#8220;Rank Math&#8221; &gt;&gt; &#8220;General Settings&#8221; &gt; &gt;&#8221;Edit robots.txt&#8221;<\/li><\/ol><p>This method is suitable for those not comfortable with directly editing files and helps prevent potential errors. Using plugins for managing WordPress Robots.txt also offers additional features that assist in SEO.<br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e79af16 elementor-widget elementor-widget-heading\" data-id=\"e79af16\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Elevate Your Website's Performance with CIPHER!<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-59074d9 elementor-widget elementor-widget-text-editor\" data-id=\"59074d9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>At CIPHER, we have a team of SEO experts ready to help optimize your Robots.txt file and other SEO components of your website to work at maximum efficiency.<br><\/p><p><strong>Our Services:<\/strong><\/p><ul><li><a href=\"https:\/\/www.cipher.co.th\/en\/services\/web-design-development\/\"><strong>Website creation with SEO-friendly structure <\/strong><\/a>from the beginning<\/li><li><a href=\"https:\/\/www.cipher.co.th\/en\/services\/seo\/\"><strong>Comprehensive SEO<\/strong><\/a> services for both On-page and Off-page<\/li><li>Maintenance and optimization of WordPress websites, including appropriate WordPress Robots.txt configuration<\/li><li>Analysis and resolution of SEO issues that may arise from incorrect Robots.txt settings<\/li><\/ul><p>SEO is an ongoing process, and Robots.txt is just one part of the overall picture. For the best results, working with experts like CIPHER will ensure that all components of your website are properly optimized.<br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5d8bede elementor-widget elementor-widget-heading\" data-id=\"5d8bede\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e8a8748 elementor-widget elementor-widget-text-editor\" data-id=\"e8a8748\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tRobots.txt may be a small file, but it has a significant impact on website SEO. Proper configuration helps search engines better understand your website structure, directly affecting your ranking in search results. Contact CIPHER to take your SEO to the next level!<br>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-722e47e elementor-widget elementor-widget-image\" data-id=\"722e47e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.cipher.co.th\/en\/services\/web-design-development\/\">\n\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"341\" src=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2024\/12\/bn-Web-Design-Development-1024x341.webp\" class=\"elementor-animation-shrink attachment-large size-large wp-image-8906\" alt=\"\u0e1a\u0e23\u0e34\u0e01\u0e32\u0e23\" srcset=\"https:\/\/www.cipher.co.th\/wp-content\/uploads\/2024\/12\/bn-Web-Design-Development-1024x341.webp 1024w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2024\/12\/bn-Web-Design-Development-300x100.webp 300w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2024\/12\/bn-Web-Design-Development-768x256.webp 768w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2024\/12\/bn-Web-Design-Development-500x167.webp 500w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2024\/12\/bn-Web-Design-Development-700x233.webp 700w, https:\/\/www.cipher.co.th\/wp-content\/uploads\/2024\/12\/bn-Web-Design-Development.webp 1200w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-6d57846 e-flex e-con-boxed e-con e-parent\" data-id=\"6d57846\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-11d5e19 elementor-widget elementor-widget-heading\" data-id=\"11d5e19\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Frequently Asked Questions<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a0919db elementor-widget elementor-widget-heading\" data-id=\"a0919db\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">What is Robots.txt?<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fe35f49 elementor-widget elementor-widget-text-editor\" data-id=\"fe35f49\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tRobots.txt is a file on websites that tells search engines which parts of the website they are allowed or not allowed to access and index.<br>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-43511df elementor-widget elementor-widget-heading\" data-id=\"43511df\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Why use a Robots.txt file?<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ed68a9d elementor-widget elementor-widget-text-editor\" data-id=\"ed68a9d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tTo control web crawlers&#8217; access, prevent server overload, and prevent indexing of content you don&#8217;t want to appear in search results.<br>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d6709ca elementor-widget elementor-widget-heading\" data-id=\"d6709ca\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">How do I create a Robots.txt file?<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ddf994c elementor-widget elementor-widget-text-editor\" data-id=\"ddf994c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tCreate a plain text file named &#8220;robots.txt&#8221; and place it in the root directory of your website, using specific syntax such as User-agent and Disallow.<br>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8f640fb elementor-widget elementor-widget-heading\" data-id=\"8f640fb\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Does Robots.txt affect website security?<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0ec21c1 elementor-widget elementor-widget-text-editor\" data-id=\"0ec21c1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tNo, because robots.txt is merely a recommendation for well-behaved bots, but cannot prevent access to information from malicious actors, so it should not be used as the sole security mechanism.<br>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>If you&#8217;re developing a website and want to improve SEO performance, one component you shouldn&#8217;t overlook is the Robots.txt file. [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":15380,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_angie_page":false,"content-type":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"page_builder":"","footnotes":""},"categories":[162,159,153],"tags":[],"class_list":["post-17104","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo-en","category-seo-tips-en","category-web-design-en"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/posts\/17104","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/comments?post=17104"}],"version-history":[{"count":0,"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/posts\/17104\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/media\/15380"}],"wp:attachment":[{"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/media?parent=17104"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/categories?post=17104"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cipher.co.th\/en\/wp-json\/wp\/v2\/tags?post=17104"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}