Online marketing can be tiring! That’s where UpCity comes in. As a trusted advocate for small businesses, we’re here to navigate you through the piles of confusing and suspicious offers. UpCity will lead you in the right direction by providing easy insights and simple solutions for your business marketing plan. Each member receives an in-depth, personalized action plan to help optimize their site, and our team is here to help you along the way. That’s why we’ve decided to launch this exciting new weekly blog series!

Each week, we’re featuring an in-depth look at one of the recommended tasks built into the UpCity software for our clients. These strategies individually contribute to your online presence and increase your chances of bringing free traffic to your site. Subscribe to the UpCity Daily Tips Newsletter for daily recommendations on how to bring customers to your site.

Background

Confused Robot_robots.txt-SEOWe recently posted a blog all about how to find out if your site is listed on search engines. Making sure your website is crawled and ranked is important for any small business so that it can appear in search engine results. If your website is not able to be found by the search engines there are several indications why this could be.

If your site isn’t listed, it could be affected by one or more of these reasons:

  • There are not many other websites that link to the site.
  • The site launched or updated after the search engine’s last round of crawling the Internet.
  • The site became unavailable or an error occurred when the engine attempted its last crawl.
  • The content of the website is in violation of the search engine policies.
  • Last, the design is difficult for the search engine to effectively crawl and database the content. (more on this subject later).

Below are some useful action items to perform on your site in order to get your site listed more easily. Although manually submitting your site to Bing and Google is an approach to get listed, it’s not going to get you any closer if you check these essential on-site items on the backend.

1. Confirm that your robots.txt settings allow search engines to crawl and database your site.

Robots.txt Summary:

Robots.txt is a key SEO concept, but it can be quite confusing. Our SEO experts get asked about a robots.txt solution often so we want to provide a quick overview for you. “Robots.txt” is a simple text file found in all standard domain that tells search engines whether they can access and crawl parts of a website. Robots.txt is often placed in the root directory of your site and can permit access to the engines crawling the web.

Why is a robots.txt file so important? If this automated text file is set up incorrectly, it may unintentionally keep a website from getting indexed in search engines. A robots.txt file is often applied to stop a page from showing up on a search engine if you don’t want it to be displayed. In addition, the file applies a level of security to keep people from accessing items on your site that you want protected. The robots.txt can restrict access in this way. The website doesn’t necessarily need a robots.txt file if you want all pages to be visible and crawled.

Robots.txt Process Tips:

  • If you edit your site’s HTML directly, you will most likely find robots.txt in the default directory. Do not worry if it does not exist, as it is not a mandatory file.
  • A basic code indicating all webpages are able to be accessed is shown below. The code includes the keyword “User-agent,” which is is the crawler looking for the specific page on your site.

User-agent: *

Disallow:

  • The below code indicates that a robot should be restricted from a specific site:

User-agent: *

Disallow: /

  • Various types of robots.txt files exist, depending what specific items you want to block on your site. Be very cautious which rules you are utilizing to ensure you’re not removing items that should be crawled by the engine. As mentioned above, if you miswrite the file, you could be preventing your site from being indexed. Therefore, it’s important to carefully review your work.
  • If using a content management system (CMS) to manage your site, refer to your individual CMS documentation about how to manage robots.txt through the CMS. Note that not all CMS providers directly control the robots.txt file.
  • When thinking about restricting access to a page with a robots.txt file, users can address more information about this task when needed in the Google Webmaster Tools Help Center. You’re able to create a robots.txt file on your own or use a pre-made generator tool.
  • View the Google Webmasters Frequently Asked Questions about robots.txt files for more information on this subject.
  • View Bing’s Webmaster Help & How-To for details on how to complete a robots.txt file for Bing and Yahoo.

2. Confirm that your website’s meta tag settings allow search engines to crawl and database your website.

Meta Tags Summary:

This task revolves around instructing the search robots how to index the pages on your website. A meta noindex tag is used to stop a site being listed in a search engine despite if other websites are using it as a link. If the page contents are currently displayed in the search results, they will most likely be remove after the next crawl. If a meta noindex tag is accidentally inserted into your website, it will prevent the page from showing up in Google’s search results. Overall, metatags provide more complete control of your website.

Meta Tags Process Tips:

  • This is a granular, technical task that requires you to edit via HTML directly on your website.
  • Begin by searching for the following text in your homepage source code to remove this tag:

<meta name=”robots” content=”noindex”>

  • Use this tag if you don’t want Google to index the page on search results but other engines can:

<meta name=”googlebot” content=”noindex”>

  • If you are certain the page should not be included in search engines, then leave it as is. If you definitely want the page to be included on the engines, then delete it and save your changes.
  • If you need the page to be removed immediately, try manually submitting the URL to Google.
  • Users can address more information about this task when needed in the Google Webmaster Tools Help Center.

UpCity members, follow this pathway to complete this task in your UpCity account: Optimize Your Presence > Establish Your Presence > Website 101: The Basics.