Logo
CAR REVIEW
Windows Vista
Windows 7
Windows Azure
Windows Server
Windows Phone
PREGNANCY
 
 
Windows Server

Microsoft Content Management Server : Influencing Search Engines with the ROBOTS META Tag

3/1/2012 6:12:35 PM
Typically, the bulk of the index contains content from pages generated by channel rendering scripts and postings. They are after all, the parts of the site that are visible to the end users. To tell search engines which pages to index and which pages to ignore and whether to follow further links from the current page you can add a special META tag to your page named ROBOTS. Most (but not all) search engines honor this META tag and evaluate its content property to see if the page should be added to the index and if links from this page should be followed.

To allow MCMS authors to benefit from this feature, channels and postings have two additional properties:

  • IsRobotIndexable

  • IsRobotFollowable

These two properties can be used to generate the ROBOTS META tag. As already mentioned, be aware that some search engines choose to ignore the ROBOTS META tag.

The table below summarizes the behavior of the different property settings:

IsRobotFollowableIsRobotIndexableMETA tag to renderRemarks
FalseFalse<meta name=“ROBOTS” content=“NOFOLLOW, NOINDEX”>Search engines that recognize the robot META tag will not index the page nor follow its hyperlinks.
FalseTrue<meta name=“ROBOTS” content=“NOFOLLOW, INDEX”>Search engines that recognize the robot META tag will index the page but not follow its hyperlinks.
TrueFalse<meta name=“ROBOTS” content=“FOLLOW, NOINDEX”>Search engines that recognize the robot META tag will not index the page but will follow its hyperlinks.
TrueTrue<meta name=“ROBOTS” content=“FOLLOW, INDEX”>Search engines that recognize the robot META tag will index the page and follow its hyperlinks.

For more information on the ROBOTS META tag, check out this resource:

http://www.robotstxt.org/wc/exclusion.html#meta

The RobotMetaTag Control and Channel Rendering Scripts

MCMS ships with a control named RobotMetaTag, which is automatically added to all template files created from the Visual Studio .NET “MCMS Template File” template. The RobotMetaTag control is used to generate the ROBOTS META tag based on the IsRobotFollowable and IsRobotIndexable properties of the page.

Unfortunately this control does not work on channel rendering scripts. If you add the control to a channel rendering script, despite setting the IsRobotFollowable and IsRobotIndexable properties of the channel, the ROBOTS META tag does not get generated. To get around this problem let’s implement an enhanced version of this control that works with both channel rendering scripts and templates.

In the WoodgroveNet sample site that comes with MCMS, the RobotMetaTag control is wrapped in a user control called HeadTags.ascx.


First, open the Tropical Green solution. Add a new class file to the TropicalGreenControlLib project with the name EnhancedRobotMetaTag.cs.

The control requires the use of methods from the namespaces highlighted below, so add them to the class file.

using System;

using System.ComponentModel;
						using System.Web.UI;
						using Microsoft.ContentManagement.Common;
						using Microsoft.ContentManagement.Publishing;
						using Microsoft.ContentManagement.WebControls;

namespace TropicalGreenControlLib
{
  . . . code continues . . .
}

Also, as we are creating a custom server control, the class will inherit from the RobotMetaTag class (in the Microsoft.ContentManagement.WebControls namespace).

namespace TropicalGreenControlLib
{
  /// <summary>
  /// Summary description for EnhancedRobotMetaTag.
  /// </summary>
  public class EnhancedRobotMetaTag : RobotMetaTag
  {
    public EnhancedRobotMetaTag()
    {
    }
  }
}

Above the EnhancedRobotMetaTag constructor, declare four constants, Index, Follow, noIndex and noFollow, which store the strings we will use later for generating the ROBOTS META tag. We will also remove the constructor, EnhancedRobotMetaTag(), as it is not required.

public class EnhancedRobotMetaTag : RobotMetaTag
{
  private const string Index = "INDEX";
						private const string Follow = "FOLLOW";
						private const string noIndex = "NOINDEX";
						private const string noFollow = "NOFOLLOW";
}

In addition to rendering the ROBOTS tag, the original RobotMetaTag control has a different feature: it provides an option to render a <base> tag, which ensures that relative links added to the template files work correctly. Here’s an example of a <base> tag:

<base
href="http://localhost/TropicalGreen/Templates/GardenPublic.aspx?NRMODE=Publis
hed&NRORIGINALURL=%2fTropicalGreen%2fGardens%2fMembers%2ehtm&NRNODEGUID=%7bB95
DD11B-2954-4F84-AB60-AC0A3D39FC49%7d&NRCACHEHINT=NoModifyGuest">

					  

When using the Word Authoring Connector and documents that use internal bookmarks, the <base> tag can cause an entire page refresh.


As a single template file can be used by multiple postings, the URL varies from posting to posting. Links relative to the template file will not work in this situation as the browser does not know that postings and channels do not exist as complete entities on the file system. The base tag tells the browser what should be treated as the current location and allows the browser to calculate the final link based on the base tag and the information in the relative link. For example, should the template file contain a relative link, say, <a href="MembersOnly/SomePage.htm">A Relative Link</a>, the browser will treat the URL as if "MembersOnly/SomePage.htm" was pre-pended with the URL stored in the href attribute of the <base> tag.

As we are going to implement a control that extends the RobotMetaTag control, we need to ensure that this feature is also included in our new control.

We will write the ROBOTS META tag and <base> tag by overriding the Render() method of the control. Notice we look at the properties of a ChannelItem object. This ensures that the code works for both postings and channel rendering scripts.

protected override void Render( HtmlTextWriter writer )
{
  ChannelItem currentItem = CmsHttpContext.Current.ChannelItem;

  if( currentItem != null )
  {
    string followKey = (currentItem.IsRobotFollowable)?Follow:noFollow;
    string indexKey = (currentItem.IsRobotIndexable)?Index:noIndex;

    // write the robot META tag
    writer.Write("<meta name=\"ROBOTS\" "
               + "content=\""+followKey+","+indexKey+"\">\n");

    if( this.RenderBaseHref )
    {
      // write the Base tag
      writer.Write("<base href=\""+Page.Request.Url.AbsoluteUri+"\">\n");
    }
  }
}

Save and build the solution. Add the control to the toolbar as you would other custom controls and it’s ready to be dropped into channel rendering scripts or template files between the <head> tags.

Other -----------------
- Windows Server 2003 : Recovering from System Failure (part 2) - Recovery Console
- Windows Server 2003 : Recovering from System Failure (part 1)
- Microsoft BizTalk 2010 : Consuming ASDK-based Adapters - The WCF-Custom adapter and SOAP actions
- Microsoft BizTalk 2010 : Consuming ASDK-based Adapters - Using the ASDK development tools
- Microsoft SQL Server 2008 R2 : SQL Server Management Studio - Administration Tools (part 2)
- Microsoft SQL Server 2008 R2 : SQL Server Management Studio - Administration Tools (part 1)
- Microsoft SQL Server 2008 R2 : SQL Server Management Studio - The Integrated Environment
- Managing Exchange Server 2010 Clients : Using Mail Profiles to Customize the Mail Environment
- Managing Exchange Server 2010 Clients : Managing the Exchange Server Configuration in Outlook
- System Center Configuration Manager 2007 : Configuration Manager and WMI (part 6) - WMI on Configuration Manager Servers
- System Center Configuration Manager 2007 : Configuration Manager and WMI (part 5) - The Configuration Manager Client WMI Namespace
- System Center Configuration Manager 2007 : Configuration Manager and WMI (part 4) - Hardware Inventory Through WMI
- System Center Configuration Manager 2007 : Configuration Manager and WMI (part 3) - Looking Inside the CIMV2 Namespace
- System Center Configuration Manager 2007 : Configuration Manager and WMI (part 2) - Managing WMI
- System Center Configuration Manager 2007 : Configuration Manager and WMI (part 1) - WMI Feature Set and Architecture
- System Center Configuration Manager 2007 : Active Directory Integration
- Deploying and Managing BizTalk Applications : Deploying a BizTalk Solution (part 2)
- Deploying and Managing BizTalk Applications : Deploying a BizTalk Solution (part 1) - Steps in Deploying a BizTalk Application
- Microsoft Dynamics CRM 4.0 Accelerators : Event Management Accelerator (part 3) - Publish the CRM Workflow & Testing the Portal
- Microsoft Dynamics CRM 4.0 Accelerators : Event Management Accelerator (part 2) - Install CRM Reports & CRM Customizations
 
 
Most view of day
- Accessing and Using Your Network : Accessing Network Resources
- Windows 7 Mobility Features : Using Windows 7 with a Netbook
- Windows Server 2012 Group Policies and Policy Management : Group Policy Policies Node
- Creating DVD Movies with Windows DVD Maker (part 3) - Understanding DVD Movie Options
- Windows Server 2003 on HP ProLiant Servers : Migration Case Studies (part 2) - Eastman Chemical Company
- Microsoft Exchange Server 2010 : Completing Transport Server Setup (part 5) - Getting Edge Subscription Details, Synchronizing Edge Subscriptions
- Microsoft Exchange Server 2010 : Getting Started with Email Archiving - Enabling Archiving (part 1) - Archive Quotas , Exchange 2010 Discovery Operation Considerations
- Reporting in Dynamics AX 2009 : Building Dynamics AX Reporting Services Reports Using Visual Studio
- Windows Server 2003 : Protecting Hosts with Windows Host Firewalls - Firewall Basics
- Windows Server 2008 R2 high-availability and recovery features : Installing and Administering Failover Clustering (part 4) - Verifying cluster configuration using the Cluster Validation Wizard
Top 10
- Microsoft Project 2010 : Linking Tasks (part 8) - Auditing Task Links,Using the Task Inspector
- Microsoft Project 2010 : Linking Tasks (part 7) - Creating Links by Using the Mouse,Working with Automatic Linking Options
- Microsoft Project 2010 : Linking Tasks (part 6) - Creating Links by Using the Entry Table
- Microsoft Project 2010 : Linking Tasks (part 5) - Creating Links by Using the Task Information Dialog Box
- Microsoft Project 2010 : Linking Tasks (part 4) - Entering Leads and Lags, Creating Links by Using the Menu or Toolbar
- Microsoft Project 2010 : Linking Tasks (part 3) - Using the Start-to-Start Relationship,Using the Finish-to-Finish Relationship
- Microsoft Project 2010 : Linking Tasks (part 2) - Using the Start-to-Start Relationship,Using the Finish-to-Finish Relationship
- Microsoft Project 2010 : Linking Tasks (part 1) - Defining Dependency Links
- Microsoft Project 2010 : Defining Task Logic - Manipulating Your Schedule
- Microsoft Lync Server 2013 : Director Troubleshooting (part 3) - Synthetic Transactions,Telnet
 
 
Windows XP
Windows Vista
Windows 7
Windows Azure
Windows Server
Windows Phone
2015 Camaro