Robots meta tags are special HTML meta tags placed in the <head> section of a webpage that give instructions to search engine crawlers (also called “robots” or “bots”) about how to handle that page.
They help control:
- Whether the page should be indexed (appear in search results).
- Whether links on the page should be followed.
- Whether cached copies, snippets, or previews should be shown.
Syntax
<meta name="robots" content="directive1, directive2">
Example:
<meta name="robots" content="noindex, nofollow">
This tells search engines not to index the page and not to follow any links on it.
Common Directives in Robots Meta Tags
| Directive | Meaning |
|---|---|
| index | Allow the page to be indexed (default). |
| noindex | Do not index the page in search results. |
| follow | Follow links on the page (default). |
| nofollow | Do not follow links on the page. |
| noarchive | Do not store a cached version of the page. |
| nosnippet | Do not show a text or video snippet in results. |
| max-snippet:[number] | Limit snippet length. |
| max-image-preview:[value] | Control image preview size (none, standard, large). |
| max-video-preview:[number] | Limit video preview length in seconds. |
Example Placement in HTML
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="robots" content="noindex, nofollow">
<title>Private Page</title>
</head>
<body>
<h1>Members Only</h1>
</body>
</html>
Types of Robots Meta Tags
Robots meta tags can be classified based on the type of crawler they target and the purpose they serve. They are mainly placed inside the <head> section of a webpage.
1. Global Robots Meta Tag
- Targets all search engine crawlers.
- Uses
name="robots". - Applies the same rule to Google, Bing, Yahoo, and others.
Example:
<meta name="robots" content="noindex, nofollow">
2. Specific Search Engine Robots Meta Tag
- Targets a particular search engine’s crawler by name.
- Useful if you want different rules for different crawlers.
- Common examples:
- Google →
googlebot - Bing →
bingbot - Yahoo →
slurp
Example:
- Google →
<meta name="googlebot" content="noindex">
<meta name="bingbot" content="index, follow">
3. Indexing Control Tags
- Control whether the page should be included in search results.
- index → allow indexing (default)
- noindex → prevent indexing
Example:
<meta name="robots" content="noindex">
4. Link Following Control Tags
- Control whether crawlers should follow the links on the page.
- follow → follow links (default)
- nofollow → do not follow links
Example:
<meta name="robots" content="index, nofollow">
5. Content Display Control Tags
- Manage how the page content is shown in search results.
- noarchive → prevent cached copy display
- nosnippet → prevent text/video snippets
- max-snippet:[number] → limit snippet length
- max-image-preview:[value] → limit image preview size (
none,standard,large) - max-video-preview:[number] → limit video preview length in seconds
Example:
<meta name="robots" content="noarchive, nosnippet">
6. Combined Robots Meta Tags
- Combine multiple directives for more precise control.
Example:
<meta name="robots" content="noindex, nofollow, noarchive">
Discover more from LearningKeeda
Subscribe to get the latest posts sent to your email.
