Definition Index

The search engines found on the Internet are often confused with web catalogs.

A conventional search engine only accepts websites that have been entered manually or sites that have been entered by a search engine robot.

A web catalog, on the other hand, can only be filled by a manual entry. This entry is then checked or indexed by an administration team and then activated.

It is also possible to prohibit the recording of robotic texts. The so-called robots.txt or the simple meta tags of the relevant page are adapted.

Indexing

There are various specification options for the Robots metatag.

1. The robot includes displayed pages and their subsequent pages in the search engine:

<meta name=“robots“ content=“index“>
<meta name=“robots“ content=“follow“>

1. No linked pages are now tracked or indexed from this page. This also applies to the robots:

<meta name=“robots“ content=“noindex“>
<meta name=“robots“ content=“nofollow“>

1. In the following example, the respective page cannot be included in the search engine by the robot. However, it is possible to include all pages that are linked to.

<meta name=“robots“ content=“noindex“>
<meta name=“robots“ content=“follow“>