Agent Commands


Navigation:  »No topics above this level«

Agent Commands

Content Grabber web-scraping agents usually contain a list of commands that execute sequentially. Sometimes, commands will execute in parallel (simultaneously). Each agent command performs an action, such as loading a new page or capturing content.


Some commands can contain sub-commands. These commands are called container commands and they execute all their sub-commands one or more times. If a container command executes its sub-commands more than one time, then the command is also called a list command. A list command iterates through a list of data elements or web elements and executes all its sub-commands once for each element in the list.



The Agent Explorer shows all commands in an agent


The most important command in a agent is the Agent command itself. This is the first command that executes. Since it contains all other commands for the agent, it is called a container command. The Agent Command loads the Start URL, the point at which data extraction starts. The Agent Command also controls many other important aspects of the agent, such as data export.


Some agent commands have a corresponding web selection that chooses an element from the current web page. One or more of these selections are put to use when the command executes. A Navigate Link command, for example, selects a link on the current web page, which will open a new web page.


See The Content Grabber Editor topic to learn how to use the Agent Explorer to add and configure agent commands.


Command Types

We classify agent commands into four types, according to function:

Capture Commands

List Commands

Action Commands

Container Commands


Capture commands do nothing but capture web content. Container, List, and Action commands may function as one or more types simultaneously. There are some special commands that don't fall into any of these 4 categories, such as the Execute Script command which simply runs a .NET script.


Agent Command Properties

All commands have a set of properties that govern the function of the command. The following is a set of basic properties common to all agent commands.





The name of the command.


The internal ID of the command.


The default value for this property is False. The system will ignore the command if this property set to True.


A command with this property set to False (default) will not save any data output. This includes all sub-commands that are part of a container command.

Recursion Disabled

A command with this property set to False (default) will only execute one time in case there is a recursive loop.

Notify On Critical Error

The default value for this property is True, which means that a notification email is sent at the end of an agent run if this command encounters a critical error. Critical errors include page load errors and mandatory web selections that are missing.

Debug Disabled

The system will ignore this command during debugging if this property set to False (default).

Debug Break Point

The default value for this property is False. During debugging, the system will break at this command if this value is True.



Web Selection Properties

All commands that perform web selection will have these properties:




Contains one or more selection paths, each of which points to a specific web element on a web page. These paths may point to elements in various locations on the web page.


A path corresponds to a web page selection. The value of the Selection Mission Option property (below) determines how Content Grabber will handle any missing selection.

Selection Missing Option

The value of this property specifies the action of Content Grabber if this selection does not exist in the current page. You can set this property to one of the following:


Default Option - The software will decide the most appropriate action depending on the type of command.

Optional Selection - The selection is optional, so the agent will continue executing any sub-commands.

Ignore Command when Selection Missing - The agent will skip this command and also skip all sub-commands.

Log Error and Ignore Command when Selection Missing - The agent will skip this command and all sub-commands, and will also log an error.


Selection Path Properties

A web selection can consist of multiple selection paths, each of which selects web elements in different locations on a web page. Each selection path will have these properties:





The selection XPath. See the topic XPath and Selection Techniques for more information about XPaths.

Wait XPath

Specify an XPath here that you want the agent to finish loading before proceeding with the other commands. This option is useful when selecting content that loads asynchronously - often after the main elements of the page have finished loading.

Wait Timeout Seconds

Specifies how long to wait for the Wait XPath web selection element to load.

Wait For Refresh Only

If you set this property to True, the agent only waits for the wait XPath to refresh. Does not verify that the selection Path exists. This setting is useful if the Path is optional.