<< Click to Display Table of Contents >> Navigation:  Data > Using Data Input > Data Providers

You can assign a data provider to any Data List command and can load data from the following data sources:

 

 

 

 

Data List commands include the following:

 

 

A list command will iterate through all the data rows given by the data provider and execute each of the sub-commands once for each row.

 

Consuming Data

Consumer commands use the data that comes from the Data Provider. The consumer must be a sub-command of the command providing the data. Data consumers include the following types of commands:

 

Most data providers are also data consumers and, consequently, many commands will retrieve data automatically. For example, an agent command will normally provide at least one start URL to itself - as shown in the figure.

 

 

You can choose the Data Provider on the Common tab of the Configure Agent Command panel. If multiple data columns are available, then you can choose a specific Data Column. The Default data column is simply the first data column that the data provider presents.

 

A data provider will often provide multiple data columns for data sets that correspond to a web form. An example would be an agent that searches for air fares, which would submit a list of departure and destination cities to a web form. The data provider would then provide two columns: one for departure city and one for destination city.

 

Consuming Data in a Script

A script can consume data from a data provider as long as the script corresponds with a command that is a sub-command of the data provider. Most scripts provide easy access to input data from the script arguments. The following example retrieves the current data from the data column DepartureCity in the CityData data provider.

 

args.GetInputData("CityData").GetStringValue("DepartureCity");

 

See the Scripting article for more information.

 

Simple Data Provider

Choosing Simple for the Data Provider will cause the agent to extract data from a CSV data set that the agent designer enters at design time. The agent contains the CSV data internally, so that there is no dependency on external files.

 

 

The CSV data set may contain multiple rows, each having multiple comma-separated data values. The data follows the same formatting rules as standard CSV data. So, you should enclose a data value in double-quotes if it contains a comma or single quote, and use two double quotes to escape any double quotes in a data value.

 

The following example contains a header row, although a header row is not mandatory. If the data contains a header row, then you need to check the Input data includes header row box on the Data tab.

 

Departure City,Destination City

Sydney,Brisbane

Sydney,Melbourne

 

CSV Data Provider

Choosing CSV for the Data Provider is similar to the Simple data provider, but uses an external CSV file.
We recommend that you choose the CSV Data Provider for large CSV files, since the CSV data provider will perform much better than the Simple data provider for large quantities of data.

 

For a CSV provider, you can choose the value Separator and the text Encoding of the CSV file.

 

You can place CSV files anywhere on your computer, but we recommend you place them in the default input data folder for your agent. Later, if you want to export your agent, then you can include these files along with the export.

 

Selection Data Provider

Choosing Selection for the Data Provider will extract data from elements on the web page. The data provider uses the selection XPath of the data provider command and then adds a relative XPath to find the web elements that you include in the selection. You can choose which HTML attributes of the web elements to include as data.

 

 

The choice of Selection for the Data Provider is almost exclusively for use with drop-down menus in web forms. A form field command selects the drop down on the web page, and a relative XPath selects the option elements within that drop down. The drop-down options then become available to the form field command, which can iterate through the options and ensure that the web form submission is done for each item in the drop down. Although this is somewhat tedious, the benefit is more precise control over which options you want to submit with the web form.

 

Database Data Providers

Choose a Database for the Data Provider to work any of these database connections:

 

In Content Grabber, you can share database connections among all agents on a computer. We recommend that you read more about create new database connections.

 

 

A database data provider requires execution of a SQL Select statement on the database connection to retrieve the input data. The following example selects the two data columns, DepatureCity and DestinationCity, from a table City.

 

SELECT DepatureCity, DestinationCity FROM City