Content Grabber uses two layers of internal data, internal data and export data. While the agent is running, it continuously saves data in an internal format. When the agent finishes, it converts the internal data into export data, and then it sends this data to the export target so it becomes external data.
The default internal database is a SQLite database, but this can be changed SQL Server or MySQL. Oracle and OleDB are not currently supported as internal databases.
Content Grabber stores internal data in the database - which contains all the data extraction elements for any agent, but also contains the data which corresponds to the settings for the agent properties. For example, if an agent stops, crashes, or otherwise fails, the internal data is used to start the agent again exactly at the point of failure. Also, the internal data will always contain a data table for each container command in the agent and at least one data field for each capture command in the agent. You can always view the internal data by clicking the Data > View Internal Data menu.
If you configure an agent to add new data to the existing data, then the internal data store will continue to grow larger. Otherwise, the internal data store will be recreated every time the agent is run.
The agent stores the export data in the internal database, and this data doesn't contain any of the property data for the agent itself. Also, the export data is more readable than the internal data. The export data will usually not contain a data table for each container command in the agent, and some capture commands may not have corresponding data fields.
Export data is always overwritten, so you cannot add data to existing export data. You can only add data to existing internal data. You can always view the internal data by clicking the Data > View Export Data menu.
External data is the same as the export data, and the agent delivers it to an external data store chosen by the user. Typically, external data is overwritten, except when you choose Export last data segment only and you are exporting to a database. In that case, the operation simply adds the data to the external database tables. You can also use a Data Export Script to customize the export process which will allow you to update or add to existing data. The export data is set in a fixed format, but you have some options for changing this format. You need to use an Data Export Script if you want to automatically create or configure your own data structures within this data.
The default internal database is a SQLite file database, but you can change it to either a SQL Server or MySQL database. You can make this change on the Internal Database screen.
WARNING: It is important to ensure that no two agents that use the same internal database have the same agent name.
Each time you run an agent, Content Grabber verifies the tables in the internal database. If the tables don't exist or are invalid, it removes and recreates all tables for that agent. If any changes have been made to the agent, there may be some inconsistencies. Instead of removing all tables, it removes all tables starting with the specific agent name.
All internal data tables have the following naming format:
VWR INTERNAL [PROJECT_NAME] [TEMPLATE_NAME]
When Content Grabber removes a table for a specific agent, it removes all tables starting with the name:
VWR INTERNAL [PROJECT_NAME]
This approach creates an obvious problem when two agents use the same internal database and also start with the same name, because Content Grabber would remove the tables for both projects when it should only remove tables for one project. It is important to ensure that no two agents that use the same internal database have the same agent name.