Optimizing Agent Commands


Navigation:  Improving Agent Performance and Reliability >

Optimizing Agent Commands

Action commands such as Navigate Link and Navigate URL commands are much slower to execute than commands that simply capture content, so it's important to try and limit the number of action commands in your agents. If you can access a target web page directly with a direct URL, rather than clicking multiple links, then you should always opt for the direct URL.


Many dynamic websites show and hide content when you click on buttons or links. Content Grabber can extract both visible and hidden content, so it's often not necessary to add an action command that simply makes content visible.


Optimizing Browser Activities

An agent spends most of its time waiting for web pages to load, and Content Grabber cannot always determine exactly when a page load or page action has completed, so it sometimes ends up waiting longer than it has to. If the agent starts extracting data too early, it may not be able to extract the data correctly, since some data may not have loaded onto the web page yet.


If an agent command executes an action that results in multiple browser activities, such as multiple page loads and AJAX callbacks, then Content Grabber will set the action option Discover activities, instead of setting the exact browser activities the action command should wait for. Content Grabber does this because the number of activities or the order of activities is likely to change, so it's safest to keep discovering the activities every time the action is executed. However, discovering activities is slower than waiting for a set number of activities, because Content Grabber has to wait to make sure no more activities occur before it can consider the action as completed. It's always best to manually set the browser activities Content Grabber should wait for, so it doesn't wait longer than absolutely necessary, but this can be difficult to get right and may not be worth the effort unless performance is very important.


When an action command uses the option Discover activities, it uses a set of timeout values to determine how long it should wait for new activities. These timeout values are set fairly conservatively to make sure they are long enough for slow computers and slow Internet connections. You may be able to lower these timeout values, so an action command completes faster without failing to extract content correctly.



Default activity timeout values


You can use the Browser Activity panel to view activities and timings for an action. Please see the topic Browser Activities for more information.