Improving Agent Performance and Reliability

<< Click to Display Table of Contents >>

Navigation:  »No topics above this level«

Improving Agent Performance and Reliability

It is important to understand that the complexity of web scraping increases significantly when extracting large amounts of data. The time it takes to perform certain operations may take so long that the whole web scraping process becomes impractical. Web scraping is unreliable in nature, and the longer a web scraping project runs, the higher the chance something fails and interrupts the web scraping process.

 

The following topics list a few steps you can take to increase performance and reliability. Performance tuning is not for the novice user, and if you have no IT experience, you may not be able to implement the recommendations in this topic.

 

Using MS SQL as Internal Database

Using the Static Parser Browser

Optimizing Agent Commands