The Remove Duplicate command checks extracted data against previously extracted data and discards the extracted data if it already exists.
The Remove Duplicate command should be added after the capture commands that extract the data that should be checked, and the Remove Duplicate command must have the same parent command as the capture commands.
When a Remove Duplicate command detects a duplicate it will exit a specified parent command. By default, a Remove Duplicate command exits the current list entry of the nearest parent list command, or the nearest parent command that exports data to a separate data table.
If the target data is displayed by descending order, then you can use a Remove Duplicate command to extract only new data by exiting the Agent command when a duplicate is detected. For example, if the target data is ordered by newest date to oldest date and the Remove Duplicate command detects that a specific date has already been extracted, then you know that all data before that date has already been extracted and you can exit the agent.