Implement Deduplication Steps

Add a deduplication step within the import process that identifies and removes duplicate records before they are committed to the database.