Hi,
I have a full refresh file with 1 million records. I should load it into a table initially and extract it in a file. From the second day I should get only the inserts and deletes. I used table comparison stage to achieve this. I tested my job with 3 lakh records and the table comparison takes approximately 10 minutes. Is there any way to improve the performance of this stage.
Note: I checked there is an option where we can set the comparison method as "sorted input" instead of row by row select. For this option, is it enough if my source file alone is sorted.