Simple scenario
Three sql transforms to execute stored procs in sqlserver as source...
query transform to join.....and continue with other stuff
Before anyone answers with why...and push down to data base and so on...not possible, a lot of complexity behind this and job is being run as part of an application splitting the data in to batches running one at a time, large data volumes. much is out of our control.
The problem is that the join is being totally ignored in first instance, from the monitor log running in to billions of rows for data sets in the region of 20,000- 40,000
It then proceeds to apply the simple join and return the correct data and continuing to complete the rest of the batch job without issue, except for the passing of a ridiculous amount of time.
Is there any way to avoid? Any ideas?