![]() ![]() ![]() Pipeline parameter, both in trigger and debug mode. If I got your question right, your key point is: how toĭynamically pass value to your pipeline parameter and after copy finishes, Has anyone found a good way to do incrementals that works well for scheduled runs and manual runs? To dynamically set the pipeline parameters within an activity task and using the if activity adds too complex of logic. Using the trigger if executed by the trigger, or using the parameter prompts if executed by debug or trigger now. To set a windowStart and windowEnd parameter using two methods depending on how the pipeline is executed. The only thing I don't like is it doesn't easily allow me to manually run the pipeline for a specific window of time without manually updating the watermark values in the Azure DB. Then I'll store these two values in the watermark table for the next day's run. So I am going to grab data between the previous High WaterMark and Today's loads start time. Then in the copy activity I am going to have a Where clauseīetween Old High water mark, and my Scheduled Trigger's windowStart time. In my pipeline I am going to lookup the previous High watermark directly before my copy activity. I am going to store the Low and High watermarks for each source table in an Azure SQL DB. I am going back and updating my pipelines to start using watermarks but want to make sure I'm doing so in a way that makes I noticed that the lookup task now allows connections other than Azure (it didn't when I developed my existing v2 pipelines). ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |