There are four types of partitions that can be done at session level.
1. Round Robin
2. Key Range
3. Hask Key
4. Pass Through
Depending on your requirement you can add partitioning scheme. When you want to group data based on some key, then you can go with Hash Key Partitioning. When you parititon data based on upper and lower limit then go for key range partition. Round robin is most useful when you can apply it on passive transformations before loading to target, it basically splits the flow into multiple chunks and creates multiple temporary files and later merges all into one at the end of session run. Pass through partitioning will run in single pipeline...which is by default for all your sessions. It is always important while choosing the partitions schemes while running sessions and also depends how your data is, if the data is very sparse and if you dont define the required partition correctly, then some pipelines will have less records whereas as others will be more...so ideally you should understand range or volume or groups before you set.
1. Round Robin
2. Key Range
3. Hask Key
4. Pass Through
Depending on your requirement you can add partitioning scheme. When you want to group data based on some key, then you can go with Hash Key Partitioning. When you parititon data based on upper and lower limit then go for key range partition. Round robin is most useful when you can apply it on passive transformations before loading to target, it basically splits the flow into multiple chunks and creates multiple temporary files and later merges all into one at the end of session run. Pass through partitioning will run in single pipeline...which is by default for all your sessions. It is always important while choosing the partitions schemes while running sessions and also depends how your data is, if the data is very sparse and if you dont define the required partition correctly, then some pipelines will have less records whereas as others will be more...so ideally you should understand range or volume or groups before you set.
Thanks for sharing,
ReplyDeleteThank you very much for this blog and it is written in a very easy way of understanding the concepts .
ReplyDelete