-
Notifications
You must be signed in to change notification settings - Fork 397
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Strange Sharding with pipeline parallel around Scatter op - TP/PP/DP #15048
Comments
Sharding metadata looks correct for my ops also. I do not understand why the all-gather is not over the TP group so {0-7} in first case |
I compared on CPU HLO runner the above HLO with those ops and one with a pass I wrote to simplify to all-gather over TP group. according to HLO runner they produce equal results
|
In general SPMD partitioner is hard to debug and personally I did not find a better method than inserting print statements to the pass methodologically. It does look like the |
I am seeing very strange sharding with pipeline parallel and tensor, data parallel.
Below is the HLO exactly before partitioning:
After partitioning I see a strange all-reduce over half the TP group and then collective-permutes. Then a strange all-gather over a group of random 8 workers. I have tried debugging this in SPMD logs with VLOG=10, but I have not found the reason behind this sharding.
I would have expected an all-gather of the TP group before the scatter op. Does anyone have any explanation or pointers on how to debug this?
After partition
The text was updated successfully, but these errors were encountered: