Fix non-deterministic output of gnn sampler

[tzemicheal]

Description

Fix the issues by updating the sampler during inference to full sampling from subsampling.

Closes #1676

By Submitting this PR I confirm:

• I am familiar with the Contributing Guidelines.
• When the PR is ready for review, new or existing tests cover these changes.
• When the PR is ready for review, the documentation is up to date with these changes.

[dagardner-nv]

LGTM, but I don't fully understand the differences in these samplers. We should probably have someone from the DS side review this as well.

[tzemicheal]

The issue is caused due to the non-determinist part introduced by dgl.dataloading.DataLoader when used along neighbor dgl.dataloading.MultiLayerNeighborSampler sampling instead of full sampling. There are two ways to fix this (which both of them fixed the issue)

• Change the neighbor sampling deterministic, and set seed every time we call dataloading.DataLoader ref: https://discuss.dgl.ai/t/reproducibility-dataloader-shuffle-true-using-seeds/4275/2 .
• Change the neighbor sampling to dgl.dataloading.MultiLayerFullNeighborSampler for the inference part. This has minimal effect on small graphs, for large graphs, it might be slower neighbor sampling. This fix uses this approach.

Ref:

• https://docs.dgl.ai/en/1.1.x/tutorials/large/L0_neighbor_sampling_overview.html

[raykallen]

lgtm

[mdemoret-nv]

/merge