GenSynth Documentation

Using Multiple GPUs

When using multiple GPUs or machines to train, each node is responsible for a portion of the data. These portions are shards.

For example, if a full dataset has 40 000 images and 4 nodes are being used, each node must produce an epoch of 10 000 images.

DarwinAI recommends that all data is available to all nodes, wherein each worker selects their shard portion—the get_train_dataset() and get_validation_dataset() functions select from the data according to the passed-in index.