This will generate a config file that will be used automatically to properly set the default options when doing accelerate launch my_script.py -args_to_my_scriptįor instance, here is how you would run the GLUE example on the MRPC task (from the root of the repo): accelerate launch examples/nlp_example.py On your machine(s) just run: accelerate configĪnd answer the questions asked. No need to remember how to use or to write a specific launcher for TPU training! □ Accelerate also provides an optional CLI tool that allows you to quickly configure and test your training environment before launching the scripts. Want to learn more? Check out the documentation or have a look at our examples. source = source.to(device) - targets = targets.to(device) optimizer.zero_grad() + from accelerate import Accelerator - device = 'cpu' + accelerator = Accelerator() - model = torch.nn.Transformer().to(device) + model = torch.nn.Transformer() optimizer = (model.parameters()) □ Accelerate even handles the device placement for you (which requires a few more changes to your code, but is safer in general), so you can even simplify your training loop further: import torch In particular, the same code can then be run without modification on your local machine for debugging or your training environment. loss.backward() + accelerator.backward(loss) optimizer.step()Īs you can see in this example, by adding 5-lines to any standard PyTorch training script you can now run on any kind of single or distributed node setting (single CPU, single GPU, multi-GPUs and TPUs) as well as with or without mixed precision (fp8, fp16, bf16). + model, optimizer, data = accelerator.prepare(model, optimizer, data) ain() Optimizer = (model.parameters())ĭata = (dataset, shuffle=True) + from accelerate import Accelerator + accelerator = Accelerator() - device = 'cpu' + device = vice model = torch.nn.Transformer().to(device) □ Accelerate abstracts exactly and only the boilerplate code related to multi-GPUs/TPU/fp16 and leaves the rest of your code unchanged. □ Accelerate was created for PyTorch users who like to write the training loop of PyTorch models but are reluctant to write and maintain the boilerplate code needed to use multi-GPUs/TPU/fp16. Run your *raw* PyTorch training script on any kind of device
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |