Apex with gradient accumulation results in inaccurate gradients if reduction is disabled for steps where optimizer is not stepping.

A clear understanding of how NVIDIA mixed precission training works. This blog also lists out official documentations necessary to understand the concepts

Suppose you did a git rebase in your local branch but mistakenly rebased to an older branch and pushed changes to remote, then here is the solution to revert your changes and go back to the previou...

Challenges and approaces to submit job in VS code with links, results and opinions

Insertion transformer paper breif summary

How to install and run spark in local windows 10 machines using two installation methods - binaries and pip.

WRITING DISTRIBUTED APPLICATIONS WITH PYTORCH The distributed package included in PyTorch (i.e., torch.distributed) enables researchers and practitioners to easily parallelize their computations a...

#!pip install pytorch_transformers #!pip install seaborn import torch from pytorch_transformers import BertConfig,BertTokenizer, BertModel BertModel model_type = 'bert-base-uncased' config = Be...

Steps to create, test and push a docker image Download and install Docker Desktop. If on Linux, download Docker Engine - Community. Create a Docker file with the name Dockerfile or download o...

LAMB paper Previous LR scaling with batch size Simple large batch training Training with extremely large batch was difficult. The researchers needed to carefully tune training hyper-parameter...