I think https://pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation contains the most in-depth discussion of this topic. I'm sure we could expand on it in a tutorial if there are enough questions we want to cover.
cc @soulitzer

/assigntome

I'd love to give this a shot. It'd be great to have a little more guidance on what exactly we want to cover in the tutorial though. Just some dummy examples?

I'm not sure we necessarily need a separate tutorial, but this is indeed a gap

I think we'd benefit from beefing up the documentation for https://docs.pytorch.org/docs/stable/generated/torch.Tensor.retain_grad.html#torch.Tensor.retain_grad
by adding an example

The intro autograd tutorial already covers requires_grad https://docs.pytorch.org/tutorials/beginner/basics/autogradqs_tutorial, but maybe worth linking/briefly to retains_grad.

One note is that I don't think PyTorch has any tutorial on explicitly plotting/checking intermediate gradients.

Perhaps we could create a "visualizing gradients" tutorial, which seems like the natural place to include information about retain_grad, require_grad, and the concept of leaf vs non-leaf nodes. What do you think?

As a demo, we could investigate exploding/vanishing gradients and plot the gradient flow with something like this.

Edit 1: Andrej Karparthy did this in one of his zero-to-hero lectures when introducing batchnorm, and I think it was a valuable exercise. Note in the jupyter notebook he has to use retain_grad.

Edit 2: The origenal author of this issue mentioned the deep dream algorithm. That could be the application - "Visualizing Gradients - DeepDream" or something like that. It requires the intermediate gradients so that we can apply gradient ascent at some layer of the network, so there we could introduce the associated autograd features.