This overload of addcmul_ is deprecated: addcmul_(Number value, Tensor tensor1, Tensor tensor2)

[neuronflow]

I get the following warning when using ranger with pytorch 1.6.0

/path/Ranger-Deep-Learning-Optimizer/ranger/ranger.py:138: UserWarning: This overload of addcmul_ is deprecated:
        addcmul_(Number value, Tensor tensor1, Tensor tensor2)
Consider using one of the following signatures instead:
        addcmul_(Tensor tensor1, Tensor tensor2, *, Number value) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
  exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)

[ioanvl]

Noticed the same as well, and its been happening since a few versions of pytorch ago.

[lessw2020]

Hi @neuronflow @ioanvl - thanks for issue.
The warning is resolved now in the file ranger2020.py
It also adds in latest gc2 from the gc developer so I wanted to test it out a bit more as it's own file before integrating but performs great on my private dataset with no warnings.

[hiyyg]

@lessw2020 I noticed in ranger2020.py, gc_loc is set to True by default. However, the original repo seems to suggest using gc_loc=False for adaptive learning rate methods. So why do you set it to True by default?

[brando90]

can one give an example of how to fix this? I tried and I still can't:

                # exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
                one_minus_beta2: torch.Tensor = torch.tensor(1 - beta2, device=grad.device)
                exp_avg_sq.mul_(beta2).addcmul_(one_minus_beta2, grad, grad)
                exp_avg.mul_(beta1).add_(1 - beta1, grad)

err

/Users/miranda9/RAdam/radam/radam.py:60: UserWarning: This overload of addcmul_ is deprecated:
	addcmul_(Number value, Tensor tensor1, Tensor tensor2)
Consider using one of the following signatures instead:
	addcmul_(Tensor tensor1, Tensor tensor2, *, Number value) (Triggered internally at  /Users/distiller/project/conda/conda-bld/pytorch_1623459065530/work/torch/csrc/utils/python_arg_parser.cpp:1025.)
  exp_avg_sq.mul_(beta2).addcmul_(one_minus_beta2, grad, grad)

[Harly-1506]

fix addcmul and add_

Try this:
exp_avg_sq.mul_(beta2).addcmul(grad, grad, value = 1 - beta2)
exp_avg.mul_(beta1).add_(grad, alpha = 1 - beta1 )