% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/optim-adagrad.R
\name{optim_adagrad}
\alias{optim_adagrad}
\title{Adagrad optimizer}
\usage{
optim_adagrad(
  params,
  lr = 0.01,
  lr_decay = 0,
  weight_decay = 0,
  initial_accumulator_value = 0,
  eps = 1e-10
)
}
\arguments{
\item{params}{(iterable): list of parameters to optimize or list parameter groups}

\item{lr}{(float, optional): learning rate (default: 1e-2)}

\item{lr_decay}{(float, optional): learning rate decay (default: 0)}

\item{weight_decay}{(float, optional): weight decay (L2 penalty) (default: 0)}

\item{initial_accumulator_value}{the initial value for the accumulator. (default: 0)

Adagrad is an especially good optimizer for sparse data.
It individually modifies learning rate for every single parameter,
dividing the original learning rate value by sum of the squares of the gradients.
It causes that the rarely occurring features get greater learning rates.
The main downside of this method is the fact that learning rate may be
getting small too fast, so that at some point a model cannot learn anymore.}

\item{eps}{(float, optional): term added to the denominator to improve
numerical stability (default: 1e-10)}
}
\description{
Proposed in \href{https://jmlr.org/papers/v12/duchi11a.html}{Adaptive Subgradient Methods for Online Learning and Stochastic Optimization}
}
\note{
Update rule:
\deqn{
\theta_{t+1} = \theta_{t} - \frac{\eta }{\sqrt{G_{t} + \epsilon}} \odot g_{t}
}
The equation above and some remarks quoted
after \href{https://web.archive.org/web/20220810011734/https://ruder.io/optimizing-gradient-descent/index.html}{\emph{An overview of gradient descent optimization algorithms}}
by Sebastian Ruder.
}
\section{Warning}{
If you need to move a model to GPU via \verb{$cuda()}, please do so before
constructing optimizers for it. Parameters of a model after \verb{$cuda()}
will be different objects from those before the call. In general, you
should make sure that the objects pointed to by model parameters subject
to optimization remain the same over the whole lifecycle of optimizer
creation and usage.
}

