In the absence of regularization an

In the absence of regularization and for certain model families,the empirical relationship between model complexity and risk is more accurately captured by the double descent curve in the figure above. There is an interpolation threshold at which a model of the given complexity can fit the training data exactly. The complexity range below the threshold is the under parameterized regime, while the one above is the over parameterized regime. Increasing model complexity in the over parameterized regime continues to decrease risk indefinitely, albeit at decreasing marginal returns, toward some convergence point. The double descent curve is not universal. In many cases, in practice we observe a single descent curve through out the entire complexity range. In other cases, we can see multiple bumps as we increase model complexity.However, the general point remains: there is no evidence that highly over parameterized models do not generalize. And, indeed, empirical evidence suggests larger models not only generalize, but that larger models make better out-of-sample predictors than smaller ones.