https://parameterfree.com/2026/04/03/from-online-learning-to-pac-bayes/2026-04-03T13:46:53+00:00monthlyhttps://parameterfree.com/2023/08/30/yet-another-icml-award-fiasco/https://parameterfree.com/wp-content/uploads/2023/08/usps_vw_100_0_16_crossentropy_none-1.pngusps_vw_100_0_16_crossentropy_none-1https://parameterfree.com/wp-content/uploads/2023/08/smallnorb_dense_100_0_16_crossentropy_none-1.pngsmallnorb_dense_100_0_16_crossentropy_none-1https://parameterfree.com/wp-content/uploads/2023/08/pendigits_vw_100_0_16_crossentropy_none-1.pngpendigits_vw_100_0_16_crossentropy_none-1https://parameterfree.com/wp-content/uploads/2023/08/letter_vw_100_0_16_crossentropy_none-1.pngletter_vw_100_0_16_crossentropy_none-1https://parameterfree.com/wp-content/uploads/2023/08/dna_vw_100_0_16_crossentropy_none-1.pngdna_vw_100_0_16_crossentropy_none-1https://parameterfree.com/wp-content/uploads/2023/08/usps_vw_100_0_16_crossentropy_none.pngusps_vw_100_0_16_crossentropy_nonehttps://parameterfree.com/wp-content/uploads/2023/08/smallnorb_dense_100_0_16_crossentropy_none.pngsmallnorb_dense_100_0_16_crossentropy_nonehttps://parameterfree.com/wp-content/uploads/2023/08/pendigits_vw_100_0_16_crossentropy_none.pngpendigits_vw_100_0_16_crossentropy_nonehttps://parameterfree.com/wp-content/uploads/2023/08/letter_vw_100_0_16_crossentropy_none.pngletter_vw_100_0_16_crossentropy_nonehttps://parameterfree.com/wp-content/uploads/2023/08/dna_vw_100_0_16_crossentropy_none.pngdna_vw_100_0_16_crossentropy_none2026-04-02T18:32:50+00:00monthlyhttps://parameterfree.com/2020/12/06/neural-network-maybe-evolved-to-make-adam-the-best-optimizer/https://parameterfree.com/wp-content/uploads/2020/12/exp_linesearch2.pngexp_linesearch2https://parameterfree.com/wp-content/uploads/2020/12/exp_linesearch1.pngexp_linesearch1https://parameterfree.com/wp-content/uploads/2020/12/adam.pngadamAdam algorithm, from [Kingma and Ba, 2015]2026-04-02T13:25:47+00:00monthlyhttps://parameterfree.com/2021/07/06/the-myth-of-the-expert-reviewer/https://parameterfree.com/wp-content/uploads/2021/06/meme-1.jpgmemehttps://parameterfree.com/wp-content/uploads/2021/06/dirac.pngdirachttps://parameterfree.com/wp-content/uploads/2021/06/pizza.jpegpizza2026-04-02T13:25:14+00:00monthlyhttps://parameterfree.com/2026/03/12/from-online-learning-to-rademacher-complexity/2026-03-12T18:53:16+00:00monthlyhttps://parameterfree.com/2026/02/26/better-optimistic-bounds-and-delays-as-bad-hints/https://parameterfree.com/wp-content/uploads/2026/02/oomd_new.pngoomd_newhttps://parameterfree.com/wp-content/uploads/2026/02/oftrl_new.pngoftrl_new2026-02-26T12:52:10+00:00monthlyhttps://parameterfree.com/2025/12/11/the-gaptron-algorithm/https://parameterfree.com/wp-content/uploads/2025/12/gaptron.pnggaptron2025-12-12T07:17:54+00:00monthlyhttps://parameterfree.com/2025/11/11/the-aggregating-algorithm/https://parameterfree.com/wp-content/uploads/2025/11/aa_logistic.pngaa_logistichttps://parameterfree.com/wp-content/uploads/2025/11/aa.pngaa2025-11-12T07:42:18+00:00monthlyhttps://parameterfree.com/2025/09/09/the-weighted-average-algorithm/https://parameterfree.com/wp-content/uploads/2025/09/waa.pngwaa2025-11-11T14:09:07+00:00monthlyhttps://parameterfree.com/2019/10/01/online-mirror-descent-ii-regret-and-mirror-version/https://parameterfree.com/wp-content/uploads/2019/10/dual_mappings.pngdual_mappings2025-10-30T07:13:42+00:00monthlyhttps://parameterfree.com/2019/11/14/multi-armed-bandit-ii/https://parameterfree.com/wp-content/uploads/2019/11/screenshot_20250404_104729.pngScreenshot_20250404_104729https://parameterfree.com/wp-content/uploads/2019/11/inf.pnginfhttps://parameterfree.com/wp-content/uploads/2019/11/exp3_fig1.pngexp3_fig12025-04-04T07:51:39+00:00monthlyhttps://parameterfree.com/2025/02/11/strongly-adaptive-regret-and-cbce/https://parameterfree.com/wp-content/uploads/2025/02/gi.pnggihttps://parameterfree.com/wp-content/uploads/2025/02/cbce2.pngcbce2https://parameterfree.com/wp-content/uploads/2025/02/cbce.pngcbce2025-02-13T07:55:05+00:00monthlyhttps://parameterfree.com/2024/05/27/black-box-reductions-sleeping-experts/https://parameterfree.com/wp-content/uploads/2024/05/sleeping_experts.pngsleeping_experts2025-02-11T09:50:33+00:00monthlyhttps://parameterfree.com/2018/06/29/the-negative-gradient-does-not-point-towards-the-minimum/https://parameterfree.com/wp-content/uploads/2018/06/gradient_not_minimum4.gifgradient_not_minimum4https://parameterfree.com/wp-content/uploads/2018/06/gradient_not_minimum31.gifgradient_not_minimum3https://parameterfree.com/wp-content/uploads/2018/06/gradient_not_minimum22.gifgradient_not_minimum2https://parameterfree.com/wp-content/uploads/2018/06/gradient_not_minimum1.gifgradient_not_minimum1https://parameterfree.com/wp-content/uploads/2018/06/figura206.jpgfigura206https://parameterfree.com/wp-content/uploads/2018/06/algo_gd.pngalgo_gd2024-12-14T07:45:53+00:00monthlyhttps://parameterfree.com/2019/11/07/optimistic-follow-the-regularized-leader/https://parameterfree.com/wp-content/uploads/2019/11/ftrl_optimistic.pngftrl_optimistic2024-07-07T08:12:00+00:00monthlyhttps://parameterfree.com/2024/07/01/dynamic-regret-and-ader/https://parameterfree.com/wp-content/uploads/2024/07/ader.pngader2024-07-01T13:45:32+00:00monthlyhttps://parameterfree.com/2024/04/29/black-box-reductions-constrained-online-learning/https://parameterfree.com/wp-content/uploads/2024/04/black_box_constrained.pngblack_box_constrained2024-09-05T07:25:03+00:00monthlyhttps://parameterfree.com/2023/11/30/is-nesterovs-universal-algorithm-really-universal/https://parameterfree.com/wp-content/uploads/2023/11/nesterov_universal2.pngnesterov_universal2https://parameterfree.com/wp-content/uploads/2023/11/nesterov_universal.pngnesterov_universal2024-03-01T05:52:42+00:00monthlyhttps://parameterfree.com/2024/02/14/a-minimizer-far-far-away/https://parameterfree.com/wp-content/uploads/2024/02/choice.jpgchoicehttps://parameterfree.com/wp-content/uploads/2024/02/image-1.pngimage (1)2025-03-10T07:26:44+00:00monthlyhttps://parameterfree.com/2019/10/03/online-mirror-descent-iii-examples-and-learning-with-expert-advice/https://parameterfree.com/wp-content/uploads/2019/10/lea.pngleahttps://parameterfree.com/wp-content/uploads/2019/10/eg.pngeg2024-02-04T08:14:31+00:00monthlyhttps://parameterfree.com/2019/09/25/lower-bounds-for-online-linear-optimization/2023-10-23T06:50:40+00:00monthlyhttps://parameterfree.com/about-me/https://parameterfree.com/wp-content/uploads/2018/04/foto_colori.pngfoto_colori2023-08-31T05:09:18+00:00weekly0.6https://parameterfree.com/2023/06/19/adapting-to-smoothness-with-normalized-gradients/https://parameterfree.com/wp-content/uploads/2023/06/meme.jpgmemehttps://parameterfree.com/wp-content/uploads/2023/06/nogd2.pngnogd2https://parameterfree.com/wp-content/uploads/2023/06/screenshot_20230619_092627.pngScreenshot_20230619_0926272023-08-15T18:17:48+00:00monthlyhttps://parameterfree.com/icml-tutorial/2023-06-06T21:58:28+00:00weekly0.6https://parameterfree.com/2022/11/07/saddle-point-optimization-with-optimism/https://parameterfree.com/wp-content/uploads/2022/11/prescient_omd.pngprescient_omdhttps://parameterfree.com/wp-content/uploads/2022/11/saddle_point_omd.pngsaddle_point_omdhttps://parameterfree.com/wp-content/uploads/2022/11/saddle_point_oftrl.pngsaddle_point_oftrl2022-11-17T22:29:17+00:00monthlyhttps://parameterfree.com/2022/09/19/a-simple-proof-for-optimistic-omd/https://parameterfree.com/wp-content/uploads/2022/09/optimistic_omd.pngoptimistic_omdOptimistic OMD2022-10-19T18:36:29+00:00monthlyhttps://parameterfree.com/2021/02/14/perceptron-is-not-sgd-a-better-interpretation-through-pseudogradients/https://parameterfree.com/wp-content/uploads/2021/02/perceptron_2-2.pngperceptron_2https://parameterfree.com/wp-content/uploads/2021/02/perceptron_2-1.pngperceptron_2https://parameterfree.com/wp-content/uploads/2021/02/perceptron_2.pngperceptron_22022-09-09T15:22:36+00:00monthlyhttps://parameterfree.com/2022/09/07/saddle-points-game-theory-and-boosting/https://parameterfree.com/wp-content/uploads/2022/09/saddle_point.pngsaddle_pointThe saddle point in the Matching Pennies game.https://parameterfree.com/wp-content/uploads/2022/09/boosting_oco.pngboosting_oco2022-10-05T17:42:59+00:00monthlyhttps://parameterfree.com/2021/12/02/saddle-point-problems-and-oco/https://parameterfree.com/wp-content/uploads/2021/12/no_saddle_point.pngno_saddle_pointhttps://parameterfree.com/wp-content/uploads/2021/12/algo_spp_1.pngalgo_spp_1https://parameterfree.com/wp-content/uploads/2021/12/algo_spp_4.pngalgo_spp_4https://parameterfree.com/wp-content/uploads/2021/12/algo_spp_3.pngalgo_spp_3https://parameterfree.com/wp-content/uploads/2021/12/algo_spp_2-1.pngalgo_spp_2https://parameterfree.com/wp-content/uploads/2021/12/algo_spp_2.pngalgo_spp_22022-09-08T21:38:13+00:00monthlyhttps://parameterfree.com/lecture-notes-on-online-learning/2022-07-24T20:49:35+00:00weekly0.6https://parameterfree.com/2021/06/12/implicit-passive-aggressive-and-aprox-updates/2022-02-15T20:48:15+00:00monthlyhttps://parameterfree.com/2019/09/17/more-online-to-batch-examples-and-strong-convexity/https://parameterfree.com/wp-content/uploads/2019/09/strongly_convex.pngstrongly_convex2021-09-06T23:24:08+00:00monthlyhttps://parameterfree.com/2019/09/20/adaptive-algorithms-l-bounds-and-adagrad/https://parameterfree.com/wp-content/uploads/2019/09/adagrad-1.pngadagradhttps://parameterfree.com/wp-content/uploads/2019/09/adagrad.pngadagrad2022-12-26T02:10:49+00:00monthlyhttps://parameterfree.com/2019/10/17/follow-the-regularized-leader-iii-more-logarithmic-bounds/https://parameterfree.com/wp-content/uploads/2019/10/ons-1.pngonshttps://parameterfree.com/wp-content/uploads/2019/10/vaw.pngvawhttps://parameterfree.com/wp-content/uploads/2019/10/ons.pngons2023-09-04T14:41:22+00:00monthlyhttps://parameterfree.com/2020/10/05/almost-sure-convergence-of-sgd-on-smooth-non-convex-functions/https://parameterfree.com/wp-content/uploads/2020/10/sgd_on_sin.pngsgd_on_sinGD vs SGD to minimize $latex {F(x)=\sin(x)}&fg=000000$. In SGD the derivative is corrupted by additive Gaussian noise.2023-06-25T20:23:33+00:00monthlyhttps://parameterfree.com/2020/05/03/adahedge/https://parameterfree.com/wp-content/uploads/2020/05/screenshot_20200503_170947.pngScreenshot_20200503_1709472025-09-10T16:52:34+00:00monthlyhttps://parameterfree.com/2020/08/07/last-iterate-of-sgd-converges-even-in-unbounded-domains/https://parameterfree.com/wp-content/uploads/2020/08/darkside.jpgdarkside2021-04-30T04:00:53+00:00monthlyhttps://parameterfree.com/2019/11/21/multi-armed-bandit-iv-ucb/https://parameterfree.com/wp-content/uploads/2019/11/ucb_alg.pngucb_alghttps://parameterfree.com/wp-content/uploads/2019/11/ucb.gifucb2020-07-20T17:25:05+00:00monthlyhttps://parameterfree.com/2019/10/08/follow-the-regularized-leader-i-regret-equality/https://parameterfree.com/wp-content/uploads/2019/10/dual_mappings_ftrl-1.pngdual_mappings_ftrlhttps://parameterfree.com/wp-content/uploads/2019/10/ftrl_algo_linear.pngftrl_algo_linearhttps://parameterfree.com/wp-content/uploads/2019/10/dual_mappings_ftrl.pngdual_mappings_ftrlhttps://parameterfree.com/wp-content/uploads/2019/10/ftrl_algo.pngftrl_algo2023-06-19T11:56:38+00:00monthlyhttps://parameterfree.com/2019/10/30/parameter-free-online-learning-i-coin-betting-and-1-d-oco/https://parameterfree.com/wp-content/uploads/2019/10/comparison_kt.pngcomparison_kthttps://parameterfree.com/wp-content/uploads/2019/10/kt_oco.pngkt_ocohttps://parameterfree.com/wp-content/uploads/2019/10/kt.pngkt2022-04-13T23:19:35+00:00monthlyhttps://parameterfree.com/2019/10/10/follow-the-regularized-leader-ii-applications/https://parameterfree.com/wp-content/uploads/2019/10/ftrl_algo_quad.pngftrl_algo_quadhttps://parameterfree.com/wp-content/uploads/2019/10/ftrl_algo_linear2.pngftrl_algo_linear22023-05-30T01:08:30+00:00monthlyhttps://parameterfree.com/2019/09/11/online-gradient-descent/https://parameterfree.com/wp-content/uploads/2019/09/pogd.pngpogdhttps://parameterfree.com/wp-content/uploads/2019/09/convex_nonconvex_epi.pngconvex_nonconvex_epihttps://parameterfree.com/wp-content/uploads/2019/09/convex_nonconvex.pngconvex_nonconvex2024-02-04T22:18:00+00:00monthlyhttps://parameterfree.com/2019/09/02/introduction-to-online-learning/https://parameterfree.com/wp-content/uploads/2019/09/sum_log.pngsum_log2023-10-17T10:05:08+00:00monthlyhttps://parameterfree.com/2019/10/31/parameter-free-online-learning-ii-reductions/https://parameterfree.com/wp-content/uploads/2019/10/magnitude_direction-1.pngmagnitude_directionhttps://parameterfree.com/wp-content/uploads/2019/10/kt_coordinate.pngkt_coordinatehttps://parameterfree.com/wp-content/uploads/2019/10/magnitude_direction.pngmagnitude_direction2019-12-04T17:26:29+00:00monthlyhttps://parameterfree.com/2019/11/27/from-online-learning-to-statistical-learning/2019-11-27T22:20:45+00:00monthlyhttps://parameterfree.com/2019/10/23/online-linear-classification/https://parameterfree.com/wp-content/uploads/2019/10/perc-1.pngperchttps://parameterfree.com/wp-content/uploads/2019/10/powers_hinge.pngpowers_hingehttps://parameterfree.com/wp-content/uploads/2019/10/perc.pngperchttps://parameterfree.com/wp-content/uploads/2019/10/rand_classif.pngrand_classif2019-11-26T18:20:39+00:00monthlyhttps://parameterfree.com/2019/11/19/multi-armed-bandit-iii-stochastic-bandits/https://parameterfree.com/wp-content/uploads/2019/11/etc.pngetc2021-01-23T15:48:15+00:00monthlyhttps://parameterfree.com/2019/09/26/online-mirror-descent-i-bregman-version/2022-02-26T12:17:47+00:00monthlyhttps://parameterfree.com/2019/11/12/multi-armed-bandit-i/https://parameterfree.com/wp-content/uploads/2019/11/simplex3.pngsimplex3https://parameterfree.com/wp-content/uploads/2019/11/simplex1.pngsimplex1https://parameterfree.com/wp-content/uploads/2019/11/simplex2.pngsimplex2https://parameterfree.com/wp-content/uploads/2019/11/exp3.pngexp3https://parameterfree.com/wp-content/uploads/2019/11/eg_bandit.pngeg_bandit2019-11-13T02:49:26+00:00monthlyhttps://parameterfree.com/2019/11/05/parameter-free-online-learning-iii-from-coins-to-experts/2021-12-13T11:39:27+00:00monthlyhttps://parameterfree.com/2019/09/13/subgradients-and-online-to-batch-conversion/https://parameterfree.com/wp-content/uploads/2019/09/posd.pngposd2020-02-27T15:32:31+00:00monthlyhttps://parameterfree.com/2018/06/20/54/https://parameterfree.com/wp-content/uploads/2018/06/omd.pngomdhttps://parameterfree.com/wp-content/uploads/2018/06/sd_approximations.pngsd_approximationshttps://parameterfree.com/wp-content/uploads/2018/06/sd_stepsizes.pngsd_stepsizeshttps://parameterfree.com/wp-content/uploads/2018/06/sd_not_descent2.pngsd_not_descent2https://parameterfree.com/wp-content/uploads/2018/06/sd_not_descent1.pngsd_not_descent1https://parameterfree.com/wp-content/uploads/2018/06/sd_algo2.pngsd_algo2https://parameterfree.com/wp-content/uploads/2018/06/algo_sd.pngalgo_sd2020-10-08T00:40:24+00:00monthlyhttps://parameterfree.com/recent-posts/2019-09-25T14:27:29+00:00weekly0.6https://parameterfree.com/welcome/2019-09-25T14:27:00+00:00weekly0.6https://parameterfree.com/2018/04/28/the-journey-begins/https://parameterfree.com/wp-content/uploads/2018/04/dsc_0446.jpgDSC_0446https://parameterfree.com/wp-content/uploads/2018/04/fly.jpgPlaceholder Image2018-04-29T22:18:07+00:00monthlyhttps://parameterfree.com/contact/https://parameterfree.com/wp-content/uploads/2018/04/person-smartphone-office-table.jpegPlaceholder Image2018-04-29T21:05:49+00:00weekly0.6https://parameterfree.comdaily1.02026-04-03T13:46:53+00:00