{"componentChunkName":"component---src-templates-post-tsx","path":"/machine_learning_math/subgradient_descent/","result":{"data":{"logo":{"childImageSharp":{"gatsbyImageData":{"layout":"fixed","backgroundColor":"#f8f8f8","images":{"fallback":{"src":"/static/e43c6530b7b2b056de6bfee951c9e393/b181d/compasslogo.png","srcSet":"/static/e43c6530b7b2b056de6bfee951c9e393/b181d/compasslogo.png 1920w","sizes":"1920px"},"sources":[{"srcSet":"/static/e43c6530b7b2b056de6bfee951c9e393/e02c9/compasslogo.webp 1920w","type":"image/webp","sizes":"1920px"}]},"width":1920,"height":472}}},"mdx":{"body":"const _excluded=[\"components\"];function _extends(){_extends=Object.assign||function(target){for(var i=1;i<arguments.length;i++){var source=arguments[i];for(var key in source){if(Object.prototype.hasOwnProperty.call(source,key)){target[key]=source[key];}}}return target;};return _extends.apply(this,arguments);}function _objectWithoutProperties(source,excluded){if(source==null)return{};var target=_objectWithoutPropertiesLoose(source,excluded);var key,i;if(Object.getOwnPropertySymbols){var sourceSymbolKeys=Object.getOwnPropertySymbols(source);for(i=0;i<sourceSymbolKeys.length;i++){key=sourceSymbolKeys[i];if(excluded.indexOf(key)>=0)continue;if(!Object.prototype.propertyIsEnumerable.call(source,key))continue;target[key]=source[key];}}return target;}function _objectWithoutPropertiesLoose(source,excluded){if(source==null)return{};var target={};var sourceKeys=Object.keys(source);var key,i;for(i=0;i<sourceKeys.length;i++){key=sourceKeys[i];if(excluded.indexOf(key)>=0)continue;target[key]=source[key];}return target;}/* @jsxRuntime classic */ /* @jsx mdx */const _frontmatter={\"layout\":\"post\",\"title\":\"Subgradient Descent Explained, Step by Step\",\"author\":[\"Boris Giba\"],\"tags\":[\"Machine Learning Math\"],\"image\":\"img/Subgradient_Descent/SubGradientDescent_Thumbnail2.png\",\"date\":\"2021-07-27T19:49:21.503Z\",\"dateModified\":\"2021-08-22T18:51:14.860Z\",\"changelog\":[[\"2021-07-31T14:28:41.400Z\",\"Added interactive, runnable code blocks.\"],[\"2021-08-21T17:17:59.435Z\",\"Reworked section 'The Fatal Flaw' to more correctly describe the issue of weights not being set exactly to zero. Added quiz questions and programming exercises.\"],[\"2021-08-22T18:51:14.860Z\",\"Added Jupyter notebook.\"]],\"draft\":false,\"imgCaption\":\"Background image by Pontus Wellgraf\",\"imgSource\":\"https://unsplash.com/photos/phdR8e9zOzg\",\"timeToRead\":18,\"uniquePostId\":13,\"excerpt\":\"Gradient descent is one of the most popular algorithms to train machine learning models. However, many of the popular machine learning models like lasso regression or support vector machines contain loss functions that are not differentiable. Because of this, regular gradient descent can not be used. One of the most commonly utilized techniques to circumvent this issue is to use subgradients instead of regular gradients. And in this article, you will learn how it's done.\",\"dependencies\":[],\"proficiencyLevel\":\"Beginner\"};const makeShortcode=name=>function MDXDefaultShortcode(props){console.warn(\"Component \"+name+\" was not imported, exported, or provided by MDXProvider as global scope\");return mdx(\"div\",props);};const NotebookLinks=makeShortcode(\"NotebookLinks\");const ConditionalLazyPlot=makeShortcode(\"ConditionalLazyPlot\");const ToolTip=makeShortcode(\"ToolTip\");const ConditionalLazyPlotInteractive=makeShortcode(\"ConditionalLazyPlotInteractive\");const WarningTip=makeShortcode(\"WarningTip\");const ConditionalLazyCodeInteractive=makeShortcode(\"ConditionalLazyCodeInteractive\");const SmoothScrollLink=makeShortcode(\"SmoothScrollLink\");const AnimationPlayer=makeShortcode(\"AnimationPlayer\");const Reinforcement=makeShortcode(\"Reinforcement\");const BlurredAnswerBox=makeShortcode(\"BlurredAnswerBox\");const layoutProps={_frontmatter};const MDXLayout=\"wrapper\";return function MDXContent(_ref){let{components}=_ref,props=_objectWithoutProperties(_ref,_excluded);return mdx(MDXLayout,_extends({},layoutProps,props,{components:components,mdxType:\"MDXLayout\"}),mdx(NotebookLinks,{ghlink:\"https://github.com/BorisGiba/machinelearningcompassnotebooks/blob/main/notebooks/machine_learning_math/subgradient_descent/Subgradient%20Descent.ipynb\",colablink:\"https://colab.research.google.com/github/BorisGiba/machinelearningcompassnotebooks/blob/main/notebooks/machine_learning_math/subgradient_descent/Subgradient%20Descent.ipynb\",mdxType:\"NotebookLinks\"}),mdx(\"h2\",{\"id\":\"outline\"},`Outline`),mdx(\"p\",null,`You’ve probably heard about gradient descent. Maybe someone even told you that `,mdx(\"em\",{parentName:\"p\"},`this`),` or `,mdx(\"em\",{parentName:\"p\"},`that`),`\nmachine learning model can be trained by using gradient descent. In many cases,\nwhat is actually used under the hood is not regular gradient descent, but `,mdx(\"em\",{parentName:\"p\"},`subgradient descent`),`.\nIn this article, we will explore what subgradients are,\nhow and why we can use them in practice, and what the\npotential dangers of subgradients are.`),mdx(\"h2\",{\"id\":\"prerequisites\"},`Prerequisites`),mdx(\"p\",null,`Before reading this article, you should already be familiar with gradients and gradient descent.\nIf you’re unfamiliar with gradient descent, don’t worry! Gradient descent\nis a lot less scary than it sounds, and it can be used for a variety of machine learning models,\nnot just neural networks. If you’re interested in an introduction to gradient descent, check out\nthe article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent_for_linear_regression\"},`Gradient Descent for Linear Regression Explained, Step by Step`),`.`),mdx(\"p\",null,`In this article, we will apply gradient descent to lasso regression,\nwhich is a slight variation of the popular linear regression algorithm.\nSo if you want to understand that example fully, I recommend you take a look at\n`,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/lasso_regression\"},`Lasso Regression Explained, Step by Step`),`.`),mdx(\"p\",null,`With that being said, let’s get started!`),mdx(\"h2\",{\"id\":\"the-problem\"},`The Problem`),mdx(\"p\",null,`So why do we even need these subgradients? What’s the issue with regular gradients?\nFor this, let’s look at a concrete example: `,mdx(\"em\",{parentName:\"p\"},`lasso regression`),`.\nImagine we have a dataset of figure prices, where each entry in the dataset contains the age of a figure as well as its price for that age in € (or any other currency).\nWe now want to predict the price of a figure given its age using `,mdx(\"em\",{parentName:\"p\"},`lasso regression`),`, to see how much the figures depreciate over time. The dataset looks like this:`),mdx(ConditionalLazyPlot,_extends({},olsVsRidgeData,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1106px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"63%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABS0lEQVQ4y52Tf3OCMAyGPUEFJy2gKIityG+9nXfb9/9u7y6ZRep2N90fOULTPEmaZDKLjnBDjVXaQuY95L6H2HeYxwWfk/0ZMXcnpEyFwjKpGLhKGwRZawHp+5eYe5PF+oQgbeCnPdyogCMVy7OZ/ciQgCJrsUx7LNYlZpF+GWYBqbQw7+HIg2V0X8xyAIb5GUHajgwKXlLD33UvQQfg+vjOwPvDGmD7PyCPS9ZCqCsWm2rUuW8YPYk8ftwdKQg5hxretkGgrlZgbkp0OGMWF+z8GHnO31ujQs3Nowoem2c1RWQdH9jjoq2LFHAa5AiLT4Y6IrdmdJhDU563OSFWFzZQ1t6m5Atvu5r/SZdZx7q/rXjUKAF6/+W2hmMyHGdBzkY3m0JrSJvjJxULnZHNT0oGUmDvFnBYPQM1JY/LoBmN1IWhY7sj9a8lfwFWWzy/hBs7RgAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/040ab5967b5b4f588dc8db80d21003fb/601b1/figurePricesData.webp 500w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/80c60/figurePricesData.webp 1000w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/e6a9b/figurePricesData.webp 1106w\"],\"sizes\":\"(max-width: 1106px) 100vw, 1106px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/040ab5967b5b4f588dc8db80d21003fb/0eb09/figurePricesData.png 500w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/1263b/figurePricesData.png 1000w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/22593/figurePricesData.png 1106w\"],\"sizes\":\"(max-width: 1106px) 100vw, 1106px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/040ab5967b5b4f588dc8db80d21003fb/22593/figurePricesData.png\",\"alt\":\"Figure prices dataset\",\"title\":\"Figure prices dataset\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,`We can now write down our lasso loss function like this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`1`)))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nLassoMSE(y,y_{pred}) = MSE(y,y_{pred}) + \\\\alpha {\\\\color{#26a6ed}||\\\\boldsymbol{\\\\theta}||_1}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#26a6ed\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))))),mdx(\"p\",null,`where `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` is the vector containing all of our model parameters.\nThe term in blue is an L1-penalty, and it can be rewritten like this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"munderover\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"munderover\"},`∑`),mdx(\"mrow\",{parentName:\"munderover\"},mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`1`)),mdx(\"mi\",{parentName:\"munderover\"},`m`)),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nLassoMSE(y,y_{pred}) = MSE(y,y_{pred}) + \\\\alpha \\\\sum_{i=1}^m{{\\\\color{#26a6ed}|\\\\theta_i|}}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.929066em\",\"verticalAlign\":\"-1.277669em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.6513970000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.872331em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel mtight\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.050005em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-symbol large-op\"},`∑`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.3000050000000005em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`m`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.277669em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#26a6ed\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`)))))))),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`{\\\\color{#26a6ed}|\\\\theta_i|}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#26a6ed\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`)))))),` is an application of the absolute function,\nwhich looks like this:`),mdx(ConditionalLazyPlot,_extends({},absoluteFunctionBase,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"971px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"63%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABWElEQVQ4y3WT65KCMAyFmRGKii5SsDDLchNQ0X3/1zs7CRQKdn9kKEn79eRSx4squJcSYf5EUv1CFi9ExQtx+UagOo7RHqtNMdfwOSOwwPHa4pT2OKoOR9UiSDv4cTMD6bs17fcMHwPP+cC37MLCesiXNdyLXaFIWrhJh/DnjVPajcAgu0NkAzzZzGmY5sf1R+qUlbj2EOmDY0LWOuUagWrhyRr+Bjor3AA1zE8f414zZSJTvfiArK1KV0BSQ8rU3ZqNs09u3JCla5PS+MZKFuAEU3c2HVtUl0uXqateVFqh9D8CCwj1sMI+gKRyDhhQkQ7cRT8qR9i1t86eHh0Gcg21QkrJHAlZsdLD93sFIxP/KdTAQ3LDOevZeUiaaV2y/ysfGLaPm3HgqXmywinruRwrIC1ooAlMqeuZovVuUkwXLCPUzCOiS2W+GseUbHtmJsQzfLYzpPAPj9M8T2RtLzcAAAAASUVORK5CYII=')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/9f5e694989c8522c2e921be6798218fd/601b1/absFunctionBase.webp 500w\",\"/static/9f5e694989c8522c2e921be6798218fd/dc04a/absFunctionBase.webp 971w\"],\"sizes\":\"(max-width: 971px) 100vw, 971px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/9f5e694989c8522c2e921be6798218fd/0eb09/absFunctionBase.png 500w\",\"/static/9f5e694989c8522c2e921be6798218fd/5f7f3/absFunctionBase.png 971w\"],\"sizes\":\"(max-width: 971px) 100vw, 971px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/9f5e694989c8522c2e921be6798218fd/5f7f3/absFunctionBase.png\",\"alt\":\"Absolute function\",\"title\":\"Absolute function\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,`To find the optimal value for our model parameters `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` we have to find the minimum\nof the `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`LassoMSE`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68333em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),`. To find the minimum of a function,\nwe usually have to take the first derivative of that function.\nIn our case we have a multivariate function, so we will have to take the gradient of it.\nBut here’s the problem: The L1 penalty inside of our `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`LassoMSE`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68333em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),` is not differentiable at 0!`),mdx(\"p\",null,`The slope of the absolute function is -1 for all of the negative\nvalues, and it is 1 for all of the positive values. But the slope at 0 is not defined!`),mdx(\"p\",null,`Because of this, we can not differentiate our `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`LassoMSE`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68333em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),`, and thus we can not find the model parameters for our\nlasso regression. Or can we?`),mdx(\"h2\",{\"id\":\"part-1-subgradients\"},`Part 1: Subgradients`),mdx(\"p\",null,`One key insight we can make is that the absolute function inside of our L1-penalty is only\nnon-differentiable at zero. So maybe we can still differentiate our function at every point except\n0, and then we’ll just find a clever way to solve the case when we get to the point at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\theta_i = 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`.`),mdx(\"p\",null,`This is the first key insight into subgradients.\nWhat is a subgradient of a function `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` at a point `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`?\nWell, if `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` is differentiable at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`, then the subgradient is just the regular gradient\nof that function at that point `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`.\nOk, so far so good.\nBut now comes the trickier part. What do we do when `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` is not differentiable at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`?`),mdx(\"div\",null,mdx(\"p\",null,`Let’s take a step back and visualize our absolute function again.\nThis time, I’ll also include the gradients for `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`>`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x > 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.5782em\",\"verticalAlign\":\"-0.0391em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`>`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),` and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`<`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x < 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.5782em\",\"verticalAlign\":\"-0.0391em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`<`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`.\nSince we’re dealing with a univariate function the gradient\nwill just be a scalar. So instead of somehow adding this scalar into the plot,\nwhat I will do is I will display the gradient as a linear function where the slope\nof that function is exactly the value of the gradient, and the intercept is 0.\nThis may sound a bit confusing, so let’s jump to the visuals to make things more clear.`,mdx(\"br\",{parentName:\"p\"}),`\n`,`The plot below contains two gradients,\nso feel free to toggle the visibility of the gradients by clicking on the labels\nin the legend at the right part of the plot.`)),mdx(ConditionalLazyPlot,_extends({},absoluteFunctionWithGrad,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"994px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"64%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABkElEQVQ4y5VTy26DMBCMAthJmoJ5GEgivDZJpPaQS4/9//+aam0MSZ/qYcV614xnZu1VVhL+HWrJ07t6qgxWMUmmyDhK42s/xdwvwv5UDVOdsPKFklB0DrlmcEKifmcYf5a9RdZYbJoLZGWxzgesuCkqi317QdEaND1hVwfQKIf7s1RFEK3F5uQgtIWsDej1DefbOzTdFsCtHrEuDGRFqLoQnDOw8GAGWUOQRwfRLQek5YDn4xXa3qBOL3eAzTjJCSBPDXm2hSaI2iLrLeTBQdQUvJvlh3VSDEgKEwBl7bDvLpO8EOnkozpZ9KND3i+1uIfjs78ToMWuPSMtTZDBMjVBHgipDtKVNqg7g21tHvy9H9IMGDw8e+osiYH8BCefuM8g24o8aNkaz84Dz9fogaHDvr9CtAayJ4jm0Scv887fPfvbDci1+Z4hJ5vO4ul4xo6ZKuMPkNN14Zy/PLidHr35bFF1cNhWkV282PGl+CvjIBvn15t56iGP1vAhUVVW2pndg+T4Npen9TX/ux8YfgCV7jxsI0ZjEwAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/dbf4a94d7f9a0353c4ae0ae1307c7b1d/601b1/absSubgradientsPlot.webp 500w\",\"/static/dbf4a94d7f9a0353c4ae0ae1307c7b1d/c8b1a/absSubgradientsPlot.webp 994w\"],\"sizes\":\"(max-width: 994px) 100vw, 994px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/dbf4a94d7f9a0353c4ae0ae1307c7b1d/0eb09/absSubgradientsPlot.png 500w\",\"/static/dbf4a94d7f9a0353c4ae0ae1307c7b1d/d75ff/absSubgradientsPlot.png 994w\"],\"sizes\":\"(max-width: 994px) 100vw, 994px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/dbf4a94d7f9a0353c4ae0ae1307c7b1d/d75ff/absSubgradientsPlot.png\",\"alt\":\"Absolute function gradients\",\"title\":\"Absolute function gradients\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"div\",null,mdx(\"p\",null,`One thing you might notice is that the linear function represented\nby one of the two gradients is always either below or exactly on our main function `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),`.\nWhat’s even better is that this property holds true for all `,mdx(\"em\",{parentName:\"p\"},`convex`),` functions.\nWhat is a convex function? In a nutshell, a convex function is a function that has\nexactly one global minimum`,mdx(ToolTip,{mdxType:\"ToolTip\"},`Technically it can also have no global minimum, but then it would be a pretty boring function.`),` and no global maximum.\nAnd as it turns out, the `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`LassoMSE`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68333em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),` is convex! For a `,mdx(\"em\",{parentName:\"p\"},`not so formal`),` but far more visual\nproof of this, read until the end of this blog post where we will visualize the `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`LassoMSE`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68333em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),`\nin 3D and see that it really has only one global minimum.`)),mdx(\"p\",null,`Ok, so we know that gradients (or rather the linear functions we define around them)\nare so-called `,mdx(\"em\",{parentName:\"p\"},`global underrestimators`),`. Now if we’re going to define a `,mdx(\"em\",{parentName:\"p\"},`subgradient`),` of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f(x) = |x|`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`))))),` at 0,\nthen it should probably look and behave somewhat similar to what a real gradient `,mdx(\"em\",{parentName:\"p\"},`would`),`\nlook like at that point, if it existed in the first place.\nWith this in mind, how could a potential subgradient of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` look like at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x = 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`?`),mdx(\"p\",null,`Since we can’t truly know how a real gradient would look like at that point,\none idea is to look at the gradients in the neighborhood around 0,\nand then define the subgradient as an `,mdx(\"em\",{parentName:\"p\"},`intermediate`),` vector between the gradients\naround our non-differentiable point.`),mdx(\"p\",null,`Right now this still sounds pretty vague, so below is a visualization\nshowing many possibilities for an `,mdx(\"em\",{parentName:\"p\"},`intermediate`),` gradient\nat `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x = 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`. At the bottom of the plot, there’s a slider which controls the value of the subgradient.`),mdx(ConditionalLazyPlot,_extends({},absoluteFunctionSlider,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"992px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"64.2%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABgUlEQVQ4y5WS2W6DMBBFIxUbG6JmMQaysTgmpFFSVV1e+v//dasxJCGQVO3DaAaPdbhzxyM2y+FNM3izHOea/aPun40uTYLeqb1ezXrn3f4NUKY1mDLwpps/KNy4u0xX4MrAj0o8TTbujgPyeQ6pt+DxHlzbiwoK6lE0deF6PLLw0xcwVSJY19DlK1R2RJjYq8JwaeHHBbiuwaLqAuVtnK3gegee1EObBiPHBnJhIJcGYlmDxzW8SXYFTjP4yd4BuxN4PT8vQF+VDsBUAZGWCLIaYnUAmxWNUoJF9qHHA4UiMp3RcnCVQ64qjM0RwfoANi8H3t6DtkspENBS5kUnGoDQJcK1RbDaum2e+74qHgPPH0KV8OnP0w0k5ckagixwkBxBbFyPfiRVR3H3Hd483sg2G9QWfvaG6PCNuHqH3n1ikp/wnJ3gxRVYsmuCfO8r7MqWSQWR1i6P7RdC8wFB8KRyD9/FYu8y3RN6+zuQRqYRZWQgKbc12eDO2nA2uBgq/AH9YD4RBYya6gAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/0e39afed662ab204ef9981e5a134dc1c/601b1/absSubgradientSlider.webp 500w\",\"/static/0e39afed662ab204ef9981e5a134dc1c/50e74/absSubgradientSlider.webp 992w\"],\"sizes\":\"(max-width: 992px) 100vw, 992px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/0e39afed662ab204ef9981e5a134dc1c/0eb09/absSubgradientSlider.png 500w\",\"/static/0e39afed662ab204ef9981e5a134dc1c/43738/absSubgradientSlider.png 992w\"],\"sizes\":\"(max-width: 992px) 100vw, 992px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/0e39afed662ab204ef9981e5a134dc1c/43738/absSubgradientSlider.png\",\"alt\":\"Absolute function subgradient with slider\",\"title\":\"Absolute function subgradient with slider\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,`Notice how the line representing our subgradient turns orange once we cross the initial function,\nmaking the subgradient invalid because it would be greater than the neighborhood gradient,\nwhich is something we don’t want. Conveniently enough this also ensures that our subgradient\nis a global underrestimator of our function, just like a regular gradient would be.\nSo any number between -1 and 1 (including both -1 and 1) is a valid subgradient for our function `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`|x|`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`))))),` at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x = 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`.`),mdx(\"div\",null,mdx(\"p\",null,`That is also why subgradients are called `,mdx(\"em\",{parentName:\"p\"},`sub`),`gradients. The prefix `,mdx(\"em\",{parentName:\"p\"},`sub`),`\ncomes from Latin and means “below”, which makes sense now that we know that\nsubgradients are global underrestimators, or in other words, lie `,mdx(\"em\",{parentName:\"p\"},`below`),` our function.`,mdx(ToolTip,{width:\"15em\",mdxType:\"ToolTip\"},`Similarly,\nyou could define `,mdx(\"em\",{parentName:\"p\"},`supergradients`),`, which are gradient approximations that lie above our function.`))),mdx(\"p\",null,`This is the second crucial insight into subgradients. They can be considered `,mdx(\"em\",{parentName:\"p\"},`intermediate`),` vectors\nof the neighborhood gradients, and they are `,mdx(\"em\",{parentName:\"p\"},`global underrestimators`),` of our function.\nThis also means that there can be an infinite number of subgradients for a function `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` at a\nnon-differentiable point `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`, which is in contrast to regular gradients, which are unique\nto every function and every point.`),mdx(\"p\",null,`And that’s it! That is how subgradients are defined.`),mdx(\"h3\",{\"id\":\"formalizing-subgradients\"},`Formalizing Subgradients`),mdx(\"div\",null,mdx(\"p\",null,`Now let’s try to come up with a mathematical\ndefinition that encapsulates this behavior.\nOur definition might start like this:`,mdx(ToolTip,{width:\"20em\",mdxType:\"ToolTip\"},`Usually `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` is just called `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`, but\nthis can be a bit confusing if the variable of your function is also named just `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`, so for the remainder\nof this article we will stick to `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` to make the two more clearly distinguishable.`))),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mtext\",{parentName:\"mrow\"},`A vector `),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\"},`∈`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mtext\",{parentName:\"mrow\"},` is a subgradient of a function `),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\"},`:`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mo\",{parentName:\"mrow\"},`→`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mtext\",{parentName:\"mrow\"},` at the point `),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mtext\",{parentName:\"mrow\"},` if `),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`.`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`.`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`.`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\text{A vector } g \\\\in \\\\mathbb{R}^d \\\\text{ is a subgradient of a function } \\\\\\\\\nf: \\\\mathbb{R}^d \\\\rightarrow \\\\mathbb{R} \\\\text{ at the point } x_0 \\\\text{ if } \\\\\\\\\n...\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8777699999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`A vector `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`∈`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.093548em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` is a subgradient of a function `))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`:`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8991079999999999em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`→`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` at the point `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` if `))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.10556em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`)))))),mdx(\"p\",null,`So far so good! We’ve seen that subgradients are global underrestimators,\nso let’s try to come up with a mathematical way of writing this property down.\nAt this point, you can pause for a second and think about how you would define this property.`),mdx(\"p\",null,`An initial idea might look similar to this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mtext\",{parentName:\"mrow\"},`A vector `),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\"},`∈`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mtext\",{parentName:\"mrow\"},` is a subgradient of a function `),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\"},`:`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mo\",{parentName:\"mrow\"},`→`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mtext\",{parentName:\"mrow\"},` at the point `),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mtext\",{parentName:\"mstyle\"},` if for all points `),mdx(\"mi\",{parentName:\"mstyle\"},`x`),mdx(\"mtext\",{parentName:\"mstyle\"},` the following holds:`)),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\"},`f`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`x`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mstyle\"},`≥`),mdx(\"mi\",{parentName:\"mstyle\"},`g`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`x`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\text{A vector } g \\\\in \\\\mathbb{R}^d \\\\text{ is a subgradient of a function } \\\\\\\\\nf: \\\\mathbb{R}^d \\\\rightarrow \\\\mathbb{R} \\\\text{ at the point } x_0 {\\\\color{#26a6ed}\\\\text{ if for all points } x \\\\text{ the following holds:}} \\\\\\\\\n{\\\\color{#26a6ed}f(x) \\\\geq  g(x)}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8777699999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`A vector `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`∈`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.093548em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` is a subgradient of a function `))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`:`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8991079999999999em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`→`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` at the point `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},` if for all points `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},` the following holds:`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\",\"color\":\"#26a6ed\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\",\"style\":{\"color\":\"#26a6ed\"}},`≥`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`))))))),mdx(\"p\",null,`This says that the value of our subgradient `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.625em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`))))),` at any point `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` has to be lower than the value of our function\n`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` at that same point `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`.\nThis definition seems intuitive, but let’s see if it really accurately describes what subgradients are.\nFor this, we will look at the absolute function once again.\nLet’s create a plot where we will see the following things:`),mdx(\"ol\",null,mdx(\"li\",{parentName:\"ol\"},`the absolute function `,mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f(x) = |x|`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`)))))),mdx(\"li\",{parentName:\"ol\"},`our subgradient `,mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.625em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`))))),` of `,mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` at the point `,mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0 = 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`)))))),mdx(\"li\",{parentName:\"ol\"},mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f(x)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),` for specific values of `,mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`)))))),mdx(\"li\",{parentName:\"ol\"},mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g(x)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),` for specific values of `,mdx(\"span\",{parentName:\"li\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))))),mdx(\"p\",null,`Since `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` is a variable, we’ll add in a slider to control the value of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`.\nWe’ll also consider just one fixed subgradient (namely 0.2),\nsince, as we have seen before, the subgradient of the absolute function at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0 = 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`\ncan take on any value between -1 and 1. Here’s how that looks like:`),mdx(ConditionalLazyPlot,_extends({},absoluteFunctionSubgradientConditionSlider,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1100px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"63.2%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABu0lEQVQ4y5VT226jMBCtVGyM05Rwv4QEDA4hm0p9qXrZ//+vs/IMUPqy2n048sEan5k5zDyIqIV3OBNWHrXEl+//4Q9/C/K2fJvwR9yJ+XxPgn7cIahuEOlAAd7h3yp0kPkIkVrIdIBKDQtKJ5j1kPmV4LKJzWMZfycgsbCBiDv4xQSZWojE4NC/4qmwc4WJQZD2JCKzCwfGZhV1CVmMk6vyBr+8QaY9vOcTVGVRf3whrKdFkCtcvBCxmR9YThJ/+6WqO/bmHV54ghee4YWcSBUdFcYtRy31v7a1VJaNEA7O09hAFhP84gJVdvBzA1X2BJl0LL78FPIwMXRJ1cwV+VmL4DhCn+/Q7R3B0WLfX5B/vrJgwZBJy0mXsXGlPtUWuh6wayx0Y7E7WQT1AJUb6GqAnxnqhPycH8sZZEd0/jmHftZBlwON0KMzOmGBx+eGuIo78tCPW8iwmXlHWMfJtbzOlPMg6SEyy0j59NIBup5waF9oTkUxQvVvc8zA50Z0synuT43Q1UQCdFYTgvIKVV5Xro+/sJ9+M3dx9Y1GbBXcbsAyjzpjOO5aUkmHXc6b4LiKeLsoZh63RfAPRP0/K5/10rAAAAAASUVORK5CYII=')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/efcecc25924f0b7a1b518c914cc8d27b/601b1/absFunctionPointSlider.webp 500w\",\"/static/efcecc25924f0b7a1b518c914cc8d27b/80c60/absFunctionPointSlider.webp 1000w\",\"/static/efcecc25924f0b7a1b518c914cc8d27b/cb2d6/absFunctionPointSlider.webp 1100w\"],\"sizes\":\"(max-width: 1100px) 100vw, 1100px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/efcecc25924f0b7a1b518c914cc8d27b/0eb09/absFunctionPointSlider.png 500w\",\"/static/efcecc25924f0b7a1b518c914cc8d27b/1263b/absFunctionPointSlider.png 1000w\",\"/static/efcecc25924f0b7a1b518c914cc8d27b/28a3f/absFunctionPointSlider.png 1100w\"],\"sizes\":\"(max-width: 1100px) 100vw, 1100px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/efcecc25924f0b7a1b518c914cc8d27b/28a3f/absFunctionPointSlider.png\",\"alt\":\"Absolute function with point slider\",\"title\":\"Absolute function with point slider\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,`So far so good! We can see that `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g(x)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),` is always greater or equal than `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f(x)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),`.`),mdx(\"h3\",{\"id\":\"refining-our-definition\"},`Refining our Definition`),mdx(\"p\",null,`Now let’s try a more complicated example.\nConsider the following function:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`f`),mdx(\"mn\",{parentName:\"msub\"},`2`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mrow\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"mrow\",\"fence\":\"true\"},`{`),mdx(\"mtable\",{parentName:\"mrow\",\"rowspacing\":\"0.24999999999999992em\",\"columnalign\":\"right left right left\",\"columnspacing\":\"0em 1em 0em\"},mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mn\",{parentName:\"mrow\"},`0.5`),mdx(\"mi\",{parentName:\"mrow\"},`x`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mtext\",{parentName:\"mrow\"},`if `),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`<`),mdx(\"mn\",{parentName:\"mrow\"},`2`))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mn\",{parentName:\"mrow\"},`2`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`3`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mtext\",{parentName:\"mrow\"},`else`)))))))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nf_2(x) = \\\\left \\\\{\n  \\\\begin{aligned}\n    &0.5x && \\\\text{if}\\\\ x \\\\lt 2 \\\\\\\\\n    &2x - 3 && \\\\text{else} \\\\\\\\\n  \\\\end{aligned} \\\\right.\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.10764em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"3.00003em\",\"verticalAlign\":\"-1.25003em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"minner\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen delimcenter\",\"style\":{\"top\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizing size4\"},`{`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mtable\"},mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.7500000000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.75em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.2500000000000002em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.7500000000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.91em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`5`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.41em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`3`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.2500000000000002em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"arraycolsep\",\"style\":{\"width\":\"1em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.7500000000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.75em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.2500000000000002em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.7500000000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.91em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`if`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`<`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.41em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`else`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.2500000000000002em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"}))))))),mdx(\"p\",null,`This function looks like this:`),mdx(ConditionalLazyPlot,_extends({},f2FunctionBase,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1089px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"64%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABNklEQVQ4y5WTb2+DIBDGSSpg9UUdiHZLtChqu2Xf/+vdxvFnaKxZX1xyePDzueeAZFUP9O2GkeavRJbkJIACLOT/iSgi+UZOlx4KZaBozOsK7X45okKuJijbGYiF5PWAkf6V7diwzjtgagbW3N3+2sBZGSCnqoezGlFleoA988u3aEGsWYBW3coqr3BEaFR4AMzbO/D3L2D1hCq3dWJbteos9BCINQ35xzfC6AYWzv4pTIHBQ2yji+az6ydQMewqWwOlH8qlw82oUGhU4oxfgLcPoEIfDMkD8do0Bsp2AlpPwP1hDDXjdWBycIPwP4qDEy5WwLCw0OK6AJMjFsrGQC41KuZCYz0OxvqO6/450B6yRdd2D1wOqAb9FBrX4Wq4tatlCSxem+3z2z5BB7/B7t70Cf4q/AHOkDqLipigvgAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/a28e15fa68c9a6420f7e3e5ac7f2a606/601b1/f2Base.webp 500w\",\"/static/a28e15fa68c9a6420f7e3e5ac7f2a606/80c60/f2Base.webp 1000w\",\"/static/a28e15fa68c9a6420f7e3e5ac7f2a606/1a410/f2Base.webp 1089w\"],\"sizes\":\"(max-width: 1089px) 100vw, 1089px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/a28e15fa68c9a6420f7e3e5ac7f2a606/0eb09/f2Base.png 500w\",\"/static/a28e15fa68c9a6420f7e3e5ac7f2a606/1263b/f2Base.png 1000w\",\"/static/a28e15fa68c9a6420f7e3e5ac7f2a606/97810/f2Base.png 1089w\"],\"sizes\":\"(max-width: 1089px) 100vw, 1089px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/a28e15fa68c9a6420f7e3e5ac7f2a606/97810/f2Base.png\",\"alt\":\"f2 function\",\"title\":\"f2 function\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`f`),mdx(\"mn\",{parentName:\"msub\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f_2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.10764em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` is not differentiable at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`2`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0 = 2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`))))),`, so let’s\nadd in a subgradient at that particular point. The slope of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`f`),mdx(\"mn\",{parentName:\"msub\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f_2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.10764em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` is `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mfrac\",{parentName:\"mrow\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mn\",{parentName:\"mfrac\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\frac{1}{2}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.190108em\",\"verticalAlign\":\"-0.345em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.845108em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.6550000000000002em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.394em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.345em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})))))),`\nfor every point less than 2, and 2 for every point after that.\nSince we can think of a subgradient as the `,mdx(\"em\",{parentName:\"p\"},`intermediate`),` vector\nbetween the gradients around it, a valid subgradient in this particular case `,mdx(\"em\",{parentName:\"p\"},`should`),` be 1.\nSo let’s add the subgradient (once again represented as a linear function, we’ll call it `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`g`),mdx(\"mn\",{parentName:\"msub\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g_2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.625em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`) to our plot\nand see how it looks like:`),mdx(ConditionalLazyPlot,_extends({},f2SubgradientPlotZero,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1097px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"63.800000000000004%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABSUlEQVQ4y42T2W6DMBBF8xC8QFVSQ2hDABuUBEj7/793K49tBZNFebhgjUdHdxZvtrsOyZcmhfMy9lAv7jcBsIaFmJPGOo+gUY7Txn6YMhBF/7bD7a51ZzWAlQb8u0eieuLcgGUMZC+Aohoh6j/I5gxZD0gKg6yd8HmcbkC5Hyg5gJha9827y1vI5orMzOCVobgz0sUOZdlHgAioNFhhwAqNVE9Iu8n3cNkGHfdQPipZWUcd2N4gv47I+hH8MJPLJSyacoCItUNfHt8bpM0Jor6AVdNTUODEwLAKeQde9lRaqq/gPxNYeUaizIvJL4C25NQORRnw6oy0nfEx/EIcLrQOtuFO+q4Kd+7ugVk9g1UjjV8eTxDKgNGd9jLgNmb/hWuP3V0L5X6Ht8uSKckDqGSb7EFh6gSkXEOuAjj8acrRM3v69N6Rm/I//TY7lXfLb30AAAAASUVORK5CYII=')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/d89df7d51a1f50c493873b31b1d87c86/601b1/f2BaseSubgradient.webp 500w\",\"/static/d89df7d51a1f50c493873b31b1d87c86/80c60/f2BaseSubgradient.webp 1000w\",\"/static/d89df7d51a1f50c493873b31b1d87c86/9186d/f2BaseSubgradient.webp 1097w\"],\"sizes\":\"(max-width: 1097px) 100vw, 1097px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/d89df7d51a1f50c493873b31b1d87c86/0eb09/f2BaseSubgradient.png 500w\",\"/static/d89df7d51a1f50c493873b31b1d87c86/1263b/f2BaseSubgradient.png 1000w\",\"/static/d89df7d51a1f50c493873b31b1d87c86/c37c4/f2BaseSubgradient.png 1097w\"],\"sizes\":\"(max-width: 1097px) 100vw, 1097px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/d89df7d51a1f50c493873b31b1d87c86/c37c4/f2BaseSubgradient.png\",\"alt\":\"f2 function with subgradient at x0 = 0\",\"title\":\"f2 function with subgradient at x0 = 0\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,`Well, that sure does not look as expected… our subgradient is at the wrong position!\nWe want our subgradient to go through the point `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mn\",{parentName:\"mrow\"},`2`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"mn\",{parentName:\"mrow\"},`1`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`(2, 1)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),`, but right now this isn’t the case.\nSo what can we do? We can move our entire subgradient by `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`2`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0 = 2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`))))),` to the right.\nMaybe you also remember from calculus that to move a function to the right by some amount,\nwe counterintuitively have to `,mdx(\"em\",{parentName:\"p\"},`subtract`),` that amount from our function.\nSo instead of calculating `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g(x)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),` we will now calculate `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g(x - x_0)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),`, which in our case will be\n`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mn\",{parentName:\"mrow\"},`2`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`2`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g2(x - 2)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),`. Now we have successfully shifted our function on the x-axis, but we still have to shift\nit on the y-axis. To do this we simply add `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f(x_0)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),` to the right-hand side of our subgradient condition.\nIn our particular case that would be `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mn\",{parentName:\"mrow\"},`2`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mn\",{parentName:\"mrow\"},`2`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`1`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f2(2) = 1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`))))),`.`),mdx(\"p\",null,`Our subgradient definition now looks like this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mtext\",{parentName:\"mrow\"},`A vector `),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\"},`∈`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mtext\",{parentName:\"mrow\"},` is a subgradient of a function `),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\"},`:`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mo\",{parentName:\"mrow\"},`→`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mtext\",{parentName:\"mrow\"},` at the point `),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mtext\",{parentName:\"mrow\"},` if for all points `),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mtext\",{parentName:\"mrow\"},` the following holds:`),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`≥`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\"},`f`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\text{A vector } g \\\\in \\\\mathbb{R}^d \\\\text{ is a subgradient of a function } \\\\\\\\\nf: \\\\mathbb{R}^d \\\\rightarrow \\\\mathbb{R} \\\\text{ at the point } x_0 \\\\text{ if for all points } x \\\\text{ the following holds:} \\\\\\\\\nf(x) \\\\geq  {\\\\color{#26a6ed}f(x_0)} + g(x {\\\\color{#26a6ed} - x_0})\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8777699999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`A vector `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`∈`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.093548em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` is a subgradient of a function `))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`:`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8991079999999999em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`→`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` at the point `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` if for all points `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` the following holds:`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`≥`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\",\"color\":\"#26a6ed\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`)))))),mdx(\"p\",null,`Ok, that seems reasonable. Let’s also visualize it to see if it actually does what we expect.\nBelow is an interactive visualization where you can visualize the two steps we have just performed\nby pressing the button at the top of the plot.`),mdx(ConditionalLazyPlotInteractive,_extends({},f2SubgradientPlotInteractive,{mdxType:\"ConditionalLazyPlotInteractive\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1083px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"64.99999999999999%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABY0lEQVQ4y4WT3XqCMAyGOaC0hW0qiOJQ2oIozu3+L+/bk5ZqQecOQv/SN1/SEMXLBmylrPl5uPfcpnfi4CxiM0g4ent17gP4dUQfnmtrfyl8XB/Acg22bkeQAssNklyNwMJArs0ESIfzVOOPBvFCQWwG8OrLgVYKSWnwrq542/YOmOQaogiA5BQC6WKhkTYdxM4g634gPnvIuoWsOySltv7EGRXqUeEdMleYkE9tIPYDMjMg00dkzfGh3s8V0t7sVeMF1a2D3F8gd8YGiBcN2Ky+t0choFOm3ejbgy4sNUTVI2u/ISpzK0OYhRczUeiiOUhSkIojeDUg1Vek6mIBVtWT3pwASaEsW7DCgG9O4NuzNVmfkR568FKDUcovGn0KLAzSakCyPUNserCVRlq1kLvWpbVU1ocyoTU9ohvvZXpImdtHOYDnLqJ1Durka0vGx7kHUgCfdjT/jcIWiOfrf4wU/gIf7TuhkEzuAgAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/c96428548448650806e12df28643ccf4/601b1/f2SubgradientAnimation.webp 500w\",\"/static/c96428548448650806e12df28643ccf4/80c60/f2SubgradientAnimation.webp 1000w\",\"/static/c96428548448650806e12df28643ccf4/fea0c/f2SubgradientAnimation.webp 1083w\"],\"sizes\":\"(max-width: 1083px) 100vw, 1083px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/c96428548448650806e12df28643ccf4/0eb09/f2SubgradientAnimation.png 500w\",\"/static/c96428548448650806e12df28643ccf4/1263b/f2SubgradientAnimation.png 1000w\",\"/static/c96428548448650806e12df28643ccf4/41289/f2SubgradientAnimation.png 1083w\"],\"sizes\":\"(max-width: 1083px) 100vw, 1083px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/c96428548448650806e12df28643ccf4/41289/f2SubgradientAnimation.png\",\"alt\":\"f2 function subgradient animation\",\"title\":\"f2 function subgradient animation\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"div\",null,mdx(\"p\",null,`Much better! Now let’s look at the actual definition of subgradients and see how it compares to our own.\nHere goes:`,mdx(ToolTip,{width:\"25em\",mdxType:\"ToolTip\"},`In literature, `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` is usually called `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`z`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`z`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.04398em\"}},`z`))))),`. So often times the definition looks like this: `,mdx(\"br\",null),mdx(\"br\",null),`\nA vector `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\"},`∈`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`g \\\\in \\\\mathbb{R}^d`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.7335400000000001em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`∈`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.849108em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.849108em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.063em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))))))),` is a subgradient of a function `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\"},`:`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mo\",{parentName:\"mrow\"},`→`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"double-struck\"},`R`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f: \\\\mathbb{R}^d \\\\rightarrow \\\\mathbb{R}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`:`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.849108em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.849108em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.063em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`→`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68889em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)))))),` at the point `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` if for all points `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`z`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`z`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.04398em\"}},`z`))))),` the following holds: `,mdx(\"br\",null),`\n`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`z`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`≥`),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\"},`g`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`z`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f(z) \\\\geq f(x) + g^T(z - x)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.04398em\"}},`z`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`≥`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.0913309999999998em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8413309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.063em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.04398em\"}},`z`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),`.`,mdx(\"br\",null),mdx(\"br\",null),`\nPersonally, I find it a lot more intuitive to call `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`z`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`z`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.04398em\"}},`z`))))),` `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` and to call `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`\nsince this (at least for me) indicates that `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` lie on the same scale\nand that `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),` is really a variable whereas `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` is just a single, fixed point.\nThat is why I chose this version for this article, but I still wanted to show you this variant as well,\nsince it is most common in literature.`))),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mtext\",{parentName:\"mrow\"},`A vector `),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mo\",{parentName:\"mrow\"},`∈`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mtext\",{parentName:\"mrow\"},` is a subgradient of a function `),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\"},`:`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mi\",{parentName:\"msup\"},`d`)),mdx(\"mo\",{parentName:\"mrow\"},`→`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"double-struck\"},`R`),mdx(\"mtext\",{parentName:\"mrow\"},` at the point `),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mtext\",{parentName:\"mrow\"},` if for all points `),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mtext\",{parentName:\"mrow\"},` the following holds:`),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`≥`),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\"},`g`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\text{A vector } g \\\\in \\\\mathbb{R}^d \\\\text{ is a subgradient of a function } \\\\\\\\\nf: \\\\mathbb{R}^d \\\\rightarrow \\\\mathbb{R} \\\\text{ at the point } x_0 \\\\text{ if for all points } x \\\\text{ the following holds:} \\\\\\\\\nf(x) \\\\geq f(x_0) + g^T(x - x_0)\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8777699999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`A vector `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`∈`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.093548em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` is a subgradient of a function `))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`:`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8991079999999999em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8991079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`→`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbb\"},`R`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` at the point `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` if for all points `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` the following holds:`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`≥`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.1413309999999999em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`)))))),mdx(\"p\",null,`Well if that doesn’t look familiar! The definition is pretty much exactly the same.\nThe only difference is that the subgradient is transposed.\nThis is necessary since our subgradient might live in more than just one dimension\nlike in our example. Because of this, we have to transpose it to make the multiplication work.\nApart from that our definition is exact, nice!`),mdx(\"p\",null,`Now we understand where subgradients come from, why they’re useful and how they are\ndefined. But you’re also probably interested in when they are actually used\nin practice. To answer this question, let’s look at `,mdx(\"em\",{parentName:\"p\"},`subgradient descent`),`!`),mdx(\"h2\",{\"id\":\"part-2-subgradient-descent\"},`Part 2: Subgradient Descent`),mdx(\"p\",null,`Million-dollar question: what’s the difference between subgradient descent and gradient descent?\nThat’s right, subgradient descent uses a subgradient instead of a regular gradient!\nThat is the only difference between the two.\nHowever, subgradient descent might not work completely as expected in some cases,\nso it’s definitely worthwhile to implement subgradient descent and take a look\nat some potential weaknesses.\nFor the implementation, we’ll reuse the code that we wrote in the article\n`,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent_for_linear_regression\"},`Gradient Descent for Linear Regression Explained, Step by Step`),`.\nHere’s how the code looks like:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def gradient_descent(X, y, theta, criterion, gradient_function, number_of_iterations, learning_rate):\n    X_b = add_intercept_ones(X)\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta) # create the current function\n        y_predicted = f(X_b) # predict our entire x\n        loss = criterion(y,y_predicted) # calculate the error\n\n        # perform optimization\n        gradient = np.array( gradient_function(...) ) # calculate gradient\n        theta = theta - learning_rate * gradient #adjust m and b\n\n    return theta\n`)),mdx(\"p\",null,mdx(\"inlineCode\",{parentName:\"p\"},`add_intercept_ones`),` is a little helper function that adds an additional column of ones to\nour `,mdx(\"inlineCode\",{parentName:\"p\"},`X`),` to make calculations with the bias (or intercept) easier. You can find it’s\ndefinition `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent_for_linear_regression/#the-code\"},`here`),`.\n`,mdx(\"inlineCode\",{parentName:\"p\"},`create_function`),` is another small helper, you can find it’s definition `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent_for_linear_regression/#making-predictions\"},`here`),`.\nNow all we need to do is rename our `,mdx(\"inlineCode\",{parentName:\"p\"},`gradient`),` to `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient`),`, like this:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\",\"metastring\":\"{1,12}\",\"{1,12}\":true},`def subgradient_descent(X, y, theta, criterion, subgradient_function, number_of_iterations, learning_rate):\n    X_b = add_intercept_ones(X)\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta)  # create the current function\n        y_predicted = f(X_b)  # predict our entire x\n        loss = criterion(y, y_predicted)  # calculate the error\n\n        # perform optimization\n        subgradient = np.array(subgradient_function(...))  # calculate gradient\n        theta = theta - learning_rate * subgradient  # adjust m and b\n\n    return theta\n`)),mdx(\"p\",null,`Technically, you could even keep the old implementation altogether, since we only changed up\nvariable names. However this way I think it’s a bit easier to know just what exactly we’re doing.\nNow that we have our general `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function defined, let’s look at a concrete example:\n`,mdx(\"em\",{parentName:\"p\"},`lasso regression`),`.`),mdx(\"h3\",{\"id\":\"example-implementation-lasso-regression\"},`Example Implementation: Lasso Regression`),mdx(\"p\",null,`To recap, here’s how the loss-function of lasso looks like:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`1`)))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nLassoMSE(y, y_{pred}, \\\\boldsymbol{\\\\theta}) = MSE(y,y_{pred}) + \\\\alpha {\\\\color{#26a6ed}||\\\\boldsymbol{\\\\theta}||_1}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#26a6ed\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))))),mdx(\"div\",null,mdx(\"p\",null,`From our observations above we figured out that the subgradient of the absolute function at `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x = 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`\ncan be any number between -1 and 1, including -1 and 1.\nAnd for every other x, our subgradient is just the regular gradient, i.e. `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`1`),mdx(\"mtext\",{parentName:\"mrow\"},` for `),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`<`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`-1 \\\\text{ for } x < 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.77777em\",\"verticalAlign\":\"-0.08333em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` for `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`<`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`\nand `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mn\",{parentName:\"mrow\"},`1`),mdx(\"mtext\",{parentName:\"mrow\"},` for `),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`>`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`+1 \\\\text{ for } x > 0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.77777em\",\"verticalAlign\":\"-0.08333em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},` for `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`>`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),`. We can write this down like so`,mdx(ToolTip,{width:\"20em\",mdxType:\"ToolTip\"},`Usually the `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\"},mdx(\"mover\",{parentName:\"mo\"},mdx(\"mo\",{parentName:\"mover\"}),mdx(\"mrow\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`u`),mdx(\"mi\",{parentName:\"mrow\"},`b`))))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\stackrel{sub}{}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.7861079999999999em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.7861079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.7em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop\"}))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.9000000000000004em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`u`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`))))))))))))),` at the top of the gradient-symbol will be ommited,\nhowever I added it in here to make it clear that we’re dealing with a subgradient.`),`:`)),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msub\"},mdx(\"mover\",{parentName:\"mo\"},mdx(\"mo\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mo\",\"mathvariant\":\"normal\"},`∇`)),mdx(\"mrow\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`u`),mdx(\"mi\",{parentName:\"mrow\"},`b`)))),mdx(\"mi\",{parentName:\"msub\"},`g`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mrow\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"mrow\",\"fence\":\"true\"},`{`),mdx(\"mtable\",{parentName:\"mrow\",\"rowspacing\":\"0.24999999999999992em\",\"columnalign\":\"right left right left\",\"columnspacing\":\"0em 1em 0em\"},mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mn\",{parentName:\"mrow\"},`1`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mtext\",{parentName:\"mrow\"},`if `),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`>`),mdx(\"mn\",{parentName:\"mrow\"},`0`))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\"},`x`),mdx(\"mo\",{parentName:\"mstyle\"},`∈`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`[`),mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"mn\",{parentName:\"mstyle\"},`1`),mdx(\"mo\",{parentName:\"mstyle\",\"separator\":\"true\"},`,`),mdx(\"mn\",{parentName:\"mstyle\"},`1`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`]`))))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mtext\",{parentName:\"mstyle\"},`if `),mdx(\"mi\",{parentName:\"mstyle\"},`x`),mdx(\"mo\",{parentName:\"mstyle\"},`=`),mdx(\"mn\",{parentName:\"mstyle\"},`0`)))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`1`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mtext\",{parentName:\"mrow\"},`if `),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`<`),mdx(\"mn\",{parentName:\"mrow\"},`0`)))))))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\stackrel{sub}{\\\\nabla} _g(x) = \\\\left \\\\{\n  \\\\begin{aligned}\n    &1 && \\\\text{if}\\\\ x>0 \\\\\\\\\n    &{\\\\color{#26a6ed}x \\\\in [-1,1]} && {\\\\color{#26a6ed}\\\\text{if}\\\\ x=0} \\\\\\\\\n    &-1 && \\\\text{if}\\\\ x<0\n  \\\\end{aligned} \\\\right.\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.6638810000000002em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.3777730000000001em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.908335em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop\"},`∇`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.791665em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`u`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.091665em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15139200000000003em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"4.500000000000002em\",\"verticalAlign\":\"-2.000000000000001em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"minner\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizing mult\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.35002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.19999em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizinginner delim-size4\"},mdx(\"span\",{parentName:\"span\"},`⎩`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.19499em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizinginner delim-size4\"},mdx(\"span\",{parentName:\"span\"},`⎪`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.20499em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizinginner delim-size4\"},mdx(\"span\",{parentName:\"span\"},`⎪`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.15001em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizinginner delim-size4\"},mdx(\"span\",{parentName:\"span\"},`⎨`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.2950099999999996em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizinginner delim-size4\"},mdx(\"span\",{parentName:\"span\"},`⎪`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.30501em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizinginner delim-size4\"},mdx(\"span\",{parentName:\"span\"},`⎪`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.60002em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"delimsizinginner delim-size4\"},mdx(\"span\",{parentName:\"span\"},`⎧`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.8500199999999998em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mtable\"},mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.5000000000000004em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.5em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.4999999999999991em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.000000000000001em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.5000000000000004em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.66em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.16em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\",\"style\":{\"color\":\"#26a6ed\"}},`∈`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`[`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\",\"style\":{\"color\":\"#26a6ed\"}},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`]`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.6599999999999993em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.000000000000001em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"arraycolsep\",\"style\":{\"width\":\"1em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.5000000000000004em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.5em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.4999999999999991em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.84em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.000000000000001em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.5000000000000004em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.66em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`if`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`>`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.16em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`if`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"color\":\"#26a6ed\"}},` `)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\",\"style\":{\"color\":\"#26a6ed\"}},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`0`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.6599999999999993em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`if`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`<`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.000000000000001em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"}))))))),mdx(\"p\",null,`Does this function look familiar to you?\nThis looks pretty similar to the sign of the number `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`x`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`))))),`. Any strictly positive number has sign 1, any strictly negative number has sign -1,\nand for 0 we can just define that the sign is 0, which, conveniently, is a number between -1 and 1, and\neven more conveniently is what `,mdx(\"inlineCode\",{parentName:\"p\"},`np.sign(0)`),` outputs.\nWith this we can now use the sign of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` as our subgradient, like so:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msub\"},mdx(\"mover\",{parentName:\"mo\"},mdx(\"mo\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mo\",\"mathvariant\":\"normal\"},`∇`)),mdx(\"mrow\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`u`),mdx(\"mi\",{parentName:\"mrow\"},`b`)))),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`))),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∇`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#54C667\"},mdx(\"mi\",{parentName:\"mstyle\"},`s`),mdx(\"mi\",{parentName:\"mstyle\"},`i`),mdx(\"mi\",{parentName:\"mstyle\"},`g`),mdx(\"mi\",{parentName:\"mstyle\"},`n`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\stackrel{sub}{\\\\nabla}_{LassoMSE} = \\\\nabla(MSE(y,y_{pred})) + \\\\alpha \\\\cdot {\\\\color{#54C667} sign(\\\\boldsymbol{\\\\theta})}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.527773em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.3777730000000001em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.908335em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop\"},`∇`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.791665em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`u`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.091665em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.32833099999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.36687em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∇`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.44445em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#54C667\"}},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#54C667\"}},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#54C667\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#54C667\"}},`n`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#54C667\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#54C667\"}},`)`))))))),mdx(\"p\",null,`And that’s it! We have “fixed” our lasso-MSE and now it’s time to perform subgradient descent!`),mdx(\"h3\",{\"id\":\"implementing-subgradient-descent-for-lasso\"},`Implementing Subgradient Descent for Lasso`),mdx(\"p\",null,`The only thing we have to implement now are our loss and (sub)gradient functions.\nIn the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/ridge_regression\"},`Ridge Regression Explained, Step by Step`),`\nwe’ve implemented these functions for ridge regression:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def get_ridge_mse_function(alpha=0.0001):\n\n    def ridge_mse(y, y_predicted, theta):\n        mse_loss = mse(y, y_predicted)\n        ridge_loss = mse_loss + alpha * np.dot(theta, theta)\n        return ridge_loss\n\n    return ridge_mse\n\ndef get_ridge_gradient_function(alpha=0.0001):\n\n    def ridge_gradient(X_b, y, y_pred, theta):\n        return  -(2/y.size) * X_b.T.dot(y - y_pred) + alpha * theta\n\n    return ridge_gradient\n`)),mdx(\"p\",null,`We can slightly modify this code to get our functions for lasso.\nInstead of writing two new functions that return us our gradient and loss functions,\nwe’ll modify our existing ones to accept parameters `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha1`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha2`),` (or `,mdx(\"inlineCode\",{parentName:\"p\"},`a1`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`a2`),` for short)\nthat will control the lasso and ridge penalties respectively. With this,\nwe technically implemented functions not for lasso, but for `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/elastic_net\"},`elastic-net`),` regression.\nHowever, since elastic-net is just a linear combination of ridge and lasso,\nthis works out just fine.\nWith this we have:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def get_elastic_mse_function(a2=1, a1=0):\n\n    def elastic_mse(y, y_predicted, theta):\n        error = y - y_predicted\n        return (\n            1 / (y.size) * np.dot(error.T, error) # mse\n            + a2 * np.dot(theta, theta)           # l2-penalty\n            + a1 * np.sum(np.abs(theta))          # l1-penalty\n        )\n\n    return elastic_mse\n\n\ndef get_elastic_gradient_function(a2=1, a1=0):\n\n    def elastic_gradient(X, y, y_pred, theta):\n        return (\n            -(2 / y.size) * X.T.dot(y - y_pred) # gradient of mse\n            + 2 * a2 * theta                    # gradient of l2-penalty\n            + a1 * np.sign(theta)               # subgradient of-l1 penalty\n        )\n\n    return elastic_gradient\n`)),mdx(\"p\",null,`By calling `,mdx(\"inlineCode\",{parentName:\"p\"},`get_elastic_mse_function`),` with `,mdx(\"inlineCode\",{parentName:\"p\"},`a1=0`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`a2=0`),`, we get the regular, unregularized MSE function.\nWith `,mdx(\"inlineCode\",{parentName:\"p\"},`a2=0`),` we get the ridgeMSE and with `,mdx(\"inlineCode\",{parentName:\"p\"},`a1=0`),` we get the lassoMSE. The same goes for our gradient function,\nwhere we can just combine the penalties to prevent us from having to write three separate functions.\nWith the loss and gradient now defined, we can use our `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function like this:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`subgradient_descent(X_train, y_train, theta,\n                    get_elastic_mse_function(a2=0, a1=1),\n                    get_elastic_gradient_function(a2=0, a1=1),\n                    number_of_iterations, learning_rate)\n`)),mdx(WarningTip,{mdxType:\"WarningTip\"},mdx(\"p\",null,`Here it is important to note that our `,mdx(\"inlineCode\",{parentName:\"p\"},`X`),` has to be standardized,\nsince we are using a regularized model.\nIf you’re unsure as to why this is truly necessary or what standardization\nreally is, I recommend you read `,mdx(\"a\",{parentName:\"p\",\"href\":\"dataset_optimization/standardization\"},`Standardization Explained, Step by Step`),`.\nFor now we will assume that our `,mdx(\"inlineCode\",{parentName:\"p\"},`X`),` was already standardized, to make the code more readable.`),mdx(\"p\",null,`Also note that we are only using a portion of the dataset for our subgradient descent,\na so-called `,mdx(\"em\",{parentName:\"p\"},`training`),` subset. You can read more about how to split your dataset correctly\nin the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/dataset_optimization/split_data_the_right_way/\"},`How to Split Your Dataset the Right Way`),`.\nFor this article, we’ll already assume that our dataset has been split properly.`)),mdx(\"p\",null,`So let’s run lasso regression and see how it goes!`),mdx(ConditionalLazyCodeInteractive,{codeSetup:setupCodeSubgradientDescent,hasPrompt:false,hasSolution:false,hasPlot:false,hasTests:false,packages:['numpy'],mdxType:\"ConditionalLazyCodeInteractive\"},mdx(\"span\",null),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`theta_lasso_subgd = subgradient_descent(\n    X_train_s, # standardized!\n    y_train,\n    np.random.rand(2), # initialize theta as random vector\n    get_elastic_mse_function(a2=0, a1=1),\n    get_elastic_gradient_function(a2=0, a1=1),\n    1200,\n    0.01,\n)\nprint(theta_lasso_subgd)\n# output: [40.75       -2.74229857]\n`))),mdx(\"p\",null,`Alright, that looks good! If we train a regular OLS regression model then our model parameters will look more\nlike this: `,mdx(\"inlineCode\",{parentName:\"p\"},`[41.25 -3.242]`),`. So lasso did make our parameters a bit smaller, nice!\nWe can plot the two models and we get:`),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"368px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"67.3913043478261%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAACYElEQVQ4y32Sa0/iQBSGS1sUFWgplHagnZnCchcFA1m8onjBddcPm/3/v+XZTEVjsmY/nJz2zMyb93IsNdsiphui8QOHZYFVTbCrbZxd/1yOl2AdRrSnN6TLF7LJOaPlA+lohV0RtEfnWJ3lK1F/RSmMiVoZYaDZ9xVOTVH8ohw/Q2QdwnZMMrrMwURvSUUMaA1WWOnplmZvhRsoii2FJzUiUngG1JM4NfkB5voSt6YIkyFes0N3tkYdXzJcPvIt/77G6ix+EQ/Ocb2UomeYpZSyIfL2D1FTceArbE/i+ilFA+hLvKhHIPrU0jF1eUxDn+AnI4RhmJw85QwdA2iY5I80rhhQThRCauqhotjo4/iGqcTxTaXYuc8JhWqCW4mp6zmWXrzSm17QTdsUvHevJMVqkkt2G4pAZnTXv6mKPnY13Xmpc/l7gcLyNJ1EoPtzrOR0i+ivOAxSXP+LIHyZSy6FilhqmrGm6CvKvqBUS3NQc89sQaDPsLrLV8TwAvtdsjH/UwgfM09i+5JKQxFFisX5d8qiR6GSshfoHLBuAE3KUW9F0U+/XJN/1+aNdTi5RnQHHMUSu6axK23q2RmWPnvJUzZLayS7uyT/X4rCkeDAT4lbmlBlOHUjeb4D7K/YDxR7O5NNmWD2A/3W6zpnZ/73dvdKjWxnjSZoaMJYEnYXRvITveVP4uEV6vSe9mRNa3xDNn8kGlzSmT/lXc8eEKPr/FydbD5mpqvZI2JwhRivsaLRLXK2JR5vkLNn2tNHWscPGObR+O6tj+5Q8x/Ek01+ls62+UzOnz+d3RMO1/wFVw1YpUrFbEAAAAAASUVORK5CYII=')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/2c3668ba00ecb540593db116a99181c0/ef070/lassoSubGDVsOLS.webp 368w\"],\"sizes\":\"(max-width: 368px) 100vw, 368px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/2c3668ba00ecb540593db116a99181c0/55493/lassoSubGDVsOLS.png 368w\"],\"sizes\":\"(max-width: 368px) 100vw, 368px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/2c3668ba00ecb540593db116a99181c0/55493/lassoSubGDVsOLS.png\",\"alt\":\"Lasso compared to OLS\",\"title\":\"Lasso compared to OLS\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `)),mdx(\"p\",null,`The lasso function is pretty similar to the regular linear regression function,\njust with slightly smaller coefficients. If we were to increase the L1-penalty coefficient,\nwe would get an even more strongly regularized result.`),mdx(\"h3\",{\"id\":\"the-fatal-flaw\"},`The Fatal Flaw`),mdx(\"p\",null,`The main property of lasso is that lasso is able to produce sparse model weights.\nIn other words, lasso is able to set weights of unhelpful model parameters all the way to zero,\nunlike ridge regression for example. Since we’ve just implemented lasso regression,\nwe should check whether our implementation zeroes out unnecessary weights,\nas we expect it to.\nWe can do this by creating a random dataset (a random X and a random y)\nand then running lasso regression on it. Since the dataset is completely random,\nthere should be no correlation between X and y, and thus lasso should set all of our\nmodel parameters to exactly zero.\nLet’s give it a try:`),mdx(ConditionalLazyCodeInteractive,{codeSetup:setupCodeSubgradientDescent,hasPrompt:false,hasSolution:false,hasPlot:false,hasTests:false,packages:['numpy'],mdxType:\"ConditionalLazyCodeInteractive\"},mdx(\"span\",null),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`theta_rand = np.random.rand(51)\nX_rand = np.random.rand(100, 50)\ny_rand = np.random.rand(100)\ntheta_rand_lasso_subgd = subgradient_descent(\n    X_rand,\n    y_rand,\n    theta_rand,\n    get_elastic_mse_function(a2=0, a1=1),\n    get_elastic_gradient_function(a2=0, a1=1),\n    3000,\n    0.01,\n)\nprint(theta_rand_lasso_subgd)\n# output similar to:\n# [ 0.01036144  0.00646848  0.00116589  0.00647136 -0.00524321  0.00290512\n#   0.00664798  0.00808131  0.00937785  0.00165406 -0.00257609  0.00490855\n#  -0.00133889 -0.00582042  0.00832424  0.00403818  0.00799567  0.00062623\n#   0.01316767 -0.0043727   0.01226832 -0.0052027   0.00092482  0.00231925\n#   0.01305558  0.00906722  0.00154304  0.00505857  0.0087923   0.00346944\n#   0.00158308  0.0112326  -0.00631912 -0.0001088  -0.00211522  0.01203972\n#   0.00760454  0.01046682  0.00350387 -0.00540258  0.00655244  0.00841913\n#   0.00930862  0.00478768 -0.00475235  0.01260794  0.01329395 -0.00388777\n#   0.00920832  0.00485297  0.00296131]\n`))),mdx(\"p\",null,`Now I don’t know about you, but those values are not exactly zero. I mean sure, they’re close.\nBut lasso should be able to set them to exactly zero, and not just close to it.\nThat’s why we’re using it in the first place, right?\nEven if we were to increase `,mdx(\"inlineCode\",{parentName:\"p\"},`a1`),` and run subgradient descent again,\nthe parameters will still not be exactly zero.\nWhy is that so? Did we do something wrong? Surely there has to be a mistake in our code.\nSurprisingly, no. Everything is working as intended.`),mdx(\"div\",null,mdx(\"p\",null,`When I first encountered this behavior, I did not know what caused the issue.\nSo naturally, I started digging deeper and I also performed some experiments.\nThe most common answer that I found online was that the problem lies in the subgradient.\nSince the subgradient is only an approximation of the real gradient`,mdx(ToolTip,{mdxType:\"ToolTip\"},`At least sort-of, since\nthe real gradient doesn’t exist at non-differentiable points.`),`, it has some\nintrinsic noise to it, and that noise is what causes the model parameters\nto not be exactly zeroed out. But at least in this case, this explanation is actually `,mdx(\"em\",{parentName:\"p\"},`incorrect`),`.\nWhy? To answer this question we must first more precisely formulate it.\n`,mdx(\"em\",{parentName:\"p\"},`If`),` the usage of a subgradient was the underlying issue, then the non-differentiable\npoints of that subgradient would need to be hit rather often. In other terms,\nif the subgradient of the L1-penalty was what causes this issue, then the subgradient of that\nL1-penalty at the point x = 0 would have to be computed rather often in our subgradient descent-algorithm.\nThis is because if that particular non-differentiable point is `,mdx(\"em\",{parentName:\"p\"},`not`),` hit frequently,\nthen our subgradient can’t be the underlying issue because for every differentiable point\nour subgradient is just the regular gradient.`)),mdx(\"p\",null,`Now the question is, how often is this point at x = 0 hit?\nIf you want to find this out for yourself, I recommend you\ntake a look at the second programming exercise of the `,mdx(SmoothScrollLink,{scrollid:\"reinforcementDiv\",mdxType:\"SmoothScrollLink\"},`Reinforcement`),`-section\nbecause\nthat is exactly the purpose of that exercise.\nIf you’re still reading, then you probably want to find out the answer.\nThe answer is that the point at x = 0 is pretty much never hit.\nWhy? The L1-penalty of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` is the sum of the absolute\nentries of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`, so for this term to be 0,\nevery entry in `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` has to be 0 as well. Or, in other terms:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`1`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"munderover\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"munderover\"},`∑`),mdx(\"mrow\",{parentName:\"munderover\"},mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`1`)),mdx(\"mi\",{parentName:\"munderover\"},`n`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`),mdx(\"mtext\",{parentName:\"mrow\"},`  `),mdx(\"mo\",{parentName:\"mrow\"},`⟺`),mdx(\"mtext\",{parentName:\"mrow\"},`  `),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∀`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mo\",{parentName:\"mrow\"},`:`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`0`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n||\\\\boldsymbol{\\\\theta}||_1 = \\\\sum^n_{i=1}|{\\\\theta_i}| = 0 \\\\iff \\\\forall i: \\\\theta_i = 0\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.929066em\",\"verticalAlign\":\"-1.277669em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.6513970000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.872331em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel mtight\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.050005em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-symbol large-op\"},`∑`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.3000050000000005em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.277669em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.66844em\",\"verticalAlign\":\"-0.024em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`⟺`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∀`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`:`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`)))))),mdx(\"h4\",{\"id\":\"the-fatal-flaw-of\"},`The Fatal Flaw of…?`),mdx(\"div\",null,mdx(\"p\",null,`So if the subgradient is not the problem, then what is?\nThe problem actually lies in gradient descent `,mdx(\"em\",{parentName:\"p\"},`itself`),`.\nThe thing is, gradient descent in itself only approximates the optimal solution.\nIn some cases, that approximation happens to be exactly the optimal solution.\nHowever, in our case, it is not. Why?\nSince there is no correlation between our randomized features,\nthe optimal solution would contain only zeros, right? This means that as we get closer and closer\nto our optimal solution, our model parameters would get closer and closer to zero,\nbut this also means that our gradient would shrink more and more! And with a smaller gradient,\ngradient descent will perform smaller and smaller updates. This means that we are\nasymptotically approaching the optimal solution and making smaller and smaller steps at\nevery iteration. So to actually reach the exact optimal solution we would have to perform\nan infinite number of gradient descent iterations`,mdx(ToolTip,{width:\"20em\",mdxType:\"ToolTip\"},`Or find exactly the right set\nof hyperparameters for which gradient descent does immediately converge to our optimum,\nwhich is as difficult as winning the lottery.`),`.`)),mdx(\"p\",null,`We can see this even better when we visualize the values of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` over time.\nTo do this, let’s generate a random dataset with only two entries, because visualizing\n50-dimensional data is a bit tricky.\nWe could either do this in 3D, where the first two dimensions would be the first and second\nvalue of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`, and the third dimension would be the `,mdx(\"em\",{parentName:\"p\"},`lassoMSE`),` for our randomized\ndataset. We will do this later when we visualize subgradient descent on our actual dataset,\nbut for now a `,mdx(\"em\",{parentName:\"p\"},`filled contour plot`),` (or contourf plot) would be a better choice. A filled contour plot is a 2D plot where the\nbackground color is the third dimension. So in our example, the `,mdx(\"em\",{parentName:\"p\"},`lassoMSE`),` would be represented by\nthe background color of the plot. If we track the value of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` at every\niteration and plot it (in red) alongside our contourf plot, we get this:`),mdx(AnimationPlayer,{path:\"../content/machine_learning_math/img/Subgradient_Descent/anim/convergence/subgd_frame_\",ending:\".png\",caption:\"Subgradient descent for lasso converging\",identifier:\"subGDconvergence\",numframes:1,mdxType:\"AnimationPlayer\"},mdx(\"img\",{id:\"full\",src:\"/13187de067ce72d097b1cf1c763a4e8b/subgradient_convergence.webp\"}),mdx(\"span\",{\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"432px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"66.66666666666666%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAADaUlEQVQ4yzWUWVPbZRSH0yxsgSwWlOwBgZYS4M+SECiEhLAmARKgLEIihYBSULBULVBbsNAOWyDLPywZHR1nOl7YcXTG8dJLb/wS3vg1HidNe3HmnXMunvM7v/O+r0ShsSDXmpEVG5EW6FAU6ZEW6pAV6ZAWlSMt0iF/VyvUc0Np4IbSiERpQFJgRJJnQpJnQaa2oNCakSjesyAvNvF+dzd3vvoU0/ICFdElahcecGtxBUs0SuXKErfWl6j5IoJzZYqOB8P4N0a598TL58cunh4LlFmrkJSYkMi1FpSaCqqjH9P4+zUN31/Qks7gOM1gj2VoTWcQvrtCeHVF868i7leHDFw/5qPXO6z9s8nJv/P89t84prpqJEWmrEIrCqUe08Q0wtUlwn4cx5Mk7Q8TODcTOLYS2HeTNL9MIpwlsWVSNP4Yp+mXU9r+OMD311Oifz+i9HZtzoqswkKNharxCM3Hlzi3ktxdT+KJJnAvJuheTdG5kaL96xRtT1PYX4g0n4kIlynqf4jT9DqG588j1DU2pFlfFVoLeSojlYFZ7u5l6FiN03s/QV8oxsBYjL6pc3rCcdzRBK61FJ2PRNq2UzgP0thjaWxiAvtPIsoP65EVvwOqjdQMhencvMQ7H2dwLMZo7yF+1wv82XP4lKGxM/pnzvHOJ3CvJHE9FOnYEnHup3HGLlBWNOSA2ZELtGaqembpW72mfzzGSO8hwZZnhJq+IWjfJeDYI9B1QOAt3DcZp2f2HO9yiu4NEdfOBSprI9ISfVahFZlKT703zEAkTWDgiFHHLsHKDXzlK4xY1wnd/pKxhm1Gsw3an7+Bjwyd4AvF6JuJ078sojK/Bco1FgpLrdS7I/inUwQ9L5kSdvBY13Ar5/DKp+lVhvGVLTFs/ox7tsfMdnzLoG2bQccuU6FzhifOKNbZkKkMuWsjVZZT4wsT3PuZYPSamTGRadchk817uWh9zoRjnxnXEZPuY0KeEyYCCUaCSXxTIv7FK1TGRqQqfc7Dkg+qqXT5uTMeRQgsYXOFcXYtUi9MI7TOIbTM0t61QEt7BHvXPK3d96nrmKO2M0KdJ4J95BPyy6qRq7MvRWMmu2llWRWyYj2FN61IS3Tkl1qQaQzklZqRaQ0UlOVyudaIXJMNA/k3zSjUBvKyf4Ha9IbzP6Pc23HFXhI+AAAAAElFTkSuQmCC')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/83637e62fbda88ddbd3a19dcf41bd089/e3a5e/subgd_frame_0.webp 432w\"],\"sizes\":\"(max-width: 432px) 100vw, 432px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/83637e62fbda88ddbd3a19dcf41bd089/dc3b1/subgd_frame_0.png 432w\"],\"sizes\":\"(max-width: 432px) 100vw, 432px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/83637e62fbda88ddbd3a19dcf41bd089/dc3b1/subgd_frame_0.png\",\"alt\":\"subgd frame 0\",\"title\":\"subgd frame 0\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `)),mdx(\"p\",null,`The dark region in the middle is where our loss is minimal.\nAs we can see the point (0,0) lies exactly in that region.\nNow it might look like we pretty quickly reach our optimal point (0,0),\nbut if we zoom in we will be able to better understand what really is going on:`),mdx(AnimationPlayer,{path:\"../content/machine_learning_math/img/Subgradient_Descent/anim/zoom/subgd_frame_\",ending:\".png\",caption:\"Zooming into (sub)gradient descent\",identifier:\"subGDzoom\",numframes:1,mdxType:\"AnimationPlayer\"},mdx(\"img\",{id:\"full\",src:\"/a94b7e8011ada3e900e3da9edfd96c53/subgradient_descent_zoom.webp\"}),mdx(\"span\",{\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"432px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"66.66666666666666%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAADb0lEQVQ4yzWUSVMaBhiGCYuoKEvVNLJr1WgEBVcwiguKCriA4jIIiQom1kmtSWON1TSuk4gSFgGXaWc6nenk0kPaQy+dybGX3nvtqX/j6VCbw3v4vsMz77cKJEoDYpUeUYkWYWElkmI1wqJKRMWVCIvvICyuRPwxV6RGWKjmllSNQKpBINUiKNAhKDAgUhiQqPQIJJ8YEJfouN3by73Nx+hWlqiKRGlY+py7y6sYIhGqV6PcXY9SuxGmbSuMfTeAZ9/H3NEgX8QcvIxZqTDWICjVIRCrDMiUVdRGHtD8yxVN3+dozV7TcXZNe/yatuw11u8usf50QfP7HK2/nzP04ZiZP79l/a/nxP5+yPt/ptA11iIo1uUdGpHI1Oim57BcXmA5TNKxk8b+NIXtWYrOrSStexmaTrLY3pzh3t9l9GSHlncxOn49wvPhJZE/vqK8voFbMu2NwyKlgZqpMC2xC2wv0txfTzMQSdG3nML+5ILBtVPmIzvMru7i3I1hTWRpvsxg/iGJ9ec4/b+doKgzIZRpEEhUBgrkWqq9Qe7vXdO1lmRwMYVzKoHLFycy+jVRzybe8Ald6znsmzk6t9PYjrK0x7OYMinaf8wg+8yMqOQjUKGlbjRE97MLnA+TuPwJZgcO2DeFWbRvMjKRwDWdZHgugfNBkr7HaRxPM3S9yGA7zGKL55BVNd0A8yUXqvTUDAQZWrtixH/K0HCC46oAG/UruOyvGe94xVj3Ed7BN3jGznDPJBkIJnCunOP4MoPjmxxyYzPCUnXeoRGRXI3ZGWI4nGV8+DWuzmMSGi+h22Hcxg389c/xN20zYd1l0n6At+eI8dFT3L44w8EUrkcZ5Pr/gWKlgaJyI+a+MJ65c3z9x/gtuywaFnAXz9AnnmdIFsJdEWVM/4SAeYtg1z4jpm1GOl4RmHiLd/otJZUmRHLNzdoIZXeoc4eY3HvHZOSKeX+GQO8JgZY9ZvJqO2C645D5fK4vhq//lOmxFOO+NO7ZDJ7lS+TaZoRy9U0PSz+tpdrh4d5UBIs3iskRwt6zhNkyh6VtAUtbELtjiVZ7mPaeRdp6F2nsWqChO0xjf5j28UdIK2oRK/KXotSTn7SsogZRiZqiMiPC0kqk5QZESg0FZXpEKg2FFTexWKVFrMxLg7RMj0ShoSD/CxS6/zj/Ahpu2+IXvuIJAAAAAElFTkSuQmCC')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/a1a412cbabf823c11c9df2f35d441fa1/e3a5e/subgd_frame_0.webp 432w\"],\"sizes\":\"(max-width: 432px) 100vw, 432px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/a1a412cbabf823c11c9df2f35d441fa1/dc3b1/subgd_frame_0.png 432w\"],\"sizes\":\"(max-width: 432px) 100vw, 432px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/a1a412cbabf823c11c9df2f35d441fa1/dc3b1/subgd_frame_0.png\",\"alt\":\"subgd frame 0\",\"title\":\"subgd frame 0\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `)),mdx(\"p\",null,`Our `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` jumps around our optimal point, but it never reaches it exactly.\nGradient descent is often explained as a traveler climbing down a hill,\nbut here I think it makes sense to think of gradient descent as a game of golf,\nwhere `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` is our golfball and the hole is our optimal solution.\nPutting the ball exactly into the hole is difficult as it requires good aim and just the right\namount of force. But here, instead of a hole, the optimal point is just a marking on the ground.\nIf putting a golf ball into a hole is difficult, then placing it onto a specific point\non the ground is nearly impossible. That is why our (sub)gradient descent is hopping around\nthe optimal point. It’s simply trying its best to get as close as possible to the optimum,\nbut the ball keeps rolling just a bit further than it has to.`),mdx(\"p\",null,`This is important because gradient descent (not stochastic gradient descent, but regular gradient descent)\nis oftentimes presented as an `,mdx(\"em\",{parentName:\"p\"},`optimal solution finder`),`, as I like to call it, even if it is a costly one.\nBut this is simply not true as we see here. Usually, gradient descent finds a solution that\nis so close to the optimal one, that the difference is completely negligible. But if you are looking\nfor some very specific values, like in the case of lasso, then this is an important fact to keep in mind,\nbecause it might explain some of the struggles you might encounter, similar to the ones presented in this article.`),mdx(\"p\",null,`So now that we know about this issue, how can we solve it?`),mdx(\"h4\",{\"id\":\"fixing-subgradient-descent-for-lasso\"},`Fixing (Sub)gradient Descent for Lasso`),mdx(\"p\",null,`This issue can actually be fixed rather easily.\nThe crucial part is that we know our weights will be set very close to zero,\neven though they will not go all the way to zero.\nSo what we can do is we can add another parameter to our `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function\ncalled `,mdx(\"inlineCode\",{parentName:\"p\"},`epsilon`),`, or `,mdx(\"inlineCode\",{parentName:\"p\"},`eps`),` for short. Then, at the end of an iteration of subgradient descent,\nwe will set all weights that are less than our `,mdx(\"inlineCode\",{parentName:\"p\"},`eps`),` to zero. So all we have to do is add this one line\nto our `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-algorithm:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\",\"metastring\":\"{9,23}\",\"{9,23}\":true},`def subgradient_descent(\n    X,\n    y,\n    theta,\n    criterion,\n    subgradient_function,\n    number_of_iterations,\n    learning_rate,\n    eps,\n):\n    X_b = add_intercept_ones(X)\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta)  # create the current function\n        y_predicted = f(X_b)  # predict our entire x\n        loss = criterion(y, y_predicted)  # calculate the error\n\n        # perform optimization\n        subgradient = np.array(subgradient_function(...))  # calculate gradient\n        theta = theta - learning_rate * subgradient  # adjust m and b\n\n        theta[theta < eps] = 0\n\n    return theta\n`)),mdx(\"p\",null,`If we now run our subgradient descent on our random dataset again we get the following output:`),mdx(ConditionalLazyCodeInteractive,{codeSetup:setupCodeSubgradientDescentEps,hasPrompt:false,hasSolution:false,hasPlot:false,hasTests:false,packages:['numpy'],mdxType:\"ConditionalLazyCodeInteractive\"},mdx(\"span\",null),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`theta_rand = np.random.rand(51)\nX_rand = np.random.rand(100, 50)\ny_rand = np.random.rand(100)\n\ntheta_rand_lasso_subgd = subgradient_descent(\n    X_rand,\n    y_rand,\n    theta_rand,\n    get_elastic_mse_function(a2=0, a1=1),\n    get_elastic_gradient_function(a2=0, a1=1),\n    3000,\n    0.01,\n    0.01,\n)\nprint(theta_rand_lasso_subgd)\n# [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n#  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n#  0. 0. 0.]\n`))),mdx(\"p\",null,`Awesome!`),mdx(\"h3\",{\"id\":\"visualizing-subgradient-descent\"},`Visualizing Subgradient Descent`),mdx(\"p\",null,`Now that we’ve fully implemented subgradient descent for lasso, let’s also visualize it!\nBefore we used a filled contour plot to visualize how subgradient descent converges\nfor a randomized dataset. Now we will do the same for our actual dataset of figure prices,\nbut this time, we will use a 3D plot instead of a contour plot.\nThe visualization will be similar to `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/ridge_regression/#visualizing-gradient-descent\"},`the one we created for ridge regression`),`.\nAt the top of the plot, there are buttons to control the current iteration\nof subgradient descent. Here’s how our algorithm performs:`),mdx(ConditionalLazyPlotInteractive,_extends({},lassoSubGDHistPlot,{mdxType:\"ConditionalLazyPlotInteractive\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"991px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"80.4%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAQCAYAAAAWGF8bAAAACXBIWXMAABYlAAAWJQFJUiTwAAADgUlEQVQ4y22U309bdRTALy3t7e1Kb3t729v2tr0/2tJCoRtQkCljZFaWJUYzmIpTyAIkbCpT10G6uWXTxWnUBeePYTI31Ama+GBm9gCJizH6oA/7B/xDfP2YeyvC4h5Ovufc8z2fc/I951zBFzVwpD2Sc2Xb3v72KH3b9isGQjBNqX+Y6vABBCmJ8KiL2+IEtIUzeMLZ//kcO6BaCEKExaXTrKx+gODXEHZn8yvmf2AxZuGRs+idfaQKe119t9+JaQGjzL/xKu9cu4LQHm9V6FwU9qQRgimkuI2UsF2gENR5/8bbPDd7HEFKuz4nsRgz3ZgWUGF6cY7zH76F4EsiuLCAxmOHnqI2WkfwaQiBFL5wlslXZlj7/RavXWsiqXYrmWoTUFsJW8AYk6emWbzaQBDTDtDAK+eo7h/jz7+2uLp2hWONk8yvXKLx402Of7XK81/fJPdEnfZQGilRQIrnCcbzLWBbjCfnZ5i6cKb1hl45S9zqRQikaXy0zC9//8HlB1tMb96jfvtb+j69Q88nG+Qb77kAt8J4wa1QdICeGLXpE4wvL+MNJBH2JArIehkpZpOtDtL87Scmfv6V/vX7lK/fxbr0HanGHTLN74kePokYTiEpOSTVwue8s1ehODXL0JvnEIMOUCuQ730cxeihLZBg6EyT0c0HGCv3yTXvUZr/BvuFNazJ21jTGyTrZwnlhxG1Ej7FpM2nkJ6Yo7RwDl/g3y7rnQOoZi+SYtJh9lK6fhfj4hb21Drlw19SGf2Mwv7PKY6sUnx6A/uldULFA7TLGTyiinJkFvPEefwBdWdsInoZ0eleKIlafxlraZPi+C0Kgx9T6LtArtok03OR4tANUn2vI4Q0fIqFNxBDPjiDPrGEPxjfGWznzJYHiRk9SKqBduQsXc/+gFF9l9LAAuNjk8QrDfRKg5BWQTW6XaAnqBIdeJFsfRF/R2Jn9ZxuRzNdFPeNIKfL7vAmR05THPsCu3aZQq1BomuBoFbF6K6hd9bwRgw8oThK1zGM4VP4Ze3hCts6dBem2XsRFWdwTeTOQ6SqcyjWUaR4mYRVQbP34Qln8EVNPOEEUfMZcr2z+KPJFnD3fjpdd6CZco1gohMpZiJG00hqhkyp3/U5d1pFmHjlJB16HdU+ii+S2gFuV+mcTnYn0JlPf8xZszyy3kU4XXZ9D//WMohqN3614ur/AOxc4nXn/3FvAAAAAElFTkSuQmCC')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/91d222c4dbf886da021a525e81d2adb3/601b1/lassoHist3d.webp 500w\",\"/static/91d222c4dbf886da021a525e81d2adb3/6e312/lassoHist3d.webp 991w\"],\"sizes\":\"(max-width: 991px) 100vw, 991px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/91d222c4dbf886da021a525e81d2adb3/0eb09/lassoHist3d.png 500w\",\"/static/91d222c4dbf886da021a525e81d2adb3/f79d8/lassoHist3d.png 991w\"],\"sizes\":\"(max-width: 991px) 100vw, 991px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/91d222c4dbf886da021a525e81d2adb3/f79d8/lassoHist3d.png\",\"alt\":\"History of subgradient descent for lasso visualized\",\"title\":\"History of subgradient descent for lasso visualized\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,`Nothing too special here, it looks mostly like regular gradient descent,\nand it also looks quite similar to the visualization we looked at earlier.\nBut if we were to zoom in `,mdx(\"em\",{parentName:\"p\"},`reeally`),` deeply (deeper than this visualization allows us to),\nwe would still see some jittering and imprecision, especially towards the end.`),mdx(\"p\",null,`Here we can also intuitively see that the `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`LassoMSE`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68333em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),` is convex because the surface plot\nrepresenting the `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`LassoMSE`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68333em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`))))),` only has one minimum. This isn’t a mathematically rigorous proof,\nbut for the intuition I think it conveys the message quite nicely.`),mdx(\"h2\",{\"id\":\"further-reading\"},`Further Reading`),mdx(\"p\",null,`Congrats on finishing this article!\nHopefully, this article helped you understand what subgradients are,\nwhen and why they are used, as well as their strengths and weaknesses.`),mdx(\"p\",null,`Subgradient descent is only one way to solve lasso regression, and there is in fact an\n`,mdx(\"em\",{parentName:\"p\"},`even better`),` way to solve lasso. It’s called `,mdx(\"em\",{parentName:\"p\"},`coordinate descent`),` and is extremely fast\n(convergence can often occur after just 2-3 epochs!). If you want to learn more about\nthis fascinating algorithm and why it is so blazingly fast, I recommend you take a look\nat the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/coordinate_descent\"},`Coordinate Descent Explained, Step by Step`),`,\nwhere you will learn everything you need to know about coordinate descent.`),mdx(\"p\",null,`Maybe you have noticed that the L1-penalty can be written both as a sum\nas well as a vector operation. This is true not only for the L1-penalty but for most\nfunctions in machine learning! However, one of the two is `,mdx(\"em\",{parentName:\"p\"},`clearly`),` better than the other\nwhen it comes to performance. If you want to know which of the two variants you should prefer\nwhen implementing machine learning models, as well as why you should do so,\ncheck out the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/vectorization\"},`Vectorization Explained, Step by Step`),`.`),mdx(Reinforcement,{elementString:\"qa,qa,mc\",mdxType:\"Reinforcement\"},mdx(\"div\",{name:\"quiz_main\"},mdx(\"div\",{name:\"quiz_question_wrapper\"},mdx(\"p\",{className:\"QA\",name:\"quiz_question\"},mdx(\"p\",null,`Briefly explain the main idea behind subgradients.`)),mdx(\"p\",null,`Subgradients can be interpreted as “intermediate” vectors\nbetween regular gradients. They only differ from regular gradients\nat non-differentiable points of a function. Any global underrestimator\nat that non-differentiable point is a valid subgradient.\nThe main purpose of subgradients is to provide a gradient-like object\nat points where an exact gradient can not be computed. This is useful\nfor algorithms like gradient descent, which rely on gradient computation.`)),mdx(\"div\",{name:\"quiz_question_wrapper\"},mdx(\"p\",{className:\"QA\",name:\"quiz_question\"},mdx(\"p\",null,`What’s the difference between subgradients and supergradients?`)),mdx(\"p\",null,`Subgradients are global underrestimators whereas supergradients are\nglobal overrestimators. Subgradients make a lot of sense for convex\nfunctions because they only have (at most) one global minimum.\nSimilarly, supergradients can be useful in the case of concave functions`,mdx(ToolTip,{mdxType:\"ToolTip\"},`A function `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` is concave\nwhen `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mi\",{parentName:\"mrow\"},`f`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`-f`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`))))),` is convex.`),`, because concave functions have at most one maximum.`),mdx(\"p\",null,`We can see why this is the case when we visualize the functions `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`|x|`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`))))),` as well as\n`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`-|x|`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`))))),` and add subgradients and supergradients into the mix:`),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1080px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"33.4%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsTAAALEwEAmpwYAAAA90lEQVQoz3VSa0/EIBC8xEJpjdcHFMxpOUppPKPx//+7Mbsc1fPxYdhkWWZnJhyq3kOOAbVe/oUYzlzv3YZmWnN/zFAmojaRZ4jrQEdrE9SUuEnk39HYjevD6QU6fKB/foXUAcpFKLtwbV26JaQtkpSMCzcLhF64T0pY2RQy3IraJggdcNfNu4udkIZZXSHtPKQmGxRFjqMhgsHzfVlMM9U1Dpr9RVh1MyRlQmpshNAe0oRdIT2ipbuD4czkfyhMOZOCR7K34Ph0wejfOZLWbqjN+iPj7Kp12xeh6LNkCpZUKLNC6cj1eLpgmN/yLzDxeh9vUN4Uwk9gd6v1blt8sgAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/1b9a258700cdf17b6c34f1487db7cd12/601b1/subAndSupergradients.webp 500w\",\"/static/1b9a258700cdf17b6c34f1487db7cd12/80c60/subAndSupergradients.webp 1000w\",\"/static/1b9a258700cdf17b6c34f1487db7cd12/5c666/subAndSupergradients.webp 1080w\"],\"sizes\":\"(max-width: 1080px) 100vw, 1080px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/1b9a258700cdf17b6c34f1487db7cd12/0eb09/subAndSupergradients.png 500w\",\"/static/1b9a258700cdf17b6c34f1487db7cd12/1263b/subAndSupergradients.png 1000w\",\"/static/1b9a258700cdf17b6c34f1487db7cd12/b4ad3/subAndSupergradients.png 1080w\"],\"sizes\":\"(max-width: 1080px) 100vw, 1080px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/1b9a258700cdf17b6c34f1487db7cd12/b4ad3/subAndSupergradients.png\",\"alt\":\"Sub- and supergradients\",\"title\":\"Sub- and supergradients\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `)),mdx(\"p\",null,`One could imagine that the supergradient for `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`|x|`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`))))),` would look rather complicated,\nas would the subgradient for `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`-|x|`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`))))),`. Since convex functions are usually more frequently\nused in machine learning, subgradients are also more frequently used than supergradients.`)),mdx(\"div\",{name:\"quiz_question_wrapper\"},mdx(\"p\",{className:\"MC\",name:\"quiz_question\"},\"What is the reason that subgradient descent for lasso generally does not produce sparse model weights?\"),mdx(\"div\",{name:\"quiz_question_option\"},mdx(\"div\",{name:\"quiz_question_answer\"},\"This is because of the inherent noise present in subgradients. They are only approximations, and thus not as \\\"precise\\\" as regular gradients.\"),mdx(\"div\",{name:\"quiz_question_solution\",isCorrect:\"false\"},mdx(\"p\",null,`Even though subgradients `,mdx(\"em\",{parentName:\"p\"},`are`),` only approximations of regular gradients,\noftentimes the points that make subgradients different from regular gradients\nare very rarely, if ever, hit. We saw this at the end of section `,mdx(SmoothScrollLink,{scrollid:\"the-fatal-flaw\",mdxType:\"SmoothScrollLink\"},`The Fatal Flaw`),`\nand you can also explore this fact further in programming exercise #2.`),mdx(\"p\",null,`So in essence, in most use cases, subgradients are treated just as regular gradients,\nbecause the very few points in which they differ are oftentimes not extremely relevant.`))),mdx(\"div\",{name:\"quiz_question_option\"},mdx(\"div\",{name:\"quiz_question_answer\"},\"This is because of gradient descent itself. Since gradient descent only approximates the optimal solution, tiny errors like these can occur.\"),mdx(\"div\",{name:\"quiz_question_solution\",isCorrect:\"true\"},mdx(\"p\",null,`Gradient descent only finds approximations of optimal solutions.\nAnd although these approximations are often times extremely precise,\nthey can still differ from the optimal solution.\nWe saw this in the section `,mdx(SmoothScrollLink,{scrollid:\"the-fatal-flaw-of\",mdxType:\"SmoothScrollLink\"},`The Fatal Flaw of…?`),`\nwhere we quite literally took a closer look at gradient descent and saw that it does in fact\nnot reach the exact, optimal solution, even though it gets extremely close.`))))),mdx(\"div\",{name:\"programming exercises\"},mdx(\"div\",null,mdx(\"strong\",null,mdx(\"p\",null,`How fast is subgradient descent?`)),mdx(\"p\",null,`In this exercise, we want to compare regular gradient descent\nwith subgradient descent and see if subgradient descent\nis noticeably slower than regular gradient descent.\nSince we can’t directly compare the two algorithms because\ntheir use cases are non-overlapping, we’ll compare gradient descent\nfor ridge regression with subgradient descent for lasso regression.\nThe only difference between the two is that ridge uses an L2-penalty (`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`||\\\\boldsymbol{\\\\theta}||_2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`)\nwhereas lasso uses an L1-penalty (`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`1`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`||\\\\boldsymbol{\\\\theta}||_1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`).\nWhat do we expect to see?\nIf we use the same dataset as well as the same learning rate and\ninitial `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`, ridge and lasso should converge similarly quickly.\nLet’s try this out and see how it goes!`),mdx(\"div\",null,mdx(\"p\",null,`To make our results a little bit more solid,\nwe’ll use a slightly larger dataset than the one presented in the article.`,mdx(ToolTip,{mdxType:\"ToolTip\"},`The original\ndataset is actually contained in this larger one. Can you spot it?`),`\nIt looks like this:`)),mdx(ConditionalLazyPlot,_extends({},polyOlsRidge,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"891px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"70.8%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB7ElEQVQ4y5VT2XLaUAx1AsErtvFufAHvG2UawiQp0+SlD8100rf+/7ecjkTx4DZNpw/iWhfp6OhcSbpZbDC115g5KZz1FrboYCcdn1pQ8n8U8y87x0n0M7FWfGHGNcy4gR5WfCpePgTS+Z6dYyQCs5YtjKiCEjYMznYR+F8M6UP1cmh+gbnYQXZzGMseU2a9eh+IilqrMeC1KbhVS/SwNrd8uTp8gx53UPwSE0tgSkYJTjokT+YJFL+AGjajwtI8qpnd1Eog9i8onn5A3L3AKR8Q9M/Q4hZGsj2Bmwkn68se5voj36tBNQb00h2cVQ8jrDjJKe7h1UdmSBLY6R6bh+9wq0focYuJKRjQrT9BixqoYT3SWlqIjl9TdjNcz5fw2yeu7Lef4TVHZkUA6/tXiP1XRLsv7BOInR0QdM9cZAAkDYmhSa8cVAgIqD4yW3NzOzwQ6euWj+zP3Ax2esdd3FyMzDCHZLNfgstegXmy5QBirMUdjOQDroyYfWIns56CC82c7I05TFrYScvVqH0CpAJk5J8kSSE7KetGPgGoQYlFdhhreN4UnsOoBkmgeBn0sOR71c/Zv6Ix8XIYYXliRm0n7YjdMNi/r9a5Gm0LbdBC9GwE/Ef8BRi3/NZ6XQbQXhMT2m2ahL+t45nhT2oxXmqYTTYCAAAAAElFTkSuQmCC')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/601b1/figurePricesDataset2.webp 500w\",\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/3e9ea/figurePricesDataset2.webp 891w\"],\"sizes\":\"(max-width: 891px) 100vw, 891px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/0eb09/figurePricesDataset2.png 500w\",\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/c7d76/figurePricesDataset2.png 891w\"],\"sizes\":\"(max-width: 891px) 100vw, 891px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/c7d76/figurePricesDataset2.png\",\"alt\":\"Figure prices dataset (large)\",\"title\":\"Figure prices dataset (large)\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `))),mdx(\"p\",null,`Your task is to modify the existing `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function\nsuch that it now returns not only the optimized parameter vector `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`,\nbut also the history of the training loss. This history can be a list which contains\nthe loss of every training iteration. Use the plotting feature to compare the histories\nof ridge and lasso.`),mdx(\"p\",null,mdx(\"em\",{parentName:\"p\"},`Bonus task`),`: Create a validation set by splitting the `,mdx(\"inlineCode\",{parentName:\"p\"},`X`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`y`),` that are passed into\nthe `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function. Then, in each epoch, compute the validation loss\nin addition to the training loss. Return both loss histories at the end of the function.\nNote: call the variables holding the validation losses `,mdx(\"inlineCode\",{parentName:\"p\"},`loss_hist_val_ridge`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`loss_hist_val_lasso`),`.`),mdx(\"p\",null,mdx(ConditionalLazyCodeInteractive,{hasPrompt:true,startFullscreen:true,codeSetup:setupCodeSubgradientDescentReinforcement,hasSolution:true,hasPlot:true,hasTests:true,plotDivName:\"plotSubGD\",testDivId:\"testsSubGD\",packages:['numpy','matplotlib'],mdxType:\"ConditionalLazyCodeInteractive\"},mdx(\"div\",null,mdx(\"strong\",null,mdx(\"p\",null,`How fast is subgradient descent?`)),mdx(\"p\",null,mdx(\"p\",null,`In this exercise, we want to compare regular gradient descent\nwith subgradient descent and see if subgradient descent\nis noticeably slower than regular gradient descent.\nSince we can’t directly compare the two algorithms because\ntheir use cases are non-overlapping, we’ll compare gradient descent\nfor ridge regression with subgradient descent for lasso regression.\nThe only difference between the two is that ridge uses an L2-penalty (`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`||\\\\boldsymbol{\\\\theta}||_2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`)\nwhereas lasso uses an L1-penalty (`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`1`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`||\\\\boldsymbol{\\\\theta}||_1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`).\nWhat do we expect to see?\nIf we use the same dataset as well as the same learning rate and\ninitial `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`, ridge and lasso should converge similarly quickly.\nLet’s try this out and see how it goes!`)),mdx(\"div\",null,mdx(\"p\",null,`To make our results a little bit more solid,\nwe’ll use a slightly larger dataset than the one presented in the article.`,mdx(ToolTip,{mdxType:\"ToolTip\"},`The original\ndataset is actually contained in this larger one. Can you spot it?`),`\nIt looks like this:`)),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"891px\"}},`\n      `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"70.8%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB7ElEQVQ4y5VT2XLaUAx1AsErtvFufAHvG2UawiQp0+SlD8100rf+/7ecjkTx4DZNpw/iWhfp6OhcSbpZbDC115g5KZz1FrboYCcdn1pQ8n8U8y87x0n0M7FWfGHGNcy4gR5WfCpePgTS+Z6dYyQCs5YtjKiCEjYMznYR+F8M6UP1cmh+gbnYQXZzGMseU2a9eh+IilqrMeC1KbhVS/SwNrd8uTp8gx53UPwSE0tgSkYJTjokT+YJFL+AGjajwtI8qpnd1Eog9i8onn5A3L3AKR8Q9M/Q4hZGsj2Bmwkn68se5voj36tBNQb00h2cVQ8jrDjJKe7h1UdmSBLY6R6bh+9wq0focYuJKRjQrT9BixqoYT3SWlqIjl9TdjNcz5fw2yeu7Lef4TVHZkUA6/tXiP1XRLsv7BOInR0QdM9cZAAkDYmhSa8cVAgIqD4yW3NzOzwQ6euWj+zP3Ax2esdd3FyMzDCHZLNfgstegXmy5QBirMUdjOQDroyYfWIns56CC82c7I05TFrYScvVqH0CpAJk5J8kSSE7KetGPgGoQYlFdhhreN4UnsOoBkmgeBn0sOR71c/Zv6Ix8XIYYXliRm0n7YjdMNi/r9a5Gm0LbdBC9GwE/Ef8BRi3/NZ6XQbQXhMT2m2ahL+t45nhT2oxXmqYTTYCAAAAAElFTkSuQmCC')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n  `,mdx(\"picture\",{parentName:\"span\"},`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/601b1/figurePricesDataset2.webp 500w\",\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/3e9ea/figurePricesDataset2.webp 891w\"],\"sizes\":\"(max-width: 891px) 100vw, 891px\",\"type\":\"image/webp\"}),`\n          `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/0eb09/figurePricesDataset2.png 500w\",\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/c7d76/figurePricesDataset2.png 891w\"],\"sizes\":\"(max-width: 891px) 100vw, 891px\",\"type\":\"image/png\"}),`\n          `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/d2b5cab4bda39bd1d25ed45eb3f556ee/c7d76/figurePricesDataset2.png\",\"alt\":\"Figure prices dataset (large)\",\"title\":\"Figure prices dataset (large)\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n        `),`\n    `)),mdx(\"p\",null,mdx(\"p\",null,`Your task is to modify the existing `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function\nsuch that it now returns not only the optimized parameter vector `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`,\nbut also the history of the training loss. This history can be a list which contains\nthe loss of every training iteration. Use the plotting feature to compare the histories\nof ridge and lasso.`)),mdx(\"p\",null,mdx(\"p\",null,mdx(\"em\",{parentName:\"p\"},`Bonus task`),`: Create a validation set by splitting the `,mdx(\"inlineCode\",{parentName:\"p\"},`X`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`y`),` that are passed into\nthe `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function. Then, in each epoch, compute the validation loss\nin addition to the training loss. Return both loss histories at the end of the function.\nNote: call the variables holding the validation losses `,mdx(\"inlineCode\",{parentName:\"p\"},`loss_hist_val_ridge`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`loss_hist_val_lasso`),`.`))),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def subgradient_descent(X, y, theta, criterion, subgradient_function, number_of_iterations, learning_rate, eps):\n    X_b = add_intercept_ones(X)\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta)  # create the current function\n        y_predicted = f(X_b)  # predict our entire x\n        loss = criterion(y, y_predicted, theta)  # calculate the error\n\n        # perform optimization\n        subgradient = subgradient_function(X_b, y, y_predicted, theta)  # calculate gradient\n        theta = theta - learning_rate * subgradient  # adjust m and b\n\n        theta[theta < eps] = 0\n\n    return theta # return loss hist as well\n\ntheta_init = np.random.rand(2)\ntheta_lasso, loss_hist_train_lasso = subgradient_descent(\n    # pass arguments here\n)\ntheta_ridge, loss_hist_train_ridge = subgradient_descent(\n    # pass arguments here\n)\n`)),mdx(\"div\",null,mdx(\"div\",null,mdx(\"strong\",null,\"With train loss\"),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\",\"metastring\":\"{4,13-14,22}\",\"{4,13-14,22}\":true},`import setup\ndef subgradient_descent(X, y, theta, criterion, subgradient_function, number_of_iterations, learning_rate, eps):\n    X_b = add_intercept_ones(X)\n    loss_hist_train = []\n\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta)  # create the current function\n        y_predicted = f(X_b)  # predict our entire x\n        loss = criterion(y, y_predicted, theta)  # calculate the error\n\n        # add loss to history\n        loss_hist_train.append(loss)\n\n        # perform optimization\n        subgradient = subgradient_function(X_b, y, y_predicted, theta)  # calculate gradient\n        theta = theta - learning_rate * subgradient  # adjust m and b\n\n        theta[theta < eps] = 0\n\n    return theta, loss_hist_train\n\ntheta_init = np.random.rand(2)\ntheta_lasso, loss_hist_train_lasso = subgradient_descent(\n    X_train,\n    y_train,\n    theta_init,\n    get_elastic_mse_function(a2=0, a1=1),\n    get_elastic_gradient_function(a2=0, a1=1),\n    300,\n    0.01,\n    0.01,\n)\ntheta_ridge, loss_hist_train_ridge = subgradient_descent(\n    X_train,\n    y_train,\n    theta_init,\n    get_elastic_mse_function(a2=1, a1=0),\n    get_elastic_gradient_function(a2=1, a1=0),\n    300,\n    0.01,\n    0.01,\n)\n`))),mdx(\"div\",null,mdx(\"strong\",null,\"With train and val loss\"),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\",\"metastring\":\"{7-8,21-23,31}\",\"{7-8,21-23,31}\":true},`import setup\ndef subgradient_descent(X, y, theta, criterion, subgradient_function, number_of_iterations, learning_rate, eps):\n    X_b = add_intercept_ones(X)\n\n    X_train_b, y_train, X_val_b, y_val = X_b[:10], y[:10], X_b[10:], y[10:]\n\n    loss_hist_train = []\n    loss_hist_val = []\n\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta)  # create the current function\n        y_train_predicted = f(X_train_b)  # predict our entire x\n        train_loss = criterion(y_train, y_train_predicted, theta)  # calculate the error\n\n        # validation loss\n        y_val_predicted = f(X_val_b)\n        val_loss = criterion(y_val, y_val_predicted, theta)\n\n        # add losses to history\n        loss_hist_train.append(train_loss)\n        loss_hist_val.append(val_loss)\n\n        # perform optimization\n        subgradient = subgradient_function(X_train_b, y_train, y_train_predicted, theta)  # calculate gradient\n        theta = theta - learning_rate * subgradient  # adjust m and b\n\n        theta[theta < eps] = 0\n\n    return theta, loss_hist_train, loss_hist_val\n\ntheta_init = np.random.rand(2)\ntheta_lasso, loss_hist_train_lasso, loss_hist_val_lasso = subgradient_descent(\n    X_train,\n    y_train,\n    theta_init,\n    get_elastic_mse_function(a2=0, a1=1),\n    get_elastic_gradient_function(a2=0, a1=1),\n    300,\n    0.01,\n    0.01,\n)\ntheta_ridge, loss_hist_train_ridge, loss_hist_val_ridge = subgradient_descent(\n    X_train,\n    y_train,\n    theta_init,\n    get_elastic_mse_function(a2=1, a1=0),\n    get_elastic_gradient_function(a2=1, a1=0),\n    300,\n    0.01,\n    0.01,\n)\n`))),mdx(BlurredAnswerBox,{visibleText:\"Hide Explanation\",blurredText:\"Show Explanation\",mdxType:\"BlurredAnswerBox\"},mdx(\"p\",null,`The solution should be mostly self-explanatory.\nWhat’s interesting is that, if you have taken a look at the visualization,\n(sub)gradient descent does not converge equivalently quickly for ridge and lasso.\nIn fact, ridge regression converges quite a bit faster than lasso.\nBut why is that? Is it because we used a subgradient, or is it an inherent property\nof the L1-penalty? To find out, take a look at the next programming exercise.`))),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def find_convergence_point(arr, eps=0.1):\n    idx = 0\n    while idx < len(arr) - 1:\n        diff = abs(arr[idx] - arr[idx + 1])\n        if diff < eps:\n            return idx\n        idx += 1\n    return -1\n\nfig, ax = plt.subplots(figsize=(7, 5))\nax.scatter(\n    [],\n    [],\n    c=\"white\",\n    alpha=1,\n    marker=r\"$\\\\downarrow$\",\n    s=100,\n    label=\"convergence point\",\n)\n\ntry:\n    hists = [loss_hist_train_lasso, loss_hist_train_ridge]\n    convergence_points = [find_convergence_point(hist) for hist in hists]\n    colors = [\"#1f77bf\", \"#ff7f0e\"]\n    names = [\"ridge | train\", \"lasso | train\",]\n    linestyles = [\"dashed\", \"solid\"]\n\n    for hist, point, color, name, linestyle in zip(\n        hists, convergence_points, colors, names, linestyles\n    ):\n        ax.plot(hist, linestyle=linestyle, label=name)\n        if point != -1:\n            ax.annotate(\n                \"\",\n                xy=(point, hist[point]),\n                xytext=(point, hist[point] + 50),\n                ha=\"center\",\n                arrowprops=dict(facecolor=color, shrink=0.05),\n                fontsize=16,\n            )\n        else:\n            print(f\"{name} not converged.\")\nexcept Exception as e:\n    print(\"Plotting training loss failed.\")\n    print(e)\n\ntry:\n    hists = [loss_hist_val_lasso, loss_hist_val_ridge]\n    convergence_points = [find_convergence_point(hist) for hist in hists]\n    colors = [\"#2ca02c\", \"#d62728\"]\n    names = [\"ridge | val\", \"lasso | val\"]\n    linestyles = [\"dashed\", \"solid\"]\n\n    for hist, point, color, name, linestyle in zip(\n        hists, convergence_points, colors, names, linestyles\n    ):\n        ax.plot(hist, linestyle=linestyle, label=name)\n        if point != -1:\n            ax.annotate(\n                \"\",\n                xy=(point, hist[point]),\n                xytext=(point, hist[point] + 50),\n                ha=\"center\",\n                arrowprops=dict(facecolor=color, shrink=0.05),\n                fontsize=16,\n            )\n        else:\n            print(f\"{name} not converged.\")\nexcept Exception as e:\n    print(\"Plotting val loss failed.\")\n    print(e)\n\nax.set_xlabel(\"Epoch\")\nax.set_ylabel(\"Loss\")\nax.legend()\nplt.show()\n`)),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def find_convergence_point(arr, eps=0.1):\n    idx = 0\n    while idx < len(arr) - 1:\n        diff = abs(arr[idx] - arr[idx + 1])\n        if diff < eps:\n            return idx\n        idx += 1\n    return -1\n\n# tests\n\ndef testOne():\n    train_convergence_ridge = find_convergence_point(loss_hist_train_ridge)\n    passed = train_convergence_ridge != -1\n    explanation = \"Ridge training loss converged at epoch below.\" if passed else \"Ridge training loss should converge, but did not.\"\n    return passed, explanation, f\"Converged at epoch {train_convergence_ridge}.\"\n\ndef testTwo():\n    train_convergence_lasso = find_convergence_point(loss_hist_train_lasso)\n    passed = train_convergence_lasso != -1\n    explanation = \"Lasso training loss converged at epoch below.\" if passed else \"Lasso training loss should converge, but did not.\"\n    return passed, explanation, f\"Converged at epoch {train_convergence_lasso}.\"\n\ndef testThree():\n    val_convergence_ridge = find_convergence_point(loss_hist_val_ridge)\n    passed = val_convergence_ridge != -1\n    explanation = \"Ridge val loss converged at epoch below.\" if passed else \"Ridge val loss should converge, but did not.\"\n    return passed, explanation, f\"Converged at epoch {val_convergence_ridge}.\"\n\ndef testFour():\n    val_convergence_lasso = find_convergence_point(loss_hist_val_lasso)\n    passed = val_convergence_lasso != -1\n    explanation = \"Lasso val loss converged at epoch below.\" if passed else \"Lasso val loss should converge, but did not.\"\n    return passed, explanation, f\"Converged at epoch {val_convergence_lasso}.\"\n\ndef testFive():\n    train_convergence_ridge = find_convergence_point(loss_hist_train_ridge)\n    train_convergence_lasso = find_convergence_point(loss_hist_train_lasso)\n    if train_convergence_ridge == -1:\n        raise ValueError('Ridge training loss did not converge.')\n    elif train_convergence_lasso == -1:\n        raise ValueError('Lasso training loss did not converge.')\n    train_convergence_diff = abs(train_convergence_ridge - train_convergence_lasso)\n\n    passed = train_convergence_diff > 50\n    explanation = \"Ridge and lasso train losses did not converge within 50 epochs of each other.\" if passed else \"Ridge and lasso train losses did converge within 50 epochs of each other, even though they should not have.\"\n    return passed, explanation, f\"Converged within {train_convergence_diff} epochs.\"\n\ndef testSix():\n    val_convergence_ridge = find_convergence_point(loss_hist_val_ridge)\n    val_convergence_lasso = find_convergence_point(loss_hist_val_lasso)\n    if val_convergence_ridge == -1:\n        raise ValueError('Ridge val loss did not converge.')\n    elif val_convergence_lasso == -1:\n        raise ValueError('Lasso val loss did not converge.')\n    val_convergence_diff = abs(find_convergence_point(loss_hist_val_lasso) - find_convergence_point(loss_hist_val_ridge))\n\n    passed = val_convergence_diff > 50\n    explanation = \"Ridge and lasso val losses did not converge within 50 epochs of each other.\" if passed else \"Ridge and lasso val losses did converge within 50 epochs of each other, even though they should not have.\"\n    return passed, explanation, f\"Converged within {val_convergence_diff} epochs.\"\n\ntests = [testOne, testTwo, testThree, testFour, testFive, testSix]\n`))))),mdx(\"div\",null,mdx(\"strong\",null,mdx(\"p\",null,`Do we actually need the subgradient?`)),mdx(\"p\",null,`In the article we have seen that subgradient descent for lasso\nproduces imprecise results, but not because of the subgradient,\nbut because of gradient descent itself. In exercise #1 we have also\nseen that (at least for our dataset) subgradient descent converged\nfaster for ridge than for lasso. Now the question is, why did ridge regression\nconverge faster? Is it because we used a subgradient, or because the L2-penalty\nnaturally converges faster than the L1-penalty?`),mdx(\"p\",null,`If you want, you can think about this question from a theoretical point of view.\nHere, we want to go with a more practical approach and see how often the non-differentiable\npoint of our L1-penalty is actually hit.\nIn other terms, we want to know how often `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msub\"},mdx(\"mover\",{parentName:\"mo\"},mdx(\"mo\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mo\",\"mathvariant\":\"normal\"},`∇`)),mdx(\"mrow\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`u`),mdx(\"mi\",{parentName:\"mrow\"},`b`)))),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mn\",{parentName:\"mrow\"},`1`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`n`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\stackrel{sub}{\\\\nabla}_{L1}(\\\\boldsymbol{\\\\theta}) = sign(\\\\boldsymbol{\\\\theta})`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.527773em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.3777730000000001em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.908335em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop\"},`∇`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.791665em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`u`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.091665em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.32833099999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),` evaluates\nto 0.`),mdx(\"p\",null,`Your task is to modify the `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function\nin such a way that it keeps track of the number of occurrences\nthis very particular point of our subgradient is hit.\nAt the end, this value should be returned by the function alongside\nthe optimized `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`.`),mdx(\"p\",null,mdx(ConditionalLazyCodeInteractive,{startFullscreen:true,hasPrompt:true,codeSetup:setupCodeSubgradientDescentReinforcement,hasSolution:true,hasPlot:false,hasTests:true,testDivId:\"testsSubGD2\",packages:['numpy'],mdxType:\"ConditionalLazyCodeInteractive\"},mdx(\"div\",null,mdx(\"p\",null,`In the article we have seen that subgradient descent for lasso\nproduces imprecise results, but not because of the subgradient,\nbut because of gradient descent itself. In exercise #1 we have also\nseen that (at least for our dataset) subgradient descent converged\nfaster for ridge than for lasso. Now the question is, why did ridge regression\nconverge faster? Is it because we used a subgradient, or because the L2-penalty\nnaturally converges faster than the L1-penalty?`),mdx(\"p\",null,`If you want, you can think about this question from a theoretical point of view.\nHere, we want to go with a more practical approach and see how often the non-differentiable\npoint of our L1-penalty is actually hit.\nIn other terms, we want to know, how often `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msub\"},mdx(\"mover\",{parentName:\"mo\"},mdx(\"mo\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mo\",\"mathvariant\":\"normal\"},`∇`)),mdx(\"mrow\",{parentName:\"mover\"},mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`u`),mdx(\"mi\",{parentName:\"mrow\"},`b`)))),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mn\",{parentName:\"mrow\"},`1`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`n`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\stackrel{sub}{\\\\nabla}_{L1}(\\\\boldsymbol{\\\\theta}) = sign(\\\\boldsymbol{\\\\theta})`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.527773em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.3777730000000001em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.908335em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop\"},`∇`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.791665em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`u`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.091665em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.32833099999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),` evaluates\nto 0.`),mdx(\"p\",null,`Your task is to modify the `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_descent`),`-function\nin such a way that it keeps track of the number of occurrences\nthis very particular point of our subgradient is hit.\nAt the end, this value should be returned by the function alongside\nthe optimized `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`.`)),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def subgradient_descent(X, y, theta, criterion, subgradient_function, number_of_iterations, learning_rate, eps):\n    X_b = add_intercept_ones(X)\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta)  # create the current function\n        y_predicted = f(X_b)  # predict our entire x\n        loss = criterion(y, y_predicted, theta)  # calculate the error\n\n        # perform optimization\n        subgradient = subgradient_function(X_b, y, y_predicted, theta)  # calculate gradient\n        theta = theta - learning_rate * subgradient  # adjust m and b\n\n        theta[theta < eps] = 0\n\n    return theta # return number of times subgradient is hit at x = 0\n\n\ntheta_init = np.random.rand(2)\n_, subgradient_hits_at_zero = subgradient_descent(\n    X_train,\n    y_train,\n    theta_init,\n    get_elastic_mse_function(a2=0, a1=1),\n    get_elastic_gradient_function(a2=0, a1=1),\n    3000,\n    0.01,\n    0.01,\n)\n`)),mdx(\"div\",null,mdx(\"div\",null,mdx(\"strong\",null,\"Solution Dropdown\"),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\",\"metastring\":\"{4,13-14,22}\",\"{4,13-14,22}\":true},`import setup\ndef subgradient_descent(X, y, theta, criterion, subgradient_function, number_of_iterations, learning_rate, eps):\n    X_b = add_intercept_ones(X)\n    zero_hits = 0\n\n    for i in range(number_of_iterations):\n\n        # predict and calculate loss\n        f = create_function(theta)  # create the current function\n        y_predicted = f(X_b)  # predict our entire x\n        loss = criterion(y, y_predicted, theta)  # calculate the error\n\n        if np.linalg.norm(theta, ord=1) == 0:\n            zero_hits += 0\n\n        # perform optimization\n        subgradient = subgradient_function(X_b, y, y_predicted, theta)  # calculate gradient\n        theta = theta - learning_rate * subgradient  # adjust m and b\n\n        theta[theta < eps] = 0\n\n    return theta, zero_hits\n\n\ntheta_init = np.random.rand(2)\n_, subgradient_hits_at_zero = subgradient_descent(\n    X_train,\n    y_train,\n    theta_init,\n    get_elastic_mse_function(a2=0, a1=1),\n    get_elastic_gradient_function(a2=0, a1=1),\n    3000,\n    0.01,\n    0.01,\n)\n`))),mdx(BlurredAnswerBox,{visibleText:\"Hide Explanation\",blurredText:\"Show Explanation\",mdxType:\"BlurredAnswerBox\"},mdx(\"p\",null,`Once again the code should be mostly self-explanatory.\nOur subgradient at the point 0 is hit when our parameter vector `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`\nonly contains zeroes as entries, i.e. when the L1-norm of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` is zero.\nWhen we run the code above and check the value of `,mdx(\"inlineCode\",{parentName:\"p\"},`subgradient_hits_at_zero`),`,\nwe can see that our subgradient is indeed never computed at 0.\nThis means that in programming exercise #1, the reason why lasso converged slower\nthan ridge was not due to us using a subgradient, but it is instead an inherent\nproperty of the L1-penalty.`),mdx(\"p\",null,`Intuitively, this can be explained by the fact that the derivative of the L1-penalty\nis just 1 or -1, while the derivative of the L2-penalty can take on any real number.\nThis means that the gradient of ridge can be scaled much more strongly by the corresponding penalty\nthan lasso can, because of the wider range of potential values our penalty derivative can take on.`))),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`# --- plot goes here ---\n# no plot, placeholder\n`)),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`# --- tests go here ---\ndef testOne():\n    passed = subgradient_hits_at_zero == 0\n    explanation = \"Subgradient was never hit at 0.\" if passed else \"Subgradient was hit at 0, even though it should not have.\"\n    return passed, explanation, f\"Subgradient was hit {subgradient_hits_at_zero} times at 0.\"\n\ntests = [testOne]\n`))))))));};MDXContent.isMDXComponent=true;","excerpt":"Gradient descent is one of the most popular algorithms to train machine learning models. However, many of the popular machine learning models like lasso regression or support vector machines contain loss functions that are not differentiable. Because of this, regular gradient descent can not be used. One of the most commonly utilized techniques to circumvent this issue is to use subgradients instead of regular gradients. And in this article, you will learn how it's done.","tableOfContents":{"items":[{"url":"#outline","title":"Outline"},{"url":"#prerequisites","title":"Prerequisites"},{"url":"#the-problem","title":"The Problem"},{"url":"#part-1-subgradients","title":"Part 1: Subgradients","items":[{"url":"#formalizing-subgradients","title":"Formalizing Subgradients"},{"url":"#refining-our-definition","title":"Refining our Definition"}]},{"url":"#part-2-subgradient-descent","title":"Part 2: Subgradient Descent","items":[{"url":"#example-implementation-lasso-regression","title":"Example Implementation: Lasso Regression"},{"url":"#implementing-subgradient-descent-for-lasso","title":"Implementing Subgradient Descent for Lasso"},{"url":"#the-fatal-flaw","title":"The Fatal Flaw","items":[{"url":"#the-fatal-flaw-of","title":"The Fatal Flaw of...?"},{"url":"#fixing-subgradient-descent-for-lasso","title":"Fixing (Sub)gradient Descent for Lasso"}]},{"url":"#visualizing-subgradient-descent","title":"Visualizing Subgradient Descent"}]},{"url":"#further-reading","title":"Further Reading"}]},"frontmatter":{"title":"Subgradient Descent Explained, Step by Step","userDate":"27 July 2021","date":"2021-07-27T19:49:21.503Z","dateModified":"2021-08-22T18:51:14.860Z","changelog":[["2021-07-31T14:28:41.400Z","Added interactive, runnable code blocks."],["2021-08-21T17:17:59.435Z","Reworked section 'The Fatal Flaw' to more correctly describe the issue of weights not being set exactly to zero. Added quiz questions and programming exercises."],["2021-08-22T18:51:14.860Z","Added Jupyter notebook."]],"tags":["Machine Learning Math"],"timeToRead":18,"excerpt":"Gradient descent is one of the most popular algorithms to train machine learning models. However, many of the popular machine learning models like lasso regression or support vector machines contain loss functions that are not differentiable. Because of this, regular gradient descent can not be used. One of the most commonly utilized techniques to circumvent this issue is to use subgradients instead of regular gradients. And in this article, you will learn how it's done.","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACaElEQVQozyXMW3eSAAAAYH5C1pB5gTkRxRt3kYvgBUXUCQqK4n2bzu3UcnPHtszWOT1YPdTpJ/fQ9wM+wCvTgyrvVHiDT5kKaxVzvVph7rQmuthXibEuLuxmTxP6VenqojjRxSaf9gzF1iSdTQALnZlo9Fhj1PRZKRup0WhHoSd1eaqLtsp5VaGtMG0pY0pZS6H7GlchYxUq1srhVRIFVCLuqpmumLAFjE/AAg535GxHJqx80i4QdQblsKBI4mo2KiTCcurMEKimSDgqrVMokEqSIfC0kEQ29ZQroJ6MfbXI3Xp2eHm+X6++PO/2n55+/Py1uX+/3z3tHjeH/Ytd4nUqOm8qAE3ncZwAfWCTQb0CPpAwmwpsltNvr4dBS9tvP67n49nAmbr2zHPdtjGyza7KLprqdtoBYmGYZfLJJOl7A/lPQuvl8uZyNrbNgdno64pdKwxa1UFTmzjtWd+SEyE1Fbm2qrc9Y6HnACIWo7FovahSBPvWF5jPL7fb7fF4vL1Z/f3ze728Ph5/PD0+vO4/P9zdGBx+aWrb0cWwkFo1BAA8CQX8MIYgllYySuVyWZekkiwVW7ox71krt33nda+c1sq1Pkz7h/Xo++1gdaGMjcKowgEJNJGK4SAYBt8Fk1GUjcfwyHkAgn2+oB8MkXGsJVLDCrdsCPemdFXnHYUcltlJhZ7rPMBTrMiJERg9hRAQhAMQkjxHOTyeiqKBAAJBCIVhppB1lWxHynRl0qtwowo7q+WGKgkgwUg4GAHBMOQPQ2AYPAn/L6g4VmXSGpMmsJgfhM9CSDOfaeTwrph25Kyrkj0p/Q8tyKY2MJ2klwAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/8d667dbc302a08dd3745c90bfb78dfd9/87706/SubGradientDescent_Thumbnail2.png","srcSet":"/static/8d667dbc302a08dd3745c90bfb78dfd9/0dee1/SubGradientDescent_Thumbnail2.png 750w,\n/static/8d667dbc302a08dd3745c90bfb78dfd9/8beaa/SubGradientDescent_Thumbnail2.png 1080w,\n/static/8d667dbc302a08dd3745c90bfb78dfd9/87706/SubGradientDescent_Thumbnail2.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/8d667dbc302a08dd3745c90bfb78dfd9/a66aa/SubGradientDescent_Thumbnail2.webp 750w,\n/static/8d667dbc302a08dd3745c90bfb78dfd9/65dd5/SubGradientDescent_Thumbnail2.webp 1080w,\n/static/8d667dbc302a08dd3745c90bfb78dfd9/71d4d/SubGradientDescent_Thumbnail2.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Boris Giba","bio":"Hi! I'm Boris and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAQFBv/EABYBAQEBAAAAAAAAAAAAAAAAAAABAv/aAAwDAQACEAMQAAAB25bcya1Bcz84HRg//8QAGxAAAwACAwAAAAAAAAAAAAAAAQIDABESMTP/2gAIAQEAAQUCo3FFo4OEbG1R8u5SXZh4f//EABURAQEAAAAAAAAAAAAAAAAAAAEg/9oACAEDAQE/ASP/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAeEAABBAEFAAAAAAAAAAAAAAABAAIQETESIUFRcf/aAAgBAQAGPwIkZTQ/Sb6ilvxBIzDPF//EABoQAQACAwEAAAAAAAAAAAAAAAERIQAQMUH/2gAIAQEAAT8hQ1PjHRkxa9E644V8JdD3BEk1JiRlbnUf/9oADAMBAAIAAwAAABADAID/xAAWEQEBAQAAAAAAAAAAAAAAAAABESD/2gAIAQMBAT8QAjcf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFdIT//EAB4QAQACAQQDAAAAAAAAAAAAAAERIQAQMUFRYYGx/9oACAEBAAE/EClkAFglYJ8Xl1kNwXZjrQ1xCEcPBl8E6A89+sAII7JzjXgEBSYlyYrZm+d/uVsWluf/2Q==","aspectRatio":1,"src":"/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg","srcSet":"/static/ba9e631272af1b925d6491b30df3b36f/b3d27/boris.jpg 40w,\n/static/ba9e631272af1b925d6491b30df3b36f/5a735/boris.jpg 80w,\n/static/ba9e631272af1b925d6491b30df3b36f/6e81a/boris.jpg 120w,\n/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg 800w,\n/static/ba9e631272af1b925d6491b30df3b36f/97bce/boris.jpg 1878w","sizes":"(max-width: 800px) 100vw, 800px"}}]}}],"imgCaption":"Background image by Pontus Wellgraf","imgSource":"https://unsplash.com/photos/phdR8e9zOzg","uniquePostId":13,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner"}},"relatedPosts":{"totalCount":3,"edges":[{"node":{"id":"d331ba94-308d-5859-8c5e-3f7526c59b0a","excerpt":"Gradient descent is one of the most popular algorithms to train machine learning models. However, many of the popular machine learning models like lasso regression or support vector machines contain loss functions that are not differentiable. Because of this, regular gradient descent can not be used. One of the most commonly utilized techniques to circumvent this issue is to use subgradients instead of regular gradients. And in this article, you will learn how it's done.","frontmatter":{"title":"Subgradient Descent Explained, Step by Step","timeToRead":18,"date":"2021-07-27T19:49:21.503Z"},"fields":{"slug":"/machine_learning_math/subgradient_descent/"}}},{"node":{"id":"340d90b8-6082-5581-a444-263bddfa3711","excerpt":"Gradient descent is one of the most famous techniques in machine learning and used for training all sorts of neural networks. But gradient descent can not only be used to train neural networks, but many more machine learning models. In particular, gradient descent can be used to train a linear regression model! If you are curious as to how this is possible, or if you want to approach gradient descent with smaller steps and not jump straight to neural networks, this post is for you. You will learn how gradient descent works from an intuitive, visual, and mathematical standpoint and we will apply it to an exemplary dataset in Python.","frontmatter":{"title":"Gradient Descent for Linear Regression Explained, Step by Step","timeToRead":23,"date":"2021-04-03T19:07:33.321Z"},"fields":{"slug":"/machine_learning_math/gradient_descent_for_linear_regression/"}}},{"node":{"id":"8eab4786-e5fd-5dd7-bc3f-bec6b2d4b411","excerpt":"Vectorization is one of the most useful techniques to make your machine learning code more efficient. In this post, you will learn everything you need to know to start using vectorization efficiently in your machine learning projects.","frontmatter":{"title":"Vectorization Explained, Step by Step","timeToRead":7,"date":"2021-03-06T18:32:05.647Z"},"fields":{"slug":"/machine_learning_math/vectorization/"}}}]}},"pageContext":{"slug":"/machine_learning_math/subgradient_descent/","prev":{"excerpt":"Standardization is one of the most useful transformations you can apply to your dataset. What is even more important is that many models, especially regularized ones, require the data to be standardized in order to function properly. In this article, you will learn everything you need to know about standardization. You will learn why it works, when you should use it, and how you can do so with just a few lines of code.","frontmatter":{"title":"When, Why, And How You Should Standardize Your Data","tags":["Dataset Optimization"],"date":"2021-07-06T11:47:10.172Z","dateModified":"2021-07-06T11:47:10.172Z","changelog":[],"draft":false,"excerpt":"Standardization is one of the most useful transformations you can apply to your dataset. What is even more important is that many models, especially regularized ones, require the data to be standardized in order to function properly. In this article, you will learn everything you need to know about standardization. You will learn why it works, when you should use it, and how you can do so with just a few lines of code.","uniquePostId":12,"timeToRead":10,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACZUlEQVQozx2PyW7TUAAA/Q90yZ4uNGspVFSKRDnQbI6z2Vn8vNux4+X5PTtJGzWhSwht0kNLRQ8gVISExKk3vhGVyxzmNEPQKm6qqCa7lAhJwS4LTom3i5xdAFaBNfPAKnJWibdJEVYlVFdwQ8ENFdOa3zaGBK2ihoJqkkuJDiXCMm+VOKsAzCJn5VnziDXzrEkKsCJASnKrkluTcV32ahJq6QMCmEPvZGJAH/RgW4W8NeT6vmL7tGx1dQ8YnuF4VFcBxoAzsNJ3geawhq+643bPJw4LjfOLi+XNzej4eHZ5eXZ+uVguZ7PZ/Ze78enp9Oz87vYWITydTpfLxWA4+nx1dTqZXF8vNPeYCERTucMiyQgKHEmWz0iW0PcE0xNMTIsmZ2DQc4Huis9m0FJswRwAHTOy09F9IryRXQnuBGPp3YMPVaB3DZ9WIKNCCvTw/Ntw8VjuakzPoxWXVlBb9xkVMSpmNO85OxTPRDZ3wxvZQDQR2cwWqh3RxF3d//jwZM8f0eK39eknB6c0r7Uks6W6HQ23NcQobkvziFA8HYpn/jMd337FslyZrOeKncHXv9z4QZh81+Z/quoQYt/BowqrtzTclGFdgg0ZPT8HY6lQ7D/j6ezrXHwjHd99D+ZPpHtPj3/Ux78qygmQDaB7ddFuSLAqOhXBoSSXiCXehnZywVgmEE0FoqkXwZdr4URoc29jvxTJHCYOyvslvsLbTclqSHZNsineaiqIVlG7h4ntvaPkfj68tbcaTqxHkuuR5FokuRZOrAa2VkPbK4GtYCz55l2RBHpdtGu8VeH6JDBaKuLN4T+RFNmEkCrl1gAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/d95f1cf4daefb962c345773fa1506879/87706/Standardization_Thumbnail.png","srcSet":"/static/d95f1cf4daefb962c345773fa1506879/0dee1/Standardization_Thumbnail.png 750w,\n/static/d95f1cf4daefb962c345773fa1506879/8beaa/Standardization_Thumbnail.png 1080w,\n/static/d95f1cf4daefb962c345773fa1506879/87706/Standardization_Thumbnail.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/d95f1cf4daefb962c345773fa1506879/a66aa/Standardization_Thumbnail.webp 750w,\n/static/d95f1cf4daefb962c345773fa1506879/65dd5/Standardization_Thumbnail.webp 1080w,\n/static/d95f1cf4daefb962c345773fa1506879/71d4d/Standardization_Thumbnail.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Boris Giba","bio":"Hi! I'm Boris and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"aspectRatio":1,"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAQFBv/EABYBAQEBAAAAAAAAAAAAAAAAAAABAv/aAAwDAQACEAMQAAAB25bcya1Bcz84HRg//8QAGxAAAwACAwAAAAAAAAAAAAAAAQIDABESMTP/2gAIAQEAAQUCo3FFo4OEbG1R8u5SXZh4f//EABURAQEAAAAAAAAAAAAAAAAAAAEg/9oACAEDAQE/ASP/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAeEAABBAEFAAAAAAAAAAAAAAABAAIQETESIUFRcf/aAAgBAQAGPwIkZTQ/Sb6ilvxBIzDPF//EABoQAQACAwEAAAAAAAAAAAAAAAERIQAQMUH/2gAIAQEAAT8hQ1PjHRkxa9E644V8JdD3BEk1JiRlbnUf/9oADAMBAAIAAwAAABADAID/xAAWEQEBAQAAAAAAAAAAAAAAAAABESD/2gAIAQMBAT8QAjcf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFdIT//EAB4QAQACAQQDAAAAAAAAAAAAAAERIQAQMUFRYYGx/9oACAEBAAE/EClkAFglYJ8Xl1kNwXZjrQ1xCEcPBl8E6A89+sAII7JzjXgEBSYlyYrZm+d/uVsWluf/2Q==","sizes":"(max-width: 800px) 100vw, 800px","src":"/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg","srcSet":"/static/ba9e631272af1b925d6491b30df3b36f/fd013/boris.jpg 200w,\n/static/ba9e631272af1b925d6491b30df3b36f/25252/boris.jpg 400w,\n/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg 800w,\n/static/ba9e631272af1b925d6491b30df3b36f/0ff54/boris.jpg 1200w,\n/static/ba9e631272af1b925d6491b30df3b36f/06655/boris.jpg 1600w,\n/static/ba9e631272af1b925d6491b30df3b36f/97bce/boris.jpg 1878w"}}]}}]},"fields":{"layout":"post","slug":"/dataset_optimization/standardization/"}},"next":null,"primaryTag":"Machine Learning Math"}},"staticQueryHashes":["13077718","1316742457","1394499604","2907728042","3762220167"]}