{"componentChunkName":"component---src-templates-post-tsx","path":"/machine_learning_models/ridge_regression/","result":{"data":{"logo":{"childImageSharp":{"gatsbyImageData":{"layout":"fixed","backgroundColor":"#f8f8f8","images":{"fallback":{"src":"/static/e43c6530b7b2b056de6bfee951c9e393/b181d/compasslogo.png","srcSet":"/static/e43c6530b7b2b056de6bfee951c9e393/b181d/compasslogo.png 1920w","sizes":"1920px"},"sources":[{"srcSet":"/static/e43c6530b7b2b056de6bfee951c9e393/e02c9/compasslogo.webp 1920w","type":"image/webp","sizes":"1920px"}]},"width":1920,"height":472}}},"mdx":{"body":"const _excluded=[\"components\"];function _extends(){_extends=Object.assign||function(target){for(var i=1;i=0)continue;if(!Object.prototype.propertyIsEnumerable.call(source,key))continue;target[key]=source[key];}}return target;}function _objectWithoutPropertiesLoose(source,excluded){if(source==null)return{};var target={};var sourceKeys=Object.keys(source);var key,i;for(i=0;i=0)continue;target[key]=source[key];}return target;}/* @jsxRuntime classic */ /* @jsx mdx */const _frontmatter={\"layout\":\"post\",\"title\":\"Ridge Regression Explained, Step by Step\",\"author\":[\"Boris Giba\"],\"tags\":[\"Machine Learning Models\"],\"image\":\"img/Ridge/RidgeRegression_Thumbnail.png\",\"date\":\"2021-05-23T19:13:45.907Z\",\"dateModified\":\"2021-11-02T08:08:00.157Z\",\"changelog\":[[\"2021-11-02T08:08:00.157Z\",\"Added 1/n to the definition of the MSE which I previously forgot in one section. Thanks to @ML_Schlagy for pointing this out!\"]],\"draft\":false,\"imgCaption\":\"Background image by Jeremy Bishop\",\"imgSource\":\"https://unsplash.com/photos/JarNBwuPHr0\",\"timeToRead\":26,\"uniquePostId\":9,\"excerpt\":\"Ridge Regression is an adaptation of the popular and widely used linear regression algorithm. It enhances regular linear regression by slightly changing its cost function, which results in less overfit models. In this article, you will learn everything you need to know about Ridge Regression, and how you can start using it in your own machine learning projects.\",\"dependencies\":[],\"proficiencyLevel\":\"Beginner\"};const makeShortcode=name=>function MDXDefaultShortcode(props){console.warn(\"Component \"+name+\" was not imported, exported, or provided by MDXProvider as global scope\");return mdx(\"div\",props);};const ConditionalLazyPlot=makeShortcode(\"ConditionalLazyPlot\");const ToolTip=makeShortcode(\"ToolTip\");const ConditionalLazyPlotInteractive=makeShortcode(\"ConditionalLazyPlotInteractive\");const InfoTip=makeShortcode(\"InfoTip\");const layoutProps={_frontmatter};const MDXLayout=\"wrapper\";return function MDXContent(_ref){let{components}=_ref,props=_objectWithoutProperties(_ref,_excluded);return mdx(MDXLayout,_extends({},layoutProps,props,{components:components,mdxType:\"MDXLayout\"}),mdx(\"h2\",{\"id\":\"outline\"},`Outline`),mdx(\"p\",null,`You have probably heard about linear regression. Maybe you have even read\nan `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression\"},`article`),` about it.\nLasso and ridge regression are two of the most popular variations of\nlinear regression which try to make it a bit more `,mdx(\"em\",{parentName:\"p\"},`robust`),`. Nowadays it is actually very uncommon\nto see regular linear regression out in the wild, and not one of it’s variations like lasso and ridge.\nThe purpose of lasso and ridge is to\nstabilize the vanilla linear regression and make it more robust against outliers, overfitting,\nand more. Lasso and ridge are very similar, but there are also some key differences between the two\nthat you really have to understand if you want to use them confidently in practice.\nThis is why, starting with this article,\nwe’ll take a deep dive into ridge and lasso regression!\nIn this article, you will learn everything you need to know to start using\nridge regression in your next machine learning project.\nFirst, we will look at a problem that may occur with regular linear regression, `,mdx(\"em\",{parentName:\"p\"},`overfitting`),`,\nand then we’ll explore ridge regression mathematically, visually, and code it up\nin raw Python as well as with scikit-learn. Let’s start!`),mdx(\"h2\",{\"id\":\"prerequisites\"},`Prerequisites`),mdx(\"p\",null,`This article assumes that you are already familiar with linear regression. If you are not,\nI very highly recommend that you read `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression\"},`Linear Regression Explained, Step by Step`),` prior to reading this article.\nIf you are unfamiliar with linear regression, this article might be very difficult to understand.\nThis article can be considered a follow-up to the article about linear regression,\nso reading this post will be much easier if you’ve read the one about linear regression as well.\nTwo other posts that will be very helpful for understanding this particular article\nare `,mdx(\"a\",{parentName:\"p\",\"href\":\"/model_optimization/bias_and_variance/\"},`Bias, Variance, and Overfitting Explained, Step by Step`),` as well as `,mdx(\"a\",{parentName:\"p\",\"href\":\"/dataset_optimization/split_data_the_right_way/\"},`How to Split Your Dataset the Right Way`),`.\nIn the first article, you will learn all about overfitting, and how it can be explained using the notions of\nbias and variance. Overfitting is one of the reasons ridge and lasso exist in the first place,\nso understanding it will make this post a bit easier to digest. The second article will teach you all about how\nyou can split your dataset into a training and testing portion. We’ll perform this splitting in this post as well,\nso it won’t hurt if you understand why exactly this is useful. You don’t `,mdx(\"em\",{parentName:\"p\"},`have`),` to read these two\narticles before this one, but they will make reading this post about ridge regression easier,\nand clear up some confusion you might have about certain sections.\nWith that being said, let’s take a look at ridge regression!`),mdx(\"h2\",{\"id\":\"the-problem\"},`The Problem`),mdx(\"p\",null,`So what is wrong with linear regression? Why do we need more machine learning algorithms\nthat do the same thing? And why are there two of them? Let’s explore these questions in more detail.`),mdx(\"p\",null,`The goal of linear regression is to fit a line to a dataset in such a way that it minimizes\nthe mean squared error of our line and our data. With regular linear regression,\nwe don’t tell our model anything about how it should achieve this goal. We just\nwant to minimize the MSE and that’s it. We don’t really care whether the model parameters\nin our model are very large or very small, we don’t care about overfitting, and so on.`),mdx(\"p\",null,`To make the distinction between linear regression and ridge regression more clear,\nwe will from now on use the term OLS regression (or `,mdx(\"em\",{parentName:\"p\"},`ordinary least squares regression`),`) to describe\nthe “normal” linear regression we already know.`),mdx(\"p\",null,`But you might say, `,mdx(\"em\",{parentName:\"p\"},`OLS regression is not that complicated, what could go wrong with\nsuch a simple model?`),`, which would be an excellent question to ask! So let’s look at a case\nwhen things don’t go as planned.\nImagine we have a very small dataset, containing figure prices. Each\nentry in the dataset contains the age of the figure as well as its price for that age in € (or any other currency).\nWe now want to predict the price of a figure, given its age, using linear regression, to see how much the figures depreciate over time.\nThe dataset looks like this:`),mdx(ConditionalLazyPlot,_extends({},olsVsRidgeData,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1106px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"63%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABS0lEQVQ4y52Tf3OCMAyGPUEFJy2gKIityG+9nXfb9/9u7y6ZRep2N90fOULTPEmaZDKLjnBDjVXaQuY95L6H2HeYxwWfk/0ZMXcnpEyFwjKpGLhKGwRZawHp+5eYe5PF+oQgbeCnPdyogCMVy7OZ/ciQgCJrsUx7LNYlZpF+GWYBqbQw7+HIg2V0X8xyAIb5GUHajgwKXlLD33UvQQfg+vjOwPvDGmD7PyCPS9ZCqCsWm2rUuW8YPYk8ftwdKQg5hxretkGgrlZgbkp0OGMWF+z8GHnO31ujQs3Nowoem2c1RWQdH9jjoq2LFHAa5AiLT4Y6IrdmdJhDU563OSFWFzZQ1t6m5Atvu5r/SZdZx7q/rXjUKAF6/+W2hmMyHGdBzkY3m0JrSJvjJxULnZHNT0oGUmDvFnBYPQM1JY/LoBmN1IWhY7sj9a8lfwFWWzy/hBs7RgAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/040ab5967b5b4f588dc8db80d21003fb/601b1/figurePricesData.webp 500w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/80c60/figurePricesData.webp 1000w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/e6a9b/figurePricesData.webp 1106w\"],\"sizes\":\"(max-width: 1106px) 100vw, 1106px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/040ab5967b5b4f588dc8db80d21003fb/0eb09/figurePricesData.png 500w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/1263b/figurePricesData.png 1000w\",\"/static/040ab5967b5b4f588dc8db80d21003fb/22593/figurePricesData.png 1106w\"],\"sizes\":\"(max-width: 1106px) 100vw, 1106px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/040ab5967b5b4f588dc8db80d21003fb/22593/figurePricesData.png\",\"alt\":\"Figure prices dataset\",\"title\":\"Figure prices dataset\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `))),mdx(\"p\",null,`If you’ve read the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/dataset_optimization/split_data_the_right_way\"},`How to Split Your Dataset the Right Way`),`,\nyou know what we’re going to do next. That’s right, we’ll split our dataset into a train\nset and a test set! The partitioned dataset looks like this:`),mdx(ConditionalLazyPlot,_extends({},olsVsRidgeDataTrainTest,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1091px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"61.6%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAABYlAAAWJQFJUiTwAAABbElEQVQoz52T63KCQAyFd0QQV+SuCC6LSEHQ6gP0/d/rdBIGClPa6fRHhp0s+ZKcZIUZFCBb+xpx8Y7D5cnfQHXwVYf48kSkH+x3Tg0MLx//NwM9ngcT/UUB09ewwgvbkGTJxv/5XGDN0BJW0McJw9fYHd8QqDsc9YT1C2wxgafhlA8Urw/uQlALdnyFc6pn2f9sfgE7K+HrDvJQQVCLu6T+PzDsq1ztFesr3PTGolPbxv5LcFe/YEXXXmwvX0xGILKpTxDIP3fYpzW2qloYQo5d2jHcnEL9ApukhDxXMyhPeRtf4WYNZF59GwoBZdrCisr5uvwEpL55yucWK0ct6mR4iiEkidfcIVUNw815bQx3vovjHlphwdPeHiq+2EQlV0tfO+6rs6MS8tRCJg37aAflseLYgSMGXcjhJDXc7IZhlchPCchPwXRHlRLMTftX42YtJx2BQ6kD1JieKSC98cKG+sGvaPrMejnm8Z99hCYjybrWZwAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/1627c02e004ae5bc1d0610489518c516/601b1/figurePricesDataTrainTest.webp 500w\",\"/static/1627c02e004ae5bc1d0610489518c516/80c60/figurePricesDataTrainTest.webp 1000w\",\"/static/1627c02e004ae5bc1d0610489518c516/c1fee/figurePricesDataTrainTest.webp 1091w\"],\"sizes\":\"(max-width: 1091px) 100vw, 1091px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/1627c02e004ae5bc1d0610489518c516/0eb09/figurePricesDataTrainTest.png 500w\",\"/static/1627c02e004ae5bc1d0610489518c516/1263b/figurePricesDataTrainTest.png 1000w\",\"/static/1627c02e004ae5bc1d0610489518c516/acfc2/figurePricesDataTrainTest.png 1091w\"],\"sizes\":\"(max-width: 1091px) 100vw, 1091px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/1627c02e004ae5bc1d0610489518c516/acfc2/figurePricesDataTrainTest.png\",\"alt\":\"Train and Test split of the figure prices dataset\",\"title\":\"Train and Test split of the figure prices dataset\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `))),mdx(\"p\",null,`We now perform linear regression on the training data. Let’s plot the\nresulting linear function alongside our partitioned dataset:`),mdx(ConditionalLazyPlot,_extends({},olsVsRidgeDataTrainTestOls,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1103px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"63%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABu0lEQVQ4y3VT2W6DMBBE4QwEfECTQMAcIQTaqqpSKY/9/7+ayktAlKYP1sJ6PMysB8MWJSyu4MgKUXpFmHZUd8cLwmMHf3+mfY17ttZ7xtS0hAI79WBZT0RRNpIuCQm3WlPfZAVVQz9oVTJ9hxWV2LCcNk32+9C/ClkB79BB1p/kjAi1mjh/w070COMBnjzDFoqWIyrY8i+Ro6ssYUUFgqJDe/9G1t5gbKJ8tJh2pM7lNcTxBpneiNjko2KLKdj8mUoFh9dg6UAjMoLDBSJ/JUJTW2QFtnmDSA1I8jtk9oUg6+BXZ2xPDWz2xLZQ86gMfQmxekd4vMDkBalw9zWclxHoiRaH7g6Rf8CLG+pZ4v+5Gl7SgOfDqJDp+JSjNV7S/CxR0Jdd3iCUA8L4FZ5oyOqSeLq8eYYsu9JBncc/FyCrWZknG+zkgOhBPCvmoyNjw4pH/q5zlpb5Wod3yqzDK+xkT2si09XQALJ96gms6/alocO6Wryg+bpxBVdWCPYt4djpCjep4Cct/OQ8zn/5p7hxTdXft6PFx++o+zqnwaOv380FTmdxwhPh2tZke3qm0Ks3Uj6RLXHrEf0A6Ys/2JEDqiMAAAAASUVORK5CYII=')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/026e7e9074d90660410322b8ef8dda0a/601b1/figurePricesDataOLS.webp 500w\",\"/static/026e7e9074d90660410322b8ef8dda0a/80c60/figurePricesDataOLS.webp 1000w\",\"/static/026e7e9074d90660410322b8ef8dda0a/8a7c0/figurePricesDataOLS.webp 1103w\"],\"sizes\":\"(max-width: 1103px) 100vw, 1103px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/026e7e9074d90660410322b8ef8dda0a/0eb09/figurePricesDataOLS.png 500w\",\"/static/026e7e9074d90660410322b8ef8dda0a/1263b/figurePricesDataOLS.png 1000w\",\"/static/026e7e9074d90660410322b8ef8dda0a/890d8/figurePricesDataOLS.png 1103w\"],\"sizes\":\"(max-width: 1103px) 100vw, 1103px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/026e7e9074d90660410322b8ef8dda0a/890d8/figurePricesDataOLS.png\",\"alt\":\"OLS for predicting figure prices\",\"title\":\"OLS for predicting figure prices\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `))),mdx(\"div\",null,mdx(\"p\",null,`Now I don’t know about you, but the function generated by our linear regression model\ndoes seem to have a pretty steep slope. A one-year-old figure worth 40€ would be worth\nonly 20€ after three years, according to our function. That doesn’t seem right!\nIf you’ve read the `,mdx(\"a\",{parentName:\"p\",\"href\":\"/model_optimization/bias_and_variance\"},`article about bias and variance`),`,\nyou know how to describe this model. It has a low bias and a high variance and is therefor\noverfit. We can confirm this by looking at the training and testing errors`,mdx(ToolTip,{mdxType:\"ToolTip\"},mdx(\"inlineCode\",{parentName:\"p\"},`mse`),` is the mean squared error which we’ve first implemented in the `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression/#adjusting-the-sosr\"},`article about linear regression`),`\nand then implemented it more efficiently in the `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/vectorization/#example-2-mse\"},`article about vectorization`),`.`),`:`)),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`mse(y_train, ols.predict(X_train))\n# output: 0.175\nmse(y_test, ols.predict(X_test))\n# output: 23.616666666666653\n`)),mdx(\"p\",null,`But how can we create a better model, one that is not overfit? And what exactly\nleads to our model overfitting?`),mdx(\"h2\",{\"id\":\"improving-ols-regression\"},`Improving OLS Regression`),mdx(\"p\",null,`Let’s first try to come up with an “imaginary” model\nthat is a better version of our current one and one that isn’t overfit. Let’s try\nand create a plot of our idealized model and display it alongside our current model.\nThe result looks like this:`),mdx(ConditionalLazyPlot,_extends({},olsVsRidgeDataTrainTestOlsRidge,{mdxType:\"ConditionalLazyPlot\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1106px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"63.6%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB/klEQVQ4y21T3ZqaMBBlEUQQIQl/KpCACijqdrsX/dqLff/HOv0mCGK3F0NCZuZk5syJYXMFsqWo4KUnuMkBXnKEE9Xw0iPc5NUohmItJjHmzs2gzyIsYXOJpLqDlz1E2SPYd2D5Bay4gBc94uoOIa9Iqnes4noCtNiQT0ZnBm38bYNN2sEMSphhMTnHwLmZD9+zKon1rsdm1+rKNSArzhC7G9b8Ak+0sOhmLrEUAxXfbNYigWcfv3C4/9GVG2ZQINx34PkJKy4RRA3S/SfC7AYzlHjTFSuYgdTtWY82rRGUSUTFFd3PL82/QcRrjvIGLC3AkhzZ8QjV/oBUH9iVPaJCIj4d4MYKrlBwI4mVUFiwocIgleg+vxCrGwwinTbr9IS3kKqoYKc1lomCF9cQ2yvk+2+IvIXYluBZAZHJh5XwYwkzVFoVmkOSAk11s21gsVLzRm3YbGjNZCVsVsNlLXzRY8UbXdmCS7iRgiOeXOopE4fBrtWANKDlf7SlEzj5KviiQxD18AQBy2961FMO87OWziiXucbG28d1QRXzCh5vEYgeDju8gE46pElrgvedHr8TVVhFtQ7yNR0SbnzQZ+T3tyc4vNb6fQEcf4hU2q+zBo6opudIUqAOCGgknlYaIsUTuM0GgU9Pb97Wv22L8opI3RDuzy++4bmOec/4v6RoQmYKoKCRAAAAAElFTkSuQmCC')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/bcd8acf2452585219b152cfe41dc8a87/601b1/figurePricesDataOLSRidge.webp 500w\",\"/static/bcd8acf2452585219b152cfe41dc8a87/80c60/figurePricesDataOLSRidge.webp 1000w\",\"/static/bcd8acf2452585219b152cfe41dc8a87/e6a9b/figurePricesDataOLSRidge.webp 1106w\"],\"sizes\":\"(max-width: 1106px) 100vw, 1106px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/bcd8acf2452585219b152cfe41dc8a87/0eb09/figurePricesDataOLSRidge.png 500w\",\"/static/bcd8acf2452585219b152cfe41dc8a87/1263b/figurePricesDataOLSRidge.png 1000w\",\"/static/bcd8acf2452585219b152cfe41dc8a87/22593/figurePricesDataOLSRidge.png 1106w\"],\"sizes\":\"(max-width: 1106px) 100vw, 1106px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/bcd8acf2452585219b152cfe41dc8a87/22593/figurePricesDataOLSRidge.png\",\"alt\":\"OLS vs Ridge for predicting figure prices\",\"title\":\"OLS vs Ridge for predicting figure prices\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `))),mdx(\"p\",null,`Alright, that looks better! Our idealized model\ndoes seem to have a slightly higher bias than our OLS model,\nbut it also has a lot lower variance. We can confirm this by looking at the\nMSE of both models on the training and testing data and\nthen compare the relative difference of the errors for training and testing,\nwhich, as we know from the `,mdx(\"a\",{parentName:\"p\",\"href\":\"/model_optimization/bias_and_variance\"},`article about bias and variance`),`,\nis a good measure to intuitively get a feel for the variance of a model.\nIf we do this, we get the following results:`),mdx(\"table\",null,mdx(\"thead\",{parentName:\"table\"},mdx(\"tr\",{parentName:\"thead\"},mdx(\"th\",{parentName:\"tr\",\"align\":null}),mdx(\"th\",{parentName:\"tr\",\"align\":null},`MSE (Train)`),mdx(\"th\",{parentName:\"tr\",\"align\":null},`MSE (Test)`),mdx(\"th\",{parentName:\"tr\",\"align\":null},`R.DIFF`))),mdx(\"tbody\",{parentName:\"table\"},mdx(\"tr\",{parentName:\"tbody\"},mdx(\"td\",{parentName:\"tr\",\"align\":null},`OLS`),mdx(\"td\",{parentName:\"tr\",\"align\":null},`0.175`),mdx(\"td\",{parentName:\"tr\",\"align\":null},`23.62`),mdx(\"td\",{parentName:\"tr\",\"align\":null},`13397.14%`)),mdx(\"tr\",{parentName:\"tbody\"},mdx(\"td\",{parentName:\"tr\",\"align\":null},`Imaginary model`),mdx(\"td\",{parentName:\"tr\",\"align\":null},`6.28`),mdx(\"td\",{parentName:\"tr\",\"align\":null},`3.77`),mdx(\"td\",{parentName:\"tr\",\"align\":null},`50.2%`)))),mdx(\"p\",null,`With this, we can confirm our assumption. Our imaginary model does\nhave a larger bias, but its variance is very low, in contrast to our\nOLS model.\nBut what exactly is the difference between our OLS model and our new,\nimaginary model? Let’s inspect the two functions that were generated by our\ntwo models. We will write down the linear functions `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`f`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`l`),mdx(\"mi\",{parentName:\"mrow\"},`s`)))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f_{ols}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.10764em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.01968em\"}},`l`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`f`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`m`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`n`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`y`)))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f_{imaginary}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.980548em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.311664em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.10764em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`m`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`n`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`f`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`l`),mdx(\"mi\",{parentName:\"mrow\"},`s`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`11.6`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mn\",{parentName:\"mrow\"},`54.3`),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"}),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`f`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`m`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`n`),mdx(\"mi\",{parentName:\"mrow\"},`a`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`y`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`2.76`),mdx(\"mi\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mn\",{parentName:\"mrow\"},`44.36`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`f_{ols}(x) = -11.6x + 54.3 \\\\\\\\\nf_{imaginary}(x) = -2.76x + 44.36`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.10764em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.01968em\"}},`l`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`s`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.72777em\",\"verticalAlign\":\"-0.08333em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`6`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`5`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`4`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`3`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.311664em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.10764em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`m`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`n`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`a`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.72777em\",\"verticalAlign\":\"-0.08333em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`7`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`6`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`4`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`4`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`.`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`3`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`6`)))))),mdx(\"p\",null,`f`),mdx(\"p\",null,`What we can see is that the (absolute) model parameters of our OLS model\nare larger than the parameters of our imaginary model.\nIn fact, the first parameter differs by around `,mdx(\"strong\",{parentName:\"p\"},`320\\\\%`),` between the models!\nSo our findings suggest that the cause for our OLS model overfitting\nare large model parameters. Maybe, if we somehow `,mdx(\"em\",{parentName:\"p\"},`force`),` our model to generate\nsmaller parameters, we can achieve better results and generate a model that looks more\nlike our imaginary one? Let’s try this out and see how it goes.`),mdx(\"h3\",{\"id\":\"regression-with-a-twist\"},`Regression With a Twist`),mdx(\"p\",null,`Ok, so we’ve said that we want to `,mdx(\"em\",{parentName:\"p\"},`force`),` our model to generate smaller model parameters.\nHow can we do that? Pause for a second and see if you can find a way in which we can\nachieve this, without changing up our model `,mdx(\"em\",{parentName:\"p\"},`too`),` much. Got an idea?\nHow does our model know what model parameters it should generate?\nHow does our model know what `,mdx(\"em\",{parentName:\"p\"},`good`),` parameters are and what `,mdx(\"em\",{parentName:\"p\"},`bad`),` parameters are?\nIt gets this information from our loss function. Currently, our loss function is the\nMSE. Recall what it looks like (n is the number of samples in our dataset):`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`)))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mfrac\",{parentName:\"mrow\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`)))),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`)))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`MSE(y,{\\\\color{#26a6ed}y_{pred}}) = \\\\frac{1}{n} \\\\cdot ( (y_i-{\\\\color{#26a6ed}y_{pred}})^T \\\\cdot (y_i-{\\\\color{#26a6ed}y_{pred}}) )`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#26a6ed\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.00744em\",\"verticalAlign\":\"-0.686em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.177439em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#26a6ed\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#26a6ed\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`)))))),mdx(\"p\",null,`If this definition does not look familiar to you, but this one does (`,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mi\",{parentName:\"msub\"},`i`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mi\",{parentName:\"msub\"},`i`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`y_i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.625em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` are just the data points of our dataset):`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`f`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mfrac\",{parentName:\"mrow\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"munderover\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"munderover\"},`∑`),mdx(\"mrow\",{parentName:\"munderover\"},mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`1`)),mdx(\"mi\",{parentName:\"munderover\"},`n`)),mdx(\"mrow\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\"},`f`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mi\",{parentName:\"msub\"},`i`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mn\",{parentName:\"msup\"},`2`)))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`MSE(f) = \\\\frac{1}{n} \\\\cdot \\\\sum_{i=1}^n{(y_i-{\\\\color{#26a6ed}f(x_i)})^2}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.00744em\",\"verticalAlign\":\"-0.686em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.929066em\",\"verticalAlign\":\"-1.277669em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.6513970000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.872331em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel mtight\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.050005em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-symbol large-op\"},`∑`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.3000050000000005em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.277669em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10764em\",\"color\":\"#26a6ed\"}},`f`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8641079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))))))))))))),mdx(\"div\",null,mdx(\"p\",null,`then I recommend you take a look at the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/vectorization\"},`Vectorization Explained, Step by Step`),`, where\nwe take this familiar definition and transform it into the vectorized form above, which can be computed `,mdx(\"em\",{parentName:\"p\"},`significantly`),` faster`,mdx(ToolTip,{trigger:\"*\",mdxType:\"ToolTip\"},`To be more precise, it is about 350 times faster.`),`.`)),mdx(\"p\",null,`This MSE is the function that we, or rather, our model, is trying to minimize. Right now, it only contains the mean sum of\nsquared residuals. But if we add another term to this function, our model will then have to minimize\n`,mdx(\"em\",{parentName:\"p\"},`both`),` the mean sum of squared residuals, as well as our additional term at the same time.\nLet’s call this additional term `,mdx(\"em\",{parentName:\"p\"},`penalty`),`.\nSo which penalty do we add? We know that we want to `,mdx(\"em\",{parentName:\"p\"},`minimize`),` our model parameters,\nand our model `,mdx(\"em\",{parentName:\"p\"},`minimizes`),` the loss function, so let’s just add the model parameters into the equation, shall we?\nOur model has parameters `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\theta_0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` (intercept) and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mn\",{parentName:\"msub\"},`1`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\theta_1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` (slope), so let’s add them to our loss like this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`N`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`w`),mdx(\"mtext\",{parentName:\"mrow\"},` `),mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mn\",{parentName:\"msub\"},`0`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mn\",{parentName:\"msub\"},`1`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nNew\\\\ Loss(y,y_{pred}) = MSE(y,y_{pred}) + \\\\theta_0 + \\\\theta_1\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`N`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02691em\"}},`w`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"}))))))))))),mdx(\"p\",null,`More broadly, if we have `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`m`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`m`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`m`))))),` model parameters, our new loss will be:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`N`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`w`),mdx(\"mtext\",{parentName:\"mrow\"},` `),mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"munderover\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"munderover\"},`∑`),mdx(\"mrow\",{parentName:\"munderover\"},mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`1`)),mdx(\"mi\",{parentName:\"munderover\"},`m`)),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mi\",{parentName:\"msub\"},`i`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nNew\\\\ Loss(y,y_{pred}) = MSE(y,y_{pred}) + \\\\sum_{i=1}^m{\\\\theta_i}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`N`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02691em\"}},`w`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.929066em\",\"verticalAlign\":\"-1.277669em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.6513970000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.872331em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel mtight\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.050005em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-symbol large-op\"},`∑`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.3000050000000005em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`m`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.277669em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.31166399999999994em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))))),mdx(\"div\",null,mdx(\"p\",null,`Ok! There’s just one problem. Since our model parameters can be negative, adding them might `,mdx(\"em\",{parentName:\"p\"},`decrease`),`\nour loss instead of increasing it. We encountered a similar problem when we built linear regression in\n`,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression/#the-problem-with-the-sor\"},`Linear Regression Explained, Step by Step`),`.\nIn order to circumvent this, we can either square our model parameters or take their absolute values.\nIn this post, we’ll only take a look at the square of the sum of model parameters.`,mdx(ToolTip,{mdxType:\"ToolTip\"},`Spoiler: We’ll come back to the sum of absolute values in the `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/lasso_regression\"},`article about lasso regression`),`. 😉`))),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`N`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`w`),mdx(\"mtext\",{parentName:\"mrow\"},` `),mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"munderover\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"munderover\"},`∑`),mdx(\"mrow\",{parentName:\"munderover\"},mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`1`)),mdx(\"mi\",{parentName:\"munderover\"},`m`)),mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\"},`θ`),mdx(\"mi\",{parentName:\"msubsup\"},`i`),mdx(\"mn\",{parentName:\"msubsup\"},`2`)),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"})),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nNew\\\\ Loss(y,y_{pred}) = MSE(y,y_{pred}) + \\\\sum_{i=1}^m{\\\\theta_i^2} \\\\\\\\\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`N`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02691em\"}},`w`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.929066em\",\"verticalAlign\":\"-1.277669em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-limits\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.6513970000000002em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.872331em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel mtight\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.050005em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mop op-symbol large-op\"},`∑`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.3000050000000005em\",\"marginLeft\":\"0em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.05em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`m`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.277669em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8641079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4530000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`i`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.247em\"}},mdx(\"span\",{parentName:\"span\"})))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}))))),mdx(\"p\",null,`We can vectorize our new loss function since the last part is just the dot product of our parameter vector `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`\nwith itself:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`N`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`w`),mdx(\"mtext\",{parentName:\"mrow\"},` `),mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mrow\"},`θ`),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"})),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nNew\\\\ Loss(y,y_{pred}) = MSE(y,y_{pred}) + \\\\boldsymbol{\\\\theta}^T\\\\theta \\\\\\\\\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`N`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02691em\"}},`w`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.9256709999999999em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9256709999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1473400000000002em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}))))),mdx(\"p\",null,`If you are unsure about this, just quickly make an exemplary calculation and confirm that this really does not change our loss function.\nIf you are familiar with `,mdx(\"em\",{parentName:\"p\"},`norms`),` in math, then you could say that this new loss contains a squared `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`l`),mdx(\"mn\",{parentName:\"msub\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`l_2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.01968em\"}},`l`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.01968em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`-norm (or Euclidian norm),\nThis is also the reason why this term is sometimes referred to as an `,mdx(\"em\",{parentName:\"p\"},`L2 regularization`),`.\nThis means that we can rewrite our new loss like this (a norm\nis denoted using two vertical lines, like this: ||):`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`N`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`w`),mdx(\"mtext\",{parentName:\"mrow\"},` `),mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msubsup\"},`2`),mdx(\"mn\",{parentName:\"msubsup\"},`2`)),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"})),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nNew\\\\ Loss(y,y_{pred}) = MSE(y,y_{pred}) + ||\\\\boldsymbol{\\\\theta}||_2^2 \\\\\\\\\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`N`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02691em\"}},`w`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.1141079999999999em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8641079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4530000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.247em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}))))),mdx(\"div\",null,mdx(\"p\",null,`Now, let’s perform one final change. Right now, we can’t control how much the model parameters should be factored into the\nequation. We want our model to minimize the MSE and the model parameters, but maybe one is more important than the other\nin a particular case? What if we want our model to focus most of its attention on the MSE and only pay a small bit of attention\nto the model parameters? To be able to control how much our new penalty term weighs into the equation, let’s add an\nadditional parameter `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),mdx(ToolTip,{width:\"20em\",mdxType:\"ToolTip\"},`Depending on the source, this parameter is sometimes also called `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`λ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\lambda`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`λ`))))),`.\nThis name is especially common in some popular books like Elements of Statistical Learning. Scikit-Learn uses the name `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),`, so that’s what I’ll be calling it as well.`),` that controls exactly that:`)),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`N`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`w`),mdx(\"mtext\",{parentName:\"mrow\"},` `),mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\"},mdx(\"msub\",{parentName:\"mi\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold-italic\"},`i`))),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msub\"},`1`)),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"})),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nNew\\\\ Loss(y,y_{pred}) = MSE(y,y_{pred}) + \\\\alpha \\\\cdot ||\\\\boldsymbol{\\\\theta_i}||_1 \\\\\\\\\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`N`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02691em\"}},`w`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.036108em\",\"verticalAlign\":\"-0.286108em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.44445em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33528199999999997em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03194em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol mtight\"},`i`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}))))),mdx(\"div\",null,mdx(\"p\",null,`Ok, enough math for now. Let’s solve the mystery and stop calling this metric `,mdx(\"em\",{parentName:\"p\"},`new loss`),`.\nWhat you see in front of you is exactly the loss function of ridge regression!`,mdx(ToolTip,{trigger:\"*\",mdxType:\"ToolTip\"},`Sometimes it’s also called Tikhonov regularization, although the term `,mdx(\"em\",{parentName:\"p\"},`ridge regression`),` is used far more frequently.`),`\nIn fact, this `,mdx(\"em\",{parentName:\"p\"},`imaginary`),` model we have been looking at so far was in truth just ridge regression!\nAwesome! As you see, ridge doesn’t differ that much from OLS regression. The only difference is the\nadded penalty which prevents the model parameters from becoming very large (or very large in the negative values, i.e. very small).\nThe only downside to this is that, since our model now has to minimize two terms instead of just one,\nit can’t minimize the MSE as effectively as OLS regression. This is the reason why ridge models\nwill have a slightly higher bias than OLS regression models. However, they compensate for that by having\nsignificantly lower variances than OLS models!\nNow that we have discovered ridge regression, let’s try to analyze its behaviour in a more `,mdx(\"em\",{parentName:\"p\"},`visual`),` way.`)),mdx(\"h3\",{\"id\":\"ols-vs-ridge\"},`OLS vs Ridge`),mdx(\"p\",null,`With the intuition and math now covered, let’s try to visually grasp the fundamental difference between OLS and ridge.\nMore specifically, let’s try to understand the difference between the two\nby looking at how strongly these models react to change in data. Since ridge has a\npenalty term in its loss function, it is not so sensitive to changes in the training data\nwhen compared to OLS regression, because ridge has to make sure that the penalty term stays\nsmall. However, OLS regression has no penalty term, which means that it will minimize only the MSE, with disregard\nto the size of its model weights. So what this means intuitively is that, if we slightly change our training data,\nour ridge model might only change its parameters a little bit, while our OLS model parameters will change a lot more.\nLet’s take just two data points and see how our two models change when we slightly shift the position\nof our data points. Below you can find a plot that shows the OLS model for the two data points. At the top of the\nplot, there is a button which you can press to slightly shift the data points across the x-dimension.\nAfter each press, a new OLS model will be displayed, while the old model will take on a more transparent\ntone. Press the button a couple of times and notice how (strongly) the functions differ from each other.`),mdx(ConditionalLazyPlotInteractive,_extends({},olsWiggle,{mdxType:\"ConditionalLazyPlotInteractive\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"978px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"62%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAABYlAAAWJQFJUiTwAAABVklEQVQoz3VS2ZKCMBC0SuUUwqECYgJBAujWPu3/f1tvZTSaZeGhKzPJpDPp6c0+bbBLBPQaX0Yk1ztS/kB6fYDVE5L6jkx8wcvlu87Azk28MYGTtQjONxwKhUM5EMJCIaoGgj43tXpdi9+Ebt5iy/gq7EtO1sA/dXB0d0nz5zEidHNJhPMvzbGjhyX8Ywc3a+GdOhxqBb+6/Sf0jnKVkPYTQTrqzsKqR8QHhLyHV0rsM0tD3a7BkuhbJqh7xh+IxYRIKAR1D+f4rN+xmYa6dfdF+CYi3QScvEXMRyRyQtp9I6jUP5LFKZOGukP23AzqEbEYkHV3RFdFxDQYxtf1tQm1yP65g1tKxM2IYvpBdFE0wW3MF/32sUzzwsyH4aVHzAdEpSI7fPz5gtXNPF/8cnju4aZPoSkvFE3WmP6jsUBwulGNIdOxGerGdrvtenvfdsHSuZ3/AjLlJOpy48fYAAAAAElFTkSuQmCC')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/2aa7171b817dee1c8423e5e188712fbc/601b1/OLSSensitivity.webp 500w\",\"/static/2aa7171b817dee1c8423e5e188712fbc/b9625/OLSSensitivity.webp 978w\"],\"sizes\":\"(max-width: 978px) 100vw, 978px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/2aa7171b817dee1c8423e5e188712fbc/0eb09/OLSSensitivity.png 500w\",\"/static/2aa7171b817dee1c8423e5e188712fbc/9b76f/OLSSensitivity.png 978w\"],\"sizes\":\"(max-width: 978px) 100vw, 978px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/2aa7171b817dee1c8423e5e188712fbc/9b76f/OLSSensitivity.png\",\"alt\":\"Sensitivity of OLS visualized\",\"title\":\"Sensitivity of OLS visualized\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `))),mdx(\"p\",null,`We can see that even if we only change our data points by just 1 point in the x-direction, our OLS model\ndoes change its shape quite a bit. Let’s create this same plot once more, but this time with ridge regression.\nTry to compare the functions generated by ridge to the ones generated by OLS.`),mdx(ConditionalLazyPlotInteractive,_extends({},ridgeWiggle,{mdxType:\"ConditionalLazyPlotInteractive\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"976px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"64.2%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABe0lEQVQ4y3VT2XKDMAxkAuFIGjDmaDKBYAwloZO0fev/f9l2ZGMPpOFBYyEkeVcrO9u0gccuoDPKJcJcqjMqtK++iw5B1to8Y8/fZI754XOBUn6jbL+QNw8U4oG8uSujGDuPtgGdc3/RkAJBJhBwgU1cwU1qdW7ievJrG19DpeKpvsQxCf6LpH9FM2Q2llyUmf+qIc0ozNvV29WFXGif6UI30c3DUiIqpa5lE2VK9vkKHdbAJfqHSjXxM4Ho2CERV6TtCC5HHKrB1jqEjBSkOc6bGCoeqxE3d2TDD5i4gstPpGLE/thrVolGaynP56OQsApB0WJ/7icUN/DujqS+qQYGsZcs52kRkrMvewX70Aw4PX6RyhFxdcXuvVPqu5PiVkC+LpqdYXwe8Hb6QFT0msYLFEu1G2XP6k972CKklzChoHkSPX9aJy2asKtF+UZ1f7YBFqFqmEtsp+Ukf1d2M4piUbR8ihd7gRXFQPZeLLB9SZn49/Q8Nvc15T+ibj1VC/oWiwAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/b76bb29ce5c05aa3446610c01d587183/601b1/ridgeSensitivity.webp 500w\",\"/static/b76bb29ce5c05aa3446610c01d587183/91b06/ridgeSensitivity.webp 976w\"],\"sizes\":\"(max-width: 976px) 100vw, 976px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/b76bb29ce5c05aa3446610c01d587183/0eb09/ridgeSensitivity.png 500w\",\"/static/b76bb29ce5c05aa3446610c01d587183/31384/ridgeSensitivity.png 976w\"],\"sizes\":\"(max-width: 976px) 100vw, 976px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/b76bb29ce5c05aa3446610c01d587183/31384/ridgeSensitivity.png\",\"alt\":\"Sensitivity of Ridge visualized\",\"title\":\"Sensitivity of Ridge visualized\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `))),mdx(\"p\",null,`Ok, there’s clearly a difference there! We can see that the functions generated by ridge vary only slightly\nin their parameters, while the functions generated by OLS differ a lot more.\nWe can see how ridge is less sensitive to changes in the input\ndata than OLS regression. Therefor we can conclude that ridge models have a lower variance\nthan OLS models. However, because of the penalty term, they usually have a slightly higher bias than OLS models.`),mdx(\"h2\",{\"id\":\"solving-ridge-regression\"},`Solving Ridge Regression`),mdx(\"p\",null,`We have now talked at length about ridge regression and how it differs from\nOLS regression. But how can you actually\ndetermine the model parameters `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`?\nJust like with `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression/#finding-the-best-function-visualize\"},`linear regression`),`,\nwe have a variety of methods at our disposal!\nThe good news is that\nwe can solve ridge regression with the same methods and techniques we used to solve\nOLS regression! These were 1. solving the normal equation and 2. using gradient descent.\nLet’s look at both options!`),mdx(\"h4\",{\"id\":\"normal-equation\"},`Normal Equation`),mdx(\"p\",null,`The good news here is that there is a normal equation for ridge regression.\nLet’s recall how the normal equation looked like for regular OLS regression:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mover\",{parentName:\"mrow\",\"accent\":\"true\"},mdx(\"mi\",{parentName:\"mover\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mover\"},`^`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`X`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mrow\",{parentName:\"msup\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`1`))),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`y`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\hat{\\\\boldsymbol{\\\\theta}} = (\\\\mathbf{X}^T\\\\mathbf{X})^{-1}\\\\mathbf{X}^T \\\\mathbf{y}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.9578799999999998em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord accent\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9578799999999998em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.26344em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"accent-body\",\"style\":{\"left\":\"-0.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`^`))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.1413309999999999em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.864108em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\",\"style\":{\"marginRight\":\"0.01597em\"}},`y`))))))),mdx(\"p\",null,`We can derive the above equation by setting the derivative of the cost function of linear\nregression (the MSE) equal to zero and solving for our model parameters.\nWe do the same thing for ridge regression. Our loss looks like this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mtable\",{parentName:\"semantics\",\"rowspacing\":\"0.24999999999999992em\",\"columnalign\":\"right left\",\"columnspacing\":\"0em\"},mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"mrow\"},`R`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`d`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mtext\",{parentName:\"mrow\"},` `),mdx(\"mi\",{parentName:\"mrow\"},`L`),mdx(\"mi\",{parentName:\"mrow\"},`o`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mi\",{parentName:\"mrow\"},`s`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mtext\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\"},`M`),mdx(\"mi\",{parentName:\"mstyle\"},`S`),mdx(\"mi\",{parentName:\"mstyle\"},`E`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mtext\",{parentName:\"mstyle\",\"mathvariant\":\"bold\"},`y`),mdx(\"mo\",{parentName:\"mstyle\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mtext\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∣`),mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\",\"mathvariant\":\"normal\"},`∣`),mdx(\"mn\",{parentName:\"msubsup\"},`2`),mdx(\"mn\",{parentName:\"msubsup\"},`2`)))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mtext\",{parentName:\"mstyle\",\"mathvariant\":\"bold\"},`y`),mdx(\"mtext\",{parentName:\"mstyle\"},`−`),mdx(\"mtext\",{parentName:\"mstyle\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"msup\",{parentName:\"mstyle\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mtext\",{parentName:\"mstyle\",\"mathvariant\":\"bold\"},`y`),mdx(\"mtext\",{parentName:\"mstyle\"},`−`),mdx(\"mtext\",{parentName:\"mstyle\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)))))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\begin{aligned}\n\nRidge\\\\ Loss(\\\\textbf{y},\\\\textbf{y}_{pred}) &= {\\\\color{#26a6ed}MSE(\\\\textbf{y},\\\\textbf{y}_{pred})} + \\\\alpha \\\\cdot ||\\\\boldsymbol{\\\\theta}||_2^2 \\\\\\\\\n\n&= {\\\\color{#26a6ed}(\\\\textbf{y} − \\\\textbf{X}\\\\boldsymbol{\\\\theta})^T (\\\\textbf{y} − \\\\textbf{X}\\\\boldsymbol{\\\\theta})} + \\\\alpha\\\\boldsymbol{\\\\theta}^T \\\\boldsymbol{\\\\theta}\n\n\\\\end{aligned}\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"3.130027em\",\"verticalAlign\":\"-1.3150135em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mtable\"},mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.8150135em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.9509055em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.00773em\"}},`R`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`d`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\"},` `),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`L`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`o`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`s`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`y`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`y`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.2419679999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4558600000000004em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.380248em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.3449865em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.3150135em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.8150135em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.9509055em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\",\"color\":\"#26a6ed\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\",\"color\":\"#26a6ed\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\",\"color\":\"#26a6ed\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\",\"style\":{\"color\":\"#26a6ed\"}},`y`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\",\"style\":{\"color\":\"#26a6ed\"}},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\",\"style\":{\"color\":\"#26a6ed\"}},`y`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.2419679999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4558600000000004em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#26a6ed\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.380248em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∣`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8641079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4530000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.247em\"}},mdx(\"span\",{parentName:\"span\"})))))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.3449865em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\",\"style\":{\"color\":\"#26a6ed\"}},`y`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\",\"style\":{\"color\":\"#26a6ed\"}},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#26a6ed\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#26a6ed\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\",\"style\":{\"color\":\"#26a6ed\"}},`y`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\",\"style\":{\"color\":\"#26a6ed\"}},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#26a6ed\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9256709999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1473400000000002em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`)))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.3150135em\"}},mdx(\"span\",{parentName:\"span\"})))))))))))),mdx(\"p\",null,`We now differentiate this function, set it equal to zero, solve for `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` and we get:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mover\",{parentName:\"mrow\",\"accent\":\"true\"},mdx(\"mi\",{parentName:\"mover\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mover\"},`^`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mtext\",{parentName:\"msup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`X`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`I`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mrow\",{parentName:\"msup\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`1`))),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mtext\",{parentName:\"msup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`y`),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"})),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\hat{\\\\boldsymbol{\\\\theta}} = (\\\\textbf{X}^T\\\\textbf{X} + \\\\alpha\\\\textbf{I})^{-1} \\\\textbf{X}^T \\\\textbf{y} \\\\\\\\\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.9578799999999998em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord accent\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9578799999999998em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.26344em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"accent-body\",\"style\":{\"left\":\"-0.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`^`))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.167341em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.917341em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1390100000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.167341em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`I`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.864108em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.917341em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1390100000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`y`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}))))),mdx(\"div\",null,mdx(\"p\",null,`where `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`I`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\textbf{I}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68611em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`I`)))))),` is an identity matrix with dimensions `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`n`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mn\",{parentName:\"mrow\"},`1`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mtext\",{parentName:\"mrow\"},`x`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`n`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mn\",{parentName:\"mrow\"},`1`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`(n+1) \\\\text{x} (n+1)`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`x`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))))),mdx(ToolTip,{mdxType:\"ToolTip\"},`A matrix full of zeros except for the diagonal, which contains only ones.`),`,\nwith a zero in the top left corner, to account for the bias (or intercept) term.\nThis mathematical derivation was pretty short, but if you’d like to read a more detailed article about\nhow exactly we can derive this normal equation, let me know in the comments below, and if there is\nenough demand then I’ll provide a more detailed walkthrough!`)),mdx(\"p\",null,`As with the normal equation for linear regression, the solution to this equation gives\nus the ideal parameters for our `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),` directly, in one step.\nIf you’ve read the `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression/#complexity\"},`section about complexity in the article about linear regression`),`,\nthen you’ll know that solving the normal equation is your best bet in most cases.\nHowever, if you have a very large number of features, you might be better off using\nan iterative algorithm, like gradient descent. Let’s take a look.`),mdx(\"h4\",{\"id\":\"gradient-descent\"},`Gradient Descent`),mdx(\"p\",null,`Alternatively to solving the normal equation, we can differentiate the ridge cost function to obtain its partial derivatives (i.e. compute its gradient)\nand then perform `,mdx(\"em\",{parentName:\"p\"},`gradient descent`),` to iteratively approach the optimal solution.\nIf you are interested in learning about how gradient descent works and how you can implement it in\nraw Python as well as with scikit-learn, I recommend that you read the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent\"},`Gradient Descent Explained, Step by Step`),`,\nwhere you will learn how to do exactly that. The implementation of gradient descent for ridge regression\nis very similar to gradient descent for linear regression, and in fact the only things that change\nare how we compute the gradients and our loss function.`),mdx(\"h2\",{\"id\":\"implementing-it-in-python\"},`Implementing it in Python`),mdx(\"p\",null,`Now that we’ve briefly looked at the two main methods of solving ridge regression, let’s also implement\nthem in Python!\nFirst we’ll implement it from scratch and then we’ll look at how we can make our lives easier by using scikit-learn.`),mdx(\"h3\",{\"id\":\"implementing-the-normal-equation\"},`Implementing the Normal Equation`),mdx(\"p\",null,`Similar to the normal equation for OLS regression, we can implement the normal equation for ridge regression as follows.\nBelow are both the normal equation as well as the implementation in code.`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mover\",{parentName:\"mrow\",\"accent\":\"true\"},mdx(\"mi\",{parentName:\"mover\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mover\"},`^`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mtext\",{parentName:\"msup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`X`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`I`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mrow\",{parentName:\"msup\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`1`))),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mtext\",{parentName:\"msup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mtext\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`y`),mdx(\"mspace\",{parentName:\"mrow\",\"linebreak\":\"newline\"})),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\n\\\\hat{\\\\boldsymbol{\\\\theta}} = (\\\\textbf{X}^T\\\\textbf{X} + \\\\alpha\\\\textbf{I})^{-1} \\\\textbf{X}^T \\\\textbf{y} \\\\\\\\\n\n`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.9578799999999998em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord accent\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9578799999999998em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.26344em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"accent-body\",\"style\":{\"left\":\"-0.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`^`))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.167341em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.917341em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1390100000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.167341em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`I`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.864108em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.917341em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1390100000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord textbf\"},`y`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace newline\"}))))),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def normal_equation_ridge_regression(X,y,alpha):\n intercept_ones = np.ones((len(X),1)) # results in array( [ [1],..,[1] ] )\n X_b = np.c_[intercept_ones,X] # we now add the additional ones as a new column to our X\n I = np.identity(X_b.shape[1]) # identity matrix with dimensions (n+1)\n I[0][0] = 0 # adjusting the first value in I to be 0, to account for the intercept term\n theta_optimal = np.linalg.inv(X_b.T.dot(X_b) + alpha * I).dot(X_b.T).dot(y) # the normal equation\n return theta_optimal\n`)),mdx(\"p\",null,`If you are wondering what exactly `,mdx(\"inlineCode\",{parentName:\"p\"},`X_b`),` means and why we have to add “intercept_ones” to our X,\nI recommend you read `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression/#adjustment-vectorization\"},`this section of the article about linear regression`),`,\nwhere this step is explained in detail.\nIf we then run the function on our training dataset, we get:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`theta_ridge_ne_1 = normal_equation_ridge_regression(X_train, y_train, 1)\nprint(theta_ridge_ne_1)\n# output: [44.35714286 -2.76190476]\n`)),mdx(\"p\",null,`We can do the same thing with scikit-learns `,mdx(\"a\",{parentName:\"p\",\"href\":\"https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html\",\"target\":\"_blank\",\"rel\":\"nofollow noopener noreferrer\"},mdx(\"inlineCode\",{parentName:\"a\"},`Ridge`)),`-class:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`ridge = Ridge() # same as Ridge(alpha=1)\nridge.fit(X_train, y_train, 1)\ntheta_ridge_ne_2 = np.array([ridge.intercept_, ridge.coef_[0]])\nprint(theta_ridge_ne_2)\n# output: [44.35714286 -2.76190476]\n`)),mdx(\"p\",null,`As you can see, it computes the exact same model parameters, awesome!\nLet’s plot our function and see how it does. We’ll also plot the OLS model alongside it,\nto more easily compare the two. Since our models produce linear functions, we only\nneed two data points to plot them. We’ll use the minimum and the maximum of our X to do that,\nbut we could also take any two other distinct points of X.`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`X_interval = np.array([min(X), max(X)]) # we'll use these points to plot our linear functions\n\nplt.plot(X_train, y_train, \"s\", markersize=12, label=\"train data\")\nplt.plot(X_test, y_test, \".\", markersize=15, label=\"test data\")\n\nplt.plot(X_interval, ols.predict(X_interval), label=\"ols\")\nplt.plot(X_interval, ridge.predict(X_interval) ,label=\"ridge\")\nplt.legend()\n`)),mdx(\"p\",null,`This yields the following plot:`),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"432px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"66.66666666666666%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAABn0lEQVQ4y5VT227bMAzN///Qngf0cdgeChhNhwbIpXFcy5Zl60JHlzNQgxOrwbCNAG2aoo7JQ3KTUsJaYoxg39/0syy+DT/8THg7XfC1OmbbGPNHtcZg0hrfdi0+WoGmuaCua4QQ7oAzEc6NwEvTwPjp4a+fxceE71WF/X6P7fYVu92uBHTOQQ0yO0xw6IjtcAMtSk0J3ntEanE6HjAMKmd+vV5XGc4zBqXgxxFx1MwBnqpXvF3OJU8LzykhGkInRL5HRBnjBsgOOQw5g0AEGI3jSaCpa6j2gKF5hxEDl4JgDCIR1mQ4Z+Gv/g5oidB/nIHpUmSyyEF2+PK8RTQjknEI0wQvZdYge9hTBR9iyWHXdUCKN55iLMcjIkDOCtqbh6Y55tD7kkMp5RK1agDKuUvAnAg/fj7j3Il7yeTKpjCHfd//80BbqyG0gCKVv8nRY5dvGf6HzHGGnCW01YhrDhlQKZV55EyZUyEExnGEtRZt20JrneeNbX7zWSe6HNu0Dabp90Js1sTzZQ5Y7KUMtnkTeM8ZLI9XCNlm4bjl3i/BXPbjgteMiQAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/7c0098e4934a4a144ed55adc0150d386/e3a5e/olsVsRidgePlot.webp 432w\"],\"sizes\":\"(max-width: 432px) 100vw, 432px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/7c0098e4934a4a144ed55adc0150d386/dc3b1/olsVsRidgePlot.png 432w\"],\"sizes\":\"(max-width: 432px) 100vw, 432px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/7c0098e4934a4a144ed55adc0150d386/dc3b1/olsVsRidgePlot.png\",\"alt\":\"OLS vs Ridge plot\",\"title\":\"OLS vs Ridge plot\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `)),mdx(\"p\",null,`This is also how I generated the plot between the OLS model\nand the “imaginary” model at the start of this article.\nI think this plot shows the difference between OLS and ridge very well.\nOur training points very much represent the shape of a linear function.\nOLS regression takes full advantage of this and therefor generates a linear function that almost perfectly\nhits every training point. Ridge regression, however, also cares about\nsmall model weights, so it’s not as steep as the OLS model.\nThe ridge line is a lot closer to our testing points than our OLS model is.\nOf course, we could take any combination of training and testing points.\nOLS would always generate the function with a minimal training error,\nwhile ridge would be a little bit more cautious. The result is that our OLS model\nhas low bias and high variance, and our ridge model has a slightly higher bias,\nbut a much lower variance. Ok! Now let’s tackle gradient descent.`),mdx(\"h3\",{\"id\":\"implementing-gradient-descent\"},`Implementing Gradient Descent`),mdx(\"p\",null,`We can reuse the gradient descent function that we’ve implemented in the\n`,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent_for_linear_regression\"},`article about Gradient Descent for Linear Regression`),`.\nHere’s how the code looked like:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def gradient_descent(X, y, theta, criterion, number_of_iterations, learning_rate):\n X_b = add_intercept_ones(X)\n loss_history=[]\n for i in range(number_of_iterations):\n\n # predict and calculate loss\n f = create_function(theta) # create the current function\n y_predicted = f(X_b) # predict our entire x\n loss = criterion(y,y_predicted) # calculate the error\n loss_history.append(loss)\n\n # perform optimization\n gradient = np.array([mse_derivative_b(y,y_predicted), mse_derivative_m(x,y,y_predicted)]) # calculate gradient\n theta = theta - learning_rate * gradient #adjust m and b\n\n if i%10==0:\n print(\"Current Epoch: {}, Current Loss: {}\".format(i,loss))\n\n return theta,loss_history\n`)),mdx(\"p\",null,`If this code doesn’t make a lot of sense to you, I recommend you give the `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent_for_linear_regression\"},`article about Gradient Descent for Linear Regression`),`,\na try, as we build out this function step by step in that article.`),mdx(\"p\",null,`Now we have to make some slight adjustments. First off we’ll have to define our new loss function\nand a new gradient function. Our loss is just the regular MSE with the added ridge penalty. So\nwe can reuse our vectorized MSE implementation from the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/vectorization/#example-2-mse\"},`Vectorization Explained, Step by Step`),`\nand create a new function:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def mse(y, y_predicted):\n error = y-y_predicted\n loss = 1/(y.size) * np.dot(error.T, error)\n return loss\n\ndef ridgeMSE(y, y_predicted, alpha, theta):\n mse = mse(y, y_predicted)\n ridge_mse = mse(y, y_predicted) + alpha * np.dot(theta,theta)\n return ridge_mse\n`)),mdx(\"p\",null,`This is alright, but when we will later pass our `,mdx(\"inlineCode\",{parentName:\"p\"},`ridgeMSE`),`-function to our gradient descent algorithm,\nwe will somehow have to provide a value for `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),`. We could use a default argument, but then we could\nonly pick one fixed value. Alternatively, we can pass in an argument for `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),` to our gradient descent function.\nBoth of these methods are not extremely fun to work with, so what we’ll do instead is create a higher-order function\nthat will receive a value for `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),` and return to us our `,mdx(\"inlineCode\",{parentName:\"p\"},`ridgeMSE`),`-function. It looks like this:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def getRidgeMSEFunction(alpha=0.0001):\n\n def ridgeMSE(y, y_predicted, theta):\n mse_loss = mse(y, y_predicted)\n ridge_loss = mse_loss + alpha * np.dot(theta,theta)\n return ridge_loss\n\n return ridgeMSE\n`)),mdx(\"p\",null,`Since our value for `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),` won’t change during training, this approach works very well.\nIt might look a bit confusing at first because we are defining a function inside of a function,\nbut once you understand what’s going on, this approach can be quite handy.`),mdx(InfoTip,{mdxType:\"InfoTip\"},mdx(\"p\",null,mdx(\"strong\",{parentName:\"p\"},`Different Values for `,mdx(\"inlineCode\",{parentName:\"strong\"},`alpha`),`.`)),mdx(\"p\",null,`You might be asking yourself why we set `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha=1`),` when we used the normal\nequation and now we set `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha=0.0001`),` when using gradient descent.\nThe reason is that with the normal equation, we perform the entire computation\nin one step, so our regularization parameter immediately affects every data point.\nAlso, every data point is only affected once.`),mdx(\"p\",null,`With gradient descent, we solve our ridge regression iteratively.\nThis means that at every iteration, we only want to apply a tiny\nbit of regularization, instead of applying it all at once.`),mdx(\"p\",null,`An analogy would be cooking. Imagine you are making a soup\nand you want to add some seasoning to it. You have two options. You can either make\na bold move and add all of your seasoning in at once,\nor you could cook the soup a bit, then add in a tiny amount of seasoning,\ntaste it, add in some more, taste it, etc.\nBut if you add in all of your seasoning at once, taste the soup,\nand then `,mdx(\"em\",{parentName:\"p\"},`again`),` add in the same amount of seasoning, your soup will not be very tasty.\nThe seasoning would completely dominate the flavor of the soup,\nand the actual soup would not be able to present its taste very well anymore.`),mdx(\"p\",null,`With our gradient descent algorithm, it’s quite similar!\nIf we would use gradient descent with `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha=1`),`,\ni.e. we were to add a large amount of seasoning at every iteration step,\nwe get the following model parameters: `,mdx(\"inlineCode\",{parentName:\"p\"},`[16.578125 14.5625 ]`),`.\nThe regularization now dominates our loss and therefor our first model parameter is\nnot being able to grow as large as it should (to around 45), and it is instead\nkept at a rather low value. This results in our second model parameter also having\nto adjust to a suboptimal value.`),mdx(\"p\",null,`This is also why the default value for `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),` is 1 in scikit-learn’s `,mdx(\"inlineCode\",{parentName:\"p\"},`Ridge`),`-class,\nbut is only 0.0001 in scikit-learn’s `,mdx(\"inlineCode\",{parentName:\"p\"},`SGDRegressor`),`-class.`)),mdx(\"p\",null,`Now we’ll also have to define a function to compute our gradients.\nWe can compute the gradient by differentiating our loss function, the ridge MSE.\nOur ridge-MSE looks like this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mtable\",{parentName:\"semantics\",\"rowspacing\":\"0.24999999999999992em\",\"columnalign\":\"right left\",\"columnspacing\":\"0em\"},mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"mrow\"},`R`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`d`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\",\"separator\":\"true\"},`,`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`))),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mi\",{parentName:\"mstyle\"},`M`),mdx(\"mi\",{parentName:\"mstyle\"},`S`),mdx(\"mi\",{parentName:\"mstyle\"},`E`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`y`),mdx(\"mo\",{parentName:\"mstyle\",\"separator\":\"true\"},`,`),mdx(\"mstyle\",{parentName:\"mstyle\",\"mathcolor\":\"#9628d9\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`)))),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#54C667\"},mdx(\"mi\",{parentName:\"mstyle\"},`α`),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"msup\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`)))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mfrac\",{parentName:\"mstyle\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`y`),mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"mstyle\",{parentName:\"mstyle\",\"mathcolor\":\"#9628d9\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`)))),mdx(\"msup\",{parentName:\"mstyle\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`y`),mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"mstyle\",{parentName:\"mstyle\",\"mathcolor\":\"#9628d9\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`)))),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#54C667\"},mdx(\"mi\",{parentName:\"mstyle\"},`α`),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"msup\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`)))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mfrac\",{parentName:\"mstyle\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`y`),mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"mstyle\",{parentName:\"mstyle\",\"mathcolor\":\"#9628d9\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"msup\",{parentName:\"mstyle\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`y`),mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"mstyle\",{parentName:\"mstyle\",\"mathcolor\":\"#9628d9\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#54C667\"},mdx(\"mi\",{parentName:\"mstyle\"},`α`),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"msup\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`))))))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\begin{aligned}\nRidgeMSE(y,y_{pred}) &= {\\\\color{#26a6ed}MSE(y,{\\\\color{9628d9}y_{pred}})} + {\\\\color{#54C667}\\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta}^T\\\\boldsymbol{\\\\theta}} \\\\\\\\\n&= {\\\\color{#26a6ed}\\\\frac{1}{n} \\\\cdot ( (y-{\\\\color{9628d9}y_{pred}})^T (y-{\\\\color{9628d9}y_{pred}}) )} + {\\\\color{#54C667}\\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta}^T\\\\boldsymbol{\\\\theta}} \\\\\\\\\n&= {\\\\color{#26a6ed}\\\\frac{1}{n} \\\\cdot ( (y- {\\\\color{9628d9}\\\\mathbf{X}_b \\\\boldsymbol{\\\\theta}} )^T (y- {\\\\color{9628d9}\\\\mathbf{X}_b \\\\boldsymbol{\\\\theta}}) )} + {\\\\color{#54C667}\\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta}^T\\\\boldsymbol{\\\\theta}}\n\\\\end{aligned}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"6.200551000000001em\",\"verticalAlign\":\"-2.8502755000000004em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mtable\"},mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"3.3502755em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-5.7460445em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.00773em\"}},`R`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`d`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\"},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.7646045em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.4571644999999998em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.8502755000000004em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"3.3502755em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-5.7460445em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\",\"color\":\"#26a6ed\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\",\"color\":\"#26a6ed\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\",\"color\":\"#26a6ed\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mpunct\",\"style\":{\"color\":\"#26a6ed\"}},`,`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.16666666666666666em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#9628d9\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#9628d9\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\",\"color\":\"#54C667\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#54C667\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9256709999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1473400000000002em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#54C667\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`)))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.7646045em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"color\":\"#26a6ed\",\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#9628d9\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#9628d9\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#26a6ed\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#9628d9\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#9628d9\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\",\"color\":\"#54C667\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#54C667\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9256709999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1473400000000002em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#54C667\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`)))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-1.4571644999999998em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"color\":\"#26a6ed\",\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\",\"style\":{\"color\":\"#9628d9\"}},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#9628d9\"}},`θ`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#26a6ed\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\",\"style\":{\"color\":\"#9628d9\"}},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#9628d9\"}},`θ`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\",\"color\":\"#54C667\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#54C667\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9256709999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1473400000000002em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#54C667\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.8502755000000004em\"}},mdx(\"span\",{parentName:\"span\"})))))))))))),mdx(\"p\",null,`Now, with the power of multivariable calculus we can compute:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mtable\",{parentName:\"semantics\",\"rowspacing\":\"0.24999999999999992em\",\"columnalign\":\"right left\",\"columnspacing\":\"0em\"},mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∇`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`R`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`d`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mfrac\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mfrac\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`2`)),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"msubsup\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msubsup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msubsup\"},`b`),mdx(\"mi\",{parentName:\"msubsup\"},`T`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`y`),mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"mstyle\",{parentName:\"mstyle\",\"mathcolor\":\"#9628d9\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\"},`y`),mdx(\"mrow\",{parentName:\"msub\"},mdx(\"mi\",{parentName:\"mrow\"},`p`),mdx(\"mi\",{parentName:\"mrow\"},`r`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`d`)))),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#54C667\"},mdx(\"mn\",{parentName:\"mstyle\"},`2`),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"mi\",{parentName:\"mstyle\"},`α`),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`)))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#26a6ed\"},mdx(\"mfrac\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mfrac\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`2`)),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"msubsup\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msubsup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msubsup\"},`b`),mdx(\"mi\",{parentName:\"msubsup\"},`T`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mstyle\"},`y`),mdx(\"mo\",{parentName:\"mstyle\"},`−`),mdx(\"mstyle\",{parentName:\"mstyle\",\"mathcolor\":\"#9628d9\"},mdx(\"msub\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`)),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"mo\",{parentName:\"mstyle\",\"stretchy\":\"false\"},`)`)),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#54C667\"},mdx(\"mn\",{parentName:\"mstyle\"},`2`),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"mi\",{parentName:\"mstyle\"},`α`),mdx(\"mo\",{parentName:\"mstyle\"},`⋅`),mdx(\"mi\",{parentName:\"mstyle\",\"mathvariant\":\"bold-italic\"},`θ`))))))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\begin{aligned}\n\\\\nabla(RidgeMSE) &= {\\\\color{#26a6ed}\\\\frac{-2}{n} \\\\mathbf{X}_b^T (y - {\\\\color{9628d9}y_{pred}} )} + {\\\\color{#54C667}2 \\\\cdot \\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta}} \\\\\\\\\n&= {\\\\color{#26a6ed}\\\\frac{-2}{n} \\\\mathbf{X}_b^T (y - {\\\\color{9628d9}\\\\mathbf{X}_b \\\\boldsymbol{\\\\theta}})} + {\\\\color{#54C667}2 \\\\cdot \\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta}}\n\\\\end{aligned}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"4.61488em\",\"verticalAlign\":\"-2.05744em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mtable\"},mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.55744em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.55744em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∇`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.00773em\"}},`R`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`d`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.05744em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.55744em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.55744em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"color\":\"#26a6ed\",\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\",\"style\":{\"color\":\"#26a6ed\"}},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4530000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`b`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#26a6ed\"}},`T`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.247em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#9628d9\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.3361079999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.03588em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`p`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.02778em\",\"color\":\"#9628d9\"}},`r`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`d`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.286108em\"}},mdx(\"span\",{parentName:\"span\"}))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#54C667\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\",\"color\":\"#54C667\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#54C667\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`)))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"color\":\"#26a6ed\"}},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"color\":\"#26a6ed\",\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\",\"style\":{\"color\":\"#26a6ed\"}},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4530000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#26a6ed\"}},`b`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\",\"color\":\"#26a6ed\"}},`T`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.247em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\",\"style\":{\"color\":\"#26a6ed\"}},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\",\"color\":\"#26a6ed\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#26a6ed\"}},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#26a6ed\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\",\"style\":{\"color\":\"#9628d9\"}},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"color\":\"#9628d9\"}},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#9628d9\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#9628d9\"}},`θ`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\",\"style\":{\"color\":\"#26a6ed\"}},`)`)),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#54C667\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\",\"color\":\"#54C667\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\",\"style\":{\"color\":\"#54C667\"}},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#54C667\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\",\"color\":\"#54C667\"}},`θ`))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.05744em\"}},mdx(\"span\",{parentName:\"span\"})))))))))))),mdx(\"p\",null,`I’ve used some colors to make the individual parts of the equation a bit clearer. The blue part (which includes\nthe purple part) is our MSE. The purple part is our predicted y, which we can compute by multiplying\nour features `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\mathbf{X}_b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.83611em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` with our model parameters `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.69444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))))),`. I’ve added a little `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mrow\",{parentName:\"msub\"}),mdx(\"mi\",{parentName:\"msub\"},`b`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`_b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.486108em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` in `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\mathbf{X}_b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.83611em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`\nto signal that this `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold\"},`X`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\mathbf{X}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.68611em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)))))),` already has `,mdx(\"em\",{parentName:\"p\"},`intercept ones`),` attached to it, to account for the bias,\nor intercept term. If you are confused as to why this is necessary and/or makes sense, I recommend\nyou read `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/linear_regression/#adjustment-vectorization\"},`this section of the article about linear regression`),`.\nThat part is not very long and in it, we will take a closer look at exactly this example,\nand how we can make sense out of it. If you’re struggling with the equation above, this should\nclear things up!`),mdx(\"p\",null,`I don’t want to get into too much detail about this differentiation, because this is what\nyou would do in a course about multivariable calculus. If you’re interested though, let me know\nand I’ll provide more info!`),mdx(InfoTip,{mdxType:\"InfoTip\"},mdx(\"p\",null,`One small `,mdx(\"em\",{parentName:\"p\"},`prettification`),` we can do is add a factor of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mfrac\",{parentName:\"mrow\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mn\",{parentName:\"mfrac\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\frac{1}{2}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.190108em\",\"verticalAlign\":\"-0.345em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.845108em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.6550000000000002em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.394em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.345em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})))))),` to our MSE,\nlike so:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`R`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`d`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mfrac\",{parentName:\"mrow\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mo\",{parentName:\"msup\",\"stretchy\":\"false\"},`)`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#FF8900\"},mdx(\"mfrac\",{parentName:\"mstyle\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mn\",{parentName:\"mfrac\"},`2`))),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"msup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msup\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mi\",{parentName:\"msup\"},`T`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\nRidgeMSE = \\\\frac{1}{n} \\\\cdot ( (y- \\\\mathbf{X}_b \\\\boldsymbol{\\\\theta} )^T (y- \\\\mathbf{X}_b \\\\boldsymbol{\\\\theta}) ) + {\\\\color{#FF8900}{\\\\frac{1}{2}}} \\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta}^T\\\\boldsymbol{\\\\theta}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.00773em\"}},`R`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`d`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.00744em\",\"verticalAlign\":\"-0.686em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1.1413309999999999em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.25em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"2.00744em\",\"verticalAlign\":\"-0.686em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},`2`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"color\":\"#FF8900\",\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.9256709999999999em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.9256709999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.1473400000000002em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`)))))))),mdx(\"p\",null,`This will result in our gradient looking like this:`),mdx(\"div\",{\"className\":\"math math-display\"},mdx(\"span\",{parentName:\"div\",\"className\":\"katex-display\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\",\"display\":\"block\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mtable\",{parentName:\"semantics\",\"rowspacing\":\"0.24999999999999992em\",\"columnalign\":\"right left\",\"columnspacing\":\"0em\"},mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"normal\"},`∇`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`R`),mdx(\"mi\",{parentName:\"mrow\"},`i`),mdx(\"mi\",{parentName:\"mrow\"},`d`),mdx(\"mi\",{parentName:\"mrow\"},`g`),mdx(\"mi\",{parentName:\"mrow\"},`e`),mdx(\"mi\",{parentName:\"mrow\"},`M`),mdx(\"mi\",{parentName:\"mrow\"},`S`),mdx(\"mi\",{parentName:\"mrow\"},`E`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`)))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mfrac\",{parentName:\"mrow\"},mdx(\"mrow\",{parentName:\"mfrac\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`2`)),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msubsup\"},`b`),mdx(\"mi\",{parentName:\"msubsup\"},`T`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mn\",{parentName:\"mrow\"},`2`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mstyle\",{parentName:\"mrow\",\"mathcolor\":\"#FF8900\"},mdx(\"mfrac\",{parentName:\"mstyle\"},mdx(\"mn\",{parentName:\"mfrac\"},`1`),mdx(\"mn\",{parentName:\"mfrac\"},`2`))),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`))))),mdx(\"mtr\",{parentName:\"mtable\"},mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"}))),mdx(\"mtd\",{parentName:\"mtr\"},mdx(\"mstyle\",{parentName:\"mtd\",\"scriptlevel\":\"0\",\"displaystyle\":\"true\"},mdx(\"mrow\",{parentName:\"mstyle\"},mdx(\"mrow\",{parentName:\"mrow\"}),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mfrac\",{parentName:\"mrow\"},mdx(\"mrow\",{parentName:\"mfrac\"},mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"mn\",{parentName:\"mrow\"},`2`)),mdx(\"mi\",{parentName:\"mfrac\"},`n`)),mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msubsup\"},`b`),mdx(\"mi\",{parentName:\"msubsup\"},`T`)),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`(`),mdx(\"mi\",{parentName:\"mrow\"},`y`),mdx(\"mo\",{parentName:\"mrow\"},`−`),mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\",\"mathvariant\":\"bold\"},`X`),mdx(\"mi\",{parentName:\"msub\"},`b`)),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`),mdx(\"mo\",{parentName:\"mrow\",\"stretchy\":\"false\"},`)`),mdx(\"mo\",{parentName:\"mrow\"},`+`),mdx(\"mi\",{parentName:\"mrow\"},`α`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mi\",{parentName:\"mrow\",\"mathvariant\":\"bold-italic\"},`θ`)))))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\begin{aligned}\n\\\\nabla(RidgeMSE) &= \\\\frac{-2}{n} \\\\mathbf{X}_b^T (y - \\\\mathbf{X}_b \\\\boldsymbol{\\\\theta} ) + 2 \\\\cdot {\\\\color{#FF8900}\\\\frac{1}{2}} \\\\cdot \\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta} \\\\\\\\\n&= \\\\frac{-2}{n} \\\\mathbf{X}_b^T (y - \\\\mathbf{X}_b \\\\boldsymbol{\\\\theta} ) + \\\\alpha \\\\cdot \\\\boldsymbol{\\\\theta} \\\\\\\\\n\\\\end{aligned}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"4.61488em\",\"verticalAlign\":\"-2.05744em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mtable\"},mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.55744em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.55744em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`∇`),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.00773em\"}},`R`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`i`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`d`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`g`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`e`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.10903em\"}},`M`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`S`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.05764em\"}},`E`),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.05744em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"col-align-l\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.55744em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-4.55744em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4530000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.247em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},`2`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"color\":\"#FF8900\",\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\",\"style\":{\"color\":\"#FF8900\"}},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"}))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.25em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3.32144em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mopen nulldelimiter\"}),mdx(\"span\",{parentName:\"span\",\"className\":\"mfrac\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"1.32144em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.314em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`n`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.23em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"frac-line\",\"style\":{\"borderBottomWidth\":\"0.04em\"}})),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.677em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"3em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.686em\"}},mdx(\"span\",{parentName:\"span\"}))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose nulldelimiter\"})),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.8913309999999999em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4530000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.113em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\",\"style\":{\"marginRight\":\"0.13889em\"}},`T`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.247em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mopen\"},`(`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.03588em\"}},`y`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`−`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathbf\"},`X`)),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.33610799999999996em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal mtight\"},`b`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`))),mdx(\"span\",{parentName:\"span\",\"className\":\"mclose\"},`)`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`+`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord boldsymbol\",\"style\":{\"marginRight\":\"0.03194em\"}},`θ`)))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"2.05744em\"}},mdx(\"span\",{parentName:\"span\"})))))))))))),mdx(\"p\",null,`As you can see, the factor 2 is removed from our gradient.\nOftentimes, in online courses or textbooks, you will see this being\ndone to make the gradient a bit simpler. It does not really make that much of a difference\nsince you can just increase or decrease `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),` to make up for the factor\n2, but it’s still nice to know why this is sometimes done.`)),mdx(\"p\",null,`We can now directly translate our gradient into code. Because we also have to pass in an argument\nfor `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),`, we can use the same approach as before:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`def getRidgeGradientFunction(alpha=0.0001):\n\n def ridgeGradient(X_b, y, y_pred, theta):\n return -(2/y.size) * X_b.T.dot((y - y_pred)) + alpha * theta\n\n return ridgeGradient\n`)),mdx(\"p\",null,`With the loss and gradient now defined, we can implement our gradient descent algorithm!\nWe’ll make a slight change - our function will now receive an additional parameter `,mdx(\"inlineCode\",{parentName:\"p\"},`gradient_function`),`\nwhich we will use to compute our gradient:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\",\"metastring\":\"{1,11}\",\"{1,11}\":true},`def gradientDescent(X, y, theta, criterion, gradientFunction, number_of_iterations, learning_rate):\n X_b = add_intercept_ones(X)\n for i in range(number_of_iterations):\n\n # predict and calculate loss\n f = create_function(theta) # create the current function\n y_predicted = X_b.dot(theta) # predict our entire x\n loss = criterion(y, y_predicted, theta) # calculate loss\n\n # perform optimization\n gradient = gradientFunction(X_b, y, y_predicted, theta) # compute gradient\n theta = theta - learning_rate * gradient # adjust theta\n\n return theta\n`)),mdx(\"p\",null,`Let’s test our implementation:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`theta_ridge_gd_1 = gradientDescent(X_train, y_train, theta, getRidgeMSEFunction(0.0001),\n getRidgeGradientFunction(0.0001), 1000, 0.03)\nprint(theta_ridge_gd_1)\n# output: [44.41801778 -3.11386069]\n`)),mdx(\"p\",null,`Great, we get almost the same results as with our normal equation!\nWe can make a small plot and compare our solution from the normal equation\nwith the results of gradient descent:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`plt.plot(X_train, y_train, \"s\", markersize=12, label=\"train data\")\nplt.plot(X_test, y_test, \".\", markersize=15, label=\"test data\")\n\nplt.plot(X, X_b.dot(theta_ridge_gd_1), label=\"ridge gd\")\nplt.plot(X, X_b.dot(theta_ridge_ne_1), label=\"ridge ne\")\nplt.legend()\nplt.show()\n`)),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"376px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"65.95744680851064%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAAB1UlEQVQ4y5WTS2/aQBSF+f8/ovsuum7X7aZtFgEVhTZBYgA/IDYePH4CnvmqO8YKTlOpPdKVx+N7z7kvT8pcM33a8uFuBfZC07S07d+taVvozjxGGR+na3T6TBCERFFEVVVMcI5U5zyblv9Bdkj49fgTpdbMZjOUUtR1zcQYQ2Fy7+Sc+ycTtG1O2yas1JrF4oEkSXrCsizJc4O7Kls5XYN88A3JLeG5rDjpnDAMWSuFJNaXDBTG0Jgj7nQWRjrAnHsBN5izL4SvshV0XYckN0nTFK01+3jJ5umeZDPnfjrj/adv0BRwukDnRv0bi1gvYAdCcajqxjtesDRdS3bcER8U4eaB7XJKvJpTxxvKeIutamilkrGIhZ4wSTOy3RZX7P+YZJ+JY51p3n2+IzMR4XZBoOYcgiUmUNT7WJiw8YKybpiIUKFTbK1HjbfXnvkBdI607Ppe4bjgODaa5zxirwMC9QOTBtRCKKPOTdFP2ZPxxpqMp/4aInVx15KPxyM6y27qdG8u8iDkMz6fSdNk/N3al7WRHcqyjDzPvYq8i5Cci6LwWyDPwcJ4x5ev3ykK42OESPzFJtZarzBcCJqm8QJ+etZyOBwY/ORe/ukhRsgE0jr59hvRJe9XyWl5jwAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/98f00c47d2a0ebdc24dd2aa8b790ad19/3e1d6/RidgeNEvsRidgeGD.webp 376w\"],\"sizes\":\"(max-width: 376px) 100vw, 376px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/98f00c47d2a0ebdc24dd2aa8b790ad19/1075d/RidgeNEvsRidgeGD.png 376w\"],\"sizes\":\"(max-width: 376px) 100vw, 376px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/98f00c47d2a0ebdc24dd2aa8b790ad19/1075d/RidgeNEvsRidgeGD.png\",\"alt\":\"Ridge Normal Equation vs Gradient Descent\",\"title\":\"Ridge Normal Equation vs Gradient Descent\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `)),mdx(\"p\",null,`Nice! You could get even closer to the results from the normal equation by tuning the hyperparameters even more.\nWe can also solve ridge regression with gradient descent by using scikit-learn’s `,mdx(\"a\",{parentName:\"p\",\"href\":\"https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.html\",\"target\":\"_blank\",\"rel\":\"nofollow noopener noreferrer\"},mdx(\"inlineCode\",{parentName:\"a\"},`SGDRegressor`)),`-class.\nScikit-learn does a bit more under the hood, and to mimic our own function’s behaviour we’ll\nhave to pass a few additional parameters. For example,\n`,mdx(\"inlineCode\",{parentName:\"p\"},`SGDRegressor`),` uses an adaptive learning rate by default.\nTo use a constant learning rate we pass the arguments `,mdx(\"inlineCode\",{parentName:\"p\"},`learning_rate=\"constant\"`),`\nand `,mdx(\"inlineCode\",{parentName:\"p\"},`eta0=0.001`),`. `,mdx(\"inlineCode\",{parentName:\"p\"},`eta0`),` sets the initial learning rate.\nWe will also increase the maximum number of iterations to 10000.\nWe have to perform these adjustments because `,mdx(\"inlineCode\",{parentName:\"p\"},`SGDRegressor`),`\nuses `,mdx(\"em\",{parentName:\"p\"},`stochastic`),` gradient descent (that’s where the `,mdx(\"inlineCode\",{parentName:\"p\"},`S`),` in `,mdx(\"inlineCode\",{parentName:\"p\"},`SGD`),` comes from),\nwhile our own implementation uses regular gradient descent.\nThere are a couple of differences between the two variants,\nbut the most important one is that SGD does not converge as “cleanly”\nas regular GD, but is computationally a lot less expensive.\nWith this we have:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`sgdreg_ridge = SGDRegressor(penalty=\"l2\", learning_rate=\"constant\", eta0=0.001, max_iter=10000, random_state=42)\nsgdreg_ridge.fit(X_train,y_train)\ntheta_ridge_gd_2 = np.array([sgdreg_ridge.intercept_[0], sgdreg_ridge.coef_[0]])\nprint(theta_ridge_gd_2)\n# output: [43.1764243 -2.0467813]\n`)),mdx(\"p\",null,`Awesome, we get almost the same result as with our own implementation!\nWe can again create a simple plot and check that our results are similar:`),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"376px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"65.95744680851064%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAAB2UlEQVQ4y42T227bMAyG/f7vsDfYXmC3xYBivVm2okmWwTnYiVMfI8vnWPoGKkmbrhk2ArQsSvxJ/qS8ShV8WwR8eliCOdK2HV13W9uug3HgbrbjYbEjT54JwpA0Tanrmqqq8LCWtFDEZcf/yibcsFou8P0lk8mEoiicvSxLPPnoUjmDtfafaiz0fU6lY56mM37O5/R9zzAMaK3xJE2lFFZuOgcj0C8BrgPJuVgGVdIelCvX932nURS5sj1xENDBji8lHS2ozpzAjPlrttcie8ehEJrnOftwTrD8QaYCvq/WfLyfu0ztDQ6tAzCnYMacKhuP56YAddNgB0OnD+jdhmQ9J/QnxOsZzS6kCld0RUxTK2zbwmjeBTHnSr0kzcn3AbaO32dyHGHoCOOMD5+/kARTtusnwtUjWfCLcruieY4ECRNNqZoWT3qhixTTFK+cGYOR9QzcG0jrU1bC9GCOFDohVhFxsSVcP1KmAY0ANk2DKvXJ2ZHNDfJP0ObM158yYl0jXcmHw4Eiz6/bdXOYL4FEZObSNHl7bszr2MhwS6dlHiWK7CWQ/MuwykuQ9aLbaM/d/Ve0Lp2PAMl9Uc+VAS8GkbZtXYBLmVmWcbkndnnXFx8BExHq5Ow36qTvB4F/s90AAAAASUVORK5CYII=')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/24a210eb6fda3cea0ebd7ac11059e120/3e1d6/ownGDvsSKlearnSGD.webp 376w\"],\"sizes\":\"(max-width: 376px) 100vw, 376px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/24a210eb6fda3cea0ebd7ac11059e120/1075d/ownGDvsSKlearnSGD.png 376w\"],\"sizes\":\"(max-width: 376px) 100vw, 376px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/24a210eb6fda3cea0ebd7ac11059e120/1075d/ownGDvsSKlearnSGD.png\",\"alt\":\"Own GD Implementation vs Scikit-Learn's SGDRegressor\",\"title\":\"Own GD Implementation vs Scikit-Learn's SGDRegressor\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `)),mdx(\"p\",null,`With scikit-learn’s `,mdx(\"inlineCode\",{parentName:\"p\"},`SGDRegressor`),`-class, if we train and evaluate it multiple times,\nthere will almost always be small differences in the final values,\nbecause of the randomness involved in stochastic gradient descent.\nThis is why we provide an argument for `,mdx(\"inlineCode\",{parentName:\"p\"},`random_state`),`, to “fix” the randomness in our training and make\nour results reproducible.`),mdx(\"h2\",{\"id\":\"visualizing-gradient-descent\"},`Visualizing Gradient Descent`),mdx(\"div\",null,mdx(\"p\",null,`Now that we’ve implemented gradient descent, let’s run the algorithm once\nmore for ridge regression and visualize the results.\nBelow you can see a 3-dimensional plot where for every combination of\nmodel parameters `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mn\",{parentName:\"msub\"},`1`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\theta_1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`θ`),mdx(\"mn\",{parentName:\"msub\"},`0`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\theta_0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.84444em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.02778em\"}},`θ`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"-0.02778em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`0`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` (I’ve named them m and b in the plot, to make them more easily distinguishable),\nthe ridge-MSE is plotted. By pressing the buttons at the top, you can perform one iteration step of gradient descent.\nAfter the first ten steps, each button press cycles through more than just one step at a time, so that you don’t have\nto press one button 300 times. Take a look and see how ridge regression converges.`,mdx(ToolTip,{width:\"25em\",mdxType:\"ToolTip\"},` Small note: The orange trajectory of gradient descent should actually be `,mdx(\"em\",null,`in`),`\nthe 3d-surface plot since the loss values lie directly on that surface. However, to improve the trajectories visibility,\nI’ve slightly raised it. But don’t worry, if you hover over it to inspect the coordinates you will see the correct values.`))),mdx(ConditionalLazyPlotInteractive,_extends({},ridgeHistPlot,{mdxType:\"ConditionalLazyPlotInteractive\"}),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"985px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"82%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAQCAYAAAAWGF8bAAAACXBIWXMAABYlAAAWJQFJUiTwAAADQElEQVQ4y22U228bVRCHN/F1bSe73ptvu+tdX1LHsWObBEoIJFUbcSmiIFW0oJY0N1GXEFpKm9JAI5pAiuAFHuiVUhXBA+KP/NDZxY1VeBjN0Zk535kzOr+RYtkyUdVF+GEb7I0qDhHF4f/ykoaPJGXZ2LrIN3duIcVzSM9DBjbYU0sNlNKRZ3vD8RCosLl9mTs/HSBFjBA4DIprHgndJ5OrIaVKrH/2Mac+OBOs01Y1iAsTuYMKV6/0ufXjLlIshyQCCT2EpMwqKauKbFSC5KRR4Yc/vmPv4S5RJQTIIsesBl42K0iSxodbK3x+8AVSNIckQN7UUZJ6CBAmW1UimRL1hSU2/7rH5b/vM7G4RDRjk7Jqzy5LBkCdkxeXWd/fZkQuhRWKm8LESlihWSWuOExu3eTlu0+Zv/uE7pe3SWheUP0gJyGePKLxytoqb12/ihQ1wx46jRfJOs2g0SmrTlK1UdsLOF/9jn3zN+ydx5T3/yR77H1kpUQqVw9aFNd9pIhG89w6C9d3iKcLSKo9SaU9h+ZMBVWKasXt3tu7lJef4px/iL38CHv1McWPfiFTnkHW3eA1MQGMarhnNmht7pBI5ZHiWplaZ55CrReAknqJQnON9olfqS/co37sAbXjD6iduE/tzSe4r+2RtiYQ5wRwJKZhntrAW7lBXLaQxKfN5CcCYDpXI+u/g9fbp977GrdzG7ezh9v5ltL095Q7B1Rnfybf6BNVHWKaTyShoS6tYp+9QVophD0cHbeDnph+h9zkCqXuNbzeJUrtPsV2n0LrEs2ZdQqtTfKta2T9d5ENPwCOyjrZufMUX98iMTaklIjq4jRmqE7Po9few+1epTKzjde7gtv9lMXFC7i9TcbKb+A2XsBpzDKquEQyJlr7LO6rn5A27PDbhFA30G2+2kVzGiSMDmb9NNbkBQqtPmp9lWTuJYxyE+fI7L9nPEbHTPTaaezpNSLj1qH0Bj6Tr2P50+T8KWLZIilLqMNHNlzylRamNx3khAPDCyCKcxJ78hzxbOHwycMDQaxFpardJGGID1xBL7cwvHYAig8NlIhSJJU/ipybIzJe+O9wGJg4KBQRaj2U2vNj7NA7xNQw9g/Fpcml6ffwuQAAAABJRU5ErkJggg==')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/bc920e08ec3b83a028e9596cc29f6b33/601b1/ridgeHist.webp 500w\",\"/static/bc920e08ec3b83a028e9596cc29f6b33/187e1/ridgeHist.webp 985w\"],\"sizes\":\"(max-width: 985px) 100vw, 985px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/bc920e08ec3b83a028e9596cc29f6b33/0eb09/ridgeHist.png 500w\",\"/static/bc920e08ec3b83a028e9596cc29f6b33/2d7cc/ridgeHist.png 985w\"],\"sizes\":\"(max-width: 985px) 100vw, 985px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/bc920e08ec3b83a028e9596cc29f6b33/2d7cc/ridgeHist.png\",\"alt\":\"Gradient descent history for ridge\",\"title\":\"Gradient descent history for ridge\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `))),mdx(\"p\",null,`You might think that our algorithm overshoots the minimum and goes unnecessarily far to the “right”,\nbut if you inspect the coordinates and loss values by hovering over the plot, you will actually see\nthat the loss really does decrease in that direction.\nAll in all, the visuals are not too surprising (which is good, because this means that our algorithm works as expected!), and look quite similar to the visualization we’ve performed\nfor regular linear regression `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_math/gradient_descent_for_linear_regression/#visualizing-in-3d\"},`here`),`.\nSo everything looks alright!\nTime to round off this article and… wait. What is this next headline?`),mdx(\"h2\",{\"id\":\"we-forgot-something-important\"},`We Forgot Something Important`),mdx(\"p\",null,`What? What are you talking about? Our gradient descent algorithm worked just fine,\nthe solution was very close to the normal equation and we even `,mdx(\"em\",{parentName:\"p\"},`visualized`),` it in\n`,mdx(\"em\",{parentName:\"p\"},`3d`),` and everything looked alright! What could we possibly forget?`),mdx(\"p\",null,`Well, technically everything did go well. But in this example we were a bit.. `,mdx(\"em\",{parentName:\"p\"},`lucky`),`.\nLet’s take a look at what happens when we slightly modify our\ndataset. Right now our X tells us the age of every figure in years, right? Now what if we change that do\n`,mdx(\"em\",{parentName:\"p\"},`days`),` instead? In other words, we do this:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`X_days = X * 365 # let's ignore leap years for now\n`)),mdx(\"p\",null,`Our dataset will still look the same and it will still hold the same information as before,\nbut if we now run our OLS and ridge algorithms again, we get a different result.\nThe model parameters for the original dataset, where our X represented the age of a\nfigure in years, look like this:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`print(theta_ridge_ne_1)\n# output: [44.36 -2.76]\nprint(theta_ols_ne_1)\n# output: [ 54.3 -11.6]\n`)),mdx(\"p\",null,`Now we create new training and test splits `,mdx(\"inlineCode\",{parentName:\"p\"},`X_days_train`),` and `,mdx(\"inlineCode\",{parentName:\"p\"},`X_days_test`),`,\ntrain our models in the exact same way we did before and our new model parameters are:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`print(theta_ols_days_ne)\n# output: [54.3 -0.03178082]\nprint(theta_ridge_days_ne)\n# output: [54.29968655 -0.03178006]\n`)),mdx(\"p\",null,`That’s right! Our ridge regression model has (almost) exactly the same model parameters as our OLS regression model!\nSo the ridge regression penalty had no effect!`),mdx(\"p\",null,`Ok.. that’s weird. What happens when we instead decide to let our X be the age of\nour figure in decades? Meaning we do this:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`X_days = X / 10\n`)),mdx(\"p\",null,`We can now again create training and testing sets, train our models and we get:`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`print(theta_ols_decades_ne)\n# output: [ 54.3 -116. ]\nprint(theta_ridge_decades_ne)\n# output: [41.29065421 -0.36137072]\n`)),mdx(\"p\",null,`Ok.. now the ridge parameters seem to go in the opposite direction.\nLet’s try to get some structure into these observations by plotting each of the OLS and\nridge models on their respective datasets (click on the image to zoom in!):`),mdx(\"p\",null,mdx(\"span\",{parentName:\"p\",\"className\":\"gatsby-resp-image-wrapper\",\"style\":{\"position\":\"relative\",\"display\":\"block\",\"marginLeft\":\"auto\",\"marginRight\":\"auto\",\"maxWidth\":\"1161px\"}},`\n `,mdx(\"span\",{parentName:\"span\",\"className\":\"gatsby-resp-image-background-image\",\"style\":{\"paddingBottom\":\"30.4%\",\"position\":\"relative\",\"bottom\":\"0\",\"left\":\"0\",\"backgroundImage\":\"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAAsTAAALEwEAmpwYAAABrElEQVQY0y3PT2sTQRgG8AXBmzeriBcPeggUTasfxW8h+C3EP0cRFG1pCpXgWouZYDdLD9XSRsGUgmh3U5o2S61UW7KbmXnn387MKxt9rj+el/cJet/6d4WQS0qpF5TShZM/py/7e3vPd75uX/mR/bynjV7kHF4BwNzR8a/5JEkePHv9/sLBye/H3tm50ShvjIpifjDMGoPB/v1glBdPjVJYGoNVlFLIGMPD7GiaUroGWqJzdmJSSRxTetrb2b0MUlChJSL6iXFGkVK6EbD87JF1DjkDAVxUVy16J9Lke01z+k4JiZSNJWNMo3NopeivR2+nSuBDIQSeFbk2UqrKDNB2QMfFE/wXW2qDinN01tosG9akFK3/5qtPQQK6sjz8svlpyhhzPCl5j8JIL41CW5ZRkOf5Q+89OudYtarUWmsASNO0BgDLlXnvOSKCtsYLLZKP6xuXlFIH3nuH3ovKVamQCUaCNE0vEkJuh2FYX2w07iyvrMyuxfF0K3xzvrfdu9rpdOqkTW6GYTgTR3F96/PW9Sj+cK7b7d4ghNRbpHWr2WzORqvRTHu1fe0vN7WWQlAmGUEAAAAASUVORK5CYII=')\",\"backgroundSize\":\"cover\",\"display\":\"block\"}}),`\n `,mdx(\"picture\",{parentName:\"span\"},`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/66d60498599506f9446da3e465c981fe/601b1/scalingEffectsOnRidge.webp 500w\",\"/static/66d60498599506f9446da3e465c981fe/80c60/scalingEffectsOnRidge.webp 1000w\",\"/static/66d60498599506f9446da3e465c981fe/79cbf/scalingEffectsOnRidge.webp 1161w\"],\"sizes\":\"(max-width: 1161px) 100vw, 1161px\",\"type\":\"image/webp\"}),`\n `,mdx(\"source\",{parentName:\"picture\",\"srcSet\":[\"/static/66d60498599506f9446da3e465c981fe/0eb09/scalingEffectsOnRidge.png 500w\",\"/static/66d60498599506f9446da3e465c981fe/1263b/scalingEffectsOnRidge.png 1000w\",\"/static/66d60498599506f9446da3e465c981fe/8cf9a/scalingEffectsOnRidge.png 1161w\"],\"sizes\":\"(max-width: 1161px) 100vw, 1161px\",\"type\":\"image/png\"}),`\n `,mdx(\"img\",{parentName:\"picture\",\"className\":\"gatsby-resp-image-image\",\"src\":\"/static/66d60498599506f9446da3e465c981fe/8cf9a/scalingEffectsOnRidge.png\",\"alt\":\"Effects of Scaling on Ridge Regression\",\"title\":\"Effects of Scaling on Ridge Regression\",\"loading\":\"lazy\",\"decoding\":\"async\",\"style\":{\"width\":\"100%\",\"height\":\"100%\",\"margin\":\"0\",\"verticalAlign\":\"middle\",\"position\":\"absolute\",\"top\":\"0\",\"left\":\"0\"}}),`\n `),`\n `)),mdx(\"p\",null,`Ok, much better! There are a few things we can see here:`),mdx(\"ol\",null,mdx(\"li\",{parentName:\"ol\"},`It seems that the ridge penalty becomes weaker when our data points are closer\ntogether, and stronger when they are further apart.`),mdx(\"li\",{parentName:\"ol\"},`The intercept term of the OLS models doesn’t change. Only the slope changes.`),mdx(\"li\",{parentName:\"ol\"},`The slope of the OLS models increases when our points are closer together.`)),mdx(\"p\",null,`Points 2 and 3 make sense together when we consider a simple example.\nTwo data points from our original X were\nabout 1 year apart from each other. Now the difference can be as high as 365 days!\nFor us, those two mean exactly the same thing. But for our model, the numbers just increased\nby a factor of `,mdx(\"strong\",{parentName:\"p\"},`365`),`!\nSince our data points are now a lot further apart from each other,\nthis also means that very small changes in our model parameters can drastically alter\nour loss. If we have two data points `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`1`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`1`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_1=1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`))))),` and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`2`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`2`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_2=2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`2`))))),`, then\nour y-value will have changed by `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mn\",{parentName:\"mrow\"},`1`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mtext\",{parentName:\"mrow\"},`slope`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`1\\\\cdot\\\\text{slope}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`1`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`slope`)))))),` if we go from `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`1`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` to `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msub\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msub\"},`x`),mdx(\"mn\",{parentName:\"msub\"},`2`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.58056em\",\"verticalAlign\":\"-0.15em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.30110799999999993em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.5500000000000003em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.15em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`.\nBut if we have two data points `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\"},`x`),mdx(\"mn\",{parentName:\"msubsup\"},`1`),mdx(\"mo\",{parentName:\"msubsup\",\"mathvariant\":\"normal\",\"lspace\":\"0em\",\"rspace\":\"0em\"},`′`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`365`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_1'=365`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.24810799999999997em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.751892em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4518920000000004em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.063em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`′`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.24810799999999997em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`3`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`6`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`5`))))),` and `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\"},`x`),mdx(\"mn\",{parentName:\"msubsup\"},`2`),mdx(\"mo\",{parentName:\"msubsup\",\"mathvariant\":\"normal\",\"lspace\":\"0em\",\"rspace\":\"0em\"},`′`)),mdx(\"mo\",{parentName:\"mrow\"},`=`),mdx(\"mn\",{parentName:\"mrow\"},`730`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_2'=730`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.24810799999999997em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.751892em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4518920000000004em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.063em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`′`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.24810799999999997em\"}},mdx(\"span\",{parentName:\"span\"})))))),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mrel\"},`=`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2777777777777778em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`7`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`3`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`0`))))),` and we go from `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\"},`x`),mdx(\"mn\",{parentName:\"msubsup\"},`1`),mdx(\"mo\",{parentName:\"msubsup\",\"mathvariant\":\"normal\",\"lspace\":\"0em\",\"rspace\":\"0em\"},`′`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_1'`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.24810799999999997em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.751892em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4518920000000004em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`1`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.063em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`′`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.24810799999999997em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),` to `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"msubsup\",{parentName:\"mrow\"},mdx(\"mi\",{parentName:\"msubsup\"},`x`),mdx(\"mn\",{parentName:\"msubsup\"},`2`),mdx(\"mo\",{parentName:\"msubsup\",\"mathvariant\":\"normal\",\"lspace\":\"0em\",\"rspace\":\"0em\"},`′`))),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`x_2'`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"1em\",\"verticalAlign\":\"-0.24810799999999997em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\"},`x`),mdx(\"span\",{parentName:\"span\",\"className\":\"msupsub\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-t vlist-t2\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.751892em\"}},mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-2.4518920000000004em\",\"marginLeft\":\"0em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`2`))),mdx(\"span\",{parentName:\"span\",\"style\":{\"top\":\"-3.063em\",\"marginRight\":\"0.05em\"}},mdx(\"span\",{parentName:\"span\",\"className\":\"pstrut\",\"style\":{\"height\":\"2.7em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"sizing reset-size6 size3 mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord mtight\"},`′`))))),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-s\"},`​`)),mdx(\"span\",{parentName:\"span\",\"className\":\"vlist-r\"},mdx(\"span\",{parentName:\"span\",\"className\":\"vlist\",\"style\":{\"height\":\"0.24810799999999997em\"}},mdx(\"span\",{parentName:\"span\"})))))))))),`,\nour y-value will have changed by `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mn\",{parentName:\"mrow\"},`365`),mdx(\"mo\",{parentName:\"mrow\"},`⋅`),mdx(\"mtext\",{parentName:\"mrow\"},`slope`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`365\\\\cdot\\\\text{slope}`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.64444em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`3`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`6`),mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`5`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mbin\"},`⋅`),mdx(\"span\",{parentName:\"span\",\"className\":\"mspace\",\"style\":{\"marginRight\":\"0.2222222222222222em\"}})),mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.8888799999999999em\",\"verticalAlign\":\"-0.19444em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord text\"},mdx(\"span\",{parentName:\"span\",\"className\":\"mord\"},`slope`)))))),`!\nThis is why the slope of our OLS models `,mdx(\"em\",{parentName:\"p\"},`increases`),` when our data points are\n`,mdx(\"em\",{parentName:\"p\"},`further apart`),` from each other. Ok. Similarly, the slope `,mdx(\"em\",{parentName:\"p\"},`decreases`),` when\nour data points are `,mdx(\"em\",{parentName:\"p\"},`closer together`),`. Makes sense, right? But what about ridge?`),mdx(\"p\",null,`Take a look at the OLS model parameters for our `,mdx(\"inlineCode\",{parentName:\"p\"},`X_decades`),`. The slope is pretty high (in the absolutes),\nit is around -116. Our ridge penalty penalizes large model weights, so our penalty hinders our\nridge model from developing such a high slope value. But our model parameters `,mdx(\"em\",{parentName:\"p\"},`need`),`\nto be this large because our points are much closer together! This unlucky combination leads to our ridge\npenalty completely taking over and our model looking like a flat line.`),mdx(\"p\",null,`If we consider `,mdx(\"inlineCode\",{parentName:\"p\"},`X_days`),`, then our data points are very far apart, which means our model needs\nsmaller weights. But because our model parameters already are so small because of our scaling,\nthe ridge penalty does not see any problem whatsoever and does not penalize our model any further,\nbecause the weights are already so small.`),mdx(\"p\",null,`This issue appears very frequently in machine learning and it can be a real pain to deal with\nbecause you might not suspect that the scaling of your data is the leading cause for this issue.\nThis issue also affects almost all regularized models, because they all share some sort of\nregularization term that punishes large model weights.`),mdx(\"p\",null,`So what can we do about this? Is there any hope left for our ridge regression model?\nYes, there is! And it is called `,mdx(\"strong\",{parentName:\"p\"},`standardization`),`.`),mdx(\"p\",null,`Originally I wanted to talk about standardization, what it is, why it can save our ridge regression model,\nand so on. But then this article got longer and longer and in the end, I decided to write a separate\narticle all about standardization! After all, this article is already pretty long\nand I wanted to make a cut here so that you can process all of the information\nfrom this article, and then move onto the next one.\nSo if you want to find out how you can save our ridge regression model\nfrom the pitfalls of different data scalings, head on to the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/dataset_optimization/standardization\"},`When You Should Standardize Your Data`),`\nand learn all about the magic of standardization!`),mdx(\"h2\",{\"id\":\"finding-the-optimal-value-for-alpha\"},`Finding the Optimal Value for `,mdx(\"span\",{parentName:\"h2\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`)))))),mdx(\"p\",null,`So far we’ve seen a number of examples that show ridge regression in action.\nIn the code examples, I’ve provided parameters for hyperparameters such as\nthe learning rate or `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),`. But the question is, how did I actually find\nthe right values for these hyperparameters? Do I just have a magical sense\nthat tells me which values will result in the lowest loss? Sadly, no.\nThe only way to find out the ideal values is to just try out a bunch of them\nand then pick the ones that result in the best models.\nThis is called `,mdx(\"em\",{parentName:\"p\"},`hyperparameter tuning`),`. For `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),`, scikit-learn offers you\na built-in class called `,mdx(\"inlineCode\",{parentName:\"p\"},`RidgeCV`),` where you don’t have to provide an argument for `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),`.\nInstead, multiple values of `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),` are tested using `,mdx(\"em\",{parentName:\"p\"},`cross validation`),` (That’s where the `,mdx(\"inlineCode\",{parentName:\"p\"},`CV`),`\nin `,mdx(\"inlineCode\",{parentName:\"p\"},`RidgeCV`),` comes from) and the ideal one is chosen. You can specify the values for `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),`\nthat should be compared to each other.\nBelow is an exemplary application of `,mdx(\"inlineCode\",{parentName:\"p\"},`RidgeCV`),`.`),mdx(\"pre\",null,mdx(\"code\",{parentName:\"pre\",\"className\":\"language-python\"},`ridge_cv = RidgeCV(alphas=[0.1, 1.0, 10])\nridge_cv.fit(X_train, y_train)\nprint(ridge_cv.alpha_)\n# output: 0.1\n`)),mdx(\"div\",null,mdx(\"p\",null,`As you can see, we compare the values 0.1, 1, and 10 for our hyperparameter `,mdx(\"inlineCode\",{parentName:\"p\"},`alpha`),` and then our\nmodel just chooses the value for alpha that results in the best model!`,mdx(ToolTip,{mdxType:\"ToolTip\"},`More specifically, it chooses the value for alpha based on the lowest cross validation loss.`))),mdx(\"h2\",{\"id\":\"further-reading\"},`Further Reading`),mdx(\"p\",null,`If you have made it until the end of this article, great job! This was my most exhaustive article so far and I really\nhope that it could help you understand ridge regression better. Now you should be able to use regularization\nto further optimize your models and prevent them from overfitting.\nFrom here, there are two ways to continue learning.`),mdx(\"h3\",{\"id\":\"improving-regularization\"},`Improving Regularization`),mdx(\"p\",null,`As we saw before, ridge regression relies on an `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),`-parameter that helps control\nthe regularization strength. But finding the optimal value for `,mdx(\"span\",{parentName:\"p\",\"className\":\"math math-inline\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex\"},mdx(\"span\",{parentName:\"span\",\"className\":\"katex-mathml\"},mdx(\"math\",{parentName:\"span\",\"xmlns\":\"http://www.w3.org/1998/Math/MathML\"},mdx(\"semantics\",{parentName:\"math\"},mdx(\"mrow\",{parentName:\"semantics\"},mdx(\"mi\",{parentName:\"mrow\"},`α`)),mdx(\"annotation\",{parentName:\"semantics\",\"encoding\":\"application/x-tex\"},`\\\\alpha`)))),mdx(\"span\",{parentName:\"span\",\"className\":\"katex-html\",\"aria-hidden\":\"true\"},mdx(\"span\",{parentName:\"span\",\"className\":\"base\"},mdx(\"span\",{parentName:\"span\",\"className\":\"strut\",\"style\":{\"height\":\"0.43056em\",\"verticalAlign\":\"0em\"}}),mdx(\"span\",{parentName:\"span\",\"className\":\"mord mathnormal\",\"style\":{\"marginRight\":\"0.0037em\"}},`α`))))),` can be a bit tricky\nbecause it’s a hyperparameter. This is where the magic of hyperparameter optimization comes in!\nHyperparameter optimization can optimize these values for you, and automatically select the values that\nresult in the lowest losses and the best models. Hyperparameter optimization is extremely useful in practice,\nand if you want to learn more about it, I recommend that you read the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/model_optimization/hyperparameter_optimization\"},`Hyperparameter Optimization Explained, Step by Step`),`,\nwhere you will learn everything you need to know about this topic.`),mdx(\"p\",null,`Another crucial aspect we’ve seen is just how important the scaling of your dataset is.\nDifferent scales can result in vastly different models, and not paying attention to the scaling\nof your data can cause a lot of pain and confusion. Luckily, there are ways to deal with this!\nOne of these ways is `,mdx(\"em\",{parentName:\"p\"},`standardization`),`, which puts all of your data values onto the same scale.\nIf you want to create reliable and consistent machine learning models with data on any scale,\nthen you should read the article `,mdx(\"a\",{parentName:\"p\",\"href\":\"/dataset_optimization/standardization\"},`When You Should Standardize Your Data`),`,\nwhere you will learn everything you need to know about standardization and how you can apply it in your own machine learning projects\n(Spoiler: It’s a lot easier than one might think!).`),mdx(\"h3\",{\"id\":\"ridge-for-other-models\"},`Ridge for Other Models`),mdx(\"p\",null,`In this article, we looked at ridge regression, which is a regularized variant of OLS regression.\nBut the ridge penalty is not just limited to plain old OLS regression!\nIn fact, we can take most other linear models and just add the ridge penalty to its respective\ncost function! We can take the ridge penalty and apply it to models like\nlogistic regression or polynomial regression. If you’re interested in these regularized models,\nI recommend you take a look at the articles `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/logistic_regression\"},`Logistic Regression Explained, Step by Step`),`\nand `,mdx(\"a\",{parentName:\"p\",\"href\":\"/machine_learning_models/polynomial_regression\"},`Polynomial Regression Explained, Step by Step`),` respectively.\nIn those articles you will learn everything about the named models as well as their regularized variants!`));};MDXContent.isMDXComponent=true;","excerpt":"Ridge Regression is an adaptation of the popular and widely used linear regression algorithm. It enhances regular linear regression by slightly changing its cost function, which results in less overfit models. In this article, you will learn everything you need to know about Ridge Regression, and how you can start using it in your own machine learning projects.","tableOfContents":{"items":[{"url":"#outline","title":"Outline"},{"url":"#prerequisites","title":"Prerequisites"},{"url":"#the-problem","title":"The Problem"},{"url":"#improving-ols-regression","title":"Improving OLS Regression","items":[{"url":"#regression-with-a-twist","title":"Regression With a Twist"},{"url":"#ols-vs-ridge","title":"OLS vs Ridge"}]},{"url":"#solving-ridge-regression","title":"Solving Ridge Regression","items":[{"items":[{"url":"#normal-equation","title":"Normal Equation"},{"url":"#gradient-descent","title":"Gradient Descent"}]}]},{"url":"#implementing-it-in-python","title":"Implementing it in Python","items":[{"url":"#implementing-the-normal-equation","title":"Implementing the Normal Equation"},{"url":"#implementing-gradient-descent","title":"Implementing Gradient Descent"}]},{"url":"#visualizing-gradient-descent","title":"Visualizing Gradient Descent"},{"url":"#we-forgot-something-important","title":"We Forgot Something Important"},{"url":"#finding-the-optimal-value-for-alpha","title":"Finding the Optimal Value for \\alpha"},{"url":"#further-reading","title":"Further Reading","items":[{"url":"#improving-regularization","title":"Improving Regularization"},{"url":"#ridge-for-other-models","title":"Ridge for Other Models"}]}]},"frontmatter":{"title":"Ridge Regression Explained, Step by Step","userDate":"23 May 2021","date":"2021-05-23T19:13:45.907Z","dateModified":"2021-11-02T08:08:00.157Z","changelog":[["2021-11-02T08:08:00.157Z","Added 1/n to the definition of the MSE which I previously forgot in one section. Thanks to @ML_Schlagy for pointing this out!"]],"tags":["Machine Learning Models"],"timeToRead":26,"excerpt":"Ridge Regression is an adaptation of the popular and widely used linear regression algorithm. It enhances regular linear regression by slightly changing its cost function, which results in less overfit models. In this article, you will learn everything you need to know about Ridge Regression, and how you can start using it in your own machine learning projects.","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACVUlEQVQozz3OWU8aUQCG4QMeBmEYhmFmmOWcWZgVDqACKkOCCzZao8SaaohCG5qmS2wv3JDlptg7TZo2aYzav+CvbDBNk+fuTb58gEopkZT6H0ypEU6lOJVKKRSnTCsr/5PMQEaEjAATAkzwkOZAVDApwYw8g1NGmNNCLAIsAowcSkghZiqclCGnQlZ+Jj2v8CAuWrMoH8UkigiFCRCsfK3Z+3hSXXlRCtYbWy3VLyN/QbAKIU6DvA7TGkxjOP0rA4bDjGgljXnaWqLdAGjlne6nu4fHi8H41++7u/v72x8//zw+tHsfQBLHNEIpHpRsKJpxxQJ0Un6m6M6cahcjvMFiz1sI3n8+ebm1ubHeHAxHJGialdUZXAzLPpScWcWlkSuYeRCNC1FaiMb5fn9w1b/qdrqt1t75ZX88HnWOj990upubW68O2tu7e+3O24Oj7v5Bm4oxaVHJOj6QkSMjR8Xul6+nw9H42/X3m5vbp6ens7PT4WAwHI0uLy+uJ5OL8/PJZDIcjXq9dzutvf3Xh43VJpAUS1KyCnL4jIF0L19YrC43avVVrDmW5WCMi8VSsTRHSCEI6mvrG4fto8Va3fGJpGLASC5tVChaTPHIcudJaTnrFCXF1owc1n0j69m2l8uTSqVaLleCel1BGgCh0Aw1Q8VAdnnHqe3auXJ1aaWxtj1XDkybYMMXJENSbd3MmxZxPOJ6Oc/PIayzXDrBpqgYE5lNAE62OcnK2iRXWPRJFelekpOFjDFfrvOioSLHsIiCXd30sZ7NKIjjRU4QYwk2TMX/AgxCcEm95chYAAAAAElFTkSuQmCC"},"images":{"fallback":{"src":"/static/bffad91bca399e12f65048348b2b2df8/87706/RidgeRegression_Thumbnail.png","srcSet":"/static/bffad91bca399e12f65048348b2b2df8/0dee1/RidgeRegression_Thumbnail.png 750w,\n/static/bffad91bca399e12f65048348b2b2df8/8beaa/RidgeRegression_Thumbnail.png 1080w,\n/static/bffad91bca399e12f65048348b2b2df8/87706/RidgeRegression_Thumbnail.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/bffad91bca399e12f65048348b2b2df8/a66aa/RidgeRegression_Thumbnail.webp 750w,\n/static/bffad91bca399e12f65048348b2b2df8/65dd5/RidgeRegression_Thumbnail.webp 1080w,\n/static/bffad91bca399e12f65048348b2b2df8/71d4d/RidgeRegression_Thumbnail.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Boris Giba","bio":"Hi! I'm Boris and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAQFBv/EABYBAQEBAAAAAAAAAAAAAAAAAAABAv/aAAwDAQACEAMQAAAB25bcya1Bcz84HRg//8QAGxAAAwACAwAAAAAAAAAAAAAAAQIDABESMTP/2gAIAQEAAQUCo3FFo4OEbG1R8u5SXZh4f//EABURAQEAAAAAAAAAAAAAAAAAAAEg/9oACAEDAQE/ASP/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAeEAABBAEFAAAAAAAAAAAAAAABAAIQETESIUFRcf/aAAgBAQAGPwIkZTQ/Sb6ilvxBIzDPF//EABoQAQACAwEAAAAAAAAAAAAAAAERIQAQMUH/2gAIAQEAAT8hQ1PjHRkxa9E644V8JdD3BEk1JiRlbnUf/9oADAMBAAIAAwAAABADAID/xAAWEQEBAQAAAAAAAAAAAAAAAAABESD/2gAIAQMBAT8QAjcf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFdIT//EAB4QAQACAQQDAAAAAAAAAAAAAAERIQAQMUFRYYGx/9oACAEBAAE/EClkAFglYJ8Xl1kNwXZjrQ1xCEcPBl8E6A89+sAII7JzjXgEBSYlyYrZm+d/uVsWluf/2Q==","aspectRatio":1,"src":"/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg","srcSet":"/static/ba9e631272af1b925d6491b30df3b36f/b3d27/boris.jpg 40w,\n/static/ba9e631272af1b925d6491b30df3b36f/5a735/boris.jpg 80w,\n/static/ba9e631272af1b925d6491b30df3b36f/6e81a/boris.jpg 120w,\n/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg 800w,\n/static/ba9e631272af1b925d6491b30df3b36f/97bce/boris.jpg 1878w","sizes":"(max-width: 800px) 100vw, 800px"}}]}}],"imgCaption":"Background image by Jeremy Bishop","imgSource":"https://unsplash.com/photos/JarNBwuPHr0","uniquePostId":9,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner"}},"relatedPosts":{"totalCount":4,"edges":[{"node":{"id":"8da3beae-1c9f-5fae-a6b9-3fd103f104ef","excerpt":"Elastic net is a combination of the two most popular regularized variants of linear regression: ridge and lasso. Ridge utilizes an L2 penalty and lasso uses an L1 penalty. With elastic net, you don't have to choose between these two models, because elastic net uses both the L2 and the L1 penalty! In practice, you will almost always want to use elastic net over ridge or lasso, and in this article you will learn everything you need to know to do so, successfully!","frontmatter":{"title":"Elastic Net Regression Explained, Step by Step","timeToRead":7,"date":"2021-06-26T08:28:02.234Z"},"fields":{"slug":"/machine_learning_models/elastic_net_regression/"}}},{"node":{"id":"f6288123-30a0-5918-8d58-1607badab75e","excerpt":"Lasso regression is an adaptation of the popular and widely used linear regression algorithm. It enhances regular linear regression by slightly changing its cost function, which results in less overfit models. Lasso regression is very similar to ridge regression, but there are some key differences between the two that you will have to understand if you want to use them effectively. In this article, you will learn everything you need to know about lasso regression, the differences between lasso and ridge, as well as how you can start using lasso regression in your own machine learning projects.","frontmatter":{"title":"Lasso Regression Explained, Step by Step","timeToRead":16,"date":"2021-06-20T16:35:36.458Z"},"fields":{"slug":"/machine_learning_models/lasso_regression/"}}},{"node":{"id":"a9afab9b-71b2-5906-a280-fb2dd2afa180","excerpt":"Ridge Regression is an adaptation of the popular and widely used linear regression algorithm. It enhances regular linear regression by slightly changing its cost function, which results in less overfit models. In this article, you will learn everything you need to know about Ridge Regression, and how you can start using it in your own machine learning projects.","frontmatter":{"title":"Ridge Regression Explained, Step by Step","timeToRead":26,"date":"2021-05-23T19:13:45.907Z"},"fields":{"slug":"/machine_learning_models/ridge_regression/"}}},{"node":{"id":"6be475fd-5ce4-55fb-a78f-49f61e026a2e","excerpt":"Linear regression is one of the most famous algorithms in statistics and machine learning. In this post you will learn how linear regression works on a fundamental level. You will also implement linear regression both from scratch as well as with the popular library scikit-learn in Python. You will learn when and how to best use linear regression in your machine learning projects. You do not need any knowledge prior to reading this article.","frontmatter":{"title":"Linear Regression Explained, Step by Step","timeToRead":23,"date":"2021-03-20T18:17:30.352Z"},"fields":{"slug":"/machine_learning_models/linear_regression/"}}}]}},"pageContext":{"slug":"/machine_learning_models/ridge_regression/","prev":{"excerpt":"If you are just starting out in machine learning and building your first real models, you will have to split your dataset into a train set as well as a test set. But what benefits does this splitting yield? How can you split your dataset optimally? In this article, we will go through these questions and explore why splitting your dataset makes sense and how you can split your dataset properly.","frontmatter":{"title":"How to Split Your Dataset the Right Way","tags":["Dataset Optimization"],"date":"2021-05-01T18:09:26.968Z","dateModified":"2021-05-01T18:09:26.968Z","changelog":[],"draft":false,"excerpt":"If you are just starting out in machine learning and building your first real models, you will have to split your dataset into a train set as well as a test set. But what benefits does this splitting yield? How can you split your dataset optimally? In this article, we will go through these questions and explore why splitting your dataset makes sense and how you can split your dataset properly.","uniquePostId":8,"timeToRead":15,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACcUlEQVQozxXPaW/SYAAA4DfA1pVCOVooPaDH27ctZZwFBgwm24QKHedidAM3sssNk0EkXlGTJWYxUWf85Ff/mb/F+PyDB7gk3S0bHsUEAsSNfDRj6NUCY7VyzfLZ+cFqXqnbla29DJlpl/fSWtkEMRlDOirKSkEFLslwK0kgqFZ3qO2P0hX286pvT7/dv+v++rTz9w+4e3v0/aNTefpbq5a2HLva73tEKZZGAR0BTFABIxadUef0AvAQgzqupoCUI1QNhxqV3mRzWsBQ1yEKGEjIiJObWaU/AELCizSwEeSImHJ8s/DpGcBpW/X45dDfb4WWh66BHSHNPpMdRzOHlGEz2Qadko1q8mSxEHK6nIfA42cSWrZ3OAEkQyYLQZTYrxPTsWQ/lpVUBBZrXGZfstpa8dGGlHcJEuDE8cVZubUFOAWs+WOsnCo32jSn/Hj4+eLy2pmeP79e3n19uFis5svb1YfXw+nx3f2Xm+XtydVVoWHsjAdiycJkCPCwgIeF2m6XjCLJKChJy8hWAnLKbDyazeetXhPVdphcsdHrJJsdy3bYfO7Js4OAoRL6/zPv9nMJVTvqpXud7Z5tOe3SZFzjN3cBzXsRXBelkAGlLDceQGfUqDs9rrjthTKQdJCQxE2dD9HUpG/OjjtvXo3mZwdX02bS6nrDfESSYQ7qJYgp8OUs/341PZ0NKTMJEqpb0kDNUnYrJsWKAGc9BBOJm0REARtRjGQISiBp3h/hfXEJk5EnoQdMizR0P1LdIiIQAsEIG4ryWFDwUQJBxbEgi4d5Hx33hgQ8xHspgWZ4X5in+bhXUYGgrEEdiHpAV8vbiX/nc3rHlqIDkgAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/e9b8f309bba40fd754df6982c7c7159d/87706/SplitDataTheRightWay_Thumbnail.png","srcSet":"/static/e9b8f309bba40fd754df6982c7c7159d/0dee1/SplitDataTheRightWay_Thumbnail.png 750w,\n/static/e9b8f309bba40fd754df6982c7c7159d/8beaa/SplitDataTheRightWay_Thumbnail.png 1080w,\n/static/e9b8f309bba40fd754df6982c7c7159d/87706/SplitDataTheRightWay_Thumbnail.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/e9b8f309bba40fd754df6982c7c7159d/a66aa/SplitDataTheRightWay_Thumbnail.webp 750w,\n/static/e9b8f309bba40fd754df6982c7c7159d/65dd5/SplitDataTheRightWay_Thumbnail.webp 1080w,\n/static/e9b8f309bba40fd754df6982c7c7159d/71d4d/SplitDataTheRightWay_Thumbnail.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Boris Giba","bio":"Hi! I'm Boris and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"aspectRatio":1,"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAQFBv/EABYBAQEBAAAAAAAAAAAAAAAAAAABAv/aAAwDAQACEAMQAAAB25bcya1Bcz84HRg//8QAGxAAAwACAwAAAAAAAAAAAAAAAQIDABESMTP/2gAIAQEAAQUCo3FFo4OEbG1R8u5SXZh4f//EABURAQEAAAAAAAAAAAAAAAAAAAEg/9oACAEDAQE/ASP/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAeEAABBAEFAAAAAAAAAAAAAAABAAIQETESIUFRcf/aAAgBAQAGPwIkZTQ/Sb6ilvxBIzDPF//EABoQAQACAwEAAAAAAAAAAAAAAAERIQAQMUH/2gAIAQEAAT8hQ1PjHRkxa9E644V8JdD3BEk1JiRlbnUf/9oADAMBAAIAAwAAABADAID/xAAWEQEBAQAAAAAAAAAAAAAAAAABESD/2gAIAQMBAT8QAjcf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFdIT//EAB4QAQACAQQDAAAAAAAAAAAAAAERIQAQMUFRYYGx/9oACAEBAAE/EClkAFglYJ8Xl1kNwXZjrQ1xCEcPBl8E6A89+sAII7JzjXgEBSYlyYrZm+d/uVsWluf/2Q==","sizes":"(max-width: 800px) 100vw, 800px","src":"/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg","srcSet":"/static/ba9e631272af1b925d6491b30df3b36f/fd013/boris.jpg 200w,\n/static/ba9e631272af1b925d6491b30df3b36f/25252/boris.jpg 400w,\n/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg 800w,\n/static/ba9e631272af1b925d6491b30df3b36f/0ff54/boris.jpg 1200w,\n/static/ba9e631272af1b925d6491b30df3b36f/06655/boris.jpg 1600w,\n/static/ba9e631272af1b925d6491b30df3b36f/97bce/boris.jpg 1878w"}}]}}]},"fields":{"layout":"post","slug":"/dataset_optimization/split_data_the_right_way/"}},"next":{"excerpt":"Lasso regression is an adaptation of the popular and widely used linear regression algorithm. It enhances regular linear regression by slightly changing its cost function, which results in less overfit models. Lasso regression is very similar to ridge regression, but there are some key differences between the two that you will have to understand if you want to use them effectively. In this article, you will learn everything you need to know about lasso regression, the differences between lasso and ridge, as well as how you can start using lasso regression in your own machine learning projects.","frontmatter":{"title":"Lasso Regression Explained, Step by Step","tags":["Machine Learning Models"],"date":"2021-06-20T16:35:36.458Z","dateModified":"2021-06-26T08:28:02.234Z","changelog":[["2021-06-26T08:28:02.234Z","Added section about SGDRegressor."],["2022-01-18T09:43:47.766Z","Fixed small typo in section about qualitative difference between Ridge and Lasso. Thanks to Tianyu for pointing this out!"]],"draft":false,"excerpt":"Lasso regression is an adaptation of the popular and widely used linear regression algorithm. It enhances regular linear regression by slightly changing its cost function, which results in less overfit models. Lasso regression is very similar to ridge regression, but there are some key differences between the two that you will have to understand if you want to use them effectively. In this article, you will learn everything you need to know about lasso regression, the differences between lasso and ridge, as well as how you can start using lasso regression in your own machine learning projects.","uniquePostId":10,"timeToRead":16,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACQElEQVQozz3OSXOaYADGcT5CzC1NqqzyIjsCyssSIyIKiIJLQNTapnGJTZpMD520nUw+eoceOvO7PYfnj0z77t0ivbtNd/k8C/tFPFyOhsu4VIyj1SRaTUalZFRMomIcribRMg6zcJBHAZKF/mF1+/x1+2P/ZZfPdsv5YZ3vi8Vxszxuiv/us+lxnc2DwaTv3uezcl0vkYHVmg+90Y21moSzQXfa7ywCL/Wus2hQxMG/hKCIg6Rrp72OB9tOU0n93maWJL6HoBdXtiI5Mu+bLZNvOBJ/3RRcTYEcMDna5gHkaFtkHVmEAtdiGc+CrtF2oaEwAKlUPrCAt5tSNryxBRoyeGxr9+v8bpX/ef35bb97eXo8bNeRqUZQUSjMksU+1Ia2YUs8cn5+geMNHG/okqyxILT0DkdFN/Yk8JfpuEii9TRehP2uxHhNTq/jjio7qtLRVQXUkUrlgiS5OsVdXhLRKHl/e3v//fpyOj49HJ5Ph912s//86fF4ePv1+v3hEPe6hsB1WpqjqQJFIGdn5TNJNEiSpUhWU40gHJ9Oj3k6TgM/HvTn4ygNh1ky9juOIQmmXIKyyKA1hCDYVvtaktqCoOE4g+NM9SOFYQCQjMoyjsxZImuKrME3TIk3RN6FRpOhWwJXv7pEyCql1FmG5lGUJgmGIssEkmgQRBnS5ITA1JOOEVr6EOqmyHm26Tum2ZQYrIqogFWBQOFMDaVrKI2VAIYBHAMYClAUUEQDyrKrK11daQtszzI9C0JZArXqX0zno6oMKKD7AAAAAElFTkSuQmCC"},"images":{"fallback":{"src":"/static/1cbee7448495e7e7fa3771995e853206/87706/LassoRegression_Thumbnail.png","srcSet":"/static/1cbee7448495e7e7fa3771995e853206/0dee1/LassoRegression_Thumbnail.png 750w,\n/static/1cbee7448495e7e7fa3771995e853206/8beaa/LassoRegression_Thumbnail.png 1080w,\n/static/1cbee7448495e7e7fa3771995e853206/87706/LassoRegression_Thumbnail.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/1cbee7448495e7e7fa3771995e853206/a66aa/LassoRegression_Thumbnail.webp 750w,\n/static/1cbee7448495e7e7fa3771995e853206/65dd5/LassoRegression_Thumbnail.webp 1080w,\n/static/1cbee7448495e7e7fa3771995e853206/71d4d/LassoRegression_Thumbnail.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Boris Giba","bio":"Hi! I'm Boris and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"aspectRatio":1,"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAQFBv/EABYBAQEBAAAAAAAAAAAAAAAAAAABAv/aAAwDAQACEAMQAAAB25bcya1Bcz84HRg//8QAGxAAAwACAwAAAAAAAAAAAAAAAQIDABESMTP/2gAIAQEAAQUCo3FFo4OEbG1R8u5SXZh4f//EABURAQEAAAAAAAAAAAAAAAAAAAEg/9oACAEDAQE/ASP/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAeEAABBAEFAAAAAAAAAAAAAAABAAIQETESIUFRcf/aAAgBAQAGPwIkZTQ/Sb6ilvxBIzDPF//EABoQAQACAwEAAAAAAAAAAAAAAAERIQAQMUH/2gAIAQEAAT8hQ1PjHRkxa9E644V8JdD3BEk1JiRlbnUf/9oADAMBAAIAAwAAABADAID/xAAWEQEBAQAAAAAAAAAAAAAAAAABESD/2gAIAQMBAT8QAjcf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFdIT//EAB4QAQACAQQDAAAAAAAAAAAAAAERIQAQMUFRYYGx/9oACAEBAAE/EClkAFglYJ8Xl1kNwXZjrQ1xCEcPBl8E6A89+sAII7JzjXgEBSYlyYrZm+d/uVsWluf/2Q==","sizes":"(max-width: 800px) 100vw, 800px","src":"/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg","srcSet":"/static/ba9e631272af1b925d6491b30df3b36f/fd013/boris.jpg 200w,\n/static/ba9e631272af1b925d6491b30df3b36f/25252/boris.jpg 400w,\n/static/ba9e631272af1b925d6491b30df3b36f/2f1b1/boris.jpg 800w,\n/static/ba9e631272af1b925d6491b30df3b36f/0ff54/boris.jpg 1200w,\n/static/ba9e631272af1b925d6491b30df3b36f/06655/boris.jpg 1600w,\n/static/ba9e631272af1b925d6491b30df3b36f/97bce/boris.jpg 1878w"}}]}}]},"fields":{"layout":"post","slug":"/machine_learning_models/lasso_regression/"}},"primaryTag":"Machine Learning Models"}},"staticQueryHashes":["13077718","1316742457","1394499604","2907728042","3762220167"]}