{"componentChunkName":"component---src-templates-post-tsx","path":"/model_optimization/bias_and_variance/","result":{"data":{"logo":{"childImageSharp":{"gatsbyImageData":{"layout":"fixed","backgroundColor":"#f8f8f8","images":{"fallback":{"src":"/static/e43c6530b7b2b056de6bfee951c9e393/b181d/compasslogo.png","srcSet":"/static/e43c6530b7b2b056de6bfee951c9e393/b181d/compasslogo.png 1920w","sizes":"1920px"},"sources":[{"srcSet":"/static/e43c6530b7b2b056de6bfee951c9e393/e02c9/compasslogo.webp 1920w","type":"image/webp","sizes":"1920px"}]},"width":1920,"height":472}}},"mdx":{"body":"const _excluded = [\"components\"];\n\nfunction _extends() { _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; }; return _extends.apply(this, arguments); }\n\nfunction _objectWithoutProperties(source, excluded) { if (source == null) return {}; var target = _objectWithoutPropertiesLoose(source, excluded); var key, i; if (Object.getOwnPropertySymbols) { var sourceSymbolKeys = Object.getOwnPropertySymbols(source); for (i = 0; i < sourceSymbolKeys.length; i++) { key = sourceSymbolKeys[i]; if (excluded.indexOf(key) >= 0) continue; if (!Object.prototype.propertyIsEnumerable.call(source, key)) continue; target[key] = source[key]; } } return target; }\n\nfunction _objectWithoutPropertiesLoose(source, excluded) { if (source == null) return {}; var target = {}; var sourceKeys = Object.keys(source); var key, i; for (i = 0; i < sourceKeys.length; i++) { key = sourceKeys[i]; if (excluded.indexOf(key) >= 0) continue; target[key] = source[key]; } return target; }\n\n/* @jsxRuntime classic */\n\n/* @jsx mdx */\nconst _frontmatter = {\n \"layout\": \"post\",\n \"title\": \"Bias, Variance, and Overfitting Explained, Step by Step\",\n \"author\": [\"Lari Giba\"],\n \"tags\": [\"Model Optimization\"],\n \"image\": \"img/BiasAndVariance/BiasAndVariance_Thumbnail.png\",\n \"date\": \"2021-04-17T19:54:40.006Z\",\n \"dateModified\": \"2021-04-17T19:54:40.006Z\",\n \"changelog\": [[\"2022-03-19T19:39:23.993Z\", \"Fixed typo, thanks to Ali Jan!\"]],\n \"draft\": false,\n \"imgCaption\": \"Background image by Sora Shimazaki\",\n \"imgSource\": \"https://www.pexels.com/de-de/foto/urteilsskala-und-hammer-im-richteramt-5669602/\",\n \"timeToRead\": 20,\n \"uniquePostId\": 7,\n \"excerpt\": \"You have likely heard about bias and variance before. They are two fundamental terms in machine learning and often used to explain overfitting and underfitting. If you're working with machine learning methods, it's crucial to understand these concepts well so that you can make optimal decisions in your own projects. In this article, you'll learn everything you need to know about bias, variance, overfitting, and the bias-variance tradeoff.\",\n \"dependencies\": [],\n \"proficiencyLevel\": \"Beginner\"\n};\n\nconst makeShortcode = name => function MDXDefaultShortcode(props) {\n console.warn(\"Component \" + name + \" was not imported, exported, or provided by MDXProvider as global scope\");\n return mdx(\"div\", props);\n};\n\nconst ConditionalLazyPlot = makeShortcode(\"ConditionalLazyPlot\");\nconst ToolTip = makeShortcode(\"ToolTip\");\nconst ConditionalLazyPlotInteractive = makeShortcode(\"ConditionalLazyPlotInteractive\");\nconst layoutProps = {\n _frontmatter\n};\nconst MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n let {\n components\n } = _ref,\n props = _objectWithoutProperties(_ref, _excluded);\n\n return mdx(MDXLayout, _extends({}, layoutProps, props, {\n components: components,\n mdxType: \"MDXLayout\"\n }), mdx(\"h2\", {\n \"id\": \"outline\"\n }, `Outline`), mdx(\"p\", null, `Bias and variance are very fundamental, and also very important concepts.\nUnderstanding bias and variance well will help you make\nmore effective and more well-reasoned decisions in your own machine learning projects, whether\nyou’re working on your personal portfolio or at a large organization.\nIn this article, you will learn what bias and variance are, what the so-called `, mdx(\"em\", {\n parentName: \"p\"\n }, `bias-variance tradeoff`), ` is, and how you can make the best\ndecisions in your own machine learning projects, to create the best-performing machine learning models.`), mdx(\"h2\", {\n \"id\": \"the-problem\"\n }, `The Problem`), mdx(\"p\", null, `Let’s start by looking at a particular scenario.\nImagine you are teaching your friend the basics of machine learning.\nYou give them a small dataset of exam results. Each entry in the dataset contains the number of hours a student has spent studying for the exam\nas well as the number of points (between 0 and 100) the student has achieved in said exam.\nYou then tell your friend to try and predict the number of points achieved based on the number of hours studied.\nThe dataset looks like this:`), mdx(ConditionalLazyPlot, _extends({}, polynomialData, {\n mdxType: \"ConditionalLazyPlot\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"604px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"66.60000000000001%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABu0lEQVQ4y4VTy3KbQBDEkpFALGh5g1hekpEiS6U8KhX7kMohl9zy/3/TqZk1ayzLlcPA1tDTdM/OWPfrChTzYIOwOiBtHxHXR2T9GU7UcH78PsWO52lQ3qLHGIuwNrGMGgN6S/D6g5lf8vvOyzHzC85bVLxKO3hpZwoJKPIdZLVntUH58E6NHTbw1RluPiDafYdfPWImClhx/QnrcsAqaW/aeEckFeZ+CVvWqL/9QTz8QPf8F2Jz1IShOsAvHuDGDQP/R3jnpbDDGrL7Atl+ZqtEPg90rRVWe4h8Cy/bwkm3Hzb8nnvcIDv+xLq5wEm2hmRaoxXmO+6ZX58hqtMNUm1T9l/Z4jJqMRM527/+sUX9E6QuUqzQSXoGG5BUxlY8PDHxqOxWWNGkh8uk59sSdGNjP6VCcfqF8vKbFWpy9XYurxW6Sce3TMqC+sykrCLYwCv2kO0Fy7iFm+7MHNqygm1ufkI4HkTW82bQRyeiwdbhlwMWocKKHISKczSzC6mwSjrtLNQ5Q0jyxwbTmdSMm0PkZJ9caIziM+VomwhDWPdljq3pHk73loJGKetOoOGPFMUBSXPk3NjH65p/yKw8xKoEiiAAAAAASUVORK5CYII=')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/ee444ab1b41797b78b129f4bc0f154fd/601b1/polynomialData.webp 500w\", \"/static/ee444ab1b41797b78b129f4bc0f154fd/88e19/polynomialData.webp 604w\"],\n \"sizes\": \"(max-width: 604px) 100vw, 604px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/ee444ab1b41797b78b129f4bc0f154fd/0eb09/polynomialData.png 500w\", \"/static/ee444ab1b41797b78b129f4bc0f154fd/1994c/polynomialData.png 604w\"],\n \"sizes\": \"(max-width: 604px) 100vw, 604px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/ee444ab1b41797b78b129f4bc0f154fd/1994c/polynomialData.png\",\n \"alt\": \"Exam Results Dataset\",\n \"title\": \"Exam Results Dataset\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `Your friend is just getting started with machine learning and recently read `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"/machine_learning_models/linear_regression\"\n }, `an article about linear regression`), `.\nNow they want to try out linear regression themselves! After some time,\nthey come back to you and say that they have had difficulty creating a model that achieves a low error.\nYou take their model and make a series of predictions.\nThen, you plot those predicted values alongside the actual exam scores for every number of hours studied in the dataset.\nTo make the visuals easier to digest, the predictions are displayed as a continuous function.\nThe resulting plot looks like this:`), mdx(ConditionalLazyPlot, _extends({}, polynomialPredUnderFitNoAnnotations, {\n mdxType: \"ConditionalLazyPlot\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1110px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"63.6%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB1UlEQVQ4y3VT226bQBBFsTHGYO5gA97FXAwYJ3IURY2qqu1Ln/rU//+ZU82wJL4kD2dH7M4cZs7MaHNPgqB7Equ4xiqpYSUN2+ntFvol/At4EpruF6OjX8DLj3DzHv5ugJv1YzC9K7z7ElyBuSfYzhToXqPDCEsso/L9YYKuSPQLMiOqYEQ1zKTFMm5gZQOcrIeTdVyhZm8OCMQJVlJjoQIXN0R8F5SYOTlc+Qzx+hfVz3+IhjcstzWi8gmb6ox12kKjwxcDzKhUmhSf4sHO2G4ef8Bvn2GLHnpAUki2kx9nSNqRpXLuSL2CA+LTd0Snb7CIyBOYOULJUFw1TFunHby8xzo7witfYG7aUWxFtEgKuM0T0vNv1uxhnUP3919WotGIUEepZBJ4Qc6BhJlWsMQB5rZGeHjDMmowc3ZYBPsrjec3FWlO2nFTlmEJK+0Q9q+wiw56KPkH+fkPkuHXlxndEdIA29sWdlbDEg3c8sxZ0Hy58gw7O/JY0d00AQxuhLwj1SYxV5saRriH4RdcPs2lGTcwgj1/EyHNGVlqINvt4T7Dj+lXlyoTM654/cZ1bGBGFRNMmZI/V+JdZ6ndajFZalQgxhX0d+NKhvKkNkp+GkOE/wGI10Lo1jgIwgAAAABJRU5ErkJggg==')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/e35271bd0b4c842f1a5c9daab74965a0/601b1/underFitNoAnnotations.webp 500w\", \"/static/e35271bd0b4c842f1a5c9daab74965a0/80c60/underFitNoAnnotations.webp 1000w\", \"/static/e35271bd0b4c842f1a5c9daab74965a0/d209b/underFitNoAnnotations.webp 1110w\"],\n \"sizes\": \"(max-width: 1110px) 100vw, 1110px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/e35271bd0b4c842f1a5c9daab74965a0/0eb09/underFitNoAnnotations.png 500w\", \"/static/e35271bd0b4c842f1a5c9daab74965a0/1263b/underFitNoAnnotations.png 1000w\", \"/static/e35271bd0b4c842f1a5c9daab74965a0/ffc38/underFitNoAnnotations.png 1110w\"],\n \"sizes\": \"(max-width: 1110px) 100vw, 1110px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/e35271bd0b4c842f1a5c9daab74965a0/ffc38/underFitNoAnnotations.png\",\n \"alt\": \"Underfitted Model\",\n \"title\": \"Underfitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `You look at the plot and notice a couple of things.`), mdx(\"ol\", null, mdx(\"li\", {\n parentName: \"ol\"\n }, mdx(\"p\", {\n parentName: \"li\"\n }, `There definitely seems to be a relationship between the\nnumber of hours studied and the number of points achieved.`)), mdx(\"li\", {\n parentName: \"ol\"\n }, mdx(\"p\", {\n parentName: \"li\"\n }, `This relationship does seem somewhat linear,\nespecially when considering the segment between 40 and 60 hours studied.`)), mdx(\"li\", {\n parentName: \"ol\"\n }, mdx(\"p\", {\n parentName: \"li\"\n }, `However, the linear relationship is not that apparent throughout the entire graph.\nWhen you take a look at the interval between 0 and 30 hours studied, it does seem to have\nmore of a downward trend, in contrast to the interval between 40 and 60 hours studied.`))), mdx(\"p\", null, `With linear regression, we can only draw a straight line (a linear function) to model the relationship between\nthe two features (number of hours studied) and the target (number of points achieved). But the data does not seem to follow the same trend throughout the entire dataset.\nWe can’t just cut off one half of the dataset, so what can we do instead?\nIn this case, we need to use a different model. No plain linear regression model can capture this\nrelationship well. However, we can perform a small adjustment and use `, mdx(\"em\", {\n parentName: \"p\"\n }, `polynomial regression`), ` instead.\nWith this, we can capture nonlinear relationships in our data using polynomial functions, instead of just\nlinear ones.\nYou explain this idea to your friend and they go ahead and try out polynomial regression.`), mdx(\"div\", null, mdx(\"p\", null, `Imagine you are in the shoes of your friend right now. You have to find out the degree of your polynomial regression model,\nmeaning the maximum power which you will apply to your features. Is a degree of 2 or 3 already enough? Do you\nneed a degree of 10 or higher? Pause for a second and think about this question.\nBelow you can find an interactive visualization where you can take a look at all the possible polynomial regression\nplots for the degrees 1-20. After you’ve come up with a number in your head, take a look at the visualization and see if your intuition was right.`, mdx(ToolTip, {\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, `The transitions between the functions might be a little bit glitchy and I have no idea why 😅. Hopefully it’s not too distracting!`))), mdx(ConditionalLazyPlotInteractive, _extends({}, polynomialPredDegreeInteractive, {\n mdxType: \"ConditionalLazyPlotInteractive\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1091px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"65.8%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABvElEQVQ4y3VTy3KjMBCkAiaAeUg2YBAY8cYkcWXJJbuH3Sqf9/+/p7ckjMxWyKFLYjT09Lw0gxTYUQGO52MFK6wV5NsdxgqL/xrLm7b+INkIen4BzWc8CB4+xvr+X5DZrulBAfNQwgqrTadNHEr5jzypuFfKX3OTDj4b4J7a7wm+oEQ8/kHY/wL/+Rd+dYUe5DMhyUeZ6j5uNmuzhu5lePIyuPkr2HQDaa8g3QTzyB8KvbSHgCAUaWylujTHK95A+wmke4eVNND9DLqfY0c4DCGGcGgBuyBgA4JsRMAnOMkwE5E7DhzPcQWvHME+biD1JFXK7tO5jirorHCAnw5wogr7dIQVdTKqGZaw0xpO1mAX5jj2n7DCFk8uUyRfsqBK4QVOVMM+9fDLK7zyBTarYYYF/OIdydsN8fh77uw39X0oTHo4cQs3bWGzCrSb4KQDjOAsUyLlD+zTC6y4nQlVjVe1vs+nsKnB9lgnZ1E82uKkXAayogYm5XdbAUc2j8OJGukvtkvYlMK13AXLkO9PLeywlsTibh0riIyWNV1KoNZ0IdxtrJOwidqKVRSDT7KLnNdD8SrJ9OC8uXr/AJWYQhednFHGAAAAAElFTkSuQmCC')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/cda451270a7756700daf6e987dc83775/601b1/polynomialDegreeInteractive.webp 500w\", \"/static/cda451270a7756700daf6e987dc83775/80c60/polynomialDegreeInteractive.webp 1000w\", \"/static/cda451270a7756700daf6e987dc83775/c1fee/polynomialDegreeInteractive.webp 1091w\"],\n \"sizes\": \"(max-width: 1091px) 100vw, 1091px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/cda451270a7756700daf6e987dc83775/0eb09/polynomialDegreeInteractive.png 500w\", \"/static/cda451270a7756700daf6e987dc83775/1263b/polynomialDegreeInteractive.png 1000w\", \"/static/cda451270a7756700daf6e987dc83775/acfc2/polynomialDegreeInteractive.png 1091w\"],\n \"sizes\": \"(max-width: 1091px) 100vw, 1091px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/cda451270a7756700daf6e987dc83775/acfc2/polynomialDegreeInteractive.png\",\n \"alt\": \"Underfitted Model\",\n \"title\": \"Underfitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `If you estimated the optimal degree to be between 3 and 5, congratulations! You’ve hit the gold spot.\nWe, with our human eyes, can tell that the model with a degree between 3 and 5 best fits the data.\nBut what’s interesting is that by increasing the degree of our polynomial regression model,\nour function always gets closer and closer to our data points. F.e. the model with degree 20\nis touching a lot more points directly in the middle than the function with degree 3. The higher\nour degree is, the “wigglier” our function can get. Because a model with a higher degree has more degrees of freedom,\nit can take on more complicated function shapes. So maybe the model with degree=20 is truly the best one? Ok, let’s take a step back.`), mdx(\"p\", null, `We have all of these different functions and some can capture the relationship between\nour features and our target better than others. If we were to compare the models with degree=1 and degree=20 respectively,\nwe would say that the one with degree=20 “captures the relationship better”. But in a practical scenario, there\nhas to be a better way of expressing this without having to say “captures this relationship better” every time.\nWouldn’t it be more satisfying if we could somehow say that the model with degree=20 has a higher/lower\n`, mdx(\"em\", {\n parentName: \"p\"\n }, `something`), ` than the model with degree=1? We can do exactly that by introducing the notion of `, mdx(\"strong\", {\n parentName: \"p\"\n }, `bias`), `.`), mdx(\"h2\", {\n \"id\": \"the-bias\"\n }, `The Bias`), mdx(\"p\", null, `We want the bias to express how well a certain machine learning model fits a particular dataset.\nSo let’s try and come up with a formal definition of bias ourselves and look at some examples. Then, we will take a look at how the bias\nis defined in statistics and whether our own definition matches the one from statistics. Our definition might look similar to this:`), mdx(\"blockquote\", null, mdx(\"p\", {\n parentName: \"blockquote\"\n }, `The bias of a specific machine learning model trained on a specific dataset describes how well this\nmachine learning model can capture the relationship between the features and the targets.\nSo for our example, the bias of any one model would tell\nus how well this particular model can predict the exam points received for any number of hours studied in our specific dataset.`)), mdx(\"p\", null, `That seems like a reasonable definition.`), mdx(\"h3\", {\n \"id\": \"practical-examples-bias\"\n }, `Practical Examples (Bias)`), mdx(\"p\", null, `Let’s take a look at three of the above models and compare them using our new\ndefinition of bias. We’ll take a look at the polynomial regression models with degrees 1,4, and 15 respectively.\nBelow is the graph showing our dataset and the predictions of the model with a degree of 1.`), mdx(ConditionalLazyPlot, _extends({}, polynomialPredUnderFit, {\n mdxType: \"ConditionalLazyPlot\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1088px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"64.2%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABvUlEQVQ4y31T226jMBREKZdwN9QBAgn3W9JtquZlq7zsSvsH+///https://misfmcwntg8jjh08mnnmrghbaql33xo8/UBfleVYn38HWauIhfzx0gFOMhE6cYcn/wh5/oD57MnLaC1JFbEwdw3MXU0EAhvvADtuER7PcJLuUVFYwkoGbHkL93iBGXVQ2USqsOwEATtqoIflckmsjbC829NoPanbX/4gfv2F5vdf+PUrCVkIRYsWr6CtZrYehcTGyUhJ0F4R/fiAmTXYxs1SRy376Qg36eGm4yz9ixFsRpCDn34iutzg5CO05wKqnxPuTBEKxZy8dAQr32k2QoU6E+m8hJGUYN0bopcbrGTExs3+67RCjiY9TF7CTkZowdT6Nq6pJfvQQecFeH+DwRty9W6uXwmD45niYu1qWPse/PwBrzrBiCtSysorGcCH28qYb3Joxx3Fxj10sLIaXvFCCsRs9LACq64weE17+uyy/kC4MkW6acfNFBOWw9o1dMmKGio2woJyKurEHp3N2RV3nLj9VLiWu86gOj9FYRilIOlhRS38bFzUylkaz9X901vnTkIUC8IlVuKNpwPY4bQQrCFb/gdCF0LCbVT2YgAAAABJRU5ErkJggg==')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/4e1869b523f8e8163cfe73070cb0ae7a/601b1/underFitScreenshot.webp 500w\", \"/static/4e1869b523f8e8163cfe73070cb0ae7a/80c60/underFitScreenshot.webp 1000w\", \"/static/4e1869b523f8e8163cfe73070cb0ae7a/62aa4/underFitScreenshot.webp 1088w\"],\n \"sizes\": \"(max-width: 1088px) 100vw, 1088px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/4e1869b523f8e8163cfe73070cb0ae7a/0eb09/underFitScreenshot.png 500w\", \"/static/4e1869b523f8e8163cfe73070cb0ae7a/1263b/underFitScreenshot.png 1000w\", \"/static/4e1869b523f8e8163cfe73070cb0ae7a/347c0/underFitScreenshot.png 1088w\"],\n \"sizes\": \"(max-width: 1088px) 100vw, 1088px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/4e1869b523f8e8163cfe73070cb0ae7a/347c0/underFitScreenshot.png\",\n \"alt\": \"Underfitted Model\",\n \"title\": \"Underfitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `As you see, this model predicts our targets very poorly. The predicted values are not that close to the data points and\ntherefor we can say that this model has a very high bias because it does not perform its task well,\nwhich is to predict the number of points achieved based on the number of hours studied.`), mdx(\"div\", null, mdx(\"p\", null, `In practice, we probably don’t want to generate a fancy graph displaying our predictions and then estimate\nhow high our bias is based on how “good” the generated graph is. If we want to compare two models, then there has to\nbe a more concrete way of measuring the bias. This is why in practice, we often use the (training) error`, mdx(ToolTip, {\n width: \"25em\",\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, `Why is it called “training” error and not just “error”? When training machine learning models, we often split our dataset into a “training” portion and a “testing” portion (and sometimes a “validation portion”).\nIn a nutshell, we use the training portion to train the model and the testing portion to evaluate the model. For this article, you don’t necessarily need to have a firm grasp of training and testing, but if you want to learn more about it, then you can check out `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"/model_optimization/train_and_test\"\n }, `this article here`), `, which covers the topic in more depth.`), ` of a particular model\nto make deductions about its bias. Since we have a regression-based task in front of us, it makes sense to use the mean squared error\nto measure the performance of a specific model. To make the results a bit easier to interpret, let’s also add a square root on top\nso we can immediately tell by how much the average point is approximately off. If you are wondering why we don’t use a different metric like the mean absolute error\nor why we don’t use the root mean squared error by default, all of this is explained in more detail in `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"/machine_learning_models/linear_regression#soar-vs-sosr\"\n }, `this segment of the article about linear regression`), `.`)), mdx(\"p\", null, `If we compute the root mean squared error (RMSE) for the predictions of this model, we get a value of 19.8.\nThis means that one single point can be off by up to ~19.8, i.e. if a student in truth achieved 80 points,\nour model might only give them 60, which would probably not make the student very happy.\nWith this we can capture the following behavior:`), mdx(\"ul\", null, mdx(\"li\", {\n parentName: \"ul\"\n }, `large training error -> large bias`)), mdx(\"p\", null, `Let’s now look at the model with degree 4:`), mdx(ConditionalLazyPlot, _extends({}, polynomialPredFit, {\n mdxType: \"ConditionalLazyPlot\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1088px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"64.6%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB00lEQVQ4y3VT2XKbQBCUJSPEIW7wysuNDozwrVTlwXYl+YL8/890akZZkGz5gdqpmaG3e6Z3ovkF6Lv2cuhhBSNusIhqmEkDVVP1S/Hn3IQC+mZuBme1Q1j08LMOYb4f8qpZ9Q4xnVzPhtxE848Ni6jixNTJMHVSzIMCftrxJcT8jKFPampoXg7jZgM9bEaGBBTmPZaCCuXw0zwoYcQ1FmHF8ZivcGWtELU/IQ9/UH/8hXh8w8xNj4A0M5JniTX0oGAZp/L4ZiXTzTC1Jax0h9XzO2zZwVp10OMapJQl0wJImlu0sOQGMyeFFhQXh26KDuHuAH/zBD1aH+fnZtC8sWdgSFuN9wfYcj8wOgWb+xXE4zui7gemS/ntppmhl97BjGrY6RZO1bMsbZhbiWsng9vcI2gPuDLF2Uy/ANLgySp0Elt/+wK3ecDMITk5jGQN+foLty+/Yd60F5mdMwwruLJlyXM/gy17uFUP47bGQpQI21d49RPbhOyjbDZs/TOg2iTZx4wbHjJdokcVjKRmhsyUfZqxEj0oGVy9JitZM/B/Y4/PRs1GmdyIGljJ8SnaYsMX2WLL9dM+qmungOe+G+Ol2LLpaSRkLU/eISp6Bv/uSf4DdLhCwzPme3wAAAAASUVORK5CYII=')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/81cb0e020dbf77772a776a22e4bb30ed/601b1/wellFitScreenshot.webp 500w\", \"/static/81cb0e020dbf77772a776a22e4bb30ed/80c60/wellFitScreenshot.webp 1000w\", \"/static/81cb0e020dbf77772a776a22e4bb30ed/62aa4/wellFitScreenshot.webp 1088w\"],\n \"sizes\": \"(max-width: 1088px) 100vw, 1088px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/81cb0e020dbf77772a776a22e4bb30ed/0eb09/wellFitScreenshot.png 500w\", \"/static/81cb0e020dbf77772a776a22e4bb30ed/1263b/wellFitScreenshot.png 1000w\", \"/static/81cb0e020dbf77772a776a22e4bb30ed/347c0/wellFitScreenshot.png 1088w\"],\n \"sizes\": \"(max-width: 1088px) 100vw, 1088px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/81cb0e020dbf77772a776a22e4bb30ed/347c0/wellFitScreenshot.png\",\n \"alt\": \"Well fitted Model\",\n \"title\": \"Well fitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `This model predicts our y a lot better than the first one. It still makes some errors,\nbut it does its job a lot better than the first model. If we compute the RMSE\nfor the predictions of this model, we get 7.68. It’s about 2.5 times better than the previous model!\nIf a student in truth achieved 80 points, our model might only give them 72 (or up to 88),\nwhich still is not ideal, but it’s considerably better than the previous model.\nWe achieved this decrease in error by just moderately increasing our degree. We’ve only increased the degree\nfrom 1 to 4, but managed to reduce our error by a factor of around 2.5! So this increase in complexity was certainly worth it.\nBecause our model has a rather small error, we can say that it has a small bias since it does its task relatively well.\nWith this we can capture the following behavior:`), mdx(\"ul\", null, mdx(\"li\", {\n parentName: \"ul\"\n }, `small training error -> small bias`)), mdx(\"p\", null, `Let’s now take a closer look at the model with degree 15:`), mdx(ConditionalLazyPlot, _extends({}, polynomialPredOverFit, {\n mdxType: \"ConditionalLazyPlot\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1096px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"62.2%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAABYlAAAWJQFJUiTwAAABz0lEQVQoz2VT246bMBBNN8vNAcwl4WYwmEsaspBqq3TVblVppapv/f/fOSs7CQvZh5FhxnPmzPHMSvNLSHv0ONzsKzzWw897eOygfPO4OuX/zH8fX8mPm5GoWdg8dkuavmmBtVvgwWaL+EoPKpBdCyfrYESVurSmhQIM+ROcdA/9jo3mlXCKE9xyRDT8hhV3qoBiaMcdQj7AYR1I1uCRcmhB+amlqTVawNgK8J//EI9/IN7+w0qveRJwE7dKt03awEwqxW6h2aQXx9pl0EOB3dMLHD7gwZHtFjDT+kPDTXQBpMUBVIzQfAF9W0FKMTELSuiBADv/RXT6BTNuLi0qfwWLNTPAuIXPethZh/D4DJv3ILyBldZKK3nxC0lAxTcE++8XnwKrpmILQHJluEkaeO0Avx9Bctl+rRKl4F79jPj0CjNqJ/Hn7O8YdvDzowIkrEb64xVuNcBIKpW8Pb6And8QdOfFXN5e+xOg1MDc1rDTFq7YgyQtjLACYQ3MXQ2vGaHRHLrHYQSViskc42bXu4vBXlMOIxSwolptgZxBGSRxq15XFrwNuhwzecrHNEOhAO2k+wCcr85iVLzysop5r07KDqDZAUFxVMXnWzTflHfI5iuz2Pa3dAAAAABJRU5ErkJggg==')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/8976ae5433aa16ddbc4181031d8a7dc6/601b1/overFitScreenshot.webp 500w\", \"/static/8976ae5433aa16ddbc4181031d8a7dc6/80c60/overFitScreenshot.webp 1000w\", \"/static/8976ae5433aa16ddbc4181031d8a7dc6/d2558/overFitScreenshot.webp 1096w\"],\n \"sizes\": \"(max-width: 1096px) 100vw, 1096px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/8976ae5433aa16ddbc4181031d8a7dc6/0eb09/overFitScreenshot.png 500w\", \"/static/8976ae5433aa16ddbc4181031d8a7dc6/1263b/overFitScreenshot.png 1000w\", \"/static/8976ae5433aa16ddbc4181031d8a7dc6/f7f62/overFitScreenshot.png 1096w\"],\n \"sizes\": \"(max-width: 1096px) 100vw, 1096px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/8976ae5433aa16ddbc4181031d8a7dc6/f7f62/overFitScreenshot.png\",\n \"alt\": \"overfitted Model\",\n \"title\": \"overfitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `This model does look a bit weirder than the previous two, but it does in fact predict our data even better than the second one.\nIf we compute the RMSE for the predictions of this model, we get 4.93. It’s about 1.5 times lower\nthan the previous error. However, we did increase our degree from 4 to 15, which is quite a lot.\nSo this increase in complexity certainly is not as good of a deal as the previous one.\nBut a lower is a lower error, right? So technically, this model is the best of the bunch.`), mdx(\"p\", null, `Because our model has a very low error, we can say\nthat it has a very low bias since it does its task very well.\nWith this we can capture the following behavior:`), mdx(\"ul\", null, mdx(\"li\", {\n parentName: \"ul\"\n }, `very small training error -> very small bias`)), mdx(\"h3\", {\n \"id\": \"revisiting-the-definition-of-bias\"\n }, `Revisiting the Definition of Bias`), mdx(\"p\", null, `Ok, so we have defined the bias in our own words, we’ve looked at a couple of examples and we know that\nthe training error is correlated with the bias. The lower the training error, the lower the bias.\nThe higher the training error, the higher the bias.`), mdx(\"p\", null, `When talking about the bias of a particular model, we always talk about one model and one dataset.\nIn our case, we’ve always talked about one particular polynomial regression model and one particular dataset of exam results.`), mdx(\"p\", null, `Let’s now look at the definition used in statistics and then try to see if our previous definition still holds.\nIn the words of `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"https://en.wikipedia.org/wiki/Bias_(statistics)\",\n \"target\": \"_blank\",\n \"rel\": \"nofollow noopener noreferrer\"\n }, `Wikipedia`), `:`), mdx(\"blockquote\", null, mdx(\"p\", {\n parentName: \"blockquote\"\n }, `Statistical bias is a feature of a statistical technique or of its results whereby the expected value of the results differs from the true underlying quantitative parameter being estimated.`)), mdx(\"p\", null, `If we look back on how we introduced the bias, this makes sense.\nWe compared the predictions of our model with the actual values present in our dataset. But the real definition is worded just a tiny\nbit differently, and contains two very important words:`), mdx(\"blockquote\", null, mdx(\"p\", {\n parentName: \"blockquote\"\n }, `Statistical bias is a feature of a statistical technique or of its results whereby the expected value of the results differs from the `, mdx(\"strong\", {\n parentName: \"p\"\n }, `true underlying`), ` quantitative parameter being estimated.`)), mdx(\"div\", null, mdx(\"p\", null, `This means that the bias is a way of describing the difference between the actual, `, mdx(\"em\", {\n parentName: \"p\"\n }, `true`), ` relationship in our data,\nand the one our model learned. In our examples, we’ve looked at the error between our predictions and the data points.\nSure, that is a very sensible way to measure the bias of our machine learning models. But what guarantees that our dataset accurately represents\nthe true relationship between our features and the target? Well, nothing. Our dataset can contain noise, meaning that, even if we `, mdx(\"em\", {\n parentName: \"p\"\n }, `could`), ` compute\nthe true relationship of our data, the points would not lie directly on it.\nSince I’ve created this dataset myself using a specific function`, mdx(ToolTip, {\n width: \"30em\",\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, ` This one, to be exact: `, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"math math-inline\"\n }, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"katex\"\n }, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"katex-mathml\"\n }, mdx(\"math\", {\n parentName: \"span\",\n \"xmlns\": \"http://www.w3.org/1998/Math/MathML\"\n }, mdx(\"semantics\", {\n parentName: \"math\"\n }, mdx(\"mrow\", {\n parentName: \"semantics\"\n }, mdx(\"mi\", {\n parentName: \"mrow\"\n }, `f`), mdx(\"mo\", {\n parentName: \"mrow\",\n \"stretchy\": \"false\"\n }, `(`), mdx(\"mi\", {\n parentName: \"mrow\"\n }, `X`), mdx(\"mo\", {\n parentName: \"mrow\",\n \"stretchy\": \"false\"\n }, `)`), mdx(\"mo\", {\n parentName: \"mrow\"\n }, `=`), mdx(\"mn\", {\n parentName: \"mrow\"\n }, `50`), mdx(\"mo\", {\n parentName: \"mrow\",\n \"stretchy\": \"false\"\n }, `(`), mdx(\"mi\", {\n parentName: \"mrow\"\n }, `sin`), mdx(\"mo\", {\n parentName: \"mrow\"\n }, `⁡`), mdx(\"mrow\", {\n parentName: \"mrow\"\n }, mdx(\"mo\", {\n parentName: \"mrow\",\n \"stretchy\": \"false\"\n }, `(`), mdx(\"mo\", {\n parentName: \"mrow\"\n }, `−`), mdx(\"mn\", {\n parentName: \"mrow\"\n }, `2.75`), mdx(\"mi\", {\n parentName: \"mrow\"\n }, `π`), mdx(\"mi\", {\n parentName: \"mrow\"\n }, `X`), mdx(\"mo\", {\n parentName: \"mrow\",\n \"stretchy\": \"false\"\n }, `)`)), mdx(\"mo\", {\n parentName: \"mrow\"\n }, `+`), mdx(\"mn\", {\n parentName: \"mrow\"\n }, `1`), mdx(\"mo\", {\n parentName: \"mrow\",\n \"stretchy\": \"false\"\n }, `)`)), mdx(\"annotation\", {\n parentName: \"semantics\",\n \"encoding\": \"application/x-tex\"\n }, `f(X) = 50(\\\\sin{(-2.75\\\\pi X)}+1)`)))), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"katex-html\",\n \"aria-hidden\": \"true\"\n }, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"base\"\n }, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"strut\",\n \"style\": {\n \"height\": \"1em\",\n \"verticalAlign\": \"-0.25em\"\n }\n }), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord mathnormal\",\n \"style\": {\n \"marginRight\": \"0.10764em\"\n }\n }, `f`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mopen\"\n }, `(`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord mathnormal\",\n \"style\": {\n \"marginRight\": \"0.07847em\"\n }\n }, `X`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mclose\"\n }, `)`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mspace\",\n \"style\": {\n \"marginRight\": \"0.2777777777777778em\"\n }\n }), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mrel\"\n }, `=`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mspace\",\n \"style\": {\n \"marginRight\": \"0.2777777777777778em\"\n }\n })), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"base\"\n }, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"strut\",\n \"style\": {\n \"height\": \"1em\",\n \"verticalAlign\": \"-0.25em\"\n }\n }), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `5`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `0`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mopen\"\n }, `(`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mop\"\n }, `sin`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mspace\",\n \"style\": {\n \"marginRight\": \"0.16666666666666666em\"\n }\n }), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mopen\"\n }, `(`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `−`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `2`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `.`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `7`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `5`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord mathnormal\",\n \"style\": {\n \"marginRight\": \"0.03588em\"\n }\n }, `π`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord mathnormal\",\n \"style\": {\n \"marginRight\": \"0.07847em\"\n }\n }, `X`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mclose\"\n }, `)`)), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mspace\",\n \"style\": {\n \"marginRight\": \"0.2222222222222222em\"\n }\n }), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mbin\"\n }, `+`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mspace\",\n \"style\": {\n \"marginRight\": \"0.2222222222222222em\"\n }\n })), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"base\"\n }, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"strut\",\n \"style\": {\n \"height\": \"1em\",\n \"verticalAlign\": \"-0.25em\"\n }\n }), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mord\"\n }, `1`), mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"mclose\"\n }, `)`))))), ` `), `, I know the true relationship between the features\nand the target. Below I have added a plot displaying this particular function. I’ve also added in the dataset and a button\nwith which you can toggle on or off the noise in the dataset.`)), mdx(ConditionalLazyPlotInteractive, _extends({}, trueRelationshipInteractive, {\n mdxType: \"ConditionalLazyPlotInteractive\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1068px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"60%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAABYlAAAWJQFJUiTwAAABoklEQVQoz3WTX0+DMBTFUbcxYWxQ/jMoUFgdY0wTNcbExCd98vt/nWN6GQu67aGU3lN+t/f0ok1sjmFMnRzj9bmW4W7FcbtUc3Zc97Fhj6Yec0/g3hen4MWhNLeC4Qs4awkjlDDiBov4AYtwQwwF1+y0wTJRwfoqcMYK3C5TuLxB+/6N3ccPsu0LnKhAUB4QVU8Iyg7LWEJz+R5WLCnzJeDMyQlmsBSie8OqfIEePJwsGFtCJdvrho5sJxJTpzgDTlYZrLRFtX+FGW97wCrttfG+I1hj2Q6LSILXLfzqGVNWYuoMGTkmrIR8/oRXHnBjran8IfGlishDM9zACgVSsT1tmrEcN1YKjzeIqz20RXKE/T3VGZDxljy89yt4eYts0+HO7mH88RPt+xfMZEd+ndlxCahuWMEMr4Du1wjFASxIwGIBLjvSdK/q22ZonxHsP1QbAla0oTLnbgnDzeGua8zUuy/ops2wJl21l86Kq22mDVkGf3S3JNPnfv+hGdSkqVlpvd7/DFeB46OrbrciCb/ooC7MSXc0vHxPa6X/L3nch7+NcCrphfhtywAAAABJRU5ErkJggg==')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/faa27055e91828f7427fbafa299beca3/601b1/trueRelationshipScreenshot.webp 500w\", \"/static/faa27055e91828f7427fbafa299beca3/80c60/trueRelationshipScreenshot.webp 1000w\", \"/static/faa27055e91828f7427fbafa299beca3/a22c7/trueRelationshipScreenshot.webp 1068w\"],\n \"sizes\": \"(max-width: 1068px) 100vw, 1068px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/faa27055e91828f7427fbafa299beca3/0eb09/trueRelationshipScreenshot.png 500w\", \"/static/faa27055e91828f7427fbafa299beca3/1263b/trueRelationshipScreenshot.png 1000w\", \"/static/faa27055e91828f7427fbafa299beca3/adc5a/trueRelationshipScreenshot.png 1068w\"],\n \"sizes\": \"(max-width: 1068px) 100vw, 1068px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/faa27055e91828f7427fbafa299beca3/adc5a/trueRelationshipScreenshot.png\",\n \"alt\": \"True relationship between parameters\",\n \"title\": \"True relationship between parameters\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `If the dataset contains no noise at all, it means that the dataset\nrepresents the `, mdx(\"em\", {\n parentName: \"p\"\n }, `exact`), ` underlying relationship. But almost every dataset out there will have some noise in it. This might be because\nthe dataset was not properly cleaned, or maybe because there is some intrinsic noise to the underlying problem itself. Often times it is\nsimply not possible to get perfect, noiseless data. For this reason, there might be an `, mdx(\"em\", {\n parentName: \"p\"\n }, `irreducible error`), `, an error, that\nno machine learning model (or even human) can undergo.\nSo if the definition of bias from statistics compares the predicted relationship with the `, mdx(\"strong\", {\n parentName: \"p\"\n }, `true`), `, underlying relationship,\nwhy don’t we do the same as well? Because oftentimes, we simply can’t.`), mdx(\"div\", null, mdx(\"p\", null, `I can show you how this exemplary\ndataset looks like without the noise because I generated the dataset using a mathematical function. However, real datasets\nare often sampled from actual events, which may or may not follow the shape of some mathematical function. So most of the time,\nit’s exactly the opposite of the scenario presented here.\nWe usually have a seemingly random dataset and we want to create a mathematical function that models a certain relationship in that dataset. Here, I’ve done the opposite.\nI’ve created a function and then generated a random dataset using that function. If this dataset was truly random, we would have\nno way of knowing how much noise it contains or what part of the dataset really is noise.`, mdx(ToolTip, {\n width: \"15em\",\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, `In some cases, if you have `, mdx(\"em\", {\n parentName: \"p\"\n }, `huge`), ` amounts of data, you can try and estimate the noisiness of the data. For the example in this post, we will not look at those methods and simply assume that we can’t separate the noise from the actual data, which is true in most applications.`))), mdx(\"p\", null, `This is the reason why we compare our dataset to our predictions. Ideally, we want our function to learn this true, underlying relationship,\nby only looking at the noisy dataset. Almost always we won’t be able to create a model that perfectly matches this relationship (and we also don’t have a way of checking if it did),\nbut oftentimes we can generate a function that is `, mdx(\"em\", {\n parentName: \"p\"\n }, `good enough`), `, assuming the dataset itself is `, mdx(\"em\", {\n parentName: \"p\"\n }, `good enough`), `. So what does this mean?\nIf our model creates a relationship that we think is not as sensible, 4/5 times it will be the dataset that is to blame, and not the model.`), mdx(\"h3\", {\n \"id\": \"but-how-do-we-tell-if-our-model-is-good\"\n }, `But how `, mdx(\"em\", {\n parentName: \"h3\"\n }, `do`), ` we tell if our model is good?`), mdx(\"p\", null, `Naturally, we are interested in keeping the bias as low as possible, because we want our model\nto make the best predictions possible. But if you take a look at the second and third model,\nwould you really choose the third model over the second one? It does have a lower bias, so this\nis certainly an argument. But there is just something `, mdx(\"em\", {\n parentName: \"p\"\n }, `off`), ` about the third model. Maybe it’s the fact that it looks a lot less\n“stable” than the second one. Maybe it’s the fact that it just seems a little too good at its task. But let’s not get ahead of ourselves.\nThere has to be a way to tell whether the third model really is not so great, or whether it’s truly the best of the bunch.\nAnd to do this, we are now going to take a look at the notion of `, mdx(\"strong\", {\n parentName: \"p\"\n }, `variance`), `.`), mdx(\"h2\", {\n \"id\": \"the-variance\"\n }, `The Variance`), mdx(\"div\", null, mdx(\"p\", null, `With the bias, we now have a way of expressing how well a machine learning model can represent the relationship\nbetween feature and target values. Now, with the bias, we always only consider one dataset,\nbut what would happen if we bring in values our model has not seen before?\nWhat happens when we bring in another dataset? Obviously, we won’t consider a dataset of\napples and oranges if our model was trained to predict exam scores, but what if we bring\nin a dataset that is very `, mdx(\"em\", {\n parentName: \"p\"\n }, `similar`), mdx(ToolTip, {\n width: \"25em\",\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, `The word “similar” conveys the meaning of how our dataset should look like pretty well, but it is a bit vague. In practice, if the dataset is generated by us, we would use the same distribution to generate the datasets. If the data is collected and not generated,\nwe would use the same data aggregation process for all datasets.`), ` to our current one? This could be the data of\na different exam and different students from the same school. Alternatively, we can split our\nexisting dataset into a `, mdx(\"em\", {\n parentName: \"p\"\n }, `training`), `-set and a `, mdx(\"em\", {\n parentName: \"p\"\n }, `testing`), `-set. We won’t go into detail about why this makes sense\nin this particular article, but if you want to read more about this, I’d recommend you take a look at the\narticle `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"/dataset_optimization/train_and_test\"\n }, `Training and Testing Datasets Explained, Step by Step`), `.`)), mdx(\"p\", null, `We want the variance to express how `, mdx(\"em\", {\n parentName: \"p\"\n }, `consistent`), ` a certain machine learning model is in its predictions when compared across `, mdx(\"em\", {\n parentName: \"p\"\n }, `similar`), ` datasets.\nSo let’s apply the same approach that we used for the bias and try to first come up with a formal definition of variance ourselves.\nIt might look like the definition below.`), mdx(\"blockquote\", null, mdx(\"p\", {\n parentName: \"blockquote\"\n }, `The variance of a specific machine learning model trained on a specific dataset describes how much the\nperformance of the machine learning model differs when evaluated on different datasets of the same origin.\nSo for our example, the variance of any one model would tell\nus how `, mdx(\"em\", {\n parentName: \"p\"\n }, `robust`), ` the performance of this particular model is, meaning if we were to slightly change the values of our data points,\nhow much would our performance fluctuate?`)), mdx(\"p\", null, `Ok, now let’s take a look at the same three polynomial regression models that we analyzed previously.`), mdx(\"h3\", {\n \"id\": \"practical-examples-variance\"\n }, `Practical Examples (Variance)`), mdx(\"div\", null, mdx(\"p\", null, `This time, I’ll add a function to each plot that slightly alters the dataset when clicked.\nLet’s also display some additional information.\nFirst, we will display the RMSE for that particular model and the (potentially altered) dataset.\nSecond, we will also track the relative difference (R.DIFF) between the current MSE\nand the initial MSE.`, mdx(ToolTip, {\n width: \"25em\",\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, `Note that we could have also looked at the rolling average of the relative difference, meaning taking the sum of every relative difference computed so far and divide that number by the number of relative differences computed. However, this would make it a bit more difficult to compare individual dataset variations to each other. I will provide the average relative differences in a following paragraph.`), `\nLet’s first look at the model with degree=1:`)), mdx(ConditionalLazyPlotInteractive, _extends({}, polynomialPredUnderFitInteractive, {\n mdxType: \"ConditionalLazyPlotInteractive\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1059px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"63%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABy0lEQVQ4y4VT2XLbMAxUY1vWZVGSddDWQV2OJR91mpk208lDXpr+/x9tB5CcsTN2+gCRhIAlsFhqM1/hbKE6YFkceI3KI/umXnG1ft5fGvk1+pwD5svqym4BXV4wERnbwyL98GtGWGMhH2EnDWZjwmXivUos2cEtThDqGVH3CiNpMRU5tGWxHwDjBnqg7oKcTQ9KPDhrCPWE7PkvVk9vyF7+QA/pIgXNz3oGtKIKUy//Emzi5tyek/VIvr/CUVvoccl5Hy37+Q5OsoGb9rDkltu+AvJG8wuI5oT85R2iPkIPy4Eacc2xFuQ72EkLd91xG/aqHzkcgCjRkBVXE+1/w8n2zNUtjhnQXW9hJxuYYQlbdtCDmhPmSQUzbWBlLYy4gpsd4dc/8c2WzONd2ZDmFivisIYpH7Hsf8Frj5jLCrNAwUo6pKd3JPs3nuz/pq/RQEySTraBldZwywPmYT1ISOQQ6gfs9Y45HLQ5AOrMbzGs456HMhlJdWQLgxJEzhIinxU3fBmBEM+0cidhzRJz5IYBKc6KGkyowltPiSohn0nJUc1nO26ZOytu2Uf/jXCII/85R7t8Sp/3Iu34TXtpD1IDSYzO5L8VTy3/A2E3Q2ui/+3oAAAAAElFTkSuQmCC')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/d70b4c6415c7495d9c39c57f0cd7bbaa/601b1/underFitInteractiveScreenshot.webp 500w\", \"/static/d70b4c6415c7495d9c39c57f0cd7bbaa/80c60/underFitInteractiveScreenshot.webp 1000w\", \"/static/d70b4c6415c7495d9c39c57f0cd7bbaa/cb660/underFitInteractiveScreenshot.webp 1059w\"],\n \"sizes\": \"(max-width: 1059px) 100vw, 1059px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/d70b4c6415c7495d9c39c57f0cd7bbaa/0eb09/underFitInteractiveScreenshot.png 500w\", \"/static/d70b4c6415c7495d9c39c57f0cd7bbaa/1263b/underFitInteractiveScreenshot.png 1000w\", \"/static/d70b4c6415c7495d9c39c57f0cd7bbaa/604b4/underFitInteractiveScreenshot.png 1059w\"],\n \"sizes\": \"(max-width: 1059px) 100vw, 1059px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/d70b4c6415c7495d9c39c57f0cd7bbaa/604b4/underFitInteractiveScreenshot.png\",\n \"alt\": \"Underfitted Model\",\n \"title\": \"Underfitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `This model predicts our y very poorly. This poor performance is also maintained\nif we slightly alter the dataset. If you press the button a couple of times and keep an eye\non the RMSE and the R.DIFF, you’ll notice that the values don’t change that much. To be more exact,\nif you compute the average relative difference of 1000 RMSE values for slightly altered datasets,\nyou will receive a value of ~1.7. This means that, if we slightly change our dataset, our MSE\nwill fluctuate by around 1.7%, which makes the model very consistent.\nIn other terms, our model performs poorly, and does so `, mdx(\"em\", {\n parentName: \"p\"\n }, `consistently`), `. It always makes bad predictions for a dataset of this origin.\nBecause the performance of this model is very consistent across multiple similar datasets,\nwe can say that it has a very low variance.`), mdx(\"p\", null, `With this we can capture the following behavior:`), mdx(\"ul\", null, mdx(\"li\", {\n parentName: \"ul\"\n }, `small fluctuation of the error -> small variance`)), mdx(\"p\", null, `In practice, we usually don’t want to randomly alter our dataset and keep track of the relative difference to find out\nwhether our variance is low or high. So how can we estimate the variance of a machine learning model? We look\nat the difference in performance between the training-set and the testing-set (or for that matter, the training set and\nany other dataset of similar origin). If this difference is high, so is the variance. If it is low, so is the variance.`), mdx(\"p\", null, `Because the model with degree=1 has a high bias but a low variance, we say that it is `, mdx(\"em\", {\n parentName: \"p\"\n }, `underfitting`), `, meaning it is not “fit enough”\nto accurately model the relationship between features and targets.`), mdx(\"p\", null, `Let’s now look at the model with degree 4:`), mdx(ConditionalLazyPlotInteractive, _extends({}, polynomialPredFitInteractive, {\n mdxType: \"ConditionalLazyPlotInteractive\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1077px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"65.4%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABxUlEQVQ4y41T2XabMBR0bMxqhMxqDMKAFZaQpG580rfmpH3v/3/O9OhSbHqSJn0YBFoG3Zm5i/X2gAlWdIStEEvY8REaLzBf/wjT3sX0ocCzDtu8hy8GbPO7y/yH8AQ0Li6kC/UwggpWVGPpCaw8gSUT9P7vG5UwgppGM5IwQnm9oZs2YPsWm1j+d3krJpA8vGB3+o765ReC7hlLNxsJVYle1mGzk59qtmI5HWTlgP3TK+xdC1c8wojKa8nevoO7a8DLFkZUkSbvEvICTjrAb7/CO36B7tf0A4U1n5mijGBpA0/08Lsn2EmH9bZ446AZ3SI7/4DfnLHcZBct1375t8tKP5a2sIMKrLqHk7XjgT8bqVQ3Q9Cf4R4G3Dgp9Nnam9iQhvsOdlTDihv47QmbvCfhVTmuuEf+/BPp6RU6OftJDpXLTiyxSSQ0loOXJ7B6gJlWhGj4BlfcUTz0oCQ59BmR/l6wFZxEUh7VASuoyCAnlTBDSYfMcFxTHaT7B1hhTVDlOxS5MeiLOfsERax5BexIEpEZ1dSOat5JbolEwfDL0bCwvsZmrsFEPI1ctWExgGc9taLSOywfqN9X1HLF5WYaH3l+A8UyQp+N3T80AAAAAElFTkSuQmCC')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/9de1fe25526656d28abae6a258fdd5ca/601b1/wellFitInteractiveScreenshot.webp 500w\", \"/static/9de1fe25526656d28abae6a258fdd5ca/80c60/wellFitInteractiveScreenshot.webp 1000w\", \"/static/9de1fe25526656d28abae6a258fdd5ca/8612f/wellFitInteractiveScreenshot.webp 1077w\"],\n \"sizes\": \"(max-width: 1077px) 100vw, 1077px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/9de1fe25526656d28abae6a258fdd5ca/0eb09/wellFitInteractiveScreenshot.png 500w\", \"/static/9de1fe25526656d28abae6a258fdd5ca/1263b/wellFitInteractiveScreenshot.png 1000w\", \"/static/9de1fe25526656d28abae6a258fdd5ca/8b081/wellFitInteractiveScreenshot.png 1077w\"],\n \"sizes\": \"(max-width: 1077px) 100vw, 1077px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/9de1fe25526656d28abae6a258fdd5ca/8b081/wellFitInteractiveScreenshot.png\",\n \"alt\": \"Well fitted Model\",\n \"title\": \"Well fitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `This model predicts our y a lot better than the first one. This good performance is also\nsomewhat stable if we alter the dataset. If you again press the button a couple of times and keep an eye\non the RMSE and the R.DIFF, you’ll notice that the values change a little bit, but not a lot.\nTo be more exact, if you compute the average relative difference of 1000 RMSE values for slightly altered datasets,\nyou will receive a value of ~13.6. This means that if we slightly change our dataset, our MSE\nwill fluctuate by around 13.6%, which makes the model `, mdx(\"em\", {\n parentName: \"p\"\n }, `somewhat`), ` consistent. It’s certainly not\nas consistent as the first model, but it’s not terrible as well.\nIn other terms, our model performs well, and does so pretty consistently. It always makes at least decent predictions for a dataset of this origin.\nBecause the performance of this model is somewhat consistent across multiple similar datasets,\nwe can say that it has a somewhat low variance.`), mdx(\"p\", null, `With this we can capture the following behavior:`), mdx(\"ul\", null, mdx(\"li\", {\n parentName: \"ul\"\n }, `medium fluctuation of the error -> medium variance`)), mdx(\"p\", null, `Because the model with degree=4 has a low bias and a low variance, we say that it is `, mdx(\"em\", {\n parentName: \"p\"\n }, `well fit`), `, meaning it has just the right balance between bias and variance.`), mdx(\"p\", null, `Let’s now look at the model with degree 15:`), mdx(ConditionalLazyPlotInteractive, _extends({}, polynomialPredOverFitInteractive, {\n mdxType: \"ConditionalLazyPlotInteractive\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1068px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"64.99999999999999%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB8klEQVQ4y3VT2W7bMBA0bMu6b9k6SEkWJdGy3cA5GgN5aFCgBfrW//+bKbjylcR9WJBcDAezw+FEC2uomgcVrGULO+1hrTo4mbz0z5hb7P96E7VRNfNLePkGYblDwAaEfIdFtL4Az7jruRzveZzqjJmojR41sNMWi+UaU5cT+dTjd5VQBRWMZYdF1MDKt7Dy4arQTjtS4zEJm3eY+xUWUX2fKKxJ9dQpkAxH8ONviJ9/UXx/x9RlI6GbSwR8ByfvYBWCCO+SRTXmSrlTwOZb5M/vMFMJM93AZC20+DSyUqi8c5mEW20w88rTWJ9M90s4bIvltzeE/QF6Ii4emrnAIhn9nriZhF8M8PiAZPcCh+1gMgEjv6pVQD0WyJ5+IN68Yh6c/FWqgxpm0V4JnbQnhVYqEPR7+P0eJmtg8e6kcvTMFweE/SMmZgotWt8orz4SkodsSx669Qar5yN88QAjbygaNtuievsD/voLZia/vPwXQms1htnOWtilQDgcYPMBZiGghRVC+Qi32kNfChpb+xwlGvnGwzGsNRSxW0oiMZMGRtpQz2GSSOxVS6vqGXEDPVbZ7ajncQk9aa7BPpPSXgU9bmgUIxEwYjFeXnWUQbWqs3bG0YM1H3/K7de63St/k/UDeazCr76lOjtZj9kdvAr2PwJ/REWx3g3dAAAAAElFTkSuQmCC')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/9a47b512ef6ae2e4400d12f1a1a09da4/601b1/overFitInteractiveScreenshot.webp 500w\", \"/static/9a47b512ef6ae2e4400d12f1a1a09da4/80c60/overFitInteractiveScreenshot.webp 1000w\", \"/static/9a47b512ef6ae2e4400d12f1a1a09da4/a22c7/overFitInteractiveScreenshot.webp 1068w\"],\n \"sizes\": \"(max-width: 1068px) 100vw, 1068px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/9a47b512ef6ae2e4400d12f1a1a09da4/0eb09/overFitInteractiveScreenshot.png 500w\", \"/static/9a47b512ef6ae2e4400d12f1a1a09da4/1263b/overFitInteractiveScreenshot.png 1000w\", \"/static/9a47b512ef6ae2e4400d12f1a1a09da4/adc5a/overFitInteractiveScreenshot.png 1068w\"],\n \"sizes\": \"(max-width: 1068px) 100vw, 1068px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/9a47b512ef6ae2e4400d12f1a1a09da4/adc5a/overFitInteractiveScreenshot.png\",\n \"alt\": \"overfitted Model\",\n \"title\": \"overfitted Model\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"p\", null, `This model predicts our data very well. However, this outstanding performance is not at all maintained\nwhen we change up our dataset a bit. If you press the button just a few times and look at\nthe RMSE and R.DIFF, chances are you’ll see some larger numbers than before. These numbers also change quite a lot between clicks.\nTo be more exact, if you compute the average relative difference of 1000 RMSE values for slightly altered datasets,\nyou will receive a value of ~169.5. 169.5! This means that if we slightly change our dataset, our RMSE\nwill fluctuate by around `, mdx(\"strong\", {\n parentName: \"p\"\n }, `169.5%`), `, which makes the predictions of this model as reliable as your office or home printer in the moment you need it most.\nThis means that our model performs well sometimes, and catastrophically other times.\nBecause the performance of this model is very inconsistent across multiple datasets,\nwe can say that it has a high variance.`), mdx(\"p\", null, `With this we can capture the following behavior:`), mdx(\"ul\", null, mdx(\"li\", {\n parentName: \"ul\"\n }, `high fluctuation of the error -> high variance`)), mdx(\"p\", null, `Because this model has a low bias but a high variance, we say that it is `, mdx(\"em\", {\n parentName: \"p\"\n }, `overfitting`), `, meaning it is “too fit” at predicting\nthis very exact dataset, so much so that it fails to model a relationship that is transferable to a similar dataset.`), mdx(\"h3\", {\n \"id\": \"revisiting-the-definition-of-variance\"\n }, `Revisiting the Definition of Variance`), mdx(\"p\", null, `Ok, so we have defined the variance, looked at examples, and we know that the error fluctuation is correlated with the variance.\nThe lower the error fluctuation, the lower the variance. The higher the error fluctuation, the higher the variance.`), mdx(\"p\", null, `When talking about the variance of a particular model, we always talk about one model, but `, mdx(\"em\", {\n parentName: \"p\"\n }, `multiple`), ` datasets.\nIn practice, you would compare the model error on the training dataset and the error on the testing (or validation) dataset.`), mdx(\"p\", null, `Let’s now also look at the definition used in statistics. In the words of `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"https://en.wikipedia.org/wiki/Variance\",\n \"target\": \"_blank\",\n \"rel\": \"nofollow noopener noreferrer\"\n }, `Wikipedia`), `:`), mdx(\"blockquote\", null, mdx(\"p\", {\n parentName: \"blockquote\"\n }, `In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its mean.\nIn other words, it measures how far a set of numbers is spread out from their average value.`)), mdx(\"p\", null, `The important part is ”`, mdx(\"em\", {\n parentName: \"p\"\n }, `spread out from their average value`), `”. This means that the variance is a way of describing the difference between the expected (or average) value\nand the predicted value of our model. This definition is similar to the one we described earlier.`), mdx(\"p\", null, `Naturally, we are interested in keeping the variance as low as possible, because we want our model\nto make consistent predictions across all datasets. If we again take a look at the second (degree=4) and third (degree=15) model,\nwe can see that the third model may have a lower bias than the second one, but it also has a lot higher\nvariance when compared to the second model. Therefor, the second model is the overall best model out of the three,\nbecause it succeeds in minimizing both bias and variance at the same time, while the first model (degree=1) minimizes just the variance,\nand the third model minimizes just the bias.`), mdx(\"h2\", {\n \"id\": \"the-tradeoff\"\n }, `The Tradeoff`), mdx(\"p\", null, `Let’s think about how we might decrease bias.\nThe easiest way to achieve a lower bias would be to pick a more complex model\nor to train our existing model for longer. In other words, we want to extract `, mdx(\"em\", {\n parentName: \"p\"\n }, `more insights`), ` from\nour dataset, we want our model to `, mdx(\"em\", {\n parentName: \"p\"\n }, `learn more`), ` from our data.\nThe more our model learns from our data, the better it will be at solving its task.`), mdx(\"p\", null, `And how might we decrease variance?\nThe easiest way to decrease variance would be to pick a more simple model\nor to train our existing model for a shorter amount of time. In other words, we want to extract `, mdx(\"em\", {\n parentName: \"p\"\n }, `fewer insights`), ` from\nour dataset, we want our model to `, mdx(\"em\", {\n parentName: \"p\"\n }, `learn less`), ` from our data.\nThe less our model learns from our data, the more general it will be, because\nit has only learned so much to base its predictions on.`), mdx(\"p\", null, `A good analogy would be to think of a student preparing for their mathematics and physics exams.\nIf the student does not prepare at all, they will do poorly in both exams (high bias).\nIf they put a lot of time into their preparation, they will probably do pretty well in both exams (low bias, low variance).\nHowever, if the student puts `, mdx(\"em\", {\n parentName: \"p\"\n }, `a lot`), ` of time into their preparation, and does the same\nexercises over and over and over again (prolonged training on the same dataset),\nthey will train themselves not to be good at solving\nmath and physics exams, but to be good at solving those particular exercises they have solved over and over again.\nIf they see a question on the exam that they have practiced, they will solve it without any error.\nHowever, if they see a question that they have not practiced, they will have a more difficult time\nanswering it (low bias, high variance).`), mdx(\"p\", null, `So there seems to be a sort of “tradeoff” between bias and variance. You can only decrease\nyour bias so much until your variance starts to increase.\nAt this point, you might think of a graph like this one:`), mdx(ConditionalLazyPlot, _extends({}, biasVarianceTradeoff, {\n mdxType: \"ConditionalLazyPlot\"\n }), mdx(\"p\", null, mdx(\"span\", {\n parentName: \"p\",\n \"className\": \"gatsby-resp-image-wrapper\",\n \"style\": {\n \"position\": \"relative\",\n \"display\": \"block\",\n \"marginLeft\": \"auto\",\n \"marginRight\": \"auto\",\n \"maxWidth\": \"1067px\"\n }\n }, `\n `, mdx(\"span\", {\n parentName: \"span\",\n \"className\": \"gatsby-resp-image-background-image\",\n \"style\": {\n \"paddingBottom\": \"64.2%\",\n \"position\": \"relative\",\n \"bottom\": \"0\",\n \"left\": \"0\",\n \"backgroundImage\": \"url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAABdElEQVQ4y5VT25KCMAxlVGhhRwtF7ncoiq5P++Ds///Y2Wm4qDu64z6cIaHpSXKSGqZXYeOW4H4DkQ7Yxge42QAnVPR/Pn8Jr6KYOc6YDSZr7NIjwSvOcMJuIfwTbom1KLDa5eQbdElWBH2wngLeItPF6EKyE/zyEytRwNiIEixswOMW2n6HZK7Mkg1YqCCqE9rLN3W2VGin7ftkk0yW38KUDUyvBPMbwqihKGFnHSy/fptQy7IrLnCiwyLRbSi67ej/bTvxAEvWD3ovU6aAQk36VLDkc5KRoAALemr5frWmKU+aTMNx8o4E1/7chuk+2pbfge0V2Tr5xhu/t7XR2UncGjzq4BQ9nLwnm4fthA48VrCTI3jcw9rXYEFNA31oeXbYvgML1Jg9UHSJpQo8VWBJC572ZNtpB5404ElLm6GJn2p4Dz01/RSr8xVZ/0WohivcdMBqm49yiJssvzQsn++YrOHmJ4hsoLctizM+IoVX8XOFPyfGPeni3FomAAAAAElFTkSuQmCC')\",\n \"backgroundSize\": \"cover\",\n \"display\": \"block\"\n }\n }), `\n `, mdx(\"picture\", {\n parentName: \"span\"\n }, `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/8f030b78e25916b2fca5182f281a7ed2/601b1/theTradeoffScreenshot.webp 500w\", \"/static/8f030b78e25916b2fca5182f281a7ed2/80c60/theTradeoffScreenshot.webp 1000w\", \"/static/8f030b78e25916b2fca5182f281a7ed2/419f7/theTradeoffScreenshot.webp 1067w\"],\n \"sizes\": \"(max-width: 1067px) 100vw, 1067px\",\n \"type\": \"image/webp\"\n }), `\n `, mdx(\"source\", {\n parentName: \"picture\",\n \"srcSet\": [\"/static/8f030b78e25916b2fca5182f281a7ed2/0eb09/theTradeoffScreenshot.png 500w\", \"/static/8f030b78e25916b2fca5182f281a7ed2/1263b/theTradeoffScreenshot.png 1000w\", \"/static/8f030b78e25916b2fca5182f281a7ed2/40547/theTradeoffScreenshot.png 1067w\"],\n \"sizes\": \"(max-width: 1067px) 100vw, 1067px\",\n \"type\": \"image/png\"\n }), `\n `, mdx(\"img\", {\n parentName: \"picture\",\n \"className\": \"gatsby-resp-image-image\",\n \"src\": \"/static/8f030b78e25916b2fca5182f281a7ed2/40547/theTradeoffScreenshot.png\",\n \"alt\": \"A graph depicting how you might imagine the tradeoff looks visually\",\n \"title\": \"A graph depicting how you might imagine the tradeoff looks visually\",\n \"loading\": \"lazy\",\n \"decoding\": \"async\",\n \"style\": {\n \"width\": \"100%\",\n \"height\": \"100%\",\n \"margin\": \"0\",\n \"verticalAlign\": \"middle\",\n \"position\": \"absolute\",\n \"top\": \"0\",\n \"left\": \"0\"\n }\n }), `\n `), `\n `))), mdx(\"h3\", {\n \"id\": \"the-tradeoff-1\"\n }, `The Tradeoff?`), mdx(\"p\", null, `If you have a rather simple model like polynomial regression, there certainly is a tradeoff between\nbias and variance. As you saw, increasing the degree up to a specific point decreased bias considerably, while only slightly affecting variance.\nAfter that specific point, the variance just started to increase dramatically, rendering the further decrease in bias meaningless.`), mdx(\"p\", null, `However, if you have a rather complicated model such as a deep neural network,\nthings get a little bit more complicated. Oftentimes it seems as if the bias-variance tradeoff is not really such “tradeoff” in these cases anymore.\nI don’t want to get into too much detail here because this topic is extensive enough to fill a separate article. However, I do want to\npoint you to some external resources if you are interested in reading more about this.\nThere is `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"https://www.bradyneal.com/bias-variance-tradeoff-textbooks-update\",\n \"target\": \"_blank\",\n \"rel\": \"nofollow noopener noreferrer\"\n }, `this blog article`), ` by Brady Neal which explains the topic in an easy-to-follow fashion. If you are looking for a more\nrigorous source, consider reading the paper `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"https://arxiv.org/abs/1810.08591\",\n \"target\": \"_blank\",\n \"rel\": \"nofollow noopener noreferrer\"\n }, `A Modern Take on the Bias-Variance Tradeoff in Neural Networks`), `,\nwhose first author is the same person who wrote the aforementioned blog post. The paper covers the topic a bit more in-depth.`), mdx(\"h2\", {\n \"id\": \"an-all-purpose-remedy\"\n }, `An All-Purpose Remedy?`), mdx(\"div\", null, mdx(\"p\", null, `No matter what machine learning model you are training and which problem you are trying to solve,\nthere is one thing you can do that will almost always result in a better, more stable machine learning model.\nWhat is it? Simply said, “getting more (and better) data”. There have been numerous papers written and studies performed`, mdx(ToolTip, {\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, ` Two of the most popular titles are the `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"https://en.wikipedia.org/wiki/No_free_lunch_theorem\",\n \"target\": \"_blank\",\n \"rel\": \"nofollow noopener noreferrer\"\n }, `No Free Lunch Theorem`), ` and `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"https://static.googleusercontent.com/media/research.google.com/de//pubs/archive/35179.pdf\",\n \"target\": \"_blank\",\n \"rel\": \"nofollow noopener noreferrer\"\n }, `The Unreasonable Effectiveness of Data`), `.`), `\nabout whether increasing data capacity (and quality) yields a greater positive effect on model performance than trying to come\nup with a more elaborate machine learning algorithm. While it’s impossible to generalize the results to each and every application, the general notion seems to be that increasing data capacity yields greater results than choosing a more complicated model.`)), mdx(\"p\", null, `So what does this mean for you? If you are unsatisfied with the performance of your machine learning model,\nand you have already tried out a variety of parameter configurations and alternative models,\nthen your 1# chance to increase model performance is to get more and better data.\nWhen your machine learning model is underperforming, the underlying issue is frequently\npresent in the dataset itself, and not in the model.`), mdx(\"h2\", {\n \"id\": \"diving-deeper\"\n }, `Diving Deeper`), mdx(\"p\", null, `In this article, I briefly talked about the concept of an `, mdx(\"em\", {\n parentName: \"p\"\n }, `irreducible error`), `, an error, that\nis present in the problem/data itself, and that no machine learning model (or even human) can undergo.\nThis is the reason why a training error of 0.15 might be amazing in one application, and horrible in another.\nTherefor it makes sense to remember this concept when you are analyzing the bias of a machine learning model. If your models\nhave been stuck at a certain bias and don’t improve any further no matter what you do,\nit may just be the case that you’ve reached the ideal bias for this particular\nproblem and that it is simply not possible to improve any further with this particular dataset that you have.`), mdx(\"div\", null, mdx(\"p\", null, `Oftentimes you won’t know the irreducible error for a problem. It is pretty difficult to compute\nand it’s even more difficult to prove that it really is irreducible. In practice,\nyou can’t compute the irreducible error for every problem you try to solve with machine learning.\nWhat you can do instead, is to train multiple different machine learning models`, mdx(ToolTip, {\n width: \"20em\",\n trigger: \"*\",\n mdxType: \"ToolTip\"\n }, `Here it is important to mention that you should use models that function differently. This means that you should not use two linear regression models but f.e. one linear regression and one decision tree model.`), `,\ncalculate the bias for each of those models and compare them to each other.\nIf you have multiple models that are all optimized for that particular problem\nand they all possess very similar biases, you might not get any better results by optimizing your models.`)), mdx(\"h2\", {\n \"id\": \"further-reading\"\n }, `Further Reading`), mdx(\"p\", null, `Bias and variance are two terms that are often used to describe overfitting and underfitting.\nIn this post, we saw what exactly over-and underfitting is, but how can you actually prevent it in practice?\nIf you want to know how you can stop overfitting (and underfitting), then I recommend you read the\npost `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"model_optimization/stop_overfitting_and_underfitting\"\n }, `How To Stop Overfitting and Underfitting`), `,\nwhere you will learn a variety of techniques that can help you make your model more robust against\nboth overfitting as well as underfitting.`), mdx(\"p\", null, `You can only improve your machine learning models so much until it becomes very difficult to improve\ntheir performance any further. However, oftentimes you will get even better results when you optimize\nthe dataset your machine learning models are trained on. There are lots of ways with which you can improve your data,\nand if you want to learn how exactly you can do so, you should take a look at the section `, mdx(\"a\", {\n parentName: \"p\",\n \"href\": \"/tags/dataset_optimization\"\n }, `Dataset Optimization`), `,\nwhere you will find many articles that explain exactly how you can do so.`));\n}\n;\nMDXContent.isMDXComponent = true;","excerpt":"You have likely heard about bias and variance before. They are two fundamental terms in machine learning and often used to explain overfitting and underfitting. If you're working with machine learning methods, it's crucial to understand these concepts well so that you can make optimal decisions in your own projects. In this article, you'll learn everything you need to know about bias, variance, overfitting, and the bias-variance tradeoff.","tableOfContents":{"items":[{"url":"#outline","title":"Outline"},{"url":"#the-problem","title":"The Problem"},{"url":"#the-bias","title":"The Bias","items":[{"url":"#practical-examples-bias","title":"Practical Examples (Bias)"},{"url":"#revisiting-the-definition-of-bias","title":"Revisiting the Definition of Bias"},{"url":"#but-how-do-we-tell-if-our-model-is-good","title":"But how do we tell if our model is good?"}]},{"url":"#the-variance","title":"The Variance","items":[{"url":"#practical-examples-variance","title":"Practical Examples (Variance)"},{"url":"#revisiting-the-definition-of-variance","title":"Revisiting the Definition of Variance"}]},{"url":"#the-tradeoff","title":"The Tradeoff","items":[{"url":"#the-tradeoff-1","title":"The Tradeoff?"}]},{"url":"#an-all-purpose-remedy","title":"An All-Purpose Remedy?"},{"url":"#diving-deeper","title":"Diving Deeper"},{"url":"#further-reading","title":"Further Reading"}]},"frontmatter":{"title":"Bias, Variance, and Overfitting Explained, Step by Step","userDate":"17 April 2021","date":"2021-04-17T19:54:40.006Z","dateModified":"2021-04-17T19:54:40.006Z","changelog":[["2022-03-19T19:39:23.993Z","Fixed typo, thanks to Ali Jan!"]],"tags":["Model Optimization"],"timeToRead":20,"excerpt":"You have likely heard about bias and variance before. They are two fundamental terms in machine learning and often used to explain overfitting and underfitting. If you're working with machine learning methods, it's crucial to understand these concepts well so that you can make optimal decisions in your own projects. In this article, you'll learn everything you need to know about bias, variance, overfitting, and the bias-variance tradeoff.","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACdElEQVQozxXOWVLaAAAA0HxVHGXJngAhLGELCYSyJYSQoAhFBK1FtCiLSFiDMyjYTqmg/rXTS9iTdvpO8AAuxHAhpl6tfK6fyJlknAtzQSbOs8l4LOSlaTse5zk+HGTcLi9NIaCFpuyimM4rOU3NA1HWHwn6ttvN8/PzcDicz41ep1MtHz0sFt+eVveGMZtO5/PZavk4m0wvLhq6Puj1ulImXdBUwOt2R0LMeKQP9cFkPDJmU31wd3py3O+29X5vNNQb52f3hmFMp6e16lm91r657nTakpguaHkAhBwESaeSyePSYU7KpOKx0oFaPixoOemqcX5cKhY1pajlU0LU56YwyMownmxWzCvy/zYEO6wgaQXtDBOWJelAzX0qFqrloiZLk+FgrN9l0wk1lxX4iN/nwSArFwnl84qiyKqaB6w2knYHr6+7HJ+w2Aguwmu5rCpLlVJx0L+VMunaSbVULMqSyAb9BAwG/D5NU5WcLIoZYMcENputxcNys3np3+mLh+V6/WO1Ws0N48/vXzet1uvry8/1ejqZLBaLt7e31Wq53W50feD3M8DePhIV0qun71qh1O70a/Xzx+XTl8ZlhItXK5VK+ei2171pXXW7HWM2vWxeTMaj9/e/xnyOIDDgdQeCbp/ZjJp2oQ8m244J2tkF98woiFD7FowgHJmEkBFYIexjGYoPuj7yoa/NhpwVKRID0lEhl5JI0mMFCQh2ICiFkx6McOOkB0EpixVHUCeJYk7URhNwKsYeKJIQCWSSQoz1AwjiwHEXCDtgxAkjFIK5UJyGYAeKu6w2wmzBEJSCYCcM4S4c9tgR1k8n43wqIUQjgX/XKpxML/eDfwAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/7e8675c2404d1fdaa12b78c05104d577/87706/BiasAndVariance_Thumbnail.png","srcSet":"/static/7e8675c2404d1fdaa12b78c05104d577/0dee1/BiasAndVariance_Thumbnail.png 750w,\n/static/7e8675c2404d1fdaa12b78c05104d577/8beaa/BiasAndVariance_Thumbnail.png 1080w,\n/static/7e8675c2404d1fdaa12b78c05104d577/87706/BiasAndVariance_Thumbnail.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/7e8675c2404d1fdaa12b78c05104d577/a66aa/BiasAndVariance_Thumbnail.webp 750w,\n/static/7e8675c2404d1fdaa12b78c05104d577/65dd5/BiasAndVariance_Thumbnail.webp 1080w,\n/static/7e8675c2404d1fdaa12b78c05104d577/71d4d/BiasAndVariance_Thumbnail.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Lari Giba","bio":"Hi! I'm Lari and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAbABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAMEAgH/xAAWAQEBAQAAAAAAAAAAAAAAAAABAgP/2gAMAwEAAhADEAAAAeDIop5sDKn0aCSgzv8A/8QAHhABAAICAQUAAAAAAAAAAAAAAQACAxEEEyEiMjP/2gAIAQEAAQUC3ZAE6ZGW8HtMw1M2O+tTlfI9Z//EABURAQEAAAAAAAAAAAAAAAAAABAR/9oACAEDAQE/AYU//8QAFxEBAAMAAAAAAAAAAAAAAAAAABAREv/aAAgBAgEBPwFbMf/EABwQAAIBBQEAAAAAAAAAAAAAAAABMQIQESFxkf/aAAgBAQAGPwLfpJvBhuRVIgXRNwpsxct//8QAHRABAAMAAgMBAAAAAAAAAAAAAQARIUFRMWGBcf/aAAgBAQABPyG78vaUcDUeYJrJrCUcHAhzK6pYRVinqYAdKT7Y0/Yi+8UGE//aAAwDAQACAAMAAAAQsO1C/8QAGBEBAQEBAQAAAAAAAAAAAAAAAREAECH/2gAIAQMBAT8QPF4uTBS7/8QAGBEBAQEBAQAAAAAAAAAAAAAAAQARIVH/2gAIAQIBAT8QO2vYEgOZf//EAB8QAQEAAgIBBQAAAAAAAAAAAAERACExQVFhcZGxwf/aAAgBAQABPxDlRMDhDreGljrJ8uK5DiW/WJE4rXSx7wgAOy8sEKuzsxERAF59XxirS0k0dz03g4T3iY5jH9MRu2FvejEgEAAM/9k=","aspectRatio":0.7407407407407407,"src":"/static/9eab840ebd4087faf31c48702c92af36/2f1b1/lari.jpg","srcSet":"/static/9eab840ebd4087faf31c48702c92af36/b3d27/lari.jpg 40w,\n/static/9eab840ebd4087faf31c48702c92af36/5a735/lari.jpg 80w,\n/static/9eab840ebd4087faf31c48702c92af36/6e81a/lari.jpg 120w,\n/static/9eab840ebd4087faf31c48702c92af36/2f1b1/lari.jpg 800w,\n/static/9eab840ebd4087faf31c48702c92af36/75b83/lari.jpg 1520w","sizes":"(max-width: 800px) 100vw, 800px"}}]}}],"imgCaption":"Background image by Sora Shimazaki","imgSource":"https://www.pexels.com/de-de/foto/urteilsskala-und-hammer-im-richteramt-5669602/","uniquePostId":7,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner"}},"relatedPosts":{"totalCount":1,"edges":[{"node":{"id":"70b388d6-57d2-555c-86b7-c8c3122d723c","excerpt":"You have likely heard about bias and variance before. They are two fundamental terms in machine learning and often used to explain overfitting and underfitting. If you're working with machine learning methods, it's crucial to understand these concepts well so that you can make optimal decisions in your own projects. In this article, you'll learn everything you need to know about bias, variance, overfitting, and the bias-variance tradeoff.","frontmatter":{"title":"Bias, Variance, and Overfitting Explained, Step by Step","timeToRead":20,"date":"2021-04-17T19:54:40.006Z"},"fields":{"slug":"/model_optimization/bias_and_variance/"}}}]}},"pageContext":{"slug":"/model_optimization/bias_and_variance/","prev":{"excerpt":"Gradient descent is one of the most famous techniques in machine learning and used for training all sorts of neural networks. But gradient descent can not only be used to train neural networks, but many more machine learning models. In particular, gradient descent can be used to train a linear regression model! If you are curious as to how this is possible, or if you want to approach gradient descent with smaller steps and not jump straight to neural networks, this post is for you. You will learn how gradient descent works from an intuitive, visual, and mathematical standpoint and we will apply it to an exemplary dataset in Python.","frontmatter":{"title":"Gradient Descent for Linear Regression Explained, Step by Step","tags":["Machine Learning Math"],"date":"2021-04-03T19:07:33.321Z","dateModified":"2021-04-03T19:07:33.321Z","changelog":[],"draft":false,"excerpt":"Gradient descent is one of the most famous techniques in machine learning and used for training all sorts of neural networks. But gradient descent can not only be used to train neural networks, but many more machine learning models. In particular, gradient descent can be used to train a linear regression model! If you are curious as to how this is possible, or if you want to approach gradient descent with smaller steps and not jump straight to neural networks, this post is for you. You will learn how gradient descent works from an intuitive, visual, and mathematical standpoint and we will apply it to an exemplary dataset in Python.","uniquePostId":6,"timeToRead":23,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACV0lEQVQozxWQWXOaUACFebexigiSVpMoakoSV2QRROSyCGi0cQHuBRNTTUxMmk477Uxm+tCf3vHtvJxvvnOwlg4FEMpG2LORaqOehWQLcQB1AFJsJJmoOThk2YCKCVUbySYUQdjWYRvEmGyEvIF4cOiow0hzIt2N+04s2xFvRpwOeXBAyFakWKhrHEBtHTW1sGtGmObEfQsKdtwxoKj7sr6UgK9ZoWqGvB6A0a1khA3Vr/f8S8WvdpeXit/sB7wO1WGM1Tpm313l3X3G2eecF8J5TXo/cs6evnkvuPvH3ff3v/82T2/rh5fN88/925/d8+s8emppgeqsMJxiCkyrLE2uhnc1b/vFvGXtdclYn2rR5SAAo2gS7saL9Sy4ny6/zYKNOQrN8eowx1ph2eMqTjFZmqmwgjK4MdzYdKO28vXG3262u83+l2RB3oCSefiFA7AxQPUBagMkWjFG0JUsXc3kykmi+CFT+lwWWeG6rsw7wAfDuTeJpMGiC3wJBB1teaHMa71Fq7/kNF8w4EE7X6zb7vR6Op8twvkivL3bTuFjavybtJ9x7T43esvZT5S+zhrbtH5PWw95sGaFaZUbYSnijKAYUQaSonOCajtTyx5XrsSj8z7JjUlrRw9fjnsR1Q1oNSr0I7rrUw2PFSZsx8OI0wZeURJ4MZEqJNIniVThCD/7SBRTmZM0USTz9VLDZvnJRWfEcl654ZTqFsu5vL48F6fYp+aQEcf48Xkyc5omSzjJ4BSTJpmPJJPMlpKZYpoqF85lpuWWWl657VU5r9abtQDM1+z/7PCizWjDAaIAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/97717e791c8dcde652d468d6f7784a65/87706/GradientDescentLinearRegression_Thumbnail.png","srcSet":"/static/97717e791c8dcde652d468d6f7784a65/0dee1/GradientDescentLinearRegression_Thumbnail.png 750w,\n/static/97717e791c8dcde652d468d6f7784a65/8beaa/GradientDescentLinearRegression_Thumbnail.png 1080w,\n/static/97717e791c8dcde652d468d6f7784a65/87706/GradientDescentLinearRegression_Thumbnail.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/97717e791c8dcde652d468d6f7784a65/a66aa/GradientDescentLinearRegression_Thumbnail.webp 750w,\n/static/97717e791c8dcde652d468d6f7784a65/65dd5/GradientDescentLinearRegression_Thumbnail.webp 1080w,\n/static/97717e791c8dcde652d468d6f7784a65/71d4d/GradientDescentLinearRegression_Thumbnail.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Lari Giba","bio":"Hi! I'm Lari and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"aspectRatio":0.7434944237918215,"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAbABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAMEAgH/xAAWAQEBAQAAAAAAAAAAAAAAAAABAgP/2gAMAwEAAhADEAAAAeDIop5sDKn0aCSgzv8A/8QAHhABAAICAQUAAAAAAAAAAAAAAQACAxEEEyEiMjP/2gAIAQEAAQUC3ZAE6ZGW8HtMw1M2O+tTlfI9Z//EABURAQEAAAAAAAAAAAAAAAAAABAR/9oACAEDAQE/AYU//8QAFxEBAAMAAAAAAAAAAAAAAAAAABAREv/aAAgBAgEBPwFbMf/EABwQAAIBBQEAAAAAAAAAAAAAAAABMQIQESFxkf/aAAgBAQAGPwLfpJvBhuRVIgXRNwpsxct//8QAHRABAAMAAgMBAAAAAAAAAAAAAQARIUFRMWGBcf/aAAgBAQABPyG78vaUcDUeYJrJrCUcHAhzK6pYRVinqYAdKT7Y0/Yi+8UGE//aAAwDAQACAAMAAAAQsO1C/8QAGBEBAQEBAQAAAAAAAAAAAAAAAREAECH/2gAIAQMBAT8QPF4uTBS7/8QAGBEBAQEBAQAAAAAAAAAAAAAAAQARIVH/2gAIAQIBAT8QO2vYEgOZf//EAB8QAQEAAgIBBQAAAAAAAAAAAAERACExQVFhcZGxwf/aAAgBAQABPxDlRMDhDreGljrJ8uK5DiW/WJE4rXSx7wgAOy8sEKuzsxERAF59XxirS0k0dz03g4T3iY5jH9MRu2FvejEgEAAM/9k=","sizes":"(max-width: 800px) 100vw, 800px","src":"/static/9eab840ebd4087faf31c48702c92af36/2f1b1/lari.jpg","srcSet":"/static/9eab840ebd4087faf31c48702c92af36/fd013/lari.jpg 200w,\n/static/9eab840ebd4087faf31c48702c92af36/25252/lari.jpg 400w,\n/static/9eab840ebd4087faf31c48702c92af36/2f1b1/lari.jpg 800w,\n/static/9eab840ebd4087faf31c48702c92af36/0ff54/lari.jpg 1200w,\n/static/9eab840ebd4087faf31c48702c92af36/75b83/lari.jpg 1520w"}}]}}]},"fields":{"layout":"post","slug":"/machine_learning_math/gradient_descent_for_linear_regression/"}},"next":{"excerpt":"If you are just starting out in machine learning and building your first real models, you will have to split your dataset into a train set as well as a test set. But what benefits does this splitting yield? How can you split your dataset optimally? In this article, we will go through these questions and explore why splitting your dataset makes sense and how you can split your dataset properly.","frontmatter":{"title":"How to Split Your Dataset the Right Way","tags":["Dataset Optimization"],"date":"2021-05-01T18:09:26.968Z","dateModified":"2021-05-01T18:09:26.968Z","changelog":[],"draft":false,"excerpt":"If you are just starting out in machine learning and building your first real models, you will have to split your dataset into a train set as well as a test set. But what benefits does this splitting yield? How can you split your dataset optimally? In this article, we will go through these questions and explore why splitting your dataset makes sense and how you can split your dataset properly.","uniquePostId":8,"timeToRead":15,"tutorialId":null,"tutorialEntryIdx":null,"dependencies":[],"proficiencyLevel":"Beginner","image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsTAAALEwEAmpwYAAACcUlEQVQozxXPaW/SYAAA4DfA1pVCOVooPaDH27ctZZwFBgwm24QKHedidAM3sssNk0EkXlGTJWYxUWf85Ff/mb/F+PyDB7gk3S0bHsUEAsSNfDRj6NUCY7VyzfLZ+cFqXqnbla29DJlpl/fSWtkEMRlDOirKSkEFLslwK0kgqFZ3qO2P0hX286pvT7/dv+v++rTz9w+4e3v0/aNTefpbq5a2HLva73tEKZZGAR0BTFABIxadUef0AvAQgzqupoCUI1QNhxqV3mRzWsBQ1yEKGEjIiJObWaU/AELCizSwEeSImHJ8s/DpGcBpW/X45dDfb4WWh66BHSHNPpMdRzOHlGEz2Qadko1q8mSxEHK6nIfA42cSWrZ3OAEkQyYLQZTYrxPTsWQ/lpVUBBZrXGZfstpa8dGGlHcJEuDE8cVZubUFOAWs+WOsnCo32jSn/Hj4+eLy2pmeP79e3n19uFis5svb1YfXw+nx3f2Xm+XtydVVoWHsjAdiycJkCPCwgIeF2m6XjCLJKChJy8hWAnLKbDyazeetXhPVdphcsdHrJJsdy3bYfO7Js4OAoRL6/zPv9nMJVTvqpXud7Z5tOe3SZFzjN3cBzXsRXBelkAGlLDceQGfUqDs9rrjthTKQdJCQxE2dD9HUpG/OjjtvXo3mZwdX02bS6nrDfESSYQ7qJYgp8OUs/341PZ0NKTMJEqpb0kDNUnYrJsWKAGc9BBOJm0REARtRjGQISiBp3h/hfXEJk5EnoQdMizR0P1LdIiIQAsEIG4ryWFDwUQJBxbEgi4d5Hx33hgQ8xHspgWZ4X5in+bhXUYGgrEEdiHpAV8vbiX/nc3rHlqIDkgAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/e9b8f309bba40fd754df6982c7c7159d/87706/SplitDataTheRightWay_Thumbnail.png","srcSet":"/static/e9b8f309bba40fd754df6982c7c7159d/0dee1/SplitDataTheRightWay_Thumbnail.png 750w,\n/static/e9b8f309bba40fd754df6982c7c7159d/8beaa/SplitDataTheRightWay_Thumbnail.png 1080w,\n/static/e9b8f309bba40fd754df6982c7c7159d/87706/SplitDataTheRightWay_Thumbnail.png 1280w","sizes":"100vw"},"sources":[{"srcSet":"/static/e9b8f309bba40fd754df6982c7c7159d/a66aa/SplitDataTheRightWay_Thumbnail.webp 750w,\n/static/e9b8f309bba40fd754df6982c7c7159d/65dd5/SplitDataTheRightWay_Thumbnail.webp 1080w,\n/static/e9b8f309bba40fd754df6982c7c7159d/71d4d/SplitDataTheRightWay_Thumbnail.webp 1280w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5625}}},"author":[{"id":"Lari Giba","bio":"Hi! I'm Lari and I run this website. You can find more information in the \"About\"-tab.","avatar":{"children":[{"fluid":{"aspectRatio":0.7434944237918215,"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAbABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAMEAgH/xAAWAQEBAQAAAAAAAAAAAAAAAAABAgP/2gAMAwEAAhADEAAAAeDIop5sDKn0aCSgzv8A/8QAHhABAAICAQUAAAAAAAAAAAAAAQACAxEEEyEiMjP/2gAIAQEAAQUC3ZAE6ZGW8HtMw1M2O+tTlfI9Z//EABURAQEAAAAAAAAAAAAAAAAAABAR/9oACAEDAQE/AYU//8QAFxEBAAMAAAAAAAAAAAAAAAAAABAREv/aAAgBAgEBPwFbMf/EABwQAAIBBQEAAAAAAAAAAAAAAAABMQIQESFxkf/aAAgBAQAGPwLfpJvBhuRVIgXRNwpsxct//8QAHRABAAMAAgMBAAAAAAAAAAAAAQARIUFRMWGBcf/aAAgBAQABPyG78vaUcDUeYJrJrCUcHAhzK6pYRVinqYAdKT7Y0/Yi+8UGE//aAAwDAQACAAMAAAAQsO1C/8QAGBEBAQEBAQAAAAAAAAAAAAAAAREAECH/2gAIAQMBAT8QPF4uTBS7/8QAGBEBAQEBAQAAAAAAAAAAAAAAAQARIVH/2gAIAQIBAT8QO2vYEgOZf//EAB8QAQEAAgIBBQAAAAAAAAAAAAERACExQVFhcZGxwf/aAAgBAQABPxDlRMDhDreGljrJ8uK5DiW/WJE4rXSx7wgAOy8sEKuzsxERAF59XxirS0k0dz03g4T3iY5jH9MRu2FvejEgEAAM/9k=","sizes":"(max-width: 800px) 100vw, 800px","src":"/static/9eab840ebd4087faf31c48702c92af36/2f1b1/lari.jpg","srcSet":"/static/9eab840ebd4087faf31c48702c92af36/fd013/lari.jpg 200w,\n/static/9eab840ebd4087faf31c48702c92af36/25252/lari.jpg 400w,\n/static/9eab840ebd4087faf31c48702c92af36/2f1b1/lari.jpg 800w,\n/static/9eab840ebd4087faf31c48702c92af36/0ff54/lari.jpg 1200w,\n/static/9eab840ebd4087faf31c48702c92af36/75b83/lari.jpg 1520w"}}]}}]},"fields":{"layout":"post","slug":"/dataset_optimization/split_data_the_right_way/"}},"primaryTag":"Model Optimization"}},"staticQueryHashes":["13077718","1316742457","1394499604","2907728042","3762220167"]}