{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"###### COMP4670/8600 - Statistical Machine Learning - Tutorial"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this lab, we will use linear regression to predict the value of a home and explore the impact of regularisation.\n",
"\n",
"### Assumed knowledge\n",
"- Maximum likelihood solution to a linear regression problem, with and without regularisation (lectures)\n",
"- Matrix calculations in numpy (lab and precourse material)\n",
"- Theory behind regularisation (lectures)\n",
"\n",
"### After this lab, you should be comfortable with:\n",
"- Practical linear regression problems\n",
"- Picking an appropriate regularisation parameter for a given problem"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$\\newcommand{\\trace}[1]{\\operatorname{tr}\\left\\{#1\\right\\}}$\n",
"$\\newcommand{\\Norm}[1]{\\lVert#1\\rVert}$\n",
"$\\newcommand{\\RR}{\\mathbb{R}}$\n",
"$\\newcommand{\\inner}[2]{\\langle #1, #2 \\rangle}$\n",
"$\\newcommand{\\DD}{\\mathscr{D}}$\n",
"$\\newcommand{\\grad}[1]{\\operatorname{grad}#1}$\n",
"$\\DeclareMathOperator*{\\argmin}{arg\\,min}$\n",
"\n",
"Setting up the environment"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The data set\n",
"\n",
"\n",
"We will use [a dataset](https://machlearn.gitlab.io/sml2020/tutorials/02-dataset.csv) on the price of housing in Boston (see [description](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)). \n",
"We aim to predict the value of a home from other factors.\n",
"In this dataset, each row is one house. The first entry is the value of the house and we will predict it from the remaining values which have been normalised to be in the range $[-1, 1]$. The column labels are\n",
"\n",
"```'medv', 'crim', 'zn', 'indus', 'chas', 'nox', 'rm', 'age', 'dis', 'rad', 'tax', 'ptratio', 'b', 'lstat'```\n",
"\n",
"Download the dataset. Read in the data using ```np.loadtxt``` with the optional argument ```delimiter=','```, as our data is comma separated rather than space separated. Remove the column containing the binary variable ```'chas'```."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that the data is as expected using ```print()```. It should have 506 rows (examples) and 13 columns (1 label and 12 features). Check that this is the case. \n",
"\n",
"Hint: use assert."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regression without regularisation\n",
"\n",
"Implement a **function** to find the maximum likelihood solution $w_{ML}$ assuming Gaussian noise for this linear regression problem. Remember from the lectures that this is equivalent to a linear regresion problem with the cost function set as the sum of squares error."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training and testing\n",
"\n",
"Use a fifth of the available data for training the model using maximum likelihood. The rest of the data is allocated to the test set. Report the root mean squared error (RMSE) for the training set and the test set.\n",
"In this cases, use the identity map as the basis function, $\\phi(x)=x$.\n",
"\n",
"Note that the data may be sorted or ordered in some way we cannot predict. How will you account for this?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Interpreting the model\n",
"\n",
"Find the feature with the biggest magnitude of weight. Using ```matplotlib``` ([docs for ```matplotlib.pyplot.plot```](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html)), create a plot of this feature against the label for the datapoints in the training set. In a different colour, plot this feature against the predicted label. Create a similar plot for the test data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regression with regularisation\n",
"\n",
"Implement a **function** to find the maximum likelihood solution $w_{reg}$ for some regularisation parameter $\\lambda > 0$ assuming Gaussian noise for this linear regression problem."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By calculating the RMSE on the training and test sets, evaluate the impact of regularisation for $\\lambda = 1.1$.\n",
"\n",
"What is the effect of regularisation?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"--- replace this with your solution, add and remove code and markdown cells as appropriate ---"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Picking a regularisation parameter\n",
"\n",
"You will now explore picking a good regularisation parameter.\n",
"\n",
"What would you expect to see if you were under-regularising (so the parameter was too small)? Over-regularising? Discuss with a partner.\n",
"\n",
"Plot the RMSE on the training and test sets against the regularisation parameter $\\lambda$ for a range of values of $\\lambda$. What is a good range of values of $\\lambda$ to check? What do you think is the best value?\n",
"\n",
"Hint: You may find you want to plot against $\\log(\\lambda)$. The functions ``np.arange`` and ``np.linspace`` could be useful here (use whichever you think is more applicable)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"--- replace this with your solution, add and remove code and markdown cells as appropriate ---"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Basis Functions\n",
"We want to use basis functions to improve our performance. Implement subroutines for polynomial basis function of degree 2. See [the feature map based on the binomial formula](http://en.wikipedia.org/wiki/Polynomial_kernel) ."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Apply this to your train and test sets, and repeat the above exercise with these new features. Report what differences you see."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"--- replace this with your solution, add and remove code and markdown cells as appropriate ---"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# replace this with your solution, add and remove code and markdown cells as appropriate"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 1
}