{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Classification with Naive Bayes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"###### COMP4670/8600 - Introduction to Statistical Machine Learning - Assignment 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Name:\n",
"\n",
"Student ID:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Instructions\n",
"\n",
"| |Notes|\n",
"|:------------|:--|\n",
"|Maximum marks| 19|\n",
"|Weight|19% of final grade|\n",
"|Format| Complete this ipython notebook. Do not forget to fill in your name and student ID above|\n",
"|Submission mode| Use [wattle](https://wattle.anu.edu.au/)|\n",
"|Formulas| All formulas which you derive need to be explained unless you use very common mathematical facts. Picture yourself as explaining your arguments to somebody who is just learning about your assignment. With other words, do not assume that the person marking your assignment knows all the background and therefore you can just write down the formulas without any explanation. It is your task to convince the reader that you know what you are doing when you derive an argument. Typeset all formulas in $\\LaTeX$.|\n",
"| Code quality | Python code should be well structured, use meaningful identifiers for variables and subroutines, and provide sufficient comments. Please refer to the examples given in the tutorials. |\n",
"| Code efficiency | An efficient implementation of an algorithm uses fast subroutines provided by the language or additional libraries. For the purpose of implementing Machine Learning algorithms in this course, that means using the appropriate data structures provided by Python and in numpy/scipy (e.g. Linear Algebra and random generators). |\n",
"| Late penalty | For every day (starts at midnight) after the deadline of an assignment, the mark will be reduced by 20%. No assignments shall be accepted if it is later than 5 days. | \n",
"| Cooperation | All assignments must be done individually. Cheating and plagiarism will be dealt with in accordance with University procedures (please see the ANU policies on [Academic Honesty and Plagiarism](http://academichonesty.anu.edu.au)). Hence, for example, code for programming assignments must not be developed in groups, nor should code be shared. You are encouraged to broadly discuss ideas, approaches and techniques with a few other students, but not at a level of detail where specific solutions or implementation issues are described by anyone. If you choose to consult with other students, you will include the names of your discussion partners for each solution. If you have any questions on this, please ask the lecturer before you act. |\n",
"| Solution | To be presented in the tutorials. |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$\\newcommand{\\dotprod}[2]{\\left\\langle #1, #2 \\right\\rangle}$\n",
"$\\newcommand{\\onevec}{\\mathbb{1}}$\n",
"$\\newcommand{\\B}[1]{\\mathbf{#1}}$\n",
"$\\newcommand{\\Bphi}{\\boldsymbol{\\mathsf{\\phi}}}$\n",
"$\\newcommand{\\BPhi}{\\boldsymbol{\\Phi}}$\n",
"$\\newcommand{\\Cond}{\\,|\\,}$\n",
"$\\newcommand{\\DNorm}[3]{\\mathcal{N}(#1\\Cond#2, #3)}$\n",
"$\\newcommand{\\DUniform}[3]{\\mathcal{U}(#1 \\Cond #2, #3)}$\n",
"$\\newcommand{\\Ex}[2][]{\\mathbb{E}_{#1} \\left[ #2 \\right]}$\n",
"$\\newcommand{\\var}[1]{\\operatorname{var}[#1]}$\n",
"$\\newcommand{\\cov}[1]{\\operatorname{cov}[#1]}$\n",
"$\\newcommand{\\Norm}[1]{\\lVert#1\\rVert}$\n",
"$\\DeclareMathOperator*{\\argmax}{arg\\,max}$\n",
"\n",
"Setting up the environment (Please evaluate this cell to activate the $\\LaTeX$ macros.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import csv, scipy, scipy.stats, collections, itertools\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The data set\n",
"\n",
"\n",
"The data set contains census information. Our task is to predict whether an indivual earns more than some amount.\n",
"\n",
"Please download the following data:\n",
"* https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\n",
"\n",
"Read and preprocess the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"raw_data = np.genfromtxt(\"adult.data\", dtype=str, delimiter=',', autostrip=True)\n",
"\n",
"# targets\n",
"target = \">50K\"\n",
"target_column = raw_data[:,-1]\n",
"Y = (target_column == target).reshape(-1,1)\n",
"assert any(Y) and any(~Y), set(target_column)\n",
"\n",
"features = [\n",
" 'age', 'workclass', 'fnlwgt','education', 'education_num', 'marital_status', \n",
" 'occupation', 'relationship', 'race', 'sex', 'capital_gain', 'capital_loss', \n",
" 'hours_per_week', 'native_country'\n",
"]\n",
"\n",
"raw_features = raw_data[:,:-1] # drop the target\n",
"assert raw_features.shape[1] == len(features)\n",
"\n",
"\n",
"def preprocess_continuous(column):\n",
" # convert continuous variables stored as strings to binary by comparing with the median\n",
" float_column = column.astype(float)\n",
" return (float_column > np.median(float_column)).astype(float).reshape(-1,1)\n",
"\n",
"\n",
"def preprocess_categorical(column):\n",
" # convert categorical variables to indicator vectors\n",
" values = sorted(set(column))\n",
" if len(values) == 2:\n",
" values = values[:1]\n",
" return np.hstack([column.reshape(-1,1)==v for v in values]).astype(float)\n",
"\n",
"\n",
"preprocessor = collections.defaultdict(lambda: preprocess_categorical)\n",
"preprocessor.update(dict(\n",
" age=preprocess_continuous, \n",
" fnlwgt=preprocess_continuous, \n",
" education_num=preprocess_continuous,\n",
" capital_gain=preprocess_continuous, \n",
" capital_loss=preprocess_continuous, \n",
" hours_per_week=preprocess_continuous,\n",
"))\n",
"\n",
"# apply appropriate preprocessor to each column of raw_features\n",
"X_list = [preprocessor[feature](raw_features[:,features.index(feature)]) for feature in features]\n",
"for feature, X in zip(features, X_list):\n",
" assert X.shape[0] == raw_features.shape[0]\n",
" print(X.shape[1], '\\t', feature)\n",
" \n",
"make_feature_names = lambda feature, dimension: ['%s_%.2i' % (feature, i) for i in range(dimension)]\n",
"binary_feature_names_list = [make_feature_names(feature, X.shape[1]) for feature, X in zip(features, X_list)]\n",
"binary_feature_names = list(itertools.chain(*binary_feature_names_list))\n",
"\n",
"X = np.hstack(X_list)\n",
"assert set(X.flatten()) == {0.0, 1.0}\n",
"assert len(binary_feature_names) == X.shape[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"print('')\n",
"print('X.shape', '\\t', X.shape)\n",
"print('Y.shape', '\\t', Y.shape)\n",
"print()\n",
"print('mean(Y)', np.mean(Y))\n",
"\n",
"figure = lambda : plt.figure(figsize=(16,5))\n",
"figure()\n",
"plt.imshow(np.hstack((X,Y)))\n",
"plt.axis('auto')\n",
"plt.ylabel('data index');\n",
"plt.gca().set_xticks(list(range(X.shape[1]+1)))\n",
"plt.gca().set_xticklabels(binary_feature_names+['y'], rotation='vertical')\n",
"\n",
"ntrain = 1200\n",
"ishuffle = np.arange(X.shape[0])\n",
"np.random.seed(0)\n",
"np.random.shuffle(ishuffle)\n",
"itrain = ishuffle[slice(0,ntrain)]\n",
"itest = ishuffle[slice(ntrain,None)]\n",
"X_train, Y_train = X[itrain,:], Y[itrain,:]\n",
"X_test, Y_test = X[itest,:], Y[itest,:]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (3 points) 1A: Naive Bayes: Maximum Likelihood (m.l.)\n",
"Assume we have dataset $\\mathcal{D}=\\left\\{(\\mathbf{x}_i,y_i\\right)\\}_{i=1,2,\\ldots,n}$ where $\\mathbf{x}_i\\in\\{0,1\\}^d$, $y_i\\in\\{0,1\\}$, $n$ is the number of data points and $d$ the number of features.\n",
"1. State the independence assumption of the naive Bayes classifier.\n",
"- Appropriately assume Bernoulli random variables. Letting $p(x_j=1|y=k) = \\rho_{j,k}$ for $k=0,1$ and $p(y=1)=\\mu$, derive the maximum likelihood $\\rho_{j,k}$ and $\\mu$ (that is, the parameters which maximise the likelihood).\n",
"- Implement a function which computes these maximum likelihood parameters, and call it on ```X_train, Y_train```.\n",
"- Plot the $\\rho_{j,k}$ (vertical) vs. $j$ (horizontal) using ```plt.plot``` with ```marker='.'```, labeling appropriately.\n",
"- Print the number of $\\rho_{j,k}$ which are zero.\n",
"- Explain the problems which zero- (or one-) valued $\\rho_{j,k}$ can lead to."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (3 points) 1B: Naive Bayes: Maximum a Posteriori (m.a.p.)\n",
"Let $\\rho_{j,k}\\sim\\text{Beta}(\\beta)$, with p.d.f. $f_{\\rho_{j,k}}(\\rho)=\\frac{\\rho^{\\beta-1}(1-\\rho)^{\\beta-1}}{Z(\\beta)}$ where $Z$ is a normalisation factor. Assume a uniform prior for $\\mu$.\n",
"1. Derive the maximum a posteriori $\\rho_{j,k}$ and $\\mu$ given the above prior.\n",
"- Implement a function which computes these maximum a posteriori parameters.\n",
"- Verify with ```assert np.allclose()``` that the m.a.p. solution with $\\beta=1$ is identical to the m.l. solution. \n",
"- Call your function on ```X_train, Y_train``` with $\\beta=10,100,1000$. For each case scatter plot the m.a.p. parameters vs. the m.l. parameters, all on one axis, colored and labelled appropriately.\n",
"- Give one example of the role of $\\beta$ as evidenced by the plot."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (2 points) 1C: Naive Bayes: Prediction\n",
"1. Derive the log predictive distribution $\\log p(y=1|\\mathbf{x},\\{\\rho_{j,k}\\},\\mu)$. Where appropriate, implement this by adding log probabilities instead of multiplying probabilities. Use ```np.logaddexp``` to add probabilities stored in log space.\n",
"- Call the above funcion on ```X_test``` using m.a.p. parameters computed with $\\beta=2$.\n",
"- Plot a histogram of the probabilities (use ```np.exp``` on the log probabilities) using ```plt.hist``` with ```normed=True,histtype='step',label='your_label'```. Do this for the predictive test probabilities corresponding to ```Y_test==1``` and ```Y_test==0```, puting the histograms on the same plot, labeling appropriately."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (2 points) 1D: Naive Bayes: Evaluation\n",
"1. Write a function ```evaluate``` which takes data ```X,Y``` and model parameters $\\rho_{j,k}$ and $\\mu$, and returns a dict with keys ```mean_logp_true``` (for the mean predictive log probability of the ground truth labels ```Y``` given the data matrix ```X``` and ```percent_correct``` (for the percent correctly classified, assuming we classify each datum into the class with maximum predictive probability).\n",
"- Write a function ```cross_validate(beta, nfolds, X, Y)``` which performs cross validation with ```nfolds``` folds, for hyper parameter $\\beta$. For each split, call your ```evaluate``` function, and return a ```dict``` with averaged results for each evaluation metric. You can make use of our ```xval_inds``` function.\n",
"- For $\\beta$ in ```np.logspace(np.log10(2),2,32)``` compute the evaluation metrics on the test set, ```X_test, Y_test```, as well as the cross-validated estimates. Make one plot for each metric, showing both results on each (for a total of four curves). Label appropriately."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def xval_inds(ndata, nfolds):\n",
" # return a list of (trainind, testind) pairs for ndata data points and nfolds xval folds\n",
" assert ndata % nfolds == 0\n",
" nchunk = int(ndata / nfolds)\n",
" itest = [list(range(i*nchunk,(i+1)*nchunk)) for i in range(int(nfolds))]\n",
" itrain = [sorted(set(range(ndata)).difference(set(i))) for i in itest]\n",
" return list(zip(itrain, itest))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (2 points) 1E: Naive Bayes: Discussion\n",
"1. Explain the shortcomings of our preprocessing of (a) continous variables, and (b) categorical variables. Make note of the assumptions of our model.\n",
"- Suggest improvements to our data processing pipeline, both in terms of the representation and the model.\n",
"- Consider the distribution $p(a,b,c)$ where all three variables are binary. How many parameters are needed to specify this distribution (a) in general and (b) if it factorises as $p(a|c)p(b|c)p(c)$?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (3 points) 2A: Representation Proof\n",
"Consider a parametric model governed by the parameter vector $\\mathbf{w}$ together with a dataset of input values $\\mathbf{x}_1,\\ldots,\\mathbf{x}_N$ and a nonlinear feature mapping $\\phi(\\mathbf{x})$. Suppose that the dependence of the error function on $\\mathbf{w}$ takes the form $J(\\mathbf{w}) = f(\\mathbf{w}^\\top \\phi(\\mathbf{x}_1), \\ldots, \\mathbf{w}^\\top \\phi(\\mathbf{x}_N)) + g(\\mathbf{w}^\\top \\mathbf{w})$\n",
"where $g(\\cdot)$ is a monotonically increasing function. By writing $\\mathbf{w}$ in the form\n",
"$$\n",
"\\mathbf{w} = \\sum_{n=1}^N \\alpha_n \\phi(\\mathbf{x}_n) + \\mathbf{w}_\\perp\n",
"$$\n",
"show that the value of $\\mathbf{w}$ that minimizes $J(\\mathbf{w})$ takes the form of a linear combination of the basis functions $\\phi(\\mathbf{x}_n)$ for $n = 1, \\ldots, N$."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"## (2 points) Maximum likelihood (ML) and Maximum A Posteriori (MAP)\n",
"We assume data samples $X_n = \\{ x_1,\\dots,x_n \\}$ are generated i.i.d. from a uniform distribution\n",
"$ \\DUniform{x}{0}{\\theta} $ between $ 0 $ and an unknown positive parameter $\\theta$:\n",
"$$\n",
" p(x \\Cond \\theta) = \\DUniform{x}{0}{\\theta} = \n",
"\\begin{cases}\n",
" 1/\\theta & 0 \\leq x \\leq \\theta \\\\\n",
" 0 & \\textrm{otherwise} \\\\\n",
"\\end{cases}\n",
"$$\n",
"\n",
"Assume the data samples $ X_4 = \\{ 5, 7, 3, 9 \\}$ have been observed.\n",
"\n",
"1. Calculate $\\theta_{ML} = \\argmax_{\\theta} p(X_4 \\Cond \\theta)$, \n",
"the maximum likelihood estimate of $\\theta$ for the observed data.\n",
"\n",
"- Calculate $p(\\theta \\Cond X_4)$, the posterior distribution of $\\theta$ given that the \n",
"data $ X_4 $ have been observed and \n",
"the initial distribution for $\\theta$ is given as $p(\\theta) = p(\\theta \\Cond X_0) = \\DUniform{x}{0}{10}$.\n",
"\n",
"- Calculate $\\theta_{MAP} = \\argmax_{\\theta} p(\\theta \\Cond X_4)$, the maximum a posteriori\n",
"estimate of $\\theta$ given the data $ X_4 $ and the initial distribution $p(\\theta)$ as in the previous question.\n",
"\n",
"- Calculate $\\theta_{ML}$, $p(\\theta \\Cond X_4)$, and $\\theta_{MAP}$ for the case that the observed data are $ X_4 = \\{ 9, 5, 7, 3 \\}$ instead of the $ X_4 = \\{ 5, 7, 3, 9 \\}$ given above."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"## (2 points) Laplace Approximation\n",
"The function\n",
"$$\n",
" f(z) = z^k e^{-z^2/2} \\qquad \\qquad z \\in [0, \\infty), \\qquad k > 0\n",
"$$\n",
"can be considered as an (unnormalised) probability density.\n",
"\n",
"1. Verify that it is possible to approximate $ f(z) $ with the Laplace Approximation.\n",
"\n",
"- Using the Laplace Approximation, find the mean and the variance of the Normal Distribution which best approximates the normalised version of $ f(z) $.\n",
"\n",
"- The analytical form of the normalisation of $ f(x) $ is not so easy to find. Therefore, use Python to implement a numerical approximation using $ N = 100 $ identically sized intervals between $ 0 $ and $ a = 10 $ to calculate the normalisation \n",
"$$\n",
" \\int_0^{\\infty} f(z) \\mathrm{d}z \n",
" \\approx \\int_0^{a} z^k e^{-z^2/2} \\mathrm{d}z \n",
" \\approx \\sum_{i=1}^{100} \\dots\n",
"$$\n",
"and report the results for the normalisation with a \n",
"precision of $5$ digits after the comma for the three cases $ k = \\{0.5, 3, 5 \\}$.\n",
" \n",
"- For each of the three cases $ k = \\{0.5, 3, 5 \\}$, plot the normalised function $ f(z) $\n",
"and the corresponding Normal Distribution with parameters resulting from the Laplace Approximation.\n",
"\n",
"- Why is it reasonable to replace the upper limit of $ \\infty$ with $ a = 10 $ ?"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### Answer\n",
"*--- replace this with your solution: add cells as necessary but leave the blue Answer above ---*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 0
}