linear regression

discriminative: SVM, LR,
generative:
smoothing,

Linear Regression
Regularized Linear Regression – Ridge regression, Lasso
Polynomial Regression
Kernel Regression
Gaussian Process Regression
Regression Trees, Splines, Wavelet es$mators, …

Linear Regression

Learn to derive the least squares estimate by optimization.

(1)另一种线性回归方法：Normal Equation；

(2)Gradient Descent与Normal Equation的优缺点；

前面我们通过Gradient Descent的方法进行了线性回归，但是梯度下降有如下特点：

(1)需要预先选定Learning rate；

(2)需要多次iteration；

(3)需要Feature Scaling；

因此可能会比较麻烦，这里介绍一种适用于Feature数量较少时使用的方法：Normal Equation；

当Feature数量小于100000时使用Normal Equation；

当Feature数量大于100000时使用Gradient Descent；

Normal Equation的特点：简单、方便、不需要Feature Scaling；

How to evaluate: 

While this largely depends on exactly what your goals are, a simple and standard way to do this would be measuring the mean squared error (MSE). So if you have your test dataset  which consist of input/output pairs, ={(x1,y1),(x2,y2),…,(xn,yn)} and your parameters a and b, then the MSE can be calculated as

MSEa,b=1n∑i=1n(yi−(axi+b))2.

This is probably a sensible way to measure your error also since this is likely the criteria you used for finding the parameters a and b. If you want to get a better idea of how well your estimated parameters generalize, you should look into something like cross validation.

Regularized Linear Regression

The big question is how do we
choose the regularization coefficient,
the width of the kernels or the
polynomial order?
Solution: cross-validation


 the L1 norm
This type of regularization is at the heart of a recent revolution in data
acquisition known as compressed sensing.