exp1: Linear Regression and Stochastic Gradient Descent

2020 PostGraduate

You can click here to get the Chinese version.

Motivation of Experiment

1. Further understand of linear regression and stochastic gradient descent.
2. Conduct some experiments under small scale dataset.
3. Realize the process of optimization and adjusting parameters.

Dataset

Linear Regression uses Housing in LIBSVM Data, including 506 samples and each sample has 13 features. You are expected to download scaled edition. After downloading, you are supposed to divide it into training set, validation set.

Environment for Experiment

python3, at least including following python package: sklearn，numpy，jupyter，matplotlib.
It is recommended to install anaconda3 directly, which has built-in python package above.

Experiment Step

Linear Regression and Stochastic Gradient Descent

1. Load the experiment data. You can use load_svmlight_file function in sklearn library.
2. Devide dataset. You should divide dataset into training set and validation set using train_test_split function. Test set is not required in this experiment.
3. Initialize linear model parameters. You can choose to set all parameter into zero, initialize it randomly or with normal distribution.
4. Choose loss function and derivation: Find more detail in slides.
5. Calculate gradient $G$ toward loss function from each sample.
6. enote the opposite direction of gradient $G$ as $D$.
7. Update model: $W_t=W_{t-1}+\eta D$, where $\eta$ is learning rate, a hyper-parameter that we can adjust.
8. Get the loss $loss\_train$ under the training set and $loss\_val$ by validating under validation set.
9. Repeate step 5 to 8 for several times, get the value of $loss\_train$ as well as $loss\_val$.

Finishing experiment report according to experiment result: The template of report can be found in here

Submission

Requirement for Submission

• You only have to submit the experiment report. The experiment codes is not necessary for the submission.
• The submission of experiment report should be in PDF format（The template of report may be not very fit for the experiments, please revise it by yourself）
• please send all experiment reports to teaching assistant (jiaju.wu@qq.com, the email name should contain your name and your student number)

Deadline

• 24:00 on Jan. 7th, 2021，please send all experiment reports to teaching assistant (jiaju.wu@qq.com) before deadline.

P.S.

• The reports can be written in Chinese or English, in LaTeX or Word（If you write reports in Word, you need to export them to PDF format.）