Statistics 3850A - Introduction to Machine Learning

Winter 2025
Homework Assignment 1
January 28, 2025
Due: February 4, 17:00

• You may create your document in Word, Google Docs, LaTeX or any other word processor or you
may write by hand. If you decide for the latter option, try to write as legible as possible.
• Write detailed solutions with as much detail as possible. Justify your answers. Round your results
to four decimal digits.
• If a task is not clear, feel free to ask questions before or after the lectures, during office hours or
via e-mail; do not wait until the deadline to ask for explanations.
• Collaboration is encouraged when completing assignments. However, each student needs to submit
their own work.
• The assignments can be dropped off until the deadline in the box outside the office C521 labelled
STAT3850A or can be handed in during the lectures. Please make sure to write your name on the
assignment.
• The assignment will be accepted with a 25% penalty if submitted within 48 hours of the due date.
Otherwise, late assignments will not be marked, except for documented medical reasons.

Problem 1 (3 points)
Consider the simple linear regression model between the quantitative response Yi and the regressor Xi ,
i = 1, . . . , n:
i.i.d.
Yi = β0 + β1 · xi + εi , εi ∼ N(0, σ 2 ),
where x1 , . . . , xn are the realizations of X1 , . . . , Xn . The least squares estimators for the slope β1 and
the intercept β1 are: Pn
(xi − xn )(Yi − Y n )
β1 = i=1Pn
b
2
,
i=1 (xi − xn )
βb0 = Y n − βb1 · xn ,
where Y n is the sample mean (estimator) of RVs Y1 , . . . , Yn and xn is the estimated sample mean of
x1 , . . . , x n .
Prove the following equivalent expression for the variance of the estimator βb0 :
Pn 2
x2n
 
c2 · P i=1 xi
Var(βb0 ) = σ = c2 1 + P
σ ,
n n
n i=1 (xi − xn )2 n i=1 (xi − xn )
2

c2 is the unbiased estimator for σ 2 .

where σ

Problem 2 (3 points)
Consider the multiple linear regression model between the quantitative response Yi and the regressors
Xi,1 , . . . , Xi,k , i = 1, . . . , n, k ≥ 2:
i.i.d.
Yi = β0 + β1 · xi,1 + . . . βk · xi,k + εi , εi ∼ N(0, σ 2 ),

1
where xi = (xi,1 , . . . , xi,k )⊤ are the realizations of Xi,j , for i = 1, . . . , n and j = 1, . . . , k.
To check whether there is a relation between the response and the regressors, the following hypothesis
is tested:
H0 : β1 = . . . = βk = 0 versus H1 : at least one βj is non-zero,
with the F -statistic
TSS − RSS
F= k ,
RSS
n−k−1
Pn Pn
where TSS = i=1 (Yi − Y n ) and RSS = i=1 (Yi − Ybi )2 are the total sum of squares and the residual
2

sum of squares, respectively.

RSS
If the assumptions of the model hold, is an unbiased estimator for σ 2 .
n−k−1
TSS − RSS
a) Prove that, if H0 holds, is an unbiased estimator for σ 2 .
k
TSS
Hint: observe that is an unbiased estimator for σ 2 .
n−1
b) If there is no relation between the response and the regressor, what is the value of F ? Justify your
answer.

Problem 3 (8 points)
The dataset mtcars in the R package datasets contains the characteristics of 32 cars, among which wt =
weight (1000 lbs), mpg = miles/gallon and hp = gross horsepower. These variables are used in a multiple
logistic regression model to predict the response am = transmission (0 = automatic, 1 = manual):
logit(am) ≈ β0 + β1 · wt + β2 · mpg + β3 · hp.
The model provides the following output:
Estimate Std. error z value p-value
Intercept -15.72137 40.00281 -0.393 0.6943
wt -6.95492 3.35297 -2.074 ?
mpg 1.22930 1.58109 ? 0.4369
hp 0.08389 0.08228 ? ?

a) Based on the estimated values for the slopes, approximate the probability of a car having a manual
transmission, given that w
ft = 3.215, mg
pg = 21.4 and hfp = 110.

b) Determine the missing values in the output of the model. You may use the z table at the end of
the assignment to approximate the values.
βb
Hint: under the null hypothesis for each regressor H0 : β = 0, the statistic ∼ N(0, 1), where βb
Sβb
is the estimator for β and S b is the standard deviation (standard error) of the estimator β.
β
b

c) If the significant level of each hypothesis is α = 0.05, decide which regressor is significant in the
model. What is the relation between the significant regressor(s) and the response am ?

Problem 4 (6 points)
 
x1
The linear discriminant analysis classifies an observation x = to class 2 if δb2 (x) > δb1 (x), or
x2
equivalently,
1 π
b1 
x⊤ Σ
b −1 (µ
b2 − µ
b 1 ) > (µ b 2 )⊤ Σ
b1 + µ b −1 (µ
b2 − µ
b 1 ) − log ,
2 π
b2

2
and to class 1, otherwise, where µb 1 and µb 2 are the estimators for the mean vector of regressors in class 1
b −1 is the estimator for the inverse covariance matrix, and π
and class 2, respectively, Σ b1 and πb2 are the
prior probabilities of belonging to class 1 and 2, respectively.
For a given sample with the same number of observations in each group, the following values are
estimated (rounded to 2 digits):
     
−0.42 0.69 −1 0.07 0
µ
b1 = , µ b2 = , Σ =
b .
2.21 4.24 0 0.07

Determine the LDA boundary.

Problem 5 (5 points)
The Euclidean distance (p = 2) between a training set of 7 observations and a new observation x̃ is given
below:
Observation 1 2 3 4 5 6 7
Distance to x̃ 25.00 33.54 68.01 14.14 61.03 47.17 45.28
Class 2 1 1 2 1 2 1
We apply a K-nearest neighbor classifier for x̃ with K = 5. Run the algorithm and decide in which class
is the new observation x̃.

3
 Cumulative distribution function of standard normal distribution

z 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09
0.0 0.5000 0.5040 0.5080 0.5120 0.5160 0.5199 0.5239 0.5279 0.5319 0.5359
0.1 0.5398 0.5438 0.5478 0.5517 0.5557 0.5596 0.5636 0.5675 0.5714 0.5753
0.2 0.5793 0.5832 0.5871 0.5910 0.5948 0.5987 0.6026 0.6064 0.6103 0.6141
0.3 0.6179 0.6217 0.6255 0.6293 0.6331 0.6368 0.6406 0.6443 0.6480 0.6517
0.4 0.6554 0.6591 0.6628 0.6664 0.6700 0.6736 0.6772 0.6808 0.6844 0.6879
0.5 0.6915 0.6950 0.6985 0.7019 0.7054 0.7088 0.7123 0.7157 0.7190 0.7224
0.6 0.7257 0.7291 0.7324 0.7357 0.7389 0.7422 0.7454 0.7486 0.7517 0.7549
0.7 0.7580 0.7611 0.7642 0.7673 0.7704 0.7734 0.7764 0.7794 0.7823 0.7852
0.8 0.7881 0.7910 0.7939 0.7967 0.7995 0.8023 0.8051 0.8078 0.8106 0.8133
0.9 0.8159 0.8186 0.8212 0.8238 0.8264 0.8289 0.8315 0.8340 0.8365 0.8389
1.0 0.8413 0.8438 0.8461 0.8485 0.8508 0.8531 0.8554 0.8577 0.8599 0.8621
1.1 0.8643 0.8665 0.8686 0.8708 0.8729 0.8749 0.8770 0.8790 0.8810 0.8830
1.2 0.8849 0.8869 0.8888 0.8907 0.8925 0.8944 0.8962 0.8980 0.8997 0.9015
1.3 0.9032 0.9049 0.9066 0.9082 0.9099 0.9115 0.9131 0.9147 0.9162 0.9177
1.4 0.9192 0.9207 0.9222 0.9236 0.9251 0.9265 0.9279 0.9292 0.9306 0.9319
1.5 0.9332 0.9345 0.9357 0.9370 0.9382 0.9394 0.9406 0.9418 0.9429 0.9441
1.6 0.9452 0.9463 0.9474 0.9484 0.9495 0.9505 0.9515 0.9525 0.9535 0.9545
1.7 0.9554 0.9564 0.9573 0.9582 0.9591 0.9599 0.9608 0.9616 0.9625 0.9633
1.8 0.9641 0.9649 0.9656 0.9664 0.9671 0.9678 0.9686 0.9693 0.9699 0.9706
1.9 0.9713 0.9719 0.9726 0.9732 0.9738 0.9744 0.9750 0.9756 0.9761 0.9767
2.0 0.9772 0.9778 0.9783 0.9788 0.9793 0.9798 0.9803 0.9808 0.9812 0.9817
2.1 0.9821 0.9826 0.9830 0.9834 0.9838 0.9842 0.9846 0.9850 0.9854 0.9857
2.2 0.9861 0.9864 0.9868 0.9871 0.9875 0.9878 0.9881 0.9884 0.9887 0.9890
2.3 0.9893 0.9896 0.9898 0.9901 0.9904 0.9906 0.9909 0.9911 0.9913 0.9916
2.4 0.9918 0.9920 0.9922 0.9925 0.9927 0.9929 0.9931 0.9932 0.9934 0.9936
2.5 0.9938 0.9940 0.9941 0.9943 0.9945 0.9946 0.9948 0.9949 0.9951 0.9952
2.6 0.9953 0.9955 0.9956 0.9957 0.9959 0.9960 0.9961 0.9962 0.9963 0.9964
2.7 0.9965 0.9966 0.9967 0.9968 0.9969 0.9970 0.9971 0.9972 0.9973 0.9974
2.8 0.9974 0.9975 0.9976 0.9977 0.9977 0.9978 0.9979 0.9979 0.9980 0.9981
2.9 0.9981 0.9982 0.9982 0.9983 0.9984 0.9984 0.9985 0.9985 0.9986 0.9986
3.0 0.9987 0.9987 0.9987 0.9988 0.9988 0.9989 0.9989 0.9989 0.9990 0.9990
3.1 0.9990 0.9991 0.9991 0.9991 0.9992 0.9992 0.9992 0.9992 0.9993 0.9993
3.2 0.9993 0.9993 0.9994 0.9994 0.9994 0.9994 0.9994 0.9995 0.9995 0.9995
3.3 0.9995 0.9995 0.9995 0.9996 0.9996 0.9996 0.9996 0.9996 0.9996 0.9997
3.4 0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9998

Quantiles of the standard normal

distribution
α 0.9 0.95 0.975 0.99 0.995
zα 1.282 1.645 1.96 2.326 2.576