\documentclass[notitlepage,12pt]{article}
%twocolumn
\usepackage{epsfig}
\usepackage{fullpage,psfrag}
\usepackage{graphicx}
\usepackage{color}
\usepackage{subfigure}
\usepackage{rotating}
\usepackage{setspace}
\usepackage{geometry}
%\usepackage{endfloat}
\usepackage{hyperref}


%\usepackage{loflot}
%\setstretch{0.8}

\textheight 8
in \oddsidemargin 0.03in \textwidth 6.0in
 \topmargin -.25in
\parskip .1in


\usepackage{natbib}

%\newcommand{\xv}{\mbox{\boldmath$x$}}
%\newcommand{\uv}{\mbox{\boldmath$u$}}
%\newcommand{\xiv}{\mbox{\boldmath$\xi$}}

\begin{document}
%\pagestyle{empty}

\begin{center}
FALL 2009\\
Stat 849: Homework Assignment 4\\
Due: November 13, 2009 \\
Total points = 250\\
\end{center}
\begin{enumerate}
\item (30) Show that the first step in the forward selection is
equivalent to selecting the variable most highly correlated with the
response.

\item (20) AIC criterion for a model $\mathcal{M}$ is given by
\[AIC = -2 \times \textnormal{log(likelihood under $\mathcal{M}$)} +  2 \times \textnormal{(\# of parameters in $\mathcal{M}$)}.\]
Find an expression for AIC in terms of residual sum of squares in
the Gaussian linear model and simplify it as much as you can.

\item (75) Design a simulation study to investigate the effects of over
and under fitting in linear regression models. Summarize and report
your conclusions at two sample sizes, $n=50$, $n=500$.

\textit{Hint:} For example, let $X_1, \cdots, X_5$ be normally
distribution random variables. Let the true regression model be \[Y
= \beta_0 + \beta_1X_1 +\beta_2X_2 + \beta_3X_3+ \epsilon,\] where
$\epsilon \sim \mathcal{N}(0, \sigma^2)$. You will need to generate
$n=50$ observations according to this mechanism. For studying the
effect of under-fitting, fit $Y \sim X_1 + X_2$ and report estimates
of $\beta_0$, $\beta_1$, $\beta_2$, $\beta_3$, and $\sigma^2$.
Similarly, for studying the effect of over-fitting, fit $Y \sim X_1
+ X_2 + X_3 + X_4 + X_5$ and report the estimates of the
coefficients in the true model. For this to be a simulation study,
you will need to repeat this process, say at least 100 times, and
report both bias and variance of your estimates.



\item (75) Consider the linear model
\begin{eqnarray} Y = X\beta +
Z\gamma + \epsilon, \label{q3}
\end{eqnarray}
with $E(\epsilon) = 0$ and $cov(\epsilon)= \sigma^2I$. $X$ is  a
known $n \times p$ matrix of rank $p<n$, and $Z$ is a known $n
\times 1$ vector that is linearly independent of the columns $X$ and
$\beta$, $\gamma$ and $\sigma^2$ are unknown parameters.

\begin{enumerate}
\item Consider fitting the above model (\ref{q3}) by ignoring the
$Z \gamma$ term. The corresponding ordinary least squares estimator
of $\beta$ is obtained as $\hat{\beta} = (X^TX)^{-1}X^T Y$. Let $r$
be the vector residuals, with $r_i = Y_i - \hat{Y}_i$, from the
fitted model. Derive $E(r)$ and $cov(r)$.

\item Consider a full least squares fit of (\ref{q3}). Let $M =
X(X^TX)^{-1}X^T$. Show that \[\hat{\gamma} = \frac{Z^T (I - M)
Y}{Z^T (I - M) Z} .\]

\textbf{Hint:} First rewrite $X\beta+ Z\gamma$ as $X\delta +
(I-M)Z\gamma$, where $\delta$ can involve both parameters and
elements of $X$ and/or $Z$.

\item Argue whether or not the following claim is correct: "If the
plot of $r$ versus $Z$ represents the influence of $Z$ after
accounting for other variables, then the slope from fitting a simple
linear regression of $r$ on $Z$ will be equal to the $\gamma$
estimate that I would get from fitting the model (\ref{q3}).


\end{enumerate}

\item  (50) \textbf{Job proficiency data (\texttt{jobdata.txt}).} A personnel officer
in a governmental agency administered four newly developed aptitude
tests to each
 of 25 applicants for entry-level clerical positions in the
 agency. For purpose of the study, all 25 applicants were accepted
 for positions irrespective of their test scores. After a
 probationary period, each applicant was rated for proficiency on
 the job. The scores on the four tests $(X_1, X_2, X_3, X_4)$ and
 the job proficiency score $(Y)$ for the 25 employees are given in
 \texttt{jobdata.txt}, where the first column represents $Y$ and the rest represent
 the test scores $(X_1, X_2, X_3, X_4)$.

Using forward selection, backward deletion, and forward selection \&
backward deletion stepwise methods to find the best subset of
predictor variables to predict job proficiency as a linear function
of test scores. Compare models obtained by each stepwise algorithm,
choose a final model and discuss how and why you chose it.

\end{enumerate}
\end{document}


 \item (120) \textbf{Ridge regression, PCR, and PLS}.   For this
question, you are going to use the Boston housing data from
\texttt{http://lib.stat.cmu.edu/datasets/boston} and you will try to
model the median value of owner-occupied homes in terms of the
covariates in the dataset. Both the data and its description are
available at this website. First, you will set aside 50 randomly
selected observations as the test data and the following model
fitting will be based on training data.

For each of the questions  below, provide the test and training
errors of the models. Don't forget to normalize the data before you
proceed with the following questions.

\begin{enumerate}

\item Perform variable selection with AIC criteria.

\item  Obtain the principal components for the explanatory variables in this dataset. Plot the eigen values
versus principal components indices and decide on the number of
principal components to use based on this plot. Fit the principal
components regression with the selected number of components and
comment on the model fit.

\item Use cross-validation to select the number of principal components (\texttt{pls.pcr} package should be helpful),
fit the PCR with the selected number of components and compare the
results with part (b).

\item Do ridge regression for a series of model complexity (regularization) parameter $\lambda$. Select a value based
on cross-validation or some other criteria and report the results
based on this value. Comment on the results. (Functions
\texttt{lm.ridge} and \texttt{select} might be useful).

\item Fit  a partial least squares regression model by  choosing the number of partial least squares directions by cross-validation (\texttt{plsr
} function might be useful).

\item Compare the test and training error of the all models you came up with in the above parts.
\end{enumerate}




\end{enumerate}

\end{document}
