Real Estate Price Prediction with Regression and Classification

CS 229 Autumn 2016 Project Final Report

Hujia Yu, Jiafu Wu
[hujiay, jiafuwu]@stanford.edu

1. Introduction 3. Models

Housing prices are an important reflection of the We would perform two types of supervised learning
economy, and housing price ranges are of great interest algorithms: classification and regression. While it seems
for both buyers and sellers. In this project. house prices more reasonable to perform regression since house prices
will be predicted given explanatory variables that cover are continuous, classifying house prices into individual
many aspects of residential houses. As continuous house ranges of prices would also provide helpful insight for the
prices, they will be predicted with various regression users; also, this helps us explore different techniques
techniques including Lasso, Ridge, SVM regression, and which might be regression- or classification-specific.
Random Forest regression; as individual price ranges, Since there are 288 features in the dataset, regularization
they will be predicted with classification methods is needed to prevent overfit. In order to determine the
including Naive Bayes, logistic regression, SVM regularization parameter, throughout the project in both
classification, and Random Forest classification. We will classification and regression parts, we would first perform
also perform PCA to improve the prediction accuracy. K-fold cross validation with k = 5 on a wide range of
The goal of this project is to create a regression model and selection of regularization parameters; this helped us to
a classification model that are able to accurately estimate select the best regularization parameters in the training
the price of the house given the features. phase. In order to further improve our models, we also
performed principal component analysis pipeline on all
2. Data and Preprocessing models, and cross validated number of components to fit
in each of the model to give the optimized results.
The dataset is the prices and features of residential houses
sold from 2006 to 2010 in Ames, Iowa, obtained from the 3.1 Classification
Ames Assessor’s Office. This dataset consists of 79
house features and 1460 houses with sold prices. Data Preprocessing
Although the dataset is relatively small with only 1460
examples, it contains 79 features such as areas of the The house prices were classified into the buckets of
houses, types of the floors, and numbers of bathrooms. prices. Based on the distribution of the housing prices in
Such large amounts of features enable us to explore the data set, the price buckets were followed: [0, 100K),
various techniques to predict the house prices. [100K, 150K), [150K, 200K), [200K, 250K), [250K,
300K), [300K, 350K), [350K, ထ), and we would need to
The dataset consists of features in various formats. It has perform multi-class classification to predict house prices
numerical data such as prices and numbers of into these seven buckets. The performance of each model
bathrooms/bedrooms/living rooms, as well as categorical can be characterized by accuracy rate, which is the
features such as zone classifications for sale, which can be number of test examples correctly classified over the
‘Agricultural’, ‘Residential High Density’, ‘Residential number of total examples.
Low Density’, ‘Residential Low Density Park’, etc. In
order to make this data with different format usable for Models and Results
our algorithms, categorical data was converted into
separated indicator data, which expands the number of Our baseline model for classification is Naive Bayes. We
features in this dataset. The final dataset has 288 features. implemented two types of Naive Bayes: Gaussian Naive
We splitted our dataset into training and testing set with a Bayes and Multinomial Naive Bayes. Our initial
roughly 70/30 split, with 1000 training examples and 460 expectation is that Multinomial might perform better than
testing examples. Besides, there were some features that Gaussian, since most of the features are binary indicator
had values of N/A; we replaced them with the mean of values, while only a minority of them are continuous. The
their columns so that they don’t influence the distribution. test result showed that Gaussian Naive Bayes had 21%

1
accuracy while Multinomial Naive Bayes had 51% Before we fit the regression models, we preprocessed the
accuracy. Optimistically speaking, even the Gaussian data with log-transform on the skewed features, including
Naive Bayes model performed better than random guess the target variable SalePrice, to have normal distributions.
(14% or 1/7 with 7 price buckets). Besides, to better
characterize how bad the Multinomial Naive Bayes Models and Results
misclassified, we would assign the indexes on price
buckets according to their orders, and we would compute For regression models, we try to solve the following
the average absolute difference between the expected problem: given a processed list of features for a house, we
indexes and the computed indexes of all examples, which would like to predict its potential sale price. Linear
can be viewed a root mean square error. The Multinomial regression is a natural choice of baseline model for
Naive Bayes had an average absolute difference of 0.689, regression problems. So we first ran linear regression
which means that in average the indexes were including all features, using our 288 features and 1000
misassigned by less than 1. training samples. The model is then used to predict sale
prices of houses given features in our test data and is
In order to improve our classification, we turned to compared to the actual sale prices of houses given in test
Multinomial Logistic Regression on the same dataset. We data set. The performance was measured by Root Mean
tuned the L2 regularization parameters using the 5-fold Square Error (rmse) of the predicted results and the actual
cross validation (we would address that with more details results. Our baseline model generated a rmse of 0.5501.
later); we also fit an intercept into the features. Note that since the target variable SalePrice is
Nevertheless, its performance was actually similar with log-transformed before model fitting, the resulting rmse is
Multinomial Naive Bayes; it had an accuracy of 50% based on differences in the log-transformed sale prices,
compared with 51% of Multinomial Naive Bayes. After which accounts for the small values of rmse for regression
the tuning of the parameters, it appeared that the models.
performance of both Naive Bayes and Multinomial
Logistic Regression was capped at around 50%. After using linear regression model as the baseline model,
we included the regularization parameters in linear
We continued to explore other models for our multiclass regression models to reduce overfitting. Linear regression
classification. One choice was Support Vector Machine with Lasso after 5-fold cross validation generated a rmse
Classification(SVC), and we chose linear kernel as well of 0.5418, which is better than our baseline model. Also,
as Gaussian kernel. Similar to Multinomial Logistic linear regression with lasso automatically picked 110
Regression, we added an L2 regularization parameter and variables and eliminated the other 178 variables to fit in
tuned it using cross validation. We found out that the the mode. The plot on selected features and their weights
SVC with linear kernel outperformed our past model with in lasso regularized model is attached in part 6.
an accuracy of 63%, while the SVC with Gaussian kernel
only had an accuracy of 41%. Other than lasso regularizer, we also applied ridge
regularizer with cross validation in our linear regression
At last, our final choice of classification model is random model, which generate a rmse of 0.5448. This rmse is also
forest classification. One important parameter to control better than our baseline model, meaning that regularized
overfitting is the maximum depth that we allow the trees linear regression helped with overfitting.
to grow; as a result, similar to the L2 regularization
parameters of Multinomial Logistic Regression and SVC, Support vector regression (SVR) with Gaussian and linear
we performed cross validation to tune this maximum kernels are also fitted to the features. Parameters Cs of
depth parameters for regularization. After tuning, we both models are cross validated to pick the best
obtain the accuracy of 67%, which is actually similar to performing parameters. SVR with Gaussian kernel model
SVC with linear kernel. generated a rmse of 0.5271, and that of linear kernel
generated a high rmse of 5.503. SVR with Gaussian
So far, we can observe that the SVC with linear kernel kernel performed 4% better than our baseline model.
and Random Forest Classification had the best Whereas SVR with linear kernel generated a relatively
performance, with the accuracy of 67%. high rmse due to the kernel’s unfit with the dataset in this
case.
3.2 Regression
Lastly, we fitted our training dataset with random forest
Data Preprocessing regression model, with max_depth parameter cross
validated to be 150. Our random forest regression model
2
generated a rmse of 0.5397, which is also better than our
baseline model.

Overall, all of the models performed better than the basic

linear regression model, except SVR with linear kernel.
Specifically, SVR with Gaussian kernel performed the
best among all models, which generated the lowest rmse
of 0.5271. We suggest that this regression model be used
for future house price predictions.

4. Performance Optimization

For our classification and regression algorithms, we found

two ways for optimizations: regularization and
dimensionality reduction; both ways were useful to
reduce overfit and increase accuracy.
Figure 1. Cross Validation RMSE versus C
4.1 Regularization
For all the algorithms except Random Forest would use
For most of our algorithms, the regularization term played cross validation to tune their regularization coefficient C,
an important role on model performance, and tuning the while Random Forest Classification and Regression
regularization term was one of the most important parts would need cross validation to tune their maximum depth,
when applying these algorithms. Cross validation were which is also a parameter to prevent overfitting.
performed during the Multinomial Logistic Regression,
SVC and Random Forest Classification on classification 4.2 Dimensionality Reduction with PCA
algorithms, as well as Lasso/Ridge Regression, SVR and
Random Forest Regression on our regression algorithms. Beside regularization, another way to prevent overfit is to
reduce the dimension of our data. As we described before,
In order to optimize our regularization term, we would our dataset contained 288 features; as a result, it might get
tune this regularization coefficient using cross validation. overfitted due to the large amount of features. One
Let us take Multinomial Logistic Regression for example. algorithm for dimensionality reduction is Principal
We would like to tune its L2 regularization coefficient C. Component Analysis (PCA).
Here coefficient C can be considered the inverse of the
regularization parameter; larger C corresponds to smaller With PCA, we would be able to use the principal
regularization. We first selected a wide range of data components instead of all the features in the dataset.
points C; for each C, we would perform K-fold cross Though the PCA is capable to produce the principal
validation and compute the cross validation error score components, we still need to decide how many
(RMSE). Then we plotted out the RMSE versus C. Based components we should include in our training and testing.
on the graph, we would hope that our range of C would As a result, we would need to tune the number of
roughly cover the particular C corresponding to the principal component and pick the number using
minimum error score, and as a result we would pick that cross-validation, using the same method as shown before
as our designated regularization coefficient for training. in the regularization part: we first selected a wide range of
Our graph is shown below. Based on the plot below, we data points N; for each N, we would perform K-fold cross
can observe a global minimum at C = 10. validation and compute the cross validation error score
(RMSE). Then we plotted out the RMSE versus N and we
would pick the N corresponded to the global minimum or
close to global minimum as our designated regularization
coefficient for training.

Using PCA with tuned numbers of principal components,

we would use the reduced-dimension dataset to perform
our classification and regression algorithms again. For
most of the algorithms we can observe improvements of
accuracy, as shown in tables in part 5.
3
5. Model Comparison significant improvement from 21% to 50%. One
interesting point to notice is that the PCA dataset cannot
work on Multinomial Naive Bayes algorithm because it
Classification Accuracy Accuracy
requires the dataset to be non-negative while the PCA
w/o PCA w/ PCA
transformation might introduce negative value inside the
dataset and formid Multinomial Naive Bayes to be usable.
Gaussian Naive Bayes 0.2087 0.4978
For regression problem, PCA improved the results of all
Multinomial Naive Bayes 0.5109 - the models slightly. For example, Our original best
algorithms is SVR with Gaussian kernel with 0.5271 rmse
Multinomial Logistic 0.5000 0.5587 without PCA. Now with the data after PCA, the RMSE of
Regression SVR with Gaussian kernel dropped slightly to 0.5269.
PCA helped improve performance of random forest
SVC linear kernel 0.6740 0.6913 regression model the most, whose rmse dropped from
0.5397 to 0.5323, showing a 1.37% decrease in rmse. We
0.4109 0.4109 did not perform pca on lasso regularized linear regression
SVC Gaussian kernel since lasso regularization already performs feature
reduction during model fitting. PCA was not fitted to the
Random Forest 0.6652 0.5674 SVR model with linear kernel either due to the kernel’s
Classification unfit for the data set.

6. Visualizations and Analysis

Table 1. Classification Results

Regression RMSE RMSE w/

w/o PCA PCA

Linear Regression w/o 0.5501 0.5473

regularization

Lasso 0.5418 -

Ridge 0.5448 0.5447

SVR (linear kernel) 5.503 -

SVR (Gaussian kernel) 0.5271 0.5269

Figure 2. First three PCA directions. The 3D axes are
0.5397 0.5323 three largest eigenvectors of the training data.
Random Forest Regression
In PCA, instead of regressing the dependent variable on
Table 2. Regression Results the explanatory variables directly, the principal
components of the explanatory variables are used as
regressors. By regressing on only a subset of all the
For classification problem, adding PCA generally helped principal components, PCA can result in dimension
the result. Our original best algorithms are SVC with reduction through substantially lowering the effective
linear kernel and Random Forest Classification with number of parameters characterizing the underlying
accuracy of 67% without PCA. Now with the data after model. According to Figure 2, the three largest
PCA, the accuracy SVC with linear kernel raised slightly eigenvectors of the PCA model above clearly
to 69% while the accuracy of Random Forest demonstrates a linear relationship between components
Classification dropped to 57%. Also, even the Gaussian and target variable.
Naive Bayes with bad performance as baseline had a

4
 very expensive sometimes) can have significant impact on
housing prices.

7. Conclusion

For classification problem, the best-performing model is

SVC with linear kernel, with the accuracy of 0.6740; with
PCA preprocessing, the accuracy can be increased to
0.6913, which is also the best among all other algorithms
with PCA preprocessing.

For regression problem, the best-performing model is

SVR with gaussian kernel, with rmse of 0.5271, however,
visualization for SVR is difficult due to its
high-dimensionality. On the other hand, lasso regression
Figure 3. Residuals from Lasso Model. model can provide insights about chosen features, which
is helpful in helping us understanding the correlations of
Figure 3 presents a visual realization of residuals obtained house features and its sale prices.
from fitted lasso regression model, we can see that, for the
predicted values, residuals are closely clustered around 0, According to our analysis, living area square feet,
which indicates that the model does not have significant material of the roof, and neighborhood have the greatest
misfit problem due to the randomly distributed residuals. statistical significance in predicting a house’s sale price.

8. Future Work

When PCA were performed, we cross-validated its

number of dimension given the tuned regularization
coefficient. To obtain an lower test error, we might need
to cross-validate both the number of dimension after PCA
as well as regularization coefficient together, though this
would be more computationally intensive.

For Random Forest Classification/Regression, besides the

depth, we might need to examine further variations to
optimize this algorithm, such as considering the splits of
nodes, the requirements of leaf nodes, etc.

9. Reference:
Figure 4. Coefficients of Covariates Selected in the Lasso
Model
[1] De Cook, Dean. “Ames, Iowa: Alternative to the
Boston Housing Data as an End of Semester
Figure 4 presents selected variables from the lasso model,
Regression Project.” ​Journal of Statistics
and value of coefficients for each fitted covariate. The
Education, vol. 19, no. 3, 2011.
variable that has the greatest coefficient is GrLivArea
(Continuous): Above grade (ground) living area square
feet. This makes intuitive sense that the sale price of a real
estate property is strongly correlated with its living area.
The variable that has the greatest negative impact on the
housing prices is Roof Matl (Nominal): Roof material :
ClyTile, which indicates the material of the roof. This
offers a new perspective of the housing prices as well,
since the cost of the materials of the roof (which could be