least squares

Examples of least squares in the following topics:

• Conditions for the least squares line

  • The variability of points around the least squares line remains roughly constant.
  • Should we have concerns about applying least squares regression to the Elmhurst data in Figure 7.12?
  • Least squares regression can be applied to these data.

• Homework

  • Calculate the least squares line.
  • Calculate the least squares line.
  • Calculate the least squares line.
  • Calculate the least squares line.
  • Calculate the least squares line.

• Least-Squares Regression

  • The criteria for determining the least squares regression line is that the sum of the squared errors is made as small as possible.
  • The criteria for the best fit line is that the sum of squared errors (SSE) is made as small as possible.
  • Therefore, this best fit line is called the least squares regression line.
  • Ordinary Least Squares (OLS) regression (or simply "regression") is a useful tool for examining the relationship between two or more interval/ratio variables assuming there is a linear relationship between said variables.
  • This method minimizes the sum of squared vertical distances between the observed responses in the dataset and the responses predicted by the linear approximation.

• Estimating the Target Parameter: Point Estimation

  • Another popular estimation approach is the linear least squares method.
  • The approach is called "linear" least squares since the assumed function is linear in the parameters to be estimated.
  • In statistics, linear least squares problems correspond to a statistical model called linear regression which arises as a particular form of regression analysis.
  • One basic form of such a model is an ordinary least squares model.
  • Contrast why MLE and linear least squares are popular methods for estimating parameters

• Introduction to fitting a line by least squares regression

  • In this section, we use least squares regression as a more rigorous approach.

• Finding the least squares line

  • For the Elmhurst data, we could write the equation of the least squares regression line as
  • The slope of the least squares line can be estimated by
  • A common exercise to become more familiar with foundations of least squares regression is to use basic summary statistics and point-slope form to produce the least squares line.
  • These lines should intersect on the least squares line.
  • Summary of least squares fit for the Elmhurst data.

• Types of outliers in linear regression

  • These points are especially important because they can have a strong influence on the least squares line.
  • There are six plots shown in Figure 7.19 along with the least squares line and residual plots.
  • For each scatterplot and residual plot pair, identify any obvious outliers and note how they influence the least squares line.
  • In these cases, the outliers influenced the slope of the least squares lines.
  • Six plots, each with a least squares line and residual plot.

• A Graph of Averages

  • A graph of averages and the least-square regression line are both good ways to summarize the data in a scatterplot.
  • The most common method of doing this is called the "least-squares" method.
  • The least-squares regression line is of the form $\hat{y} = a+bx$, with slope $b = \frac{rs_y}{s_x}$ ($r$ is the correlation coefficient, $s_y$ and $s_x$ are the standard deviations of $y$ and $x$).
  • The points on a graph of averages do not usually line up in a straight line, making it different from the least-squares regression line.
  • The graph of averages plots a typical $y$ value in each interval: some of the points fall above the least-squares regression line, and some of the points fall below that line.

• Using R2 to describe the strength of a fit

  • However, it is more common to explain the strength of a linear fit using R2, called R-squared.
  • The R2 of a linear model describes the amount of variation in the response that is explained by the least squares line.
  • However, if we apply our least squares line, then this model reduces our uncertainty in predicting aid using a student's family income.
  • This corresponds exactly to the R-squared value:
  • Gift aid and family income for a random sample of 50 freshman students from Elmhurst College, shown with the least squares regression line.

• Finding a p-value for a chi-square distribution

  • This is where the chi-square distribution becomes useful.
  • Each expected count must be at least 5.
  • If each expected count is at least 5 and the null hypothesis is true, then the test statistic below follows a chi-square distribution with k − 1 degrees of freedom:
  • Sample size / distribution: Each particular scenario (i.e. cell count) must have at least 5 expected cases.
  • Degrees of freedom: We only apply the chi-square technique when the table is associated with a chi-square distribution with 2 or more degrees of freedom.