Regression analysis is probably the most commonly used statistic in the social sciences. Regression is used to evaluate relationships between two or more feature attributes. Identifying and measuring relationships lets you better understand what's going on in a place, predict where something is likely to occur, or begin to examine causes of why things occur where they do.
Ordinary Least Squares (OLS) is the best known of all regression techniques. It is also the proper starting point for all spatial regression analyses. It provides a global model of the variable or process you are trying to understand or predict; it creates a single regression equation to represent that process.
There are a number of good resources to help you learn more about both OLS regression and Geographically Weighted Regression. Start by reading the Regression Analysis Basics documentation and/or watching the free one-hour ESRI Virtual Campus Regression Analysis Basics Web seminar. Next, work through a Regression Analysis tutorial. Once you begin creating your own regression models, you may want to refer to the Interpreting OLS Regression Results documentation to help you understand OLS output and diagnostics.
Additional resources
Mitchell, Andy. The ESRI Guide to GIS Analysis, Volume 2. ESRI Press, 2005.
Wooldridge, J. M. Introductory Econometrics: A Modern Approach. South-Western, Mason, Ohio, 2003.
Hamilton, Lawrence C. Regression with Graphics. Brooks/Cole, 1992.