First, Happy New Year!
In my job, I often have to explain why a piece of machinery behaves differently than the engineering equations used to describe the behavior. Usually, it becomes a problem of real world behavior and human recording error. When you are trying to read a dial that is vibrating between several readings in an extremely noisy and occasionally scarey environment, what gets written down is a best estimate of what is true, while trying to escape the environment in one piece. Sometimes a bias is built into the reading by the standard operating procedure. (e.g. Take the reading at the maximum value during the test, or take the reading of the maximum and minimum of the vibrating dial and divide by two, etc.) In general, you have to be careful when fitting real world data for these types of reasons.
I recently modified a physical model to fit reality, and came across an unusual (to me) situation. I generally look at two values to determine goodness of fit. I like to use the root mean square error (RMS) and Pearson's coefficient of correlation (r2). However, in this case, I had two similar models to compare, and I had successfully modified both to fit machine behavior. One model gave me a very good RMS value, and the other had a better r2. I was surprised by this, because these two values generally give similar indications. I decided to go with the model that gave the better RMS value. However, this whole goodness of fit exercise caused me to waste spend a lot of time on "goodness of fit" searches.
I came across this simple, but practical paper and spreadsheet: "Goodness of Fit Metrics in Comparing Models to Data"
The spreadsheet introduced me to weighting of data in a model. In examining the spreadsheet the author introduces weighting factors (the Data SE column), but makes no mention in his paper or the spreadsheet how one goes about determining the weighting factors. It's definitely peaked my interest, but I need to understand the concepts of weighting data (like why would you do this?) If anyone has any references on goodness of fit and using weighted values for goodness of fit calculations, I'd appreciate it if you could leave me a comment.