Loss/Cost Function
The loss function refers to the error associated with a single training example in the context of machine learning. It quantifies how well the model's predictions match the actual target values for that specific instance.
The cost function refers to the average of the loss function over the entire training dataset. It provides a measure of the model's overall performance across all training examples. The cost function is what is minimized during the training process using optimization algorithms like gradient descent.
Mean Squared Error (MSE)
In regression tasks, a common loss function is the mean squared error (MSE). It measures the average squared difference between the predicted values generated by a model and the actual target values.
It is used because it penalizes larger errors (due to the squaring), making it sensitive to outliers. A lower MSE indicates a better fit of the model to the data, while an MSE of zero indicates perfect predictions.
The formula for MSE is:
\[ MSE = \frac{1}{n} \sum_{i=1}^{n}(y_{i} - \hat{y}_{i})^{2} \]
n := number of observations yi := actual target values \(\hat{y}_{i}\) := predicted values