2022-05-14

tensorflow automatic accuracy calculation for multilabel classifier

I am fitting a multilabel classifier to (train_x, train_y) while monitoring the loss and accuracy on a validation set (val_x, val_y):

classification_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0002),
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])
classification_model.fit(train_x, train_y, validation_data=(val_x, val_y), \
        epochs=10,
        batch_size=10
)

This gives the following output:

Epoch 1/10
50/50 [==============================] - ETA: 0s - loss: 0.1186 - accuracy: 0.7094
Epoch 1: val_loss improved from 0.15329 to 0.11998, saving model to best_classification_model.tf
50/50 [==============================] - 12s 186ms/step - loss: 0.1186 - accuracy: 0.7094 - val_loss: 0.1200 - val_accuracy: 0.6280
Epoch 2/10
50/50 [==============================] - ETA: 0s - loss: 0.0848 - accuracy: 0.7776
Epoch 2: val_loss improved from 0.11998 to 0.10281, saving model to best_classification_model.tf
50/50 [==============================] - 8s 167ms/step - loss: 0.0848 - accuracy: 0.7776 - val_loss: 0.1028 - val_accuracy: 0.7200
Epoch 3/10
50/50 [==============================] - ETA: 0s - loss: 0.0652 - accuracy: 0.8176
Epoch 3: val_loss improved from 0.10281 to 0.09259, saving model to best_classification_model.tf
50/50 [==============================] - 10s 202ms/step - loss: 0.0652 - accuracy: 0.8176 - val_loss: 0.0926 - val_accuracy: 0.7560
Epoch 4/10
50/50 [==============================] - ETA: 0s - loss: 0.0522 - accuracy: 0.8236
Epoch 4: val_loss improved from 0.09259 to 0.08710, saving model to best_classification_model.tf
50/50 [==============================] - 10s 206ms/step - loss: 0.0522 - accuracy: 0.8236 - val_loss: 0.0871 - val_accuracy: 0.7480
Epoch 5/10
50/50 [==============================] - ETA: 0s - loss: 0.0418 - accuracy: 0.8337
Epoch 5: val_loss improved from 0.08710 to 0.08441, saving model to best_classification_model.tf
50/50 [==============================] - 10s 209ms/step - loss: 0.0418 - accuracy: 0.8337 - val_loss: 0.0844 - val_accuracy: 0.7640

I am wondering how this accuracy is actually calculated. Does it count the total number of correct labels, or the total number of rows for which all labels? And what is a 'correct label'? Is (internally) the maximum taken per output row?

To clarify what I mean with each option:
The total number of correct labels: for each image, 20 labels are outputted of which some are 0 and some are 1. Report the total number of correct labels (= number of correct 0s + number of correct 1s) and divide it by the total number of labels (= 20*num_images). I don’t think this is what happens, as that would probably lead to a higher accuracy. Just predicting 0's everywhere would already give an accuracy of over 90%! And that is not what happens, even after training for a longer time.
The total number of rows for which all labels are correct: count the number of images for which all labels are correct (0s and 1s) and divide by the number of images

The model output and the validation labels look as follows

>>> classification_model.predict(val_x)    # shape: (250, 20)
array([[ -9.385,  -5.443,  -8.274, ...,   1.936, -11.607,  -1.867],
       [-10.523,   3.074,  -7.765, ...,  -2.925, -10.35 ,  -2.602],
       [ -7.872,  -7.525,  -4.877, ...,  -6.434,  -9.063,  -8.485],
       ...,
       [ -6.04 ,  -4.826,   3.537, ...,  -5.68 ,  -7.276,  -6.05 ],
       [ -5.734,  -6.399,  -5.288, ...,  -5.495,  -6.673,   0.06 ],
       [ -9.458,  -7.892,   1.391, ...,  -6.422,  -9.75 ,  -7.702]],
      dtype=float32)
>>> val_y    # also shape: (250,20)
array([[0., 0., 0., ..., 0., 0., 1.],
       [0., 1., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 1., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 1.],
       [0., 0., 1., ..., 0., 0., 0.]])


No comments:

Post a Comment