Why is validation accuracy stuck at 33% w/ relu activation and 100% at first epoch w/ softmax?

TLDR: When my tensorflow model’s last dense layer uses relu activation, val_accuracy is stuck at about 0.3396 (from epoch 1-10). When I modify the last dense layer’s activation to softmax, val_Accuracy is 1.00 starting at the first epoch. Why is val_accuracy so high at the first epoch? My model incorrectly predicts all new images.

My attempt to fix incorrectly high val_accuracy: I’ve ensured that each class has the same amount of data. (Each has about 500 images). I know overfitting can be a problem. I have added a dropout layer to try to resolve this.

I would appreciate your help.

This is my model:

def model():
  model_input = tf.keras.layers.Input(shape=(h, w, 3)) 
  x = tf.keras.layers.Rescaling(rescale_factor)(model_input) 
  x = tf.keras.layers.Conv2D(16, 3, activation='relu',padding='same')(x)
  x = tf.keras.layers.Dropout(.5)(x)
  x = tf.keras.layers.MaxPooling2D()(x) 
  x = tf.keras.layers.Flatten()(x)
  x = tf.keras.layers.Dense(128, activation='relu')(x)
  outputs = tf.keras.layers.Dense(num_classes, activation = 'relu')(x)

These are my training results. Why is val_accuracy stuck at 0.3396?

Epoch 1/10
41/41 [==============================] - 3s 62ms/step - loss: 6.1832 - accuracy: 0.3310 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 2/10
41/41 [==============================] - 2s 56ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 3/10
41/41 [==============================] - 2s 57ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 4/10
41/41 [==============================] - 2s 57ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 5/10
41/41 [==============================] - 2s 57ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 6/10
41/41 [==============================] - 2s 57ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 7/10
41/41 [==============================] - 2s 57ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 8/10
41/41 [==============================] - 2s 57ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 9/10
41/41 [==============================] - 2s 58ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396
Epoch 10/10
41/41 [==============================] - 2s 57ms/step - loss: 6.1537 - accuracy: 0.3326 - val_loss: 5.9654 - val_accuracy: 0.3396

When I change the last line to outputs = tf.keras.layers.Dense(num_classes, activation = 'softmax')(x) my training results are this. Why is val_accuracy different for softmax v. rely and why is it 1.000 val accuracy at first epoch?

41/41 [==============================] - 3s 47ms/step - loss: 0.1347 - accuracy: 0.9735 - val_loss: 1.2451e-05 - val_accuracy: 1.0000
Epoch 2/10
41/41 [==============================] - 2s 43ms/step - loss: 2.1454e-06 - accuracy: 1.0000 - val_loss: 1.5783e-06 - val_accuracy: 1.0000
Epoch 3/10
41/41 [==============================] - 2s 43ms/step - loss: 1.2228e-06 - accuracy: 1.0000 - val_loss: 1.2556e-06 - val_accuracy: 1.0000
Epoch 4/10
41/41 [==============================] - 2s 43ms/step - loss: 1.0604e-06 - accuracy: 1.0000 - val_loss: 1.1204e-06 - val_accuracy: 1.0000
Epoch 5/10
41/41 [==============================] - 2s 44ms/step - loss: 9.8337e-07 - accuracy: 1.0000 - val_loss: 1.0190e-06 - val_accuracy: 1.0000
Epoch 6/10
41/41 [==============================] - 2s 43ms/step - loss: 9.1662e-07 - accuracy: 1.0000 - val_loss: 9.6480e-07 - val_accuracy: 1.0000
Epoch 7/10
41/41 [==============================] - 2s 43ms/step - loss: 8.4151e-07 - accuracy: 1.0000 - val_loss: 8.8682e-07 - val_accuracy: 1.0000
Epoch 8/10
41/41 [==============================] - 2s 43ms/step - loss: 7.7699e-07 - accuracy: 1.0000 - val_loss: 8.3223e-07 - val_accuracy: 1.0000
Epoch 9/10
41/41 [==============================] - 2s 43ms/step - loss: 7.1349e-07 - accuracy: 1.0000 - val_loss: 7.2788e-07 - val_accuracy: 1.0000
Epoch 10/10
41/41 [==============================] - 2s 43ms/step - loss: 6.6057e-07 - accuracy: 1.0000 - val_loss: 6.6252e-07 - val_accuracy: 1.0000

Edit: Model.fit and model.compile parameters:

model = tf.keras.Model(model_input, outputs)
  
 model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])
  
 
hist = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=10
)

from Recent Questions - Stack Overflow https://ift.tt/3E368PK
https://ift.tt/eA8V8J