Model experiment notes

[ebgoldstein]

This is a place to discuss model experiments and architectures

[somyamohanty]

Ok, I was finally able to commit with some brief cleanup. Some lessons learnt.

• Multi-output does not seem to work as well as I initially thought. MAPE of Skew is really bad, but distribution and kurt are fine.
• Sequential model without batch norm and with more drop out is better. Small networks works pretty well.
• Residual connections are not that beneficial.
• There seems to be a lot of variation between the kernel size and the sparsification. For example, keeping kernel size at 3 for input and 6 for later layers, with the general model we see MAPE stabilizing at 38-40, but with sparse networks this drops down to 27-28. This probably indicates we can generalize the network even further. Maybe even have regularization (l1,l2) to get some accuracy benefits.
• Quantization with float 16 bit reduces the size but MAPE increases to 40. We can potentially look at training aware quant. https://www.tensorflow.org/model_optimization/guide/quantization/training. I think I had done this before with dune collision, but didnt see any benefits. Maybe a regression model can benefit.

[ebgoldstein]

some ideas for the future:

• AdamW - https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW
• cosine decay w/ restarts - https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/CosineDecayRestarts

[ebgoldstein]

Enforce monotonicity

[ebgoldstein]

I am not sure how to use TF lattice (which enforces monotonicity), but monotonicity could potentially be enforced with a custom loss that penalized when something is non-monotonic...

[ebgoldstein]

the other option is to just swap values/sort if monotonicity does not occur..

[ebgoldstein]

see #136 for the np.sort() solution

[ebgoldstein]

I have tried to pretrain a network using the SediNet dataset. (https://github.com/MARDAScience/SediNet).
the steps to do this are :

• use imagemagick to make sure all sedinet pictures are cropped or padded to 1024x1024px
• construct the model, with regression head that has 9 classes, based on Sedinet data, which has 9 values from the continuous distribution of grain size, but in pixel dimensions
• train model
• freeze the base model
• replace last dense layers with new ones that have 11 outputs for SandCam dataset
• retrain with instagrain data (i.e., retrain just the dense layers to preserve features learned in the conv blocks)
• (optional) unfreeze basemodel (conv blocks) and train whole model w/ very low lr...
• continue on with Quantization Aware Training and TF Lite workflow

the results look similar to training from scratch... but this might be something to revisit in the future..

[ebgoldstein]

Some way to leverage grain size pictures... maybe pretrain w/ an autoencoder ( SandSnap, sedinet + instagrain)?