Design for NAS model constructor

[andreyvelich]

/kind feature

Currently, for ENAS we have model constructor inside training container. I think, we can create Model somewhere else. In that case, user doesn't need to create model constructor inside training container.

Few thoughts:

1. As @gaocegege mentioned here: [discussion] Support Advanced NAS Algorithms #461, we can use ONNX representation and create mode inside Suggestion. In that case, we can extend GetSuggestionsReply with additional field: bytes onnx_model. Or convert bytes to string, if it is possible.

2. We can use JSON representation for the model (https://www.tensorflow.org/guide/keras/save_and_serialize#architecture-only_saving), but it will not save the weights. For enas it doesn't matter, but for other algorithms, maybe, we have to send model to Trials with predefined weights.

3. I think we should create model inside Suggestion, since Trial is just a training job. For NAS algorithms user wants to receive the model in the training job. Also, for example darts algorithm mades some modifications with current running model during the training.

/cc @johnugeorge @gaocegege

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.72

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

/area nas

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/lifecycle frozen