Add transpose convolution layers #315

juliabeliaeva · 2021-12-20T22:32:56Z

This PR adds implementations for Conv1DTranspose, Conv2DTranspose and Conv3DTranspose layers.

Caveats:

Dilation values greater than 1 are not supported in tensorflow on cpu: CPU support for dilation rates larger than 1 tensorflow/tensorflow#28264. Therefore, I have not added any tests for dilations as they do not work.
Output padding is not supported in org.tensorflow.op.NnOps#conv3dBackpropInput. Conv3DTranspose still has the parameter, it just does not do anything. I'm not sure if this parameter is needed or if it is, where to add a warning about its usage.

Fixes #124

zaleslaw · 2021-12-27T07:38:49Z

I suggest to be an honest library for a user and remove unsupported parameters by adding to the KDoc the NOTE about this removal (with a link to the known bug).

But it could be a problem during the model convertation Keras->KotlinDL and KotlinDL->Keras and in the case
Keras->KotlinDL we could just skip values and the case KotlinDL->Keras write some default values from Keras layers contructors

zaleslaw · 2021-12-27T08:30:15Z

@juliabeliaeva I made an integration example with auto-encoder based on the following article https://www.machinecurve.com/index.php/2019/12/10/conv2dtranspose-using-2d-transposed-convolutions-with-keras/

The Colab is available here, you could a copy of it and download the JSON config and h5 weights put it to examples/resources, and run the following example

fun loadAutoencoderModelWithWeightsAndEvaluate() {
    val (_, test) = fashionMnist()

    val jsonConfigFile = getAutoencoderJSONConfigFile()
    val model = Sequential.loadModelConfiguration(jsonConfigFile)

    model.use {
        for(layer in it.layers) {
            layer.isTrainable = false
        }
        it.compile(
            optimizer = Adam(),
            loss = Losses.MAE,
            metric = Metrics.ACCURACY
        )

        it.logSummary()

        val hdfFile = getAutoencoderWeightsFile()
        it.loadWeights(hdfFile)

        val result = it.predict(test.x[0])
    }
}

/** */
fun main(): Unit = loadAutoencoderModelWithWeightsAndEvaluate()

/** Returns JSON file with model configuration, saved from Keras 2.x. */
fun getAutoencoderJSONConfigFile(): File {
    val pathToConfig = "models/mnist/autoencoder/modelConfig.json"
    val realPathToConfig = OnHeapDataset::class.java.classLoader.getResource(pathToConfig).path.toString()

    return File(realPathToConfig)
}

/** Returns .h5 file with model weights, saved from Keras 2.x. */
fun getAutoencoderWeightsFile(): HdfFile {
    val pathToWeights = "models/mnist/autoencoder/weights.h5"
    val realPathToWeights = OnHeapDataset::class.java.classLoader.getResource(pathToWeights).path.toString()
    val file = File(realPathToWeights)
    return HdfFile(file)
}

It fails with the following error message

Exception in thread "main" java.lang.IllegalStateException: Attempting to use uninitialized value Exception in thread "main" java.lang.IllegalStateException: Attempting to use uninitialized value conv2d_transpose_6_conv2d_transpose_kernel
	 [[{{node Conv2dBackpropInput}}]]

Looks like something was missed during the loading of the weights or initialization of the variables.

To reproduce the bug, download the following archive (json + h5)
weights.zip

juliabeliaeva · 2021-12-27T11:21:09Z

@zaleslaw thank you, I forgot about loading weights from h5 files completely. I'll implement it.

juliabeliaeva · 2021-12-27T11:22:51Z

But it could be a problem during the model convertation Keras->KotlinDL and KotlinDL->Keras and in the case
Keras->KotlinDL we could just skip values and the case KotlinDL->Keras write some default values from Keras layers contructors

That was my concern: if someone loads a model from keras and saves it back, they will loose their parameters in the process.

juliabeliaeva · 2022-01-18T15:46:01Z

@zaleslaw I just pushed a new version of this PR. Apart from the added loading weights for transposed convolutions, there are some other notable changes:

I moved expansion of kernel, strides and dilations directly to org.jetbrains.kotlinx.dl.api.core.layer.convolutional.Conv1D#convImplementation instead of doing it in constructor. One of the side effects of this change is that kernel variable has a correct shape now and weights could be loaded for it without problems.
The changes in Conv1D allowed to get rid of the duplicated sets of properties ("internal" properties in AbstractConv vs non-internal properties in the implementations). Having two sets of properties was really confusing, it was hard to figure out which to use as they were sometimes the same and sometimes different. This change makes convolutional layer hierarchy much simpler.
I removed outputPadding from Conv3DTranspose as suggested. I kept the dilations since they should on gpu.
It turned out that conv2dBackpropInput/conv3dBackpropInput need to have a specific batch size, so I had to add a hack for it (I would appreciate any suggestions on how to make this simpler). See: https://github.com/JetBrains/KotlinDL/blob/1d5492e5ef51b2d7eb7066e35d8a82934ca390e2/api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/ConvTranspose.kt#L86
After these changes the "Autoencoder" example still does not work completely: org.jetbrains.kotlinx.dl.api.core.GraphTrainableModel#internalPredict function does not expect to have an image as a prediction. I'm going to send a separate PR with the fix.

zaleslaw · 2022-01-24T15:43:12Z

@juliabeliaeva Is this the final version of this PR or is it required any PR should be merged before?

juliabeliaeva · 2022-01-24T16:38:32Z

@juliabeliaeva Is this the final version of this PR or is it required any PR should be merged before?

It's the final version, can be merged right now.

zaleslaw

Please have a look at the left comments and add the general example related to the autoencoder based on the Colab shared earlier in the issue comments.
No big changes, just a few minor things for future developers.

At this moment #328 is merged to the master, you could merge in this PR and finish the Autoencoder example

zaleslaw · 2022-01-31T08:37:51Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/Conv1D.kt

@@ -57,42 +56,27 @@ private const val EXTRA_DIM = 1L
 * @since 0.3
 */
 public class Conv1D(
-    public val filters: Int = 32,
-    public val kernelSize: Int = 3,


Looks like kernel size is a stable term in Deep Learning for CNN

We can't use kernelSize here since kernelSize is an array and we want to allow passing a single integer here.

zaleslaw · 2022-01-31T08:40:54Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/Conv1D.kt

    }

    override fun toString(): String =
        "Conv1D(filters=$filters, kernelSize=$kernelSize, strides=${strides.contentToString()}, " +
                "dilation=${dilations.contentToString()}, activation=$activation, kernelInitializer=$kernelInitializer, " +
                "biasInitializer=$biasInitializer, kernelShape=${kernel.shape}, biasShape=${bias?.shape}, padding=$padding, " +
                "biasRegularizer=$biasRegularizer, kernelRegularizer=$kernelRegularizer, activityRegularizer=$activityRegularizer)"
+
+    internal companion object {


Should these functions be companion functions or could be just private functions in the Conv1D?

These functions are used in both Conv1D and Conv1DTranspose since 1D convolutions are implemented with 2D convolutions. They can't be private, but they are only needed for convolutions implementation, so they are internal.

zaleslaw · 2022-01-31T08:41:33Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/Conv1D.kt

+            return kernel.withAdded(EXTRA_DIM - 1, 1)
+        }
+
+        internal fun Ops.expandKernel(kernel: Operand<Float>): Operand<Float> {


Is it Conv1D specific functions, could they be a part of our local helper framework over the TensorFlow Java API

These functions are specific for Conv1D/Conv1DTranspose, I don't think they are needed outside of these operations.

zaleslaw · 2022-01-31T08:41:53Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/Conv1DTranspose.kt

+ */
+public class Conv1DTranspose(
+    public override val filters: Int = 3,
+    public val kernelLength: Int = 3,


Kernel Size is better

zaleslaw · 2022-01-31T08:44:14Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/Conv1DTranspose.kt

+    override fun convImplementation(tf: Ops, input: Operand<Float>): Operand<Float> {
+        return tf.withExpandedDimensions(input) { expandedInput ->
+            val expandedOutputPadding = outputPadding?.withAdded(EXTRA_DIM * 2, listOf(0, 0))
+            return@withExpandedDimensions tf.nn.conv2dBackpropInput(


very complex function call, could we extract some logic to the separate variables and leave some comments for the future (why, for example, we did tf.shapeWithDynamicBatchSize(outputShape.expand(), input) for the specific parameter and so on) At this moment this knowledge is hidden in the issue comments.
I suggest doing this, because it's enough far from the initial Keras code

why, for example, we did tf.shapeWithDynamicBatchSize(outputShape.expand(), input)

shapeWithDynamicBatchSize has a javadoc with explanation and links to the relevant issues, I'll add some comments to this method as well.

zaleslaw · 2022-01-31T08:47:01Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/Conv2DTranspose.kt

+        )
+    }
+
+    internal companion object {


Methods of this companion are used in the both, Conv1DTranspose and Conv2DTranspose, probably need we need a separate place with top-level functions a la Util class.

zaleslaw · 2022-01-31T08:48:11Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/Conv3DTranspose.kt

+ * C - number of channels
+ * ```
+ *
+ * Note: providing explicit output padding is currently not supported.


Should we throw an exception on that or add a contract to init section?

There is no outputPadding parameter, so there is no situation where we could throw an exception. The note is here since keras does have this parameter (see https://keras.io/api/layers/convolution_layers/convolution3d_transpose/).

zaleslaw · 2022-01-31T08:49:02Z

api/src/main/kotlin/org/jetbrains/kotlinx/dl/api/core/layer/convolutional/ConvTranspose.kt

+    }
+
+    internal companion object {
+        /**


Could we move here other companion functions mentioned earlier?

zaleslaw · 2022-01-31T08:51:48Z

examples/src/main/kotlin/examples/cnn/fsdd/SoundNet.kt

@@ -41,7 +41,7 @@ internal fun soundBlock(filters: Int, kernelSize: Int, poolStride: Int): Array<L
    arrayOf(
        Conv1D(
            filters = filters,
-            kernelSize = kernelSize,
+            kernelLength = kernelSize,


kernelSize!

…olution to the convImplementation method

juliabeliaeva · 2022-02-17T12:59:42Z

Pushed a new version of this PR.

Rebased on master, resolved conflicts.
Rewritten Conv1DTranspose implementation to be more clear.
Moved helper methods.
Discovered that my implementation computes the output size a bit differently than keras, so I rewritten it to match (see the last commit in the branch).

I think it might be more convenient to implement the example as another PR.

juliabeliaeva mentioned this pull request Dec 28, 2021

Simplify loading layer weights support for new layers #318

Merged

zaleslaw added the Review This PR is under review label Jan 12, 2022

juliabeliaeva force-pushed the conv-transpose branch from 62281e3 to 1d5492e Compare January 18, 2022 15:25

juliabeliaeva mentioned this pull request Jan 19, 2022

GraphTrainableModel#internalPredict works only with one-dimensional predictions #327

Closed

zaleslaw requested changes Jan 31, 2022

View reviewed changes

juliabeliaeva added 8 commits February 16, 2022 18:27

Simplify loading parameters for convolutional and pooling layers

0a16dcb

Implement toString in L2L1 regularizer

c3af0e0

Move dimension expansion for kernel, strides and dilations in 1D conv…

9bf1f9a

…olution to the convImplementation method

Do not duplicate internal properties of the AbstractConv implementations

947279d

Add transposed convolution layers

5132291

Move helper functions to ConvTranspose

050d80e

Improve Conv1DTranspose implementation readability

b19a00c

Compute transposed convolution output size the same way as keras does

0a8a32d

juliabeliaeva force-pushed the conv-transpose branch from 1d5492e to 0a8a32d Compare February 17, 2022 12:43

zaleslaw added LGTM PR reviewed and is ready to merge and removed Review This PR is under review labels Feb 18, 2022

zaleslaw merged commit 016450c into Kotlin:master Feb 18, 2022

juliabeliaeva deleted the conv-transpose branch April 15, 2022 23:53

juliabeliaeva mentioned this pull request Jan 10, 2023

[WiP] Add ConvTranspose Layers #150

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add transpose convolution layers #315

Add transpose convolution layers #315

juliabeliaeva commented Dec 20, 2021

zaleslaw commented Dec 27, 2021 •

edited

Loading

zaleslaw commented Dec 27, 2021 •

edited

Loading

juliabeliaeva commented Dec 27, 2021

juliabeliaeva commented Dec 27, 2021

juliabeliaeva commented Jan 18, 2022

zaleslaw commented Jan 24, 2022

juliabeliaeva commented Jan 24, 2022

zaleslaw left a comment •

edited

Loading

zaleslaw Jan 31, 2022

juliabeliaeva Feb 16, 2022

zaleslaw Jan 31, 2022

juliabeliaeva Feb 16, 2022 •

edited

Loading

zaleslaw Jan 31, 2022

juliabeliaeva Feb 16, 2022 •

edited

Loading

zaleslaw Jan 31, 2022

zaleslaw Jan 31, 2022

juliabeliaeva Feb 16, 2022

zaleslaw Jan 31, 2022

zaleslaw Jan 31, 2022

juliabeliaeva Feb 16, 2022

zaleslaw Jan 31, 2022

zaleslaw Jan 31, 2022

juliabeliaeva commented Feb 17, 2022 •

edited

Loading

Add transpose convolution layers #315

Add transpose convolution layers #315

Conversation

juliabeliaeva commented Dec 20, 2021

zaleslaw commented Dec 27, 2021 • edited Loading

zaleslaw commented Dec 27, 2021 • edited Loading

juliabeliaeva commented Dec 27, 2021

juliabeliaeva commented Dec 27, 2021

juliabeliaeva commented Jan 18, 2022

zaleslaw commented Jan 24, 2022

juliabeliaeva commented Jan 24, 2022

zaleslaw left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliabeliaeva Feb 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliabeliaeva Feb 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliabeliaeva commented Feb 17, 2022 • edited Loading

zaleslaw commented Dec 27, 2021 •

edited

Loading

zaleslaw commented Dec 27, 2021 •

edited

Loading

zaleslaw left a comment •

edited

Loading

juliabeliaeva Feb 16, 2022 •

edited

Loading

juliabeliaeva Feb 16, 2022 •

edited

Loading

juliabeliaeva commented Feb 17, 2022 •

edited

Loading