Is there a way to introduce sparsity constraint on an autoencoder to achieve compressions in the Cosine/Fourier domain? I want to use the encoder part of the Auto encoder as the feature extractor from the STL-10 dataset. https://www.kaggle.com/residentmario/autoencoders The features learned should be more optimized than the compression algorithms like JPEG.

  • $\begingroup$ you're stating a conflicting demand: you want your encoder to produce sparse data, but you want that data to be highly compressed. That is in conflict; or did I misunderstand what is supposed to be sparse in your model? $\endgroup$ Commented Jul 31, 2021 at 8:35
  • $\begingroup$ I want the encoder to highly compress the data like in the JPEG and then using the data to decode back the original image. The weights learn should be similar to the DFT basis. $\endgroup$
    – Yvon
    Commented Jul 31, 2021 at 9:34
  • $\begingroup$ yeah. so, that's what you basically always do when designing an autoencoder, you've got data->encoder->penalty->decoder, where "penalty" is typically a low-dimensional vector, and potentially a function applied to that. That's the most classical autoencoder thing I can think of! $\endgroup$ Commented Jul 31, 2021 at 9:47
  • $\begingroup$ Ya. That is exactly what I want to do. I am trying to replicate the compressions with some learned basis like DFT/others. Do you have any good idea on enforcing the weights of the encoder to some good basis? $\endgroup$
    – Yvon
    Commented Jul 31, 2021 at 9:58
  • $\begingroup$ Do the machine learning thing and initialize them randomly. If that doesn't work, you can put expert knowledge in it. $\endgroup$ Commented Aug 2, 2021 at 12:50

1 Answer 1


When we train Auto Encoder we usually apply the loss function on the result obtained from the decoder and not the encoder.
Hence the obvious step, adding a sparsity promoting regularization to the loss function, won't do what we're after.

What you may do is adding to the loss function the $ {L}_{1} $ norm, or any other differential regularization which fits you model (Maybe the KL Divergence), based on the output of the encoder.

Adding this penalty will promote sparsity at the output of the encoder.

  • $\begingroup$ Have you ever implemented something like that? $\endgroup$
    – Mark
    Commented Dec 11, 2022 at 18:12
  • $\begingroup$ It shouldn't be hard to do. If you have separate backbones for the encoder and decoder, all needed is to hook the encoder output to the loss. $\endgroup$
    – Royi
    Commented Dec 12, 2022 at 6:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.