Share AI Models and Methods

Understand Parametrization Strategy in Deep Learning

Parametrization strategy can be found in paper:

UserAdapter: Few-Shot User Learning in Sentiment Analysis

Prefix-Tuning: Optimizing Continuous Prompts for Generation

It can make the optimization more stable and improve the efficiency when file-tuning a model.

What is parametrization strategy?

Usually, we can train user-specific vector, product-specific vector, position-specific vector et al in a model. For example: we can add user-specific vector directly in a attention layer to get user-specific attention score. However, this direct way may lead to unstable optimization and a slight drop in performance. So, we can use parametrization strategy.

parametrization strategy can be viewed as:


Here we will use MLP layer to map our specific vector.

For example: if \(P_\theta\) is 1*768, we can use a 768*768 matrix to map it.

Leave a Reply

Your email address will not be published. Required fields are marked *