Parametrization strategy can be found in paper:
UserAdapter: Few-Shot User Learning in Sentiment Analysis
Prefix-Tuning: Optimizing Continuous Prompts for Generation
It can make the optimization more stable and improve the efficiency when file-tuning a model.
What is parametrization strategy?
Usually, we can train user-specific vector, product-specific vector, position-specific vector et al in a model. For example: we can add user-specific vector directly in a attention layer to get user-specific attention score. However, this direct way may lead to unstable optimization and a slight drop in performance. So, we can use parametrization strategy.
parametrization strategy can be viewed as:
\(P_\theta=MLP(P_\theta^`)\)
Here we will use MLP layer to map our specific vector.
For example: if \(P_\theta\) is 1*768, we can use a 768*768 matrix to map it.