New embedding layers

[soran-ghaderi]

A list of new embedding layers to be added

To contribute, please:

1. Create a new issue, copy an available (not closed and not opened by someone else) subtask name and paste it in the title followed by the reference of the task. (ex. subtask_name #source_issue_number). Make sure the subtask is not already opened by someone else. Another way is to hover over the subtasks and click on the Open convert to issue
2. Copy and paste your new issue link here in the comments.
3. Fork the repository
4. Add your changes
5. Create a pull request and mention your issue link

Embedding:

• TokenEmbedding

Positional embedding:

• Rename PositionalEncoding to AbsolutePositionalEmbedding #47
• Fixed Positional Embedding
• Relative Positional Embedding
• Dynamic Position Bias
• Alibi Positional Bias https://arxiv.org/pdf/2108.12409v2.pdf
• Learned Alibi PositionalBias
• Rotary Embedding
• Conditional Positional Encoding

Please refer to the following useful links:

• https://paperswithcode.com/methods/category/position-embeddings