The proposed model was tested on machine translation tasks of IWSLT2016 DE-EN, WMT17 EN-DE, and WMT17 EN-CS. Therefore, this paper proposes the interacting-head attention mechanism, which induces deeper and wider interactions across the attention heads by low-dimension computations in different subspaces of all the tokens, and chooses the appropriate number of heads to avoid low-rank bottleneck. However, these methods are challenged by the variable sequence length in the corpus and the sheer number of parameters to be learned. To address the low-rank bottleneck, the two mainstream methods make the head size equal to the sequence length and complicate the distribution of self-attention heads. On the other hand, the low-rank bottleneck may occur, when the number of heads surpasses a threshold. On the one hand, the computations of each head attention are conducted in the same subspace, without considering the different subspaces of all the tokens. But this is not always the case in practice. In theory, more heads enhance the expressive power of the NMT model. The main component of the transformer is the multihead attention layer. Transformer-based models have gained significant advances in neural machine translation (NMT).

Kannada to english pdf converter