【NLP】Transformer自注意机制代码实现
机器学习初学者
共 3908字,需浏览 8分钟
·
2021-08-10 02:56
编译 | VK
来源 | Towards Data Science
![](https://filescdn.proginn.com/8a99a353ed22ffc74aa73c86dfc2128d/e83e75be8aebba69d18a33e594a46cc2.webp)
Transformers — Yo_just need Attention(https://machinelearningmarvel.in/transformers-you-just-need-attention/) Intuitive Maths and Code behind Self-Attention Mechanism of Transformers(https://machinelearningmarvel.in/intuitive-maths-and-code-behind-self-attention-mechanism-of-transformers-for-dummies/) Concepts about Positional Encoding Yo_Might Not Know About(https://machinelearningmarvel.in/concepts-about-positional-encoding-you-might-not-know-about/)
![](https://filescdn.proginn.com/a1dd915afeeab3ccf532ad139fd2f23b/4a32678be2e61e83f360e25907bb0e9a.webp)
注意机制概念 自注意机制的步骤(直觉数学理论和代码)
输入预处理 查询、键和值矩阵 注意分数的概念
多头自注意机制
注意机制概念
![](https://filescdn.proginn.com/2d2bc00e29dc7d26b87ca15f93d9cff2/b9a9018e8789d29c454320b8715618ef.webp)
自注意机制的步骤
1.输入正确的格式
print(f”Shape is :- {np.random.randn(3,5).shape}”)
X=np.random.randn(3,5)
X
![](https://filescdn.proginn.com/296bb212f5b7a4143bdb8748806c106e/8c36154ebc0b22eea81a2f4941aad199.webp)
2.获取查询、键值矩阵
![](https://filescdn.proginn.com/13bf5d6deda42ad7c9f397489d438c9e/b0dbb535658c86da4ebcfdff73962f66.webp)
weight_of_query=np.random.randn(5,3)
weight_of_query
![](https://filescdn.proginn.com/13cd40a77f7eabc3bbb079d53f2b8834/c17a5bb1d8bf63fd95ac42a4df193c3b.webp)
weight_of_key=np.random.randn(5,3)
weight_of_key
![](https://filescdn.proginn.com/4bc008d6bd6836b9e4ade403f7f03535/131d809996527194858728c03e894f38.webp)
weight_of_values=np.random.randn(5,3)
weight_of_values
![](https://filescdn.proginn.com/9215314bab0f43a1ed1d8454799005cc/04bc82ec5fa7e0f051659b5f7fd2869e.webp)
Key=np.matmul(X,weight_of_key)
Key
![](https://filescdn.proginn.com/d2f6bc5f5b422ba7b44b69cbae09c9f4/a0aff9a06b39b4b224fc2f57bd267085.webp)
Query=np.matmul(X,weight_of_query)
Query
![](https://filescdn.proginn.com/6bbdb2ef9a11364b11b8556ad8265b29/0919c183dbe0fa8826b00211982f116b.webp)
Values=np.matmul(X,weight_of_values)
Values
![](https://filescdn.proginn.com/e7ecfd3ef42f61989625bd0190f2e77c/e37915861cac236fefd0241e5617be92.webp)
3.注意得分
![](https://filescdn.proginn.com/2b478b7c00f95dd7cac3e427d4ae3a71/0aadadcdf3e740dad3d9407aaedbb76c.webp)
dimension=5
Scores=np.matmul(Query,Key.T)/np.sqrt(dimension)
Scores
![](https://filescdn.proginn.com/fb5dec4675a4a2f906aae9bcd7a0625f/2f612d9566762de0aea2d08e95a4fca2.webp)
![](https://filescdn.proginn.com/ba9099247a11420b8b5ad3425118bf4d/18572b82e6f4eb1d2f4f55e1c6076947.webp)
from scipy.special import softmax
Softmax_attention_scores=np.array([softmax(x) for x in Scores])
Softmax_attention_scores
![](https://filescdn.proginn.com/e96774a4ff24723ffc2a2ead97fb677a/d51763552a880e017d8db79ba611d9eb.webp)
Softmax_attention_scores[0][0]*Values[0]+\
Softmax_attention_scores[0][1]*Values[1]+\
Softmax_attention_scores[0][2]*Values[2]
![](https://filescdn.proginn.com/5ceb61b673b6d10b7644f7dbc3cb0e60/496b5f6f3557c22b4e5da0dbd340a55d.webp)
多头自注意机制
Transformers — Yo_just need Attention(https://machinelearningmarvel.in/transformers-you-just-need-attention/) Intuitive Maths and Code behind Self-Attention Mechanism of Transformers(https://machinelearningmarvel.in/intuitive-maths-and-code-behind-self-attention-mechanism-of-transformers-for-dummies/) Concepts about Positional Encoding Yo_Might Not Know About(https://machinelearningmarvel.in/concepts-about-positional-encoding-you-might-not-know-about/)
往期精彩回顾 本站qq群851320808,加入微信群请扫码:
评论