0% found this document useful (0 votes)
119 views38 pages

Attention in Deep Networks: Ishan Misra

Uploaded by

Zu Ki
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
119 views38 pages

Attention in Deep Networks: Ishan Misra

Uploaded by

Zu Ki
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 38

Attention in Deep

Networks

Ishan Misra
Fine-Grained Crowdsourcing for Fine-Grained Recognition - Deng et al., 2013
Fine-Grained Crowdsourcing for Fine-Grained Recognition - Deng et al., 2013
Self-supervised pipeline

Jigsaw/Colorization

Pre-train data ConvNet

4
"Standard" pipeline
Image Classification

Classes

ConvNet
Image

Machine Translation

Input (English): Output (French):


How are you? Comment vas-tu

RNN
How to "focus" on parts of input?
Image Classification

Classes

ConvNet
Image

Machine Translation

Input (English): Output (French):


How are you? Comment vas-tu

RNN
Attention
Image Classification

Focus on parts
Classes
of image
ConvNet
Image

Machine Translation
Focus on parts
of sentence Output (French):
Input (English):
How are you? Comment vas-tu
Attention
• Old idea in neuroscience computer vision
• Feature aggregation view from the 80s
• Recently shows great promise in NLP - machine translation and
sentence classification and vision + language

Shifts in Selective Visual Attention: Toward the Underlying Neural Circuitry - Koch, Ullman - 1987
A feature integration theory of attention - Treisman, Gelade - 1980
Perceptual organization and attention - Kahneman, Henik - 1981
VQA - Visual Question Answering
Question : What color is the hydrant? Question : Is the man skateboarding on a boardwalk?

Antol et al., 2015

Malinowski et al., 2014


VQA - Visual Question Answering
Question : What color is the hydrant? Question : Is the man skateboarding on a boardwalk?

Antol et al., 2015

Malinowski et al., 2014


VQA + Attention

FiLM: Visual Reasoning with a General Conditioning Layer

Perez et al., 2017


VQA + Attention

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering - Anderson et al., 2017
VQA + Attention

Attention: Given a set of vector values, and a vector query, attention is a


technique to compute a weighted sum of the values, dependent on the query.

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering - Anderson et al., 2017
Simple VQA model
Input
Question : What color is the hydrant?

Image
Simple VQA model
Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q)
Image
RNN

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I)
ConvNet
Simple VQA model
Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q)
Image
RNN

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I)
ConvNet Concatenate

FC layer
Predict Answer
How to add attention?
Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q)
Image
RNN

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I)
ConvNet Concatenate

FC layer
Predict Answer
How to add attention?
Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q) Query

Image

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I) Value
How to add attention?
Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q)
Image Attention Mask

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I)
How to add attention?
Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q)
A
<latexit sha1_base64="Tt4H7rxa0PpGz9XO7vh5H/PiHt0=">AAAB6HicbZA9TwJBEIbn8AvxC7W02UhMrMidjZSojSUk8pHAhewtc7Cyt3fZ3TMhF36BjYXG2PqT7Pw3LnCFgm+yyZN3ZrIzb5AIro3rfjuFjc2t7Z3ibmlv/+DwqHx80tZxqhi2WCxi1Q2oRsEltgw3AruJQhoFAjvB5G5e7zyh0jyWD2aaoB/RkeQhZ9RYq3kzKFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSENT/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPctNr1K/zeMowhmcwyV4cA11uIcGtIABwjO8wpvz6Lw4787HsrXg5DOn8EfO5w+R74zC</latexit>
<latexit sha1_base64="XA0+O1Jd0wn7spVAElsB0dzDNyw=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UaM2lgmYB6QHGFvM5es2ds7dveEEAL2NhaK2PrP2Nv537h5FJr4wcKP75thZyZMBdfG876dpeWV1bX13Ia7ubW9s5vf26/pJFMMqywRiWqEVKPgEquGG4GNVCGNQ4H1sH8zzusPqDRP5J0ZpBjEtCt5xBk11qpctfMFr+hNRBbBn0Hh8tO9eASAcjv/1eokLItRGiao1k3fS00wpMpwJnDktjKNKWV92sWmRUlj1MFwMuiIHFunQ6JE2ScNmbi/O4Y01noQh7Yypqan57Ox+V/WzEx0Hgy5TDODkk0/ijJBTELGW5MOV8iMGFigTHE7K2E9qigz9jauPYI/v/Ii1E6LvuWKXyhdw1Q5OIQjOAEfzqAEt1CGKjBAeIIXeHXunWfnzXmfli45s54D+CPn4wcBNo6P</latexit>
sha1_base64="5jhxeIZiPp+wZ6YJ19RXzNTZRxs=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8EgWIVdG23EqI1lAuYCyRJmJ2eTMbMXZmaFsOQJbCwUsdWHsbcR38bJpdDEHwY+/v8c5pzjJ4Ir7TjfVm5peWV1Lb9ub2xube8UdvfqKk4lwxqLRSybPlUoeIQ1zbXAZiKRhr7Ahj+4HueNe5SKx9GtHibohbQX8YAzqo1VvewUik7JmYgsgjuD4sWHfZ68f9mVTuGz3Y1ZGmKkmaBKtVwn0V5GpeZM4MhupwoTyga0hy2DEQ1Redlk0BE5Mk6XBLE0L9Jk4v7uyGio1DD0TWVIdV/NZ2Pzv6yV6uDMy3iUpBojNv0oSAXRMRlvTbpcItNiaIAyyc2shPWppEyb29jmCO78yotQPym5hqtusXwFU+XhAA7hGFw4hTLcQAVqwADhAZ7g2bqzHq0X63VamrNmPfvwR9bbD/K2kAM=</latexit>

Image Attention Mask

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I)

AT (I)
<latexit sha1_base64="wqCWwLR0cYfQxuln5U851zDJvpQ=">AAAB9HicbZDLTgIxFIbP4A3xhrp000hMcENm3OgSdaM7TLglMJJO6UBDpx3bDgmZ8BxuXGiMWx/GnW9jgVko+CdNvvznnJzTP4g508Z1v53c2vrG5lZ+u7Czu7d/UDw8amqZKEIbRHKp2gHWlDNBG4YZTtuxojgKOG0Fo9tZvTWmSjMp6mYSUz/CA8FCRrCxln/9mNanqBsPWfn+vFcsuRV3LrQKXgYlyFTrFb+6fUmSiApDONa647mx8VOsDCOcTgvdRNMYkxEe0I5FgSOq/XR+9BSdWaePQqnsEwbN3d8TKY60nkSB7YywGerl2sz8r9ZJTHjlp0zEiaGCLBaFCUdGolkCqM8UJYZPLGCimL0VkSFWmBibU8GG4C1/eRWaFxXP8oNXqt5kceThBE6hDB5cQhXuoAYNIPAEz/AKb87YeXHenY9Fa87JZo7hj5zPH3qLkTs=</latexit>
sha1_base64="uX6nZvk2keZmKR0noQiW9vTMBWE=">AAAB9HicbZDLSgMxFIbP1Futt6pLN6FFqAhlphtdVt3orkJv0BlLJs20oZnMmGQKZehzuHFREbc+jDvfxvSy0NYfAh//OYdz8vsxZ0rb9reV2djc2t7J7ub29g8Oj/LHJ00VJZLQBol4JNs+VpQzQRuaaU7bsaQ49Dlt+cO7Wb01olKxSNT1OKZeiPuCBYxgbSzv5imtT5AbD1jp4aKbL9pley60Ds4SitWCezkFgFo3/+X2IpKEVGjCsVIdx461l2KpGeF0knMTRWNMhrhPOwYFDqny0vnRE3RunB4KImme0Gju/p5IcajUOPRNZ4j1QK3WZuZ/tU6ig2svZSJONBVksShIONIRmiWAekxSovnYACaSmVsRGWCJiTY55UwIzuqX16FZKTuGH51i9RYWysIZFKAEDlxBFe6hBg0g8AwvMIU3a2S9Wu/Wx6I1Yy1nTuGPrM8fjQqSxA==</latexit>
sha1_base64="CGj61RfJ8i57+q3oCvw2LsN3acE=">AAAB9HicbZDLSsNAFIZP6q3WW9Wlm6FFqAglcaPLqBvdVegNmlgm00k7dHJxZlIIoW8huHGhiFsfxl3fxulloa0/DHz85xzOmd+LOZPKNCdGbm19Y3Mrv13Y2d3bPygeHjVllAhCGyTikWh7WFLOQtpQTHHajgXFgcdpyxveTuutERWSRWFdpTF1A9wPmc8IVtpyrx+z+hg58YBV7s+6xbJZNWdCq2AtoGyXnPPniZ3WusVvpxeRJKChIhxL2bHMWLkZFooRTscFJ5E0xmSI+7SjMcQBlW42O3qMTrXTQ34k9AsVmrm/JzIcSJkGnu4MsBrI5drU/K/WSZR/5WYsjBNFQzJf5CccqQhNE0A9JihRPNWAiWD6VkQGWGCidE4FHYK1/OVVaF5ULc0PVtm+gbnycAIlqIAFl2DDHdSgAQSe4AXe4N0YGa/Gh/E5b80Zi5lj+CPj6weWdJRK</latexit>

Re-weighted features
T
aij =
<latexit sha1_base64="Lly6la+/qtdEpLdpovTOMnbnh0U=">AAACK3icbVDLSsNAFJ34tr6iLt0MFqFFkMSNboRSN7pTsK3QxnAznbajk0k6MxFKyP+48Vdc6MIHbv0PJ2kXWj0wcDjnXO7cE8ScKe0479bM7Nz8wuLScmlldW19w97caqookYQ2SMQjeR2AopwJ2tBMc3odSwphwGkruDvN/dY9lYpF4kqPYuqF0BesxwhoI/l2HfyU3Wb4BLf8FLKb9CrDnRD0QIapBjHIKkY/zzrxgBXBynkV7+fZodEUqwyrVd8uOwdOAfyXuBNSRhNc+PZzpxuRJKRCEw5KtV0n1l4KUjPCaVbqJIrGQO6gT9uGCgip8tLi1gzvGaWLe5E0T2hcqD8nUgiVGoWBSeZnqGkvF//z2onuHXspE3GiqSDjRb2EYx3hvDjcZZISzUeGAJHM/BWTAUgg2tRbMiW40yf/Jc3DA9fwS7dcq0/qWEI7aBdVkIuOUA2doQvUQAQ9oCf0it6sR+vF+rA+x9EZazKzjX7B+voGq4enQg==</latexit>
sha1_base64="+Pd6EhawBNQ1MdIomnDdDDYRtsc=">AAACK3icbVC7SgNBFL3r2/iKWtoMRiFBkF0bbYSgjXYKxghJXO5OJmZ0dnadmRXCsv9jY2vtF1ho4QNbP8He2cTC14GBwznncueeIBZcG9d9cYaGR0bHxicmC1PTM7NzxfmFYx0lirIajUSkTgLUTHDJaoYbwU5ixTAMBKsHF7u5X79iSvNIHplezFohnkne4RSNlfziDvopP8/INqn7KWan6VFGmiGargpTg7Kbla2+nzXjLu8Hy/sVspZnL62mefmyUvGLJXfd7YP8Jd4XKVVXPm7vAODALz402xFNQiYNFah1w3Nj00pRGU4FywrNRLMY6QWesYalEkOmW2n/1oysWqVNOpGyTxrSV79PpBhq3QsDm8zP0L+9XPzPaySms9VKuYwTwyQdLOokgpiI5MWRNleMGtGzBKni9q+EdlEhNbbegi3B+33yX3K8se5ZfuiVqjswwAQswTKUwYNNqMIeHEANKFzDPTzBs3PjPDqvztsgOuR8zSzCDzjvn3X3qhE=</latexit>
sha1_base64="rQR1SDos6hDqfBCICO2g0B2246Q=">AAACK3icbVC7TsMwFHV40/IIMLJYFKRWSChhgQWpgoVuRaIUqS3RjetSU8cJtoNURfkfFlZ+gJ0BBh5i5RNgxmkZgHIkS0fnnKvre/yIM6Ud58UaG5+YnJqemc3l5+YXFu2l5RMVxpLQGgl5KE99UJQzQWuaaU5PI0kh8Dmt+72DzK9fUalYKI51P6KtAM4F6zAC2kievQ9ewi5SvIfrXgLpWXKc4mYAuiuDRIPopkWjV9Jm1GWDYLFSwptZ9tJoihUvSyXPLjhbzgB4lLjfpFBe/7i9u8p/Vj37odkOSRxQoQkHpRquE+lWAlIzwmmaa8aKRkB6cE4bhgoIqGolg1tTvGGUNu6E0jyh8UD9OZFAoFQ/8E0yO0P99TLxP68R685uK2EiijUVZLioE3OsQ5wVh9tMUqJ53xAgkpm/YtIFCUSbenOmBPfvyaPkZHvLNfzILZT30RAzaBWtoSJy0Q4qo0NURTVE0DW6R0/o2bqxHq1X620YHbO+Z1bQL1jvX26Xq4s=</latexit>
Wa tanh(WI ij (I) + Wq (q))

Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q)
A
<latexit sha1_base64="Tt4H7rxa0PpGz9XO7vh5H/PiHt0=">AAAB6HicbZA9TwJBEIbn8AvxC7W02UhMrMidjZSojSUk8pHAhewtc7Cyt3fZ3TMhF36BjYXG2PqT7Pw3LnCFgm+yyZN3ZrIzb5AIro3rfjuFjc2t7Z3ibmlv/+DwqHx80tZxqhi2WCxi1Q2oRsEltgw3AruJQhoFAjvB5G5e7zyh0jyWD2aaoB/RkeQhZ9RYq3kzKFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSENT/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPctNr1K/zeMowhmcwyV4cA11uIcGtIABwjO8wpvz6Lw4787HsrXg5DOn8EfO5w+R74zC</latexit>
<latexit sha1_base64="XA0+O1Jd0wn7spVAElsB0dzDNyw=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UaM2lgmYB6QHGFvM5es2ds7dveEEAL2NhaK2PrP2Nv537h5FJr4wcKP75thZyZMBdfG876dpeWV1bX13Ia7ubW9s5vf26/pJFMMqywRiWqEVKPgEquGG4GNVCGNQ4H1sH8zzusPqDRP5J0ZpBjEtCt5xBk11qpctfMFr+hNRBbBn0Hh8tO9eASAcjv/1eokLItRGiao1k3fS00wpMpwJnDktjKNKWV92sWmRUlj1MFwMuiIHFunQ6JE2ScNmbi/O4Y01noQh7Yypqan57Ox+V/WzEx0Hgy5TDODkk0/ijJBTELGW5MOV8iMGFigTHE7K2E9qigz9jauPYI/v/Ii1E6LvuWKXyhdw1Q5OIQjOAEfzqAEt1CGKjBAeIIXeHXunWfnzXmfli45s54D+CPn4wcBNo6P</latexit>
sha1_base64="5jhxeIZiPp+wZ6YJ19RXzNTZRxs=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8EgWIVdG23EqI1lAuYCyRJmJ2eTMbMXZmaFsOQJbCwUsdWHsbcR38bJpdDEHwY+/v8c5pzjJ4Ir7TjfVm5peWV1Lb9ub2xube8UdvfqKk4lwxqLRSybPlUoeIQ1zbXAZiKRhr7Ahj+4HueNe5SKx9GtHibohbQX8YAzqo1VvewUik7JmYgsgjuD4sWHfZ68f9mVTuGz3Y1ZGmKkmaBKtVwn0V5GpeZM4MhupwoTyga0hy2DEQ1Redlk0BE5Mk6XBLE0L9Jk4v7uyGio1DD0TWVIdV/NZ2Pzv6yV6uDMy3iUpBojNv0oSAXRMRlvTbpcItNiaIAyyc2shPWppEyb29jmCO78yotQPym5hqtusXwFU+XhAA7hGFw4hTLcQAVqwADhAZ7g2bqzHq0X63VamrNmPfvwR9bbD/K2kAM=</latexit>

Image Attention Mask

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I)

A · (I)
Re-weighted features
<latexit sha1_base64="9jo9WMRT48PnmFqqkN3qbCcicwM=">AAACZnicbVFNa9wwFJTdpE3cJtkmlB56EV0WdikEO5f2UkibS3NLIZsNrLfmWStnlejDkZ5LF+M/2VvPufRnVHaW0HwMCIaZN+hplJdSOIzjP0H4bG39+YuNzejlq63tnd7r3TNnKsv4mBlp7HkOjkuh+RgFSn5eWg4ql3ySXx21/uQnt04YfYrLks8UXGhRCAbopazXDCCrxWVDP9NUAS6sqp0pUMGvZjjJamh+1KfNnYWgF51+3KTlQnTJ4fGIfqBeu/aaE8Pr0WgUDe6y96LRF5qyuUHapn0w6/Xj/bgDfUySFemTFU6y3u90bliluEYmwblpEpc4q8GiYJI3UVo5XgK7ggs+9VSD4m5WdzU1dOCVOS2M9Ucj7dT/EzUo55Yq95Pt0u6h14pPedMKi0+zWuiyQq7Z7UVFJSka2nZO58JyhnLpCTAr/K6ULcACQ/8zkS8hefjkx+TsYD/x/HvSP/y6qmODvCPvyZAk5CM5JN/ICRkTRm6CzWA32Av+htvhm/Dt7WgYrDJ75B5C+g9Mjrgk</latexit>
sha1_base64="ZJimfBzOunvmnbFBkOMBT1E2QvE=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASClE26KbjtptmlEMcByxVX41E8zfwoM1chRuhd+kzddd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxryB89PjJ02cbm9HzFy+3XvVeb586U1nGJ8xIY89ycFwKzScoUPKz0nJQueTT/OJz60+vuHXC6BNclXyu4FyLQjBAj7JeM4CsFt8b+oGmCnBpVe1MgQqum+E0q6H5Vp80txaCXnb8qEnLpeiSw6MR3aOeXXrmxPByNBpFg9vsnWj0kaZsYZC2aR/Mev14P+6KPhTJjeiPd9O9H4SQ46z3M10YVimukUlwbpbEJc5rsCiY5E2UVo6XwC7gnM+81KC4m9fdmho68GRBC2P90Ug7+m+iBuXcSuW+sx3a3fda+D9vVmHxfl4LXVbINVtfVFSSoqHtzulCWM5QrrwAZoWflbIlWGDofybyS0juP/mhOD3YT7z+mvTHn8i6Nsg7skuGJCGHZEy+kGMyIYz8DjaD7WAn+BNuhW/Ct+vWMLjJ7JA7FdK/Xw25rQ==</latexit>
sha1_base64="E+aOEDdjm4snoMJV6Vg9dH2YVk8=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASC1E2zCbjtptmlEMcByxVX41E8yfwoM1elRugt+mTdZd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxdRA+evzk6bONzej5i5dbr3qvt0+dqSzjE2aksWc5OC6F5hMUKPlZaTmoXPJpfvm59affuXXC6BNclXyu4FyLQjBAj7JeM4CsFhcNPaSpAlxaVTtToIIfzXCa1dB8q0+aWwtBLzt+1KTlUnTJ4dGI7lHPrjxzYng1Go2iwW32TjT6SFO2MEjbtA9mvX68H3dFH4rkRvTHu+nez+vx6jjr/UoXhlWKa2QSnJslcYnzGiwKJnkTpZXjJbBLOOczLzUo7uZ1t6aGDjxZ0MJYfzTSjv6bqEE5t1K572yHdve9Fv7Pm1VYHMxrocsKuWbri4pKUjS03TldCMsZypUXwKzws1K2BAsM/c9EfgnJ/Sc/FKfv9xOvvyb98Seyrg3yjuySIUnIBzImX8gxmRBGfgebwXawE/wJt8I34dt1axjcZHbInQrpX2h3uzM=</latexit>
T
aij =
<latexit sha1_base64="Lly6la+/qtdEpLdpovTOMnbnh0U=">AAACK3icbVDLSsNAFJ34tr6iLt0MFqFFkMSNboRSN7pTsK3QxnAznbajk0k6MxFKyP+48Vdc6MIHbv0PJ2kXWj0wcDjnXO7cE8ScKe0479bM7Nz8wuLScmlldW19w97caqookYQ2SMQjeR2AopwJ2tBMc3odSwphwGkruDvN/dY9lYpF4kqPYuqF0BesxwhoI/l2HfyU3Wb4BLf8FLKb9CrDnRD0QIapBjHIKkY/zzrxgBXBynkV7+fZodEUqwyrVd8uOwdOAfyXuBNSRhNc+PZzpxuRJKRCEw5KtV0n1l4KUjPCaVbqJIrGQO6gT9uGCgip8tLi1gzvGaWLe5E0T2hcqD8nUgiVGoWBSeZnqGkvF//z2onuHXspE3GiqSDjRb2EYx3hvDjcZZISzUeGAJHM/BWTAUgg2tRbMiW40yf/Jc3DA9fwS7dcq0/qWEI7aBdVkIuOUA2doQvUQAQ9oCf0it6sR+vF+rA+x9EZazKzjX7B+voGq4enQg==</latexit>
sha1_base64="+Pd6EhawBNQ1MdIomnDdDDYRtsc=">AAACK3icbVC7SgNBFL3r2/iKWtoMRiFBkF0bbYSgjXYKxghJXO5OJmZ0dnadmRXCsv9jY2vtF1ho4QNbP8He2cTC14GBwznncueeIBZcG9d9cYaGR0bHxicmC1PTM7NzxfmFYx0lirIajUSkTgLUTHDJaoYbwU5ixTAMBKsHF7u5X79iSvNIHplezFohnkne4RSNlfziDvopP8/INqn7KWan6VFGmiGargpTg7Kbla2+nzXjLu8Hy/sVspZnL62mefmyUvGLJXfd7YP8Jd4XKVVXPm7vAODALz402xFNQiYNFah1w3Nj00pRGU4FywrNRLMY6QWesYalEkOmW2n/1oysWqVNOpGyTxrSV79PpBhq3QsDm8zP0L+9XPzPaySms9VKuYwTwyQdLOokgpiI5MWRNleMGtGzBKni9q+EdlEhNbbegi3B+33yX3K8se5ZfuiVqjswwAQswTKUwYNNqMIeHEANKFzDPTzBs3PjPDqvztsgOuR8zSzCDzjvn3X3qhE=</latexit>
sha1_base64="rQR1SDos6hDqfBCICO2g0B2246Q=">AAACK3icbVC7TsMwFHV40/IIMLJYFKRWSChhgQWpgoVuRaIUqS3RjetSU8cJtoNURfkfFlZ+gJ0BBh5i5RNgxmkZgHIkS0fnnKvre/yIM6Ud58UaG5+YnJqemc3l5+YXFu2l5RMVxpLQGgl5KE99UJQzQWuaaU5PI0kh8Dmt+72DzK9fUalYKI51P6KtAM4F6zAC2kievQ9ewi5SvIfrXgLpWXKc4mYAuiuDRIPopkWjV9Jm1GWDYLFSwptZ9tJoihUvSyXPLjhbzgB4lLjfpFBe/7i9u8p/Vj37odkOSRxQoQkHpRquE+lWAlIzwmmaa8aKRkB6cE4bhgoIqGolg1tTvGGUNu6E0jyh8UD9OZFAoFQ/8E0yO0P99TLxP68R685uK2EiijUVZLioE3OsQ5wVh9tMUqJ53xAgkpm/YtIFCUSbenOmBPfvyaPkZHvLNfzILZT30RAzaBWtoSJy0Q4qo0NURTVE0DW6R0/o2bqxHq1X620YHbO+Z1bQL1jvX26Xq4s=</latexit>
Wa tanh(WI ij (I) + Wq (q))

Input
Question : What color is the hydrant? (q)
Wq
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>

<latexit sha1_base64="Q+Bgi41fYCiHukIv6o1OToZV6zg=">AAACMHicbVDLSgMxFM3UV62vqks3wVJoEWTGjW6Eogt1p9AXtHW4k6ZtbCYzTTJCGeaT3PgpulFQxK1fYabtQq0HAodzzuXmHi/kTGnbfrUyC4tLyyvZ1dza+sbmVn57p66CSBJaIwEPZNMDRTkTtKaZ5rQZSgq+x2nDG56nfuOeSsUCUdXjkHZ86AvWYwS0kdz8RRHcmN0l+BQ33BiS27ia4LYPeiD9WIMYJCWjXyXtcMAmwdJVGR+k2ZHRFCuNyuVcwx25+YJ9aE+A54kzIwU0w7Wbf2p3AxL5VGjCQamWY4e6E4PUjHCa5NqRoiGQIfRpy1ABPlWdeHJwgotG6eJeIM0TGk/UnxMx+EqNfc8k01PUXy8V//Nake6ddGImwkhTQaaLehHHOsBpe7jLJCWajw0BIpn5KyYDkEC06ThnSnD+njxP6keHjuE3TqFyNqsji/bQPiohBx2jCrpE16iGCHpAz+gNvVuP1ov1YX1OoxlrNrOLfsH6+gbZGqjK</latexit>
sha1_base64="mg6DsTv05V3MjkyhJ4eQrH72JZ8=">AAACMHicbVA9TxtBFHznADHmIw4paVYYJFtI1h1N0kSyoEjoQPKXZJvTu/UaL97bO+/uIVmn+0lpUlDzG5IGJBBKmz599mwXCTDSSqOZeXr7JogF18Z1753Cm5XVtbfF9dLG5tb2u/L7nbaOEkVZi0YiUt0ANRNcspbhRrBurBiGgWCdYHKS+51rpjSPZNPMYjYI8VLyEadorOSXvxygn/KrjHwmHT/F7CJtZqQfohmrMDUox1nV6qdZPx7zebB6WiOHeXZqNc2r01qt1PGnfrni1t05yEviLUmlsf/n5hYAzvzyj/4woknIpKECte55bmwGKSrDqWBZqZ9oFiOd4CXrWSoxZHqQzg/OyIFVhmQUKfukIXP134kUQ61nYWCT+Sn6uZeLr3m9xIw+DVIu48QwSReLRokgJiJ5e2TIFaNGzCxBqrj9K6FjVEiN7bhkS/Cen/yStI/qnuXnXqVxDAsUYRf2oAoefIQGfIUzaAGFb/ATHuDR+e7cOU/Or0W04CxnPsB/cH7/BaOKq5k=</latexit>
sha1_base64="M4mt7+xNEQ8FI7ExLJc3ii4CbkQ=">AAACMHicbVC7SgNBFJ31GeMramkzGIUEQXZttBGCFmqnkIeQxOXuZGLGzM5uZmYDYdlPsrHwB/wDbRQUsbXX2tnEwteBgcM553LnHi/kTGnbfrTGxicmp6YzM9nZufmFxdzSclUFkSS0QgIeyDMPFOVM0IpmmtOzUFLwPU5rXvcg9Wt9KhULRFkPQtr04UKwNiOgjeTmDjfAjdllgvdwzY0hOY/LCW74oDvSjzWITlIw+nHSCDtsGCwcF/Fmmu0ZTbFCr1jM1tyem8vbW/YQ+C9xvki+tP5+c9uf/Thxc3eNVkAinwpNOChVd+xQN2OQmhFOk2wjUjQE0oULWjdUgE9VMx4enOANo7RwO5DmCY2H6veJGHylBr5nkukp6reXiv959Ui3d5sxE2GkqSCjRe2IYx3gtD3cYpISzQeGAJHM/BWTDkgg2nScNSU4v0/+S6rbW47hp06+tI9GyKBVtIYKyEE7qISO0AmqIIKu0D16Qs/WtfVgvVivo+iY9TWzgn7AevsEnCqtEw==</latexit>

Image

(I)
<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>

WI
<latexit sha1_base64="9igV3A1AkAC8JvlbCipvy0MjsXM=">AAACMHicbVDLSsNAFJ34rPFVdelmsAgtgiRudCOILrS7Cn1BW8PNdGrHTiZxZiKUkE9y46foRkERt36Fk7QLXwcGDuecy517/IgzpR3nxZqZnZtfWCws2csrq2vrxY3NpgpjSWiDhDyUbR8U5UzQhmaa03YkKQQ+py1/dJb5rTsqFQtFXY8j2gvgWrABI6CN5BXPd8FL2E2Kj3HLSyC9Suop7gaghzJINIhhWjZ6Ne1GQ5YHy9UK3suyt0ZTrHxbqdgtr+oVS86+kwP/Je6UlNAUNa/42O2HJA6o0ISDUh3XiXQvAakZ4TS1u7GiEZARXNOOoQICqnpJfnCKd43Sx4NQmic0ztXvEwkESo0D3ySzU9RvLxP/8zqxHhz1EiaiWFNBJosGMcc6xFl7uM8kJZqPDQEimfkrJkOQQLTp2DYluL9P/kuaB/uu4Zdu6eR0WkcBbaMdVEYuOkQn6ALVUAMRdI+e0Ct6sx6sZ+vd+phEZ6zpzBb6AevzC5x6qKI=</latexit>
sha1_base64="mnuFSGX/iJ9J2tGIoUWcYkcemzo=">AAACMHicbVBNa9tAFHxy0sZ1m9ZtoZdclhqDTcBIuTSXgkkPqW8J+AtsR3lar+2NVyt5dxUwQj8pl/yU9tJCP8i1v6Ir24fU7sDCMDOPt2+CWHBtXPe7U9jbf/L0oPis9PzF4ctX5ddvujpKFGUdGolI9QPUTHDJOoYbwfqxYhgGgvWC+afc790ypXkk22YZs1GIU8knnKKxkl8+r6Kf8puMfCQ9P8XsKm1nZBiimakwNShnWc3qrWwYz/gqWGvVyXGeXVhN89qiXi/1/JZfrrgNdwWyS7wNqTTfXf66BoALv/xlOI5oEjJpqECtB54bm1GKynAqWFYaJprFSOc4ZQNLJYZMj9LVwRmpWmVMJpGyTxqyUh9PpBhqvQwDm8xP0dteLv7PGyRmcjpKuYwTwyRdL5okgpiI5O2RMVeMGrG0BKni9q+EzlAhNbbjki3B2z55l3RPGp7ll16leQZrFOEI3kMNPPgATfgMF9ABCnfwFX7AT+fe+eb8dh7W0YKzmXkL/8D58xcrIaqH</latexit>
sha1_base64="gHM39sLqAUGfbxUuTwtadSChPwo=">AAACMHicbVDLSgMxFM34rPVVFdy4CRahRZAZN7oRSl1ody1YK7R1uJOmNjaTmSYZoQzzSd34DX6BbhR84NavMNO68HUgcDjnXG7u8ULOlLbtJ2tqemZ2bj6zkF1cWl5Zza2tn6sgkoTWScADeeGBopwJWtdMc3oRSgq+x2nD6x+nfuOGSsUCcaaHIW37cCVYlxHQRnJzJzvgxuw6wUe44caQXMZnCW75oHvSjzWIXlIweiVphT02DhYqRbybZgdGU6wwKBazDbfi5vL2nj0G/kucL5IvbdZe2ah8V3Vz961OQCKfCk04KNV07FC3Y5CaEU6TbCtSNATShyvaNFSAT1U7Hh+c4B2jdHA3kOYJjcfq94kYfKWGvmeS6Snqt5eK/3nNSHcP2zETYaSpIJNF3YhjHeC0PdxhkhLNh4YAkcz8FZMeSCDadJw1JTi/T/5Lzvf3HMNrTr5URhNk0BbaRgXkoANUQqeoiuqIoBF6QM/oxbq1Hq03630SnbK+ZjbQD1gfn3xirEM=</latexit>
T
aij =
<latexit sha1_base64="Lly6la+/qtdEpLdpovTOMnbnh0U=">AAACK3icbVDLSsNAFJ34tr6iLt0MFqFFkMSNboRSN7pTsK3QxnAznbajk0k6MxFKyP+48Vdc6MIHbv0PJ2kXWj0wcDjnXO7cE8ScKe0479bM7Nz8wuLScmlldW19w97caqookYQ2SMQjeR2AopwJ2tBMc3odSwphwGkruDvN/dY9lYpF4kqPYuqF0BesxwhoI/l2HfyU3Wb4BLf8FLKb9CrDnRD0QIapBjHIKkY/zzrxgBXBynkV7+fZodEUqwyrVd8uOwdOAfyXuBNSRhNc+PZzpxuRJKRCEw5KtV0n1l4KUjPCaVbqJIrGQO6gT9uGCgip8tLi1gzvGaWLe5E0T2hcqD8nUgiVGoWBSeZnqGkvF//z2onuHXspE3GiqSDjRb2EYx3hvDjcZZISzUeGAJHM/BWTAUgg2tRbMiW40yf/Jc3DA9fwS7dcq0/qWEI7aBdVkIuOUA2doQvUQAQ9oCf0it6sR+vF+rA+x9EZazKzjX7B+voGq4enQg==</latexit>
sha1_base64="+Pd6EhawBNQ1MdIomnDdDDYRtsc=">AAACK3icbVC7SgNBFL3r2/iKWtoMRiFBkF0bbYSgjXYKxghJXO5OJmZ0dnadmRXCsv9jY2vtF1ho4QNbP8He2cTC14GBwznncueeIBZcG9d9cYaGR0bHxicmC1PTM7NzxfmFYx0lirIajUSkTgLUTHDJaoYbwU5ixTAMBKsHF7u5X79iSvNIHplezFohnkne4RSNlfziDvopP8/INqn7KWan6VFGmiGargpTg7Kbla2+nzXjLu8Hy/sVspZnL62mefmyUvGLJXfd7YP8Jd4XKVVXPm7vAODALz402xFNQiYNFah1w3Nj00pRGU4FywrNRLMY6QWesYalEkOmW2n/1oysWqVNOpGyTxrSV79PpBhq3QsDm8zP0L+9XPzPaySms9VKuYwTwyQdLOokgpiI5MWRNleMGtGzBKni9q+EdlEhNbbegi3B+33yX3K8se5ZfuiVqjswwAQswTKUwYNNqMIeHEANKFzDPTzBs3PjPDqvztsgOuR8zSzCDzjvn3X3qhE=</latexit>
sha1_base64="rQR1SDos6hDqfBCICO2g0B2246Q=">AAACK3icbVC7TsMwFHV40/IIMLJYFKRWSChhgQWpgoVuRaIUqS3RjetSU8cJtoNURfkfFlZ+gJ0BBh5i5RNgxmkZgHIkS0fnnKvre/yIM6Ud58UaG5+YnJqemc3l5+YXFu2l5RMVxpLQGgl5KE99UJQzQWuaaU5PI0kh8Dmt+72DzK9fUalYKI51P6KtAM4F6zAC2kievQ9ewi5SvIfrXgLpWXKc4mYAuiuDRIPopkWjV9Jm1GWDYLFSwptZ9tJoihUvSyXPLjhbzgB4lLjfpFBe/7i9u8p/Vj37odkOSRxQoQkHpRquE+lWAlIzwmmaa8aKRkB6cE4bhgoIqGolg1tTvGGUNu6E0jyh8UD9OZFAoFQ/8E0yO0P99TLxP68R685uK2EiijUVZLioE3OsQ5wVh9tMUqJ53xAgkpm/YtIFCUSbenOmBPfvyaPkZHvLNfzILZT30RAzaBWtoSJy0Q4qo0NURTVE0DW6R0/o2bqxHq1X620YHbO+Z1bQL1jvX26Xq4s=</latexit>
Wa tanh(WI ij (I) + Wq (q))

Input
Question : What color is the hydrant? (q)
Wq
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>

<latexit sha1_base64="Q+Bgi41fYCiHukIv6o1OToZV6zg=">AAACMHicbVDLSgMxFM3UV62vqks3wVJoEWTGjW6Eogt1p9AXtHW4k6ZtbCYzTTJCGeaT3PgpulFQxK1fYabtQq0HAodzzuXmHi/kTGnbfrUyC4tLyyvZ1dza+sbmVn57p66CSBJaIwEPZNMDRTkTtKaZ5rQZSgq+x2nDG56nfuOeSsUCUdXjkHZ86AvWYwS0kdz8RRHcmN0l+BQ33BiS27ia4LYPeiD9WIMYJCWjXyXtcMAmwdJVGR+k2ZHRFCuNyuVcwx25+YJ9aE+A54kzIwU0w7Wbf2p3AxL5VGjCQamWY4e6E4PUjHCa5NqRoiGQIfRpy1ABPlWdeHJwgotG6eJeIM0TGk/UnxMx+EqNfc8k01PUXy8V//Nake6ddGImwkhTQaaLehHHOsBpe7jLJCWajw0BIpn5KyYDkEC06ThnSnD+njxP6keHjuE3TqFyNqsji/bQPiohBx2jCrpE16iGCHpAz+gNvVuP1ov1YX1OoxlrNrOLfsH6+gbZGqjK</latexit>
sha1_base64="mg6DsTv05V3MjkyhJ4eQrH72JZ8=">AAACMHicbVA9TxtBFHznADHmIw4paVYYJFtI1h1N0kSyoEjoQPKXZJvTu/UaL97bO+/uIVmn+0lpUlDzG5IGJBBKmz599mwXCTDSSqOZeXr7JogF18Z1753Cm5XVtbfF9dLG5tb2u/L7nbaOEkVZi0YiUt0ANRNcspbhRrBurBiGgWCdYHKS+51rpjSPZNPMYjYI8VLyEadorOSXvxygn/KrjHwmHT/F7CJtZqQfohmrMDUox1nV6qdZPx7zebB6WiOHeXZqNc2r01qt1PGnfrni1t05yEviLUmlsf/n5hYAzvzyj/4woknIpKECte55bmwGKSrDqWBZqZ9oFiOd4CXrWSoxZHqQzg/OyIFVhmQUKfukIXP134kUQ61nYWCT+Sn6uZeLr3m9xIw+DVIu48QwSReLRokgJiJ5e2TIFaNGzCxBqrj9K6FjVEiN7bhkS/Cen/yStI/qnuXnXqVxDAsUYRf2oAoefIQGfIUzaAGFb/ATHuDR+e7cOU/Or0W04CxnPsB/cH7/BaOKq5k=</latexit>
sha1_base64="M4mt7+xNEQ8FI7ExLJc3ii4CbkQ=">AAACMHicbVC7SgNBFJ31GeMramkzGIUEQXZttBGCFmqnkIeQxOXuZGLGzM5uZmYDYdlPsrHwB/wDbRQUsbXX2tnEwteBgcM553LnHi/kTGnbfrTGxicmp6YzM9nZufmFxdzSclUFkSS0QgIeyDMPFOVM0IpmmtOzUFLwPU5rXvcg9Wt9KhULRFkPQtr04UKwNiOgjeTmDjfAjdllgvdwzY0hOY/LCW74oDvSjzWITlIw+nHSCDtsGCwcF/Fmmu0ZTbFCr1jM1tyem8vbW/YQ+C9xvki+tP5+c9uf/Thxc3eNVkAinwpNOChVd+xQN2OQmhFOk2wjUjQE0oULWjdUgE9VMx4enOANo7RwO5DmCY2H6veJGHylBr5nkukp6reXiv959Ui3d5sxE2GkqSCjRe2IYx3gtD3cYpISzQeGAJHM/BWTDkgg2nScNSU4v0/+S6rbW47hp06+tI9GyKBVtIYKyEE7qISO0AmqIIKu0D16Qs/WtfVgvVivo+iY9TWzgn7AevsEnCqtEw==</latexit>

Image

WaT tanh
<latexit sha1_base64="Qh7XP7JvGARh+ibZRwA2Vp/LHto=">AAACQ3icbVDLSgMxFM3UV62vqks3wVJoEcqMG90Iohu7U7BWbOtwJ03b2ExmTDJCGebf3PgD7vwBNy4UcSuYqYNo64XA4Ty4N8cLOVPatp+s3Mzs3PxCfrGwtLyyulZc37hQQSQJbZCAB/LSA0U5E7Shmeb0MpQUfI/Tpjc8TvXmHZWKBeJcj0La8aEvWI8R0IZyi1dlcGN2k+AD3HRjSK7j8wS3fdAD6ccaxCCpGL6etMMBGxsr9SreSb23hlOsclutFn6Sf4JusWTX7PHgaeBkoISyOXWLj+1uQCKfCk04KNVy7FB3YpCaEU6TQjtSNAQyhD5tGSjAp6oTjztIcNkwXdwLpHlC4zH7OxGDr9TI94wzPVJNain5n9aKdG+/EzMRRpoK8r2oF3GsA5wWirtMUqL5yAAgkplbMRmABKJN7QVTgjP55WlwsVtzDD5zSodHWR15tIW2UQU5aA8dohN0ihqIoHv0jF7Rm/VgvVjv1se3NWdlmU30Z6zPLxahsaI=</latexit>
sha1_base64="cLgsr6E/EaE1aA7ivU9aP1seAcc=">AAACQ3icbVDLSiNBFL3tazS+Mrp0UxiEDoJ0u9GNEMaN7iIkRkxic7tSMaXV1W1V9UBo+t/c+APuBBezmo0LRdwKVicyjI8LBYfz4N46YSK4Np5350xMTk3P/JidK80vLC4tl3+uHOs4VZQ1aSxidRKiZoJL1jTcCHaSKIZRKFgrvNwv9NZvpjSPZcMME9aN8FzyPqdoLBWUTzcwyPhFTvZIK8gwP8saOelEaAYqygzKQe5a/jDvJAM+MrqHVbJZeK8sp7l7Va2W/iU/BINyxdvyRkO+Av8dVGrr7v0fAKgH5dtOL6ZpxKShArVu+15iuhkqw6lgeamTapYgvcRz1rZQYsR0Nxt1kJMNy/RIP1b2SUNG7P+JDCOth1FoncWR+rNWkN9p7dT0d7sZl0lqmKTjRf1UEBOTolDS44pRI4YWIFXc3kroABVSY2sv2RL8z1/+Co63t3yLj/xK7ReMZxbWYB1c8GEHanAAdWgChWv4C4/w5Nw4D86z8zK2TjjvmVX4MM7rGydws+g=</latexit>
sha1_base64="1LUnge7oK30a2hA8v2b8Kh72GcA=">AAACQ3icbVC7SgNBFJ31bXxFLW2GBGGDILs22ghBG+0U8sIkLncnEzM6O7vOzAph2S/xZ2wsbO1EaxsLRWwFZxMRXxcGDufBvXP8iDOlHefOGhkdG5+YnJrOzczOzS/kF5dqKowloVUS8lA2fFCUM0GrmmlOG5GkEPic1v2z3UyvX1CpWCgquh/RdgAngnUZAW0oL3+0Cl7CTlO8jeteAulxUklxKwDdk0GiQfRS2/D7aSvqsYHR3i/htcx7bjjF7PNSKfeV/BH08kVn3RkM/gvcT1AsF+yH+82bywMvf9vqhCQOqNCEg1JN14l0OwGpGeE0zbViRSMgZ3BCmwYKCKhqJ4MOUrxqmA7uhtI8ofGA/Z5IIFCqH/jGmR2pfmsZ+Z/WjHV3q50wEcWaCjJc1I051iHOCsUdJinRvG8AEMnMrZj0QALRpvacKcH9/eW/oLax7hp86BbLO2g4U2gFFZCNXLSJymgPHaAqIugKPaJn9GJdW0/Wq/U2tI5Yn5ll9GOs9w9FhLV+</latexit>

(I)
<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>

WI
<latexit sha1_base64="9igV3A1AkAC8JvlbCipvy0MjsXM=">AAACMHicbVDLSsNAFJ34rPFVdelmsAgtgiRudCOILrS7Cn1BW8PNdGrHTiZxZiKUkE9y46foRkERt36Fk7QLXwcGDuecy517/IgzpR3nxZqZnZtfWCws2csrq2vrxY3NpgpjSWiDhDyUbR8U5UzQhmaa03YkKQQ+py1/dJb5rTsqFQtFXY8j2gvgWrABI6CN5BXPd8FL2E2Kj3HLSyC9Suop7gaghzJINIhhWjZ6Ne1GQ5YHy9UK3suyt0ZTrHxbqdgtr+oVS86+kwP/Je6UlNAUNa/42O2HJA6o0ISDUh3XiXQvAakZ4TS1u7GiEZARXNOOoQICqnpJfnCKd43Sx4NQmic0ztXvEwkESo0D3ySzU9RvLxP/8zqxHhz1EiaiWFNBJosGMcc6xFl7uM8kJZqPDQEimfkrJkOQQLTp2DYluL9P/kuaB/uu4Zdu6eR0WkcBbaMdVEYuOkQn6ALVUAMRdI+e0Ct6sx6sZ+vd+phEZ6zpzBb6AevzC5x6qKI=</latexit>
sha1_base64="mnuFSGX/iJ9J2tGIoUWcYkcemzo=">AAACMHicbVBNa9tAFHxy0sZ1m9ZtoZdclhqDTcBIuTSXgkkPqW8J+AtsR3lar+2NVyt5dxUwQj8pl/yU9tJCP8i1v6Ir24fU7sDCMDOPt2+CWHBtXPe7U9jbf/L0oPis9PzF4ctX5ddvujpKFGUdGolI9QPUTHDJOoYbwfqxYhgGgvWC+afc790ypXkk22YZs1GIU8knnKKxkl8+r6Kf8puMfCQ9P8XsKm1nZBiimakwNShnWc3qrWwYz/gqWGvVyXGeXVhN89qiXi/1/JZfrrgNdwWyS7wNqTTfXf66BoALv/xlOI5oEjJpqECtB54bm1GKynAqWFYaJprFSOc4ZQNLJYZMj9LVwRmpWmVMJpGyTxqyUh9PpBhqvQwDm8xP0dteLv7PGyRmcjpKuYwTwyRdL5okgpiI5O2RMVeMGrG0BKni9q+EzlAhNbbjki3B2z55l3RPGp7ll16leQZrFOEI3kMNPPgATfgMF9ABCnfwFX7AT+fe+eb8dh7W0YKzmXkL/8D58xcrIaqH</latexit>
sha1_base64="gHM39sLqAUGfbxUuTwtadSChPwo=">AAACMHicbVDLSgMxFM34rPVVFdy4CRahRZAZN7oRSl1ody1YK7R1uJOmNjaTmSYZoQzzSd34DX6BbhR84NavMNO68HUgcDjnXG7u8ULOlLbtJ2tqemZ2bj6zkF1cWl5Zza2tn6sgkoTWScADeeGBopwJWtdMc3oRSgq+x2nD6x+nfuOGSsUCcaaHIW37cCVYlxHQRnJzJzvgxuw6wUe44caQXMZnCW75oHvSjzWIXlIweiVphT02DhYqRbybZgdGU6wwKBazDbfi5vL2nj0G/kucL5IvbdZe2ah8V3Vz961OQCKfCk04KNV07FC3Y5CaEU6TbCtSNATShyvaNFSAT1U7Hh+c4B2jdHA3kOYJjcfq94kYfKWGvmeS6Snqt5eK/3nNSHcP2zETYaSpIJNF3YhjHeC0PdxhkhLNh4YAkcz8FZMeSCDadJw1JTi/T/5Lzvf3HMNrTr5URhNk0BbaRgXkoANUQqeoiuqIoBF6QM/oxbq1Hq03630SnbK+ZjbQD1gfn3xirEM=</latexit>
T
aij =
<latexit sha1_base64="7kChU7GasXkFr57wu6Oh6xMi3kE=">AAACVXicbVFNSwMxFMyu3/Wr6tFLsAgtgux60YsgetGbgrVCW5e3adZGk+yavBXLsn/Si/hPvAimaxG/BgLDzBvyMokzKSwGwavnT03PzM7NL9QWl5ZXVutr61c2zQ3jbZbK1FzHYLkUmrdRoOTXmeGgYsk78f3J2O88cmNFqi9xlPG+glstEsEAnRTVJUSFuCvpIe0pwKFRhU0TVPBUNjtRAeVNcVl+WQh6WOlnZS8biirZPGvRHeq0B6dZ0XxotVq17a/sj2hUbwS7QQX6l4QT0iATnEf1594gZbniGpkEa7thkGG/AIOCSV7WernlGbB7uOVdRzUobvtF1UpJt50yoElq3NFIK/V7ogBl7UjFbnK8pP3tjcX/vG6OyUG/EDrLkWv2eVGSS4opHVdMB8JwhnLkCDAj3K6UDcEAQ/cRNVdC+PvJf8nV3m7o+EXYODqe1DFPNskWaZKQ7JMjckrOSZsw8kzePM/zvRfv3Z/2Zz9HfW+S2SA/4K9+AOMOtIs=</latexit>
sha1_base64="QH6QhqfK9L7nVHBGR783RDVQtyA=">AAACVXicbZHPSxtBFMffbq3V9IdpPXoZmgoJBdn10l4KwV7qTcEYIUmXt5NZM3Vmdp15WwzL/ls992/wIv4hQi8FJxuRqn0w8OXzfV/mzZu0UNJRFF0H4bOV56sv1tZbL1+9frPRfvvu2OWl5WLAc5XbkxSdUNKIAUlS4qSwAnWqxDA9+7rwhz+FdTI3RzQvxETjqZGZ5EgeJW2FSSV/1OwLG2ukmdWVyzPSeFF3h0mF9ffqqL63CM2s4fv1uJjJJtnd77GPzLNzz5zsnvd6vdb2ffZBNGl3op2oKfZUxHei0/9w8+s3ABwk7cvxNOelFoa4QudGcVTQpEJLkitRt8alEwXyMzwVIy8NauEmVbOVmm17MmVZbv0xxBr6b6JC7dxcp75zMaR77C3g/7xRSdnnSSVNUZIwfHlRVipGOVusmE2lFZzU3AvkVvpZGZ+hRU7+I1p+CfHjJz8Vx7s7sdeHcae/B8tagy14D12I4RP04RscwAA4XMKfIAjC4Cr4G66Eq8vWMLjLbMKDCjduAa1+t1o=</latexit>
sha1_base64="P10p8SuTWYjxJ2bu6/sj8kelRSg=">AAACVXicbZHfShtBFMZn1z+NUWuql70ZGoUEQXZ7094UQr2pdxaMEZK4nJ3Mmqkzs+vMWWlY9rV8gb5BbsQHUbwROtmI+O/AwMfvOx9z5kycSWExCG48f2FxaflDbaW+urb+caPxafPYprlhvMtSmZqTGCyXQvMuCpT8JDMcVCx5Lz7fn/m9S26sSPURTjI+VHCmRSIYoENRQ0JUiD8l/UEHCnBsVGHTBBX8LVu9qIDytDgqnywEPa74QTnIxqJKtg7adJc6duGYFa2Ldrtd33nKvohGjWawF1RF34rwUTQ727dX/y5X7w6jxnQwSlmuuEYmwdp+GGQ4LMCgYJKX9UFueQbsHM5430kNitthUW2lpDuOjGiSGnc00oo+TxSgrJ2o2HXOhrSvvRl8z+vnmHwfFkJnOXLN5hcluaSY0tmK6UgYzlBOnABmhJuVsjEYYOg+ou6WEL5+8ltx/HUvdPp32Oz8JPOqkc/kC2mRkHwjHfKLHJIuYWRK7j3P871r78Ff9Jfnrb73mNkiL8rf+A+mHrjU</latexit>
softmax(Wa tanh(WI ij (I) + Wq (q)))

Input
Question : What color is the hydrant?
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>
(q)
A
<latexit sha1_base64="Tt4H7rxa0PpGz9XO7vh5H/PiHt0=">AAAB6HicbZA9TwJBEIbn8AvxC7W02UhMrMidjZSojSUk8pHAhewtc7Cyt3fZ3TMhF36BjYXG2PqT7Pw3LnCFgm+yyZN3ZrIzb5AIro3rfjuFjc2t7Z3ibmlv/+DwqHx80tZxqhi2WCxi1Q2oRsEltgw3AruJQhoFAjvB5G5e7zyh0jyWD2aaoB/RkeQhZ9RYq3kzKFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSENT/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPctNr1K/zeMowhmcwyV4cA11uIcGtIABwjO8wpvz6Lw4787HsrXg5DOn8EfO5w+R74zC</latexit>
<latexit sha1_base64="XA0+O1Jd0wn7spVAElsB0dzDNyw=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UaM2lgmYB6QHGFvM5es2ds7dveEEAL2NhaK2PrP2Nv537h5FJr4wcKP75thZyZMBdfG876dpeWV1bX13Ia7ubW9s5vf26/pJFMMqywRiWqEVKPgEquGG4GNVCGNQ4H1sH8zzusPqDRP5J0ZpBjEtCt5xBk11qpctfMFr+hNRBbBn0Hh8tO9eASAcjv/1eokLItRGiao1k3fS00wpMpwJnDktjKNKWV92sWmRUlj1MFwMuiIHFunQ6JE2ScNmbi/O4Y01noQh7Yypqan57Ox+V/WzEx0Hgy5TDODkk0/ijJBTELGW5MOV8iMGFigTHE7K2E9qigz9jauPYI/v/Ii1E6LvuWKXyhdw1Q5OIQjOAEfzqAEt1CGKjBAeIIXeHXunWfnzXmfli45s54D+CPn4wcBNo6P</latexit>
sha1_base64="5jhxeIZiPp+wZ6YJ19RXzNTZRxs=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8EgWIVdG23EqI1lAuYCyRJmJ2eTMbMXZmaFsOQJbCwUsdWHsbcR38bJpdDEHwY+/v8c5pzjJ4Ir7TjfVm5peWV1Lb9ub2xube8UdvfqKk4lwxqLRSybPlUoeIQ1zbXAZiKRhr7Ahj+4HueNe5SKx9GtHibohbQX8YAzqo1VvewUik7JmYgsgjuD4sWHfZ68f9mVTuGz3Y1ZGmKkmaBKtVwn0V5GpeZM4MhupwoTyga0hy2DEQ1Redlk0BE5Mk6XBLE0L9Jk4v7uyGio1DD0TWVIdV/NZ2Pzv6yV6uDMy3iUpBojNv0oSAXRMRlvTbpcItNiaIAyyc2shPWppEyb29jmCO78yotQPym5hqtusXwFU+XhAA7hGFw4hTLcQAVqwADhAZ7g2bqzHq0X63VamrNmPfvwR9bbD/K2kAM=</latexit>

Image Attention Mask

<latexit sha1_base64="SqJiGd6RT500XSMOUnFA2iTlae8=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6EVvFewHtEvJptM2NJsNSVYoS3+EFw+KePX3ePPfmLZ70NYXAg/vzJCZN1KCG+v7397a+sbm1nZhp7i7t39wWDo6bpok1QwbLBGJbkfUoOASG5ZbgW2lkcaRwFY0vp3VW0+oDU/ko50oDGM6lHzAGbXOanXViFfuL3qlsl/15yKrEORQhlz1Xumr209YGqO0TFBjOoGvbJhRbTkTOC12U4OKsjEdYsehpDGaMJuvOyXnzumTQaLdk5bM3d8TGY2NmcSR64ypHZnl2sz8r9ZJ7eA6zLhUqUXJFh8NUkFsQma3kz7XyKyYOKBMc7crYSOqKbMuoaILIVg+eRWal9XA8UNQrt3kcRTgFM6gAgFcQQ3uoA4NYDCGZ3iFN095L96797FoXfPymRP4I+/zB3BFjvQ=</latexit>
sha1_base64="GN+wnayZplx7hZ6W7N8q1VE0ajQ=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui250V8FeoDOUTJppQzOZkGSEMvQh3LhQRJc+jzvfxnTahbb+EPj4/3PIOSeUnGnjut9OYW19Y3OruF3a2d3bPygfHrV1kipCWyThieqGWFPOBG0ZZjjtSkVxHHLaCcc3s7zzSJVmiXgwE0mDGA8FixjBxlodX45Y7e6sX666dTcXWgVvAdVGxT//AIBmv/zlDxKSxlQYwrHWPc+VJsiwMoxwOi35qaYSkzEe0p5FgWOqgywfd4pOrTNAUaLsEwbl7u+ODMdaT+LQVsbYjPRyNjP/y3qpia6CjAmZGirI/KMo5cgkaLY7GjBFieETC5goZmdFZIQVJsZeqGSP4C2vvArti7pn+d6rNq5hriKcQAVq4MElNOAWmtACAmN4ghd4daTz7Lw57/PSgrPoOYY/cj5/AILEkH0=</latexit>
sha1_base64="bETkKI1BsLdOsiZNgW5hJFV48Gs=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBvdVbAX6Awlk2ba0EwmJBmhDH0IQVwo4tbncde3Mb0stPWHwMf/n0POOZHkTBvXnThr6xubW9uFneLu3v7BYenouKnTTBHaIClPVTvCmnImaMMww2lbKoqTiNNWNLyd5q0nqjRLxaMZSRomuC9YzAg21moFcsCq9+fdUsWtuTOhVfAWUPHLwcXLxB/Vu6XvoJeSLKHCEI617niuNGGOlWGE03ExyDSVmAxxn3YsCpxQHeazccfozDo9FKfKPmHQzP3dkeNE61ES2coEm4Fezqbmf1knM/F1mDMhM0MFmX8UZxyZFE13Rz2mKDF8ZAETxeysiAywwsTYCxXtEbzllVeheVnzLD94Ff8G5irAKZShCh5cgQ93UIcGEBjCM7zBuyOdV+fD+ZyXrjmLnhP4I+frB4wukgM=</latexit>
(I)

A · (I)
Re-weighted features
<latexit sha1_base64="9jo9WMRT48PnmFqqkN3qbCcicwM=">AAACZnicbVFNa9wwFJTdpE3cJtkmlB56EV0WdikEO5f2UkibS3NLIZsNrLfmWStnlejDkZ5LF+M/2VvPufRnVHaW0HwMCIaZN+hplJdSOIzjP0H4bG39+YuNzejlq63tnd7r3TNnKsv4mBlp7HkOjkuh+RgFSn5eWg4ql3ySXx21/uQnt04YfYrLks8UXGhRCAbopazXDCCrxWVDP9NUAS6sqp0pUMGvZjjJamh+1KfNnYWgF51+3KTlQnTJ4fGIfqBeu/aaE8Pr0WgUDe6y96LRF5qyuUHapn0w6/Xj/bgDfUySFemTFU6y3u90bliluEYmwblpEpc4q8GiYJI3UVo5XgK7ggs+9VSD4m5WdzU1dOCVOS2M9Ucj7dT/EzUo55Yq95Pt0u6h14pPedMKi0+zWuiyQq7Z7UVFJSka2nZO58JyhnLpCTAr/K6ULcACQ/8zkS8hefjkx+TsYD/x/HvSP/y6qmODvCPvyZAk5CM5JN/ICRkTRm6CzWA32Av+htvhm/Dt7WgYrDJ75B5C+g9Mjrgk</latexit>
sha1_base64="ZJimfBzOunvmnbFBkOMBT1E2QvE=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASClE26KbjtptmlEMcByxVX41E8zfwoM1chRuhd+kzddd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxryB89PjJ02cbm9HzFy+3XvVeb586U1nGJ8xIY89ycFwKzScoUPKz0nJQueTT/OJz60+vuHXC6BNclXyu4FyLQjBAj7JeM4CsFt8b+oGmCnBpVe1MgQqum+E0q6H5Vp80txaCXnb8qEnLpeiSw6MR3aOeXXrmxPByNBpFg9vsnWj0kaZsYZC2aR/Mev14P+6KPhTJjeiPd9O9H4SQ46z3M10YVimukUlwbpbEJc5rsCiY5E2UVo6XwC7gnM+81KC4m9fdmho68GRBC2P90Ug7+m+iBuXcSuW+sx3a3fda+D9vVmHxfl4LXVbINVtfVFSSoqHtzulCWM5QrrwAZoWflbIlWGDofybyS0juP/mhOD3YT7z+mvTHn8i6Nsg7skuGJCGHZEy+kGMyIYz8DjaD7WAn+BNuhW/Ct+vWMLjJ7JA7FdK/Xw25rQ==</latexit>
sha1_base64="E+aOEDdjm4snoMJV6Vg9dH2YVk8=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASC1E2zCbjtptmlEMcByxVX41E8yfwoM1elRugt+mTdZd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxdRA+evzk6bONzej5i5dbr3qvt0+dqSzjE2aksWc5OC6F5hMUKPlZaTmoXPJpfvm59affuXXC6BNclXyu4FyLQjBAj7JeM4CsFhcNPaSpAlxaVTtToIIfzXCa1dB8q0+aWwtBLzt+1KTlUnTJ4dGI7lHPrjxzYng1Go2iwW32TjT6SFO2MEjbtA9mvX68H3dFH4rkRvTHu+nez+vx6jjr/UoXhlWKa2QSnJslcYnzGiwKJnkTpZXjJbBLOOczLzUo7uZ1t6aGDjxZ0MJYfzTSjv6bqEE5t1K572yHdve9Fv7Pm1VYHMxrocsKuWbri4pKUjS03TldCMsZypUXwKzws1K2BAsM/c9EfgnJ/Sc/FKfv9xOvvyb98Seyrg3yjuySIUnIBzImX8gxmRBGfgebwXawE/wJt8I34dt1axjcZHbInQrpX2h3uzM=</latexit>
(q)
FC layer
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>

Predict Answer
P
<latexit sha1_base64="IRE+WBfaQbF0wYsLhZdQBD5cHNc=">AAACgHicbVFdS+NAFJ3E3VXrflR99GXYUkjZXU18UQRB9GV9c8Faoa3hZjqxs52ZxJkbsYT8Dv+Xb/4YYadpET/2wsDhnDn3M8mlsBiGj56/9OHjp+WV1cba5y9fvzXXNy5sVhjGuyyTmblMwHIpNO+iQMkvc8NBJZL3ksnJTO/dcmNFps9xmvOhgmstUsEAHRU379sQl+JvRQ/pQAGOjSptlqKCuyroxSVUV+V59Swh6HHNn1aDfCxqZ3DaoT+o424cZ0Vw0+l0Gu1n7ytrY+6C2uQq2kLVOeY9vEwZN1vhdlgHfQ+iBWiRRZzFzYfBKGOF4hqZBGv7UZjjsASDgknuKheW58AmcM37DmpQ3A7LeoEVbTtmRNPMuKeR1uxLRwnK2qlK3M/ZOPatNiP/p/ULTPeHpdB5gVyzeaG0kBQzOrsGHQnDGcqpA8CMcL1SNgYDDN3NGm4J0duR34OL3e3I4T9R6+h4sY4VskW+k4BEZI8ckd/kjHQJI09ey/vp/fJ9P/B3/Gj+1fcWnk3yKvyDfw9kwp4=</latexit>
sha1_base64="Qqmu3kkY+vJ6TaADdy1egrh7sEE=">AAACgHicbVHLattAFB2pTZs6Ses2y2yGGINMXlI3LYWCaTfJLoE4DtiOuBqP4klmRsrMVakRgv5Ftv2WfkJ3/ZhAxlIIefTCwOGcOfeZ5FJYDMN/nv/i5dKr18tvWiura2/ftd9/OLFZYRgfsExm5jQBy6XQfIACJT/NDQeVSD5MLr8v9OEPbqzI9DHOcz5RcK5FKhigo+L2dRfiUlxU9CsdK8CZUaXNUlTwswqGcQnVWXlc3UsIelbzB9U4n4naGRz06BZ13JXjrAiuer1eq3vvfWRtNS6oTa6iLVSdo+nhYcq43Ql3wzrocxDdgU5/78/vX4SQw7j9dzzNWKG4RibB2lEU5jgpwaBgkrvKheU5sEs45yMHNShuJ2W9wIp2HTOlaWbc00hr9qGjBGXtXCXu52Ic+1RbkP/TRgWmnyel0HmBXLOmUFpIihldXINOheEM5dwBYEa4XimbgQGG7mYtt4To6cjPwcnH3cjho6jT/0aaWCYbZJMEJCKfSJ/sk0MyIIzceB1v29vxfT/w9/yo+ep7d5518ij8L7cwGsTv</latexit>
sha1_base64="FrjvqgAkJz+wsuSErm9FYCygwI4=">AAACgHicbVHLTttAFB27UGhK20CX3YyIIjlqC3Y3oEpIUbspO5AIQUpS63oyJgMzYzNzXRFZXvcTuu23dNlld/0YJCY2QryuNNLROXPuM8mlsBiG/z3/2dLy85XVF62Xa69ev2mvbxzbrDCMD1gmM3OSgOVSaD5AgZKf5IaDSiQfJudfF/rwBzdWZPoI5zmfKDjVIhUM0FFx+1cX4lKcVXSPjhXgzKjSZikquKyCYVxC9b08qm4lBD2r+f1qnM9E7Qz2e/Q9ddyF46wILnq9Xqt7671nbTUuqE2uoi1UnaPp4W7KuN0Jt8I66GMQ3YBOf/vP759/p3sHcfvfeJqxQnGNTIK1oyjMcVKCQcEkd5ULy3Ng53DKRw5qUNxOynqBFe06ZkrTzLinkdbsXUcJytq5StzPxTj2obYgn9JGBaa7k1LovECuWVMoLSTFjC6uQafCcIZy7gAwI1yvlM3AAEN3s5ZbQvRw5Mfg+NNW5PBh1Ol/IU2skndkkwQkIjukT76RAzIgjFx5He+D99H3/cDf9qPmq+/deN6Se+F/vgYfH8Zh</latexit>
a (I) = ij aij ij (I)
Sum

Re-weighted features
A · (I)
<latexit sha1_base64="9jo9WMRT48PnmFqqkN3qbCcicwM=">AAACZnicbVFNa9wwFJTdpE3cJtkmlB56EV0WdikEO5f2UkibS3NLIZsNrLfmWStnlejDkZ5LF+M/2VvPufRnVHaW0HwMCIaZN+hplJdSOIzjP0H4bG39+YuNzejlq63tnd7r3TNnKsv4mBlp7HkOjkuh+RgFSn5eWg4ql3ySXx21/uQnt04YfYrLks8UXGhRCAbopazXDCCrxWVDP9NUAS6sqp0pUMGvZjjJamh+1KfNnYWgF51+3KTlQnTJ4fGIfqBeu/aaE8Pr0WgUDe6y96LRF5qyuUHapn0w6/Xj/bgDfUySFemTFU6y3u90bliluEYmwblpEpc4q8GiYJI3UVo5XgK7ggs+9VSD4m5WdzU1dOCVOS2M9Ucj7dT/EzUo55Yq95Pt0u6h14pPedMKi0+zWuiyQq7Z7UVFJSka2nZO58JyhnLpCTAr/K6ULcACQ/8zkS8hefjkx+TsYD/x/HvSP/y6qmODvCPvyZAk5CM5JN/ICRkTRm6CzWA32Av+htvhm/Dt7WgYrDJ75B5C+g9Mjrgk</latexit>
sha1_base64="ZJimfBzOunvmnbFBkOMBT1E2QvE=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASClE26KbjtptmlEMcByxVX41E8zfwoM1chRuhd+kzddd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxryB89PjJ02cbm9HzFy+3XvVeb586U1nGJ8xIY89ycFwKzScoUPKz0nJQueTT/OJz60+vuHXC6BNclXyu4FyLQjBAj7JeM4CsFt8b+oGmCnBpVe1MgQqum+E0q6H5Vp80txaCXnb8qEnLpeiSw6MR3aOeXXrmxPByNBpFg9vsnWj0kaZsYZC2aR/Mev14P+6KPhTJjeiPd9O9H4SQ46z3M10YVimukUlwbpbEJc5rsCiY5E2UVo6XwC7gnM+81KC4m9fdmho68GRBC2P90Ug7+m+iBuXcSuW+sx3a3fda+D9vVmHxfl4LXVbINVtfVFSSoqHtzulCWM5QrrwAZoWflbIlWGDofybyS0juP/mhOD3YT7z+mvTHn8i6Nsg7skuGJCGHZEy+kGMyIYz8DjaD7WAn+BNuhW/Ct+vWMLjJ7JA7FdK/Xw25rQ==</latexit>
sha1_base64="E+aOEDdjm4snoMJV6Vg9dH2YVk8=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASC1E2zCbjtptmlEMcByxVX41E8yfwoM1elRugt+mTdZd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxdRA+evzk6bONzej5i5dbr3qvt0+dqSzjE2aksWc5OC6F5hMUKPlZaTmoXPJpfvm59affuXXC6BNclXyu4FyLQjBAj7JeM4CsFhcNPaSpAlxaVTtToIIfzXCa1dB8q0+aWwtBLzt+1KTlUnTJ4dGI7lHPrjxzYng1Go2iwW32TjT6SFO2MEjbtA9mvX68H3dFH4rkRvTHu+nez+vx6jjr/UoXhlWKa2QSnJslcYnzGiwKJnkTpZXjJbBLOOczLzUo7uZ1t6aGDjxZ0MJYfzTSjv6bqEE5t1K572yHdve9Fv7Pm1VYHMxrocsKuWbri4pKUjS03TldCMsZypUXwKzws1K2BAsM/c9EfgnJ/Sc/FKfv9xOvvyb98Seyrg3yjuySIUnIBzImX8gxmRBGfgebwXawE/wJt8I34dt1axjcZHbInQrpX2h3uzM=</latexit>
(q)
FC layer
<latexit sha1_base64="TIijAf6xwyNUdLTqXqoS74pSLiA=">AAAB7nicbZBNSwMxEIZn/az1q+rRS7AI9VJ2veix6MVjBfsB7VKy6bQNzWZjkhXK0h/hxYMiXv093vw3pu0etPWFwMM7M2TmjZTgxvr+t7e2vrG5tV3YKe7u7R8clo6OmyZJNcMGS0Si2xE1KLjEhuVWYFtppHEksBWNb2f11hNqwxP5YCcKw5gOJR9wRq2zWl1leOXxolcq+1V/LrIKQQ5lyFXvlb66/YSlMUrLBDWmE/jKhhnVljOB02I3NagoG9MhdhxKGqMJs/m6U3LunD4ZJNo9acnc/T2R0diYSRy5zpjakVmuzcz/ap3UDq7DjEuVWpRs8dEgFcQmZHY76XONzIqJA8o0d7sSNqKaMusSKroQguWTV6F5WQ0c3wfl2k0eRwFO4QwqEMAV1OAO6tAABmN4hld485T34r17H4vWNS+fOYE/8j5/AL3ljyc=</latexit>
sha1_base64="NKA9SV2dMMmqmgfBUmGqVGyaGuA=">AAAB7nicbZDLSgMxFIbP1Futt6pLN6FFqAhlxo0ui25cVrAX6Awlk2ba0EwmJhmhDH0INy4U0aXP4863MdN2oa0/BD7+/xxyzgklZ9q47rdTWFvf2Nwqbpd2dvf2D8qHR22dpIrQFkl4oroh1pQzQVuGGU67UlEch5x2wvFNnnceqdIsEfdmImkQ46FgESPYWKvjS81qD2f9ctWtuzOhVfAWUG1U/PMPAGj2y1/+ICFpTIUhHGvd81xpggwrwwin05KfaioxGeMh7VkUOKY6yGbjTtGpdQYoSpR9wqCZ+7sjw7HWkzi0lTE2I72c5eZ/WS810VWQMSFTQwWZfxSlHJkE5bujAVOUGD6xgIlidlZERlhhYuyFSvYI3vLKq9C+qHuW77xq4xrmKsIJVKAGHlxCA26hCS0gMIYneIFXRzrPzpvzPi8tOIueY/gj5/MH0GSQsA==</latexit>
sha1_base64="1CoPvRUhhJFoJEBI0S69bIhfd9I=">AAAB7nicbZDLSgMxFIbPeK31VnXpJrQIFaHMuNHloBuXFewFOkPJpJk2NJOJSUYoQx9CEBeKuPV53PVtTC8Lbf0h8PH/55BzTiQ508Z1J87a+sbm1nZhp7i7t39wWDo6buo0U4Q2SMpT1Y6wppwJ2jDMcNqWiuIk4rQVDW+neeuJKs1S8WBGkoYJ7gsWM4KNtVqB1Kz6eN4tVdyaOxNaBW8BFb8cXLxM/FG9W/oOeinJEioM4VjrjudKE+ZYGUY4HReDTFOJyRD3aceiwAnVYT4bd4zOrNNDcarsEwbN3N8dOU60HiWRrUywGejlbGr+l3UyE1+HORMyM1SQ+UdxxpFJ0XR31GOKEsNHFjBRzM6KyAArTIy9UNEewVteeRWalzXP8r1X8W9grgKcQhmq4MEV+HAHdWgAgSE8wxu8O9J5dT6cz3npmrPoOYE/cr5+ANnOkjY=</latexit>

Predict Answer
P
<latexit sha1_base64="IRE+WBfaQbF0wYsLhZdQBD5cHNc=">AAACgHicbVFdS+NAFJ3E3VXrflR99GXYUkjZXU18UQRB9GV9c8Faoa3hZjqxs52ZxJkbsYT8Dv+Xb/4YYadpET/2wsDhnDn3M8mlsBiGj56/9OHjp+WV1cba5y9fvzXXNy5sVhjGuyyTmblMwHIpNO+iQMkvc8NBJZL3ksnJTO/dcmNFps9xmvOhgmstUsEAHRU379sQl+JvRQ/pQAGOjSptlqKCuyroxSVUV+V59Swh6HHNn1aDfCxqZ3DaoT+o424cZ0Vw0+l0Gu1n7ytrY+6C2uQq2kLVOeY9vEwZN1vhdlgHfQ+iBWiRRZzFzYfBKGOF4hqZBGv7UZjjsASDgknuKheW58AmcM37DmpQ3A7LeoEVbTtmRNPMuKeR1uxLRwnK2qlK3M/ZOPatNiP/p/ULTPeHpdB5gVyzeaG0kBQzOrsGHQnDGcqpA8CMcL1SNgYDDN3NGm4J0duR34OL3e3I4T9R6+h4sY4VskW+k4BEZI8ckd/kjHQJI09ey/vp/fJ9P/B3/Gj+1fcWnk3yKvyDfw9kwp4=</latexit>
sha1_base64="Qqmu3kkY+vJ6TaADdy1egrh7sEE=">AAACgHicbVHLattAFB2pTZs6Ses2y2yGGINMXlI3LYWCaTfJLoE4DtiOuBqP4klmRsrMVakRgv5Ftv2WfkJ3/ZhAxlIIefTCwOGcOfeZ5FJYDMN/nv/i5dKr18tvWiura2/ftd9/OLFZYRgfsExm5jQBy6XQfIACJT/NDQeVSD5MLr8v9OEPbqzI9DHOcz5RcK5FKhigo+L2dRfiUlxU9CsdK8CZUaXNUlTwswqGcQnVWXlc3UsIelbzB9U4n4naGRz06BZ13JXjrAiuer1eq3vvfWRtNS6oTa6iLVSdo+nhYcq43Ql3wzrocxDdgU5/78/vX4SQw7j9dzzNWKG4RibB2lEU5jgpwaBgkrvKheU5sEs45yMHNShuJ2W9wIp2HTOlaWbc00hr9qGjBGXtXCXu52Ic+1RbkP/TRgWmnyel0HmBXLOmUFpIihldXINOheEM5dwBYEa4XimbgQGG7mYtt4To6cjPwcnH3cjho6jT/0aaWCYbZJMEJCKfSJ/sk0MyIIzceB1v29vxfT/w9/yo+ep7d5518ij8L7cwGsTv</latexit>
sha1_base64="FrjvqgAkJz+wsuSErm9FYCygwI4=">AAACgHicbVHLTttAFB27UGhK20CX3YyIIjlqC3Y3oEpIUbspO5AIQUpS63oyJgMzYzNzXRFZXvcTuu23dNlld/0YJCY2QryuNNLROXPuM8mlsBiG/z3/2dLy85XVF62Xa69ev2mvbxzbrDCMD1gmM3OSgOVSaD5AgZKf5IaDSiQfJudfF/rwBzdWZPoI5zmfKDjVIhUM0FFx+1cX4lKcVXSPjhXgzKjSZikquKyCYVxC9b08qm4lBD2r+f1qnM9E7Qz2e/Q9ddyF46wILnq9Xqt7671nbTUuqE2uoi1UnaPp4W7KuN0Jt8I66GMQ3YBOf/vP759/p3sHcfvfeJqxQnGNTIK1oyjMcVKCQcEkd5ULy3Ng53DKRw5qUNxOynqBFe06ZkrTzLinkdbsXUcJytq5StzPxTj2obYgn9JGBaa7k1LovECuWVMoLSTFjC6uQafCcIZy7gAwI1yvlM3AAEN3s5ZbQvRw5Mfg+NNW5PBh1Ol/IU2skndkkwQkIjukT76RAzIgjFx5He+D99H3/cDf9qPmq+/deN6Se+F/vgYfH8Zh</latexit>
a (I) = ij aij ij (I)
Sum

Re-weighted features
A · (I)
<latexit sha1_base64="9jo9WMRT48PnmFqqkN3qbCcicwM=">AAACZnicbVFNa9wwFJTdpE3cJtkmlB56EV0WdikEO5f2UkibS3NLIZsNrLfmWStnlejDkZ5LF+M/2VvPufRnVHaW0HwMCIaZN+hplJdSOIzjP0H4bG39+YuNzejlq63tnd7r3TNnKsv4mBlp7HkOjkuh+RgFSn5eWg4ql3ySXx21/uQnt04YfYrLks8UXGhRCAbopazXDCCrxWVDP9NUAS6sqp0pUMGvZjjJamh+1KfNnYWgF51+3KTlQnTJ4fGIfqBeu/aaE8Pr0WgUDe6y96LRF5qyuUHapn0w6/Xj/bgDfUySFemTFU6y3u90bliluEYmwblpEpc4q8GiYJI3UVo5XgK7ggs+9VSD4m5WdzU1dOCVOS2M9Ucj7dT/EzUo55Yq95Pt0u6h14pPedMKi0+zWuiyQq7Z7UVFJSka2nZO58JyhnLpCTAr/K6ULcACQ/8zkS8hefjkx+TsYD/x/HvSP/y6qmODvCPvyZAk5CM5JN/ICRkTRm6CzWA32Av+htvhm/Dt7WgYrDJ75B5C+g9Mjrgk</latexit>
sha1_base64="ZJimfBzOunvmnbFBkOMBT1E2QvE=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASClE26KbjtptmlEMcByxVX41E8zfwoM1chRuhd+kzddd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxryB89PjJ02cbm9HzFy+3XvVeb586U1nGJ8xIY89ycFwKzScoUPKz0nJQueTT/OJz60+vuHXC6BNclXyu4FyLQjBAj7JeM4CsFt8b+oGmCnBpVe1MgQqum+E0q6H5Vp80txaCXnb8qEnLpeiSw6MR3aOeXXrmxPByNBpFg9vsnWj0kaZsYZC2aR/Mev14P+6KPhTJjeiPd9O9H4SQ46z3M10YVimukUlwbpbEJc5rsCiY5E2UVo6XwC7gnM+81KC4m9fdmho68GRBC2P90Ug7+m+iBuXcSuW+sx3a3fda+D9vVmHxfl4LXVbINVtfVFSSoqHtzulCWM5QrrwAZoWflbIlWGDofybyS0juP/mhOD3YT7z+mvTHn8i6Nsg7skuGJCGHZEy+kGMyIYz8DjaD7WAn+BNuhW/Ct+vWMLjJ7JA7FdK/Xw25rQ==</latexit>
sha1_base64="E+aOEDdjm4snoMJV6Vg9dH2YVk8=">AAACZnicbZHNattAFIVH6l+itqmbULroZogx2ASC1E2zCbjtptmlEMcByxVX41E8yfwoM1elRugt+mTdZd1NH6MjOYQm6YWBw3fuYe7cyUspHMbxdRA+evzk6bONzej5i5dbr3qvt0+dqSzjE2aksWc5OC6F5hMUKPlZaTmoXPJpfvm59affuXXC6BNclXyu4FyLQjBAj7JeM4CsFhcNPaSpAlxaVTtToIIfzXCa1dB8q0+aWwtBLzt+1KTlUnTJ4dGI7lHPrjxzYng1Go2iwW32TjT6SFO2MEjbtA9mvX68H3dFH4rkRvTHu+nez+vx6jjr/UoXhlWKa2QSnJslcYnzGiwKJnkTpZXjJbBLOOczLzUo7uZ1t6aGDjxZ0MJYfzTSjv6bqEE5t1K572yHdve9Fv7Pm1VYHMxrocsKuWbri4pKUjS03TldCMsZypUXwKzws1K2BAsM/c9EfgnJ/Sc/FKfv9xOvvyb98Seyrg3yjuySIUnIBzImX8gxmRBGfgebwXawE/wJt8I34dt1axjcZHbInQrpX2h3uzM=</latexit>

Similar formulation for machine translation models


Source: http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture08-nmt.pdf
Source: http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture08-nmt.pdf
Attention
• General technique - not limited to computer vision or NLP
• Improves performance
• More Interpretable models
Self-attention
Self-attention (Key-value)
Very successful attention formulation by Vaswani et al.
Called the Transformer

Attention is all you need - Vaswani et al., 2017


Self-attention

Value Compute attention using the input itself


Split input into "query" and "key"
Compute attention and weigh the input with it

Query

Key
Self-attention

Value

Query

Key
Multiple heads

Value

Query

Key
Self-Attention for image generation

Image Transformer :Parmar*, Vaswani*, Uszkoreit, Kaiser, Shazeer, Ku, and Tran. ICML 2018
Self-attention everywhere
Non-local neural networks - Wang et al., 2017
• Image classification
• Action recognition

Music generation - Huang et al., 2019


Self-attention everywhere
BERT - Bi-directional Encoder Representations from Transformers (Devlin et
al., 2018)
• State-of-the-art pre-training in NLP tasks
Thanks!

You might also like