Attention map visualization

[wofmanaf]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

[zhoudaquan]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning.

[xwan6266]

Thanks for this nice work. I'm also interested with your attention map visualisation, did you use Seaborn heatmap to visualise this after training? I tried to visualise my fully trained attention weights but the outcome failed to show similar texture like the Figure 6 shows in the paper. Could you please share some details of your implementation?

[zhoudaquan]

Thanks for this nice work. I'm also interested with your attention map visualisation, did you use Seaborn heatmap to visualise this after training? I tried to visualise my fully trained attention weights but the outcome failed to show similar texture like the Figure 6 shows in the paper. Could you please share some details of your implementation?

Hi, thanks for your interest and sorry for the delay on uploading the visualization script. I am doing some other projects. Just a quick reply, I take the average over all heads before visualization and yes, I use seaborn for visualization. With default setting, there are only some white lines along the diagonal and the range keeps increasing with a diagonal center. The more clear visualization need to adjust the contrast settings carefully. Can you please show some examples of your plots? I can help with debugging a little bit hopefully. Later I can also upload mine for comparisons.

[xwan6266]

Thanks for this nice work. I'm also interested with your attention map visualisation, did you use Seaborn heatmap to visualise this after training? I tried to visualise my fully trained attention weights but the outcome failed to show similar texture like the Figure 6 shows in the paper. Could you please share some details of your implementation?

Hi, thanks for your interest and sorry for the delay on uploading the visualization script. I am doing some other projects. Just a quick reply, I take the average over all heads before visualization and yes, I use seaborn for visualization. With default setting, there are only some white lines along the diagonal and the range keeps increasing with a diagonal center. The more clear visualization need to adjust the contrast settings carefully. Can you please show some examples of your plots? I can help with debugging a little bit hopefully. Later I can also upload mine for comparisons.

Dear Daquan,
Thanks a lot for your quick reply. I finally managed to make it work similarly as yours. It turns out that I was choosing inappropriate colour during visualisation, and maybe not fully trained model for visualisation.

[cenchaojun]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning.
Dear zhoudaquan
Do you upload the code of visualization feature map?

[zhoudaquan]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning.
Dear zhoudaquan
Do you upload the code of visualization feature map?

Hi Chaojun, Thanks for your interest and really sorry for this really late reply! You can take a look at this link first where I have uploaded the code for visualization as well as the pretrained model and the saved attention map data in pickle file. I will try to update the README this weekend.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

[zhoudaquan]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi,

thanks for your interest. Please check the link for the visualizations.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

[cenchaojun]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning.
Dear zhoudaquan
Do you upload the code of visualization feature map?

Hi Chaojun, Thanks for your interest and really sorry for this really late reply! You can take a look at this link first where I have uploaded the code for visualization as well as the pretrained model and the saved attention map data in pickle file. I will try to update the README this weekend.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

thank you so much, this code will help me a lot

[ychengrong]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi,

thanks for your interest. Please check the link for the visualizations.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

Hi, Daquan! I tried the code and .pth.tar file you provided above. However, I got the output visualization for layer 1 like this.

The key to the model I used was "blocks.{layer_index}.attn.qkv.weight". Can you give me some advice about this? Appreciate that!

[zhoudaquan]

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi,
thanks for your interest. Please check the link for the visualizations.
https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

Hi, Daquan! I tried the code and .pth.tar file you provided above. However, I got the output visualization for layer 1 like this. The key to the model I used was "blocks.{layer_index}.attn.qkv.weight". Can you give me some advice about this? Appreciate that!

Please see my reply in another thread. Thanks.