NGC4929 | 195.6851 | 28.0453 | 6796 | -24.19 | 0.08 |  $<$ 0.4 |  $<$ 20.67 |
CGCG160-119 | 195.7906 | 28.5835 | 7499 | -23.82 | 0.08 |  $<$ 0.4 |  $<$ 20.74 |
NGC4934 | 195.8176 | 28.0304 | 6627 | -23.79 | 0.08 | 1.9 $\pm$ 0.0 | 21.33 | P
IC4133 | 195.9616 | 27.9883 | 6942 | -23.66 | 0.07 |  $<$ 0.4 |  $<$ 20.65 |
NGC4956 | 196.2540 | 35.1781 | 5314 | -24.86 | 0.08 |  $<$ 0.4 |  $<$ 20.43 |
NGC4971 | 196.7290 | 28.5480 | 6853 | -24.26 | 0.07 |  $<$ 0.4 |  $<$ 20.64 |
NGC4987 | 196.9961 | 51.9292 | 5209 | -23.93 | 0.06 |  $<$ 0.3 |  $<$ 20.29 |
CGCG271-017 | 197.1755 | 52.0169 | 5215 | -23.96 | 0.06 | 0.3 $\pm$ 0.0 | 20.32 | P
CGCG189-038 | 197.9894 | 34.3652 | 5585 | -23.92 | 0.07 | 4.2 $\pm$ 0.0 | 21.53 | P
ARK407 | 198.0330 | 27.3305 | 6741 | -24.09 | 0.09 |  $<$ 0.4 |  $<$ 20.70 |
NGC5032B | 198.3570 | 27.7635 | 6762 | -23.54 | 0.07 | 5.0 $\pm$ 0.5 | 21.77 | P
NGC5052 | 198.8952 | 29.6761 | 7212 | -24.57 | 0.13 | 4.6 $\pm$ 0.0 | 21.79 | P
IC4226 | 200.1267 | 32.0039 | 6093 | -23.52 | 0.09 |  $<$ 0.5 |  $<$ 20.64 |
NGC5142 | 201.2547 | 36.3996 | 6181 | -24.68 | 0.09 |  $<$ 0.5 |  $<$ 20.65 |
NGC5198 | 202.5475 | 46.6708 | 3062 | -24.37 | 0.05 | 2.0 $\pm$ 0.0 | 20.68 | P
NGC5216 | 203.0286 | 62.7008 | 3399 | -23.62 | 0.08 | 8.5 $\pm$ 0.0 | 21.40 | P
NGC5225 | 203.3339 | 51.4903 | 5115 | -23.80 | 0.06 | 6.4 $\pm$ 0.0 | 21.63 | E
UGC08564 | 203.7406 | 38.4575 | 6799 | -24.79 | 0.06 |  $<$ 0.3 |  $<$ 20.54 |
NGC5250 | 204.0306 | 51.2358 | 5025 | -24.29 | 0.05 | 0.6 $\pm$ 0.2 | 20.59 | P
CGCG161-101 | 204.6797 | 31.2706 | 5342 | -23.77 | 0.10 | 1.2 $\pm$ 0.5 | 20.94 | P
CGCG161-102 | 204.7460 | 31.3652 | 5182 | -23.55 | 0.10 |  $<$ 0.5 |  $<$ 20.53 |
NGC5283 | 205.2739 | 67.6722 | 3546 | -23.61 | 0.06 | 63.7 $\pm$ 0.3 | 22.31 | P
NGC5308 | 206.7517 | 60.9731 | 2493 | -24.46 | 0.05 |  $<$ 0.3 |  $<$ 19.63 |
NGC5311 | 207.2336 | 39.9852 | 3266 | -24.15 | 0.07 | 146.9 $\pm$ 0.0 | 22.60 | P
NGC5312 | 207.4608 | 33.6217 | 4897 | -23.70 | 0.07 | 9.4 $\pm$ 0.2 | 21.76 | P
CGCG218-055 | 207.5414 | 38.2196 | 6364 | -23.75 | 0.08 | 2.5 $\pm$ 2.1 | 21.41 | P
NGC5318 | 207.6501 | 33.7048 | 4919 | -24.81 | 0.11 | 352.2 $\pm$ 0.0 | 23.34 | P
UGC08754 | 207.6629 | 35.0382 | 6905 | -24.65 | 0.08 |  $<$ 0.4 |  $<$ 20.66 |
NGC5370 | 208.5390 | 60.6780 | 3530 | -23.54 | 0.06 |  $<$ 0.3 |  $<$ 19.97 |
NGC5361 | 208.6467 | 38.4495 | 6212 | -23.56 | 0.07 | 19.3 $\pm$ 0.0 | 22.28 | E
UGC08854 | 208.9727 | 37.1965 | 6551 | -24.60 | 0.06 | 1.0 $\pm$ 0.0 | 21.04 | P
NGC5380 | 209.2362 | 37.6104 | 3749 | -24.36 | 0.08 |  $<$ 0.4 |  $<$ 20.14 |
UGC08923 | 210.0654 | 38.5031 | 5650 | -23.93 | 0.08 | 0.8 $\pm$ 0.3 | 20.82 | P
NGC5422 | 210.1752 | 55.1645 | 2366 | -23.94 | 0.05 | 0.3 $\pm$ 0.0 | 19.63 | P
NGC5407 | 210.2088 | 39.1562 | 5966 | -24.37 | 0.10 |  $<$ 0.5 |  $<$ 20.65 |
CGCG219-044 | 210.3954 | 38.6861 | 6218 | -24.27 | 0.08 | 1.7 $\pm$ 0.0 | 21.23 | P
NGC5473 | 211.1801 | 54.8926 | 2548 | -24.50 | 0.05 | 0.3 $\pm$ 0.9 | 19.70 | P
NGC5485 | 211.7972 | 55.0016 | 2455 | -24.38 | 0.05 | 4.5 $\pm$ 0.0 | 20.84 | P
NGC5580 | 215.4100 | 35.2049 | 3795 | -24.08 | 0.09 |  $<$ 0.5 |  $<$ 20.23 |
NGC5598 | 215.6179 | 40.3198 | 6026 | -24.71 | 0.15 |  $<$ 0.8 |  $<$ 20.85 |
NGC5603 | 215.7563 | 40.3773 | 6221 | -24.82 | 0.13 | 3.0 $\pm$ 0.0 | 21.47 | P
UGC09223 | 215.9370 | 40.0238 | 5789 | -24.35 | 0.15 |  $<$ 0.8 |  $<$ 20.81 |
NGC5631 | 216.6387 | 56.5826 | 2465 | -24.33 | 0.06 | 3.3 $\pm$ 0.0 | 20.71 | P
UGC09284 | 217.0974 | 33.2537 | 4753 | -23.52 | 0.08 | 2.9 $\pm$ 0.0 | 21.23 | P
NGC5642 | 217.3062 | 30.0264 | 4916 | -24.95 | 0.09 |  $<$ 0.5 |  $<$ 20.45 |
NGC5687 | 218.7183 | 54.4757 | 2669 | -24.01 | 0.06 |  $<$ 0.3 |  $<$ 19.75 |
NGC5684 | 218.9590 | 36.5432 | 4668 | -24.46 | 0.06 | 7.6 $\pm$ 0.6 | 21.63 | P
CGCG220-034 | 219.1776 | 39.9436 | 6563 | -23.94 | 0.08 | 13.9 $\pm$ 0.8 | 22.19 | P
CGCG318-001 | 219.5392 | 66.3380 | 7100 | -24.29 | 0.09 | 10.2 $\pm$ 0.0 | 22.12 | P
UGC09473 | 220.3874 | 38.8514 | 5268 | -24.56 | 0.07 | 23.9 $\pm$ 0.2 | 22.23 | E
NGC5771 | 223.0597 | 29.8454 | 7463 | -24.41 | 0.08 |  $<$ 0.4 |  $<$ 20.75 |
NGC5820 | 224.6659 | 53.8861 | 3843 | -24.62 | 0.07 | 1.3 $\pm$ 0.3 | 20.69 | P
UGC09641 | 224.7622 | 41.4468 | 5468 | -23.94 | 0.07 |  $<$ 0.3 |  $<$ 20.40 |
NGC5860 | 226.6400 | 42.6403 | 5958 | -23.93 | 0.08 | 34.9 $\pm$ 0.0 | 22.50 | P
NGC5943 | 232.4338 | 42.7779 | 6204 | -24.98 | 0.06 | 6.4 $\pm$ 0.0 | 21.80 | P
CGCG222-025 | 233.3049 | 41.0766 | 6161 | -23.89 | 0.07 |  $<$ 0.3 |  $<$ 20.53 |
NGC5966 | 233.9671 | 39.7689 | 5034 | -24.67 | 0.08 | 4.2 $\pm$ 0.0 | 21.44 | P
CGCG194-013 | 234.6603 | 36.9583 | 6150 | -24.01 | 0.09 | 18.4 $\pm$ 0.0 | 22.25 | P
CGCG166-067 | 238.5479 | 30.1559 | 6605 | -23.63 | 0.08 |  $<$ 0.4 |  $<$ 20.67 |
CGCG250-023 | 239.0321 | 48.0932 | 6291 | -24.03 | 0.06 |  $<$ 0.3 |  $<$ 20.48 |
IC1152 | 239.1805 | 48.0950 | 6370 | -24.94 | 0.06 | 1.1 $\pm$ 0.4 | 21.06 | P
UGC10118 | 239.6678 | 48.6824 | 6494 | -24.46 | 0.06 |  $<$ 0.3 |  $<$ 20.50 |
IC1214 | 244.0488 | 65.9688 | 7364 | -24.76 | 0.09 |  $<$ 0.4 |  $<$ 20.78 |
NGC6187 | 247.9033 | 57.7067 | 5899 | -23.69 | 0.06 |  $<$ 0.3 |  $<$ 20.42 |
NGC6198 | 248.8777 | 57.4868 | 5716 | -23.61 | 0.05 | 0.9 $\pm$ 0.0 | 20.88 | P
UGC10486 | 249.3930 | 50.3455 | 6537 | -24.82 | 0.06 | 0.4 $\pm$ 0.0 | 20.64 | P
NGC6206 | 250.0329 | 58.6174 | 5927 | -24.26 | 0.07 |  $<$ 0.3 |  $<$ 20.48 |
NGC6211 | 250.3652 | 57.7836 | 5693 | -24.92 | 0.06 | 103.6 $\pm$ 1.9 | 22.94 | E
CGCG251-041 | 250.7894 | 45.8458 | 5782 | -23.67 | 0.07 | 2.0 $\pm$ 0.0 | 21.23 | P
CGCG299-027 | 251.5536 | 62.2301 | 6048 | -23.59 | 0.07 |  $<$ 0.3 |  $<$ 20.49 |
NGC6291 | 255.2330 | 58.9375 | 5596 | -23.67 | 0.09 |  $<$ 0.4 |  $<$ 20.56 |
CGCG226-037 | 265.1584 | 41.3014 | 5805 | -23.74 | 0.09 |  $<$ 0.5 |  $<$ 20.60 |
CGCG171-025 | 268.7725 | 28.9595 | 7225 | -24.11 | 0.19 |  $<$ 1.0 |  $<$ 21.11 |
NGC6518 | 269.9323 | 28.8667 | 7189 | -24.87 | 0.24 |  $<$ 1.2 |  $<$ 21.20 |
CGCG254-009 | 270.4529 | 45.3105 | 6206 | -23.72 | 0.08 |  $<$ 0.4 |  $<$ 20.61 |
CGCG171-048 | 270.7378 | 29.3107 | 7420 | -24.57 | 0.23 | 11.9 $\pm$ 0.0 | 22.23 | P
ARK538 | 273.7654 | 29.7697 | 6146 | -23.82 | 0.19 | 43.4 $\pm$ 0.4 | 22.62 | P
CGCG227-023 | 274.4821 | 41.3557 | 6259 | -23.70 | 0.17 |  $<$ 0.9 |  $<$ 20.94 |
UGC11228 | 276.1928 | 41.4927 | 6167 | -24.84 | 0.15 | 9.8 $\pm$ 0.0 | 21.98 | P
UGC11233 | 276.4343 | 32.1788 | 7212 | -24.69 | 0.13 |  $<$ 0.6 |  $<$ 20.92 |
IC4772 | 279.9854 | 40.0264 | 7470 | -24.35 | 0.41 |  $<$ 2.0 |  $<$ 21.47 |
NGC6686 | 280.0292 | 40.1376 | 7430 | -24.56 | 0.41 | 4.0 $\pm$ 0.0 | 21.75 | P
CGCG228-027 | 281.2488 | 42.7197 | 6941 | -23.95 | 0.14 |  $<$ 0.7 |  $<$ 20.93 |
CGCG228-030 | 281.5922 | 40.5821 | 6941 | -24.42 | 0.34 | 6.9 $\pm$ 0.0 | 21.93 | P
CGCG493-015 | 329.2996 | 30.7303 | 7107 | -24.55 | 0.11 | 1.3 $\pm$ 0.0 | 21.23 | P
UGC11903 | 331.3274 | 34.9892 | 4890 | -23.75 | 0.09 |  $<$ 0.4 |  $<$ 20.44 |
CGCG513-017 | 333.4549 | 37.0267 | 6334 | -24.51 | 0.15 | 15.5 $\pm$ 0.0 | 22.20 | E
WISEAJ221408.43+365115.7 | 333.5352 | 36.8544 | 6101 | -24.14 | 0.10 |  $<$ 0.5 |  $<$ 20.69 |
NGC7240 | 333.8440 | 37.2807 | 6051 | -24.52 | 0.13 |  $<$ 0.7 |  $<$ 20.80 |
WISEAJ221534.77+371106.7 | 333.8949 | 37.1853 | 5328 | -23.59 | 0.13 | 0.8 $\pm$ 0.0 | 20.77 | P
IC5193 | 333.9314 | 37.2429 | 5661 | -24.21 | 0.13 |  $<$ 0.6 |  $<$ 20.71 |
UGC11990 | 335.1254 | 35.9736 | 5895 | -24.60 | 0.10 |  $<$ 0.5 |  $<$ 20.65 |
CGCG494-019 | 335.7883 | 32.1487 | 6531 | -24.18 | 0.09 |  $<$ 0.5 |  $<$ 20.70 |
NGC7273 | 336.0384 | 36.1998 | 5144 | -24.61 | 0.12 | 3.2 $\pm$ 0.2 | 21.34 | P
NGC7276 | 336.0598 | 36.0875 | 6033 | -24.52 | 0.11 | 84.5 $\pm$ 0.0 | 22.90 | E
CGCG514-029 | 336.5029 | 35.9537 | 6184 | -24.42 | 0.13 | 1.0 $\pm$ 0.0 | 20.99 | P
UGC12040 | 336.7741 | 36.3616 | 6435 | -24.93 | 0.16 | 24.2 $\pm$ 0.1 | 22.41 | P
CGCG514-036 | 337.1428 | 33.9992 | 6132 | -24.48 | 0.19 |  $<$ 0.9 |  $<$ 20.96 |
WISEAJ222843.62+362955.8 | 337.1817 | 36.4989 | 7338 | -24.54 | 0.16 | 3.8 $\pm$ 0.4 | 21.72 | P
CGCG514-041 | 337.4832 | 36.2613 | 7149 | -24.43 | 0.16 |  $<$ 0.8 |  $<$ 21.03 |
NGC7317 | 338.9662 | 33.9449 | 6621 | -24.36 | 0.12 |  $<$ 0.6 |  $<$ 20.84 |
NGC7332 | 339.3522 | 23.7983 | 1383 | -23.53 | 0.16 | 2.7 $\pm$ 0.4 | 20.12 | P
UGC12121 | 339.4184 | 34.8465 | 6954 | -24.64 | 0.13 |  $<$ 0.6 |  $<$ 20.90 |
NGC7340 | 339.4342 | 34.4100 | 6424 | -24.76 | 0.11 |  $<$ 0.6 |  $<$ 20.77 |
CGCG514-079 | 339.6269 | 34.2406 | 6030 | -24.38 | 0.10 | 28.9 $\pm$ 0.0 | 22.43 | P
CGCG495-013 | 340.3148 | 33.6237 | 6575 | -23.84 | 0.22 |  $<$ 1.1 |  $<$ 21.09 |
CGCG514-094 | 340.3452 | 34.6217 | 6854 | -24.55 | 0.08 | 3.5 $\pm$ 0.0 | 21.62 | P
MRK0918 | 340.5420 | 35.9276 | 7259 | -24.31 | 0.11 |  $<$ 0.6 |  $<$ 20.88 |
CGCG495-033 | 342.3888 | 32.3674 | 6495 | -24.09 | 0.09 |  $<$ 0.4 |  $<$ 20.68 |
NGC7395 | 342.7621 | 37.0879 | 5700 | -24.85 | 0.18 |  $<$ 0.9 |  $<$ 20.86 |
ARK566 | 342.7647 | 31.3744 | 7057 | -23.91 | 0.09 |  $<$ 0.4 |  $<$ 20.76 |
CGCG515-010 | 343.3530 | 37.1753 | 5691 | -24.45 | 0.12 |  $<$ 0.6 |  $<$ 20.70 |
UGC12235 | 343.5175 | 32.3734 | 6666 | -24.26 | 0.09 | 4.7 $\pm$ 0.0 | 21.73 | P
NGC7411 | 343.6455 | 20.2362 | 6810 | -24.76 | 0.28 | 19.9 $\pm$ 0.0 | 22.37 | P
CGCG496-020 | 343.7761 | 32.1847 | 6533 | -24.27 | 0.10 | 4.3 $\pm$ 0.0 | 21.67 | P
CGCG496-031 | 345.1618 | 30.7418 | 6296 | -24.32 | 0.09 |  $<$ 0.4 |  $<$ 20.65 |
NGC7454 | 345.2772 | 16.3884 | 2122 | -23.63 | 0.18 |  $<$ 0.9 |  $<$ 20.01 |
NGC7473 | 345.9879 | 30.1602 | 6929 | -24.63 | 0.08 |  $<$ 0.4 |  $<$ 20.71 |
UGC12356 | 346.4541 | 31.0872 | 6886 | -24.91 | 0.09 |  $<$ 0.5 |  $<$ 20.75 |
NGC7485 | 346.5202 | 34.1077 | 5707 | -24.82 | 0.11 | 104.7 $\pm$ 0.0 | 22.94 | P
CGCG475-041 | 347.5689 | 26.6210 | 7417 | -24.30 | 0.17 | 1.6 $\pm$ 0.0 | 21.35 | P
NGC7516 | 348.2161 | 20.2484 | 4980 | -24.53 | 0.12 |  $<$ 0.6 |  $<$ 20.59 |
CGCG496-062 | 348.5978 | 28.4445 | 6864 | -24.29 | 0.15 |  $<$ 0.8 |  $<$ 20.96 |
CGCG496-077 | 349.4413 | 29.0272 | 6420 | -23.98 | 0.14 |  $<$ 0.7 |  $<$ 20.86 |
CGCG496-078 | 349.5641 | 29.2306 | 6678 | -24.81 | 0.15 | 6.1 $\pm$ 0.0 | 21.84 | P
WISEAJ231854.65+290123.5 | 349.7277 | 29.0232 | 7226 | -24.19 | 0.11 |  $<$ 0.6 |  $<$ 20.88 |
WISEAJ232001.11+271247.8 | 350.0046 | 27.2133 | 5565 | -23.78 | 0.11 |  $<$ 0.5 |  $<$ 20.63 |
NGC7628 | 350.2287 | 25.8986 | 4176 | -24.20 | 0.11 | 2.1 $\pm$ 0.0 | 20.97 | P
IC5315 | 350.3260 | 25.3854 | 4435 | -23.55 | 0.10 | 3.3 $\pm$ 0.0 | 21.22 | P
CGCG497-012 | 350.8932 | 29.1824 | 6193 | -24.32 | 0.13 | 19.7 $\pm$ 0.0 | 22.29 | P
CGCG454-072 | 352.1258 | 17.2779 | 7251 | -24.33 | 0.13 |  $<$ 0.6 |  $<$ 20.93 |
CGCG497-026 | 352.2046 | 29.7353 | 5555 | -24.17 | 0.10 | 1.3 $\pm$ 0.3 | 21.01 | P
UGC12636 | 352.5797 | 27.1337 | 7261 | -23.85 | 0.10 |  $<$ 0.5 |  $<$ 20.81 |
UGC12655 | 353.1211 | 23.9311 | 5140 | -24.59 | 0.09 | 24.2 $\pm$ 0.0 | 22.21 | E
UGC12657 | 353.1812 | 29.4607 | 5557 | -24.71 | 0.10 | 1.3 $\pm$ 0.0 | 21.01 | P
NGC7698 | 353.5065 | 24.9446 | 6951 | -24.95 | 0.08 |  $<$ 0.4 |  $<$ 20.69 |
CGCG497-046 | 354.7851 | 32.0856 | 5105 | -24.75 | 0.12 |  $<$ 0.6 |  $<$ 20.60 |
WISEAJ233932.46+285506.2 | 354.8853 | 28.9183 | 6460 | -24.13 | 0.09 |  $<$ 0.4 |  $<$ 20.66 |
CGCG498-007 | 356.2788 | 28.2055 | 6919 | -24.58 | 0.09 | 13.4 $\pm$ 0.0 | 22.22 | P
NGC7760 | 357.2997 | 30.9832 | 5203 | -24.54 | 0.08 |  $<$ 0.4 |  $<$ 20.46 |
UGC12835 | 358.4864 | 28.4927 | 6810 | -24.92 | 0.09 | 1.7 $\pm$ 0.0 | 21.31 | P
UGC12840 | 358.6246 | 28.8715 | 6765 | -24.53 | 0.12 |  $<$ 0.6 |  $<$ 20.85 |

Column description: (1) name, (2 and 3) right ascension and declination, (4)
recession velocity ($\rm{\,km\,s}^{-1}$), (5) K-band absolute magnitude, (6)
r.m.s. noise of the LOFAR image (mJy beam-1), (7 and 8) flux density (mJy) and
luminosity (W Hz-1) at 150 MHz from LOFAR.


# AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker
Extraction

###### Abstract

Visual information can serve as an effective cue for target speaker extraction
(TSE) and is vital to improving extraction performance. In this paper, we
propose AV-SepFormer, a SepFormer-based attention dual-scale model that
utilizes cross- and self-attention to fuse and model features from audio and
visual. AV-SepFormer splits the audio feature into a number of chunks,
equivalent to the length of the visual feature. Then self- and cross-attention
are employed to model the multi-modal features. Furthermore, we use a novel 2D
positional encoding, that introduces the positional information between and
within chunks and provides significant gains over the traditional positional
encoding. Our model has two key advantages: the time granularity of audio
chunked feature is synchronized to the visual feature, which alleviates the
harm caused by the inconsistency of audio and video sampling rate; by
combining self- and cross-attention, feature fusion and speech extraction
processes are unified within an attention paradigm. The experimental results
show that AV-SepFormer significantly outperforms other existing methods.

Index Terms—  Target speech extraction, multi-modal fusion, cross-attention,
2D positional encoding.

##  1 Introduction

Target speaker extraction (TSE) aims to extract a specific character’s speech
signal in the presence of multiple talkers and background noises. It has a
wide range of real-world applications, such as hearing aids [1], speech
recognition [2], speaker verification [3], speaker diarization [4] and voice
surveillance [5]. Unlike blind speech separation (BSS), TSE methods are not
restricted by permutation invariant training (PIT) [6] and require additional
clue information to identify the target speaker. Numerous techniques have been
created by researchers to make use of various forms of cues, including
reference speech [7, 8, 9], still speaker face image [10], and lip movement
[11, 12, 13, 14, 10].

Among these cues, lip movement is more advantageous than the others. The
reasons are as follows: 1) Audible background noise and reverberation have no
impact on the visual lip movement information. 2) Speech can be directly
inferred from lip movement, and thus spoken content information can be
effectively extracted. 3) Approaches that use visual cues are enrollment-free,
making them easier to use in practice. In this paper, we focus on enhancing
the performance of lip movement based Audio-Visual TSE, for convenience we
refer to this lip movement based Audio-Visual TSE as Audio-Visual TSE in the
following.

Existing Audio-Visual TSE models typically up-sample the visual feature to
align with the audio feature in time dimension [10, 11], which means these two
modal features are fused without taking any difference in temporal granularity
between visual and audio into account. This may lead to a degradation of TSE
performance since the audio feature is fine-grained, whereas the visual
feature tends to be coarse-grained. Regarding the data types frequently
employed in Audio-Visual TSE tasks, the face track video stream is available
at 25 FPS (Frames Per Second), and the audio data commonly have a sampling
rate of 8 or 16 kHz, which commonly is processed by a short time Fourier
transform (STFT) to about 50 to 125 FPS [15, 16]. Meanwhile, other time domain
approaches using a trainable front-end instead of STFT commonly produce
features at 200 FPS [12, 13], leading to a more significant disparity of time
granularity between audio and visual features.

Therefore, it is vital to consider integrating audio and visual features at
the appropriate temporal granularity for Audio-Visual TSE models to perform
even better.

To this end, we propose AV-SepFormer, a dual-scale model for TSE at both
visual and audio scales. Inspired by DPRNN [17] and SepFormer [18], we split
the audio feature sequence into shorter chunks such that the number of audio
feature chunks matches the length of the visual feature sequence. In this way,
the audio chunked feature has comparable temporal granularity to the visual
feature. Three modules—IntraTransformer, InterTransformer and
CrossModalTransformer, are interleaved for further modeling. The former two
modules use self-attention at intra- and inter-chunk levels for short- and
long-distance modeling, while the latter uses cross-attention for feature
fusion at the inter-chunk level for the reason that long-term temporal context
is essential to learn the audio-visual relationship [19]. Moreover, we
introduce a 2D positional encoding method [20] to help our attention-based
model learn the 2D spatial relationship at intra- and inter-chunk levels,
making it possible for cross-modal fusion to differentiate consecutive audio
frames more clearly at the intra-chunk level. Experimental results on
VoxCeleb2 [21], LRS3 [22] and TCD-TIMIT [23] datasets show that our AV-
SepFormer consistently outperforms other existing advanced Audio-Visual TSE
models in terms of signal and perceptual quality.

##  2 Methodology

Fig. 1:  Overall structure of AV-SepFormer. AV-SepFormer consists of six
parts: Visual Encoder, Audio Encoder, Chunk, Separator, OverlapAdd and Audio
Decoder.

Following the paradigm of the time-domain TSE model [11], our proposed AV-
SepFormer employs an Audio Encoder, a Visual Encoder, an Audio Decoder, and a
Separator. The overall structure of AV-SepFormer is shown in Figure 1.

###  2.1 Audio Encoder

The Audio Encoder extracts the audio feature $\boldsymbol{H_{x}}$ from a
speech mixture $\boldsymbol{x}\in\mathbb{R}^{1\times T}$ using the 1D
convolution operation with a kernel size $L$ and stride $L/2$:

$\boldsymbol{H_{x}}=\text{Conv1D}(\boldsymbol{x},L,L/2)\in\mathbb{R}^{N\times
K},$ (1)

where $N$ is the audio feature dimension and $K=\frac{T-L}{L/2}+1$. This
approach is also known as adaptive front-end [17], which seeks to replace the
short-time Fourier transform (STFT) with a differentiable transform to learn a
trainable feature representation from the .

For the 2D audio feature $\boldsymbol{H_{x}}$, Chunk operation splits
$\boldsymbol{H_{x}}$ into chunks of length $C$ and hop size $C/2$. The padding
operation is the same as which in SepFormer [18]. All chunks are then
concatenated together to form a 3D audio chunked feature
$\boldsymbol{H_{x}^{\prime}}$:

$\boldsymbol{H_{x}^{\prime}}=\textrm{Chunk}\left(\boldsymbol{H_{x}}\right)\in\mathbb{R}^{N\times
C\times I},$ (2)

where $I$ indicates the number of chunks, it is designed to be exactly
equivalent to the length of visual feature $\boldsymbol{H_{v}}$.

###  2.2 Visual Encoder

Our Visual Encoder follows previous Audio-Visual TSE [11, 13, 12] methods,
which use a pre-trained fixed-parameter lip embedding extractor consisting of
a 3D convolution layer and an 18-layer ResNet [24] combined with multi-layer
TCN networks. Its detailed structure is shown in Figure 2. Specifically, the
Visual Encoder takes in the grey-scale image sequence $\boldsymbol{v}$ as
input and outputs visual feature $\boldsymbol{H_{v}}\in\mathbb{R}^{N\times
I}$, where $N$ represents the visual feature dimension (equivalent to the
audio feature dimension), and $I$ indicates the length of visual feature.

Fig. 2: Detailed structure of the Visual Encoder.

###  2.3 Separator

As shown in Figure 3, the Separator of our AV-SepFormer consists of three
parts: IntraTransformer, CrossModalTransformer and InterTransformer. The
number of heads in each multi-head attention in these modules is abbreviated
as $N_{\text{head}}$. IntraTransformer aims to model the audio chunked feature
$\boldsymbol{H^{\prime}_{x}}$ at fine-grained intra-chunk level. The audio
chunked feature $\boldsymbol{H^{\prime}_{x}}$ is progressively refined through
$N_{\text{intra}}$ repeated blocks. Inside each block, self-attention is
applied to each of the $C$ chunks respectively:

$\boldsymbol{H^{\prime\prime}_{x}}[:,\ :,\
i]={\textrm{IntraTransformer}}(\boldsymbol{H^{\prime}_{x}}[:,\ :,\ i]),$ (3)

where $\boldsymbol{H^{\prime\prime}_{x}}\in\mathbb{R}^{N\times C\times I}$ ,
$i\in\\{1,2,\cdots,I\\}$.

CrossModalTransformer seeks to fuse audio and visual features at the same time
granularity. Before the modal fusion phase, $\boldsymbol{H_{v}}$ is linearly
converted to $\boldsymbol{Q_{v}}$, while $\boldsymbol{H_{x}^{\prime\prime}}$
is linearly converted to $\boldsymbol{K_{x}},\boldsymbol{V_{x}}$:

$\boldsymbol{Q_{v}}=\textrm{Linear}(\boldsymbol{H_{v}})\in\mathbb{R}^{N\times
I},$ (4) $\boldsymbol{K_{xc}},\
\boldsymbol{V_{xc}}=\textrm{Linear}(\boldsymbol{H_{x}^{\prime\prime}}[:,\ c,\
:])\in\mathbb{R}^{N\times I},$ (5)

where $c\in\\{1,2,\cdots,C\\}$. After that,
$\boldsymbol{K_{x}},\boldsymbol{V_{x}},\boldsymbol{Q_{x}}$ are fed as the key,
value and query respectively into CrossModalTransformer to output the fusion
feature $\boldsymbol{H_{f}}\in\mathbb{R}^{N\times C\times I}$:

$\boldsymbol{H_{f}}[:,\ c,\
:]=\text{CrossModalTransformer}(\boldsymbol{Q}_{v},\boldsymbol{K}_{xc},\boldsymbol{V}_{xc}).$
(6)

InterTransformer focuses on modeling the fusion feature at the coarse-grained
inter-chunk level. Similar to the IntraTransformer, InterTransformer consists
of $N_{\text{inter}}$ blocks, self-attention is applied inside each block. The
difference is that we do self-attention for each slice on the $C$ dimension,
and get chunked mask $\boldsymbol{M^{\prime}}$:

$\boldsymbol{M^{\prime}}[:,\ c,\
:]=\textrm{InterTransformer}(\boldsymbol{H_{f}}[:,\ c,\ :]),$ (7)

where $\boldsymbol{M^{\prime}}\in[0,1]^{N\times C\times I}$,
$c\in\\{1,2,\cdots,C\\}$.

Fig. 3: Detailed structure of Separator. The whole Separator consists of three
parts: IntraTransformer, InterTransformer and CrossModalTransformer.

###  2.4 2D Positional Encoding

Positional encoding is considered essential when it comes to transformer-based
models. However, the original positional encoding in [25] is usually
considered for encoding 2D features like $\boldsymbol{H_{x}}$ and
$\boldsymbol{H_{v}}$. Due to the 3D shape of $\boldsymbol{H^{\prime}_{x}}$, it
is not applicable in AV-SepFormer, otherwise, positional encoding applied to
every chunk of $\boldsymbol{H^{\prime}_{x}}$ would be identical as is in
Figure 4(a). This can not help us to take advantage of more detailed
positional information inside every chunk, and thus may lead to a degradation
of performance.

Thus we introduce 2D positional encoding [20], which uses the horizontal and
vertical coordinates of the vectors in the 2D matrix to calculate the
positional encoding values:

$\displaystyle PE(c,\ i,\ 2u)=\sin\frac{c}{10000^{\frac{4u}{N}}},$ (8)
$\displaystyle PE(c,\ i,\ 2u+1)=\cos\frac{c}{10000^{\frac{4u}{N}}},$
$\displaystyle PE\left(c,\ i,\
2v+\frac{N}{2}\right)=\sin\frac{i}{10000^{\frac{4v}{N}}},$ $\displaystyle
PE\left(c,\ i,\ 2v+1+\frac{N}{2}\right)=\cos\frac{i}{10000^{\frac{4v}{N}}},$

where $c\in\\{1,2,\cdots,C\\}$ and $i\in\\{1,2,\cdots,I\\}$ represent the
position within and between the chunks, $u,\ v\in\left[0,\frac{N}{4}\right]$
denote the position in feature dimension. For audio chunked feature
$\boldsymbol{H^{\prime}_{x}}\in\mathbb{R}^{N\times C\times I}$, simply add
$PE(c,i,\cdot)$ to it by dimension. For visual feature
$\boldsymbol{H_{v}}\in\mathbb{R}^{N\times I}$, due to its 2D shape, we choose
to fix $c=\frac{C}{2}$ and remain $i$ corresponding to its $I$ dimension, add
$PE(\frac{C}{2},i,\cdot)$ to it by dimension at last. Figure 4(b) shows the
detailed process.

###  2.5 Audio Decoder

OverlapAdd operation seeks to reconstruct 2D feature from chunked 3D feature.
It can be regarded as the inverse of the Chunk operation in Section 2.1,

$\boldsymbol{M}=\textrm{OverlapAdd}(\boldsymbol{M^{\prime}})\in[0,1]^{N\times
K}.$ (9)

The input to the Audio Decoder is the element-wise multiplication between the
mask $\boldsymbol{M}$ of the target speaker and the output
$\boldsymbol{H_{x}}$ of the Audio Encoder:

$\vspace{-3mm}\boldsymbol{{H}_{\hat{s}}}=\boldsymbol{H_{x}}\otimes\boldsymbol{M}\in\mathbb{R}^{N\times
K},\vspace{-3mm}$ (10)

where $\otimes$ means element-wise multiplication.

(a) 1D positional encoding in AV-SepFormer

(b) 2D positional encoding in AV-SepFormer

Fig. 4: Difference between using 1D and 2D positional encoding in our AV-
SepFormer. $PE(\cdot)$ means positional encoding. Every chunks’
($\boldsymbol{H_{x}^{\prime}}$) positional encoding is identical in the 1D
case. Note that this is a slice on feature dimension $N$.

The Audio Decoder utilizes a transposed convolution layer, with the same
stride and kernel size of the Audio Encoder in Section 2.1 to reconstruct the
target speech $\hat{s}$ from $\boldsymbol{H_{\hat{s}}}$:

$\hat{s}=\textrm{TransposedConv1d}(\boldsymbol{{H}_{\hat{s}}})\in\mathbb{R}^{1\times
T}.$ (11)

##  3 Experiments

###  3.1 Datasets

We first evaluate AV-SepFormer on the VoxCeleb2 dataset [21]. The identities
of speakers in the train and test sets have no overlap. All utterances have a
duration between 4 and 6 seconds. We select 48,000 utterances from 800
speakers in the train set and 36,237 utterances from 118 speakers in the test
set. Then we simulate a 2-speaker mixture speech dataset of 20,000, 5,000, and
3,000 utterances for train, validation and test sets, respectively. Each
interfering speech is mixed with the target speech at a random Signal-to-Noise
ratio (SNR) set between -10 and 10 dB. We also compare AV-SepFormer with
VisualVoice [10], AV-ConvTasnet [11] and MuSE [12] on cross-domain datasets,
namely LRS3 [22] and TCD-TIMIT [23], which are created from TED videos and
studio videos respectively.

Table 1: Hyperparameters of AV-SepFormer

Symbol | Description | Value
---|---|---
$L$ | Kernel size in Conv1D of audio encoder | 16
$C$ | Length of the chunk | 160
$N_{\text{intra}}$ | Number of the IntraTransformer | 8
$N_{\text{inter}}$ | Number of the InterTransformer | 7
$N$ | Feature dimension of visual and audio feature | 256
$N_{\text{head}}$ | Number of head in multi-head attention | 8

Table 2: The comparison across various Audio-Visual TSE models on VoxCeleb2
2-speaker mixture.

Network | Domain | Auxiliary input | SI-SDR (dB) | PESQ
---|---|---|---|---
VisualVoice [10] | T-F | L(lip) + F(face) | 9.732 | 1.965
AV-ConvTasNet [11] | T | L(lip) | 10.381 | 1.973
AV-ConvTasnet (att) | T | L(lip) | 11.332 | 2.237
MuSE [12] | T | L(lip) + S(label) | 11.240 | 2.201
AV-SepFormer (ours) | T | L(lip) | 12.130 | 2.313

###  3.2 Implementation setup

We implement an AV-SepFormer111Code and demo available at
https://github.com/lin9x/AV-SepFormer system as described in Section 2. The
visual feature is extracted from the input video and resampled to 25 FPS. The
audio is synchronized with the video and sampled at 16 kHz. The hyper-
parameters of AV-SepFormer are shown in Table 1. Note that here we fix $C=160$
to ensure that the audio chunked feature and the visual feature are aligned.

The model is trained using an Adam optimizer [26] with an initial learning
rate of $1.5\text{\times}{10}^{-4}$. The learning rate will get halved if
there is no loss decrease on validation set for $3$ epochs. If there is no
loss decrease for $5$ epochs, the whole training process stops.

###  3.3 Comparison with other state-of-the-art methods

We first compare the objective performance of AV-SepFormer with other state-
of-the-art methods on VoxCeleb2. The methods include one time-frequency domain
method (VisualVoice [10]) and two time domain methods (AV-ConvTasnet [11],
MuSE [12]). Note that time-frequency domain method means using STFT as the
audio feature extractor to predict $\boldsymbol{M}$, while time-domain method
replaces STFT with a learnable Audio Encoder as in 2.1. Scale-invariant
signal-to-distortion ratio (SI-SDR) measures the speech signal quality, while
perceptual evaluation of speech quality (PESQ) measures the overall perceptual
quality. They’re all the higher, the better.

Table 2 presents the results of the comparison experiment, where “T-F” means
time-frequency method, while “T” stands for time domain method. In addition to
the use of lip information as an auxiliary input, VisualVoice [10] also uses
extra information about the speaker’s face as an auxiliary input. Meanwhile,
MuSE [12] requires an additional speaker label, while most models require only
visual features from a pre-trained extractor. Results demonstrate that our
proposed AV-SepFormer has a relative improvement of 0.9 dB and 0.1 in terms of
SI-SDR and PESQ respectively, over MuSE. Besides, we employ cross-attention in
AV-ConvTasNet (att) to fuse the audio and visual features [27] instead of
direct concatenation in the original AV-ConvTasnet. It’s worth pointing out
that AV-SepFormer outperforms the AV-ConvTasnet (att) by 0.8 dB and 0.1 in
terms of SI-SDR and PESQ. This is due to the dual-scale design of AV-SepFormer
helping to solve the problem of different temporal granularity between the
audio and visual features and the Transformer’s superior sequence modeling
capability over TCN.

On LRS3 and TCD-TIMIT, AV-SepFormer still shows strong competitiveness
compared to other models, and results are shown in Table 3.

Table 3: Cross-datasets evaluations of various Audio-Visual TSE models. Models
are trained on VoxCeleb2 and tested on LRS3 and TCD-TIMIT.

Network | LRS3 | TCD-TIMIT
---|---|---
SI-SDR (dB) | PESQ | SI-SDR (dB) | PESQ
VisualVoice [10] | 11.603 | 2.269 | 10.883 | 2.246
AV-ConvTasNet [11] | 12.128 | 2.326 | 11.532 | 2.210
MuSE [12] | 12.971 | 2.558 | 12.497 | 2.447
AV-SepFormer (ours) | 13.815 | 2.668 | 13.436 | 2.567

Table 4: The ablation study on VoxCeleb2 dataset, CA and 2DPos means cross-
modal attention and 2D positional encoding respectively.

Models | CA | 2DPos | SI-SDR (dB) | PESQ
---|---|---|---|---
AV-SepFormer | ✔ | ✔ | 12.130 | 2.313
- w/o CA | ✗ | ✔ | 11.701 | 2.268
- w/o 2DPos | ✔ | ✗ | 12.020 | 2.295
- w/o CA & 2DPos | ✗ | ✗ | 11.522 | 2.261

###  3.4 Ablation study

To determine the effectiveness of the improved method proposed in this paper,
we study variants of AV-SepFormer. Table 4 shows the performances of these
variants, where “w/o” means without, “CA” and “2DPos” are the abbreviations of
cross-modal attention and 2D positional encoding, respectively.

Results show that after using cross-modal attention instead of directly
concatenating different modal features, the model can obtain about 0.4 dB and
0.05 performance improvement in SI-SDR and PESQ. Using 2D instead of 1D
positional encoding improves the model performance by about 0.1 dB and 0.02 in
terms of SI-SDR and PESQ. Using both methods, the model achieves about 0.61 dB
and 0.05 in terms of SI-SDR and PESQ compared to the baseline model. The above
experiments demonstrate that cross-modal attention (CA) and 2D positional
encoding (2DPos), alone and in combination, can lead to a significant
improvement in the performance of AV-SepFormer.

##  4 Conclusions

In this paper, we propose a dual-scale transformer-based model AV-SepFormer to
overcome the degradation of Audio-Visual TSE tasks caused by temporal
granularity misalignment of audio and visual features. Specifically, we employ
CrossModalTransformer to fuse the multi-modal features at coarse-grained
level. Fine- and coarse-grained feature information is modeled by
IntraTransformer and InterTransformer, respectively. Furthermore, we introduce
2D positional encoding to help our model focus more on the positional
relationships of the audio chunked feature at intra- and inter-chunk levels.
Experimental results demonstrate the effectiveness of the proposed AV-
SepFormer with a combination of the above three modules and the special
positional encoding. We also compare AV-SepFormer with other top-ranked Audio-
Visual TSE methods on three public datasets, which shows the superior
performance of the proposed AV-SepFormer on both speech signal quality and
overall perceptual quality.

Acknowledgements: This work is supported by National Natural Science
Foundation of China (62076144), Major Key Project of PCL (PCL2021A06,
PCL2022D01), AMiner.Shenzhen SciBrain.

## References

  * [1] DeLiang Wang,  “Deep learning reinvents the hearing aid,”  IEEE spectrum, vol. 54, no. 3, pp. 32–37, 2017.
  * [2] Zixing Zhang, Jürgen Geiger, Jouni Pohjalainen, Amr El-Desoky Mousa, Wenyu Jin, and Björn Schuller,  “Deep learning for environmentally robust speech recognition: An overview of recent developments,”  ACM Transactions on Intelligent Systems and Technology (TIST), vol. 9, no. 5, pp. 1–28, 2018.
  * [3] Daniel Michelsanti and Zheng-Hua Tan,  “Conditional generative adversarial networks for speech enhancement and noise-robust speaker verification,”  in Interspeech 2017. ISCA, 2017, pp. 2008–2012.
  * [4] Gregory Sell, David Snyder, Alan McCree, Daniel Garcia-Romero, Jesús Villalba, Matthew Maciejewski, Vimal Manohar, Najim Dehak, Daniel Povey, Shinji Watanabe, et al.,  “Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural dihard challenge.,”  in Interspeech, 2018, pp. 2808–2812.
  * [5] Jürgen T Geiger and Karim Helwani,  “Improving event detection for audio surveillance using gabor filterbank features,”  in 2015 23rd European Signal Processing Conference (EUSIPCO). IEEE, 2015, pp. 714–718.
  * [6] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen,  “Permutation invariant training of deep models for speaker-independent multi-talker speech separation,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 241–245.
  * [7] Chenglin Xu, Wei Rao, Eng Siong Chng, and Haizhou Li,  “Spex: Multi-scale time domain speaker extraction network,”  IEEE/ACM transactions on audio, speech, and language processing, vol. 28, pp. 1370–1384, 2020.
  * [8] Meng Ge, Chenglin Xu, Longbiao Wang, Eng Siong Chng, Jianwu Dang, and Haizhou Li,  “Spex+: A complete time domain speaker extraction network,”  Proc. Interspeech 2020, pp. 1406–1410, 2020.
  * [9] Meng Ge, Chenglin Xu, Longbiao Wang, Eng Siong Chng, Jianwu Dang, and Haizhou Li,  “Multi-stage speaker extraction with utterance and frame-level reference signals,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6109–6113.
  * [10] Ruohan Gao and Kristen Grauman,  “Visualvoice: Audio-visual speech separation with cross-modal consistency,”  in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021, pp. 15490–15500.
  * [11] Jian Wu, Yong Xu, Shi-Xiong Zhang, Lian-Wu Chen, Meng Yu, Lei Xie, and Dong Yu,  “Time domain audio visual speech separation,”  in 2019 IEEE automatic speech recognition and understanding workshop (ASRU). IEEE, 2019, pp. 667–673.
  * [12] Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li,  “Muse: Multi-modal target speaker extraction with visual cues,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6678–6682.
  * [13] Zexu Pan, Meng Ge, and Haizhou Li,  “Usev: Universal speaker extraction with visual cue,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 3032–3045, 2022.
  * [14] Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li,  “Selective listening by synchronizing speech with lips,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 1650–1664, 2022.
  * [15] Lei Yang, Wei Liu, and Weiqin Wang,  “Tfpsnet: Time-frequency domain path scanning network for speech separation,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6842–6846.
  * [16] Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe,  “Tf-gridnet: Making time-frequency domain models great again for monaural speaker separation,”  arXiv preprint arXiv:2209.03952, 2022.
  * [17] Yi Luo, Zhuo Chen, and Takuya Yoshioka,  “Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 46–50.
  * [18] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong,  “Attention is all you need in speech separation,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 21–25.
  * [19] Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li,  “Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection,”  in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 3927–3935.
  * [20] Zobeir Raisi, Mohamed A Naiel, Paul Fieguth, Steven Wardell, and John Zelek,  “2d positional embedding-based transformer for scene text recognition,”  Journal of Computational Vision and Imaging Systems, vol. 6, no. 1, pp. 1–4, 2020.
  * [21] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “Voxceleb2: Deep speaker recognition,”  arXiv preprint arXiv:1806.05622, 2018.
  * [22] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman,  “Lrs3-ted: a large-scale dataset for visual speech recognition,”  arXiv preprint arXiv:1809.00496, 2018.
  * [23] Naomi Harte and Eoin Gillen,  “Tcd-timit: An audio-visual corpus of continuous speech,”  IEEE Transactions on Multimedia, vol. 17, no. 5, pp. 603–615, 2015\.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [26] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in ICLR (Poster), 2015.
  * [27] Hiroshi Sato, Tsubasa Ochiai, Keisuke Kinoshita, Marc Delcroix, Tomohiro Nakatani, and Shoko Araki,  “Multimodal attention fusion for target speaker extraction,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 778–784.