/admin/home-ckadirt/miniconda3/envs/mindeye/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:165: PossibleUserWarning: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /admin/home-ckadirt/miniconda3/envs/mindeye/lib/pyth ... rank_zero_warn( Using 16bit Automatic Mixed Precision (AMP) GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs /admin/home-ckadirt/miniconda3/envs/mindeye/lib/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:165: PossibleUserWarning: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /admin/home-ckadirt/miniconda3/envs/mindeye/lib/pyth ... rank_zero_warn( You are using a CUDA device ('NVIDIA A100-SXM4-40GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision The len is 480 The len is 60 Epoch 0: 1%|▉ | 1/120 [00:00<00:24, 4.77it/s, v_num=eix7] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] | Name | Type | Params ------------------------------------------------------------------------- 0 | brain_network | BrainNetwork | 474 M 1 | ridge_regression | RidgeRegression | 248 M 2 | loss | CrossEntropyLoss | 0 3 | pseudo_text_encoder | Sequential | 723 M 4 | musicgen_decoder | MusicgenForConditionalGeneration | 588 M ------------------------------------------------------------------------- 723 M Trainable params 588 M Non-trainable params 1.3 B Total params 5,250.392 Total estimated model params size (MB) /admin/home-ckadirt/miniconda3/envs/mindeye/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:432: PossibleUserWarning: The dataloader, val_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 96 which is the number of cpus on this machine) in the `DataLoader` init to improve performance. rank_zero_warn( /admin/home-ckadirt/miniconda3/envs/mindeye/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:432: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 96 which is the number of cpus on this machine) in the `DataLoader` init to improve performance. Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.91it/s, v_num=eix7] Epoch 1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.92it/s, v_num=eix7] Epoch 4: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.89it/s, v_num=eix7] Epoch 5: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 7: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.89it/s, v_num=eix7] Epoch 8: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 11: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:25<00:00, 4.75it/s, v_num=eix7] Epoch 12: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.89it/s, v_num=eix7] Epoch 14: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.89it/s, v_num=eix7] Epoch 16: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.86it/s, v_num=eix7] Epoch 17: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 19: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 20: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.86it/s, v_num=eix7] Epoch 26: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 28: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.88it/s, v_num=eix7] Epoch 29: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.88it/s, v_num=eix7] Epoch 32: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.92it/s, v_num=eix7] Epoch 33: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.92it/s, v_num=eix7] Epoch 35: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.88it/s, v_num=eix7] Epoch 36: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.90it/s, v_num=eix7] Epoch 38: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:25<00:00, 4.66it/s, v_num=eix7] Epoch 39: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:24<00:00, 4.88it/s, v_num=eix7] Epoch 40: 75%|████████████████████████████████████████████████████████████████████████████████████ | 90/120 [00:18<00:06, 4.89it/s, v_num=eix7] /admin/home-ckadirt/miniconda3/envs/mindeye/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py:52: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown... Epoch 40: 79%|████████████████████████████████████████████████████████████████████████████████████████▋ | 95/120 [00:19<00:05, 4.86it/s, v_num=eix7]Fri Sep 8 05:01:25 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A100-SXM... On | 00000000:10:1C.0 Off | 0 | | N/A 39C P0 75W / 400W | 29319MiB / 40960MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA A100-SXM... On | 00000000:10:1D.0 Off | 0 | | N/A 42C P0 207W / 400W | 39999MiB / 40960MiB | 95% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 2 NVIDIA A100-SXM... On | 00000000:20:1C.0 Off | 0 | | N/A 47C P0 239W / 400W | 29903MiB / 40960MiB | 99% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 3 NVIDIA A100-SXM... On | 00000000:20:1D.0 Off | 0 | | N/A 44C P0 246W / 400W | 28433MiB / 40960MiB | 100% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 4 NVIDIA A100-SXM... On | 00000000:90:1C.0 Off | 0 | | N/A 46C P0 264W / 400W | 28285MiB / 40960MiB | 88% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 5 NVIDIA A100-SXM... On | 00000000:90:1D.0 Off | 0 | | N/A 44C P0 254W / 400W | 40365MiB / 40960MiB | 98% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 6 NVIDIA A100-SXM... On | 00000000:A0:1C.0 Off | 0 | | N/A 48C P0 293W / 400W | 28537MiB / 40960MiB | 99% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 7 NVIDIA A100-SXM... On | 00000000:A0:1D.0 Off | 0 | | N/A 41C P0 196W / 400W | 28295MiB / 40960MiB | 99% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 895711 C ...3/envs/mindeye/bin/python 29316MiB | | 1 N/A N/A 899054 C ...ari/llama_env/bin/python3 39996MiB | | 2 N/A N/A 899055 C ...ari/llama_env/bin/python3 29900MiB | | 3 N/A N/A 899056 C ...ari/llama_env/bin/python3 28430MiB | | 4 N/A N/A 899057 C ...ari/llama_env/bin/python3 28282MiB | | 5 N/A N/A 899058 C ...ari/llama_env/bin/python3 40362MiB | | 6 N/A N/A 899059 C ...ari/llama_env/bin/python3 28534MiB | | 7 N/A N/A 899060 C ...ari/llama_env/bin/python3 28292MiB | +-----------------------------------------------------------------------------+ torch.Size([1920, 749]) torch.Size([1920, 749]) torch.Size([1920, 749]) Fri Sep 8 05:08:00 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A100-SXM... On | 00000000:10:1C.0 Off | 0 | | N/A 31C P0 71W / 400W | 29323MiB / 40960MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA A100-SXM... On | 00000000:10:1D.0 Off | 0 | | N/A 40C P0 204W / 400W | 40217MiB / 40960MiB | 100% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 2 NVIDIA A100-SXM... On | 00000000:20:1C.0 Off | 0 | | N/A 47C P0 243W / 400W | 40421MiB / 40960MiB | 99% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 3 NVIDIA A100-SXM... On | 00000000:20:1D.0 Off | 0 | | N/A 42C P0 194W / 400W | 37547MiB / 40960MiB | 100% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 4 NVIDIA A100-SXM... On | 00000000:90:1C.0 Off | 0 | | N/A 45C P0 282W / 400W | 29223MiB / 40960MiB | 100% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 5 NVIDIA A100-SXM... On | 00000000:90:1D.0 Off | 0 | | N/A 39C P0 179W / 400W | 24387MiB / 40960MiB | 100% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 6 NVIDIA A100-SXM... On | 00000000:A0:1C.0 Off | 0 | | N/A 43C P0 232W / 400W | 29819MiB / 40960MiB | 88% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 7 NVIDIA A100-SXM... On | 00000000:A0:1D.0 Off | 0 | | N/A 38C P0 166W / 400W | 28583MiB / 40960MiB | 100% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 895711 C ...3/envs/mindeye/bin/python 29320MiB | | 1 N/A N/A 899054 C ...ari/llama_env/bin/python3 40214MiB | | 2 N/A N/A 899055 C ...ari/llama_env/bin/python3 40418MiB | | 3 N/A N/A 899056 C ...ari/llama_env/bin/python3 37544MiB | | 4 N/A N/A 899057 C ...ari/llama_env/bin/python3 29220MiB | | 5 N/A N/A 899058 C ...ari/llama_env/bin/python3 24384MiB | | 6 N/A N/A 899059 C ...ari/llama_env/bin/python3 29816MiB | | 7 N/A N/A 899060 C ...ari/llama_env/bin/python3 28580MiB | +-----------------------------------------------------------------------------+ torch.Size([1920, 749]) torch.Size([1920, 749]) torch.Size([1920, 749]) torch.Size([240, 749]) torch.Size([240, 749]) torch.Size([1, 60784]) 0 1 2 3 4 5 6 7 8 9 10 11 12 0 1 2 3 4 5 6 7 8 9 10 11 12 0 torch.Size([4, 2]) 1 torch.Size([4, 4]) 2 torch.Size([4, 8]) 3 torch.Size([4, 16]) 4 torch.Size([4, 32]) 5 torch.Size([4, 64]) 6 torch.Size([4, 128]) 7 torch.Size([4, 256]) 8 torch.Size([4, 512]) 9 torch.Size([4, 1024]) 10 torch.Size([4, 2048]) 11 torch.Size([4, 4096]) 12 0 torch.Size([4, 1]) torch.Size([4, 2]) 1 torch.Size([4, 2]) torch.Size([4, 4]) 2 torch.Size([4, 4]) torch.Size([4, 8]) 3 torch.Size([4, 8]) torch.Size([4, 16]) 4 torch.Size([4, 16]) torch.Size([4, 32]) 5 torch.Size([4, 32]) torch.Size([4, 64]) 6 torch.Size([4, 64]) torch.Size([4, 128]) 7 torch.Size([4, 128]) torch.Size([4, 256]) 8 torch.Size([4, 256]) torch.Size([4, 512]) 9 torch.Size([4, 512]) torch.Size([4, 1024]) 10 torch.Size([4, 1024]) torch.Size([4, 2048]) 11 torch.Size([4, 2048]) torch.Size([4, 4096]) 12 0 torch.Size([4, 1, 2048]) torch.Size([4, 1]) torch.Size([4, 2]) 1 torch.Size([4, 2, 2048]) torch.Size([4, 2]) torch.Size([4, 4]) 2 torch.Size([4, 4, 2048]) torch.Size([4, 4]) torch.Size([4, 8]) 3 torch.Size([4, 8, 2048]) torch.Size([4, 8]) torch.Size([4, 16]) 4 torch.Size([4, 16, 2048]) torch.Size([4, 16]) torch.Size([4, 32]) 5 torch.Size([4, 32, 2048]) torch.Size([4, 32]) torch.Size([4, 64]) 6 torch.Size([4, 64, 2048]) torch.Size([4, 64]) torch.Size([4, 128]) 7 torch.Size([4, 128, 2048]) torch.Size([4, 128]) torch.Size([4, 256]) 8 torch.Size([4, 256, 2048]) torch.Size([4, 256]) torch.Size([4, 512]) 9 torch.Size([4, 512, 2048]) torch.Size([4, 512]) torch.Size([4, 1024]) 10 torch.Size([4, 1024, 2048]) torch.Size([4, 1024]) torch.Size([4, 2048]) 11 torch.Size([4, 2048, 2048]) torch.Size([4, 2048]) torch.Size([4, 4096]) 12 torch.Size([1, 128, 1024]) 0 torch.Size([4, 1, 2048]) torch.Size([4, 1]) torch.Size([4, 2]) 1 torch.Size([4, 2, 2048]) torch.Size([4, 2]) torch.Size([4, 4]) 2 torch.Size([4, 4, 2048]) torch.Size([4, 4]) torch.Size([4, 8]) 3 torch.Size([4, 8, 2048]) torch.Size([4, 8]) torch.Size([4, 16]) 4 torch.Size([4, 16, 2048]) torch.Size([4, 16]) torch.Size([4, 32]) 5 torch.Size([4, 32, 2048]) torch.Size([4, 32]) torch.Size([4, 64]) 6 torch.Size([4, 64, 2048]) torch.Size([4, 64]) torch.Size([4, 128]) 7 torch.Size([4, 128, 2048]) torch.Size([4, 128]) torch.Size([4, 256]) 8 torch.Size([4, 256, 2048]) torch.Size([4, 256]) torch.Size([4, 512]) 9 torch.Size([4, 512, 2048]) torch.Size([4, 512]) torch.Size([4, 1024]) 10 torch.Size([4, 1024, 2048]) torch.Size([4, 1024]) torch.Size([4, 2048]) 11 torch.Size([4, 2048, 2048]) torch.Size([4, 2048]) torch.Size([4, 4096]) 12 torch.Size([1, 128, 1024]) 0 torch.Size([4, 1, 2048]) torch.Size([1, 128, 1024]) 0 torch.Size([4, 1, 2048]) torch.Size([1, 128, 1024]) 0 torch.Size([4, 1, 2048]) torch.Size([4]) torch.Size([1, 128, 1024]) 0 torch.Size([4, 1, 2048]) torch.Size([4]) torch.Size([1, 128, 1024]) 0 torch.Size([4, 1, 2048]) torch.Size([4]) torch.Size([4, 2]) 1 torch.Size([4, 2, 2048]) torch.Size([4]) torch.Size([4, 3]) 2 torch.Size([4, 3, 2048]) torch.Size([4]) torch.Size([4, 4]) 3 torch.Size([4, 4, 2048]) torch.Size([4]) torch.Size([4, 5]) 4 torch.Size([4, 5, 2048]) torch.Size([4]) torch.Size([4, 6]) 5 torch.Size([4, 6, 2048]) torch.Size([4]) torch.Size([4, 7]) 6 torch.Size([4, 7, 2048]) torch.Size([4]) torch.Size([4, 8]) 7 torch.Size([4, 8, 2048]) torch.Size([4]) torch.Size([4, 9]) 8 torch.Size([4, 9, 2048]) torch.Size([4]) torch.Size([4, 10]) 9 torch.Size([4, 10, 2048]) torch.Size([4]) torch.Size([4, 11]) 10 torch.Size([4, 11, 2048]) torch.Size([4]) torch.Size([4, 12]) 11 torch.Size([4, 12, 2048]) torch.Size([4]) torch.Size([4, 13]) 12 torch.Size([4, 13, 2048]) torch.Size([4]) torch.Size([4, 14]) 13 torch.Size([4, 14, 2048]) torch.Size([4]) torch.Size([4, 15]) 14 torch.Size([4, 15, 2048]) torch.Size([4]) torch.Size([4, 16]) 15 torch.Size([4, 16, 2048]) torch.Size([4]) torch.Size([4, 17]) 16 torch.Size([4, 17, 2048]) torch.Size([4]) torch.Size([4, 18]) 17 torch.Size([4, 18, 2048]) torch.Size([4]) torch.Size([4, 19]) 18 torch.Size([4, 19, 2048]) torch.Size([4]) torch.Size([4, 20]) 19 torch.Size([4, 20, 2048]) torch.Size([4]) torch.Size([4, 21]) 20 torch.Size([4, 21, 2048]) torch.Size([4]) torch.Size([4, 22]) 21 torch.Size([4, 22, 2048]) torch.Size([4]) torch.Size([4, 23]) 22 torch.Size([4, 23, 2048]) torch.Size([4]) torch.Size([4, 24]) 23 torch.Size([4, 24, 2048]) torch.Size([4]) torch.Size([4, 25]) 24 torch.Size([4, 25, 2048]) torch.Size([4]) torch.Size([4, 26]) 25 torch.Size([4, 26, 2048]) torch.Size([4]) torch.Size([4, 27]) 26 torch.Size([4, 27, 2048]) torch.Size([4]) torch.Size([4, 28]) 27 torch.Size([4, 28, 2048]) torch.Size([4]) torch.Size([4, 29]) 28 torch.Size([4, 29, 2048]) torch.Size([4]) torch.Size([4, 30]) 29 torch.Size([4, 30, 2048]) torch.Size([4]) torch.Size([4, 31]) 30 torch.Size([4, 31, 2048]) torch.Size([4]) torch.Size([4, 32]) 31 torch.Size([4, 32, 2048]) torch.Size([4]) torch.Size([4, 33]) 32 torch.Size([4, 33, 2048]) torch.Size([4]) torch.Size([4, 34]) 33 torch.Size([4, 34, 2048]) torch.Size([4]) torch.Size([4, 35]) 34 torch.Size([4, 35, 2048]) torch.Size([4]) torch.Size([4, 36]) 35 torch.Size([4, 36, 2048]) torch.Size([4]) torch.Size([4, 37]) 36 torch.Size([4, 37, 2048]) torch.Size([4]) torch.Size([4, 38]) 37 torch.Size([4, 38, 2048]) torch.Size([4]) torch.Size([4, 39]) 38 torch.Size([4, 39, 2048]) torch.Size([4]) torch.Size([4, 40]) 39 torch.Size([4, 40, 2048]) torch.Size([4]) torch.Size([4, 41]) 40 torch.Size([4, 41, 2048]) torch.Size([4]) torch.Size([4, 42]) 41 torch.Size([4, 42, 2048]) torch.Size([4]) torch.Size([4, 43]) 42 torch.Size([4, 43, 2048]) torch.Size([4]) torch.Size([4, 44]) 43 torch.Size([4, 44, 2048]) torch.Size([4]) torch.Size([4, 45]) 44 torch.Size([4, 45, 2048]) torch.Size([4]) torch.Size([4, 46]) 45 torch.Size([4, 46, 2048]) torch.Size([4]) torch.Size([4, 47]) 46 torch.Size([4, 47, 2048]) torch.Size([4]) torch.Size([4, 48]) 47 torch.Size([4, 48, 2048]) torch.Size([4]) torch.Size([4, 49]) 48 torch.Size([4, 49, 2048]) torch.Size([4]) torch.Size([4, 50]) 49 torch.Size([4, 50, 2048]) torch.Size([4]) torch.Size([4, 51]) 50 torch.Size([4, 51, 2048]) torch.Size([4]) torch.Size([4, 52]) 51 torch.Size([4, 52, 2048]) torch.Size([4]) torch.Size([4, 53]) 52 torch.Size([4, 53, 2048]) torch.Size([4]) torch.Size([4, 54]) 53 torch.Size([4, 54, 2048]) torch.Size([4]) torch.Size([4, 55]) 54 torch.Size([4, 55, 2048]) torch.Size([4]) torch.Size([4, 56]) 55 torch.Size([4, 56, 2048]) torch.Size([4]) torch.Size([4, 57]) 56 torch.Size([4, 57, 2048]) torch.Size([4]) torch.Size([4, 58]) 57 torch.Size([4, 58, 2048]) torch.Size([4]) torch.Size([4, 59]) 58 torch.Size([4, 59, 2048]) torch.Size([4]) torch.Size([4, 60]) 59 torch.Size([4, 60, 2048]) torch.Size([4]) torch.Size([4, 61]) 60 torch.Size([4, 61, 2048]) torch.Size([4]) torch.Size([4, 62]) 61 torch.Size([4, 62, 2048]) torch.Size([4]) torch.Size([4, 63]) 62 torch.Size([4, 63, 2048]) torch.Size([4]) torch.Size([4, 64]) 63 torch.Size([4, 64, 2048]) torch.Size([4]) torch.Size([4, 65]) 64 torch.Size([4, 65, 2048]) torch.Size([4]) torch.Size([4, 66]) 65 torch.Size([4, 66, 2048]) torch.Size([4]) torch.Size([4, 67]) 66 torch.Size([4, 67, 2048]) torch.Size([4]) torch.Size([4, 68]) 67 torch.Size([4, 68, 2048]) torch.Size([4]) torch.Size([4, 69]) 68 torch.Size([4, 69, 2048]) torch.Size([4]) torch.Size([4, 70]) 69 torch.Size([4, 70, 2048]) torch.Size([4]) torch.Size([4, 71]) 70 torch.Size([4, 71, 2048]) torch.Size([4]) torch.Size([4, 72]) 71 torch.Size([4, 72, 2048]) torch.Size([4]) torch.Size([4, 73]) 72 torch.Size([4, 73, 2048]) torch.Size([4]) torch.Size([4, 74]) 73 torch.Size([4, 74, 2048]) torch.Size([4]) torch.Size([4, 75]) 74 torch.Size([4, 75, 2048]) torch.Size([4]) torch.Size([4, 76]) 75 torch.Size([4, 76, 2048]) torch.Size([4]) torch.Size([4, 77]) 76 torch.Size([4, 77, 2048]) torch.Size([4]) torch.Size([4, 78]) 77 torch.Size([4, 78, 2048]) torch.Size([4]) torch.Size([4, 79]) 78 torch.Size([4, 79, 2048]) torch.Size([4]) torch.Size([4, 80]) 79 torch.Size([4, 80, 2048]) torch.Size([4]) torch.Size([4, 81]) 80 torch.Size([4, 81, 2048]) torch.Size([4]) torch.Size([4, 82]) 81 torch.Size([4, 82, 2048]) torch.Size([4]) torch.Size([4, 83]) 82 torch.Size([4, 83, 2048]) torch.Size([4]) torch.Size([4, 84]) 83 torch.Size([4, 84, 2048]) torch.Size([4]) torch.Size([4, 85]) 84 torch.Size([4, 85, 2048]) torch.Size([4]) torch.Size([4, 86]) 85 torch.Size([4, 86, 2048]) torch.Size([4]) torch.Size([4, 87]) 86 torch.Size([4, 87, 2048]) torch.Size([4]) torch.Size([4, 88]) 87 torch.Size([4, 88, 2048]) torch.Size([4]) torch.Size([4, 89]) 88 torch.Size([4, 89, 2048]) torch.Size([4]) torch.Size([4, 90]) 89 torch.Size([4, 90, 2048]) torch.Size([4]) torch.Size([4, 91]) 90 torch.Size([4, 91, 2048]) torch.Size([4]) torch.Size([4, 92]) 91 torch.Size([4, 92, 2048]) torch.Size([4]) torch.Size([4, 93]) 92 torch.Size([4, 93, 2048]) torch.Size([4]) torch.Size([4, 94]) 93 torch.Size([4, 94, 2048]) torch.Size([4]) torch.Size([4, 95]) 94 torch.Size([4, 95, 2048]) torch.Size([4]) torch.Size([4, 96]) 95 torch.Size([4, 96, 2048]) torch.Size([4]) torch.Size([4, 97]) 96 torch.Size([4, 97, 2048]) torch.Size([4]) torch.Size([4, 98]) 97 torch.Size([4, 98, 2048]) torch.Size([4]) torch.Size([4, 99]) 98 torch.Size([4, 99, 2048]) torch.Size([4]) torch.Size([4, 100]) 99 torch.Size([4, 100, 2048]) torch.Size([4]) torch.Size([4, 101]) 100 torch.Size([4, 101, 2048]) torch.Size([4]) torch.Size([4, 102]) 101 torch.Size([4, 102, 2048]) torch.Size([4]) torch.Size([4, 103]) 102 torch.Size([4, 103, 2048]) torch.Size([4]) torch.Size([4, 104]) 103 torch.Size([4, 104, 2048]) torch.Size([4]) torch.Size([4, 105]) 104 torch.Size([4, 105, 2048]) torch.Size([4]) torch.Size([4, 106]) 105 torch.Size([4, 106, 2048]) torch.Size([4]) torch.Size([4, 107]) 106 torch.Size([4, 107, 2048]) torch.Size([4]) torch.Size([4, 108]) 107 torch.Size([4, 108, 2048]) torch.Size([4]) torch.Size([4, 109]) 108 torch.Size([4, 109, 2048]) torch.Size([4]) torch.Size([4, 110]) 109 torch.Size([4, 110, 2048]) torch.Size([4]) torch.Size([4, 111]) 110 torch.Size([4, 111, 2048]) torch.Size([4]) torch.Size([4, 112]) 111 torch.Size([4, 112, 2048]) torch.Size([4]) torch.Size([4, 113]) 112 torch.Size([4, 113, 2048]) torch.Size([4]) torch.Size([4, 114]) 113 torch.Size([4, 114, 2048]) torch.Size([4]) torch.Size([4, 115]) 114 torch.Size([4, 115, 2048]) torch.Size([4]) torch.Size([4, 116]) 115 torch.Size([4, 116, 2048]) torch.Size([4]) torch.Size([4, 117]) 116 torch.Size([4, 117, 2048]) torch.Size([4]) torch.Size([4, 118]) 117 torch.Size([4, 118, 2048]) torch.Size([4]) torch.Size([4, 119]) 118 torch.Size([4, 119, 2048]) torch.Size([4]) torch.Size([4, 120]) 119 torch.Size([4, 120, 2048]) torch.Size([4]) torch.Size([4, 121]) 120 torch.Size([4, 121, 2048]) torch.Size([4]) torch.Size([4, 122]) 121 torch.Size([4, 122, 2048]) torch.Size([4]) torch.Size([4, 123]) 122 torch.Size([4, 123, 2048]) torch.Size([4]) torch.Size([4, 124]) 123 torch.Size([4, 124, 2048]) torch.Size([4]) torch.Size([4, 125]) 124 torch.Size([4, 125, 2048]) torch.Size([4]) torch.Size([4, 126]) 125 torch.Size([4, 126, 2048]) torch.Size([4]) torch.Size([4, 127]) 126 torch.Size([4, 127, 2048]) torch.Size([4]) torch.Size([4, 128]) 127 torch.Size([4, 128, 2048]) torch.Size([4]) torch.Size([4, 129]) 128 torch.Size([4, 129, 2048]) torch.Size([4]) torch.Size([4, 130]) 129 torch.Size([4, 130, 2048]) torch.Size([4]) torch.Size([4, 131]) 130 torch.Size([4, 131, 2048]) torch.Size([4]) torch.Size([4, 132]) 131 torch.Size([4, 132, 2048]) torch.Size([4]) torch.Size([4, 133]) 132 torch.Size([4, 133, 2048]) torch.Size([4]) torch.Size([4, 134]) 133 torch.Size([4, 134, 2048]) torch.Size([4]) torch.Size([4, 135]) 134 torch.Size([4, 135, 2048]) torch.Size([4]) torch.Size([4, 136]) 135 torch.Size([4, 136, 2048]) torch.Size([4]) torch.Size([4, 137]) 136 torch.Size([4, 137, 2048]) torch.Size([4]) torch.Size([4, 138]) 137 torch.Size([4, 138, 2048]) torch.Size([4]) torch.Size([4, 139]) 138 torch.Size([4, 139, 2048]) torch.Size([4]) torch.Size([4, 140]) 139 torch.Size([4, 140, 2048]) torch.Size([4]) torch.Size([4, 141]) 140 torch.Size([4, 141, 2048]) torch.Size([4]) torch.Size([4, 142]) 141 torch.Size([4, 142, 2048]) torch.Size([4]) torch.Size([4, 143]) 142 torch.Size([4, 143, 2048]) torch.Size([4]) torch.Size([4, 144]) 143 torch.Size([4, 144, 2048]) torch.Size([4]) torch.Size([4, 145]) 144 torch.Size([4, 145, 2048]) torch.Size([4]) torch.Size([4, 146]) 145 torch.Size([4, 146, 2048]) torch.Size([4]) torch.Size([4, 147]) 146 torch.Size([4, 147, 2048]) torch.Size([4]) torch.Size([4, 148]) 147 torch.Size([4, 148, 2048]) torch.Size([4]) torch.Size([4, 149]) 148 torch.Size([4, 149, 2048]) torch.Size([4]) torch.Size([4, 150]) 149 torch.Size([4, 150, 2048]) torch.Size([4]) torch.Size([4, 151]) 150 torch.Size([4, 151, 2048]) torch.Size([4]) torch.Size([4, 152]) 151 torch.Size([4, 152, 2048]) torch.Size([4]) torch.Size([4, 153]) 152 torch.Size([4, 153, 2048]) torch.Size([4]) torch.Size([4, 154]) 153 torch.Size([4, 154, 2048]) torch.Size([4]) torch.Size([4, 155]) 154 torch.Size([4, 155, 2048]) torch.Size([4]) torch.Size([4, 156]) 155 torch.Size([4, 156, 2048]) torch.Size([4]) torch.Size([4, 157]) 156 torch.Size([4, 157, 2048]) torch.Size([4]) torch.Size([4, 158]) 157 torch.Size([4, 158, 2048]) torch.Size([4]) torch.Size([4, 159]) 158 torch.Size([4, 159, 2048]) torch.Size([4]) torch.Size([4, 160]) 159 torch.Size([4, 160, 2048]) torch.Size([4]) torch.Size([4, 161]) 160 torch.Size([4, 161, 2048]) torch.Size([4]) torch.Size([4, 162]) 161 torch.Size([4, 162, 2048]) torch.Size([4]) torch.Size([4, 163]) 162 torch.Size([4, 163, 2048]) torch.Size([4]) torch.Size([4, 164]) 163 torch.Size([4, 164, 2048]) torch.Size([4]) torch.Size([4, 165]) 164 torch.Size([4, 165, 2048]) torch.Size([4]) torch.Size([4, 166]) 165 torch.Size([4, 166, 2048]) torch.Size([4]) torch.Size([4, 167]) 166 torch.Size([4, 167, 2048]) torch.Size([4]) torch.Size([4, 168]) 167 torch.Size([4, 168, 2048]) torch.Size([4]) torch.Size([4, 169]) 168 torch.Size([4, 169, 2048]) torch.Size([4]) torch.Size([4, 170]) 169 torch.Size([4, 170, 2048]) torch.Size([4]) torch.Size([4, 171]) 170 torch.Size([4, 171, 2048]) torch.Size([4]) torch.Size([4, 172]) 171 torch.Size([4, 172, 2048]) torch.Size([4]) torch.Size([4, 173]) 172 torch.Size([4, 173, 2048]) torch.Size([4]) torch.Size([4, 174]) 173 torch.Size([4, 174, 2048]) torch.Size([4]) torch.Size([4, 175]) 174 torch.Size([4, 175, 2048]) torch.Size([4]) torch.Size([4, 176]) 175 torch.Size([4, 176, 2048]) torch.Size([4]) torch.Size([4, 177]) 176 torch.Size([4, 177, 2048]) torch.Size([4]) torch.Size([4, 178]) 177 torch.Size([4, 178, 2048]) torch.Size([4]) torch.Size([4, 179]) 178 torch.Size([4, 179, 2048]) torch.Size([4]) torch.Size([4, 180]) 179 torch.Size([4, 180, 2048]) torch.Size([4]) torch.Size([4, 181]) 180 torch.Size([4, 181, 2048]) torch.Size([4]) torch.Size([4, 182]) 181 torch.Size([4, 182, 2048]) torch.Size([4]) torch.Size([4, 183]) 182 torch.Size([4, 183, 2048]) torch.Size([4]) torch.Size([4, 184]) 183 torch.Size([4, 184, 2048]) torch.Size([4]) torch.Size([4, 185]) 184 torch.Size([4, 185, 2048]) torch.Size([4]) torch.Size([4, 186]) 185 torch.Size([4, 186, 2048]) torch.Size([4]) torch.Size([4, 187]) 186 torch.Size([4, 187, 2048]) torch.Size([4]) torch.Size([4, 188]) 187 torch.Size([4, 188, 2048]) torch.Size([4]) torch.Size([4, 189]) 188 torch.Size([4, 189, 2048]) torch.Size([4]) torch.Size([4, 190]) 189 torch.Size([4, 190, 2048]) torch.Size([4]) torch.Size([4, 191]) 190 torch.Size([4, 191, 2048]) torch.Size([4]) torch.Size([4, 192]) 191 torch.Size([4, 192, 2048]) torch.Size([4]) torch.Size([4, 193]) 192 torch.Size([4, 193, 2048]) torch.Size([4]) torch.Size([4, 194]) 193 torch.Size([4, 194, 2048]) torch.Size([4]) torch.Size([4, 195]) 194 torch.Size([4, 195, 2048]) torch.Size([4]) torch.Size([4, 196]) 195 torch.Size([4, 196, 2048]) torch.Size([4]) torch.Size([4, 197]) 196 torch.Size([4, 197, 2048]) torch.Size([4]) torch.Size([4, 198]) 197 torch.Size([4, 198, 2048]) torch.Size([4]) torch.Size([4, 199]) 198 torch.Size([4, 199, 2048]) torch.Size([4]) torch.Size([4, 200]) 199 torch.Size([4, 200, 2048]) torch.Size([4]) torch.Size([4, 201]) 200 torch.Size([4, 201, 2048]) torch.Size([4]) torch.Size([4, 202]) 201 torch.Size([4, 202, 2048]) torch.Size([4]) torch.Size([4, 203]) 202 torch.Size([4, 203, 2048]) torch.Size([4]) torch.Size([4, 204]) 203 torch.Size([4, 204, 2048]) torch.Size([4]) torch.Size([4, 205]) 204 torch.Size([4, 205, 2048]) torch.Size([4]) torch.Size([4, 206]) 205 torch.Size([4, 206, 2048]) torch.Size([4]) torch.Size([4, 207]) 206 torch.Size([4, 207, 2048]) torch.Size([4]) torch.Size([4, 208]) 207 torch.Size([4, 208, 2048]) torch.Size([4]) torch.Size([4, 209]) 208 torch.Size([4, 209, 2048]) torch.Size([4]) torch.Size([4, 210]) 209 torch.Size([4, 210, 2048]) torch.Size([4]) torch.Size([4, 211]) 210 torch.Size([4, 211, 2048]) torch.Size([4]) torch.Size([4, 212]) 211 torch.Size([4, 212, 2048]) torch.Size([4]) torch.Size([4, 213]) 212 torch.Size([4, 213, 2048]) torch.Size([4]) torch.Size([4, 214]) 213 torch.Size([4, 214, 2048]) torch.Size([4]) torch.Size([4, 215]) 214 torch.Size([4, 215, 2048]) torch.Size([4]) torch.Size([4, 216]) 215 torch.Size([4, 216, 2048]) torch.Size([4]) torch.Size([4, 217]) 216 torch.Size([4, 217, 2048]) torch.Size([4]) torch.Size([4, 218]) 217 torch.Size([4, 218, 2048]) torch.Size([4]) torch.Size([4, 219]) 218 torch.Size([4, 219, 2048]) torch.Size([4]) torch.Size([4, 220]) 219 torch.Size([4, 220, 2048]) torch.Size([4]) torch.Size([4, 221]) 220 torch.Size([4, 221, 2048]) torch.Size([4]) torch.Size([4, 222]) 221 torch.Size([4, 222, 2048]) torch.Size([4]) torch.Size([4, 223]) 222 torch.Size([4, 223, 2048]) torch.Size([4]) torch.Size([4, 224]) 223 torch.Size([4, 224, 2048]) torch.Size([4]) torch.Size([4, 225]) 224 torch.Size([4, 225, 2048]) torch.Size([4]) torch.Size([4, 226]) 225 torch.Size([4, 226, 2048]) torch.Size([4]) torch.Size([4, 227]) 226 torch.Size([4, 227, 2048]) torch.Size([4]) torch.Size([4, 228]) 227 torch.Size([4, 228, 2048]) torch.Size([4]) torch.Size([4, 229]) 228 torch.Size([4, 229, 2048]) torch.Size([4]) torch.Size([4, 230]) 229 torch.Size([4, 230, 2048]) torch.Size([4]) torch.Size([4, 231]) 230 torch.Size([4, 231, 2048]) torch.Size([4]) torch.Size([4, 232]) 231 torch.Size([4, 232, 2048]) torch.Size([4]) torch.Size([4, 233]) 232 torch.Size([4, 233, 2048]) torch.Size([4]) torch.Size([4, 234]) 233 torch.Size([4, 234, 2048]) torch.Size([4]) torch.Size([4, 235]) 234 torch.Size([4, 235, 2048]) torch.Size([4]) torch.Size([4, 236]) 235 torch.Size([4, 236, 2048]) torch.Size([4]) torch.Size([4, 237]) 236 torch.Size([4, 237, 2048]) torch.Size([4]) torch.Size([4, 238]) 237 torch.Size([4, 238, 2048]) torch.Size([4]) torch.Size([4, 239]) 238 torch.Size([4, 239, 2048]) torch.Size([4]) torch.Size([4, 240]) 239 torch.Size([4, 240, 2048]) torch.Size([4]) torch.Size([4, 241]) 240 torch.Size([4, 241, 2048]) torch.Size([4]) torch.Size([4, 242]) 241 torch.Size([4, 242, 2048]) torch.Size([4]) torch.Size([4, 243]) 242 torch.Size([4, 243, 2048]) torch.Size([4]) torch.Size([4, 244]) 243 torch.Size([4, 244, 2048]) torch.Size([4]) torch.Size([4, 245]) 244 torch.Size([4, 245, 2048]) torch.Size([4]) torch.Size([4, 246]) 245 torch.Size([4, 246, 2048]) torch.Size([4]) torch.Size([4, 247]) 246 torch.Size([4, 247, 2048]) torch.Size([4]) torch.Size([4, 248]) 247 torch.Size([4, 248, 2048]) torch.Size([4]) torch.Size([4, 249]) 248 torch.Size([4, 249, 2048]) torch.Size([4]) torch.Size([4, 250]) 249 torch.Size([4, 250, 2048]) torch.Size([4]) torch.Size([4, 251]) 250 torch.Size([4, 251, 2048]) torch.Size([4]) torch.Size([4, 252]) 251 torch.Size([4, 252, 2048]) torch.Size([4]) torch.Size([4, 253]) 252 torch.Size([4, 253, 2048]) torch.Size([4]) torch.Size([4, 254]) 253 torch.Size([4, 254, 2048]) torch.Size([4]) torch.Size([4, 255]) 254 torch.Size([4, 255, 2048]) torch.Size([4]) torch.Size([4, 256]) 255 torch.Size([4, 256, 2048]) torch.Size([4]) torch.Size([4, 257]) torch.Size([1, 128, 1024]) ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ../aten/src/ATen/native/cuda/Indexing.cu:1093: indexSelectSmallIndex: block: [0,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed. Fri Sep 8 05:39:28 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A100-SXM... On | 00000000:10:1C.0 Off | 0 | | N/A 30C P0 70W / 400W | 29365MiB / 40960MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA A100-SXM... On | 00000000:10:1D.0 Off | 0 | | N/A 41C P0 240W / 400W | 33897MiB / 40960MiB | 92% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 2 NVIDIA A100-SXM... On | 00000000:20:1C.0 Off | 0 | | N/A 44C P0 208W / 400W | 37457MiB / 40960MiB | 82% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 3 NVIDIA A100-SXM... On | 00000000:20:1D.0 Off | 0 | | N/A 38C P0 189W / 400W | 40233MiB / 40960MiB | 76% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 4 NVIDIA A100-SXM... On | 00000000:90:1C.0 Off | 0 | | N/A 41C P0 242W / 400W | 34031MiB / 40960MiB | 95% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 5 NVIDIA A100-SXM... On | 00000000:90:1D.0 Off | 0 | | N/A 38C P0 201W / 400W | 30723MiB / 40960MiB | 100% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 6 NVIDIA A100-SXM... On | 00000000:A0:1C.0 Off | 0 | | N/A 39C P0 183W / 400W | 38225MiB / 40960MiB | 97% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 7 NVIDIA A100-SXM... On | 00000000:A0:1D.0 Off | 0 | | N/A 38C P0 197W / 400W | 40121MiB / 40960MiB | 79% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 895711 C ...3/envs/mindeye/bin/python 29362MiB | | 1 N/A N/A 919504 C ...ari/llama_env/bin/python3 33894MiB | | 2 N/A N/A 919505 C ...ari/llama_env/bin/python3 37454MiB | | 3 N/A N/A 919506 C ...ari/llama_env/bin/python3 40230MiB | | 4 N/A N/A 919507 C ...ari/llama_env/bin/python3 34028MiB | | 5 N/A N/A 919508 C ...ari/llama_env/bin/python3 30720MiB | | 6 N/A N/A 919509 C ...ari/llama_env/bin/python3 38222MiB | | 7 N/A N/A 919510 C ...ari/llama_env/bin/python3 40118MiB | +-----------------------------------------------------------------------------+