Add checkpoint files
f9dc37b
verified
bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_100_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_101_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_102_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_103_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_104_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_105_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_106_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_107_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_108_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_109_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_110_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_111_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_112_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_113_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_114_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_115_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_116_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_117_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_118_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_119_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "collections.OrderedDict",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "collections.OrderedDict",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "collections.OrderedDict",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "collections.OrderedDict"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "torch._utils._rebuild_tensor_v2",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "deepspeed.utils.tensor_fragment.fragment_address",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch.FloatStorage"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
Detected Pickle imports (6)
- "deepspeed.utils.tensor_fragment.fragment_address",
- "torch.FloatStorage",
- "deepspeed.runtime.fp16.loss_scaler.LossScaler",
- "collections.OrderedDict",
- "deepspeed.runtime.zero.config.ZeroStageEnum",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
803 MB
Add checkpoint files
