| 10/15/2023 00:08:47 - INFO - app - Distributed environment: NO | |
| Num processes: 1 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda | |
| Mixed precision type: no | |
| 10/15/2023 00:08:52 - INFO - app - Num examples = 5 | |
| 10/15/2023 00:08:52 - INFO - app - Num batches each epoch = 5 | |
| 10/15/2023 00:08:52 - INFO - app - Num Epochs = 40 | |
| 10/15/2023 00:08:52 - INFO - app - Instantaneous batch size per device = 1 | |
| 10/15/2023 00:08:52 - INFO - app - Total train batch size (w. parallel, distributed & accumulation) = 1 | |
| 10/15/2023 00:08:52 - INFO - app - Gradient Accumulation steps = 1 | |
| 10/15/2023 00:08:52 - INFO - app - Training Steps = 200 | |
| 10/15/2023 00:08:52 - INFO - app - STEP = 1 | |
| 10/15/2023 00:09:00 - INFO - app - STEP = 2 | |
| 10/15/2023 00:09:07 - INFO - app - STEP = 3 | |
| 10/15/2023 00:09:13 - INFO - app - STEP = 4 | |
| 10/15/2023 00:09:20 - INFO - app - STEP = 5 | |
| 10/15/2023 00:09:26 - INFO - app - Running validation... | |
| Generating 4 images with prompt: A photo of sks dog in a bucket. | |
| 10/15/2023 00:09:52 - INFO - app - STEP = 6 | |
| 10/15/2023 00:09:58 - INFO - app - STEP = 7 | |
| 10/15/2023 00:10:05 - INFO - app - STEP = 8 | |
| 10/15/2023 00:10:12 - INFO - app - STEP = 9 | |
| 10/15/2023 00:10:18 - INFO - app - STEP = 10 | |
| 10/15/2023 00:10:25 - INFO - app - STEP = 11 | |
| 10/15/2023 00:10:32 - INFO - app - STEP = 12 | |
| 10/15/2023 00:10:39 - INFO - app - STEP = 13 | |
| 10/15/2023 00:10:45 - INFO - app - STEP = 14 | |
| 10/15/2023 00:10:52 - INFO - app - STEP = 15 | |
| 10/15/2023 00:10:59 - INFO - app - STEP = 16 | |
| 10/15/2023 00:11:06 - INFO - app - STEP = 17 | |
| 10/15/2023 00:11:12 - INFO - app - STEP = 18 | |
| 10/15/2023 00:11:19 - INFO - app - STEP = 19 | |
| 10/15/2023 00:11:26 - INFO - app - STEP = 20 | |
| 10/15/2023 00:11:32 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-20 | |
| 10/15/2023 00:11:32 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-20/optimizer.bin | |
| 10/15/2023 00:11:32 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-20/scheduler.bin | |
| 10/15/2023 00:11:32 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-20/random_states_0.pkl | |
| 10/15/2023 00:11:32 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-20 | |
| 10/15/2023 00:11:33 - INFO - app - STEP = 21 | |
| 10/15/2023 00:11:39 - INFO - app - STEP = 22 | |
| 10/15/2023 00:11:46 - INFO - app - STEP = 23 | |
| 10/15/2023 00:11:53 - INFO - app - STEP = 24 | |
| 10/15/2023 00:11:59 - INFO - app - STEP = 25 | |
| 10/15/2023 00:12:06 - INFO - app - STEP = 26 | |
| 10/15/2023 00:12:13 - INFO - app - STEP = 27 | |
| 10/15/2023 00:12:20 - INFO - app - STEP = 28 | |
| 10/15/2023 00:12:27 - INFO - app - STEP = 29 | |
| 10/15/2023 00:12:33 - INFO - app - STEP = 30 | |
| 10/15/2023 00:12:40 - INFO - app - STEP = 31 | |
| 10/15/2023 00:12:47 - INFO - app - STEP = 32 | |
| 10/15/2023 00:12:53 - INFO - app - STEP = 33 | |
| 10/15/2023 00:13:00 - INFO - app - STEP = 34 | |
| 10/15/2023 00:13:07 - INFO - app - STEP = 35 | |
| 10/15/2023 00:13:14 - INFO - app - STEP = 36 | |
| 10/15/2023 00:13:20 - INFO - app - STEP = 37 | |
| 10/15/2023 00:13:27 - INFO - app - STEP = 38 | |
| 10/15/2023 00:13:34 - INFO - app - STEP = 39 | |
| 10/15/2023 00:13:40 - INFO - app - STEP = 40 | |
| 10/15/2023 00:13:47 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-40 | |
| 10/15/2023 00:13:47 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-40/optimizer.bin | |
| 10/15/2023 00:13:47 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-40/scheduler.bin | |
| 10/15/2023 00:13:47 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-40/random_states_0.pkl | |
| 10/15/2023 00:13:47 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-40 | |
| 10/15/2023 00:13:47 - INFO - app - STEP = 41 | |
| 10/15/2023 00:13:54 - INFO - app - STEP = 42 | |
| 10/15/2023 00:14:01 - INFO - app - STEP = 43 | |
| 10/15/2023 00:14:08 - INFO - app - STEP = 44 | |
| 10/15/2023 00:14:14 - INFO - app - STEP = 45 | |
| 10/15/2023 00:14:21 - INFO - app - STEP = 46 | |
| 10/15/2023 00:14:28 - INFO - app - STEP = 47 | |
| 10/15/2023 00:14:35 - INFO - app - STEP = 48 | |
| 10/15/2023 00:14:41 - INFO - app - STEP = 49 | |
| 10/15/2023 00:14:48 - INFO - app - STEP = 50 | |
| 10/15/2023 00:14:55 - INFO - app - STEP = 51 | |
| 10/15/2023 00:15:01 - INFO - app - STEP = 52 | |
| 10/15/2023 00:15:08 - INFO - app - STEP = 53 | |
| 10/15/2023 00:15:15 - INFO - app - STEP = 54 | |
| 10/15/2023 00:15:22 - INFO - app - STEP = 55 | |
| 10/15/2023 00:15:28 - INFO - app - STEP = 56 | |
| 10/15/2023 00:15:35 - INFO - app - STEP = 57 | |
| 10/15/2023 00:15:42 - INFO - app - STEP = 58 | |
| 10/15/2023 00:15:49 - INFO - app - STEP = 59 | |
| 10/15/2023 00:15:55 - INFO - app - STEP = 60 | |
| 10/15/2023 00:16:02 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-60 | |
| 10/15/2023 00:16:02 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-60/optimizer.bin | |
| 10/15/2023 00:16:02 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-60/scheduler.bin | |
| 10/15/2023 00:16:02 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-60/random_states_0.pkl | |
| 10/15/2023 00:16:02 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-60 | |
| 10/15/2023 00:16:02 - INFO - app - STEP = 61 | |
| 10/15/2023 00:16:09 - INFO - app - STEP = 62 | |
| 10/15/2023 00:16:16 - INFO - app - STEP = 63 | |
| 10/15/2023 00:16:22 - INFO - app - STEP = 64 | |
| 10/15/2023 00:16:29 - INFO - app - STEP = 65 | |
| 10/15/2023 00:16:36 - INFO - app - STEP = 66 | |
| 10/15/2023 00:16:42 - INFO - app - STEP = 67 | |
| 10/15/2023 00:16:49 - INFO - app - STEP = 68 | |
| 10/15/2023 00:16:56 - INFO - app - STEP = 69 | |
| 10/15/2023 00:17:03 - INFO - app - STEP = 70 | |
| 10/15/2023 00:17:09 - INFO - app - STEP = 71 | |
| 10/15/2023 00:17:16 - INFO - app - STEP = 72 | |
| 10/15/2023 00:17:23 - INFO - app - STEP = 73 | |
| 10/15/2023 00:17:30 - INFO - app - STEP = 74 | |
| 10/15/2023 00:17:36 - INFO - app - STEP = 75 | |
| 10/15/2023 00:17:43 - INFO - app - STEP = 76 | |
| 10/15/2023 00:17:50 - INFO - app - STEP = 77 | |
| 10/15/2023 00:17:56 - INFO - app - STEP = 78 | |
| 10/15/2023 00:18:03 - INFO - app - STEP = 79 | |
| 10/15/2023 00:18:10 - INFO - app - STEP = 80 | |
| 10/15/2023 00:18:16 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-80 | |
| 10/15/2023 00:18:16 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-80/optimizer.bin | |
| 10/15/2023 00:18:16 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-80/scheduler.bin | |
| 10/15/2023 00:18:16 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-80/random_states_0.pkl | |
| 10/15/2023 00:18:16 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-80 | |
| 10/15/2023 00:18:17 - INFO - app - STEP = 81 | |
| 10/15/2023 00:18:23 - INFO - app - STEP = 82 | |
| 10/15/2023 00:18:30 - INFO - app - STEP = 83 | |
| 10/15/2023 00:18:37 - INFO - app - STEP = 84 | |
| 10/15/2023 00:18:44 - INFO - app - STEP = 85 | |
| 10/15/2023 00:18:50 - INFO - app - STEP = 86 | |
| 10/15/2023 00:18:57 - INFO - app - STEP = 87 | |
| 10/15/2023 00:19:04 - INFO - app - STEP = 88 | |
| 10/15/2023 00:19:11 - INFO - app - STEP = 89 | |
| 10/15/2023 00:19:17 - INFO - app - STEP = 90 | |
| 10/15/2023 00:19:24 - INFO - app - STEP = 91 | |
| 10/15/2023 00:19:31 - INFO - app - STEP = 92 | |
| 10/15/2023 00:19:37 - INFO - app - STEP = 93 | |
| 10/15/2023 00:19:44 - INFO - app - STEP = 94 | |
| 10/15/2023 00:19:51 - INFO - app - STEP = 95 | |
| 10/15/2023 00:19:58 - INFO - app - STEP = 96 | |
| 10/15/2023 00:20:04 - INFO - app - STEP = 97 | |
| 10/15/2023 00:20:11 - INFO - app - STEP = 98 | |
| 10/15/2023 00:20:18 - INFO - app - STEP = 99 | |
| 10/15/2023 00:20:24 - INFO - app - STEP = 100 | |
| 10/15/2023 00:20:31 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-100 | |
| 10/15/2023 00:20:31 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-100/optimizer.bin | |
| 10/15/2023 00:20:31 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-100/scheduler.bin | |
| 10/15/2023 00:20:31 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-100/random_states_0.pkl | |
| 10/15/2023 00:20:31 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-100 | |
| 10/15/2023 00:20:31 - INFO - app - STEP = 101 | |
| 10/15/2023 00:20:38 - INFO - app - STEP = 102 | |
| 10/15/2023 00:20:45 - INFO - app - STEP = 103 | |
| 10/15/2023 00:20:51 - INFO - app - STEP = 104 | |
| 10/15/2023 00:20:58 - INFO - app - STEP = 105 | |
| 10/15/2023 00:21:05 - INFO - app - STEP = 106 | |
| 10/15/2023 00:21:12 - INFO - app - STEP = 107 | |
| 10/15/2023 00:21:18 - INFO - app - STEP = 108 | |
| 10/15/2023 00:21:25 - INFO - app - STEP = 109 | |
| 10/15/2023 00:21:32 - INFO - app - STEP = 110 | |
| 10/15/2023 00:21:39 - INFO - app - STEP = 111 | |
| 10/15/2023 00:21:45 - INFO - app - STEP = 112 | |
| 10/15/2023 00:21:52 - INFO - app - STEP = 113 | |
| 10/15/2023 00:21:59 - INFO - app - STEP = 114 | |
| 10/15/2023 00:22:05 - INFO - app - STEP = 115 | |
| 10/15/2023 00:22:12 - INFO - app - STEP = 116 | |
| 10/15/2023 00:22:19 - INFO - app - STEP = 117 | |
| 10/15/2023 00:22:26 - INFO - app - STEP = 118 | |
| 10/15/2023 00:22:32 - INFO - app - STEP = 119 | |
| 10/15/2023 00:22:39 - INFO - app - STEP = 120 | |
| 10/15/2023 00:22:46 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-120 | |
| 10/15/2023 00:22:46 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-120/optimizer.bin | |
| 10/15/2023 00:22:46 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-120/scheduler.bin | |
| 10/15/2023 00:22:46 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-120/random_states_0.pkl | |
| 10/15/2023 00:22:46 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-120 | |
| 10/15/2023 00:22:46 - INFO - app - STEP = 121 | |
| 10/15/2023 00:22:53 - INFO - app - STEP = 122 | |
| 10/15/2023 00:22:59 - INFO - app - STEP = 123 | |
| 10/15/2023 00:23:06 - INFO - app - STEP = 124 | |
| 10/15/2023 00:23:13 - INFO - app - STEP = 125 | |
| 10/15/2023 00:23:20 - INFO - app - STEP = 126 | |
| 10/15/2023 00:23:26 - INFO - app - STEP = 127 | |
| 10/15/2023 00:23:33 - INFO - app - STEP = 128 | |
| 10/15/2023 00:23:40 - INFO - app - STEP = 129 | |
| 10/15/2023 00:23:46 - INFO - app - STEP = 130 | |
| 10/15/2023 00:23:53 - INFO - app - STEP = 131 | |
| 10/15/2023 00:24:00 - INFO - app - STEP = 132 | |
| 10/15/2023 00:24:07 - INFO - app - STEP = 133 | |
| 10/15/2023 00:24:13 - INFO - app - STEP = 134 | |
| 10/15/2023 00:24:20 - INFO - app - STEP = 135 | |
| 10/15/2023 00:24:27 - INFO - app - STEP = 136 | |
| 10/15/2023 00:24:34 - INFO - app - STEP = 137 | |
| 10/15/2023 00:24:40 - INFO - app - STEP = 138 | |
| 10/15/2023 00:24:47 - INFO - app - STEP = 139 | |
| 10/15/2023 00:24:54 - INFO - app - STEP = 140 | |
| 10/15/2023 00:25:00 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-140 | |
| 10/15/2023 00:25:00 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-140/optimizer.bin | |
| 10/15/2023 00:25:00 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-140/scheduler.bin | |
| 10/15/2023 00:25:00 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-140/random_states_0.pkl | |
| 10/15/2023 00:25:00 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-140 | |
| 10/15/2023 00:25:01 - INFO - app - STEP = 141 | |
| 10/15/2023 00:25:07 - INFO - app - STEP = 142 | |
| 10/15/2023 00:25:14 - INFO - app - STEP = 143 | |
| 10/15/2023 00:25:21 - INFO - app - STEP = 144 | |
| 10/15/2023 00:25:27 - INFO - app - STEP = 145 | |
| 10/15/2023 00:25:34 - INFO - app - STEP = 146 | |
| 10/15/2023 00:25:41 - INFO - app - STEP = 147 | |
| 10/15/2023 00:25:48 - INFO - app - STEP = 148 | |
| 10/15/2023 00:25:54 - INFO - app - STEP = 149 | |
| 10/15/2023 00:26:01 - INFO - app - STEP = 150 | |
| 10/15/2023 00:26:08 - INFO - app - STEP = 151 | |
| 10/15/2023 00:26:15 - INFO - app - STEP = 152 | |
| 10/15/2023 00:26:21 - INFO - app - STEP = 153 | |
| 10/15/2023 00:26:28 - INFO - app - STEP = 154 | |
| 10/15/2023 00:26:35 - INFO - app - STEP = 155 | |
| 10/15/2023 00:26:42 - INFO - app - STEP = 156 | |
| 10/15/2023 00:26:48 - INFO - app - STEP = 157 | |
| 10/15/2023 00:26:55 - INFO - app - STEP = 158 | |
| 10/15/2023 00:27:02 - INFO - app - STEP = 159 | |
| 10/15/2023 00:27:08 - INFO - app - STEP = 160 | |
| 10/15/2023 00:27:15 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-160 | |
| 10/15/2023 00:27:15 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-160/optimizer.bin | |
| 10/15/2023 00:27:15 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-160/scheduler.bin | |
| 10/15/2023 00:27:15 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-160/random_states_0.pkl | |
| 10/15/2023 00:27:15 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-160 | |
| 10/15/2023 00:27:15 - INFO - app - STEP = 161 | |
| 10/15/2023 00:27:22 - INFO - app - STEP = 162 | |
| 10/15/2023 00:27:29 - INFO - app - STEP = 163 | |
| 10/15/2023 00:27:36 - INFO - app - STEP = 164 | |
| 10/15/2023 00:27:42 - INFO - app - STEP = 165 | |
| 10/15/2023 00:27:49 - INFO - app - STEP = 166 | |
| 10/15/2023 00:27:56 - INFO - app - STEP = 167 | |
| 10/15/2023 00:28:03 - INFO - app - STEP = 168 | |
| 10/15/2023 00:28:09 - INFO - app - STEP = 169 | |
| 10/15/2023 00:28:16 - INFO - app - STEP = 170 | |
| 10/15/2023 00:28:23 - INFO - app - STEP = 171 | |
| 10/15/2023 00:28:29 - INFO - app - STEP = 172 | |
| 10/15/2023 00:28:36 - INFO - app - STEP = 173 | |
| 10/15/2023 00:28:43 - INFO - app - STEP = 174 | |
| 10/15/2023 00:28:50 - INFO - app - STEP = 175 | |
| 10/15/2023 00:28:56 - INFO - app - STEP = 176 | |
| 10/15/2023 00:29:03 - INFO - app - STEP = 177 | |
| 10/15/2023 00:29:10 - INFO - app - STEP = 178 | |
| 10/15/2023 00:29:17 - INFO - app - STEP = 179 | |
| 10/15/2023 00:29:23 - INFO - app - STEP = 180 | |
| 10/15/2023 00:29:30 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-180 | |
| 10/15/2023 00:29:30 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-180/optimizer.bin | |
| 10/15/2023 00:29:30 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-180/scheduler.bin | |
| 10/15/2023 00:29:30 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-180/random_states_0.pkl | |
| 10/15/2023 00:29:30 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-180 | |
| 10/15/2023 00:29:30 - INFO - app - STEP = 181 | |
| 10/15/2023 00:29:37 - INFO - app - STEP = 182 | |
| 10/15/2023 00:29:44 - INFO - app - STEP = 183 | |
| 10/15/2023 00:29:50 - INFO - app - STEP = 184 | |
| 10/15/2023 00:29:57 - INFO - app - STEP = 185 | |
| 10/15/2023 00:30:04 - INFO - app - STEP = 186 | |
| 10/15/2023 00:30:10 - INFO - app - STEP = 187 | |
| 10/15/2023 00:30:17 - INFO - app - STEP = 188 | |
| 10/15/2023 00:30:24 - INFO - app - STEP = 189 | |
| 10/15/2023 00:30:31 - INFO - app - STEP = 190 | |
| 10/15/2023 00:30:37 - INFO - app - STEP = 191 | |
| 10/15/2023 00:30:44 - INFO - app - STEP = 192 | |
| 10/15/2023 00:30:51 - INFO - app - STEP = 193 | |
| 10/15/2023 00:30:58 - INFO - app - STEP = 194 | |
| 10/15/2023 00:31:04 - INFO - app - STEP = 195 | |
| 10/15/2023 00:31:11 - INFO - app - STEP = 196 | |
| 10/15/2023 00:31:18 - INFO - app - STEP = 197 | |
| 10/15/2023 00:31:25 - INFO - app - STEP = 198 | |
| 10/15/2023 00:31:31 - INFO - app - STEP = 199 | |
| 10/15/2023 00:31:38 - INFO - app - STEP = 200 | |
| 10/15/2023 00:31:44 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-200 | |
| 10/15/2023 00:31:45 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-200/optimizer.bin | |
| 10/15/2023 00:31:45 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-200/scheduler.bin | |
| 10/15/2023 00:31:45 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-200/random_states_0.pkl | |
| 10/15/2023 00:31:45 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-200 | |