Spaces:
Running
Running
gradio.exceptions.Error: 'GPU task aborted'
#119
by
robinwitch
- opened
I meet a strange problem while using zerogpu.
Beacuse I find I have a gpu use time limitation, so I just using @spaces.GPU(duration=80) on only one function _warp at row 337 in app.py.
The error like this:
File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 211, in gradio_handler
raise gr.Error("GPU task aborted")
│ └ <class 'gradio.exceptions.Error'>
└ <module 'gradio' from '/usr/local/lib/python3.10/site-packages/gradio/__init__.py'>
gradio.exceptions.Error: 'GPU task aborted'
When completed this decorator function _warp, my process seems like been kill by the error GPU task aborted.
Note that I'm sure my _warp function has complete run, because I add a print sentence in the end of this function, and I can see it has been print in the log!
Hum, can anyone give me some advice? You can simply try my app by click any examples.
My gradio link is https://huggingface.co/spaces/robinwitch/SynTalker.
And more detail question description is in https://huggingface.co/spaces/robinwitch/SynTalker/discussions/1#6707ab0c800440ceb27ba21e
By the way, you may meet error "You have exceeded your GPU quota (80s requested vs. 80s left).", just try another times!
I tried it too, but it still aborted with an error on the way.
The current Zero GPU space' @spaces.GPU decorator is multiprocessing internally, so it doesn't seem to work well with multi-threading and multi-processing...😓
debug5:finish it!
2024-10-10 14:53:47.112 | ERROR | gradio.utils:wrapped:1325 - An error has been caught in function 'wrapped', process 'MainProcess' (1), thread 'AnyIO worker thread' (140271096407744):
Traceback (most recent call last):
File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()
│ └ <function Thread._bootstrap_inner at 0x7f960e11df30>
└ <WorkerThread(AnyIO worker thread, started daemon 140271096407744)>
File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
│ └ <function WorkerThread.run at 0x7f93faa2c550>
└ <WorkerThread(AnyIO worker thread, started daemon 140271096407744)>
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 914, in run
result = context.run(func, *args)
│ │ │ └ (<generator object Examples.sync_lazy_cache at 0x7f93fa931770>,)
│ │ └ <function run_sync_iterator_async at 0x7f95627b03a0>
│ └ <method 'run' of '_contextvars.Context' objects>
└ <_contextvars.Context object at 0x7f93fa7c2380>
File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 647, in run_sync_iterator_async
return next(iterator)
└ <generator object Examples.sync_lazy_cache at 0x7f93fa931770>
File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 809, in gen_wrapper
response = next(iterator)
└ <generator object Examples.sync_lazy_cache at 0x7f93fa933ae0>
File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 451, in sync_lazy_cache
for output in fn(*input_values):
│ │ └ ((16000, array([2134, 4394, 3727, ..., 29, 26, 25], dtype=int32)), 0)
│ └ <function sync_fn_to_generator.<locals>.wrapped at 0x7f93fa915ab0>
└ [None, None]
> File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 1325, in wrapped
yield fn(*args, **kwargs)
│ │ └ {}
│ └ ((16000, array([2134, 4394, 3727, ..., 29, 26, 25], dtype=int32)), 0)
└ <function syntalker at 0x7f93fa0491b0>
File "/home/user/app/app.py", line 755, in syntalker
result = trainer.test_demo(999)
│ └ <function BaseTrainer.test_demo at 0x7f93fab6b370>
└ <__main__.BaseTrainer object at 0x7f93fa8f0310>
File "/home/user/app/app.py", line 270, in test_demo
net_out = _warp(self.args,self.model, batch_data,self.joints,self.joint_mask_upper,self.joint_mask_hands,self.joint_mask_lower,self.use_trans,self.mean_upper,self.mean_hands,self.mean_lower,self.std_upper,self.std_hands,self.std_lower,self.trans_mean,self.trans_std)
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([0.00434298, 0.01658506, 0.00604226])
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([-1.03093567e-05, 1.30673285e+00, 3.21644151e-05])
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([0.15210988, 0.09043588, 0.32370892, 0.04405705, 0.02246166,
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 0.08144826, 0.02109734, 0.10316863, 0.06898032, 0.0...
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([0.06018772, 0.23905125, 0.11087885, 0.24111511, 0.0542646 ,
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 0.05752097, 0.125199 , 0.22886859, 0.06037393, 0.2...
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([0.00276254, 0.04254209, 0.04602624, 0.04832461, 0.02070488,
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 0.07023287, 0.0021067 , 0.03852088, 0.03396116, 0.0...
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([ 0.9268511 , 0.06455171, 0.0322784 , -0.06788209, 0.9549398 ,
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 0.27286723, 0.9663208 , -0.09199008, 0.2048...
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([ 0.93831116, 0.07900303, 0.20092662, -0.06968854, 0.96385115,
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ -0.04870924, 0.89996815, 0.31663465, -0.1351...
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ array([ 0.998115 , 0.00287484, 0.04198178, 0.00790095, 0.96294385,
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ -0.25487295, 0.9989194 , 0.01088245, -0.0377...
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ True
│ │ │ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ │ │ └ array([1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 1., 1., 1., 1., 1.,
│ │ │ │ │ │ │ │ │ │ │ │ │ 1., 0., 0., 0., 1., 1., 1., 1., 1., 1., 0.,...
│ │ │ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ │ │ └ array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
│ │ │ │ │ │ │ │ │ │ │ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,...
│ │ │ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ │ │ └ array([0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0., 0., 0.,
│ │ │ │ │ │ │ │ │ 0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 1.,...
│ │ │ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ │ │ └ 55
│ │ │ │ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ │ │ │ └ {'pose': tensor([[[-0.1800, -0.0463, -0.0809, ..., 1.0000, 1.0000, 1.0000],
│ │ │ │ │ [-0.1806, -0.0473, -0.0812, ..., 1...
│ │ │ │ └ MDM(
│ │ │ │ (WavEncoder): WavEncoder(
│ │ │ │ (feat_extractor): Sequential(
│ │ │ │ (0): BasicBlock(
│ │ │ │ (conv1): Conv1d(2, 64, kern...
│ │ │ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
│ │ └ Namespace(config='configs/diffusion_rvqvae_128_hf.yaml', project='s2g', stat='ts', csv_name='a2g_0', notes='', trainer='diffu...
│ └ <__main__.BaseTrainer object at 0x7f93fa8f0310>
└ <function _warp at 0x7f93fab6b400>
File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 211, in gradio_handler
raise gr.Error("GPU task aborted")
│ └ <class 'gradio.exceptions.Error'>
└ <module 'gradio' from '/usr/local/lib/python3.10/site-packages/gradio/__init__.py'>
gradio.exceptions.Error: 'GPU task aborted'
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/gradio/queueing.py", line 536, in process_events
response = await route_utils.call_process_api(
File "/usr/local/lib/python3.10/site-packages/gradio/route_utils.py", line 322, in call_process_api
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1935, in process_api
result = await self.call_function(
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1532, in call_function
prediction = await utils.async_iteration(iterator)
File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 671, in async_iteration
return await iterator.__anext__()
File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 664, in __anext__
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2405, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 914, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 647, in run_sync_iterator_async
return next(iterator)
File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 809, in gen_wrapper
response = next(iterator)
File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 452, in sync_lazy_cache
output = client_utils.synchronize_async(self._postprocess_output, output)
File "/usr/local/lib/python3.10/site-packages/gradio_client/utils.py", line 855, in synchronize_async
return fsspec.asyn.sync(fsspec.asyn.get_loop(), func, *args, **kwargs) # type: ignore
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 355, in _postprocess_output
return await demo.postprocess_data(demo.default_config.fns[0], output, None)
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1717, in postprocess_data
self.validate_outputs(block_fn, predictions) # type: ignore
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1691, in validate_outputs
raise ValueError(
ValueError: An event handler (syntalker) didn't receive enough output values (needed: 2, received: 1).
Wanted outputs:
[<gradio.components.video.Video object at 0x7f93fa0be860>, <gradio.components.file.File object at 0x7f93fa0bde10>]
Received outputs:
[None]
Emm, sounds like I can't expect to get the return value of @spaces.GPU decorator function, is it right?
Right. I don't know if it's a time-out or some other error, but it aborts in the middle of the process.
So it must be returning strange values.
Compared to the summer before, the timing of when CUDA can be called within the Zero GPU space is much more severe, and many libraries virtually do not work.😓
Yeah, I have fix it out!
I use pickle to save the return object as a file.
Then I load it in my main process.
robinwitch
changed discussion status to
closed
Good job!
By the way, isn't this a typo? Not defined.
# style_feature = motionclip.encoder(batch)['mu'].detach().float() # typo?
style_feature = motionclip_feat.encoder(batch)['mu'].detach().float()
Yeah, it is used in my history code, I forget to delete it. I promise it won't be excuted in this code.
I see. There seems to be no operational problem.