{"action":"created","issue":{"url":"https://api.github.com/repos/borgbackup/borg/issues/4070","repository_url":"https://api.github.com/repos/borgbackup/borg","labels_url":"https://api.github.com/repos/borgbackup/borg/issues/4070/labels{/name}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/4070/comments","events_url":"https://api.github.com/repos/borgbackup/borg/issues/4070/events","html_url":"https://github.com/borgbackup/borg/issues/4070","id":356377246,"node_id":"MDU6SXNzdWUzNTYzNzcyNDY=","number":4070,"title":"How to solve this locking problem?","user":{"login":"ho1ger","id":8535968,"node_id":"MDQ6VXNlcjg1MzU5Njg=","avatar_url":"https://avatars0.githubusercontent.com/u/8535968?v=4","gravatar_id":"","url":"https://api.github.com/users/ho1ger","html_url":"https://github.com/ho1ger","followers_url":"https://api.github.com/users/ho1ger/followers","following_url":"https://api.github.com/users/ho1ger/following{/other_user}","gists_url":"https://api.github.com/users/ho1ger/gists{/gist_id}","starred_url":"https://api.github.com/users/ho1ger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ho1ger/subscriptions","organizations_url":"https://api.github.com/users/ho1ger/orgs","repos_url":"https://api.github.com/users/ho1ger/repos","events_url":"https://api.github.com/users/ho1ger/events{/privacy}","received_events_url":"https://api.github.com/users/ho1ger/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-03T06:46:24Z","updated_at":"2018-09-03T08:59:52Z","closed_at":null,"author_association":"NONE","body":"... yes, I checked the documentation, and google, and also in the IRC.\r\n\r\nI am running borg 1.1.5 on an Ubuntu 18.04.1 LTS machine. The backup target is a cloud storage that I mounted using davfs. The machine starts a cron job every 12 hours and pushes a backup to the cloud. This cron job is running as root.\r\n\r\nI have a bit of an odd situation since a couple of days that I cannot understand or fix: a few days ago I logged into the machine and tried to `borg list` the contents of my repo. I did not get any result but only an error related to locking the repo. The same happened when I tried to manually create a backup, etc.\r\n\r\nInterestingly, the cron job still seems to work (more or less), as it sent me this mail some days earlier:\r\n\r\n    Removed stale shared roster lock for pid 28964.   <-- ??\r\n    Removed stale shared roster lock for pid 28964.   <-- ??\r\n    Initializing cache transaction: Reading config\r\n    Initializing cache transaction: Reading chunks\r\n    Initializing cache transaction: Reading files\r\n\r\n     0% Syncing chunks cache. Processing archive 2018_05_27_18_20_02\r\n     2% Syncing chunks cache. Processing archive 2018_05_31_18_20_01\r\n    ...\r\n    Compacting segments  94%\r\n    Compacting segments  97%\r\n\r\n    Saving files cache\r\n    Saving chunks cache\r\n    Saving cache config\r\n\r\n\r\n    ------------------------------------------------------------------------------\r\n    Archive name: 2018_09_02_12_20_01\r\n    Archive fingerprint: dd199d7a69e0bc4bdcb2865fc07c62f2d08a6116bfd5a210862733974ca1ac60\r\n    Time (start): Sun, 2018-09-02 12:20:11\r\n    Time (end):   Sun, 2018-09-02 12:34:31\r\n    Duration: 14 minutes 19.86 seconds\r\n    Number of files: 11169\r\n    Utilization of max. archive size: 0%\r\n    ------------------------------------------------------------------------------\r\n                          Original size      Compressed size    Deduplicated size\r\n    This archive:               18.50 GB             17.90 GB             46.32 MB\r\n    All archives:              889.64 GB            854.94 GB             20.48 GB\r\n\r\n                          Unique chunks         Total chunks\r\n    Chunk index:                   26141               908537\r\n    ------------------------------------------------------------------------------\r\n\r\n12 hours later, I got a similar mail but without the hint that it removed a lock for a stale backup session. So I actually thought my problem would have been solved by a miracle or so. When I now do `borg list`, I get the complete list of backups but at the end there is a problem:\r\n\r\n    (list of backups)\r\n    Local Exception\r\n    Traceback (most recent call last):\r\n      File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 4253, in main\r\n        exit_code = archiver.run(args)\r\n      File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 4185, in run\r\n        return set_ec(func(args))\r\n      File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 152, in wrapper\r\n        return method(self, args, repository=repository, **kwargs)\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 198, in __exit__\r\n        self.close()\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 415, in close\r\n        self.lock.release()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 387, in release\r\n        self._roster.remove()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 116, in __exit__\r\n        self.release()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 149, in release\r\n        os.rmdir(self.path)\r\n    OSError: [Errno 22] Invalid argument: '/media/PATH_TO_MOUNT/backup/lock.exclusive'\r\n\r\n    Platform: Linux cubi 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64\r\n    Linux: Ubuntu 18.04 bionic\r\n    Borg: 1.1.5  Python: CPython 3.6.5\r\n    PID: 4996  CWD: /root/scripts/backup\r\n    sys.argv: ['/usr/bin/borg', 'list', '/media/PATH_TO_MOUNT/backup']\r\n    SSH_ORIGINAL_COMMAND: None\r\n\r\n    Exception ignored in: <bound method Repository.__del__ of <Repository /media/PATH_TO_MOUNT/backup>>\r\n    Traceback (most recent call last):\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 173, in __del__\r\n        self.close()\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 415, in close\r\n        self.lock.release()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 384, in release\r\n        with self._lock:\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 113, in __enter__\r\n        return self.acquire()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 135, in acquire\r\n        raise LockTimeout(self.path)\r\n    borg.locking.LockTimeout: Failed to create/acquire the lock /media/PATH_TO_MOUNT/backup/lock.exclusive (timeout).\r\n\r\nWhen running the same backup script as used by cron manually, the process fails immediately:\r\n\r\n    Failed to create/acquire the lock /media/PATH_TO_MOUNT/backup/lock.exclusive (timeout).\r\n\r\nThings are getting weirder: I tried to initialize a new borg repo on the webdav and I get the following error, again \r\n\r\n    By default repositories initialized with this version will produce security\r\n    errors if written to with an older version (up to and including Borg 1.0.8).\r\n\r\n    If you want to use these older versions, you can disable the check by running:\r\n    borg upgrade --disable-tam /media/1und1/test\r\n\r\n    See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.\r\n    Exception ignored in: <bound method Repository.__del__ of <Repository /media/1und1/test>>\r\n    Traceback (most recent call last):\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 173, in __del__\r\n        self.close()\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 415, in close\r\n        self.lock.release()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 379, in release\r\n        self._roster.modify(EXCLUSIVE, REMOVE)\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 278, in modify\r\n        elements.remove(self.id)\r\n    KeyError: (('cubi@238368719996246', 5245, 0),)\r\n    Local Exception\r\n    Traceback (most recent call last):\r\n      File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 4253, in main\r\n        exit_code = archiver.run(args)\r\n      File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 4185, in run\r\n        return set_ec(func(args))\r\n      File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 152, in wrapper\r\n        return method(self, args, repository=repository, **kwargs)\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 198, in __exit__\r\n        self.close()\r\n      File \"/usr/lib/python3/dist-packages/borg/repository.py\", line 415, in close\r\n        self.lock.release()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 382, in release\r\n        self._lock.release()\r\n      File \"/usr/lib/python3/dist-packages/borg/locking.py\", line 149, in release\r\n        os.rmdir(self.path)\r\n    OSError: [Errno 22] Invalid argument: '/media/PATH_TO_MOUNT/test/lock.exclusive'\r\n\r\n    Platform: Linux cubi 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64\r\n    Linux: Ubuntu 18.04 bionic\r\n    Borg: 1.1.5  Python: CPython 3.6.5\r\n    PID: 5245  CWD: /root/scripts/backup_test\r\n    sys.argv: ['/usr/bin/borg', 'init', '--encryption=keyfile', '/media/1und1/test']\r\n    SSH_ORIGINAL_COMMAND: None\r\n\r\nHowever, it seems like the repo is created correctly, ...\r\n\r\n    /media/PATH_TO_MOUNT/test/\r\n    ├── config\r\n    ├── data\r\n    │   └── 0\r\n    │       ├── 0\r\n    │       └── 1\r\n    ├── hints.1\r\n    ├── index.1\r\n    ├── integrity.1\r\n    ├── lock.exclusive\r\n    ├── nonce\r\n    └── README\r\n\r\nbut an attempt to run a backup fails with the same problem as described earlier. I absolutely do not understand what is going on here, so any help would be appreciated.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/borgbackup/borg/issues/comments/418049114","html_url":"https://github.com/borgbackup/borg/issues/4070#issuecomment-418049114","issue_url":"https://api.github.com/repos/borgbackup/borg/issues/4070","id":418049114,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODA0OTExNA==","user":{"login":"ThomasWaldmann","id":356103,"node_id":"MDQ6VXNlcjM1NjEwMw==","avatar_url":"https://avatars0.githubusercontent.com/u/356103?v=4","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T08:59:52Z","updated_at":"2018-09-03T08:59:52Z","author_association":"MEMBER","body":"Stale lock means that the lock is there, but the process that created it is not running (any more).\r\n\r\nUsually borg cleans up locks when it terminates, but sudden death (like OS/hw crash, power failure) or a borg bug can leave locks behind.\r\n\r\nThe stuff seen above doesn't look like a well-behaved filesystem (or a stable connection), maybe that is the problem."}}
{"actor":{"display_login":"ThomasWaldmann"}}