{"action":"created","issue":{"url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/1842","repository_url":"https://api.github.com/repos/QubesOS/qubes-issues","labels_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/1842/labels{/name}","comments_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/1842/comments","events_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/1842/events","html_url":"https://github.com/QubesOS/qubes-issues/issues/1842","id":141255474,"number":1842,"title":"Storage Subsystem Design R4","user":{"login":"kalkin","id":145191,"avatar_url":"https://avatars.githubusercontent.com/u/145191?v=3","gravatar_id":"","url":"https://api.github.com/users/kalkin","html_url":"https://github.com/kalkin","followers_url":"https://api.github.com/users/kalkin/followers","following_url":"https://api.github.com/users/kalkin/following{/other_user}","gists_url":"https://api.github.com/users/kalkin/gists{/gist_id}","starred_url":"https://api.github.com/users/kalkin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kalkin/subscriptions","organizations_url":"https://api.github.com/users/kalkin/orgs","repos_url":"https://api.github.com/users/kalkin/repos","events_url":"https://api.github.com/users/kalkin/events{/privacy}","received_events_url":"https://api.github.com/users/kalkin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/QubesOS/qubes-issues/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/QubesOS/qubes-issues/milestones/17","html_url":"https://github.com/QubesOS/qubes-issues/milestones/Release%204.0","labels_url":"https://api.github.com/repos/QubesOS/qubes-issues/milestones/17/labels","id":1083579,"number":17,"title":"Release 4.0","description":"* core-admin rewrite AKA core3","creator":{"login":"marmarek","id":726704,"avatar_url":"https://avatars.githubusercontent.com/u/726704?v=3","gravatar_id":"","url":"https://api.github.com/users/marmarek","html_url":"https://github.com/marmarek","followers_url":"https://api.github.com/users/marmarek/followers","following_url":"https://api.github.com/users/marmarek/following{/other_user}","gists_url":"https://api.github.com/users/marmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/marmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmarek/subscriptions","organizations_url":"https://api.github.com/users/marmarek/orgs","repos_url":"https://api.github.com/users/marmarek/repos","events_url":"https://api.github.com/users/marmarek/events{/privacy}","received_events_url":"https://api.github.com/users/marmarek/received_events","type":"User","site_admin":false},"open_issues":113,"closed_issues":11,"state":"open","created_at":"2015-04-25T20:32:26Z","updated_at":"2016-04-08T13:24:01Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2016-03-16T12:14:29Z","updated_at":"2016-04-10T20:41:10Z","closed_at":null,"body":"# Design\r\nThis is a deisign proposal for the storage subsystem for QubesOS R4.0. For previous discussions see also the PR: [Allow defining and using custom Storage types](https://github.com/QubesOS/qubes-core-admin/pull/6)\r\n\r\n# Requirements\r\n- Multiple storage implementations like file based  (current storage method), LVM Thin Pools, Btrfs, ZFS ...\r\n- [X] 3rd parties can develop custom storage implementations\r\n- [X] uses entry-points for discovery of available storage backends #1238 \r\n- [X] Be independent enough to work on it's own in a separate storage domain\r\n- [X] The AppVM needs to be able to access a read only root image of the TempalteVM, so it can verify it.\r\n- [ ] Import/Export storage volumes between pools\r\n- [X] Domains can reuse different pools for different `StorageVolumes`\r\n- [ ] Differentiate between permanent and temporary attached devices\r\n- [ ] Handle removable devices (or should we instead save permanent?)\r\n- [ ] Pool should know all devices it handles.\r\n\r\n\r\n## Volume\r\n\r\nEncapsulates all data about a volume for serialization to qubes.xml and libvirt config.\r\n\r\n### Interface\r\n\r\n```python\r\nclass Volume:\r\n    devtype = 'disk'  # Used for libvirt <disk device=\"devtype\">\r\n    domain = None  # None is currently the same as dom0\r\n    name = 'root' #  (root|private|volatile|linux-kernel|foo)\r\n    pool = 'qubes_dom0'  # Pool name\r\n    rw = True\r\n    script = None  # Libvirt' s <disk><script /></disk>\r\n    size = 0\r\n    usage = 0\r\n    vid = None # some kind of id must be unique in a pool. XenStorage uses path for that\r\n    volume_type = 'read-write'  # (read-write|read-only|origin|snapshot|volatile)\r\n\r\n    @property\r\n    def config(self):\r\n        ''' return config data for serialization to qubes.xml as (<volume_config>) '''\r\n        return {'name': self.name,\r\n                'pool': self.pool,\r\n                'volume_type': self.volume_type}\r\n\r\n    def block_device(self):\r\n        ''' Return :py:class:`qubes.devices.BlockDevice` for serialization in\r\n            the libvirt XML template as <disk>.\r\n        '''\r\n        return BlockDevice(self.path, self.name, self.script, self.rw,\r\n                           self.domain, self.devtype)\r\n\r\n\r\n```\r\n## Pool\r\nA *Pool* is used to manage different kind of *volumes* (File based/LVM/Btrfs/...).\r\n\r\n### Interface\r\n3rd Parties providing own storage implementations will need to implement the following interface.\r\n\r\n```python\r\nclass Pool:\r\n    def init(self, vm, name, **kwargs) # TODO Refactor vm out\r\n    def init_volume(volume_config) # Creates a Volume object from given volume_config\r\n    def create(self, volume)  # Create volume on disk\r\n    def commit_template_changes(volume)\r\n    def clone(source_volume, target_volume)\r\n    def rename(volume, old_name, new_name)\r\n    def start(volume)\r\n    def stop(volume) \r\n    @property\r\n    def volumes(self): # all volumes managed by this pool, TODO imlement this\r\n```\r\n\r\n## Storage\r\nThe `Storage` class provides managment methods for  domain's volumes. The method are called by the volume at the appropriate time. Currently it's in `qubes/storage/__init__.py`, but I'm considering to move it somewhere else, or make it a part of `QubesVM`, because most of the methods just iterate over `self.vm.volumes` and execute a method. See also [my current Storage version](https://github.com/kalkin/qubes-core-admin/blob/master/qubes/storage/__init__.py#L92-L238)\r\n\r\n### Interface\r\n```python\r\nclass Storage:\r\n    def __init__(vm):\r\n        for conf in vm.volumes_config:  # simplified!\r\n            vm.volumes[conf['name']] = pool.init_volume(conf)\r\n\r\n    def kernels_dir() # backward compatibility will be removed\r\n\r\n    def create_on_disk(self)  # Create volumes on disk, TODO rename to create()\r\n        os.makedirs(self.vm.dir_path)\r\n        for volume in self.vm.volumes.values():\r\n            self.get_pool(volume).create(volume)\r\n\r\n        os.umask(old_umask)\r\n\r\n    def commit_template_changes()\r\n        for v in self.vm.volumes.values():\r\n            if v.volume_type == 'origin':\r\n                self.get_pool(v).commit_template_changes(v)\r\n    def clone(self, src_vm)\r\n    def rename(old_name, new_name)\r\n    def start()\r\n    def stop() \r\n    def verify_files(): # Just does some sanity checks currently.\r\n    def get_pool(volume): # Helper method\r\n        return qubes.storage.get_pool(volume, self.vm) \r\n       # The above will be replaced once vm is refactored out of the pool with: \r\n       return qubes.storage.get_pool(volume, vm.pool_config[volume.pool])\r\n```\r\n\r\n## Further Details\r\n\r\n* `QubesVM.volume_config` would contain a dict {'volume_name':{config}} from the  xml configuration for the current domain.\r\n* `QubesVM.volumes` is a dict containing `{ 'root_img' : Volume, 'private_img' : Volume, ...}`\r\n* `Qubes.pool_config` contains the pool config parsed from qubes.xml. Will be replaced `Qubes.pool_config` with `Qubes.pools` containing `*Pool`\r\n\r\n## Open Questions\r\n- What should be the parameter to `Volume.resize()`? There might be volume implementations which have strategies for shrinking. Should resize also accept smaller sizes as the current one? Or should we even have `extend()`   and `shrink()`\r\n\r\n## EDITS:\r\n* *2016-03-17* Change signature of `Storage.import()`\r\n* *2016-04-10* Document the current API"},"comment":{"url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/comments/208065062","html_url":"https://github.com/QubesOS/qubes-issues/issues/1842#issuecomment-208065062","issue_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/1842","id":208065062,"user":{"login":"marmarek","id":726704,"avatar_url":"https://avatars.githubusercontent.com/u/726704?v=3","gravatar_id":"","url":"https://api.github.com/users/marmarek","html_url":"https://github.com/marmarek","followers_url":"https://api.github.com/users/marmarek/followers","following_url":"https://api.github.com/users/marmarek/following{/other_user}","gists_url":"https://api.github.com/users/marmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/marmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmarek/subscriptions","organizations_url":"https://api.github.com/users/marmarek/orgs","repos_url":"https://api.github.com/users/marmarek/repos","events_url":"https://api.github.com/users/marmarek/events{/privacy}","received_events_url":"https://api.github.com/users/marmarek/received_events","type":"User","site_admin":false},"created_at":"2016-04-10T20:41:10Z","updated_at":"2016-04-10T20:41:10Z","body":"Ok, now it is much clearer for me. Few more questions:\r\n\r\nHow `Volume.vid` got initialized and then saved? Is it simply missing in above example xml? Or maybe it is calculated dynamically by `vm.volumes[name] = pool.init_volume(conf)`?\r\nHow `root` volume of AppVM know which template's root image should be used? Is it done by `pool.init_volume(conf)` too (how?)? or maybe some other method? The same question for `kernel` volume (how does it get `vm.kernel`). \r\nWill `vm1.volumes['root']` be different than `vm2.volumes['root']` (having the same template)? Will it be different than `template.volumes['root']` (I guess so)? How about its `vid`? Will it be the same, but differ in `volume_type` attribute?\r\nI assume for template switch, `vm.volumes['root']` will be replaced by appropriate event handler (`property-set:template` event)."}}