{"action":"created","issue":{"url":"https://api.github.com/repos/obspy/obspy/issues/2206","repository_url":"https://api.github.com/repos/obspy/obspy","labels_url":"https://api.github.com/repos/obspy/obspy/issues/2206/labels{/name}","comments_url":"https://api.github.com/repos/obspy/obspy/issues/2206/comments","events_url":"https://api.github.com/repos/obspy/obspy/issues/2206/events","html_url":"https://github.com/obspy/obspy/issues/2206","id":351745661,"node_id":"MDU6SXNzdWUzNTE3NDU2NjE=","number":2206,"title":"Proposed API for new Client & Indexer submodules based on IRIS time series index","user":{"login":"chad-iris","id":16736618,"node_id":"MDQ6VXNlcjE2NzM2NjE4","avatar_url":"https://avatars1.githubusercontent.com/u/16736618?v=4","gravatar_id":"","url":"https://api.github.com/users/chad-iris","html_url":"https://github.com/chad-iris","followers_url":"https://api.github.com/users/chad-iris/followers","following_url":"https://api.github.com/users/chad-iris/following{/other_user}","gists_url":"https://api.github.com/users/chad-iris/gists{/gist_id}","starred_url":"https://api.github.com/users/chad-iris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chad-iris/subscriptions","organizations_url":"https://api.github.com/users/chad-iris/orgs","repos_url":"https://api.github.com/users/chad-iris/repos","events_url":"https://api.github.com/users/chad-iris/events{/privacy}","received_events_url":"https://api.github.com/users/chad-iris/received_events","type":"User","site_admin":false},"labels":[{"id":289116380,"node_id":"MDU6TGFiZWwyODkxMTYzODA=","url":"https://api.github.com/repos/obspy/obspy/labels/.clients","name":".clients","color":"5882FA","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/obspy/obspy/milestones/11","html_url":"https://github.com/obspy/obspy/milestone/11","labels_url":"https://api.github.com/repos/obspy/obspy/milestones/11/labels","id":1248945,"node_id":"MDk6TWlsZXN0b25lMTI0ODk0NQ==","number":11,"title":"Future release","description":"A release some time in the future, but not the next one.","creator":{"login":"QuLogic","id":302469,"node_id":"MDQ6VXNlcjMwMjQ2OQ==","avatar_url":"https://avatars2.githubusercontent.com/u/302469?v=4","gravatar_id":"","url":"https://api.github.com/users/QuLogic","html_url":"https://github.com/QuLogic","followers_url":"https://api.github.com/users/QuLogic/followers","following_url":"https://api.github.com/users/QuLogic/following{/other_user}","gists_url":"https://api.github.com/users/QuLogic/gists{/gist_id}","starred_url":"https://api.github.com/users/QuLogic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/QuLogic/subscriptions","organizations_url":"https://api.github.com/users/QuLogic/orgs","repos_url":"https://api.github.com/users/QuLogic/repos","events_url":"https://api.github.com/users/QuLogic/events{/privacy}","received_events_url":"https://api.github.com/users/QuLogic/received_events","type":"User","site_admin":false},"open_issues":31,"closed_issues":0,"state":"open","created_at":"2015-08-10T20:33:00Z","updated_at":"2018-08-21T09:43:23Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2018-08-17T21:34:41Z","updated_at":"2018-08-21T18:12:18Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Howdy.  We will very soon begin development on a new Client (and related Indexer) submodules that will allow ObsPy users to transparently read data from a repository of miniSEED that has been indexed with IRIS' [mseedindex](https://github.com/iris-edu/mseedindex) program.  At IRIS we are building a data access client that will download data and construct such an index and these modules will allow ObsPy users to efficiently access their local repository of data.\r\n\r\nThese modules are in the same spirit as `obspy.db` and `obspy.clients.filesystem.sds`, in that they abstract the access to a local repository of data.\r\n\r\nThoughts and feedback welcome.\r\n\r\nWe do not have strong feelings on the suggested module locations, the suggestion in the proposal is just a guess at best fit.\r\n\r\nAre there any guidelines/requirements for connections to databases, i.e. use SQLAlchemy?  By far, the most common use case is with SQLite, but in theory the same system could be used with PostgreSQL.  We will probably write directly for SQLite initially (as most of the code already exists in [another project](https://iris-edu.github.io/portable-fdsnws-dataselect/)) but wanted to think ahead. \r\n\r\n# Proposed APIs:\r\n\r\n## obspy.clients.filesystem.tsindex.Client\r\n\r\n**class Client(sqlitedb, datapath_replace=None)**\r\n\r\nBases: builtins.object\r\n\r\n### Description\r\nTime series extraction client for a database created by the IRIS mseedindex program.  If `datapath_replace` is specified it must be a 2-value tuple, where any occurance of the first value will be replaced with the second value in filename paths from the index. \r\n\r\n### Public Methods\r\n\r\n|Method|Return type|Description|\r\n|--------|------------|-----------|\r\n|**get_waveforms**(network, station, location, channel, starttime, endtime)|obspy.core.stream.Stream|Query tsindex database and read miniSEED data from local indexed directory tree.|\r\n|**get_nslc**(network, station, location, channel, starttime, endtime)|list(tuple)|Return a list of tuples [(net, sta, loc, cha),...] containing information on what streams are included in the tsindex database.|\r\n|**get_availability_extent**(network, station, location, channel, starttime, endtime)|list(tuple)|Return a list of tuples [(net, sta, loc, cha, earliest, latest)] containing data extent info for time series included in the tsindex database.|\r\n|**get_availability**(network, station, location, channel, starttime, endtime, include_sample_rate=False, merge_overlap=False)|list(tuple)|Return a list of tuples [(net, sta, loc, cha, start, end),...] containing data availability info for time series included in the tsindex database.<br><br>If include_sample_rate=True, then a tuple containing the sample rate [(net, sta, loc, cha, start, end, sample_rate),...] is returned. <br><br>If merge_overlap=True, then all time spans that overlap are merged.|\r\n|**get_availability_percentage**(network, station, location, channel, starttime, endtime)|2-tuple of percentage of available data (`0.0` to `1.0`) and number of gaps/overlaps.| Get percentage of available data for a specified network, station, location, channel, starttime, endtime combination.|\r\n|**has_data**(network, station, location, channel, starttime, endtime)|boolean|Check if specified stream has any data using the tsindex database.|\r\n\r\n## obspy.clients.filesystem.tsindex.Indexer\r\n\r\n**class Indexer(root_path, sqlitedb, index_cmd=’mseedindex’, filename_pattern=’*’, parallel=5)**\r\n\r\nBases: builtins.object\r\n\r\n### Description\r\n\r\nBuild an index for miniSEED data using IRIS’s mseedindex program.  Recursively search for files matching `filename_pattern` starting from `root_path` and run `index_cmd` for each target file found that is not already in the index.  After all new files are indexed a summary table is generated with the extents of each time series.\r\n\r\n### Public Methods\r\n\r\n|Method|Return type|Description|\r\n|--------|------------|-----------|\r\n|**run**(build_summary=True, relative_paths=False, reindex=False)|boolean|Execute the file discovery and indexing.<br><br>By default, a summary table is (re)generated containing the extents for each time series in the index.  This can be turned off by setting `build_summary` to False.<br><br>By default, the absolute path to each file is stored in the index.  If `relative_paths` is True, the file paths will be relative to the `root_path`.<br><br>By default, files are not indexed that are already in the index and have not been modified.  The `reindex` option can be set to True to force a re-indexing of all files regardless.|\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/obspy/obspy/issues/comments/414770268","html_url":"https://github.com/obspy/obspy/issues/2206#issuecomment-414770268","issue_url":"https://api.github.com/repos/obspy/obspy/issues/2206","id":414770268,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNDc3MDI2OA==","user":{"login":"chad-iris","id":16736618,"node_id":"MDQ6VXNlcjE2NzM2NjE4","avatar_url":"https://avatars1.githubusercontent.com/u/16736618?v=4","gravatar_id":"","url":"https://api.github.com/users/chad-iris","html_url":"https://github.com/chad-iris","followers_url":"https://api.github.com/users/chad-iris/followers","following_url":"https://api.github.com/users/chad-iris/following{/other_user}","gists_url":"https://api.github.com/users/chad-iris/gists{/gist_id}","starred_url":"https://api.github.com/users/chad-iris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chad-iris/subscriptions","organizations_url":"https://api.github.com/users/chad-iris/orgs","repos_url":"https://api.github.com/users/chad-iris/repos","events_url":"https://api.github.com/users/chad-iris/events{/privacy}","received_events_url":"https://api.github.com/users/chad-iris/received_events","type":"User","site_admin":false},"created_at":"2018-08-21T18:12:18Z","updated_at":"2018-08-21T18:12:18Z","author_association":"CONTRIBUTOR","body":"@Jollyfant \r\n> I'm wondering has anyone ever tried indexing individual mSEED records before? Just the header metadata and a 512 byte blob.\r\n\r\nIt is not something I would recommend for a data set of any non-trivial size as it's simply too much overhead (and redundancy) to be worth it in most cases.  Any row-per-record scheme will not scale very far.\r\n\r\n> One question about the timeindex field. When you're using a hash map you would need the exact timestamp (key) to get the trace byte-offset. So can you round to the nearest key within the database somehow, or will the Client somehow handle this?\r\n\r\nThe `timeindex` field will be used transparently by the Client to narrow the read range of the file for efficiency.  In general, ObsPy users should not need to know any details of the index schema beyond the name of the SQLite file that contains it.  More details on the [timeindex field](https://github.com/iris-edu/mseedindex/wiki/Database-Schema#2-about-timeindex) if you're interested.\r\n\r\n"}}
{"actor":{"display_login":"chad-iris"}}