{"action":"created","issue":{"url":"https://api.github.com/repos/terraref/computing-pipeline/issues/81","repository_url":"https://api.github.com/repos/terraref/computing-pipeline","labels_url":"https://api.github.com/repos/terraref/computing-pipeline/issues/81/labels{/name}","comments_url":"https://api.github.com/repos/terraref/computing-pipeline/issues/81/comments","events_url":"https://api.github.com/repos/terraref/computing-pipeline/issues/81/events","html_url":"https://github.com/terraref/computing-pipeline/issues/81","id":144067568,"number":81,"title":"Add hyperspectral data extractor to pipeline","user":{"login":"dlebauer","id":464871,"avatar_url":"https://avatars.githubusercontent.com/u/464871?v=3","gravatar_id":"","url":"https://api.github.com/users/dlebauer","html_url":"https://github.com/dlebauer","followers_url":"https://api.github.com/users/dlebauer/followers","following_url":"https://api.github.com/users/dlebauer/following{/other_user}","gists_url":"https://api.github.com/users/dlebauer/gists{/gist_id}","starred_url":"https://api.github.com/users/dlebauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dlebauer/subscriptions","organizations_url":"https://api.github.com/users/dlebauer/orgs","repos_url":"https://api.github.com/users/dlebauer/repos","events_url":"https://api.github.com/users/dlebauer/events{/privacy}","received_events_url":"https://api.github.com/users/dlebauer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/terraref/computing-pipeline/labels/2%20-%20Working%20%3C=%205","name":"2 - Working <= 5","color":"CCCCCC"},{"url":"https://api.github.com/repos/terraref/computing-pipeline/labels/Review","name":"Review","color":"009800"},{"url":"https://api.github.com/repos/terraref/computing-pipeline/labels/sensor%20data","name":"sensor data","color":"bfe5bf"}],"state":"open","locked":false,"assignee":{"login":"Zodiase","id":1815174,"avatar_url":"https://avatars.githubusercontent.com/u/1815174?v=3","gravatar_id":"","url":"https://api.github.com/users/Zodiase","html_url":"https://github.com/Zodiase","followers_url":"https://api.github.com/users/Zodiase/followers","following_url":"https://api.github.com/users/Zodiase/following{/other_user}","gists_url":"https://api.github.com/users/Zodiase/gists{/gist_id}","starred_url":"https://api.github.com/users/Zodiase/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zodiase/subscriptions","organizations_url":"https://api.github.com/users/Zodiase/orgs","repos_url":"https://api.github.com/users/Zodiase/repos","events_url":"https://api.github.com/users/Zodiase/events{/privacy}","received_events_url":"https://api.github.com/users/Zodiase/received_events","type":"User","site_admin":false},"assignees":[{"login":"Zodiase","id":1815174,"avatar_url":"https://avatars.githubusercontent.com/u/1815174?v=3","gravatar_id":"","url":"https://api.github.com/users/Zodiase","html_url":"https://github.com/Zodiase","followers_url":"https://api.github.com/users/Zodiase/followers","following_url":"https://api.github.com/users/Zodiase/following{/other_user}","gists_url":"https://api.github.com/users/Zodiase/gists{/gist_id}","starred_url":"https://api.github.com/users/Zodiase/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zodiase/subscriptions","organizations_url":"https://api.github.com/users/Zodiase/orgs","repos_url":"https://api.github.com/users/Zodiase/repos","events_url":"https://api.github.com/users/Zodiase/events{/privacy}","received_events_url":"https://api.github.com/users/Zodiase/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/terraref/computing-pipeline/milestones/25","html_url":"https://github.com/terraref/computing-pipeline/milestones/July%202016","labels_url":"https://api.github.com/repos/terraref/computing-pipeline/milestones/25/labels","id":1870076,"number":25,"title":"July 2016","description":"","creator":{"login":"rachelshekar","id":11740606,"avatar_url":"https://avatars.githubusercontent.com/u/11740606?v=3","gravatar_id":"","url":"https://api.github.com/users/rachelshekar","html_url":"https://github.com/rachelshekar","followers_url":"https://api.github.com/users/rachelshekar/followers","following_url":"https://api.github.com/users/rachelshekar/following{/other_user}","gists_url":"https://api.github.com/users/rachelshekar/gists{/gist_id}","starred_url":"https://api.github.com/users/rachelshekar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rachelshekar/subscriptions","organizations_url":"https://api.github.com/users/rachelshekar/orgs","repos_url":"https://api.github.com/users/rachelshekar/repos","events_url":"https://api.github.com/users/rachelshekar/events{/privacy}","received_events_url":"https://api.github.com/users/rachelshekar/received_events","type":"User","site_admin":false},"open_issues":24,"closed_issues":2,"state":"open","created_at":"2016-07-07T14:20:02Z","updated_at":"2016-07-19T16:06:23Z","due_on":"2016-07-31T05:00:00Z","closed_at":null},"comments":49,"created_at":"2016-03-28T19:55:36Z","updated_at":"2016-07-28T17:00:07Z","closed_at":null,"body":"The steps for converting the sensor binary output to a netcdf data product are documented (https://github.com/terraref/documentation/blob/master/hyperspectral_data_pipeline.md).\r\n\r\nNext steps:\r\n\r\n* [ ] Create extractor (docker image?) contact @czender w/ questions (#86)\r\n  * requires development version of nco installed, on Roger or in Docker container (?)\r\n```sh\r\ngit clone git@github.com:/nco/nco.git\r\ncd nco\r\nconfigure --prefix=/usr/local\r\nmake install.\r\n```\r\n* [ ] add converter to Clowder\r\n* [ ] run on files on Roger \r\n* [ ] document"},"comment":{"url":"https://api.github.com/repos/terraref/computing-pipeline/issues/comments/235957861","html_url":"https://github.com/terraref/computing-pipeline/issues/81#issuecomment-235957861","issue_url":"https://api.github.com/repos/terraref/computing-pipeline/issues/81","id":235957861,"user":{"login":"Zodiase","id":1815174,"avatar_url":"https://avatars.githubusercontent.com/u/1815174?v=3","gravatar_id":"","url":"https://api.github.com/users/Zodiase","html_url":"https://github.com/Zodiase","followers_url":"https://api.github.com/users/Zodiase/followers","following_url":"https://api.github.com/users/Zodiase/following{/other_user}","gists_url":"https://api.github.com/users/Zodiase/gists{/gist_id}","starred_url":"https://api.github.com/users/Zodiase/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zodiase/subscriptions","organizations_url":"https://api.github.com/users/Zodiase/orgs","repos_url":"https://api.github.com/users/Zodiase/repos","events_url":"https://api.github.com/users/Zodiase/events{/privacy}","received_events_url":"https://api.github.com/users/Zodiase/received_events","type":"User","site_admin":false},"created_at":"2016-07-28T17:00:07Z","updated_at":"2016-07-28T17:00:07Z","body":"@czender I changed to using `apt-get install nco` instead of building it myself and passed that error. Now here comes a new one:\r\n\r\n```\r\ninvoking terraref.sh to create: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083.nc4\r\ndirname: missing operand\r\nTry 'dirname --help' for more information.\r\nInput #00: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw\r\ntrn(in)  : /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw\r\ntrn(out) : /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp\r\nncks: unrecognized option '--trr_wxy=955,1600,0'\r\nncks Command line options cheatsheet (full details at http://nco.sf.net/nco.html#ncks):\r\nncks [-3] [-4] [-5] [-6] [-7] [-A] [-a] [-b fl_bnr] [--bfr sz] [-C] [-c] [--cdl] [--cnk_byt sz] [--cnk_dmn nm,sz] [--cnk_map map] [--cnk_min min] [--cnk_plc plc] [--cnk_scl sz] [-D nco_dbg_lvl] [-d ...] [-F] [--fix_rec_dmn dim] [--fl_fmt fmt] [-G grp:lvl] [-g ...] [--glb ...] [--grp_xtr_var_xcl] [-H] [-h] [--hdn] [--hdr_pad nbr] [-L lvl] [-l path] [-M] [-m] [--map map.nc] [--md5_dgs] [--md5_wrt] [--mk_rec_dmn dim] [--msa] [--no_blank] [--no_tmp_fl] [-O] [-o out.nc] [-P] [-p path] [--ppc ...] [-Q] [-q] [-R] [-r] [--rad] [--rgr] [--ram_all] [--rnr=wgt] [-s format] [-t thr_nbr] [-u] [--unn] [-V] [-v ...] [-X box] [-x] [--xml] [--xml_no_loc] [--xml_spr_chr sng] [--xml_spr_nmr sng] [--xtn_var ...] in.nc [[out.nc]]\r\n\r\n-3, --3, --fl_fmt=classic\tOutput file in netCDF3 CLASSIC (32-bit offset) storage format\r\n-4, --4, --netcdf4\t\tOutput file in netCDF4 (HDF5) storage format\r\n-5, --5, \t\tPrint alphabetically by group then variable\r\n-6, --6, --64, --fl_fmt=64bit\tOutput file in netCDF3 64-bit offset storage format\r\n-7, --7, --fl_fmt=netcdf4_classic\tOutput file in netCDF4 CLASSIC format (3+4=7)\r\n-A, --apn, --append\tAppend to existing output file, if any\r\n-a, --abc, --alphabetize\tDisable alphabetization of extracted variables\r\n-b, --fl_bnr, --binary-file fl_bnr\tUnformatted binary file to write\r\n    --bfr_sz, --buffer_size sz\tBuffer size to open files with\r\n-C, --nocoords\t\tAssociated coordinate variables should not be processed\r\n-c, --crd, --coords\tCoordinate variables will all be processed\r\n    --cdl\t\tPrint CDL (netCDF lingua franca used by ncdump/ncgen)\r\n    --cnk_dmn, --chunk_dimension nm,sz\tChunksize of dimension nm is sz\r\n    --cnk_map, --chunk_map map\t\tChunking map [dmn,lfp,prd,rd1,rew,scl,xpl,xst]\r\n    --cnk_min, --chunk_min min\t\tMinimum size [B] of variable to chunk\r\n    --cnk_plc, --chunk_policy plc\tChunking policy [all,g2d,g3d,xpl,xst,uck]\r\n    --cnk_scl, --chunk_scalar sz\tChunksize scalar (for all dimensions)\r\n-D, --dbg_lvl, --debug-level lvl\tDebug-level is lvl\r\n-d, --dmn, --dimension dim,[min][,[max]][,[stride]] Dimension's limits and stride in hyperslab\r\n-F, --ftn, --fortran\tFortran indexing conventions (1-based) for I/O\r\n    --fix_rec_dmn dim\tChange dimension dim (or all) to fixed dimension in output file\r\n    --fl_fmt, --file_format fmt\tFile format [classic,64bit,netcdf4,netcdf4_classic]\r\n-G, --gpe [grp_nm][:[lvl]]\tGroup Path Editing path, levels to replace\r\n-g, --grp grp1[,grp2[...]] Group(s) to process (regular expressions supported)\r\n    --glb_att_add nm=val\tGlobal attribute to add\r\n-H, --data, --hieronymus\tToggle printing data\r\n-h, --hst, --history\tDo not append to \"history\" global attribute\r\n    --hdn, --hidden\tPrint hidden (aka special) attributes\r\n    --hdr_pad, --header_pad\tPad output header with nbr bytes\r\n-L, --dfl_lvl, --deflate lvl\tLempel-Ziv deflation/compression (lvl=0..9) for netCDF4 output\r\n-l, --lcl, --local path\tLocal storage path for remotely-retrieved files\r\n-M, --Mtd, --Metadata\tToggle printing global metadata\r\n-m, --mtd, --metadata\tToggle printing variable metadata\r\n    --map, --rgr_map map.nc\tFile containing (ESMF- or SCRIP-format) weights to regrid input to output grid\r\n    --mk_rec_dmn dim\tDefine dim as record dimension in output file\r\n    --no_blank\t\tPrint numeric missing values instead of blanks (underscores)\r\n    --no_tmp_fl\t\tDo not write output to temporary file\r\n-o, --output, --fl_out out.nc\tOutput file name (or use last positional argument)\r\n-O, --ovr, --overwrite\tOverwrite existing output file, if any\r\n-P, --prn, --print\tPrint data, metadata, and units. Abbreviation for -C -H -M -m -u.\r\n-p, --pth, --path path\tPath prefix for all input filenames\r\n    --ppc v1[,v2[,...]]=prc\tPrecision-Preserving Compression: Number of total or decimal significant digits\r\n-Q, --quiet\t\tQuiet printing of dimension indices and coordinate values\r\n-q, --quench\t\tQuench (turn-off) all printing to screen\r\n-R, --rtn, --retain\tRetain remotely-retrieved files after use\r\n-r, --revision, --version\tCompile-time configuration and/or program version\r\n    --orphan, --rph_dmn\tRetain or print all (including orphaned) dimensions\r\n    --ram_all, --diskless_all\tOpen netCDF3 files and create output files in RAM\r\n    --rgr key=value\t\tSee http://nco.sf.net/nco.html#regrid for valid keys\r\n    --rnr=wgt, --renormalize\t\tWeight threshold for valid destination value\r\n-s, --sng_fmt, --string format\tString format for text output\r\n-t, --thr_nbr, --threads, --omp_num_threads thr_nbr\tThread number for OpenMP\r\n-u, --units\t\tToggle printing units of variables, if any\r\n    --unn, --union\tSelect union of specified groups and variables\r\n-V, --var_val\t\tPrint variable values only\r\n-v, --variable var1[,var2[...]] Variable(s) to process (regular expressions supported)\r\n-X, --auxiliary lon_min,lon_max,lat_min,lat_max\tAuxiliary coordinate bounding box\r\n-x, --xcl, --exclude\tExtract all variables EXCEPT those specified with -v\r\n    --xml\t\tPrint XML (NcML, netCDF Markup Language)\r\n    --xml_no_location\tOmit NcML location element\r\n    --xml_spr_chr sng\tSeparator for NcML character types\r\n    --xml_spr_nmr sng\tSeparator for NcML numeric types\r\n    --xtn_var, --extensive var\tExtensive variables for regridding (summed not averaged)\r\nin.nc\t\t\tInput file name\r\nout.nc\t\t\tOutput file name (or use -o switch)\r\n\r\nEight ways to find more help on ncks and/or NCO:\r\n1. Examples:     http://nco.sf.net/nco.html#xmp_ncks\r\n2. Ref. manual:  http://nco.sf.net/nco.html#ncks\r\n3. User Guide:   http://nco.sf.net#RTFM\r\n4. Manual pages: 'man ncks', 'man nco', ...\r\n5. Homepage:     http://nco.sf.net\r\n6. FAQ:          http://nco.sf.net#FAQ\r\n7. Help Forum:   http://sf.net/p/nco/discussion/9830\r\n8. Publications: http://nco.sf.net#pub\r\nPost questions, suggestions, patches at http://sf.net/projects/nco\r\nterraref.sh: ERROR Failed to translate raw data. Debug this:\r\nncks -O --trr_wxy=955,1600,0 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw ~zender/nco/data/in.nc /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp\r\n```\r\n\r\nI'm still very concerned about the first error `dirname: missing operand` which seems to be from this line: https://github.com/terraref/computing-pipeline/blob/master/scripts/hyperspectral/terraref.sh#L65. "}}
{"actor":{"display_login":"Zodiase"}}