{"push_id":3331683950,"size":8,"distinct_size":4,"ref":"refs/heads/update-index-mappings","head":"ae0de03e61c55aaf824f6c1dadd1ceb3a575aca3","before":"1380e0e98220edc08b6e8b23dafc3f4c9dcb3739","commits":[{"sha":"521cd430c64e2c497b761bf1f5f3a78dc1fb807b","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Reduce # of util-scripts mock'd in bench-utils\n\nRemove symlinks to the util-scripts directory and just include\nutil-scripts itself in PATH of unit tests.  We also remove the\nsymlink to pbench-collect-sysinfo to help expose its bad behavior\nof also calling pbench-metadata-log (causing it to be called\ntwice as most bench-scripts call it directly).","distinct":false,"url":"https://api.github.com/repos/portante/pbench/commits/521cd430c64e2c497b761bf1f5f3a78dc1fb807b"},{"sha":"66a1c883c75e73efa6d470956cb675fb0eae0370","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Stop calling metadata-log from collect-sysinfo","distinct":false,"url":"https://api.github.com/repos/portante/pbench/commits/66a1c883c75e73efa6d470956cb675fb0eae0370"},{"sha":"0fd5919dadf0792422f53d3fd6d9ee9a2b5421ee","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Add additional error checking to pbench-metadata-log\n\nPerform some additional clean up where possible.","distinct":false,"url":"https://api.github.com/repos/portante/pbench/commits/0fd5919dadf0792422f53d3fd6d9ee9a2b5421ee"},{"sha":"c26c354eea5242a55a7b6e1c0434e3454d8bc92a","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Move sed commands into common shared function","distinct":false,"url":"https://api.github.com/repos/portante/pbench/commits/c26c354eea5242a55a7b6e1c0434e3454d8bc92a"},{"sha":"abf8bcb82ab8c25071ecc4473a31008d4d9115ab","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Add ignore for *~","distinct":true,"url":"https://api.github.com/repos/portante/pbench/commits/abf8bcb82ab8c25071ecc4473a31008d4d9115ab"},{"sha":"009aa9030188ed1b5b2940a119ee7583c5032a76","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Fix server unittests to emit diff properly\n\nAnd only match multiple tests when test argument ends in \".\". This avoids\nan argument like, \"test-7.1\", matching test-7.1, and test-7.10, etc.","distinct":true,"url":"https://api.github.com/repos/portante/pbench/commits/009aa9030188ed1b5b2940a119ee7583c5032a76"},{"sha":"c30c253fda68ba8e4130cc80150830828a371589","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Remove spaces from field names\n\nUnfortunately, we had to update the samples by hand since the new code\nexpects them removed.","distinct":true,"url":"https://api.github.com/repos/portante/pbench/commits/c30c253fda68ba8e4130cc80150830828a371589"},{"sha":"ae0de03e61c55aaf824f6c1dadd1ceb3a575aca3","author":{"name":"Peter Portante","email":"1768998915a9d75a38099dd28e7e828cd5a8c6fe@redhat.com"},"message":"Update index document models to fix data problems\n\nThis is a significant refactoring of the index mappings for all document\ntypes in order to fix problems with missing fields, adjust how\ndata is indexed for usability, correcting data types, fixing field name\nconflicts, while updating various fields using proper tokenizers and/or\nanalyzers to ensure their data is indexed in a more natural or usable\nway.\n\nThe driving factor for this work was the fact that there were no\nresult-data templates being created, and what we had in the code base\nwas completely wrong.  We completely replaced the existing result-data\nmapping with one which matches the documents currently generated. And\nsince it wasn't actually being used, we add code to ensure that template\nis in place.\n\nWe add `_meta` version fields to the templates to record the indexer\nversion and mapping version. We now query to see if the previous\nversion is older, and update it accordingly.\n\nAs a result of adding the version, the index names have been changed to\nadd a version number that matches the mapping in the template.  This\nlets us easily re-index data unto the new mapping, while allowing the\nconsumers of the index to use the new and old version indices at the\nsame time opportunistically. We now enforce the index prefix setting\nnot contain the period character, since that separates the various\nparts of the index name, <prefix>.<version>.<name>.<YYYY-MM-DD>.\n\nWhile doing this work, we found a bug in discovering the tool data\nfiles in the tar ball. Sometimes the full hostnam is being used instead\nof the short hostname.  So now we look for tool data by short and long\nhostname.\n\nWe turn off the use of the _all field for all document types, and add a\n`default_field` setting for each type so that unspecified searches don't\ntouch every field.  This required us to add a \"settings\" infrastructure\nper index name.\n\nWe fix the indexing for mpstat to emit floats, prevent duplicate mpstat\ndata, have prometheus-metrics emit fields in its namespace, have\nproc-vmstat index similar to the way the proc-vmstat-postprocess works\n(although we index both the original value and the computed rate), add\nproper indexing of the proc-interrupts data, and generally correct many\ndata types.\n\nIn order to ensure and test all these changes, we re-worked the unit\ntest mock infrastructure to work with less specific interactions in\nour normal code, moving it to its own module, and adding the ability\nto compare generated JSON documents against the mappings to flag\nproblems. We also moved more Elasticsearch infrastructure into the\npbench library module.\n\nWe add a unit test, 7.17, to verify vmstat tool data indexing works.\n\nSince we are re-working toc-entries in this effort, we take the\nopportunity to add missing fields for directories, and ensure all empty\ndirectories are indexed as well. This is in preparation for viewing tar\nball contents via the JSON documents only.\n\nWhen we get BadDate exceptions from processing the pbench tar ball we will\nexit with that status.  But when we encounter these errors while dealing with\nresult and tool data, we have already indexed the run document, and other tool\ndata, so ending early is not too useful.  We now continue to index so that we\ncan report all the errors, indexing what we can.\n\nFIXME: fix tar file usage for sosreports\n\nWIP: use sample parent docs, fix controller_host, run/iteration/sample\nnamespaces, add @idx field to record index (or offset) of data in source\nfiles","distinct":true,"url":"https://api.github.com/repos/portante/pbench/commits/ae0de03e61c55aaf824f6c1dadd1ceb3a575aca3"}]}
{"actor":{"display_login":"portante"}}