{"action":"created","issue":{"url":"https://api.github.com/repos/threatgrid/ctia/issues/645","repository_url":"https://api.github.com/repos/threatgrid/ctia","labels_url":"https://api.github.com/repos/threatgrid/ctia/issues/645/labels{/name}","comments_url":"https://api.github.com/repos/threatgrid/ctia/issues/645/comments","events_url":"https://api.github.com/repos/threatgrid/ctia/issues/645/events","html_url":"https://github.com/threatgrid/ctia/issues/645","id":293664294,"number":645,"title":"CTIA ignoring external-key-prefixes query parameter","user":{"login":"saintx","id":711320,"avatar_url":"https://avatars1.githubusercontent.com/u/711320?v=4","gravatar_id":"","url":"https://api.github.com/users/saintx","html_url":"https://github.com/saintx","followers_url":"https://api.github.com/users/saintx/followers","following_url":"https://api.github.com/users/saintx/following{/other_user}","gists_url":"https://api.github.com/users/saintx/gists{/gist_id}","starred_url":"https://api.github.com/users/saintx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saintx/subscriptions","organizations_url":"https://api.github.com/users/saintx/orgs","repos_url":"https://api.github.com/users/saintx/repos","events_url":"https://api.github.com/users/saintx/events{/privacy}","received_events_url":"https://api.github.com/users/saintx/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-01T20:00:51Z","updated_at":"2018-02-01T20:01:49Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"CTIA appears to be ignoring my external-key-prefixes query param and creating duplicate objects from identical data, ignoring the external ID.  In some cases, it is giving the objects their own UUID.  In other cases, it is creating records in the store for the transient IDs.\r\n\r\nSee notes below for detail."},"comment":{"url":"https://api.github.com/repos/threatgrid/ctia/issues/comments/362384768","html_url":"https://github.com/threatgrid/ctia/issues/645#issuecomment-362384768","issue_url":"https://api.github.com/repos/threatgrid/ctia/issues/645","id":362384768,"user":{"login":"saintx","id":711320,"avatar_url":"https://avatars1.githubusercontent.com/u/711320?v=4","gravatar_id":"","url":"https://api.github.com/users/saintx","html_url":"https://github.com/saintx","followers_url":"https://api.github.com/users/saintx/followers","following_url":"https://api.github.com/users/saintx/following{/other_user}","gists_url":"https://api.github.com/users/saintx/gists{/gist_id}","starred_url":"https://api.github.com/users/saintx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saintx/subscriptions","organizations_url":"https://api.github.com/users/saintx/orgs","repos_url":"https://api.github.com/users/saintx/repos","events_url":"https://api.github.com/users/saintx/events{/privacy}","received_events_url":"https://api.github.com/users/saintx/received_events","type":"User","site_admin":false},"created_at":"2018-02-01T20:01:49Z","updated_at":"2018-02-01T20:01:49Z","author_association":"CONTRIBUTOR","body":"## Setup\r\n\r\nI have a file of preprocessed SSLBL entries which I can convert into bundles and import into my local CTIA server.  This lets me perform some integration testing locally, to ensure that I'm not creating duplicates, the import mechanism is adequately performant, and that the imported data itself conforms to my expectations.\r\n\r\nEach bundle is posted using the following function:\r\n\r\n```\r\n(defn post-bundle\r\n  [ctia-base-uri ctia-api-key bundle]\r\n  (let [options {:throw-exceptions true\r\n                 :body (pr-str bundle)\r\n                 :content-type :edn\r\n                 :accept :edn\r\n                 :as :clojure\r\n                 :query-params {\"external-key-prefixes\" \"hydrant-\"}\r\n                 :headers {\"Authorization\" ctia-api-key}}\r\n        results (try (http/post (str ctia-base-uri \"/ctia/bundle/import\")\r\n                                options)\r\n                     (catch Exception e\r\n                       (log/error e)))]\r\n    results))\r\n```\r\n\r\nNone of my requests yield errors.\r\n\r\nAs you can see, I include `:query-params {\"external-key-prefixes\" \"hydrant-\"}` in order to prevent duplication of documents in storage that use the `hydrant-` external ID prefix.\r\n\r\n## Problem\r\n\r\nI am seeing duplication of certain objects.  I have verified that this is not due to subtle differences between the bundle objects between runs.\r\n\r\n## First Run\r\n\r\nAs one would expect, the first run of my importer against my local CTIA server transformed the transient IDs of each relationship into UUIDs:\r\n\r\n```\r\n{\r\n  \"took\": 1,\r\n  \"timed_out\": false,\r\n  \"_shards\": {\r\n    \"total\": 5,\r\n    \"successful\": 5,\r\n    \"failed\": 0\r\n  },\r\n  \"hits\": {\r\n    \"total\": 1,\r\n    \"max_score\": 7.729472,\r\n    \"hits\": [\r\n      {\r\n        \"_index\": \"ctia_relationship\",\r\n        \"_type\": \"relationship\",\r\n        \"_id\": \"relationship-cbbb3244-57b7-4ad4-bacb-de90d86f9971\",\r\n        \"_score\": 7.729472,\r\n        \"_source\": {\r\n          \"schema_version\": \"0.4.23\",\r\n          \"target_ref\": \"http://localhost:3000/ctia/indicator/indicator-b7f8235a-4f54-48ee-8d31-4a94ed1b0863\",\r\n          \"type\": \"relationship\",\r\n          \"created\": \"2018-02-01T19:04:45.683Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-aea5c8de8a2b66f47457c8c93b2615cc646eb735eccd635549560481dec07978\"\r\n          ],\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"source_ref\": \"http://localhost:3000/ctia/sighting/sighting-df3292cb-64f8-4303-8bb3-f85727ddcfde\",\r\n          \"id\": \"relationship-cbbb3244-57b7-4ad4-bacb-de90d86f9971\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"owner\": \"Unknown\",\r\n          \"relationship_type\": \"element-of\"\r\n        }\r\n      }\r\n    ]\r\n  }\r\n}\r\n```\r\n\r\n## Second Run\r\n\r\nRunning my batch import a 2nd time created duplicate relationship documents in ES, which share the external ID of the original, and bear a distinct \"created\" date.  Instead of using the existing ID, however, they retain their transient IDs:\r\n\r\n```\r\n{\r\n  \"took\": 1,\r\n  \"timed_out\": false,\r\n  \"_shards\": {\r\n    \"total\": 5,\r\n    \"successful\": 5,\r\n    \"failed\": 0\r\n  },\r\n  \"hits\": {\r\n    \"total\": 2,\r\n    \"max_score\": 7.729472,\r\n    \"hits\": [\r\n      {\r\n        \"_index\": \"ctia_relationship\",\r\n        \"_type\": \"relationship\",\r\n        \"_id\": \"relationship-cbbb3244-57b7-4ad4-bacb-de90d86f9971\",\r\n        \"_score\": 7.729472,\r\n        \"_source\": {\r\n          \"schema_version\": \"0.4.23\",\r\n          \"target_ref\": \"http://localhost:3000/ctia/indicator/indicator-b7f8235a-4f54-48ee-8d31-4a94ed1b0863\",\r\n          \"type\": \"relationship\",\r\n          \"created\": \"2018-02-01T19:04:45.683Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-aea5c8de8a2b66f47457c8c93b2615cc646eb735eccd635549560481dec07978\"\r\n          ],\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"source_ref\": \"http://localhost:3000/ctia/sighting/sighting-df3292cb-64f8-4303-8bb3-f85727ddcfde\",\r\n          \"id\": \"relationship-cbbb3244-57b7-4ad4-bacb-de90d86f9971\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"owner\": \"Unknown\",\r\n          \"relationship_type\": \"element-of\"\r\n        }\r\n      },\r\n      {\r\n        \"_index\": \"ctia_relationship\",\r\n        \"_type\": \"relationship\",\r\n        \"_id\": \"relationship-f644df64-ea63-4156-887a-145e44ac2f0d\",\r\n        \"_score\": 7.729472,\r\n        \"_source\": {\r\n          \"schema_version\": \"0.4.23\",\r\n          \"target_ref\": \"transient:2bc9fe1d-260b-4635-be33-85d7f53ecc31\",\r\n          \"type\": \"relationship\",\r\n          \"created\": \"2018-02-01T19:13:38.904Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-aea5c8de8a2b66f47457c8c93b2615cc646eb735eccd635549560481dec07978\"\r\n          ],\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"source_ref\": \"transient:39ce0976-6dc6-4ec9-b106-4ccc2d369f7a\",\r\n          \"id\": \"relationship-f644df64-ea63-4156-887a-145e44ac2f0d\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"owner\": \"Unknown\",\r\n          \"relationship_type\": \"element-of\"\r\n        }\r\n      }\r\n    ]\r\n  }\r\n}\r\n```\r\n\r\nNote the entry that uses the transient IDs for the source_ref and target_ref.\r\n\r\n## Third run\r\n\r\nThe third run added another duplicate of every relationship, again using only the transient IDs.  Here are the results of the query after a third import of the same entries:\r\n\r\n```\r\n{\r\n  \"took\": 1,\r\n  \"timed_out\": false,\r\n  \"_shards\": {\r\n    \"total\": 5,\r\n    \"successful\": 5,\r\n    \"failed\": 0\r\n  },\r\n  \"hits\": {\r\n    \"total\": 3,\r\n    \"max_score\": 7.711932,\r\n    \"hits\": [\r\n      {\r\n        \"_index\": \"ctia_relationship\",\r\n        \"_type\": \"relationship\",\r\n        \"_id\": \"relationship-cbbb3244-57b7-4ad4-bacb-de90d86f9971\",\r\n        \"_score\": 7.711932,\r\n        \"_source\": {\r\n          \"schema_version\": \"0.4.23\",\r\n          \"target_ref\": \"http://localhost:3000/ctia/indicator/indicator-b7f8235a-4f54-48ee-8d31-4a94ed1b0863\",\r\n          \"type\": \"relationship\",\r\n          \"created\": \"2018-02-01T19:04:45.683Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-aea5c8de8a2b66f47457c8c93b2615cc646eb735eccd635549560481dec07978\"\r\n          ],\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"source_ref\": \"http://localhost:3000/ctia/sighting/sighting-df3292cb-64f8-4303-8bb3-f85727ddcfde\",\r\n          \"id\": \"relationship-cbbb3244-57b7-4ad4-bacb-de90d86f9971\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"owner\": \"Unknown\",\r\n          \"relationship_type\": \"element-of\"\r\n        }\r\n      },\r\n      {\r\n        \"_index\": \"ctia_relationship\",\r\n        \"_type\": \"relationship\",\r\n        \"_id\": \"relationship-f644df64-ea63-4156-887a-145e44ac2f0d\",\r\n        \"_score\": 7.711932,\r\n        \"_source\": {\r\n          \"schema_version\": \"0.4.23\",\r\n          \"target_ref\": \"transient:2bc9fe1d-260b-4635-be33-85d7f53ecc31\",\r\n          \"type\": \"relationship\",\r\n          \"created\": \"2018-02-01T19:13:38.904Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-aea5c8de8a2b66f47457c8c93b2615cc646eb735eccd635549560481dec07978\"\r\n          ],\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"source_ref\": \"transient:39ce0976-6dc6-4ec9-b106-4ccc2d369f7a\",\r\n          \"id\": \"relationship-f644df64-ea63-4156-887a-145e44ac2f0d\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"owner\": \"Unknown\",\r\n          \"relationship_type\": \"element-of\"\r\n        }\r\n      },\r\n      {\r\n        \"_index\": \"ctia_relationship\",\r\n        \"_type\": \"relationship\",\r\n        \"_id\": \"relationship-a29b3ba2-df2f-47db-9671-e2df5949583b\",\r\n        \"_score\": 7.711932,\r\n        \"_source\": {\r\n          \"schema_version\": \"0.4.23\",\r\n          \"target_ref\": \"transient:8e3daf50-07aa-47a7-b57f-fbfde1f3c677\",\r\n          \"type\": \"relationship\",\r\n          \"created\": \"2018-02-01T19:22:51.921Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-aea5c8de8a2b66f47457c8c93b2615cc646eb735eccd635549560481dec07978\"\r\n          ],\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"source_ref\": \"transient:45280dd1-527c-4b38-b7a8-22089f24e2fc\",\r\n          \"id\": \"relationship-a29b3ba2-df2f-47db-9671-e2df5949583b\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"owner\": \"Unknown\",\r\n          \"relationship_type\": \"element-of\"\r\n        }\r\n      }\r\n    ]\r\n  }\r\n}\r\n```\r\n\r\n## Second Example: Judgements\r\n\r\nI've added the same bundle twice, and found that CTIA is creating duplicate Judgement objects, identical in every way except for their `id` and `created` attributes:\r\n\r\n```\r\n{\r\n  \"took\": 5,\r\n  \"timed_out\": false,\r\n  \"_shards\": {\r\n    \"total\": 5,\r\n    \"successful\": 5,\r\n    \"failed\": 0\r\n  },\r\n  \"hits\": {\r\n    \"total\": 2,\r\n    \"max_score\": 0.6931472,\r\n    \"hits\": [\r\n      {\r\n        \"_index\": \"ctia_judgement\",\r\n        \"_type\": \"judgement\",\r\n        \"_id\": \"judgement-cb1f8742-abbc-449d-98e0-33be68631c0e\",\r\n        \"_score\": 0.6931472,\r\n        \"_source\": {\r\n          \"valid_time\": {\r\n            \"end_time\": \"2018-02-28T12:23:07.000Z\",\r\n            \"start_time\": \"2018-01-29T12:23:07.000Z\"\r\n          },\r\n          \"schema_version\": \"0.4.23\",\r\n          \"observable\": {\r\n            \"value\": \"c1325ee5bcff5bba65155cdb21f72d78\",\r\n            \"type\": \"md5\"\r\n          },\r\n          \"type\": \"judgement\",\r\n          \"created\": \"2018-02-01T19:50:18.130Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-ab288ff48dba9925f2fe4fd3757f3311b1a1a6981f516cca94364e08153ac302\"\r\n          ],\r\n          \"disposition\": 2,\r\n          \"reason\": \"Sample used or installed a known malicious SSL certificate.\",\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"disposition_name\": \"Malicious\",\r\n          \"priority\": 90,\r\n          \"id\": \"judgement-cb1f8742-abbc-449d-98e0-33be68631c0e\",\r\n          \"severity\": \"High\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"confidence\": \"Medium\",\r\n          \"owner\": \"Unknown\"\r\n        }\r\n      },\r\n      {\r\n        \"_index\": \"ctia_judgement\",\r\n        \"_type\": \"judgement\",\r\n        \"_id\": \"judgement-c380789a-3619-4122-bbf1-748bf2dc6a89\",\r\n        \"_score\": 0.2876821,\r\n        \"_source\": {\r\n          \"valid_time\": {\r\n            \"end_time\": \"2018-02-28T12:23:07.000Z\",\r\n            \"start_time\": \"2018-01-29T12:23:07.000Z\"\r\n          },\r\n          \"schema_version\": \"0.4.23\",\r\n          \"observable\": {\r\n            \"value\": \"c1325ee5bcff5bba65155cdb21f72d78\",\r\n            \"type\": \"md5\"\r\n          },\r\n          \"type\": \"judgement\",\r\n          \"created\": \"2018-02-01T19:49:08.263Z\",\r\n          \"source\": \"Abuse.ch SSL Blacklist\",\r\n          \"external_ids\": [\r\n            \"hydrant-ab288ff48dba9925f2fe4fd3757f3311b1a1a6981f516cca94364e08153ac302\"\r\n          ],\r\n          \"disposition\": 2,\r\n          \"reason\": \"Sample used or installed a known malicious SSL certificate.\",\r\n          \"source_uri\": \"https://sslbl.abuse.ch\",\r\n          \"disposition_name\": \"Malicious\",\r\n          \"priority\": 90,\r\n          \"id\": \"judgement-c380789a-3619-4122-bbf1-748bf2dc6a89\",\r\n          \"severity\": \"High\",\r\n          \"tlp\": \"green\",\r\n          \"groups\": [\r\n            \"Administrators\"\r\n          ],\r\n          \"confidence\": \"Medium\",\r\n          \"owner\": \"Unknown\"\r\n        }\r\n      }\r\n    ]\r\n  }\r\n}\r\n```\r\n"}}
{"actor":{"display_login":"saintx"}}