{"action":"created","issue":{"url":"https://api.github.com/repos/mitre/cti/issues/17","repository_url":"https://api.github.com/repos/mitre/cti","labels_url":"https://api.github.com/repos/mitre/cti/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/mitre/cti/issues/17/comments","events_url":"https://api.github.com/repos/mitre/cti/issues/17/events","html_url":"https://github.com/mitre/cti/issues/17","id":326710727,"node_id":"MDU6SXNzdWUzMjY3MTA3Mjc=","number":17,"title":"Pandas 0.23.0 - json_normalize - 'STIXdatetime' object has no attribute 'nanosecond'","user":{"login":"Cyb3rWard0g","id":9653181,"node_id":"MDQ6VXNlcjk2NTMxODE=","avatar_url":"https://avatars0.githubusercontent.com/u/9653181?v=4","gravatar_id":"","url":"https://api.github.com/users/Cyb3rWard0g","html_url":"https://github.com/Cyb3rWard0g","followers_url":"https://api.github.com/users/Cyb3rWard0g/followers","following_url":"https://api.github.com/users/Cyb3rWard0g/following{/other_user}","gists_url":"https://api.github.com/users/Cyb3rWard0g/gists{/gist_id}","starred_url":"https://api.github.com/users/Cyb3rWard0g/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cyb3rWard0g/subscriptions","organizations_url":"https://api.github.com/users/Cyb3rWard0g/orgs","repos_url":"https://api.github.com/users/Cyb3rWard0g/repos","events_url":"https://api.github.com/users/Cyb3rWard0g/events{/privacy}","received_events_url":"https://api.github.com/users/Cyb3rWard0g/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2018-05-26T05:03:55Z","updated_at":"2018-06-08T17:09:00Z","closed_at":"2018-06-08T17:09:00Z","author_association":"NONE","body":"Good evening Team,\r\n\r\nI hope you guys are having a good day. I have been playing with the ATT&CK STIX content for the past week and I wanted to report an issue that I am not sure if this is an issue with Pandas or the STIX library. However, I figured it would be good to share it here first just in case I am missing something and also if anyone is having an issue when using pandas 0.23.0 (Latest Version) with ATT&CK STIX content via TAXII.\r\n\r\nI tested ATT&CK STIX content with Pandas 0.21.0 and 0.22.0 and everything was working fine. I was getting everything fine like this:\r\n\r\n```\r\n{'contributors': 'NA',\r\n 'data sources': [u'File monitoring',\r\n                  u'Process monitoring',\r\n                  u'Process command-line parameters'],\r\n 'defense bypassed': 'NA',\r\n 'description': u'Collected data is staged in a central location or directory prior to Exfiltration. Data may be kept in separate files or combined into one file through techniques such as Data Compressed or Data Encrypted.\\n\\nInteractive command shells may be used, and common functionality within cmd and bash may be used to copy data into a staging location.\\n\\nDetection: Processes that appear to be reading files from disparate locations and writing them to the same directory or file may be an indication of data being staged, especially if they are suspected of performing encryption or compression on the files.\\n\\nMonitor processes and command-line arguments for actions that could be taken to collect and combine files. Remote access tools with built-in features may interact directly with the Windows API to gather and copy to a location. Data may also be acquired and staged through Windows system management tools such as Windows Management Instrumentation and PowerShell.\\n\\nPlatforms: Linux, macOS, Windows\\n\\nData Sources: File monitoring, Process monitoring, Process command-line parameters',\r\n 'detectable': 'NA',\r\n 'detectable description': 'NA',\r\n 'difficulty': 'NA',\r\n 'difficulty description': 'NA',\r\n 'effective permissions': 'NA',\r\n 'matrix': u'mitre-attack',\r\n 'network requirements': 'NA',\r\n 'object created': '2017-05-31T21:30:58.938Z',\r\n 'object created by ref': 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5',\r\n 'object id': u'attack-pattern--7dd95ff6-712e-4056-9626-312ea4ab4c5e',\r\n 'object marking refs': ['marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168'],\r\n 'object modified': '2018-04-18T17:59:24.739Z',\r\n 'object type': u'attack-pattern',\r\n 'permission required': 'NA',\r\n 'platforms': [u'Linux', u'macOS', u'Windows'],\r\n 'references': 'NA',\r\n 'remote support': 'NA',\r\n 'system requirements': 'NA',\r\n 'tactic': [u'collection'],\r\n 'tactic type': 'NA',\r\n 'technique': u'Data Staged',\r\n 'technique id': u'T1074',\r\n 'url': u'https://attack.mitre.org/wiki/Technique/T1074'}\r\n```\r\nHowever, when I tested it with Pandas 0.23.0, I got the following error:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in try_datetime(v)\r\n    913                                         require_iso8601=True,\r\n--> 914                                         errors='raise')\r\n    915         except ValueError:\r\n\r\npandas/_libs/tslib.pyx in pandas._libs.tslib.array_to_datetime()\r\n\r\npandas/_libs/tslib.pyx in pandas._libs.tslib.array_to_datetime()\r\n\r\nValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nAttributeError                            Traceback (most recent call last)\r\n<ipython-input-5-e6f5fde53c7b> in <module>()\r\n      3 print(\" \")\r\n      4 df = all_attack['techniques']\r\n----> 5 df = json_normalize(df)\r\n      6 df.reindex(['matrix', 'object created','tactic', 'technique', 'technique id', 'data sources'], axis=1)[0:5]\r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/io/json/normalize.py in json_normalize(data, record_path, meta, meta_prefix, record_prefix, errors, sep)\r\n    202             #       reasonably\r\n    203             data = nested_to_record(data, sep=sep)\r\n--> 204         return DataFrame(data)\r\n    205     elif not isinstance(record_path, list):\r\n    206         record_path = [record_path]\r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)\r\n    385                     if is_named_tuple(data[0]) and columns is None:\r\n    386                         columns = data[0]._fields\r\n--> 387                     arrays, columns = _to_arrays(data, columns, dtype=dtype)\r\n    388                     columns = _ensure_index(columns)\r\n    389 \r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in _to_arrays(data, columns, coerce_float, dtype)\r\n   7435     elif isinstance(data[0], collections.Mapping):\r\n   7436         return _list_of_dict_to_arrays(data, columns,\r\n-> 7437                                        coerce_float=coerce_float, dtype=dtype)\r\n   7438     elif isinstance(data[0], Series):\r\n   7439         return _list_of_series_to_arrays(data, columns,\r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in _list_of_dict_to_arrays(data, columns, coerce_float, dtype)\r\n   7558     content = list(lib.dicts_to_array(data, list(columns)).T)\r\n   7559     return _convert_object_array(content, columns, dtype=dtype,\r\n-> 7560                                  coerce_float=coerce_float)\r\n   7561 \r\n   7562 \r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in _convert_object_array(content, columns, coerce_float, dtype)\r\n   7578         return arr\r\n   7579 \r\n-> 7580     arrays = [convert(arr) for arr in content]\r\n   7581 \r\n   7582     return arrays, columns\r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in <listcomp>(.0)\r\n   7578         return arr\r\n   7579 \r\n-> 7580     arrays = [convert(arr) for arr in content]\r\n   7581 \r\n   7582     return arrays, columns\r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/frame.py in convert(arr)\r\n   7575         if dtype != object and dtype != np.object:\r\n   7576             arr = lib.maybe_convert_objects(arr, try_float=coerce_float)\r\n-> 7577             arr = maybe_cast_to_datetime(arr, dtype)\r\n   7578         return arr\r\n   7579 \r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in maybe_cast_to_datetime(value, dtype, errors)\r\n   1086         elif not (is_array and not (issubclass(value.dtype.type, np.integer) or\r\n   1087                                     value.dtype == np.object_)):\r\n-> 1088             value = maybe_infer_to_datetimelike(value)\r\n   1089 \r\n   1090     return value\r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in maybe_infer_to_datetimelike(value, convert_dates)\r\n    948         value = try_datetime(v)\r\n    949     elif inferred_type == 'datetime':\r\n--> 950         value = try_datetime(v)\r\n    951     elif inferred_type == 'timedelta':\r\n    952         value = try_timedelta(v)\r\n\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in try_datetime(v)\r\n    922                 from pandas import DatetimeIndex\r\n    923 \r\n--> 924                 values, tz = conversion.datetime_to_datetime64(v)\r\n    925                 return DatetimeIndex(values).tz_localize(\r\n    926                     'UTC').tz_convert(tz=tz)\r\n\r\npandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.datetime_to_datetime64()\r\n\r\npandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_datetime_to_tsobject()\r\n\r\nAttributeError: 'STIXdatetime' object has no attribute 'nanosecond'\r\n```\r\n\r\nThe reason why at the beginning I thought it was STIX library was due to the following error message at the end:\r\n```\r\nAttributeError: 'STIXdatetime' object has no attribute 'nanosecond'\r\n```\r\n\r\nThis is a very specific error in version 0.23.0 so I checked the changes to that specific definitions in pandas:\r\n\r\n```\r\n~/Library/Python/3.6/lib/python/site-packages/pandas/core/dtypes/cast.py in try_datetime(v)\r\n    922                 from pandas import DatetimeIndex\r\n    923 \r\n--> 924                 values, tz = conversion.datetime_to_datetime64(v)\r\n    925                 return DatetimeIndex(values).tz_localize(\r\n    926                     'UTC').tz_convert(tz=tz)\r\n```\r\n** Pandas Version 0.22.0:**\r\nhttps://github.com/pandas-dev/pandas/blob/0.22.x/pandas/core/dtypes/cast.py#L879\r\n\r\n** Pandas Version 0.23.0:**\r\nhttps://github.com/pandas-dev/pandas/blob/0.23.x/pandas/core/dtypes/cast.py#L908\r\n\r\nSo they added the following in version 0.23.0:\r\nhttps://github.com/pandas-dev/pandas/blob/0.23.x/pandas/core/dtypes/cast.py#L920\r\n```\r\n# we might have a sequence of the same-datetimes with tz's\r\n            # if so coerce to a DatetimeIndex; if they are not the same,\r\n            # then these stay as object dtype, xref GH19671\r\n            try:\r\n                from pandas._libs.tslibs import conversion\r\n                from pandas import DatetimeIndex\r\n\r\n                values, tz = conversion.datetime_to_datetime64(v)\r\n                return DatetimeIndex(values).tz_localize(\r\n                    'UTC').tz_convert(tz=tz)\r\n            except (ValueError, TypeError):\r\n                pass\r\n```\r\nI checked the STIXdateTime class arguments and I dont see nanoseconds as an option\r\nhttps://github.com/oasis-open/cti-python-stix2/blob/master/stix2/utils.py#L24\r\n\r\nI am not sure if there is anything that needs to be done on the STIX library side. \r\n\r\nI downgraded the Python3 Pandas package to 0.22.0 and it worked fine. I didnt want to start an issue in Pandas before asking you guys if this makes sense and if it is possible that nanoseconds needs to be defined as an argument for the STIXdatetime class. \r\n\r\nI hope you all have a great weekend! No rush at all on this one. I will keep working with Pandas 0.22.0 for now. I dont need to use pandas to collect or filter the data initially. I use it for a better representation of the results after collecting everything via STIX and TAXII libraries. Therefore, if you want to close this issue since I am using an external library, I would understand. It is just that the STIXdatetime error message caught my attention and I wasnt sure if nanosecond is an standard or anything that needs to be defined on the STIX side. If not, then this issue can be close 😄 \r\n\r\nOnce again guys, great job and thank you for all your help!! I hope you all have a great weekend!!! "},"comment":{"url":"https://api.github.com/repos/mitre/cti/issues/comments/395825723","html_url":"https://github.com/mitre/cti/issues/17#issuecomment-395825723","issue_url":"https://api.github.com/repos/mitre/cti/issues/17","id":395825723,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTgyNTcyMw==","user":{"login":"emmanvg","id":8574582,"node_id":"MDQ6VXNlcjg1NzQ1ODI=","avatar_url":"https://avatars3.githubusercontent.com/u/8574582?v=4","gravatar_id":"","url":"https://api.github.com/users/emmanvg","html_url":"https://github.com/emmanvg","followers_url":"https://api.github.com/users/emmanvg/followers","following_url":"https://api.github.com/users/emmanvg/following{/other_user}","gists_url":"https://api.github.com/users/emmanvg/gists{/gist_id}","starred_url":"https://api.github.com/users/emmanvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmanvg/subscriptions","organizations_url":"https://api.github.com/users/emmanvg/orgs","repos_url":"https://api.github.com/users/emmanvg/repos","events_url":"https://api.github.com/users/emmanvg/events{/privacy}","received_events_url":"https://api.github.com/users/emmanvg/received_events","type":"User","site_admin":false},"created_at":"2018-06-08T17:09:00Z","updated_at":"2018-06-08T17:09:00Z","author_association":"NONE","body":"No problem @Cyb3rWard0g! I think it is OK to close. If you have [stix2](https://github.com/oasis-open/cti-python-stix2) library problems in the future I would recommend opening them there to better address it."}}
{"actor":{"display_login":"emmanvg"}}