{"action":"created","issue":{"url":"https://api.github.com/repos/cf-convention/cf-conventions/issues/148","repository_url":"https://api.github.com/repos/cf-convention/cf-conventions","labels_url":"https://api.github.com/repos/cf-convention/cf-conventions/issues/148/labels{/name}","comments_url":"https://api.github.com/repos/cf-convention/cf-conventions/issues/148/comments","events_url":"https://api.github.com/repos/cf-convention/cf-conventions/issues/148/events","html_url":"https://github.com/cf-convention/cf-conventions/issues/148","id":374463611,"node_id":"MDU6SXNzdWUzNzQ0NjM2MTE=","number":148,"title":"Add calendars gregorian_tai and gregorian_utc","user":{"login":"JimBiardCics","id":17711910,"node_id":"MDQ6VXNlcjE3NzExOTEw","avatar_url":"https://avatars2.githubusercontent.com/u/17711910?v=4","gravatar_id":"","url":"https://api.github.com/users/JimBiardCics","html_url":"https://github.com/JimBiardCics","followers_url":"https://api.github.com/users/JimBiardCics/followers","following_url":"https://api.github.com/users/JimBiardCics/following{/other_user}","gists_url":"https://api.github.com/users/JimBiardCics/gists{/gist_id}","starred_url":"https://api.github.com/users/JimBiardCics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JimBiardCics/subscriptions","organizations_url":"https://api.github.com/users/JimBiardCics/orgs","repos_url":"https://api.github.com/users/JimBiardCics/repos","events_url":"https://api.github.com/users/JimBiardCics/events{/privacy}","received_events_url":"https://api.github.com/users/JimBiardCics/received_events","type":"User","site_admin":false},"labels":[{"id":220395606,"node_id":"MDU6TGFiZWwyMjAzOTU2MDY=","url":"https://api.github.com/repos/cf-convention/cf-conventions/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":{"login":"JimBiardCics","id":17711910,"node_id":"MDQ6VXNlcjE3NzExOTEw","avatar_url":"https://avatars2.githubusercontent.com/u/17711910?v=4","gravatar_id":"","url":"https://api.github.com/users/JimBiardCics","html_url":"https://github.com/JimBiardCics","followers_url":"https://api.github.com/users/JimBiardCics/followers","following_url":"https://api.github.com/users/JimBiardCics/following{/other_user}","gists_url":"https://api.github.com/users/JimBiardCics/gists{/gist_id}","starred_url":"https://api.github.com/users/JimBiardCics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JimBiardCics/subscriptions","organizations_url":"https://api.github.com/users/JimBiardCics/orgs","repos_url":"https://api.github.com/users/JimBiardCics/repos","events_url":"https://api.github.com/users/JimBiardCics/events{/privacy}","received_events_url":"https://api.github.com/users/JimBiardCics/received_events","type":"User","site_admin":false},"assignees":[{"login":"JimBiardCics","id":17711910,"node_id":"MDQ6VXNlcjE3NzExOTEw","avatar_url":"https://avatars2.githubusercontent.com/u/17711910?v=4","gravatar_id":"","url":"https://api.github.com/users/JimBiardCics","html_url":"https://github.com/JimBiardCics","followers_url":"https://api.github.com/users/JimBiardCics/followers","following_url":"https://api.github.com/users/JimBiardCics/following{/other_user}","gists_url":"https://api.github.com/users/JimBiardCics/gists{/gist_id}","starred_url":"https://api.github.com/users/JimBiardCics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JimBiardCics/subscriptions","organizations_url":"https://api.github.com/users/JimBiardCics/orgs","repos_url":"https://api.github.com/users/JimBiardCics/repos","events_url":"https://api.github.com/users/JimBiardCics/events{/privacy}","received_events_url":"https://api.github.com/users/JimBiardCics/received_events","type":"User","site_admin":false}],"milestone":null,"comments":157,"created_at":"2018-10-26T16:45:42Z","updated_at":"2018-11-18T23:03:22Z","closed_at":null,"author_association":"NONE","body":"### Introduction\r\nThe current CF time system does not address the presence or absence of leap seconds in data with a standard name of **`time`**. This is not an issue for model runs or data with time resolutions on the order of hours, days, etc, but it can be an issue for modern satellite swath data and other systems with time resolutions of tens of seconds or finer.\r\n\r\nI have written a background section for this proposal, but I have put it at the end so that people don't have to scroll through it in order to get to proposal itself. If something about the proposal seems unclear, I hope the background will help resolve your question.\r\n### Proposal\r\nAfter past discussions with @JonathanGregory and again with he and @marqh at the 2018 CF Workshop, I propose the new calendars listed below and a change to existing calendar definitions.\r\n\r\n- **`gregorian_tai`** - When this calendar is called out, the epoch date and time stated in the **`units`** attribute are required to be Coordinated Universal Time (UTC) and the time values in the variable are required to be fully metric, representing the the advance in International Atomic Time (TAI) since that epoch. Conversion of a time value in the variable to a UTC date and time must account for any leap seconds between the epoch date and the time being converted.\r\n - **`gregorian_utc`** - When this calendar is called out, the epoch date and time stated in the **`units`** attribute are required to be in UTC and the time values in the variable are assumed to be conversions from UTC dates and times that did not account for leap seconds. As a consequence, the time values may not be fully metric. Conversion of a time value in the variable to a UTC date and time must not use leap seconds.\r\n- **`gregorian`** - When this calendar is called out, the epoch date stated in the **`units`** attribute is required to be in mixed Gregorian/Julian form. The epoch date and time have an unknown relationship to UTC. The time values in the variable may not be fully metric, and conversion of a time value in the variable to a date and time produces results of unknown precision.\r\n- **`the others`** - The other calendars all have an unknown relationship to UTC, similar to the **`gregorian`** calendar above.\r\n\r\nThe large majority of existing files (past and future) are based on artificial model time or don't need to record time precisely enough to require either of the new calendars (**`gregorian_tai`** or **`gregorian_utc`**). The modified definition of the **`gregorian`** calendar won't pose any problem for them. For users that know exactly how they obtained their times and how they processed them to get time values in a variable, the two new calendars allow them to tell users how to handle (and not handle) those time values.\r\n\r\nOnce we come to an agreement on the proposal, we can work out wording Section 4.4 to reflect these new/changed calendar definitions.\r\n### Background\r\nThere are three parts to the way people deal with time. The first part is the counting of the passing of time, the second part is the representation of time for human consumption, and the third is the relationship between the representation of time and the orbital and rotational cycles of the earth. This won't be a deep discussion, but I want to define a few terms here in the hopes that it will help make things clearer. For gory details, please feel free to consult Google and visit places such as the NIST and US Naval Observatory websites. I'm glossing over some things here, and many of my definitions are not precise. My goal is to provide a common framework for thinking about the proposal, as opposed to writing a textbook on the topic.\r\n\r\nThe first part is the simplest. This is time as a scalar quantity that grows at a fixed rate. This, precisely measured, is what people refer to as 'atomic time' - a count of cycles of an oscillator tuned to resonate with an electron level transition in a sample of super-cooled atoms. The international standard atomic time is known as International Atomic Time (TAI). So time in this sense is a counter that advances by one every SI second. (For simplicity, I am going to speak in terms of counts of seconds throughout this proposal.) No matter how you may represent time, whether with or without leap days or seconds, this time marches on at a fixed pace. This time is metric. You can do math operations on pairs or other groups of these times and get consistently correct results. In the rest of this proposal I'm going to refer to this kind of time as 'metric time'.\r\n\r\nThe second part, the representation of time, is all about how we break time up into minutes, hours, days, months, and years. Astronomy, culture, and history have all affected the way we represent time. When we display a time as YYYY-MM-DD HH:MM:SS, we are representing a point in time with a label. In the rest of this proposal I'm going to refer to this labeling of a point in time as a time stamp.\r\n\r\nThe third part, the synchronization of time stamps with the cycles of the planet, is where calendars come into play, and this is where things get ugly. Reaching way back in time, there were three basic units for time - the solar year, the lunar month, and the solar day. Unfortunately, these three units of time are not compatible with each other or with counts of seconds. A solar day is not (despite our definitions) an integer number of seconds in length, a lunar month is not an integer number of solar days (and we pretty much abandoned them in Western culture), and a solar year is not an integer number of solar days or lunar months in length. If you attempt to count time by incrementing a time stamp like an odometer - having a given element increment once each time the element below it has 'rolled over', you find that the time stamps pretty quickly get out of synchronization with the sun and the seasons.\r\n\r\nThe first attempts to address this asynchrony were leap days. The Julian calendar specified that every four years February would wait an extra day to roll over to March. The Gregorian calendar addressed a remaining asynchrony by specifying that this only happens on the last year of a century (when it normally would) every fourth century. That was close enough for the technology of those days. Clocks weren't accurate enough at counting seconds to worry about anything else. But the addition of leap days (as well as months with random lengths) means that time stamps aren't metric. You can't do straightforward math with them.\r\n\r\nIn more recent times technology and science have advanced to the point that we can count seconds quite accurately, and we found that keeping the time stamp hours, minutes, and seconds sufficiently aligned with the rising of the sun each day requires the addition (or subtraction) of leap seconds. On an irregular, potentially bi-yearly, basis, the last minute of a day is allowed to run to 60 before rolling over instead of 59 (or rolls over after 58, though it's lately been only additions). Coordinated Universal Time (UTC) is the standard for time stamps that include both leap days and leap seconds.\r\n\r\nUTC time stamps represent the time in a human-readable form that is precise and synchronized with the cycles of the earth. But they aren't metric. It's not hard to deal with the leap days part because they follow a fixed pattern. But the leap seconds don't. If you try to calculate the interval between 2018-01-01 00:00:00 and 1972-01-01 00:00:00 without consulting a table of leap seconds and when they were applied, you will have a difference of 27 seconds between the time you get from your calculation and the time has actually elapsed between those two time stamps. This isn't enough of a discrepancy to worry about for readings from rain gauges or measurements of daily average temperature, but an error of even one second can make a big difference for data from a polar-orbiting satellite moving at a rate of 7 km/second.\r\n\r\nThe clocks in our computers can add further complexity to measuring time. The vast majority of computers don't handle leap seconds. We typically attempt to address this by using time servers to keep our computer clocks synchronized, but this is done by altering the metric time count in the computer rather than modifying the time stamps by updating a table of leap seconds.\r\n\r\nFurthermore, most computer software doesn't have 'leap second aware' libraries. When you take a perfectly exact UTC time stamp (perhaps taken from a GPS unit) and convert it to a count of seconds since an epoch using a time calculation function in your software, you are highly likely to have introduced an error of however many leap seconds that have been added between your epoch and the time represented by the time stamp.\r\n\r\nAs a result of all this, many of the times written in netCDF files are not metric times, and there is no good way to know how to produce accurate time stamps from them. They may be perfectly metric within a given file or dataset, they may include skips or repeats, or they may harbor non-linearities where there are one or more leap seconds between two time values.\r\n\r\nWe have another minor issue for times prior to 1972-01-01. There's not much way to relate times prior to that epoch to times since - not to the tens of seconds or better level. I'd be surprised if this would ever be a significant problem in our domain.\r\n\r\nTo summarize, we have TAI, which is precise metric time. We have UTC, which is a precise, non-metric sequence of time stamps that are tied to TAI, and we have a whole host of ways that counts time since epoch stored in netCDF files can be inaccurate to a level as high as 37 seconds (the current leap seconds offset between TAI and UTC).\r\n\r\nMost uses of time in netCDF aren't concerned with this level of accuracy, but for those that are, it can be critical."},"comment":{"url":"https://api.github.com/repos/cf-convention/cf-conventions/issues/comments/439734161","html_url":"https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-439734161","issue_url":"https://api.github.com/repos/cf-convention/cf-conventions/issues/148","id":439734161,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTczNDE2MQ==","user":{"login":"graybeal","id":1266896,"node_id":"MDQ6VXNlcjEyNjY4OTY=","avatar_url":"https://avatars0.githubusercontent.com/u/1266896?v=4","gravatar_id":"","url":"https://api.github.com/users/graybeal","html_url":"https://github.com/graybeal","followers_url":"https://api.github.com/users/graybeal/followers","following_url":"https://api.github.com/users/graybeal/following{/other_user}","gists_url":"https://api.github.com/users/graybeal/gists{/gist_id}","starred_url":"https://api.github.com/users/graybeal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/graybeal/subscriptions","organizations_url":"https://api.github.com/users/graybeal/orgs","repos_url":"https://api.github.com/users/graybeal/repos","events_url":"https://api.github.com/users/graybeal/events{/privacy}","received_events_url":"https://api.github.com/users/graybeal/received_events","type":"User","site_admin":false},"created_at":"2018-11-18T23:03:22Z","updated_at":"2018-11-18T23:03:22Z","author_association":"NONE","body":"An outsider's view, staying at a high level and focusing mostly on social/usability/adoption aspects. (I was following the technical details for the first hundred posts or so, honest. We're way past that now.)\r\n\r\nShould you have a telecon? I can't imagine that would be helpful, unless you started with high-level objectives. Every word and sentence is thoughtful but prone to misunderstanding _via voce_.\r\n\r\n`Given that leap seconds are almost always completely immaterial to \"climate and forecasts\" (the focus of the CF conventions), I don't think we should complicate the conventions in any significant way to accommodate leap seconds.` As an observationalist supporting CF use a long time, I naturally object to the cited motivation, and the implication that we can be sure that no leap-second-involved use case will have anything to do with climate and forecast. But it is fair to say that these solutions all complicate the CF convention, and to question whether they are absolutely necessary—at least, *if there are workarounds that acceptably support the needs of prospective CF users*.\r\n\r\nSo I favor a solution that is the most transparent to those who are trying to assess their need for it. To me that suggests solutions that most closely (and obviously) match existing usage in the real world. So I like *tai*, *gps*, and *utc* as well-defined, not-too-hard-to-use calendars; *gregorian* as the pre-existing \"We don't know or we don't care\" calendar (and worth keeping because of its usefulness to large number of current real-world adopters); and *ntp* as the \"we do know and we do care\" calendar.  \r\n\r\nI _think_ I understand the arguments about whether or not to add ntp, and find the ability to succinctly capture the differences between gregorian and ntp to be valuable for future users, without causing excessive anguish for current users.  But some data will be labeled ntp needlessly, and some software may not be able to handle it; I don't think  \r\n\r\n(Addressing calendar issues with changes involving attributes and similar tweaks may be equally valid technically, but I feel they require a more nuanced handling in the documentation and in the software systems, a handling that will be at risk of being missed. @taylor13's units proposal may be a viable exception to this concern, and that single-UTC approach may be resonant enough to be as good as my first choice above.)\r\n\r\nI know this adds complexity to the spec, and especially to any software *that has to precisely understand these differences* in order to work (but that software had no clear way to proceed previously, so it seems a reasonable cost). I do not think the change adds significant complexity to, for example, cf checkers (which will have to be changed in any case, yes?).\r\n\r\nAssuming that the documentation for these calendar options can give clear use cases and concrete examples, it seems like they would help CF get over the longstanding 'time confusion hump' in a way that contributes to understanding those issues. \r\n\r\n"}}
{"actor":{"display_login":"graybeal"}}