{"action":"created","issue":{"url":"https://api.github.com/repos/HERA-Team/hera_qm/issues/170","repository_url":"https://api.github.com/repos/HERA-Team/hera_qm","labels_url":"https://api.github.com/repos/HERA-Team/hera_qm/issues/170/labels{/name}","comments_url":"https://api.github.com/repos/HERA-Team/hera_qm/issues/170/comments","events_url":"https://api.github.com/repos/HERA-Team/hera_qm/issues/170/events","html_url":"https://github.com/HERA-Team/hera_qm/issues/170","id":338709069,"node_id":"MDU6SXNzdWUzMzg3MDkwNjk=","number":170,"title":"does red_corr_metrics actually measure what we think?","user":{"login":"jaguirre","id":2090374,"node_id":"MDQ6VXNlcjIwOTAzNzQ=","avatar_url":"https://avatars0.githubusercontent.com/u/2090374?v=4","gravatar_id":"","url":"https://api.github.com/users/jaguirre","html_url":"https://github.com/jaguirre","followers_url":"https://api.github.com/users/jaguirre/followers","following_url":"https://api.github.com/users/jaguirre/following{/other_user}","gists_url":"https://api.github.com/users/jaguirre/gists{/gist_id}","starred_url":"https://api.github.com/users/jaguirre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaguirre/subscriptions","organizations_url":"https://api.github.com/users/jaguirre/orgs","repos_url":"https://api.github.com/users/jaguirre/repos","events_url":"https://api.github.com/users/jaguirre/events{/privacy}","received_events_url":"https://api.github.com/users/jaguirre/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-07-05T20:06:55Z","updated_at":"2018-08-08T20:13:38Z","closed_at":null,"author_association":"NONE","body":"The question has been raised whether average data0 * conj(data1) over time, median over frequency\r\n\r\n```\r\ncorr = np.median(np.abs(np.mean(data0 * data1.conj(),axis=0)))\r\ncorr /= np.sqrt(autoPower[ant0_i, ant0_j, pol0] * autoPower[ant1_i, ant1_j, pol1])\r\n```\r\n\r\nin `ant_metrics.py` actually detects a failure of redundancy between baselines in a group, since it is not terribly sensitive to frequency structure, and because over a single file, the visibilities are not changing fast, much less differences between them.  \r\n\r\nPerhaps the idea was to correlate over frequency and median over time (this is similar to JEA's suggestion that |data0|/|data1| = constant, without having to worry about nulls). Assuming axis=0 is time and axis=1 is frequency, this would look like\r\n\r\n```\r\ncorr = np.median(np.mean(np.abs(data0) * np.abs(data1),axis=1))\r\ncorr /= np.sqrt(autoPower[ant0_i, ant0_j, pol0] * autoPower[ant1_i, ant1_j, pol1])\r\n\r\n```\r\n\r\nTo test the difference, Is there an example data set where a baseline is known not to be redundant that can be used for testing?"},"comment":{"url":"https://api.github.com/repos/HERA-Team/hera_qm/issues/comments/411536974","html_url":"https://github.com/HERA-Team/hera_qm/issues/170#issuecomment-411536974","issue_url":"https://api.github.com/repos/HERA-Team/hera_qm/issues/170","id":411536974,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTUzNjk3NA==","user":{"login":"ShaneBechtel","id":22246419,"node_id":"MDQ6VXNlcjIyMjQ2NDE5","avatar_url":"https://avatars3.githubusercontent.com/u/22246419?v=4","gravatar_id":"","url":"https://api.github.com/users/ShaneBechtel","html_url":"https://github.com/ShaneBechtel","followers_url":"https://api.github.com/users/ShaneBechtel/followers","following_url":"https://api.github.com/users/ShaneBechtel/following{/other_user}","gists_url":"https://api.github.com/users/ShaneBechtel/gists{/gist_id}","starred_url":"https://api.github.com/users/ShaneBechtel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ShaneBechtel/subscriptions","organizations_url":"https://api.github.com/users/ShaneBechtel/orgs","repos_url":"https://api.github.com/users/ShaneBechtel/repos","events_url":"https://api.github.com/users/ShaneBechtel/events{/privacy}","received_events_url":"https://api.github.com/users/ShaneBechtel/received_events","type":"User","site_admin":false},"created_at":"2018-08-08T20:13:38Z","updated_at":"2018-08-08T20:13:38Z","author_association":"NONE","body":"That is what appears to be the case. In a case where everything is equal save the metric used, it appears that an antenna with excessive nonredundant data is more likely to be flagged using James' metric than the current metric. As for the histogram, I tried offering a tighter view after the initial histogram, but I'll include a view of the distribution on a range from -5 to 5 with more narrow bin widths here. \r\n\r\n![james_metric_12552_data_zoom_1](https://user-images.githubusercontent.com/22246419/43861451-be75474e-9b0b-11e8-9b50-c1b0d602f398.PNG)\r\n\r\nI poorly described the behavior of the distribution in my post. A more accurate description is that the same probability is distributed more widely on the negative side than the positive side. Meaning, it is more likely for a high negative value to occur than a high positive value. I found this to possibly be significant as, it has been questioned what a very large z-score means (an antenna that is too redundant?). The fact that this metric is more biased towards larger negative numbers may be indicative of it measuring what we would like it to. "}}
{"actor":{"display_login":"ShaneBechtel"}}