{"action":"created","issue":{"url":"https://api.github.com/repos/bgottula/track/issues/94","repository_url":"https://api.github.com/repos/bgottula/track","labels_url":"https://api.github.com/repos/bgottula/track/issues/94/labels{/name}","comments_url":"https://api.github.com/repos/bgottula/track/issues/94/comments","events_url":"https://api.github.com/repos/bgottula/track/issues/94/events","html_url":"https://github.com/bgottula/track/issues/94","id":330122425,"node_id":"MDU6SXNzdWUzMzAxMjI0MjU=","number":94,"title":"Stuttering and stalling with Gemini and G11","user":{"login":"bgottula","id":7323674,"node_id":"MDQ6VXNlcjczMjM2NzQ=","avatar_url":"https://avatars0.githubusercontent.com/u/7323674?v=4","gravatar_id":"","url":"https://api.github.com/users/bgottula","html_url":"https://github.com/bgottula","followers_url":"https://api.github.com/users/bgottula/followers","following_url":"https://api.github.com/users/bgottula/following{/other_user}","gists_url":"https://api.github.com/users/bgottula/gists{/gist_id}","starred_url":"https://api.github.com/users/bgottula/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgottula/subscriptions","organizations_url":"https://api.github.com/users/bgottula/orgs","repos_url":"https://api.github.com/users/bgottula/repos","events_url":"https://api.github.com/users/bgottula/events{/privacy}","received_events_url":"https://api.github.com/users/bgottula/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"bgottula","id":7323674,"node_id":"MDQ6VXNlcjczMjM2NzQ=","avatar_url":"https://avatars0.githubusercontent.com/u/7323674?v=4","gravatar_id":"","url":"https://api.github.com/users/bgottula","html_url":"https://github.com/bgottula","followers_url":"https://api.github.com/users/bgottula/followers","following_url":"https://api.github.com/users/bgottula/following{/other_user}","gists_url":"https://api.github.com/users/bgottula/gists{/gist_id}","starred_url":"https://api.github.com/users/bgottula/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgottula/subscriptions","organizations_url":"https://api.github.com/users/bgottula/orgs","repos_url":"https://api.github.com/users/bgottula/repos","events_url":"https://api.github.com/users/bgottula/events{/privacy}","received_events_url":"https://api.github.com/users/bgottula/received_events","type":"User","site_admin":false},"assignees":[{"login":"bgottula","id":7323674,"node_id":"MDQ6VXNlcjczMjM2NzQ=","avatar_url":"https://avatars0.githubusercontent.com/u/7323674?v=4","gravatar_id":"","url":"https://api.github.com/users/bgottula","html_url":"https://github.com/bgottula","followers_url":"https://api.github.com/users/bgottula/followers","following_url":"https://api.github.com/users/bgottula/following{/other_user}","gists_url":"https://api.github.com/users/bgottula/gists{/gist_id}","starred_url":"https://api.github.com/users/bgottula/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgottula/subscriptions","organizations_url":"https://api.github.com/users/bgottula/orgs","repos_url":"https://api.github.com/users/bgottula/repos","events_url":"https://api.github.com/users/bgottula/events{/privacy}","received_events_url":"https://api.github.com/users/bgottula/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-06-07T05:30:09Z","updated_at":"2018-07-14T23:22:04Z","closed_at":null,"author_association":"OWNER","body":"Sometimes for no apparent reason when attempting to track with Gemini the G11 mount will stutter violently, sometimes badly enough to induce motor stalls. This makes practical tracking impossible and is probably hard on the motors. There is no known reliable way to recover from this issue, including power cycling Gemini. Yet, somehow, the issue will then spontaneously vanish and smooth performance returns. This has happened on multiple occasions while testing the software indoors and as of 8 June 2018 has also cost us a 80-degree elevation ISS pass when it started happening just when the hybrid_track program was started up to track ISS.\r\n\r\nSome ideas for what might be going on:\r\n- Our software (either point or track) is doing something wrong. Malformed commands, commanding erratic slew rates with high accelerations, long pauses between slew commands, etc.\r\n- Gemini firmware has a bug or has some behavior we don't understand. On 8 June while attempting to track ISS I tried to recover from this problem by power-cycling Gemini and that did not seem to help which should rule out some classes of bugs.\r\n- Gemini software/firmware is fighting our software for control of the slew rate. It is not entirely clear to me how we know what is in control of the slew rate at any given moment. For example, if we leave the tracking mode as \"sidereal,\" does that mean Gemini is going to periodically try to override our divisor commands?\r\n\r\nThings that have been ruled out:\r\n\r\n- Errant detections in the guidescope camera. This behavior has been observed when using blind_track, when the camera is not involved at all.\r\n- Hardware problem, such as damage to the servo motors, gears, or wiring. This almost certainly is not the case because when the mount is symptomatic, the problem vanishes when performing a goto using the hand controller. The problem also goes away when slewing in the problem direction after going back the other way and then returning.\r\n- Hardware dynamics (gears too loose or too tight, not lubricated properly, unbalanced load). This seems unlikely based on the symptoms unless something really counter-intuitive is going on, like lubrication being redistributed after moving one direction in RA and thereby reducing friction on subsequent passes of that part of the gear. I took pains to test on 9 June with very minimal load on the RA axis. Inspection of the worm did not reveal anything concerning. The drivetrain upstream of the worm is fully exonerated because it turns multiple full revolutions with the issue and without the issue.\r\n- Errant detection of RA axis limit. I enabled debug messages in the gamepad program and did not see any evidence that the axis limit was tripped when it should not have been.\r\n\r\nAs part of the resolution of this issue, it might be prudent to detect and handle motor stalls (if possible). When Gemini detects a motor stall the following things seem to happen:\r\n\r\n- Motion on that axis stops (not sure about other axis)\r\n- Beep sound\r\n- Get NACKs on UDP interface for some stupid reason\r\n- Track program is unaware and keeps trying to command potentially high slew rates which causes further motor stalls\r\n\r\nThe appropriate response is probably to reset the loop filter integrator on the affected axis to 0 to sync up with the fact that motion on the axis has stopped. This will hopefully allow that axis to accelerate smoothly back up to full rate without causing further stalls."},"comment":{"url":"https://api.github.com/repos/bgottula/track/issues/comments/405055984","html_url":"https://github.com/bgottula/track/issues/94#issuecomment-405055984","issue_url":"https://api.github.com/repos/bgottula/track/issues/94","id":405055984,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTA1NTk4NA==","user":{"login":"bgottula","id":7323674,"node_id":"MDQ6VXNlcjczMjM2NzQ=","avatar_url":"https://avatars0.githubusercontent.com/u/7323674?v=4","gravatar_id":"","url":"https://api.github.com/users/bgottula","html_url":"https://github.com/bgottula","followers_url":"https://api.github.com/users/bgottula/followers","following_url":"https://api.github.com/users/bgottula/following{/other_user}","gists_url":"https://api.github.com/users/bgottula/gists{/gist_id}","starred_url":"https://api.github.com/users/bgottula/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgottula/subscriptions","organizations_url":"https://api.github.com/users/bgottula/orgs","repos_url":"https://api.github.com/users/bgottula/repos","events_url":"https://api.github.com/users/bgottula/events{/privacy}","received_events_url":"https://api.github.com/users/bgottula/received_events","type":"User","site_admin":false},"created_at":"2018-07-14T23:22:04Z","updated_at":"2018-07-14T23:22:04Z","author_association":"OWNER","body":"I have started performing additional experimentation. This time rather than using higher level control software or the gamepad_control program I am invoking the slew() method on the LosmandyGeminiMount class in the track package. This should (I think) only be sending a single native Gemini command (451 or 452) to the mount per invocation. I have observed the following:\r\n\r\n* Additional confirmation that only the RA axis seems to be affected.\r\n* After setting the slew rate on the RA axis to a value such as +1 or -1, the mount will slew for some time and then stop. Inspection of the Gemini State webpage shows that the slew rate has not actually stopped but has instead changed to a very slow rate. The slow slew rate always continues in the same direction as the original slew. If the slew rate is explicitly set to 0, the mount motion will actually come to a stop.\r\n* The issue is present even when the tracking mode is set to \"Comet/User Defined\". René had recommended using Comet tracking mode to make sure our divisors are preserved.\r\n\r\nThese observations suggest that the stuttering behavior is in fact resulting from something in the Gemini firmware setting the RA divisor to values that are in conflict with the values we are trying to set using our software. I attempted to confirm this by reading the divisors but the native 451 and 452 commands are evidently write-only.\r\n\r\nI also ruled out the hand controller by performing the same experiments with the hand controller disconnected from Gemini 2. I did not observe any change in behavior with the hand controller disconnected."}}
{"actor":{"display_login":"bgottula"}}