title: How perform a drag (based in X,Y mouse coordinates) on Android using AccessibilityService?
body: <p>I want know how to perform a drag on android based in X, Y mouse coordinates? consider as two simple examples, the Team Viewer/QuickSupport drawing the "password pattern" on remote smartphone and the Pen of Windows Paint respectively.</p>
  
  <p><a href="https://i.stack.imgur.com/i5oDk.gif" rel="noreferrer"><img src="https://i.stack.imgur.com/i5oDk.gif" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/Pn3b0.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/Pn3b0.jpg" alt="enter image description here"></a></p>
  
  <p>All that i'm able of make is <a href="https://stackoverflow.com/a/51706446/9708179">simulate touch</a> (with <code>dispatchGesture()</code> and also <code>AccessibilityNodeInfo.ACTION_CLICK</code>).</p>
  
  <p>I found these relevants links, but not know if they can be useful:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/a/44448362/9708179">Perform swipe on screen using AccessibilityService</a></li>
  <li><a href="https://www.programcreek.com/java-api-examples/index.php?api=android.accessibilityservice.GestureDescription" rel="noreferrer"><strong>Example 1</strong></a></li>
  <li><a href="https://eventandtech.blogspot.com/p/fingerprint-gestures-accessibility.html" rel="noreferrer"><strong>Continued gestures</strong></a></li>
  </ul>
  
  <p>Below is my working code that is used to send mouse coordinates (inside of <code>PictureBox</code> control) to remote phone and simulate touch.</p>
  
  <p><strong>Windows Forms Application:</strong></p>
  
  <pre><code>private void pictureBox1_MouseDown(object sender, MouseEventArgs e)
  {
      foreach (ListViewItem item in lvConnections.SelectedItems)
      {
          // Remote screen resolution
          string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920
  
          int xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width);
          int yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);
  
          Client client = (Client)item.Tag;
  
          if (e.Button == MouseButtons.Left)
              client.sock.Send(Encoding.UTF8.GetBytes("TOUCH" + xClick + "&lt;|&gt;" + yClick + Environment.NewLine));
      }
  }
  </code></pre>
  
  <hr>
  
  <p><strong>Edit:</strong></p>
  
  <p>My last attempt was a "swipe screen" using mouse coordinates (C# Windows Forms Application) and a custom android routine (with reference to code of "swipe screen" linked above), respectively:</p>
  
  <pre><code>private Point mdownPoint = new Point();
  
  private void pictureBox1_MouseDown(object sender, MouseEventArgs e)
  {
      foreach (ListViewItem item in lvConnections.SelectedItems)
      {
          // Remote screen resolution
          string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920
  
          Client client = (Client)item.Tag;
  
          if (e.Button == MouseButtons.Left)
          {
              xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width); 
              yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);
  
              // Saving start position:
  
              mdownPoint.X = xClick; 
              mdownPoint.Y = yClick; 
  
              client.sock.Send(Encoding.UTF8.GetBytes("TOUCH" + xClick + "&lt;|&gt;" + yClick + Environment.NewLine));
          }
      }
  }
  
  private void PictureBox1_MouseMove(object sender, MouseEventArgs e)
  {
      foreach (ListViewItem item in lvConnections.SelectedItems)
      {
          // Remote screen resolution
          string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920
  
          Client client = (Client)item.Tag;
  
          if (e.Button == MouseButtons.Left)
          {
              xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width);
              yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);
  
              client.sock.Send(Encoding.UTF8.GetBytes("MOUSESWIPESCREEN" + mdownPoint.X + "&lt;|&gt;" + mdownPoint.Y + "&lt;|&gt;" + xClick + "&lt;|&gt;" + yClick + Environment.NewLine));
          }
      }
  }
  </code></pre>
  
  <p>android <em>AccessibilityService</em>:</p>
  
  <pre><code>public void Swipe(int x1, int y1, int x2, int y2, int time) {
  
  if (android.os.Build.VERSION.SDK_INT &gt;= android.os.Build.VERSION_CODES.N) {
      System.out.println(" ======= Swipe =======");
  
      GestureDescription.Builder gestureBuilder = new GestureDescription.Builder();
      Path path = new Path();
      path.moveTo(x1, y1);
      path.lineTo(x2, y2);
  
      gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 100, time));
      dispatchGesture(gestureBuilder.build(), new GestureResultCallback() {
          @Override
          public void onCompleted(GestureDescription gestureDescription) {
              System.out.println("SWIPE Gesture Completed :D");
              super.onCompleted(gestureDescription);
          }
      }, null);
  }
  
  }
  </code></pre>
  
  <p>that produces the following result (but still not is able to draw "pattern password" like TeamViewer for example). But like said on comment below, I think that with a similar approach this can be achieved using <a href="https://eventandtech.blogspot.com/p/fingerprint-gestures-accessibility.html" rel="noreferrer">Continued gestures</a> probably. Any suggestions in this direction will be welcome.</p>
  
  <p><a href="https://i.stack.imgur.com/7gDiC.gif" rel="noreferrer"><img src="https://i.stack.imgur.com/7gDiC.gif" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/QvgOs.gif" rel="noreferrer"><img src="https://i.stack.imgur.com/QvgOs.gif" alt="enter image description here"></a></p>
  
  <hr>
  
  <p><strong>Edit 2:</strong></p>
  
  <p>Definitely, the solution is <a href="https://developer.android.com/reference/android/accessibilityservice/GestureDescription.StrokeDescription#continueStroke(android.graphics.Path,%20long,%20long,%20boolean)" rel="noreferrer">continued gestures</a> like said on previous <strong>Edit</strong>.</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/57195814/simulating-joystick-movement-using-accessibilityservice"><strong>Simulating joystick movement using AccessibilityService</strong></a></li>
  <li><a href="https://stackoverflow.com/questions/56389826/why-the-continuestroke-function-is-not-work"><strong>Why the continueStroke function is not work</strong></a></li>
  </ul>
  
  <p>And below is a supposed fixed code that i found <a href="http://www.mysql5.com/thread-496-1-1.html" rel="noreferrer">here</a> =></p>
  
  <p><em>android AccessibilityService:</em></p>
  
  <pre><code>// Simulates an L-shaped drag path: 200 pixels right, then 200 pixels down.
  Path path = new Path();
  path.moveTo(200,200);
  path.lineTo(400,200);
  
  final GestureDescription.StrokeDescription sd = new GestureDescription.StrokeDescription(path, 0, 500, true);
  
  // The starting point of the second path must match
  // the ending point of the first path.
  Path path2 = new Path();
  path2.moveTo(400,200);
  path2.lineTo(400,400);
  
  final GestureDescription.StrokeDescription sd2 = sd.continueStroke(path2, 0, 500, false); // 0.5 second
  
  HongBaoService.mService.dispatchGesture(new GestureDescription.Builder().addStroke(sd).build(), new AccessibilityService.GestureResultCallback(){
  
  @Override
  public void onCompleted(GestureDescription gestureDescription){
  super.onCompleted(gestureDescription);
  HongBaoService.mService.dispatchGesture(new GestureDescription.Builder().addStroke(sd2).build(),null,null);
  }
  
  @Override
  public void onCancelled(GestureDescription gestureDescription){
  super.onCancelled(gestureDescription);
  }
  },null);
  </code></pre>
  
  <p>Then, my doubt is: <strong>how send correctly mouse coordinates for code above, of the way that can perform drag to any direction?</strong> Some idea?</p>
  
  <hr>
  
  <p><strong>Edit 3:</strong></p>
  
  <p>I found two routines that are used to perform drag, but they are using <a href="http://developer.android.com/reference/android/app/UiAutomation.html" rel="noreferrer">UiAutomation</a> + <a href="https://stackoverflow.com/questions/23159265/how-to-inject-click-event-with-android-uiautomation-injectinputevent"><code>injectInputEvent()</code></a>. AFAIK, injection of event works only in a system app like said <a href="https://stackoverflow.com/a/33443102/9708179">here</a> and <a href="https://gist.github.com/oneyoung/a14c99856c3984263734" rel="noreferrer">here</a> and i not want it.</p>
  
  <p>This are routines found:</p>
  
  <ul>
  <li><a href="https://github.com/USC-NSL/PUMA/blob/2545e58b22f421f387a52f73fa893cf7716bab3c/src/nsl/stg/uiautomator/core/MyInteractionController.java#L403" rel="noreferrer">public boolean swipe(int downX, int downY, int upX, int upY, int steps, boolean drag)</a></li>
  <li><a href="https://github.com/USC-NSL/PUMA/blob/2545e58b22f421f387a52f73fa893cf7716bab3c/src/nsl/stg/uiautomator/core/MyInteractionController.java#L442" rel="noreferrer">public boolean swipe(Point[] segments, int segmentSteps)</a></li>
  </ul>
  
  <p>Then to achieve my goal, i think that 2rd routine is more appropriated to use (following the logic, excluding event injection) with code showed on <strong>Edit 2</strong> and send all points of <code>pictureBox1_MouseDown</code> and <code>pictureBox1_MouseMove</code> (C# Windows Forms Application) respectively to fill <code>Point[]</code> dynamically and on <code>pictureBox1_MouseUp</code> send cmd to execute the routine and use this array filled. If you have a idea to 1st routine, let me know :D.</p>
  
  <p>If after read this <strong>Edit</strong> you have a possible solution, show me in a answer please, while i will try and test this idea.</p>