{"action":"created","issue":{"url":"https://api.github.com/repos/lvonasek/tango/issues/25","repository_url":"https://api.github.com/repos/lvonasek/tango","labels_url":"https://api.github.com/repos/lvonasek/tango/issues/25/labels{/name}","comments_url":"https://api.github.com/repos/lvonasek/tango/issues/25/comments","events_url":"https://api.github.com/repos/lvonasek/tango/issues/25/events","html_url":"https://github.com/lvonasek/tango/issues/25","id":327254758,"node_id":"MDU6SXNzdWUzMjcyNTQ3NTg=","number":25,"title":"Projection from mesh to rgb images incorrect","user":{"login":"miguelriemoliveira","id":11350998,"node_id":"MDQ6VXNlcjExMzUwOTk4","avatar_url":"https://avatars1.githubusercontent.com/u/11350998?v=4","gravatar_id":"","url":"https://api.github.com/users/miguelriemoliveira","html_url":"https://github.com/miguelriemoliveira","followers_url":"https://api.github.com/users/miguelriemoliveira/followers","following_url":"https://api.github.com/users/miguelriemoliveira/following{/other_user}","gists_url":"https://api.github.com/users/miguelriemoliveira/gists{/gist_id}","starred_url":"https://api.github.com/users/miguelriemoliveira/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miguelriemoliveira/subscriptions","organizations_url":"https://api.github.com/users/miguelriemoliveira/orgs","repos_url":"https://api.github.com/users/miguelriemoliveira/repos","events_url":"https://api.github.com/users/miguelriemoliveira/events{/privacy}","received_events_url":"https://api.github.com/users/miguelriemoliveira/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-05-29T10:11:48Z","updated_at":"2018-06-06T12:05:47Z","closed_at":"2018-06-06T11:43:07Z","author_association":"NONE","body":"Hi @lvonasek ,\r\n\r\nas I told you I am back working on this. I am having some problems and wanted to get your opinion. So please, if you have some time read through this long (sorry) text and tell me what you think I am doing wrong.\r\n\r\nMy first task was to create a python script (that's my programming language) that would read an open constructor dataset, in particular:\r\n\r\n1. The obj file in which the mesh is stored (given in your dataset folder, I am using the obj inside the dataset folder, I assume it is the non-simplified mesh)\r\n2. The set of jpg rgb image files and corresponding txt files indicating the camera pose for each image (given in your dataset folder, in #20 you told me how they work)\r\n3. The rgb camera's intrinsic parameters (I was not sure here, see bellow)\r\n4. The per image ply or depth map (did not get this far yet)\r\n\r\nI implemented the code reading all this stuff. As far as I can tell it should be working. To verify, what I do is to project the triangles in the mesh (1) to all the images described in (2). \r\n\r\nTo get a better feeling, I select only a few triangles on the mesh which belong to an object (i.e. a door or a book), that is, I create a new mesh with meshlab by erasing all other faces,  and see if those triangles are well projected.\r\n\r\nThe odd thing is that they seem to be correctly projected for some images and incorrectly projected for others ... here's an example \r\n\r\n![screenshot from 2018-05-29 10-50-46](https://user-images.githubusercontent.com/11350998/40651673-32f09a12-632e-11e8-9c8b-71d6f25bf544.png)\r\n\r\n... the object is the red book. For some images, e.g. 5,8,14,19,20 the projection seems ok. For others, e.g. 28,32,38 the triangles are just not well aligned with the book.\r\n\r\nI see some reasons that could explain why this is not working, can you tell me your opinion?\r\n\r\na. There is something wrong with the timestamps -> It could be that, for some images the sync between the time of capturing the image and the time of getting the camera pose would fail. However, since it is the images which are close that are always wrong, it does not seem like a random unaligned timestamp problem.\r\nb. I am doing something wrong with the projection -> It is very likely but I don's understand why the projection works for some images and not others ... In any case, my next step should be to  try to implement the projection as explained in #19 \r\nc. I am getting the wrong intrinsics -> long explanation, see bellow. \r\n\r\nAs for c. what I did was to use an app called \"TangoROSStreamer\" to publish camera_info_messages and then used it to create a yaml configuration file which I read. The content of the file is shown bellow. \r\n\r\n```yaml\r\nimage_width: 1920\r\nimage_height: 1080\r\ncamera_name: camera_color\r\ncamera_matrix:\r\n  rows: 3\r\n  cols: 3\r\n  data: [1484.120151053405, 0.0, 982.1869163422028, 0.0, 1483.296808359715, 543.0569322723705, 0.0, 0.0, 1.0]\r\ndistortion_model: rational_polynomial\r\ndistortion_coefficients:\r\n  rows: 1\r\n  cols: 5\r\n  data: [0.09370951394834764, -0.1640721737187069, 0.0, 0.0, 0.1224192868301668]\r\nrectification_matrix:\r\n  rows: 3\r\n  cols: 3\r\n  data: [1, 0, 0, 0, 1, 0, 0, 0, 1]\r\nprojection_matrix:\r\n  rows: 3\r\n  cols: 4\r\n  data: [1484.120151053405, 0.0, 982.1869163422028, 0.0, 0.0, 1483.296808359715, 543.0569322723705, 0.0, 0.0, 0.0, 1.0, 0.0]\r\n```\r\n\r\nThis calibration is for a 1920x1080 image, which makes sense since the images you have in the dataset are this size. However, when I use the Google Constructor they have a calibration file for a 960x540 camera: \r\n\r\n```yaml\r\ncamera_id: TANGO_3DR_CAMERA_COLOR\r\ncamera_intrinsics:\r\n  calibration_type: TANGO_3DR_CALIBRATION_POLYNOMIAL_2_PARAMETERS\r\n  width: 960\r\n  height: 540\r\n  model: [742.060075526702, 741.648404179857, 490.843458171101, 271.278466136185]\r\n  distortion: [0.0937095139483476, -0.164072173718707]\r\ncamera_extrinsics:\r\n  orientation: [-0.706785140012307, 0.707372138513566, 0.00647751405612829, 0.00612089243330192]\r\n  translation: [-0.0178448539838816, 0.00721245490079159, -0.00481233577874809]\r\n```\r\n\r\n... my guess is that they use the camera at half the resolution. Do these values make sense for you ... do you have similar values?\r\n\r\nThanks in advance,\r\nMiguel\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/lvonasek/tango/issues/comments/395045196","html_url":"https://github.com/lvonasek/tango/issues/25#issuecomment-395045196","issue_url":"https://api.github.com/repos/lvonasek/tango/issues/25","id":395045196,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTA0NTE5Ng==","user":{"login":"lvonasek","id":6472545,"node_id":"MDQ6VXNlcjY0NzI1NDU=","avatar_url":"https://avatars3.githubusercontent.com/u/6472545?v=4","gravatar_id":"","url":"https://api.github.com/users/lvonasek","html_url":"https://github.com/lvonasek","followers_url":"https://api.github.com/users/lvonasek/followers","following_url":"https://api.github.com/users/lvonasek/following{/other_user}","gists_url":"https://api.github.com/users/lvonasek/gists{/gist_id}","starred_url":"https://api.github.com/users/lvonasek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvonasek/subscriptions","organizations_url":"https://api.github.com/users/lvonasek/orgs","repos_url":"https://api.github.com/users/lvonasek/repos","events_url":"https://api.github.com/users/lvonasek/events{/privacy}","received_events_url":"https://api.github.com/users/lvonasek/received_events","type":"User","site_admin":false},"created_at":"2018-06-06T12:05:47Z","updated_at":"2018-06-06T12:05:47Z","author_association":"OWNER","body":"Hi Miguel,\r\n\r\nI am using raw images. Distortion is applied during texturing (postprocess), in my dataset viewerI ignore it.\r\n\r\nI forgot to tell you one important detail - the obj model is little bit simplified because there is no standart how to save per vertex colors and generating texture does not work with too detailed geometry.\r\n\r\nIf you want to do a research connected to 3D scanning and improving the scan quality, I have one problem:\r\nif I scan flat surface with a grid pattern (floor in my kitchen) then the photos are not applied correctly - the motion tracking generates some small mistake which makes in the end from the grid pattern some kind of waves.\r\nThe first attempt to solve this was adding into settings \"use sharp photos only\" (because I thought the source of the problem are just blured photos but it is just one part of the problem).\r\n\r\nThe goal of the research would be to correct the camera matrices in the dataset, also in the end the texturing should result completely. I would provide all you ask for (if I am able to deliver it).\r\n\r\nI am developing last months an ARcore version of the scanner:\r\nhttps://plus.google.com/106761694899705891407/posts/V35843UquAz\r\nand there is the problem with motion tracking much bigger.\r\n\r\nRegards\r\n\r\nLuboš"}}
{"actor":{"display_login":"lvonasek"}}