Research by rexagod Open source environmental science research at Public Lab Google Summer of Code 2019 Public Lab Final Work Product: Pranshu Srivastava rexagod Sun, 25 Aug 2019 23:43:13 +0000 <h2>Project Details</h2> <ul> <li>Title: <a href=""><code>matcher.js</code> library and <code>Leaflet.DistortableImage</code> overhaul</a></li> <li>Repositories created (and contributed to): <a href=""><code>matcher-core</code></a> and <a href=""><code>matcher-cli</code></a></li> <li>Repositories contributed to: <a href=""><code>Leaflet.DistortableImage</code></a></li> </ul> <h2>Project Description</h2> <h3><a href=""><code>matcher-core</code></a></h3> <p><code>matcher-core</code> essentially employs the ORB<a href="">(Oriented FAST and Rotated BRIEF)</a> algorithm to mine patterns using the well-known <a href="">FAST(Features from Accelerated Segment Test)</a> keypoint detector and the <a href="">BRIEF(Binary Robust Independent Elementary Features)</a> descriptor technologies, which provide appreciable performance boosts on low computational costs.</p> <h3><a href=""><code>matcher-cli</code></a></h3> <p>A Node-based CLI utility for headlessly testing custom matcher-core configurations, this module eases any development process on the matcher-core library by allowing the user to run a set of commands on their locally deployed Chromium environment in a headless manner, i.e., right from the CLI, without the need to open the browser whatsoever, be it testing, demonstration, debugging issues, etc. <code>matcher-cli</code> enables headless support for all of these tasks, and can easily integrate with your choice of CI to automate any of the above tasks right inside your CI.</p> <h3><a href=""><code>Leaflet.DistortableImage</code></a></h3> <p>A Leaflet extension to distort images -- "rubbersheeting" -- for the <a href=""></a> (<a href="">src</a>) image georectification service by <a href="">Public Lab</a>. Leaflet.DistortableImage allows for perspectival distortions of images on the client-side.</p> <h2>Project Demonstrations</h2> <h3><a href=""><code>matcher-core</code></a></h3> <p>Watch the <code>matcher-core</code> in action by mapping two 4K images this <a href="">demo</a>.</p> <h3><a href=""><code>matcher-cli</code></a></h3> <p>Run the following command inside your terminal for an interactive demo.</p> <pre><code class="bash">$ bash &lt;(curl -s <a href=""></a>) </code></pre> <h3><a href=""><code>Leaflet.DistortableImage</code></a></h3> <p>Checkout the <code>stitcher</code> and <code>projector</code> helper modules for <code>matcher-core</code> in action <a href="">here</a> by enabling <code>matcher.js</code> from the bottom-left. The current progress is demonstrated <a href="">here</a> which I got the opportunity to contribute to for some time as well.</p> <h2>Project Involvement</h2> <p>Being welcoming in nature, the <code></code>s below have been catered from a contributor's perspective, and include almost every resource a contributor would require in order to get started with these.</p> <ul> <li><a href=""><code>matcher-core</code></a></li> <li><a href=""><code>matcher-cli</code></a></li> <li><a href=""><code>Leaflet.DistortableIamge</code></a></li> </ul> <p>FTOs can be found below.</p> <ul> <li><a href=""><code>matcher-core</code></a></li> <li><a href=""><code>matcher-cli</code></a></li> <li><a href=""><code>Leaflet.DistortableIamge</code></a></li> </ul> <h2>Project Leftovers and Future Plans</h2> <ul> <li><a href=""><code>matcher-core</code></a> <ul> <li>Implementing streams for data flow between sockets. This would immensely reduce noise, and speed up the load time. The idea's pretty simple, send the <code>utils</code> promise directly to the socket in the form of a stream and resolve it there and then.</li> <li>Rebasing <a href=""><code>rexa-soc-ldi</code></a> after we shift to ES6.</li> <li>Containerized build through Docker on the CI.</li> <li>More (unique) headless automated tasks (other than tests, for eg., crossposting back to the PR on each build, etc.) on the CI.</li> <li>Rebasing <a href=""><code>matcher-module</code></a> once we modify image-sequencer for asychronous modules (as proposed to <code><a href="/profile/jywarren">@jywarren</a></code>).</li> </ul> </li> <li><a href=""><code>matcher-cli</code></a> <ul> <li>Vorpal was incorporated for a better understanding of the codebase, but takes a bit to load. Implementing Vorpal's APIs purely in Node would definitely bring up the loading time, but it would be a tradeoff for code abstractions.</li> <li>Generalize standard methods and expose APIs to gain flexibility.</li> </ul> </li> <li><a href=""><code>Leaflet.DistortableIamge</code></a> <ul> <li>PWA conversion! Perhaps take a step further and integrate electron as well.</li> <li>In-browser drag and drop functionality.</li> <li>Headless tests on CI.</li> </ul> </li> </ul> <h2>Project Overall Contribution</h2> <ul> <li><code>matcher-core</code> <ul> <li><a href="">PRs</a></li> <li><a href="">Issues</a></li> <li><a href="">Commits</a></li> </ul> </li> <li><code>matcher-cli</code> <ul> <li><a href="">PRs</a></li> <li><a href="">Issues</a></li> <li><a href="">Commits</a></li> </ul> </li> <li><code>Leaflet.DistortableImage</code> <ul> <li><a href="">PRs</a></li> <li><a href=";q=is%3Aissue+rexagod+">Issues</a></li> <li>Commits on my <a href="">fork</a> and on <a href="">main</a>.</li> </ul> </li> </ul> <h2>Project Experience</h2> <p>The Public Lab community has supported me througout the duration, and even before this Summer of Code began, and granted me such a great opportunity to work among such brilliant and supportive people, for which I'll always be obliged to them. I've undoubtedly learnt numerous stupendous concepts throughout my time here, and perfected my passion for JavaScript through the guidance of my mentors, who were always there to support me, motivate me, and help me push my limits, for which I'll always be thankful.</p> <p>Thank you, Public Lab, for such a fascinating experience, I'd love to continue being a part of and keep on contributing to this amazing community!</p> Google Summer of Code: Second Evaluations rexagod Sun, 21 Jul 2019 18:23:39 +0000 <h1>Public Lab: Second Evaluations</h1> <blockquote><p>Programs must be written for people to read, and only incidentally for machines to execute.</p></blockquote> <p>And just like that we've arrived upon the second evaluations! I've had the immense pleasure and opportunity to switch and work between a vast array of issues and projects during the past weeks, which made me realize the importance for a developer to be well-acquainted with a diverse set of tools and expertise in utilizing them to their needs! In this post, I'll throw some light on the different things I've had the pleasure of working on, with due respect to all of my mentors, who made these humongous tasks seem like a cool summer breeze!</p> <p>Let's get started!</p> <p>Okay, so other than the regular reviewing and first-timers stuff, I was engaged in the following projects.</p> <ul> <li><strong><a href="">Jest integration for CI testing automation</a></strong></li> </ul> <p>Wrote Jest tests for verifying successful fetch attempts for the <code>matcher.js</code> module. These tests run on a virtual page that is setup in the CI's localhost and on a successful exit code, a screenshot of that page is posted back into the respective PR on GitHub via Travis (or whatever CI you're using). A snippet of two of such tests is given below.</p> <pre><code class="javascript">describe('Built-in modules', () =&gt; { beforeAll(async () =&gt; { await page.goto(`<a href="`);">`);</a> }); it('corners: all key-points detected', async () =&gt; { expect(context('corners')).not.toBeNull(); }); it('matches: all rich matches found', async () =&gt; { expect(context('matches')).not.toBeNull(); }); }); </code></pre> <ul> <li><strong><a href="">Keymapper improvements to pave way for stalled PRs</a></strong></li> </ul> <p>Made final changes to the keymapper PR, which has been open for a while now. Getting this one in will allow merging of many of my other PRs that have been around for the same time and require very little or no changes. These are:</p> <ul> <li><a href="">Sub-menus implementation</a></li> <li><a href="">EXIF functionality</a></li> <li><a href="">Reveal functionality</a></li> <li><strong><a href="">matcher.js integration into Leaflet.DistortableImage</a></strong></li> </ul> <p>Implemented <a href=""><code>stitcher</code></a> and <a href=""><code>projector</code></a> modules for auto-locating overlays based on their EXIF coordinates and Leaflet UI helpers for graphic renders on overlays, respecively. For more details, <a href="">see this</a>.</p> <ul> <li><strong><a href="">Getters and Setters for <code>matcher.js</code></a></strong></li> </ul> <p>Refactored older <code>tick</code> implementation in the <code>matcher.js</code> library into smaller methods, namely <code>findPoints</code> and <code>findMatchedPoints</code> which would allow for improved code readability and specificity. These methods return all the detected corners, and the overlapping matches between the two images, respectively. <code>findPoints</code> is actually faster by a few hundred milliseconds as compared to <code>findMatchedPoints</code>, since the latter has the computational overhead of running the algorithm to overlap images and find matches among them, which takes more time.</p> <ul> <li><strong>Image-sequencer integration of matcher.js</strong></li> </ul> <p>A demonstrative module integration for <code>matcher.js</code> to show beginners and experienced devs the practical steps to follow in order to integrate it into a library. The implementation is in it's final stages, and I'll soon push a PR for the same once my <a href="">doubt</a> is resolved. I also had the idea of making a <code>matcher.js</code> API that people could straight away pass images and params in, and get points back! Let's see what feedback this gets, which will be essential for its implementation, if it happens.</p> <ul> <li><strong>Caching mechanism for matcher.js</strong></li> </ul> <p>Implemented a caching mechanism that allows the algorithm to only run once, that's it! For a bit more detail and interesting insights on the same, <a href="">read my comment here</a>.</p> <ul> <li><strong><a href="">Prepping up docs from a lesser in-depth but more user-friendly POV</a></strong></li> </ul> <p>Complex libraries require super simple documentation, which I think the <code>matcher.js</code> library could finally be close to having thanks to <a href="/profile/jywarren">@jywarren</a> who constantly guided me along the way! This may not seem much, having to do nothing with code but only docs, but a library is only as good as its documentation, which is indeed critical in a library's overall growth.</p> <ul> <li><strong>Simpler constructor and corresponding documentation</strong></li> </ul> <p>Again, thanks to <a href="/profile/jywarren">@jywarren</a>'s observant eye, I was able to change the initial <code>matcher.js</code> constructor from this,</p> <pre><code class="javascript">Promise.resolve(new orbify('/path/to/img1.jpg', '/path/to/img2.jpg', { browser: true, params: { lap_thres: 35, eigen_thres: 40, blur_size: 4, matchThreshold: 50 } }).utils).then(function (utils) { callback(utils); }); </code></pre> <p>to this,</p> <pre><code class="javascript"> new Matcher('/path/to/img1.jpg', '/path/to/img2.jpg', async function (r) { document.getElementById('res').innerHTML = JSON.stringify(await r); }, { browser: true, caching: true, leniency: 30, params: { lap_thres: 30, eigen_thres: 35 } }); </code></pre> <p>with the addition of type checks and a simplistic callback function that simplifies one's engagement with this library. This, although a small change, makes the library look really nice from a semantic POV while keeping a check on the order of things, thanks a ton, <a href="/profile/jywarren">@jywarren</a>!</p> <ul> <li><strong><a href="">Fix user reported bugs</a></strong></li> </ul> <p>User engagement is something that is of immense value here and Public Lab strives to resolve any bugs or issues that the users might be facing. Recently, <a href="/profile/molangmuir10">@molangmuir10</a> suggested some changes and reported some bugs for the MK/LDI UI/UX (thanks!) which (documented <a href="">here</a>) were acted upon today <a href="">here</a>, and are up for <a href="/profile/jywarren">@jywarren</a>'s and <a href="/profile/sashadev-sky">@sashadev-sky</a>'s review.</p> <p>Yup! Pretty much this is almost everything I've been around for the past few weeks, and these are in the absolute final stages, if not done with already, which makes me excited all the amazing stuff that <code>async</code>s us to be <code>await</code>ed for the next evaluations (get it?)! It's been an exciting journey till now, and I'm affirmative that it'll be even more astounding and full of commits, reviews, and PRs for the next month as well!</p> <p>And as always, thank you for giving this a read, ciao!</p> Google Summer of Code @ Public Lab: 3rd and 4th Week rexagod Sat, 22 Jun 2019 21:06:45 +0000 <h1>Public Lab: 3rd &amp; 4th Week</h1> <blockquote><p>Having a robust initial codebase is all that matters. Integration will be frictionless, and you can be sure of a much more stable environment, right from the beginning.</p></blockquote> <p>The third and forth week was an interesting time slice. What made it so, was the fact that the codebase now had a sturdy foundation, so in a way the hard part was over. It was really relaxing to see that the long nights of laboring had been done with, which meant I could now shift my topic of concern from establishing a strong codebase to working on its integrations and add-ons, the interesting stuff!</p> <p>Other than reviewing PRs, opening FTOs, and general debugging, these two weeks can be best characterized by the undertaking of the following tasks, in this order, and detailed further below, in the same order.</p> <ul> <li>Integrate <a href=""><code>matcher-core</code></a> algorithm into <a href=""><code>Leaflet.DistortableImage</code></a>.</li> <li>: Setup a UI for visualizing <a href=""><code>matcher-core</code></a>'s results inside <a href=""><code>Leaflet.DistortableImage</code></a> using only <code>Leaflet</code> as the helper library.</li> <li>Setting up Babel, Browserify, ESLint, and Terser in both of the modules to abide by code standards, and to ensure a fine level of code quality in the future.</li> <li>Finishing up on <a href=""><code>matcher-cli</code></a>, i.e., incorporating support for background processing, dynamic command assignments, removing <a href="">unwanted dependencies</a>, and adding necessary <code>npm scripts</code> in general.</li> <li>Setting up support for Continuous Integration platforms as discussed in the previous post, and actually demonstrating the same on a PR, in order to expose its dynamic nature.</li> <li>Setting up Jest as a testing, mocking, and visualizing framework for the <a href=""><code>matcher-cli</code></a> library, and demonstrate how the users can exploit this new "headless" dimension for expanding on the traditional approaches towards testing.</li> </ul> <p>Starting off with integrating the <code>core</code> algorithm in the <a href=""><code>Leaflet.DistortableImage</code></a> library, in doing so I didn't face any issues whatsoever, and you should not too if you, one, modularize your code into logical components that actually make sense, and not because everyone's talking about it, and two, embrace functional programming, and for the love of god, use only a limited amount of syntactic sugar, only as much as you need. You can't go around town telling people you've actually embraced functional programming, while there still exist big clunky <code>class</code>es in your code. So just keep that in mind, or not, since this is just how I feel, and it's just my take on it.</p> <pre><code class="javascript">if(document.querySelectorAll("svg &gt; g &gt; path").length) { [].slice .call(document.querySelectorAll("svg &gt; g &gt; path")) .slice(0,3) .map(function(x) { x.parentNode.removeChild(x); }); } var lat_offset = +; var lng_offset = -best_point.lng + corresponding_best_point.lng; // relocation code // getCorners -&gt; formal values -&gt; copy of values -&gt; not useful here // _corners -&gt; actual values -&gt; copy of reference // hence can be useful to mutate states (position of the images) overlay._corners[0] = [ processedPoints.images[1].getCorner(0).lat - lat_offset, processedPoints.images[1].getCorner(0).lng - lng_offset ]; overlay._corners[1] = [ processedPoints.images[1].getCorner(1).lat - lat_offset, processedPoints.images[1].getCorner(1).lng - lng_offset ]; overlay._corners[2] = [ processedPoints.images[1].getCorner(2).lat - lat_offset, processedPoints.images[1].getCorner(2).lng - lng_offset ]; overlay._corners[3] = [ processedPoints.images[1].getCorner(3).lat - lat_offset, processedPoints.images[1].getCorner(3).lng - lng_offset ]; var zoom_level = map.getZoom(); map.setView(overlay.getCenter(), (zoom_level%2?zoom_level+1:zoom_level-1)); L.DomEvent.on(overlay._image, 'mousedrag', overlay.editing.enable, overlay.editing); </code></pre> <p>Setting up the <code>auto-stitcher</code> module was all about converting independent image-pixel space coordinates back to their respective latitudes and longitudes, then "projecting" the identified points of interests, along with connecting <code>polylines</code> for the top three most confident matches, onto the images, while keeping away any induced latency that might have crept in due to the drag, so I had to account for that as well. After that, came the "stitching" part, which essentially required the mutation of the overlay corners, translating them along certain "connecting paths" connecting the two.</p> <p>Setting up Babel, Browserify, ESLint, and Terser was pretty straight-forward and there isn't much to discuss here, although one should never undermine the importance of such "quality-maintaining" modules.</p> <p><a href="">Automating the CLI on CI</a> was one of the things I had in mind for a while, and was really interested to see its potential as well, so without wasting any time, I spun up a <code>.travis.yml</code> script that allowed me to deploy <a href=""><code>matcher-cli</code></a> on <code>Travis</code> exactly how I wanted to. The testing however required some victim issues to checkout the automated cross-posting tech before finally making it dynamic for PRs, and so <a href="">issue#2</a> was, I think over a span of a few hours, spammed with exactly 36 messages right form the CI, in sets of three (Node 8, 10, and 12) due to the 12 different configurations of the same script I had to test, in order to get a perfect CI log, along with a snapshot of the virtual page right in the CI! Among many other possibilities, this would also allow the user to change the configurations right in their forks, submit a PR, all inside GitHub, and without even cloning the repository on their local, and <code>Travis</code> would report every thing they need to know right in their PRs. Headless automation FTW!</p> <p>Going with Jest as the ultimate testing, mocking, and visualizing framework for the <code>matcher-cli</code> library was actually something we discussed about on one of the <a href="">issues</a> and seeing the maximum amount of advantages along with headless and async compatibility, with support for almost anything you'd expect, or rather want out of a test runner, made this my first choice fot the same module.</p> <p>This pretty much summarizes my third and fourth week of Google Summer of Code at Public Lab, and be sure to chime in for the future ones!</p> <p>Thank you for giving this a read, ciao!</p> Google Summer of Code @ Public Lab: 1st and 2nd Week rexagod Sat, 22 Jun 2019 21:03:37 +0000 <h1>Public Lab: 1st &amp; 2nd Week</h1> <blockquote><p>From refactoring an 7-year-old library, to embedding node support into it, while optimizing it for browser, the first two weeks started off as one of the most interesting but crucial stages of my timeline, and yet somehow, we made it work.</p></blockquote> <p>I wouldn't be wrong to say that this period of Google Summer of Code at Public Lab was indeed, in my case, rather epiphanic. I'll explain.</p> <p>Other than reviewing PRs and making FTOs, most of my time spent at Public Lab right till the first evaluation, required me to essentially work on a "pattern-mining" library, called <code>matcher.js</code>, which would initially consist of just a single module called <a href=""><code>matcher-core</code></a> that would help users in searching for rich matches across two images, but interestingly grew to encompass another library, a <a href=""><code>matcher-cli</code></a> module, which was originally built to pass custom commands to manipulate the test environment and monitor changes on a local server, and with time, would power Public Lab's first ever headless-driven test written to support Continuous Integration platforms, including cross-posting.</p> <h2><a href=""><code>matcher-core</code></a></h2> <p>The <a href=""><code>matcher-core</code></a> module works on <a href="">Oriented FAST and Rotated BRIEF (ORB)</a> algorithm and was developed at OpenCV labs by Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R. Bradski in 2011, as an efficient and viable alternative to SIFT and SURF. ORB was conceived mainly because SIFT and SURF are patented algorithms. ORB, however, is free to use.</p> <p>The idea of abstracting such a sophisticated piece of code, and mending it to Public Lab's custom needs seemed ambitious to say the least, and of course, the implementation proved to be my primary concern for the first few days.</p> <pre><code class="javascript"> /* a simpler, yet much better way * that Eugene kept in mind during * the initial implementations */ // the snippet below analyzed all pixels for corners, but initialized `matchT` only for the rich matches let i = 640 * 480; while (--i &gt;= 0) { screenCorners[i] = new jsfeat.keypoint_t(0, 0, 0, 0, -1); matches[i] = new matchT(); } </code></pre> <p>As a matter of fact, all the code that was written in the first week unfortunately did not meet my expectations, i.e., I had initially tried to couple ORB's FAST and BRIEF features with <code>Uint8ClampedArray</code>s instead of the original <code>matchT</code> structures, which did not seem to work out very well, because although the time complexity was reduced (in some cases, by ~700ms), I had to add a property for every paramenter which was passed, which ended up taking a lot of space than I anticipated initially. Imagine processing a small <code>640x320</code> image with a 307200-sized array each element consisting of its repective <code>RGBA</code> data along with pixel-dependent 4 parameters, that's about 307200 * 4 values * 4 parameters * 8 Bits = 4.915 Megabytes! That was way more than expected and indicated the fact that I had to shift all my calculations, modifications, everything while keeping the <code>matchT</code> structure in mind, because in all honesty, this was the only solution that efficiently employed <code>JavaScript-specific</code> features to reduce the aforementioned induced complexity, so I went with that.</p> <p>Now that I come to think of it, maybe Eugene faced the same problem as well?</p> <p>So more or less, this meant that any change that I brought to the library, which affected the pixel arrays in an way whatsoever, had to consider the negative affects (if any), that it caused to them, and if the these were more than I could mend, I had to come up with a different approach.</p> <p>By the next mid-week I had a custom optimized version of the algorithm, and the only reason I believe things moved so fast was because my first step was to modularize the whole library, each component isolated, yet working together, without which, this would have taken me entire two weeks!</p> <p>Altering each component one-by-one, I felt a need to observe the changes that would be incurred if I changed each parameter to every finite value they could possibly be assigned to, but to do that I would've needed to individually each different set of configuration on my rig atleast a 5 blur_sizes * 100 lp_thresholds * 100 eigen_values * 100 matchThresholds = 5,000,000 runs for every configuration! As surprising as it may sound, I actually did not have that kind of time. At all.</p> <p>So to automate tasks such as the above, and even more complex ones as they came, to find out the best configuration (from variables, functions to non-primitive types) for the <a href=""><code>matcher-core</code></a>, I came up with an idea.</p> <h2><a href=""><code>matcher-cli</code></a></h2> <p>The <a href=""><code>matcher-cli</code></a> module was never originally included in my proposal, but it was a beneficial add-on that had to be made, and also interested me a great deal, so I got to work.</p> <p>What <a href=""><code>matcher-cli</code></a> aims to do is pretty simple and straight-forward: Control every aspect of <a href=""><code>matcher-core</code></a>, right from the command-line interface. In detail, however, this means that <a href=""><code>matcher-cli</code></a> will utilize <code>puppeteer</code> to take absolute control over the headless environment it creates. This environment is then deployed on a port which will house a <code>Chromium</code> instance, running the user's configuration of the <code>matcher-core</code> library in one of it's pages. These "virtual pages" can then be scraped for immense testing improvements and debugging, since one will be able to <em>actually see</em> the visual impacts of their code base inside the headless environment once it gets deployed.</p> <pre><code class="javascript"> // allows for dynamic assignment of commands to their respective functions if (eval(`commands.${query}`) !== undefined) { eval(`commands.${query}()`); } else if (eval(`summoner.${query}`)) { console.log(eval(`summoner.${query}`)); } else { error(`Invalid command: "${query}", exiting...\n`); process.exit(); } </code></pre> <p>This module will work based on different "commands" as they get defined in the <a href=""><code>matcher-commands.js</code></a> entry point. Commands will execute to trigger their respective linked functions, which can be any operation the user wants to perform once the virtual pages are setup. In order to generalize this library for the users, I started of by removing off all my older custom code to conduct experiments with the core, and rather including useful and simple commands that the users will frequent.</p> <p>So I began adding the basic <code>clear</code> and <code>update</code> functionality for the CLI. <code>clear</code>, as one can easily discern, cleans out the console screen and is a pretty simple command itself, nothing complex. Although, my main motive behind setting up such a basic command was so that users can actually look at its implementation and learn how easy it is to ship your own commands to <code>matcher-cli</code>! Moving on to my favorite command, if that's a thing, the <code>update</code> command can prove helpful for all the non-technical users who don't want to <code>git pull</code> or <code>npm update</code> every release of the module, or spend hours fixing bugs, all they need to do is run <code>matcher update</code> and all of their files, along with the latest (or pinned) versions of dependencies will be compared against the code base, and all the discrepancies can be resolved. Other than these, some of the more "complex" commands can be found in <a href=""><code>matcher-summon.js</code></a>, namely, <code>corners</code> and <code>matches</code>, that search the virtual pages for detected matches and rich matches, respectively, under the current configuration of the core.</p> <p>This has been pretty much what I've been doing for the past two weeks, and to for the future, I'll focus on setting this up on Continuous Integration platforms, integrating <code>core</code> and a Leaflet UI for the same into <a href=""><code>Leaflet.DistortableImage</code></a>, setting up Jest to run (headless) tests, and get both of my libraries forked!</p> <p>Thank you for giving this a read, ciao!</p> GSoC proposal: Mapknitter ORB Descriptor (w/ auto-stitching, pattern training, and live video support) and LDI revamp (major UI enhancements) rexagod Mon, 11 Mar 2019 15:08:16 +0000 <h2>About me</h2> <p>Name : Pranshu Srivastava</p> <p>Email : <a href=""></a></p> <p>Github : <a href=""></a></p> <p>Proposal: Includes <a href="">MapKnitter UI</a> and <a href=",+auto-stitch+in+MapKnitter+(magnetic+attraction">Microscope live stitching, auto-stitch in MapKnitter (magnetic attraction)</a>)</p> <p>Student year: Sophomore (second-year student)</p> <p>Field of study: Computer Science Engineering, B.Tech.</p> <p>Affiliation : Jaypee Institute of Information Technology, India</p> <p>Location : Noida, India</p> <h2>Project description</h2> <h3>Abstract</h3> <p>Employing machine learning techniques to incorporate an ORB desciptor (w/ auto-stitching, live video support) in order to enable better pattern matching along with some major LDI UI (and API) enhancements.</p> <h3>Aims to solve</h3> <p><strong>Both of the projects listed below are detailed descriptively in the implementations section</strong></p> <p><strong><a href=",+auto-stitch+in+MapKnitter+(magnetic+attraction">Oriented FAST and Rotated BRIEF (ORB) Descriptor</a>)</strong></p> <ul> <li>Currently, the user has to manually "stitch" similar looking images in order to form a larger one. We aim to automate this cumbersome process in the following sequence. <ul> <li>First, using some of the advanced machine learning techniques, mostly "Oriented FAST and Rotated BRIEF", i.e., ORB detection, offered by the JSFeat (JavaScript Computer Vision) Library the algorithm will understand <em>how similiar</em> any two images passed to it are, showing the corresponding outputs (keypoints, matches, good matches, gaussian blur and grayscale intensity) for each pair to the user. Also, a "training process" will be initiated every x seconds to better recognize and discover even more keypoints in the image, leaving lesser room for failed attempts at each iteration.</li> <li>Second, on the basis of the above details the user can choose to "stitch" the images which itself, will be an automated process, and the "auto-stitcher" <code>ToolbarAction</code> will proceed to automatically stitch them together (using the Auto-Stitcher module, detailed below).</li> <li>Lastly, for better accuracy, the user themselves can intervene and make small adjustments if they wish to obtain precise results.</li> </ul> </li> <li>Public Lab will also showcase a live video ORB matcher, that learns from an image passed to it, and goes on to find that particular pattern in the live video feed! At each step, the most efficient params will be considered, so as to maintain the maximum render FPS possible. This can be applied to various areas of interest, for eg., a particular minute organism can easily be detected if the video feed from the microscope was supplied to this module and compared against an image of it (downloaded from the net, or uploaded from user's local), or a prominent object in space using the satellite's live feed, the possibilities are endless!</li> </ul> <p><strong><a href="">Mapknitter UI enhancements</a></strong></p> <p>Currently, the <code>Leaflet.DistortableImage</code> repository serves as the supporting image distortion library of the <code>Mapknitter</code> repository hence proving itself to be an essential part of it. We'll be implementing some major core changes as per the planning issue describes <a href="" title="">here</a>, which will definitely prove beneficial to the user in terms of better accessibility and interactivity. Work regarding this already in progress and can be checked out <a href=";q=is%3Apr+author%3Arexagod+" title=";q=is%3Apr+author%3Arexagod+">here</a>.</p> <h3>Implementation</h3> <p><strong><a href=",+auto-stitch+in+MapKnitter+(magnetic+attraction">Oriented FAST and Rotated BRIEF (ORB) Descriptor</a>)</strong></p> <p><strong>Dividing this into subparts,</strong></p> <p>a) <strong>Microscope live-tracking:</strong> We can modify the algorithm to adapt to just a single image and train only on that rather than depending on the live video feed to train itself, in short, this will allow the user to provide an image (pattern) as an input and the tracker will then attempt to find that pattern in the feed every x seconds in a loop (currently set to 2s). On finding appropriate matches the tracker will filter the number of "good matches" and if they are above a certain threshold, display "matched".</p> <p><img src="" alt="Screenshot from 2019-03-09 19-36-13" /></p> <p><strong>Live-tracking in action!</strong> (do let me know if the .gif is taking way too long to load)</p> <p><a href=""><img src="" alt="Peek-2019-03-09-18-10.gif" /></a> Few points of interest above that one should notice are that the no matter the orientation, ORB will recognize image pattern if they are even partially visible. Also, notice how it confirms a match case even when some portion of the coaster was cut off.</p> <p>b) <strong>ORB Algorithm (currently in beta, <a href="">click here</a> to check it out!):</strong> We can enhance the current ORB algorithm in the following ways in order to better fit Public Lab's utilization. - [a simplified version of the] Currently proposed ORB structure, built on <a href="/profile/warren">@warren</a>'s suggestions, is depicted below. Note that this is based off of the review <a href="">here</a>, and satisfies the queries in that particular comment. The actual implementation in all practicality, will be much more detailed, but still draws inspiration from the snippet below.</p> <pre><code class="js">function ORBMatcher() { this.findPoints = findPoints; this.findMatchedPoints = findMatchedPoints; this.projectPointsInto = projectPointsInto; this.renderer = renderer; return this; } function findPoints(img) { //extracts [RGBA] info about every pixel in an image passed to it //img is an `Image()` instance var canvas = document.createElement("CANVAS"); //notice we aren't appending this anywhere (offscreen) //TC is a temporary canvas used to extract [RGBA] data for each pixel of image var TC = node.getContext("2d"); TC.drawImage(img,0,0,img.width,img.height); //faster than putImageData canvas.parentElement.childElement(canvas); //remove canvas since we are done with canvas APIs //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ //imgData contains `0-255` "clamped" points in a Uint8ClampedArray (refer MDN) //for eg., if the image is of the resolution 640x480, this array will have 1228800 //elements (307200x4), where all are "clamped" (rounded off) to have a value //between 0-255, and every set of 4 elements represents the R,G,B, and A values //for every single pixel, i.e., 307200 pixels var imgData = TC.getImageData(0, 0, primary_image.width, primary_image.height); //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ //finally, we can extract the data about every single pixel, which also removes //the need for having a `Stateful` module (saving space as well) since we //now already have everything we need } // construct correspondences for inliers // var correspondences_obj; // for (var i = 0; i &lt; count; ++i) { // var m = matches[i]; //`match_t()` construct:176 // var s_kp = screen_corners[m.screen_idx]; //`keypoint_t()` construct:175 // var p_kp = pattern_corners[m.pattern_lev][m.pattern_idx]; // pattern_xy[i] = { "x": p_kp.x, "y": p_kp.y }; //==&gt;X (img1points) // screen_xy[i] = { "x": s_kp.x, "y": s_kp.y }; //==&gt;Y (img2points) // } // correspondences_obj = {"X":pattern_xy,"Y":screen_xy}; function findMatchedPoints(X,Y) { var good_matches_locatorX=[]; var good_matches_locatorY=[]; var strong_inliers=0; var xk=0; var yk=0; //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ //assuming we have already constructed correspondences from imageData as //described above and X and Y hold the `pattern_idx`(image1points) and //`screen_idx`(image2points) of the matches array for every //pixel respectively for each pixel in { "x": p_kp.x, "y": p_kp.y } //and { "x": s_kp.x, "y": s_kp.y } formats for X and Y respectively (please refer to the image above) //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ //find respective homographic transforms for both of the models (X and Y) //run ransac homographic motion model helper on motion kernel to find //*only* the strong inliers (good matches) excluding away all the others //(works on static models as well) //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ var mm_kernel = new jsfeat.motion_model.homography2d(); var homo3x3 = new jsfeat.matrix_t(3, 3, jsfeat.F32C1_t); var ransac_param = new jsfeat.ransac_params_t(num_model_points,reproj_threshold, 0.5, 0.99); var match_mask = new jsfeat.matrix_t(500, 1, jsfeat.U8C1_t) //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ if (jsfeat.motion_estimator.ransac //runs once for static models and iterates if motion is detected (ransac_param, mm_kernel, X, Y, strong_inliers, homo3x3, match_mask)) { for (var i = 0; i &lt; count; ++i) { if ([i]) { X[strong_inliers].x = X[i].x; X[strong_inliers].y = X[i].y; Y[strong_inliers].x = Y[i].x; Y[strong_inliers].y = Y[i].y; good_matches_locatorX[xk++]= X[i]; console.log(X[strong_inliers]); //==&gt;{x: 359, y: 48},i.e., a matched point in X space good_matches_locatorY[yk++]= Y[i]; console.log(Y[strong_inliers]); //==&gt;{x: 65, y: 309},i.e., the corresponing point in Y space strong_inliers++; } }, Y, homo3x3, strong_inliers); //run kernel directly with inliers only } var matched_points = { "good_matches_X": good_matches_locatorX, "good_matches_Y": good_matches_locatorY, "matches_count": strong_inliers }; return matched_points; //store location of all good matches in both models (images), X and Y, as well as their count } function projectPointsInto(matched_points,canvas,match_thres_percentage=30,expected_good_matches=5) { var good_matches = matched_points.matches_count; //from above fn. if (num_matches) { //from `match_pattern` utility fn. if (good_matches &gt;= num_matches * match_thres_percentage/100) //we have a match! renderer.lines(canvas, matches, num_matches); //picks X/Y data from matches and num_matches which are GLOBALS if (good_matches &gt;= expected_good_matches) //in case of a solid match when results are as we expected or even better; //we can stop iterations here completely for performance issues } } var renderer = { //gets hoisted lines: function(canvas, matches, num_matches) { render_matches(canvas, matches, num_matches); }, box: function(canvas) { render_pattern_shape(canvas); } } //declarations and calls var matcher = new ORBMatcher(); var imageA = new Image(); imageA.src = "imageA.jpg"; var imageB = new Image(); imageB.src = "imageB.jpg"; var imageDataA = matcher.findPoints(imageA); var imageDataB = matcher.findPoints(imageB); //construct correspondences var matcherData = matcher.findMatchedPoints(X_,Y_); var canvas = querySelector("#deploy-canvas").getContext("2d"); matcher.projectPointsInto(matcherData,canvas); </code></pre> <p><img src="" alt="mnpyr" /></p> <ul> <li>Large Data files solution: For large images ranging from 10-15mb, we can downsample the image data matrix into a smaller one, while carefully preserving most of the inliers. <code>JSFeat</code> implements the <code>Gaussian Pyramid</code> method of downsampling a large image to a smaller one in it's <code>pyrdown</code> API, i.e., <code>jsfeat.imgproc.pyrdown(source:matrix_t, dest:matrix_t);</code>. Here we can set the destination matrix to as few rows and columns which will allow us to process things faster while not sacrificing the important inliers in the image, as is evident from the final (smallest) image in the image pyramid above.</li> <li>Proposed change in the search algorithm: Instead of starting from the first row and column, I believe a better approach would be starting from the means, i.e, if the image is of the resoulution 640x480, we can start our search from the <code>imgData</code>'s pixel info at row,col of [320][240] and moving out in a circular (vignette) fashion. This approach is based on the fact that majorly the images have meaningful data, or inlier clusters count maximum at the center, rather that at the borders. To stop the execution, refer to the <code>projectPointsInto</code> fn. in which we can do the same in case <code>good_matches &gt;= expected_good_matches</code>, where <code>expected_good_matches</code> is a custom threshold, and can be adjusted to reduce performance issues as well.</li> <li>Determining the "confidence" of each point match structure: The "confidence" for each point is currently being calculated in the <code>match_pattern()</code> util. fn. and can be returned to the <code>getMatches</code> fn. (refer snippet above). We might want to note that the "confidence" we are talking about here corresponds to the standards set by the <code>match_threshold</code> and ranges between 0-127. The measure of a point being above or below the <code>match_threshold</code> is currently a property of <strong>every</strong> single point, so if two points were, say, to be in a pair, both of their <code>best_dist</code> (confidence factors) would be the same.</li> <li>Completely remove all the video instances since we won't be working with any video formats whatsoever <em>inside</em> the LDI UI.</li> <li>Add support for training process to occur b/w images rather than a video and image.</li> <li>Calibrations and adjustments below can be employed to maximize ORB coverage while reducing outliers at the same time. But what exactly are these and why do we need them? <ul> <li><strong>Gaussian blurring:</strong> Reduces noise (outliers) by blurring the modified input <code>img_u8</code>(8-bit conversion). <img src="" alt="Screenshot from 2019-03-09 20-18-41" /></li> <li><strong>Grayscale method:</strong> Converting every pixel to a 0 (black) or 1 (white) value for way better performace and evaluation. <img src="" alt="Screenshot from 2019-03-09 20-31-14" /></li> <li><strong>Match threshold application:</strong> Specify matching standards (how high/low should the ORB expect the match instensity to be in order for it to be a good match). <img src="" alt="Screenshot from 2019-03-09 20-37-39" /></li> </ul> </li> </ul> <blockquote><p>The idea is to separate the image into two parts; the background and foreground. Select initial threshold value, typically the mean 8-bit value of the original image. Divide the original image into two portions; Pixel values that are less than or equal to the threshold; background.</p> <p>Based on this, we need to consider two more thresholds, which are available under the YAPE06 module, originally designed by CVLab, Switzerland.</p></blockquote> <ul> <li><strong>Eigen threshold application:</strong> Specify how similar two pixels should be in order for the ORB to perceive them as "lying in the same cluster". <img src="" alt="Screenshot from 2019-03-09 23-58-04" /></li> <li><strong>Laplacian threshold application:</strong> Defines the rate of change of intensity of the pixels that should be perceived by the ORB as "noisy". <img src="" alt="Screenshot from 2019-03-09 23-55-57" /></li> <li>Finally, we can modularize this technique into a custom PL module that accepts two images and runs the algorithm against them. Refer (d) to look over the current proposed steps we can take after this.</li> </ul> <p>c) <strong>Auto-Stitcher module:</strong> The auto-stitcher module will accept two images, the two most recently clicked images (as of now) and then then proceed to "stitch" those together as per the instructions passed on to it. We can, as an initial implementation, add a functionality to this module that allows the user to attach an image in any of the four major directions, after which the user themselves can make little changes to those images if need be. Since it wasn't merged at the time of writing this, I didn't implement this using the "multiple-image" selection feature, but am in the favor of shifting from the "last two latest-clicked images" to selecting any two images as per user's convenience and passing those down to the Stitcher, thus improving the module's overall accessibility. An <code>undo</code> function can be easily implemented as well using the recent <code>addTool</code> API of the <code>Leafet.DistortableImage</code> which triggers and adds an undo button to the menu bar every time the user <code>stitches</code> two images, and is removed upon undoing the stitch action. We can display a soft popup here to notify the user of the stitch since alert would be a bit "obstructing". I also believe that we should implement the <code>AS</code> module until after we have successfully abstracted away our polylines one, then using that build the <code>AS</code> on top of it.</p> <pre><code class="javascript">// an instance that "traces" the last two latest-clicked images var Tracer = L.DistortableImage.Tracer = L.DistortableImage.Tracer || {}; </code></pre> <pre><code class="javascript">initialize: ... //previous image (constructing param being {}) is available in this scope overlay.Tracer = Tracer; Tracer = overlay._corners; //current image that is clicked is available in this scope </code></pre> <p><img src="" alt="carbon" /></p> <p><a href=""><img src="" alt="autostitcher--road.gif" /></a></p> <p>d) <strong>Integrating (b) and (c):</strong> We can incorporate the ORB and Stitcher modules to work together in real-time in the MK UI. Whenever the user selects two images for comparison, the images are first passed down to the ORB module that displays all the relevant information about that particular pair of images in a modal. After this the user can select whatever pair fits best and choose to "stitch" them together in a user-defined orientation (right, left, top, bottom) with respect to the original image using the Auto-Stitcher.</p> <p><img src="" alt="modal" /></p> <p>That being said, I am well familiar with <a href="">Leaflet's polylines</a> and think that replacing <code>JSFeat</code>'s inbuilt methods with Leaflet's (<a href="/profile/warren">@warren</a>'s suggestion) shouldn't be a problem at all, since we already have the indices of all the good matches, which brings me to another very important aspect of this library, the {x,y} image space conversion to <code>LatLng</code> objects. This can be achieved by a <code>Scale function</code>, that takes in the {x,y} coordinates of the matched points, and encapsulates them to <LatLng> points. What I'm proposing here is, the extensive use of the following "scaling" formula, which can be used to convert the image space points to latitudes and longitutes, hence easily being able to pinpoint the desired location on the image using the Leaflet library, and thus being able to use those coordinates in a ton of different Leaflet APIs.</p> <p><img src="" alt="Scaling" /></p> <p>Converting image coordinates to <code>LatLng</code> object properties: For the "scaled" width we can, <code>x_image_to_lat = {|obj2.x-obj1.x|/image_width} * imagespace_x</code>. Similarly, for the "scaled" height we can, <code>y_image_to_lng = {|obj3.x-obj1.y|/image_height} * imagespace_y</code>.</p> <p><strong><a href="">Mapknitter UI enhancements</a></strong></p> <p>This is currently under progress, and most of the points in the <strong><a href="">projects and compatibility section</a></strong> have been or are getting resolved <a href=";q=is%3Apr+author%3Arexagod+">over in the PRs</a> which I've submitted till now. I do aim to incorporate the appropriate modifications to my remaining open PRs as per <a href="/profile/warren">@warren</a>'s suggestions and get them merged speedily and the remaining ones as well. You can checkout the follow-ups on the <a href="">original planning issue</a> as well.</p> <p>Also, based on <a href="/profile/warren">@warren</a>'s review, I think we can actually <strong>encapsulate all the "matching functions" in a different module, while constructing the UI (<code>AutoStitcher</code>+<code>Lines</code>+<code>Boxes</code>) using a different one</strong>. This would also make the testing part more convenient, and improve the resuseability of the components as well.</p> <h3>Timeline</h3> <ul> <li><strong>27th May to 23rd June: Introduce and integrate two new PL modules (w/ blog posts on both dates)</strong> <ul> <li><strong>Week 1 and 2: ORB and Auto-Stitcher modules (w/ FTOs on each weekend)</strong> <ul> <li>Starting off with the most exciting aspects of my proposal, the ORB and Auto-Stitcher modules will be implemented from scratch and solidified along the way (details above), while keeping in mind to make these effective yet simplistic, so that newer contributors who find this interesting can easily contribute to this. I will also be <strong>fully</strong> documenting both of these modules to make the code as readable and easily understandable as possible, especially the Oriented FAST and Rotated BRIEF (ORB) module. That being said, I, under the guidance of my mentor, will execute appropriate measures to refine these modules so that they perfectly fit in with Public Lab's coding standards, practices, and community expectations.</li> </ul> </li> <li><strong>Week 3 and 4: Integrate modularized ORB+AS and Microscope web-based live demo (w/ PR reviews on each weekend)</strong> <ul> <li>Although I've proposed a way to integrate ORB+AS into LDI above, I am and will be open to newer and better ideas that might be suggested by my mentor during my GSoC period. As is discernible, this will be a more work-focused period, and throughout this duration I will work on a <strong>modularized</strong> integration in order to maintain the readability of the codebase.</li> <li>The Microscope web-based demo will implement an abstract flavor of the ORB different from its native or LDI implementation. For this purpose, I'll be customizing the ORB almost from the bottom-up, to change its current behavior of depending on user to train itself whenever prompted from the video feed to completely discarding that approach and rather train itself from a specific image supplied to it, so that it could find the desired patterns in the live feed, as is demonstrated the <code>.gif</code> above. It may also be worth noting that this will be a PL standalone module, unless until suggested otherwise.</li> </ul> </li> </ul> </li> <li><strong>24th June to 28th June: Evaluations</strong></li> <li><strong>29th June to 21st July: Testing and Debugging Phase (w/ blog posts on both dates)</strong> <ul> <li><strong>Week 1 and 2: General Debugging at LDI's <a href="/n/87">#87</a> (w/ FTOs on each weekend)</strong> <ul> <li>This period will consist of different approaches to resolve the issues (todos, lower priority, and leftovers) over at LDI <a href="/n/87">#87</a>. These bugs have been in the system for a while now, and need to eradicated as soon as possible to build a neat codebase to attract more contributions in the future and less dangling errors in the future.</li> </ul> </li> <li><strong>Week 3: Unit Tests w/ Jasmine (w/ PR reviews on each weekend)</strong> <ul> <li>Initially proposing the opportunity to write extensive tests for the ORB, AS, and the Microscopic live-feed modules, but willing to write exhaustive unit tests for different potential LDI modular fragments, however my mentor sees fit.</li> </ul> </li> </ul> </li> <li><strong>22nd July to 26th July: Evaluations</strong></li> <li><strong>27th July to 25th August: LDI UI revamp (w/ blog posts on both dates)</strong> <ul> <li><strong>Week 1 and 2: LDI <a href="/n/126">#126</a> (w/ FTOs on each weekend)</strong> <ul> <li>During this period I will be looking forward to completely solving the remaining planning issues, i.e., finishing off the projects section and implementing the assorted utility functions, a step closer to the next major release of LDI!</li> </ul> </li> <li><strong>Week 3 and 4: Uncommon issues (w/ PR reviews on each weekend)</strong> <ul> <li>I will finish off my GSoC journey by working on some of the uncommon issues such as Cross-browser testing for the <code>Toolbar</code> module by generating heap snapshots across various combinations of OS and browser versions. I'm currently considering different ways to implement this but this method looks the most promising and solid. Other than this, I will work on implementing a couple of <a href="<a href="/profile/sarahelson81">@sarahelson81</a>/save-time-on-manual-cross-browser-testing-3772be756e68">unique approaches</a> that'll further guarantee the end-to-end functionality of PL's components and will also be easier for the new comers to quickly participate in! Also, I'll also look over the menu placement and image deletion (and specific ordering) issues which have been showing unexpected behaviour for a while now.</li> </ul> </li> </ul> </li> </ul> <h3>Needs</h3> <p>Only the guidance of my mentor, everything else that I require is either online, or on my local.</p> <h3>Contributions</h3> <p><strong>At the time of writing this down, my contributions have been as follows.</strong></p> <p><img src="" alt="Screenshot from 2019-03-10 02-41-12" /></p> <ul> <li>Pull Requests (53 total, 46 closed, 7 open): <a href="" title="">Navigate</a></li> <li>Issues -- FTOs (17 issues): <a href="">Navigate</a></li> <li>Issues -- Overall (31 issues): <a href="">Navigate</a></li> <li>Comments (on 148 issues): <a href=";q=commenter%3Arexagod+org%3Apubliclab&amp;type=Issues">Navigate</a></li> <li>Review Requests (19 PRs): <a href="">Navigate</a></li> </ul> <h3>Experience</h3> <p><strong>Comfortable with:</strong> C, CPP, JS, SH.</p> <p><strong>Achievements</strong></p> <ul> <li>Mentored at <a href="">KWoC</a> organized by KOSS, IIT Kharagpur, accessible <a href="">here</a> (2018).</li> <li>JS Core at <a href="">GDG JIIT Noida</a> (2019) and <a href="" title="">JIITODC Noida</a> (2019).</li> <li>Secured 1st rank at DevFest organized by <a href="">GDG JIIT Noida</a> (Google Developer Groups, now Developer Student Clubs) (2017).</li> <li>Secured 1st rank at Website Designing Competition (WDC) organized by <a href="">GDG JIIT Noida</a> (2017).</li> </ul> <h3>Teamwork</h3> <ul> <li>Interned towards the end of my first year as a Front-end developer at Profformance Technologies Pvt. Ltd. for their product, Xane, a React chatbot for HR purposes. The team strength wasn't much, about 25 people or so, and after a couple of months, I adapted to a work-from-home option to manage it with my studies, while visiting occasionally to attend meetings and other events. Throughout the duration of my internship here, I was introduced to the corporate etiquettes and productive practices that helped me realize the importance of time and priority management. I employed these techniques into my daily lifestyle to stay motivated and focused towards my project, as is discernable <a href="">here</a> too.</li> <li>I am a member of JavaScript core team at Google DSC JIIT. I've been teaching JS (Vanilla, React, and Node.js) to my peers for some time now which has helped me to stay updated with the latest API changes as well, while coordinating with rest of the team throughout.</li> <li>I also mentor a lot of enthusiastic students about OSS through JODC, working collaboratively with a lot of local tech societies like DSC-BVP, I take workshops and talks to bridge the gap between newcomers and development and spread awareness for FOSS. I am a teaching assistant here as well.</li> </ul> <h3>Passion</h3> <p><img src="" alt="Screenshot from 2019-03-10 02-37-11" /></p> <p>To start this off, I cannot stress how joyous I am when I realize that my code is actually improving the very surroundings that we live in! It's really an amazing feeling to "give back" to nature in such a unique way, which has become my passion and should be everyone else's too! Having said that, what interested me most about Public Lab's projects was the substantial, caring, and progressive community, which actively helped me clear away my doubts and learn new and interesting things throughout my journey with Public Lab, which I very much plan to extend for the many years to come!</p> <h3>Audience</h3> <p>My project will directly help all users (students, researchers, etc.) working in all kinds of fields that relate to studying, examining and experimenting with maps in many different ways possible, as described in various sections above. Also, it will help them recognize patterns that might often be "overlooked" by the human eye and thus extract credible information from live-video feeds, ranging from microscopic experiments to satellite broadcasts to cost-effective visual sensors for the visually impaired.</p> <p>The best thing about this is that all of it is open-sourced and free, so people from all over the world, no matter what their current social or financial status is, can use this to meet their needs, and even make it better!</p> <h3>Commitment</h3> <p>I do understand that this requires serious commitment, and since the day I joined Public Lab I've structured every single day to revolve around only two priorities, i.e., PL and college, actually in that order (I like it here!), which has enabled me to make about 185 contributions since mid-December! Also, I'd like to mention that I will, as I have done previously, continue to actively interact with the newer contributors and provide insight and any help that I can regarding their PRs and issues during (and after) my GSoC period. Hence, I firmly believe that I will deliver my assignments with commitment and promptness, given that I get selected!</p> <p><strong>Thank you!</strong></p>