<![CDATA[Rcd.AI]]>https://rcd.ai/https://rcd.ai/favicon.pngRcd.AIhttps://rcd.ai/Ghost 2.1Tue, 03 Sep 2019 22:08:37 GMT60<![CDATA[Deploying Your Data Science Projects in JavaScript]]>Pushing a React client, Python JSON API, and SSL certificate from Let's Encrypt to DigitalOcean in an hour

For my latest project, I decided to use React for most of my exploratory data analysis (EDA) and needed a drop-dead simple JSON API for serving up the necessary data to avoid

https://rcd.ai/quick-deploy-react-visualization-python-json-api/5c9ce592dd54971c616d57cfThu, 28 Mar 2019 23:57:31 GMTPushing a React client, Python JSON API, and SSL certificate from Let's Encrypt to DigitalOcean in an hourDeploying Your Data Science Projects in JavaScript

For my latest project, I decided to use React for most of my exploratory data analysis (EDA) and needed a drop-dead simple JSON API for serving up the necessary data to avoid loading a +70 MB page. In this tutorial, I'll walk you through the process of deploying a sample app with create-react-app, fastapi, and Nginx on DigitalOcean. You can explore the real app as it currently stands at https://ecce.rcd.ai. The production deploy process for this is a bit manual, but it certainly could be automated if needed.

Here are the top-level steps to make this happen:

  1. Create the Python API with fastapi
  2. Create the client app with create-react-app (and pass requests to the API)
  3. Create DigitalOcean droplet and install dependencies
  4. Configure Nginx (to unify React and Python as single web service)
  5. Build client app and copy to droplet
  6. Clone client API and install as cronjob
  7. Setup domain and obtain SSL certificate from Let's Encrypt
  8. Grab a coffee ☕ for a job well done! 🎉

Python API with fastapi

As far as I know, fastapi is pretty much the easiest, fastest way to create a Python API. It's a one file server and can be easily expanded and configured as necessary. For this example, we'll be serving up verses from the ESV (legally permissible since this a non-commercial project). We'll start the project from scratch with these commands:

# Create directory and jump into it
mkdir -p fastapi-react-demo/app
cd fastapi-react-demo

# Download data source
curl -LO https://github.com/honza/bibles/raw/master/ESV/ESV.json
mv ESV.json app/esv.json

# Install dependencies
pip install fastapi uvicorn toolz

# Save dependencies for later use
pip freeze | grep "fastapi\|uvicorn\|toolz" > requirements.txt

# Create server file
touch app/server.py

In app/server.py:

from fastapi import FastAPI
from starlette.middleware.cors import CORSMiddleware
import json
import os
from toolz import memoize

app = FastAPI()

# TODO: Change origin to real domain to reject Ajax requests from elsewhere
app.add_middleware(CORSMiddleware, allow_origins=['*'])

def data():
    with open(os.path.join(os.path.dirname(__file__), 'esv.json')) as f:
        return json.load(f)

def load_text(book: str, chapter: int, verse: int):
        return {'text': data()[book][str(chapter)][str(verse)]}
    except KeyError as e:
        return {'error': str(e), 'type': 'KeyError'}

From this simple file, we have a JSON API, CORS, parameter validation, and so much more. Let's go ahead and run this in development mode to check things out:

# Start app
uvicorn app.server:app --reload

# Send API request (in separate window)
curl http://localhost:8000/api/verse/Genesis/1/1
# => {"text":"In the beginning, God created the heavens and the earth."}

With the server running, we not only have the JSON API fully operational in development, but it even live reloads if we change anything in that file. Let's move onto the client.

Client app with create-react-app

With the API running in development mode, let's go ahead and create a client app. I am going to assume familiarity with the JavaScript ecosystem if you're doing data science visualizations in React.

npx create-react-app client

Before adding our ajax code, we'll need to configure the proxy for create-react-app to forward requests it doesn't handle to our API server running on port 8000.

In client/package.json

  "proxy": "http://localhost:8000",
  "scripts": ...

We'll go ahead and do a simple API call from the client:

In client/src/App.js

import React, { Component } from 'react';

class App extends Component {
  constructor(props) {
    this.state = { verse: 'Loading...' };

  componentDidMount() {
      .then(r => r.json())
      .then(data => this.setState({ text: data.text }));

  render() {
    return (
        <p>Result of API call: {this.state.text}</p>

export default App;

With that file updated, we can go ahead and start up the server.

cd client && yarn start
Deploying Your Data Science Projects in JavaScript
Result of the most amazing visualization app... ever.

Hurray! It's certainly not going to win any design awards, but we now have the two apps talking to each other. Let's talk about how to actually deploy these apps.

Make sure you have pushed this up to a Git repository that we can easily clone later. (If not, don't worry–you can just use the sample repository at https://github.com/rcdilorenzo/fastapi-react-demo).


With the app up and running, we now have to deploy our code for the world! We'll start by creating a DigitalOcean droplet. In this case, I'll choose the smallest disk size with Ubuntu 18 to serve the application.

Deploying Your Data Science Projects in JavaScript
Configuring droplet options

With the SSH keys in place, we can go ahead and jump into the droplet described in the settings.

Deploying Your Data Science Projects in JavaScript
ssh root@<DROPLET_IP xxx.xxx.xxx.128>

# (on server)
apt-get update
apt-get install -y nginx-full

Configure Nginx

We now need to setup Nginx to accomplish three items.

  1. Serve Python app whenever route starts with /api
  2. Fallback to React app for any specific asset or if a route isn't found
  3. Force SSL and use certificate from Let's Encrypt

I'm not going to jump through all of the details of this configuration, but it should accomplish our goals and you learn more about the details later. For editing, go ahead use your favorite command line editor. (Mine is vim for quick edits; easiest is nano.) Be sure to replace demo.rcd.ai with your custom domain.

In /etc/nginx/sites-available/demo.rcd.ai.conf

server {
    listen [::]:80;
    listen 80;
    server_name demo.rcd.ai;
    location / {
        return 301 https://$host$request_uri;
server {
    listen [::]:443 ssl http2;
    listen 443 ssl http2;
    server_name demo.rcd.ai;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ssl_certificate /etc/letsencrypt/live/demo.rcd.ai/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/demo.rcd.ai/privkey.pem;

    root /var/www/demo/;

    index index.html;

    location / {
      try_files $uri $uri/ /index.html;

    location /api {
      proxy_pass http://localhost:1234; # Port of Python server

    # JavaScript/CSS
        try_files $uri =404;
        expires 1y;
        access_log off;
        add_header Cache-Control "public";

    # Any file route
    location ~ ^.+\..+$ {
        try_files $uri =404;

Let's also symlink this file into the sites-enabled folder so that Nginx will know to serve it when receiving requests from the domain demo.rcd.ai.

ln -s /etc/nginx/sites-available/demo.rcd.ai.conf /etc/nginx/sites-enabled/demo.rcd.ai

Although we could go ahead and tell Nginx to reload, we still have a couple of things remaining before it will work properly.

  1. Add demo.rcd.ai with our domain registrar to point to this IP address
  2. Copy the production build of the client to /var/www/demo
  3. Clone the Python server and start it on port 1234
  4. Request Let's Encrypt certificate

I won't go too much into the details of setting the A record of a subdomain as it depends on your provider. Here's what I added to NameCheap since that's my registrar.

Deploying Your Data Science Projects in JavaScript
Add an "A Record" to point to the droplet

Note that this may take a while before the change is reflected across the "interwebs." You can always check it with a simple ping:

❯ ping demo.rcd.ai
# PING demo.rcd.ai ( 56(84) bytes of data.
# 64 bytes from ( icmp_seq=1 ttl=48 time=88.4 ms
# 64 bytes from ( icmp_seq=2 ttl=48 time=85.4 ms
# ^C
# --- demo.rcd.ai ping statistics ---
# 2 packets transmitted, 2 received, 0% packet loss, time 1000ms
# rtt min/avg/max/mdev = 85.411/86.921/88.432/1.539 ms

Build and Deploy Client

Back in the client/ folder on the development machine, creating a production build is as easy as issuing a single command. We'll then package it up and send it to the DigitalOcean droplet in the /var/www/demo directory.

# Build production-optimized assets
yarn build

# Zip contents
zip -r build.zip build/

# Upload with scp
scp build.zip root@<DROPLET_IP>:/var/www/

Then on the droplet:

# Unzip and rename folder
apt-get install -y unzip
cd /var/www && unzip build.zip
mv build demo

At this point, Nginx should serve the assets properly. However, because the certificate isn't in place yet, it won't allow us to even load the configuration.

SSL Certificate from Let's Encrypt

Although we could setup the Python server, it would be nice to get some feedback to make sure at least our client is being served properly. In the spirit of quick iteration, let's setup that certificate first. Instructions are easily obtainable from https://certbot.eff.org. On the droplet, go ahead and install certbot.

apt-get update
apt-get install software-properties-common
add-apt-repository universe
add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install certbot python-certbot-nginx 

From here, the certificate can be obtained via a single command.

certbot certonly --standalone \
  --pre-hook "service nginx stop" \
  --post-hook "service nginx start" \
  --preferred-challenges http -d demo.rcd.ai
# ...
# Running pre-hook command: service nginx stop
# Obtaining a new certificate
# Performing the following challenges:
# http-01 challenge for demo.rcd.ai
# Waiting for verification...
# Cleaning up challenges
# Running post-hook command: service nginx start
#  - Congratulations! Your certificate and chain have been saved at:
#    /etc/letsencrypt/live/demo.rcd.ai/fullchain.pem
#    Your key file has been saved at:
#    /etc/letsencrypt/live/demo.rcd.ai/privkey.pem
#    Your cert will expire on 2019-06-26. To obtain a new or tweaked
#    version of this certificate in the future, simply run certbot
#    again. To non-interactively renew *all* of your certificates, run
#    "certbot renew"
#  - Your account credentials have been saved in your Certbot
#    configuration directory at /etc/letsencrypt. You should make a
#    secure backup of this folder now. This configuration directory will
#    also contain certificates and private keys obtained by Certbot so
#    making regular backups of this folder is ideal.
#    ...

Because this command stops and restarts Nginx, we should be able to go directly to the app and see something (https://demo.rcd.ai). We can also specifically request Nginx to reload the configuration.

nginx -s reload
Deploying Your Data Science Projects in JavaScript
Client is working... now onto the Python API!

Deploy the API

With the Nginx configuration expecting an internal server listening on port 1234, all that remains is running the Python server on the droplet. Because we pushed the code to GitHub, the remaining commands can be run from the droplet.

# Clone code to folder
cd /var/www
git clone https://github.com/rcdilorenzo/fastapi-react-demo.git demo-server

# Install Python-3.6 based virtualenv (to avoid version conflicts)
apt-get install -y python3.6-venv python3-venv python3.6-dev

# Jump into server folder
cd demo-server

# Create virtual environment in /var/www/demo-server/demo_3.6
python3.6 -m venv demo_3.6

# Install a couple of prerequisites for compiling some dependencies
./demo_3.6/bin/pip install wheel
apt-get install -y gcc

# Install dependencies
./demo_3.6/bin/pip install -r requirements.txt

With the dependencies installed, the server can be started inline for testing.

./demo_3.6/bin/uvicorn app.server:app --port 1234
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
INFO:uvicorn:Started server process [9357]
INFO:uvicorn:Waiting for application startup.
INFO:uvicorn:Uvicorn running on (Press CTRL+C to quit)

Revisiting the page, we can see that the verse now loads properly.

Deploying Your Data Science Projects in JavaScript
Our working application - React, Python, and Nginx

However, this Python server will only run while we have the SSH session open. To have it actually persist even across reboots, we can add in to the crontab. The only command we have to run is changing to the server root directory and running uvicorn. Kill the server with CTRL-c and open the crontab with crontab -e.

@reboot cd /var/www/demo-server && ./demo_3.6/bin/uvicorn app.server:app --port 1234

Save and close that file. Restart the server with reboot. Once the droplet restarts, the application should be automatically started.


If you have come this far, then congratulations. 🎉 Go grab a coffee! This process may seem a bit tedious, but it shows that for only $5 a month you can have a production-level, interactive web app of your data project without forcing the data science process through a more narrow set of visualization tools. This level of flexibility certainly won't be necessary in many cases, but now you know how to set everything up for yourself.

There's much more that could be said on each one of these steps, but I wanted to focus on the primary workflow. Of course, updating the client or server requires a couple of commands, but that could easily be automated. If you really want a long term, production-ready system, deploying a CI/CD process with a team is certainly worth the time investment.

<![CDATA[Speed Up Your Data Processing with Macros]]>https://rcd.ai/speed-up-your-data-processing-with-macros/5c910dfefd84a81d842a03c8Tue, 19 Mar 2019 20:51:00 GMT

Data engineers and data scientists are all about data. Often, however, it's those Linux guys that are really good at transferring data formats with their cool utilities like awk, sed, tr, and nc to mention a few. When it comes to preprocessing, data scientists can jump a bit too quickly to a full blown script in their favorite language (Python, R, Julia, etc). I have been recently working on a project that requires importing many Biblical references, and I was able to employ a quick Vim macro to quickly transform some structured text into a Python dictionary that I could use for preprocessing references.

Here's how it works. On that first line, I begin at the first character and initialize a macro to the a key using q a. Because some of the books have spaces in them, I need to keep those in mind when I use a set of keystrokes that get replayed with the macro.

| Ge     | Genesis         |

What I need to do here is surround the "Ge" with quotes and not assume I can jump by spaces. Therefore, I'll use d W to delete until the "G", insert a quote, then jump to the pipe with f | and get to the end of the "Ge" using b (back a word) and e (end of the word). At that point, I can insert another quotation mark and delete until the pipe d t |. I can then replace that pipe with a colon r :.

For the expanded book, I can take a similar approach. I'll jump to beginning of "Genesis" with w, insert a quote, jump to the final pipe f | and then get to the end of the final word (in this case just "Genesis") with b e. Then, I insert a quote and delete to the end of the line D. To prepare for the macro being run over and over, I'll also jump back to the beginning of the line and go to the beginning of the next line with ^ j.

Phew! With the movements completed, I press q to stop the macro recording and save it off to a. With that complete, I can then simply tell it how many times to repeat the macro. In this case, I repeated it 71 times with 7 1 @ a. As you can see, the magic immediately begins. Afterwards, I just clean up a couple of the books I don't want, and add some curly brackets to make it into an actual Python structure.

This process may seem slightly tedious, but I can promise that if you practice using a tool like Spacemacs (or plain Vim) you'll be ready for these tasks. It'll save you plenty of time just fiddling around with open, map, trim, and the other Python commands you might need. If you're interested in installing vim, I've got a one-line installation (macOS and Linux) that gets you setup with most of the things you need: https://github.com/rcdilorenzo/vimfiles. I actually use Spacemacs (http://spacemacs.org) for my everyday work, but the same Vim commands I've discussed will work the same.

Happy data mining!

<![CDATA[Data Science from Concept to Production: A Case Study of Automatic Building Footprint Segmentation]]>https://rcd.ai/abfs-project/5c7e8bfc2b3b847ad2f2b2dfThu, 07 Mar 2019 21:07:19 GMT

Photo by Ahmet Sali on Unsplash


As satellite imagery has improved over the past ten years, applications of aerial images have dramatically increased. From crop yield to war devastation, these images can tell a detailed story even from a single snapshot. Buildings are perhaps the most common landscape feature in aerial imagery.

Automatic prediction of a building footprint can give rough estimates for both specific and general applications. In general, insights about house size and population density enable broad cross-section comparison of regions. More specifically, determining roof area from a particular building footprint enables a rough estimate of potential work without a tedious outlining process of that building.

Using Deep Learning and Computer Vision techniques, aerial imagery can be analyzed to automatically predict or augment structure annotations in both form and area. As part of my M.S. in Data Science from Regis University, this project employs aerial imagery to automatically predict building footprints from a given latitude and longitude.

SpaceNet Data

Started last year, the SpaceNet challenge open sourced a set of manually labeled building footprints over five regions that includes visible spectrum (3-band), 8-band, and off-nadir tiles. In pursuit of an entire data science pipeline from raw annotated data to production-level API, I have focused on using only the tiles that include the visible spectrum in order to use a global tile-provider for the API.

Due to the 8-week timeline of this entire project, I employed the +250,000 building annotations found in the SpaceNet Rio de Janeiro data set that covers over 2,500 square kilometers. Since they stored the information in a public AWS S3 storage bucket, the 3.4 GB of data was easily downloadable for local exploration of the links, formats, and imagery.

Production-Oriented Approach

Before jumping into the project details, a simple elaboration of my approach will help clarify later discussion. Although my masters focuses on the technical aspects of modeling and data engineering, I am also deeply interested in how to build machine learning (ML) systems end-to-end from data to production since I work for a software firm. Therefore, my entire process has centered around reproducibility, maintainability, and interoperability.

For reproducibility, I have written automated unit tests for most of the core system including data extraction, train/val/test splits, training augmentation, and batch options. This value also extends to the top-level documentation, naming conventions, and accessibility through the command line interface. I have written most of the code to be operable from the abfs command with tasks such as train, evaluate, and export. For the several Jupyter notebooks employed, I have extracted most of the code to external files that can be tested to keep the notebooks as easy to read and up-to-date as possible.

Towards maintainability, I have kept as much as possible to industry code review standards of domain modeling often found in object-oriented systems, in order to facilitate extensions by others and machine learning model changes without rework most of the existing system. For example, when making an polygon API request, it goes through a logical progression of subsystems:

 -> cli                     # CLI functions
 -> api.server              # Web server entry
 -> api.polygon_prediction  # Top-level handler
 -> api.prediction.lat_lng  # Prediction for coordinates
 -> api.prediction.image    # Prediction from image
 -> keras.Model             # Raw Keras model prediction

For interoperability, I have attempted to keep pieces as independent as possible without compromising functionality. Extensions should be easily achievable as well as integration of the JSON API into any language. Likewise, it should help to define some best practices for next projects. This emphasis on interoperability is motivated by my desire to help standardize the data science process for myself, my co-workers, and the community at large.

Data Preparation

SpaceNet Format

Diving into the Rio data set, all of necessary file references are contained in a single structure called DataConfig. The primary summary file contains the polygons and identifiers to link directly to the processed image tiles. Below is a sample of the first five entries. (See this link for a larger sample.)

ImageId BuildingId PolygonWKT_Pix PolygonWKT_Geo geometry
0 AOI_1_
1 ...((408.210... ...((-43.541... ...Z ((-43.541...
1 AOI_1_
2 ...((389.833... ...((-43.541... ...Z ((-43.541...
2 AOI_1_
3 ...((242.119... ...((-43.542... ...Z ((-43.542...
3 AOI_1_
4 ...((311.733... ...((-43.542... ...Z ((-43.542...
4 AOI_1_
5 ...((350.582... ...((-43.542... ...Z ((-43.542...

Although some utilities are available from SpaceNetChallenge/utilities, I wanted to understand the data format fully and specifically use only the 3-band (visible spectrum) imagery. After an examination of the geocoded information in those three columns, I found that the PolygonWKT_Pix column is needed to assemble training examples from the tiles. Despite working with GeoJSON previously, I discovered from the utility code that this data is in a Well-Known Text (WKT) format.

As seen in the data folder structure below, the ImageId column corresponds with the TIF images in the 3-band folder. Because the summary CSV contains the WKT polygons, the individual GeoJSON feature files in the vectordata/geojson folder are not required to assemble training examples.

└── processedData
    └── processedBuildingLabels
        ├── 3band [6940 entries]
            ├── 3band_AOI_1_Rio_img1000.tif
            ├── 3band_AOI_1_Rio_img1001.tif
            └── ...
        ├── 8band [6940 entries]
        └── vectordata
            ├── geojson [6940 entries]
            └── summarydata
                ├── AOI_1_RIO_polygons_solution_3band.csv
                └── AOI_1_RIO_polygons_solution_8band.csv

Additionally, the TIF images include geospatial data that link them directly to a WGS 84 Mercator projection (typically used in GPS). This metadata is extremely important as it will allow converting to and from geographic coordinates when performing pixel analysis.

❯ gdalinfo 3band_AOI_1_RIO_img1000.tif 
Size is 438, 406
Coordinate System is:
        SPHEROID["WGS 84",6378137,298.257223563,
Corner Coordinates:
Upper Left  ( -43.7125239, -22.9499059) ...


Because the goal here is more than just drawing a bounding box around buildings, an image segmentation approach is critical for determining an exact polygon. For a given pixel, therefore, the model should predict whether it is part of a building. For preprocessing, pixel-based predictions mean that the polygons need to be "stamped" into a single image to create a mask of black and white pixels where the white pixels indicate the buildings.

Creating these pixel masks from the polygons was perhaps one of the most difficult steps of the preprocessing due in part to the Python to C++ interop with osgeo and the lack of internet resources for how to use the Python library properly. For example, simply refactoring the method into multiple calls often caused a segmentation fault (not to be confused with image segmentation) because C++ had released the memory despite Python still needing it to exist.

Image Masks from WKT Polygons

Exploratory Data Analysis (EDA)

With the initial data preprocessing in place, I explored the data a bit further in a Jupyter notebook. Having imported the PolygonWKT_Geo data as shapely objects, the area of each building is calculated via a degrees to feet projection. Using a few matplotlib and seaborn commands, a visualization of the area distribution reveals a healthy cross-section of the data.

Logarithmic Distribution of Building Area (>= 10 sq.ft.)

From these graphs, the median of 227 square feet is seen in the smallest building category. As it turns out, 98.1% of the buildings are under a thousand square feet. To determine how these actually appear relative to the tiles, labels overlaid onto the aerial imagery for buildings from several categories give a decent sample of the proportions.


Because the EDA forced a calculation of building area, I opted to model only on the tiles with buildings (>= 0 square feet) since the intended use of the API is for a requested building location.


Before actually writing the neural network, I still had several tasks to complete. Keras expects to be handed batches of inputs and outputs (as numpy arrays) unique to the given test split (train/validation/test). To this end, I updated the Data class and and created a generic GroupDataSplit class to manage data splits and create data frames for each phase (train/val/test). Because the Keras generator worked extremely well for previous projects, I created an extension to this system with a class called Generator that inherits from keras.Sequence and simply provides the interface expected. I also implemented segmentation augmentation that optionally creates "new examples" by rotating and scaling the tiles and the associated labels.

With all of this behavior unit-tested and ready for prime time, the model setup becomes extremely easy to discern.

# Use Rio 3band data

# Train (70%), Val (10%), Test (20%), Fixed seed
split_config = DataSplitConfig(0.1, 0.2, 1337)

# Create data with batch size
data = Data(config, split_config,

# Exclude empty polygons
data.data_filter = lambda df: df.sq_ft > 0

# Create model
# ...

These are just some of the features that have now been enabled:

  • Consistent split of data (doesn't change between runs)
  • Train data randomization
  • Train data augmentation
  • Filtering data examples
  • Scaled inputs
  • Configurable batch size
  • Configurable input data image size
  • Keras integration

U-Net & Training Process

For the modeling process, I was eager to experiment with fully-convolutional neural networks, particularly the U-Net architecture. After a bit of tweaking and discussion with my professor, I added layers through the training process until I reached my 5-level deep U-Net with 31,032,837 parameters and frequent 25% dropout layers. I experimented with different hidden activation functions, kernel initializers, and fewer layers. With this architecture, the highest level features are max-pooled into a 32x32 filter before being up-sampled to the original size after being combined with the relevant skip connections.

(Click to view full network)

For network input and output, I began with a 512x512x3 input and predicted a 512x512 mask of building probabilities by pixel using a sigmoid activation with an SGD (stochastic gradient descent) optimizer. As will be later seen, these decimal values will need to be converted to polygons for actual use from an external application. For the convolutional layers, I used an he_normal kernel initializer for the initial weights, which samples from a truncated gaussian distribution.

To facilitate the training process, I employed a TensorBoard callback that plots the network loss, metrics, and even a custom image visualization of validation set examples. This visualization was a key element from one of my computer vision projects last year that revealed a Keras issue only late in the semester because I didn't put a visualization together earlier. Although it wasn't 100% effective, it did give a bit of sanity check to the network training before the official evaluation step.

Training visualization from a intermediary iteration

As seen above, the green corresponds to where the pixels were correctly predicted, the red to where it mistakenly predicted a building, and blue to where it missed an actual building. Combining this visualization with TensorBoard's scalar graphs, I had good insight into the training progress (even remotely).

Viewing training and validation loss in TensorBoard for two models

For metrics, I attempted several variations of binary mean intersection-over-union statistic. After writing my own as well as trying to the TensorFlow implementation, I eventually abandoned it in favor of the SpaceNet metric of an F1-score for my final evaluation. However, by monitoring the binary cross-entropy loss function in tandem with the image validation in TensorBoard, I was still able to accurately experiment with the network hyperparameters.

TensorBoard (Comparison of similar networks with vastly different epoch sizes) 

Through the training process, I experimented with different epoch sizes. Although the training data was both shuffled and augmented during each epoch, training on the entire 755 batches (8 per batch) for a single epoch more quickly achieved convergence. This experiment was made possible by training across two 1080 Ti's. During the course of the project, I installed liquid cooling to keep the temperatures down on both GPUs (from 91°C to 69°C for training).

Custom liquid cooling for 2-GPU training (See https://youtu.be/Qdmf7I5BPtk)

Because of my background in software engineering, I knew that my biggest opportunity for learning would be during the modeling process. To this end, I created a CLI for the basic training in order to more quickly experiment with various hyperparameters.

❯ abfs train -h
Using TensorFlow backend.
usage: abfs train [-h] [-lr LEARNING_RATE] [-s SIZE] [-e EPOCHS]
                  [-b BATCH_SIZE] [-mb MAX_BATCHES] [-gpus GPU_COUNT]

optional arguments:
  -h, --help            show this help message and exit
  -lr LEARNING_RATE, --learning-rate LEARNING_RATE
  -s SIZE, --size SIZE  Size of image
  -e EPOCHS, --epochs EPOCHS
  -b BATCH_SIZE, --batch-size BATCH_SIZE
                        Number of examples per batch
  -mb MAX_BATCHES, --max-batches MAX_BATCHES
                        Maximum batches per epoch
  -gpus GPU_COUNT, --gpu-count GPU_COUNT

❯ abfs train -lr 0.02 --batch-size 8 --epochs 150 -gpus 2



According the example SpaceNet implementation for building extraction from the Rio data set that I found after my modeling, they achieved an F1-score of 0.729 on the entire public data set. My raw F1-score was 0.692. However, there are several key differences that distinguish my work.

  1. My model was trained using only the 3-band imagery. The example uses both 3-band and 8-band imagery.
  2. My model employed with a 60-10-20 split (60% train, 10% validation, 20% test). The example was evaluated on the entire public data set (and does not state what it was trained on).
  3. My model filters out tiles with no buildings. The example does not seem to filter out any examples.

However, mu final F1-score metric is not directly calculated from the network training process. This raw F1-score is a combination of the probabilities. To actually produce polygons, the model must use a threshold to find the relevant contours in the mask image.

Tune Tolerance

Although the threshold tolerance is not directly implemented in the neural network, it is still a parameter that must be tuned. Using the validation data, the output masks of varying grey-to-white pixels must therefore be converted at different tolerances (1 - threshold) to binary white/black pixels. Once converted, each sample can then be assessed with the F1-score metric. However, it is not simply enough to choose the highest median F1-score. By visualizing these distributions as a series of box plots (see figure below), the best tolerance can be manually determined as well as programmatically found via a combination of high median and low standard deviation.

Tuning the tolerance parameter (best is 0.70 on the x-axis)

As seen in the tuning notebook, this calculation can be quite computationally expensive. I started with a CPU implementation that was going to take about an hour and cut it down to to 7.4 seconds by writing it in TensorFlow using CUDA. Below is a sample run on the selected model.

❯ abfs tune -w checkpoints/unet-f63fce-0024-0.19.hdf5 -gpus 2
Calculating F1-Scores... This may take perhaps even an hour if no GPU.
F1-Score calculation complete: 7.39 seconds
Plot has been saved to /tmp/tmp_xjeqphr.png. Please open to view.
Tuned tolerance: 0.70 w/ median=0.6974 stdev=0.1722

From this tuning process, the final F1-score was 0.6974.

Building a GeoJSON API

With the model built, I crafted a pyramid web server with two API endpoints–one that returns the raw masked image with the varying gray pixels and the other one that uses the tuned tolerance parameter to create and return GeoJSON polygons.

Using the polygon GeoJSON API (with Paw for Mac)

As previously mentioned, my goal was to swap out the tile provider from SpaceNet to one that I could map directly from latitude and longitude. To this end, I employed MapBox since I could easily stay with their free limits for testing this project.

Since the focus of this project is not primarily the software engineering, I will avoid details of the pixel-to-coordinate projection mapping and GeoJSON creation. Like the other parts of this project, the server is easily accessible and configurable from the CLI.

❯ abfs serve -h
Using TensorFlow backend.
usage: abfs serve [-h] [-w WEIGHTS_PATH] [-ws3 WEIGHTS_S3] [-m MODEL_PATH]
                  [-ms3 MODEL_S3] [-a ADDRESS] [-p PORT] [-mb MAPBOX_API_KEY]
                  [-t TOLERANCE]

optional arguments:
  -h, --help            show this help message and exit
  -w WEIGHTS_PATH, --weights-path WEIGHTS_PATH
                        Path to hdf5 model weights
  -ws3 WEIGHTS_S3, --weights-s3 WEIGHTS_S3
                        S3 path to hdf5 model weights
  -m MODEL_PATH, --model-path MODEL_PATH
                        Path to keras model JSON
  -ms3 MODEL_S3, --model-s3 MODEL_S3
                        S3 path to keras model JSON
  -a ADDRESS, --address ADDRESS
                        Address to bind server to
  -p PORT, --port PORT  Port for server to listen on
  -mb MAPBOX_API_KEY, --mapbox-api-key MAPBOX_API_KEY
                        Mapbox API key
  -t TOLERANCE, --tolerance TOLERANCE
                        Default tolerance parameter (output from tune)

Deployment to Production

Using environment variables in tandem with this CLI, I set out to create an entire CI/CD pipeline. For unit testing, I created a public project on Semaphore CI to automatically test for code regressions. After many failed production attempts (or excessively expensive options) with AWS Lambda, Heroku, and a Python environment on AWS Elastic Beanstalk, I succeeded with a Docker container that runs the necessary environment in Elastic Beanstalk.

Although I am sure that most Python projects with work with the stock Python environment, this project required certain versions of C-based libraries to be installed for the geospatial calculations. For the minimum processing requirements, I used a t2.medium EC2 instance that returns the API results in about 11 seconds. For reference, my GPU-build performs it in 900ms and my Mac (no CUDA) returns in 4 seconds on its i9 6-core CPU. For $33 a month, the t2.medium instance was a reasonable option for testing this public API.


The last eight weeks has held a significant amount of learning. From a data science pipeline perspective, I was able to go from raw data to production. Although the model could certainly be tuned further, model improvements should not require  a significant amount of change to the system even if an entirely different network is selected.

In consideration of further efforts, one of the elements that the model could not take into account is different environmental settings. When, for example, I ran it in my home state of North Carolina, the heavy vegetation seemed to throw it off. My preliminary theory is that model's data source of Rio looks significantly different from an aerial perspective with its commonly dusty streets, packed houses, and little vegetation. Solving this problem would likely require artificial augmentation (adding trees and masking obscured labels) or tiles from a different region. Additional work in this direction should significantly improve the model's ability to generalize.

Considering the network itself, I only had the time to explore the U-Net architecture. Pursuing a different approach with perhaps an auto-encoder or XGBoost would likely be helpful. Of course, ensembling these models would likely improve the accuracy. Aside from the model improvements, performance optimization would be a significant prerequisite to employing it in a client project or as an external API.

If you've read this far, then congratulations! I appreciate the interest. If this particular project seems beneficial to you or a project of yours, feel free to drop me a note at rcddeveloper@icloud.com or checkout the entirety of the source code at github.com/rcdilorenzo/abfs. 🌟 Please star the repository to show your appreciation! 🎉

<![CDATA[Transfer Files Using Netcat]]>nc is an excellent command line tool available on *nix systems. One of the quick tips is using it to transfer from one machine to another. Assuming your receiving machine is at and your sending machine is, we can use the following

https://rcd.ai/transfer-files-using-netcat/5c64c2353abcdb0d4b428c1bThu, 14 Feb 2019 01:24:59 GMTnc is an excellent command line tool available on *nix systems. One of the quick tips is using it to transfer from one machine to another. Assuming your receiving machine is at and your sending machine is, we can use the following to stream it using a socket (using 1337 as an arbitrarily chosen yet available port).

# on receiving machine
nc -l 1337 > my-large-file.sql

# on sending machine
nc 10.0.02 1337 < my-large-file.sql

Likewise, we can actually flip this in case it's easier to tell a co-worker to execute the command after you have a stream ready to send the file.

# on sending machine
nc -l 1337 < my-large-file.sql

# on receiving machine
nc 1337 > my-large-file.sql
<![CDATA[Testing for the Data Scientist]]>https://rcd.ai/testing-for-the-data-scientist/5c185d2eab98076cb2f92b9dWed, 19 Dec 2018 14:22:27 GMT

Data collection, modeling, and analysis. These are tasks that the modern data scientist is expected to perform with statistical precision and ethical decision making. The code we write is not for accomplishing tasks that are easily verifyable like, "As a user, I want to login to my account using GitHub." As data science begins to mature, more and more articles are being written about ensuring proper behavior for complex machine learning systems.

As I delve into data science more myself, I am surprised at the unfortunate inattention paid to the lessons already learned by software engineers, particularly around testing and clean code. For those unfamiliar with tests, I simply mean automated methods that verify behavior in a given piece of software–whether that be a calculator app or an AI-based intrusion detection system.

Think about this. Data scientists, ML engineers, and data engineers are expected to often write more logically difficult code than the average developer and yet rarely spend time writing tests to ensure proper behavior. It simply doesn't make sense. As Edmund Burke famously said, "Those who don't know history are doomed to repeat it."

Lightning Introduction to Unit Testing

(Feel free to skip to the next section if you're already familiar with unit testing.)

What are unit tests? These are automated tests that focus on ensuring proper behavior for a given set of functions or methods within a given context such as an OOP class. Generally speaking, each test assertion is a simple, easy-to-read description of a function's output given a particular input. Here's a ridiculously simple test in Python of an add function:

# In calculator.py
def add(x, y):
    return x + y

# In calculator_test.py
from calculator import add

def test_addition():
    assert add(1, 2) == 3
❯ pytest calculator_test.py

calculator_test.py .                                                     [100%]


Case Study: A Tensorflow Loss Function

Those involved in Deep Learning and more specifically Tensorflow will know that building up computational graphs is not terribly easy. Providing a differentiable loss function for backpropogation can sometimes require direct calls to the Tensorflow API. I recently had to write an intersection-over-union (IoU) function that compared a set of predicted versus actual bounding boxes as opposed to the built-in mean_iou function that compares bounding boxes row-by-row. Here's an example of the algorithm using three sample bounding boxes:

Testing for the Data Scientist
Visualization of Spatial Intersection-over-Union (IoU)

Apparently, this type of loss function is far from standard. Because this determines the entire process of training, it was essential for the calculations to be accurate. To this end, I did some calculations by hand to make sure I got an expected result (as seen in the poster image). Therefore, the goal of the automated test is this:

Write a test that passes both predictions and actual values to the loss function and returns a tensor that evaluates to the proper IoU score.

To maintain proper isolation of testing a single loss function "unit," I take advantage of pytest's fixture syntax to have a sample of predicted and actual values for a single batch:

# In conftest.py
import pytest
import numpy as np

def sample_pred():
    return np.array([
        # x1, y1, x2, y2, prob
        [[ 32,  12,  46,  18, 0.7],
         [ 21,  10,  22,  30, 0.8],
         [100, 100, 100, 100, 0.2],
         [ 14,  15,  24,  44, 0.1],
         [200, 200, 200, 200, 0.2]]
    ], dtype=np.float32)

def sample_actual():
    return np.array([
        # x1, y1, x2, y2, prob
        [[33, 10, 46, 18, 0.8],
         [21, 10, 23, 31, 0.3],
         [ 0,  0,  0,  0, 0.0],
         [18, 13, 21, 41, 0.2],
         [ 0,  0,  0,  0, 0.0]]
    ], dtype=np.float32)

Unfortunately, the test cannot simply call the function like the calculator example. Loss functions with Tensorflow describe the GPU-operation but don't actually run them. In other words, the return value of this spatial_iou function is actually a Tensor, not a numerical result. Thankfully, Tensorflow has an tf.InteractiveSession that allows explicit evaluation of a Tensor with given inputs.

# In loss_test.py
def test_spatial_iou(sample_pred, sample_actual):
    sess = tf.InteractiveSession()

    # Mix up inputs (shouldn't be index dependent)
    pred_t = tf.constant(sample_pred[:, [0, 1, 3], :])
    actual_t = tf.constant(sample_actual[:, [1, 3, 0], :])

    # Pass image shape and tensors to loss function
    tensor = spatial_iou((50, 50), actual_t, pred_t)

    # Initialize `tf.Variable`s

    # Explicitly evaluate using the tf.InteractiveSession
    result = tensor.eval()

    # Result calculated by hand
    assert result == approx(0.463942308, abs=0.00001)

    # Close the interactive session

As seen in the comments, the test requires a little bit of setup. However, with this test written, I can now freely write the implementation with the confidence that I have a secondary check on my logic. Additionally, I now have an automated check in case a new Tensorflow API change necessitates changing my implementation or another programmer wants to understand the purpose of this loss function.

For reference, here is the final implementation:

import tensorflow as tf
import keras.backend as K
from toolz.curried import curry

def spatial_iou(image_shape, actual, pred):
    def _spatial_iou(tensor):
        actual, pred = tensor

        pred_empty_canvas = tf.Variable(lambda: tf.zeros(image_shape))
        actual_empty_canvas = tf.Variable(lambda: tf.zeros(image_shape))

        pred_masks = tf.equal(tf.map_fn(mask_true(pred_empty_canvas), tf.round(pred)), 1)
        pred_canvas = tf.reduce_any(pred_masks, axis=0)

        actual_masks = tf.equal(tf.map_fn(mask_true(actual_empty_canvas), tf.round(actual)), 1)
        actual_canvas = tf.reduce_any(actual_masks, axis=0)

        intersection_mask = tf.logical_and(pred_canvas, actual_canvas)
        union_mask = tf.logical_or(pred_canvas, actual_canvas)

        intersection = tf.reduce_sum(tf.cast(intersection_mask, tf.float32), name='intersection')
        union = tf.reduce_sum(tf.cast(union_mask, tf.float32), name='union')

        return intersection / union

    iou = tf.map_fn(_spatial_iou, (actual, pred), dtype=tf.float32)

    return K.mean(iou)

def mask_true(canvas, t):
    x1 = tf.to_int32(t[0])
    y1 = tf.to_int32(t[1])
    x2 = tf.to_int32(t[2])
    y2 = tf.to_int32(t[3])

    shape = tf.shape(canvas)
    x_start = tf.maximum(0, x1)
    y_start = tf.maximum(0, y1)

    x_delta = tf.cond(tf_cond_all([
        tf.greater_equal(x1, 0),
        tf.greater_equal(x2, x1),
        tf.less_equal(x1, shape[1]),
        tf.less_equal(x2, shape[1])
    ]), lambda: x2 - x1, lambda: tf.constant(0))

    y_delta = tf.cond(tf_cond_all([
        tf.greater_equal(y1, 0),
        tf.greater_equal(y2, y1),
        tf.less_equal(y1, shape[0]),
        tf.less_equal(y2, shape[0])
    ]), lambda: y2 - y1, lambda: tf.constant(0))

    area = tf.ones((y_delta, x_delta))

    return canvas[y_start:(y_start + y_delta),
                  x_start:(x_start + x_delta)].assign(area)

For more details on how to write Python tests, check out the pytest documentation.


Writing tests is not a natural discipline; it is not an innate desire. Most coding bootcamps barely cover it if at all–data science programs even less. While it is important to understand algorithms and statistics for data science, our goal as data scientists is to accurately and confidently use data to assist fellow humans. Having rigorous practices that provide concrete steps to achieving this goal ought to be embraced.

The primary purpose of testing is to increase confidence and maintainability. It doesn't mean mandatory 100% test coverage. Some systems are more difficult to test than others. Using statistics for model accuracy is a perfect example. Proper metrics give greater confidence that the model will perform as expected. Likewise, unit tests where possible increase our confidence that we don't have bugs that artificially inflate assurance in a given model or analysis.

Do yourself a favor--write tests and don't assume you can remember everything.

Venners, B. (2002). Test-driven development. Retrieved on Dec 17, 2018, from https://www.artima.com/intv/testdrivenP.html

<![CDATA[Quick Tip: Image Resize Prep for Training]]>https://rcd.ai/image-resize-prep-for-training/5bddc9b0ab98076cb2f92b84Sat, 03 Nov 2018 17:40:26 GMT

(Image from techsparks.co.in)


When designing and working with neural networks, a common requirement is having a fixed image dimensions for inputs. Having been working recently on exploring AAEs (adversarial autoencoder neural networks), I ran into this need while working with a Kaggle data set of car images.

Like most data sets, the Stanford Cars Dataset includes two folders with many images–approximately 8,000 for each in this case. In browsing through the files, I could quickly see that the images had different dimensions (as seen below). The goal, therefore, was twofold: find a reasonable aspect ratio (and size) based on all of the images and then batch resize all of them. By handling the image resizing beforehand, the training phase will be able to progress without as much overhead and thereby allow for tighter feedback loops.

Quick Tip: Image Resize Prep for Training
Sample of car images from the Stanford Cars Dataset that have different dimensions

Find Proper Size & Aspect Ratio

To determine an appropriate size and aspect ratio for the images, some basic statistics of all 16,000+ images are needed. Although the file command does print out the dimensions of the given image, it fails to actually extract the given information:

❯ file cars_train/00001.jpg

cars_train/00001.jpg: JPEG image data, JFIF standard 1.01, resolution (DPI), density 72x72, segment length 16, baseline, precision 8, 600x400, frames 3

Thankfully, ImageMagick's identify command does just this. If you haven't used ImageMagick, I would highly recommend installing it and becoming familiar with its tools from convert to montage. With identify, even the specific dimension–width or height–can now be specified.

❯ identify -format %w cars_train/00001.jpg

❯ identify -format %h cars_train/00001.jpg

Using ls and xargs (with a little bit of bash command line piping), this command can be performed against all of the images in a given folder.

❯ ls cars_train/* | xargs -L1 identify -format "%wx%h\n"

For those unfamiliar with xargs, it essentially allows customization of what happens with the bash pipe operation. In this case, we pass it -L1 to indicate that the command should be run on every line. By default it passes the input as the last argument to the specified command.

# Iterative calls from xargs
identify -format "%wx%h\n" cars_train/00001.jpg  # first time
identify -format "%wx%h\n" cars_train/00002.jpg  # second time
identify -format "%wx%h\n" cars_train/00003.jpg  # third time

The format %wx%h\n simply tells identify to print width + "x" + height + newline for each image.

Although this command does print all of the dimensions, the goal here is really to get a sense for the average, min, and max values for width and height in order to make a good decision about what a happy medium is for the given images. For this final step, then, the datamash command gives an easy way to perform basic statistics by just piping numbers into it. (Check out GNU datamash website for easy installation.)

# Pipe five through ten into datamash for basic stats
❯ seq 5 10 | datamash min 1 max 1 mean 1 median 1
5       10      7.5     7.5

Armed with all of these individual commands, the statistics for width and height can now be easily obtained.

# Stats for width
❯ ls cars_**/*.jpg |
  xargs -L1 identify -format "%w\n" |
  datamash min 1 max 1 mean 1 median 1

78      7800    700.49255483472 640

# Stats for height
❯ ls cars_**/*.jpg |
  xargs -L1 identify -format "%h\n" |
  datamash min 1 max 1 mean 1 median 1

41      5400    483.24584491813 426

Wow. With only two lines of bash commands, the min, max, median, and mean dimensions of 16,000 images have been obtained. I personally find this to be a compelling argument for not only using bash where needed but investing more time in it to learn some really fantastic tools. Looking at this output, it seems that the standard size of 640x480 may be a decent size to employ.

Batch Resize 16k Images

Perhaps not surprisingly at this point, ImageMagick is the tool to turn to yet again. Its mogrify command is just what is needed. The goal here is to resize all images with a scale-to-fit, centered approach and add a neutral background to fill as needed. Additionally, copies of the images are desired so that later they can be used at full resolution (or resized to a different size).

Since the split of the training and test folders are still relevant, this resizing command should be run in each directory. Due to how mogrify works, the subfolder for this images need to be created separately:

mkdir cars_test/resized
mkdir cars_train/resized

Inside of each folder, the mogrify command can now be run.

mogrify -resize 640x480 \  # resize to this size
        -background gray \ # fill with gray as needed
        -gravity center \  # keep image centered
        -extent 640x480 \  # set the image size
        -format jpg \      # keep jpg format
        -path resized \    # put images in a subfolder (no overwrite)
        *.jpg              # files to resize

After pulling a shot of espresso or taking a short walk, the files should be now resized for processing without the overhead of resizing on-the-fly or writing a bunch of Python hoping that you won't run out of memory!

Using in the Data Science Process

If you have read thus far, you may be excited to try and reduce the amount of Python required for image processing. As a software engineer who values maintainability, I highly recommend capturing these commands at least for documentation if not in a shell script. Here's how I might capture it if this Kaggle data set was a real project at my company.

<![CDATA[Eye Tracking: Activation Functions]]>https://rcd.ai/eye-tracking-nn-activation-functions/5bc24d6cab98076cb2f92b70Sun, 14 Oct 2018 01:44:49 GMT

Photo by Joshua Sortino on Unsplash

This blog post is weekly update from a project I've been working on as part of my graduate studies at Regis University for an M.S. in Data Science. Visit the repository for more details.

This week has been filled with fluctuating loss values ranging from <100 to >1,000,000,000 which has made my progress a bit discouraging. However, while experimenting with multiple architectures, I took the opportunity to step back and consider how I might confine the output of the neural network to be a bit more specific to the GazeCapture of iOS devices. Because I performed an initial exploratory data analysis (EDA) of the data, I already knew that the maximum extents of XCam and YCam values that measure the distance in centimeters from the lens. (In this case, this maximum distance is the measure from an 12.9" iPad Pro's lens to its furthest corner.) This can be clearly seen by the following extract from the EDA notebook.

dotInfo.XPts dotInfo.YPts dotInfo.XCam dotInfo.YCam
count 2258008.00 2258008.00 2258008.00 2258008.00
mean 294.22 260.50 0.12 -1.85
std 200.75 190.86 6.08 4.87
min 40.00 40.00 -26.52 -26.52
25% 139.73 116.07 -3.72 -4.23
50% 261.64 220.00 0.40 -1.24
75% 414.28 335.00 3.67 0.94
max 1326.00 1326.00 26.52 26.52

However, at this point, the neural network was simply using a linear function, thereby producing any value from (-∞, +∞) since it just passed the value of the last neurons as the network output. Although the eye tracking project might be theoretically extended to show really "wherever" the person is looking, it is technically limited by this data set to values between [-26.52, 26.52] for both  x and y.

Determining a Custom Activation Function

So, we know that the network should never actually produce values outside of the  [-26.52, 26.52] range. The question is: How should the coordinate output layers be constrained? Should it just cap the values at that extent? The reality is there are several requirements for the optimal function being sought:

  1. Restricts the range. The goal is to predict where a person could be looking as constrained by an iOS device, not anywhere in space. The function should therefore cap at somewhere near 26 above and below zero.
  2. Must be differentiable. Although there are activation functions that are not "technically differentiable" can be used such as ReLU, keras can do the differentiation automatically if given a nicely-behaved function.
  3. Accepts a wide domain to product values less than the extents. For example, if a step-wise function were used, only the maximum or minimum values could be produced. This requirement is that a wide number of values can be used to product values well within the possible range.

To determine what function might work, the sigmoid function is a great place to start. Although it is typically used as an activation function for classification problems, it might be easily adjusted to fulfill our requirements. Below is an example of the base sigmoid function.

Eye Tracking: Activation Functions
Basic Sigmoid Function

Even without modification, this activation function caps the range, is differentiable, and produces values less than its extents. However, the range still needs to be adjusted and a larger range of x-values should be accepted to produce values less than the maximum extent. Thankfully, the Wikipedia article on sigmoid functions also listed a graph of similar functions submitted by an extremely helpful contributor (as seen below).

Eye Tracking: Activation Functions
By Georg-Johann, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=11498624

Armed with the basic sigmoid function and other "logistic-like" functions, I was able to quickly experiment with some different functions using the built-in Mac utility Grapher. Below are the many variations I graphed for experimentation.

Eye Tracking: Activation Functions
Activation Functions (Gray line indicates default linear activation.)

Looking at these functions, several qualities are apparent. The cubic root function (in orange), although producing a good range of outputs, does not actually cap the range. On the other end of the spectrum, the modified sigmoid function (red) and its square root counterpart (blue) properly cap the extents at 30 but only accept values between approximately -5 and +5 to produce anything aside from the maximum extents. However, the base absolute value equation (brown) and its variant (black) overcome this problem since they accept values beyond even -80 and +80 that product values less than the maximum extents.

Conclusion and Results (Ongoing)

Sticking with the absolute value equations as variants to test, I immediately saw the neural network loss values in a friendlier neighborhood (<2000). However, as the activation function is only one of the myriad of hyperparametrs that can be changed for the network, my analysis continues in an effort to have the loss actually converge to a local minima without continuously changing.

Hopefully, this article has given you an idea of how activation functions might be tweaked for your use case. Although the common ones will often suffice (Leaky ReLU, Softmax, Sigmoid, etc.), sometimes a custom activation function is needed because of the given problem's specificity.

Thoughts? Comments? Do you use custom activation functions for certain regression problems? I'd love to hear from you in the comments below.

<![CDATA[Eye Tracking Update: Neural Network Architecture and Loss Functions]]>Background

As a part of my graduate studies at Regis University, I have had the opportunity to begin exploring my own project during my Deep Learning class. (Data Source) By using this data set before my practicum (Spring 2019), I hope to not only explore and model some interesting data

https://rcd.ai/eye-tracking-nn-and-loss/5bb40cdfcdbdfe04bac3a4e6Sat, 06 Oct 2018 14:11:00 GMTBackgroundEye Tracking Update: Neural Network Architecture and Loss Functions

As a part of my graduate studies at Regis University, I have had the opportunity to begin exploring my own project during my Deep Learning class. (Data Source) By using this data set before my practicum (Spring 2019), I hope to not only explore and model some interesting data but also be prepared to solve the eye tracking problem by combining known solutions as well as current research in Deep Learning (e.g. synthetic data training, JavaScript Convnets, pre-trained model adaptation) and thereby bring some of the concepts presented by Krafka et al. (2016) to the web.

Repository: https://github.com/rcdilorenzo/msds-686-eye-tracking

Begin Then Backtrack

After discovering the face-alignment project several weeks ago, I ran it against the entire 2.4M images available in the GazeCapture data set. Below is a sample of the output of Adrian Bulat's deep neural network (written using PyTorch).

Eye Tracking Update: Neural Network Architecture and Loss Functions
Face alignment output (3D points)

For my purposes, I focused this week on modeling the predicted (x, y) distance from the camera lense based on the output of the facial landmarks (68 points in 3D space) as well as the cutout images of each eye. After much debugging in my attempts to coerce the dimensional shapes of the keras layers to cooperate, I found that I could only combine the left and right eye images with the (x, y, z) points of the face-alignment library by having a predefined input shape for each eye. To this end, I spent some time backtracking to produce a consistent 128x128 pixel image (as seen below).

Eye Tracking Update: Neural Network Architecture and Loss Functions
128x128 input to the neural network derived from landmarks

Although I was basing each eye image off of the coordinates from the face-alignment library, I had to do some resizing to make sure that I had a consistent input size. Accomplishing this resize with tensorflow turned to not be the difficult part. The issue primarily arose with images where the projected face landmarks indicated that at least one of the eyes was not even visible in the image. This meant that I needed to still produce a 128x128 image since my neural network would expect that as an input. For the tensorflow code, therefore, I required at least a 1x1 pixel image was sliced off from the image based on the coordinates before being scaled to the appropriate size.

Eye Tracking Update: Neural Network Architecture and Loss Functions
Landmarks projections indicating an additional eye not visible

Thankfully, I was able to backtrack a bit to and revamp the input preparation such that the network could reliably depend on only 128x128 images for the right and left eyes. Below is the tensorflow code for extracting a single eye from the image and landmark points.

def eye_tensor(image, predictions, factor, index = LEFT):
    eye_points = predictions[index:(index + 6), :]
    x = eye_points[:, 0]
    y = eye_points[:, 1]

    # Image dimensions
    image_shape = tf.shape(image)
    image_height = image_shape[0]
    image_width = image_shape[1]

    # Find bounding box
    min_x_raw, max_x_raw = tf.reduce_min(x), tf.reduce_max(x)
    min_y_raw, max_y_raw = tf.reduce_min(y), tf.reduce_max(y)

    # Expand by factor and reform as square
    width = tf.to_float(max_x_raw - min_x_raw)
    height = tf.to_float(max_y_raw - min_y_raw)
    sq_size = tf.to_int32(tf.round(tf.reduce_max([width * factor, height * factor])))

    # Compute deltas
    width_delta = tf.to_int32(tf.round((tf.to_float(sq_size) - width) / 2))
    height_delta = tf.to_int32(tf.round((tf.to_float(sq_size) - height) / 2))

    # Pre-compute max_x and max_y
    max_x = max_x_raw + width_delta
    max_y = max_y_raw + height_delta

    # Calculate whether eye visible within image
    both_eyes_visible = tf.logical_and(max_x < image_width, max_y < image_height)

    # Update frame based on delta (but with min/max boundaries)
    max_x = tf.reduce_min([tf.reduce_max([max_x, 1]), tf.shape(image)[1]])
    min_x = tf.reduce_max([tf.reduce_min([min_x_raw - width_delta, max_x - 1]), 0])
    max_y = tf.reduce_min([tf.reduce_max([max_y, 1]), tf.shape(image)[0]])
    min_y = tf.reduce_max([tf.reduce_min([min_y_raw - height_delta, max_y - 1]), 0])

    # Create image and scale to (128, 128)
    unscaled_shape = tf.stack([max_y - min_y, max_x - min_x, tf.constant(3)])
    eye = tf.reshape(image[min_y:max_y, min_x:max_x] / 255, unscaled_shape)
    scaled_eye = tf.image.resize_images(eye, IMAGE_SIZE)

    # Return original bounding box and resized image
    return (scaled_eye, (min_x, max_x, min_y, max_y), both_eyes_visible)

For those who haven't seen this type of tensorflow code before, it is essentially defining the operation to perform across each input. Checkout my GPUs or CPUs for Deep Learning? post for why this is an tremendous computational advantage.

The "Invisible Eye" Problem

Even though the input image to the network could be "garbage" (or an expanded 1x1 pixel image), the network still technically had a dot that the person is supposedly staring at. As I began to iterate over various architecture types, it became increasingly clear how the network really needed to "know" whether it should be actually predicting a point or not even based just on the image.

How can this be accomplished? Essentially, an additional probabilistic output was needed to tell the neural network whether its regressed (x, y) output is actually valuable. To recap therefore, here are the desired inputs and outputs to the network:

Inputs: Left Eye (128x128), Right Eye (128x128), 3D Landmarks (68 points)
Outputs: Gaze Coordinate (x and y relative to lense), Gaze Likelihood (probability between 0 and 1)

As seen in the above tensorflow code, this likelihood value is not terribly difficult to produce from the coordinates. I did, however, need to verify the code for correctness so I did it TDD-style using the following tests (test_generator.py).


# Load sample data frame and add landmarks as column
df = sample_df()
df['Landmarks'] = np.load('02-facial-landmarks/sample_landmarks.npy')

generator = InspectNNGenerator(session, df, 2, set_type=SET_TYPE_TRAIN)

# Remove randomization
generator.data_frame = df

inputs, outputs = generator.__getitem__(0)

def test_input_size():
    assert len(inputs) == 3

def test_input_row_counts_match():
    assert list(map(len, inputs)) == [2, 2, 2]

def test_eye_inputs():
    left_eyes = inputs[0]
    right_eyes = inputs[1]

    assert left_eyes[BOTH_EYES_IDX].shape == (128, 128, 3)
    assert left_eyes[ONE_EYE_IDX].shape == (128, 128, 3)

    assert right_eyes[BOTH_EYES_IDX].shape == (128, 128, 3)
    assert right_eyes[ONE_EYE_IDX].shape == (128, 128, 3)

def test_landmark_inputs():
    landmarks = inputs[2]

    assert landmarks.shape == (2, 68, 3)

def test_output_shape():
    assert outputs.shape == (2, 3)

def test_gaze_likelihood_output():
    # 1 = both eyes visible
    # 0 = one or both eyes missing
    assert outputs[BOTH_EYES_IDX, 2] == 1.0
    assert outputs[ONE_EYE_IDX, 2] == 0.0

Writing a Loss Function

So, here's the problem. It's not terribly complicated to generate the probability as a value from 0 to 1. The more difficult piece is coming up with a single loss function that describes how "off" the prediction of the network really is. Prior to this point, I had been using the mean-squared error (MSE) of the distance between the predicted and actual points (a.k.a. Euclidean distance or Pythagorean theorem).

(Feel free to skip to the next section is math isn't your cup of tea; you won't miss much.)

Conceptually, the goal is to penalize that Euclidean distance by a factor that describes the difference in predicted versus actual probability that the coordinate is actually meaningful. If the network predicts that the coordinate matters but the truth is that one eye isn't visible, the loss should be described as the Euclidean distance times a factor larger than 1. The other requirement of this loss function is that the raw Euclidean distance should be returned if the probability output is perfectly correct. Putting these requisites together, a single equation can be determined by combining a probability delta between 1 and 2 (the factor) and the existing loss function. Here is one method of accomplishing this without the need of an absolute value operation.

    \text{Loss} &= \text{MSE}(\text{Euclidean distance (D) and Gaze likelihood (G)})\\
    G &\in [0, 1]\; \text{(where 0 means one or more eyes are not visible)}\\\\
    G_{\text{Loss}} &= (\hat{G} - G)^2 + 1\; \text{(as multiplicative error factor)}\\
    D_{\text{Loss}} &= \sqrt{(\hat{y} - y)^2 + (\hat{x} - x)^2}\\
    \text{MSE} &= \frac{1}{b}\sum_{i=1}^b(\text{distance}^2)\\\\
    \text{Loss} &= \frac{1}{b}\sum_{i=1}^b\left[\left((\hat{{}_{i}G_L} - {}_{i}G_L)^2 + 1\right)\cdot\sqrt{(\hat{y_i} - y_i)^2 + (\hat{x_i} - x_i)^2}\right]^2\\
    &= \frac{1}{b}\sum_{i=1}^b\left((\hat{{}_{i}G_L} - {}_{i}G_L)^2 + 1\right)^2\cdot\left((\hat{y_i} - y_i)^2 + (\hat{x_i} - x_i)^2\right)

Additionally, here is the tensorflow implementation (which ironically looks a bit cleaner):

# (1/b) ∑^b_(i=1)([(^G-G)^2 + 1]^2 * [(^y - y)^2 + (^x - x)^2])
def loss_func(actual, pred):
    x_diff = tf.square(pred[:, 0] - actual[:, 0])
    y_diff = tf.square(pred[:, 1] - actual[:, 1])
    g_diff = tf.square(pred[:, 2] - actual[:, 2])

    return K.mean(tf.square(g_diff + 1) * (y_diff + x_diff))

Check Those Activation Functions

With the loss function in place, I was able to continue experimenting with simpler versions of proven image-based neural network designs like Inception and ResNet50. Unfortunately, for many iterations even before adding the third output, I could not get the loss to go below 28 but doggedly hovered around 28-33. Finally, when I began adding the third output, I realized that I had a sigmoid activation function for the (x, y) coordinate. For those not immediately familiar with this function, here is its graph:

Eye Tracking Update: Neural Network Architecture and Loss Functions
Sigmoid (Source: Wolfram MathWorld)

The problem is that it can never a value less than 0 or above 1! Feeling much humiliation after this silly mistake, I quickly adjusted the functions to use linear activation functions since the (x, y) coordinate is not technically limited to any particular range. (Although I could arbitrarily constrain it to the largest iOS device for this case since those are the only devices from GazeCapture, I have preferred to keep the neural network more general.)

Build, Measure, Learn

Throughout this process of output modification and loss adaption, I worked on various architectures. Although it is still too early to determine whether the discussed loss function will need tweaking, the current model training sessions have had measure of success though still limited. Many of the architecture iterations I was able to throw out quickly since the loss function was resulting in values in the trillions or quadrillions (even with the scale change of the new loss output which is much larger than 28-33).

With two GPUs (Nvidia 1080 Ti's) at my disposal in my custom build, I am currently running two slight variations of an architecture while measuring the results with TensorBoard as well as monitoring the output. I am still unsure about some of the results but am continuing to debug the extreme fluctuations that seem to occur in some of loss and val_loss outputs. Because many previous iterations have been upwards of 10^24 values, I am continuing with these models for a full 100 epoch cycle to see if they eventually converge.

Eye Tracking Update: Neural Network Architecture and Loss Functions
Loss values for v6 and v7 builds (ε = 0.1 vs 0.01)

Although these results may not seem promising, several characteristics make v6 in particular (denoted by red) a bit more interesting. Per epoch, each model uses the entire available training set as well as the predetermined validation set established by the original GazeCapture researchers (where facial landmarks have been detected).

Eye Tracking Update: Neural Network Architecture and Loss Functions
Data split between train/test/validation

While the output of v6 seems to wildly vary per epoch, watching the results in person present a different story. The loss value steadily decreases perhaps 95% of the time and then spikes up by sometimes several orders of magnitude for remaining 5% of the time. The TensorBoard representation only captures the loss at the end of each epoch which only happens every ~1.5M rows (plus the validation set). The other consideration that must be made is that most of the activation functions need to use a linear activation function and do not therefore constrain the output of each neuron. I will just have to sit tight and wait to see how the model progresses especially considering the entire model right now is only 467KB.

# train.py (excerpt) - https://git.io/fxsO2

left_eye_input = Input(shape=(128,128,3))
right_eye_input = Input(shape=(128,128,3))
landmark_input = Input(shape=(68,3))

def eye_path(input_layer, prefix='na'):
    return pipe(
        Conv2D(8, (3, 3), activation='relu', padding='same', name=(prefix + '_3x3conv1')),
        MaxPooling2D(pool_size=(3, 3), padding='same', name=(prefix + '_max1')),
        Conv2D(8, (3, 3), activation='relu', padding='same', name=(prefix + '_3x3conv2')),
        MaxPooling2D(pool_size=(3, 3), padding='same', name=(prefix + '_max2')),
        Conv2D(4, (2, 2), activation='relu', padding='same', name=(prefix + '_2x2conv1')),
        MaxPooling2D(pool_size=(2, 2), padding='same', name=(prefix + '_max3')),
        Flatten(name=(prefix + '_flttn'))

left_path = eye_path(left_eye_input, prefix='left')
right_path = eye_path(right_eye_input, prefix='right')

landmarks = pipe(
    Dense(16, activation='linear'),
    Dense(8, activation='linear'),

grouped = concatenate([left_path, right_path, landmarks])

coordinate = pipe(
    Dense(16, activation='linear'),
    Dense(8, activation='linear'),
    Dense(2, activation='linear', name='coord_output')

gaze_likelihood = pipe(
    Dense(8, activation='relu'),
    Dense(4, activation='relu'),
    Dense(1, activation='sigmoid', name='gaze_likelihood')
output = concatenate([coordinate, gaze_likelihood])

model = Model(inputs=[left_eye_input, right_eye_input, landmark_input],

Thoughts or questions? Something not explained well? I am just starting into this deep learning world and would love to hear what you have to say.

<![CDATA[GPUs or CPUs for Deep Learning?]]>Photo by Jordan Harrison from Pexels

The repurposing and expansion of GPUs for neural network calculations has revolutionized the possibilities of deep neural network architectures and made large, general-purpose models like Inception and RestNet computationally feasible. Why is this? What makes GPUs so much better suited than CPUs for these

https://rcd.ai/gpu-or-cpu-deep-learning/5bb92d51ed70e35e656200d9Thu, 27 Sep 2018 21:46:00 GMT

Photo by Jordan Harrison from Pexels


When explaining this in person, I find it most helpful to consider the similarities to 3D rendering. As some may already know, any 3D model is composed of tessellations (or tiny triangles) that are all rendered using the appropriate material, lightning, position, and its other attributes. In other words, the same exact type of calculation must be performed on each tessellation. Likewise, the same differentiated loss function has to be back-propagated through a neural network — against every neuron. In this similar manner, therefore, a single (grouped) operation has to be performed on a large set of neurons just as it must on the tessellations.

Although this may seem to make sense thus far, what attributes of a GPU make it better at doing the same operation across many data points? The answer lies in the hardware tradeoffs that a GPU can make but that a CPU cannot. While modern CPUs have much hardware devoted towards the classic stage of fetching and decoding operations, GPUs have a fraction of that hardware that focus on understanding the actually computation being performed. Since the GPU’s primary difference is performing the same operation on different data points, many more cores and more specifically ALUs (arithmetic logical units) can be packed into the same chip. This is why GPUs often have orders of magnitude more cores than CPUs. On one hand, CPUs can optimized general operation performance by using tricks such as out-of-order execution or speculative execution, GPUs need not have this hardware at all.

To put it another way — GPUs and CPUs have distinct purposes. Performing a graphics task on a CPU or doing general purpose calculations on a GPU will result in extremely poor performance. The trick is knowing when to use which one.

<![CDATA[An Intro into Ethical Issues of Data Science]]>https://rcd.ai/ethical-issues-data-science/5bb3e5e5cdbdfe04bac3a4e0Mon, 02 Oct 2017 21:42:00 GMT

Photo by rawpixel / Unsplash

Data is everywhere. Now more than ever, the internet has exploded with almost inconceivable amounts of data just waiting to be mined by the eager researcher. The industry is continually discovering new and innovative ways to apply this data from music recommendations to disease prediction. As all of these applications evolve, the time is ripe for a consideration of various ethical problems that are sometimes subtly presented not only to the researcher but also other interested parties. In this short discussion, four of the many ethical issues related to data science will be presented as based on the article presented by Michael Fuller this year entitled Big Data, Ethics and Religion: New Questions from a New Science.

First, any use of data requires assumed or explicit consent. So often, though, users see this consent in the form of a privacy policy or EULA that is not only incredibly long but also just as confusing to understand with the legal terminology and release of rights. These documents are far too often focused on the mitigation of liability rather than genuinely explaining what can or will be done with the user’s data. This ethical inconsistency of informing yet not actually informing the user could be solved with readable explanations as well as more granularity of consent rather than a single agreement to the entire lengthy document. Far too often, the disciplines of law and software become isolated from each other such that neither can coordinate enough to care for their customers.

Second, an ethical contention exists between data-based intelligence and individual privacy. As Fuller presents, aggregation and analysis of individual medical information can be extremely useful particularly for future research and development of new treatments. Additionally, an individual can be easily identifiable by only three traits: gender, zip code, and year of birth. Understanding this fine balance will be key to the necessary discussion and disclosure of what data can be used and how it may be applied. The solution here, unfortunately, will not be an easy one.

Third, ownership and rights to share data pose a significant ethical question. Is identifiable personal data always personal or can it always be shared as it is now by data brokers? Some countries have actually mandated web-based services to keep surveillance data on their users for government access. Like other dilemmas, these data “rights” must be responsibly disclosed and open for discussion. Individual privacy has to be balanced with security concerns particularly at the civil level.

Fourth and perhaps most subtle, research bias can creep in perhaps even more than in typical scientific journals. Who performs the data? What data is cleaned from the set? What metrics are used for analysis? Does the presentation of results accurately portray the information? As Fuller recognizes, bias, either intentional or not, can appear in data science at nearly every step of the process. Although bias will always be present, peer review and disclosure of methodologies as applied by other scientific disciplines can assist in keeping these decisions accountable and more ethically sound.

In short, data science is inseparable from ethics. The Bible says that money is not evil but rather the love of money is the root of all evil (1 Timothy 6:10). Likewise, data is not the problem; every concern revolves around how it is used. Open discussion and responsible transparency will be the major weapons in this battle of data control and regulation.

What are your concerns with Big Data? What do you think about cloud-based computing or data brokering? Add to the discussion by leaving a comment.