1. Add the pdf-reader gem to your Gemfile

gem "pdf-reader", "~> 2.14"

2. Add the analyzer

# /lib/pdf_analyzer.rb

class PdfAnalyzer < ActiveStorage::Analyzer
  def self.accept?(blob)
    blob.content_type == "application/pdf"
  end

  def metadata
    download_blob_to_tempfile do |file|
      reader = PDF::Reader.new(file)

      reader.info
        .transform_keys { |key| key.to_s.underscore }
        .transform_values { |value| parse_pdf_datetime(value) }
        .merge({
          page_count: reader.page_count
        })
    end
  end

  private

  def parse_pdf_datetime(time)
    match = time.match(/D:(\d{14})([+-]\d{2})'(\d{2})'/)

    if match
      datetime_str, offset_hours, offset_minutes = match.captures

      # Parse the datetime portion
      time = Time.strptime(datetime_str, "%Y%m%d%H%M%S")

      # Convert offset to ActiveSupport::TimeZone format
      offset_seconds = offset_hours.to_i * 3600 + offset_minutes.to_i * 60
      time_with_zone = time.in_time_zone(offset_seconds)

      time_with_zone
    else
      time
    end
  end
end

3. Register the analyzer with ActiveStorage

# /config/application.rb

config.active_storage.analyzers << ::PdfAnalyzer

Usage

document = Document.first
document.file.metadata
# {
#     "identified"=>true,
#     "analyzed"=>true,
#     "producer"=>"PFU PDF Library 1.2.1",
#     "creator"=>"ScandAll PRO V2.1.5",
#     "creation_date"=>"2023-11-01T11:32:25.000-06:00",
#     "mod_date"=>"2023-11-01T11:32:25.000-06:00",
#     "metadata_date"=>"2023-11-01T11:32:25.000-06:00",
#     "page_count"=>16
#  }

The Setup

My setup has been in contant flux but lately I’ve started copying all the icon files into app/assets/images and rendering them with inline_svg and a custom helper.

module ApplicationHelper
  def svg(name, **args)
    return if name.blank?
    filename = "#{name}.svg" unless name.end_with?(".svg")
    inline_svg(filename, **args)
  end
end

The icons are stored with in directories based on the icon source and variant.

$ find app/assets/images -name "*.svg"
# app/assets/images/fontawesome/regular/*.svg
# app/assets/images/heroicons/micro/*.svg
# app/assets/images/heroicons/mini/*.svg
# app/assets/images/heroicons/outline/*.svg
# app/assets/images/heroicons/solid/*.svg

Using this helper and this file structure makes it quick to add icons to views and in other helper methods.

<%= link_to post, class: 'flex items-center' do %>
  <span>View More</span>
  <%= svg "heroicons/micro/chevron-right", class: 'size-4 ml-1' %>
<% end %>

The Problem

One of the projects I’m working on right now has almost 1,600 svg icons saved into the project. I’m perfectly happy with this during development but it is very slow during deployment when precompiling assets.

The solution I came up with the help of ChatGPT :) was to remove the icons that are not referenced in app/controllers, app/helpers, and app/views. This script purges 1,500+ icons in less than a second and allows me to be fast during development and pushing to production.

#!/usr/bin/env bash

set -euo pipefail

namespaces=("heroicons" "fontawesome" "phosphor")
pattern="$(IFS='|'; echo "${namespaces[*]}")"

# Step 1: Find all SVG files under the selected namespaces
mapfile -t svg_files < <(find app/assets/images -type f -name "*.svg" | grep -E "$pattern")

# Step 2: Extract all possible used keys from views using grep
# This builds a list like: heroicons/outline/arrow-left, etc.
mapfile -t used_keys < <(grep -rhoE "($pattern)/[a-zA-Z0-9_-]+/[a-zA-Z0-9_-]+" app/controllers app/helpers app/views | sort -u)

# Step 3: Put keys into a lookup set (associative array for fast matching)
declare -A used_lookup
for key in "${used_keys[@]}"; do
  used_lookup["$key"]=1
done

# Step 4: Check all SVGs in one loop (still O(n), but lookup is O(1))
unused_files=()
for path in "${svg_files[@]}"; do
  key="${path#app/assets/images/}"
  key="${key%.svg}"

  if ! [[ ${used_lookup["$key"]+_} ]]; then
    unused_files+=("$path")
  fi
done

# Step 5: Delete unused files in batches (fast)
if [[ ${#unused_files[@]} -gt 0 ]]; then
  printf "%s\n" "${unused_files[@]}" | xargs -P 4 rm
fi

# Format the USB drive as MS-DOS and name it "WIN10"
$ diskutil eraseDisk MS-DOS "WIN10" MBR /dev/disk127

# Copy files from mounted Windows 10 ISO to new drive
#   > Make sure to preserve trailing slash (/)
$ rsync -vha --exclude=sources/install.wim /Volumes/CCCOMA_X64FRE_EN-US_DV9/ /Volumes/WIN10

# Split and copy the last file
$ wimlib-imagex split /Volumes/CCCOMA_X64FRE_EN-US_DV9/sources/install.wim /Volumes/WIN10/sources/install.wim 4000

The prerequisites for the above commands are knowing the id of the mounted USB drive and wimlib-imagex. MacOS only has write capabilities for the older MS-DOS file format which has a 4GB single file size limit. This is why we use wimlib-imagex to split the 5GB install.wim into two files.

You can find the ID of your USB drive with Disk Utility.

It looks like I installed wimlib-imagex with HomeBrew.

$ brew install wimlib

See you later future Baylor!

The end result shows how they built an incredibly complex cohort query with efficiency and maintainablity using Arel.

The first thing I learned is that HEIF is not shipped with image processing libraries, ImageMagick or Sharp. From what I can tell this is mostly because of licensing issues that thankfully don’t apply to me. As a result, the first requirement is to compile a Vips with a self install version of libheif and libde265.

This is the largest hurdle since AWS Lambda runs application code in a stateless, isolated environment. To overcome this we need to build and package these libraries and attach them to our Lambda function. I use the word “attach” since I uploaded these binaries as a Lambda layer to lighten my application code.

I used a multi-stage Dockerfile to build the binaries, copy them into another isolated environment and install the sharp node module. I then wrapped these steps in a shell script to further the portability of it.

Dockerfile

# Use environment that closely matches AWS Lambda
FROM lambci/lambda:build-nodejs12.x AS builder

WORKDIR /var/task

# Download dependencies for vips, libheif and libde265
RUN curl -L https://github.com/libvips/libvips/releases/download/v8.9.0/vips-8.9.0.tar.gz | tar xz && \
    curl -L https://github.com/strukturag/libde265/releases/download/v1.0.4/libde265-1.0.4.tar.gz | tar xz && \
    curl -L https://github.com/strukturag/libheif/releases/download/v1.6.1/libheif-1.6.1.tar.gz | tar xz && \
    yum install -y gtk-doc gobject-introspection gobject-introspection-devel expat-devel libjpeg-turbo libjpeg-turbo-devel libpng libpng-devel

# Install h.265 video codec library
RUN cd libde265-1.0.4 && \
    ./autogen.sh && \
    ./configure && \
    make && \
    make install

# Install HEIF library
RUN cd libheif-1.6.1 && \
    ./autogen.sh && \
    PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure && \
    make && \
    make install

# Install vips
RUN cd vips-8.9.0 && \
    ./autogen.sh && \
    PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure --prefix=/var/task/vendor && \
    make && \
    make install

# Copy pkg-config files to working directory
RUN cp /usr/local/lib/pkgconfig/* vendor/lib/pkgconfig/ && \
    cp /usr/lib64/pkgconfig/libpcre* vendor/lib/pkgconfig/ && \
    cp /usr/lib64/pkgconfig/glib* vendor/lib/pkgconfig/ && \
    cp /usr/lib64/pkgconfig/gobject* vendor/lib/pkgconfig/ && \
    cp /usr/lib64/pkgconfig/gmodule* vendor/lib/pkgconfig/ && \
    cp /usr/lib64/pkgconfig/gthread* vendor/lib/pkgconfig/ && \
    cp /usr/lib64/pkgconfig/libpng* vendor/lib/pkgconfig/ && \
    cp /usr/lib64/pkgconfig/libjpeg* vendor/lib/pkgconfig/

# Copy compiled binaries to working directory
RUN cp /usr/local/lib/libheif.* vendor/lib/ && \
    cp /usr/local/lib/libde265.* vendor/lib/ && \
    cp /usr/lib64/libpng* vendor/lib/ && \
    cp /usr/lib64/libjpeg* vendor/lib/ && \
    cp -r /usr/lib64/glib*/include/* vendor/include/ && \
    cp -r /usr/include/glib*/* vendor/include/ && \
    cp -r /usr/lib64/gobject* vendor/lib/ && \
    cp /usr/lib64/libexpat* vendor/lib/ && \
    cp /usr/lib64/libgthread* vendor/lib/ && \
    cp /usr/lib64/libgmodule* vendor/lib/ && \
    cp /usr/lib64/libpcre* vendor/lib/ && \
    cp /usr/lib64/libgobject* vendor/lib/ && \
    cp /usr/lib64/libglib* vendor/lib/


# Starting in a new image ensures we have everything for
# an isolated environment.
FROM lambci/lambda:build-nodejs12.x

WORKDIR /var/task

# Copy results from builder image
COPY --from=builder /var/task/vendor ./vendor

# Install sharp with node
RUN LD_LIBRARY_PATH=/var/task/vendor/include PKG_CONFIG_PATH=/var/task/vendor/lib/pkgconfig npm install sharp

Compile and Copy Script

The following script builds and runs the docker image and copies the necessary files to the host machine. It’s important to note that the files are copied to a directory structure that is matches $PATH, $NODE_PATH and $LD_LIBRARY_PATH inside the Lambda environment.

#!/usr/bin/env bash

main() {
  echo "Building Image"
  IMAGE_ID=$(docker build . -q)

  echo "Starting Container"
  CONTAINER_ID=$(docker run -d -it $IMAGE_ID bash)

  echo "Copying node_modules into node_modules"
  docker cp $CONTAINER_ID:/var/task/node_modules ./vips-sharp-layer-env/nodejs

  echo "Copying vendor into vendor"
  docker cp $CONTAINER_ID:/var/task/vendor/bin ./vips-sharp-layer-env
  docker cp $CONTAINER_ID:/var/task/vendor/lib ./vips-sharp-layer-env

  echo "Removing container"
  docker kill $CONTAINER_ID
  docker rm $CONTAINER_ID
}

main

Creating the Lambda Layer

I’m using CloudFormation and the CDK to build out my application. The following will upload the results from the above script as a Layer and add it to a Lambda Function.

const uploadBucket = new s3.Bucket(this, "Upload")
const outputBucket = new s3.Bucket(
  this,
  "Output",
  {
    blockPublicAccess: BlockPublicAccess.BLOCK_ACLS,
  }
)

outputBucket.addToResourcePolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  principals: [new iam.AnyPrincipal()],
  actions: ['s3:GetObject'],
  resources: [`arn:aws:s3:::${outputBucket.bucketName}/*`]
}))

const vipsSharpLayer = new lambda.LayerVersion(
  this,
  "VipsSharpLayer",
  {
    compatibleRuntimes: [lambda.Runtime.NODEJS_12_X],
    code: lambda.Code.fromAsset("lambda/process-images/vips-sharp-layer-env")
  }
)

const resizeHandler = new lambda.Function(
  this,
  "ResizeHandler",
  {
    runtime: lambda.Runtime.NODEJS_12_X,
    code: lambda.Code.fromAsset("lambda/process-images/process-images"),
    handler: "resize.handler",
    timeout: Duration.seconds(30),
    layers: [vipsSharpLayer],
    environment: {
      OUTPUT_BUCKET: outputBucket.bucketName
    }
  }
)

uploadBucket.grantRead(resizeHandler)
outputBucket.grantWrite(resizeHandler)

resizeHandler.addEventSource(new S3EventSource(uploadBucket, {
  events: [s3.EventType.OBJECT_CREATED],
}))

The Lambda Function

This code barely scratches the surface of what I want it to do long term. Right now it simply loads the S3 object into sharp, formats it to JPG and uploads it into an output bucket.

I’ve also introduced ExifReader to read the orientation of the image to ensure it is rotated correctly during the upload. Sharp has support for auto rotating the image but I found it couldn’t read the orientation data from HEIC images. This is unfortunate as it requires me to load the entire image into memory rather than stream it into the output bucket.

const AWS = require('aws-sdk')
const sharp = require('sharp')
const ExifReader = require('exifreader')

const S3 = new AWS.S3({
  maxRetries: 0,
  region: 'us-east-1'
})

const ORIENTATION_MAP = {
  auto: undefined,
  3: 180,
  6: 90,
  8: -90
}

exports.handler = async (event, context) => {
  const srcBucket = event.Records[0].s3.bucket.name;
  const srcKey = event.Records[0].s3.object.key;

  console.log(`Getting ${srcKey} from s3://${srcBucket}`)
  const imageObject = await S3.getObject({
    Bucket: srcBucket,
    Key: srcKey
  }).promise()

  const image = imageObject.Body

  console.log('Read exif data')
  let orientation = 'auto'
  try {
    const exif = ExifReader.load(image)
    orientation = exif && exif.Orientation && exif.Orientation.value || 'auto'
  }catch (e) {
    console.log('Failed reading exif data')
    console.log(e)
  }

  console.log('Building Sharp Pipeline \'toJPG\'')
  const output = sharp(image)
    .rotate(ORIENTATION_MAP[orientation])
    .toFormat('jpg')

  console.log('Upload image')
  const uploadData = await S3.upload({
    Body: await output.toBuffer(),
    Bucket: process.env.OUTPUT_BUCKET,
    ContentType: 'image/jpg',
    Key: srcKey.split('.')[0] + '.jpg'
  }).promise()

  console.log('Data: ', {
    ...uploadData
  })

  return {
    message: 'Success',
    timeRemainingMs: context.getRemainingTimeInMillis()
  }
}

And that’s it. Aside from locating all the necessary binaries to create a portable version of Vips it is pretty straight forward. Hopefully you can use this as a reference for your own project and you findit helpful.

Usage

import React, { useState } from 'react'

const Form = () => {
  const [body, setBody] = useState('')

  return (
    <Editor value={body} onChange={setBody} />
  )
}

The Editor Component

import React, { useEffect, useRef } from 'react'
import 'trix'
import 'trix/dist/trix.css'

const Editor = ({ value, onChange }) => {
  const trixEditor = useRef(null)

  useEffect(() => {
    trixEditor.current.addEventListener('trix-change', (e) => {
      onChange(trixEditor.current.value)
    })
  }, [trixEditor])

  useEffect(() => {
    if (!trixEditor.current) return
    if (trixEditor.current.inputElement.value === value) return
    trixEditor.current.value = value
  }, [value])

  return (
    React.createElement('trix-editor', {ref: trixEditor})
  )
}

export default Editor

It’s important to note that this configuration runs Postgres inside a docker container with the port forwarded to a random, available port within the build’s VM. Don’t forget to modify your config/database.yml to read the configuration from the assigned environment variables.

name: Ruby

on: [push]

jobs:
  rspec:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: RAILS_APP_DB_NAME_test
        ports:
        # will assign a random free host port
        - 5432/tcp
        # needed because the postgres container does not provide a healthcheck
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

    steps:
    - uses: actions/checkout@v1

    - name: Install Postgres Libraries
      run: |
        sudo apt-get install libpq-dev

    - name: Set up Ruby 2.6
      uses: actions/setup-ruby@v1
      with:
        ruby-version: 2.6.x

    - name: Bundle Install
      run: |
        gem install bundler
        bundle install --jobs 4 --retry 3

    - name: Build and test with RSpec
      env:
        POSTGRES_HOST: localhost
        POSTGRES_PORT: ${{ job.services.postgres.ports[5432] }} # get randomly assigned published port
        POSTGRES_USER: postgres
        POSTGRES_PASS: postgres
      run: |
        bundle exec rspec

I’m using #fetch to have sensible defaults if the environment variables aren’t defined.

test:
  <<: *default
  database: RAILS_APP_DB_NAME_test
  username: <%= ENV.fetch('POSTGRES_USER', '') %>
  password: <%= ENV.fetch('POSTGRES_PASS', '') %>
  host: <%= ENV.fetch('POSTGRES_HOST', 'localhost') %>
  port: <%= ENV.fetch('POSTGRES_PORT', 5432) %>

• Rails Helper
• Matcher Class
• Usage
• Spec

Rails Helper

# app/helpers/active_link_helper.rb
# frozen_string_literal: true

require_relative "../../lib/active_link_path"

module ActiveLinkHelper
  def nav_link_to(path, matches: [], &block)
    matcher = ActiveLinkPath.new(params)
    active = matcher.active_match?(matches) ? 'active' : nil
    link_to path, class: ['nav-link', active].compact, &block
  end
end

Matcher Class

# lib/active_link_path.rb

class ActiveLinkPath
  attr_reader :context

  def initialize(context)
    @context = context.stringify_keys
  end

  def active_match?(match)
    matches = match.is_a?(Hash) ? [match] : match

    matches.any? do |match|
      match.stringify_keys!
      values_match(match, context.slice(*match.keys))
    end
  end

  private

  def values_match(match, sliced_context)
    match.all? do |(key, value)|
      value = Array(value)
      value.include?(sliced_context[key])
    end
  end
end

Usage

The API is somewhat complex to accommodate most use cases.

1. Basic resource resources :products
2. Nested resources resources :collections { resources :products }
3. Top level REST actions get '/export/products', to: 'export#products'

In the above list, #3 is the most interesting since you’d probably access this action from the products index page (/products).

nav_link_to(some_path,
            matches: {controller: 'products'})

# 👍   /products                 products#index
# 👍   /products/1               products#show
# 👍   /products/1/edit          products#edit
# 👍   /collections/1/products   products#index
# ❌   /collections              collections#index
# ❌   /collections/1            collections#show
# ❌   /export/products          export#products

nav_link_to(some_path,
            matches: {controller: %w(products collections)})

# 👍   /products                 products#index
# 👍   /products/1               products#show
# 👍   /products/1/edit          products#edit
# 👍   /collections/1/products   products#index
# 👍   /collections              collections#index
# 👍   /collections/1            collections#show
# ❌   /export/products          export#products

nav_link_to(some_path,
            matches: {controller: 'products',
                      action: 'show'})

# 👍   /products/1               products#show
# ❌   /products                 products#index
# ❌   /products/1/edit          products#edit
# ❌   /collections/1/products   products#index
# ❌   /collections              collections#index
# ❌   /collections/1            collections#show
# ❌   /export/products          export#products

nav_link_to(some_path,
            matches: {controller: %w(products collections),
                      action: 'show'})

# 👍   /products/1               products#show
# 👍   /collections/1            collections#show
# ❌   /products                 products#index
# ❌   /products/1/edit          products#edit
# ❌   /collections/1/products   products#index
# ❌   /collections              collections#index
# ❌   /export/products          export#products

nav_link_to(some_path,
            matches: [
              {controller: %w(products collections), action: 'index'},
              {controller: 'products', action: 'edit'},
              {controller: 'export', action: 'products'},
             ])

# 👍   /products                 products#index
# 👍   /products/1/edit          products#edit
# 👍   /collections/1/products   products#index
# 👍   /collections              collections#index
# 👍   /export/products          export#products
# ❌   /products/1               products#show
# ❌   /collections/1            collections#show

Spec

I’ve also written a spec for this if you’d like to include it in your application.

# spec/lib/active_link_path_spec.rb
# frozen_string_literal: true

require "spec_helper"
require "active_support/hash_with_indifferent_access"
require "./lib/active_link_path"

describe ActiveLinkPath do
  context "active_match?" do
    it "finds overlappings key value pairs" do
      matcher = ActiveLinkPath.new({'key' => 'value1', 'key2' => 'value2'})
      expect(matcher.active_match?(key: 'value1')).to be_truthy
      expect(matcher.active_match?(key: 'value2')).to be_falsey
    end

    it "finds from multiple matching pairs" do
      matcher = ActiveLinkPath.new({'key' => 'value1', 'key2' => 'value2'})
      expect(matcher.active_match?(key: 'value1', key2: 'value2')).to be_truthy
      expect(matcher.active_match?(key: 'value1', key2: 'value1')).to be_falsey
      expect(matcher.active_match?(key: 'value1', key3: 'value3')).to be_falsey
    end

    it "allows an array of values" do
      matcher = ActiveLinkPath.new({'key' => 'value1', 'key2' => 'value2'})
      expect(matcher.active_match?(key: ['value', 'value1'])).to be_truthy
      expect(matcher.active_match?(key: ['value', 'value2'])).to be_falsey
    end

    it "allows an array of matchers" do
      matcher = ActiveLinkPath.new({'key' => 'value1', 'key2' => 'value2'})
      expect(
        matcher.active_match?([
          {key: ['value'], key2: 'value3'},
          {key: 'value1', key2: ['value', 'value2']}
        ])
      ).to be_truthy

      expect(
        matcher.active_match?([
          {key: ['value'], key2: 'value3'},
          {key: 'value1', key2: ['value', 'value1']}
        ])
      ).to be_falsey
    end
  end
end

While this is a problem I still don’t fully understand it was especially difficult for me back then. I’m writing this post for my future self and for others facing a similar issue.

The Solution

I think the end goal to this problem is basically building a graph database that has the following connections user -> (person, company or location). With this design it is possible to perform a reverse lookup to find the fellow user with the contacts you are looking for.

Problem #1: Classifying Texts

The platform I worked on was strictly driven by unlabelled text. Posts could be any of the following types Question, Announcement, Search, Poll and didn’t follow a template.

1. “Does anyone have any recommendations for restaurants in X city?”

2. “The company I work for is wanting to hire an intern for the summer. Please contact me at “x@y.com”

3. “I’m looking for a contact at X company to promote Y product.”

4. “Which of these products do you think provide the best lift X, Y or Z?”

All of these texts are hand written without any required structure. This makes classifying and therefore indexing the important information very difficult.

I think the way we solve this first problem is is by applying Part of Speech tagging overlayed with an intent parser. The idea is we look at the verbs in correlation to types of entities they are modifying/connected to. These two techniques should help us label the text.

Problem #2: Extracting Entities

Once we have labelled the text, next we need to extract the entities in question.

All of the above texts have important information to build our connections database. In the first text, a question, we ask for recommendations in a city. This is helps us identity that the user is likely making a trip to X city and probably knows or would like to meet people in that city.

This information compounds itself since we will have likely extracted entities from previous posts which would help us determine their likes and interests.

“The following people are from X city and also like A, B and C.”

This makes it realistic for us to create ‘answers’ or help other users find questions they may be able to answer.

Example Entity Extraction

Problem #3: Finding Answers

The last step to finding answers comes from inferring connections. This is probably best solved with word embeddings such as Word2Vec.

I think there’s a good possibility we can start with a pretrained model from the Wikipedia and/or Google News datasets. These models will help us gain insight into the people, companies or locations and figure out how they are related.

The idea being that since we know the entities in a question (both the term and the label) we can query our datasets to make predictions of who might have a connection based on the relationships found in an unsupervised model from a larger dataset.

It becomes even more interesting when you factor in that Word2Vec can also perform mathematical operations to determine the resulting point. As we improve our tagging of text it might be possible to create positive and negative terms when searching for the most similar results.

Conclusion

This post has been a bit abstract mostly because I haven’t actually implemented any of the things I mentioned above. After spending the last few months working on document classification and working with Spacy I was reminded of this previous project and thought I should log my initial ideas on solving this problem again.

Sources:
Model: https://codepen.io/explosion/pen/991f245ef90debb78c8fc369294f75ad
Image: https://www.samyzaf.com/ML/nlp/word2vec2.png
Image: https://www.tensorflow.org/tutorials/representation/word2vec

Interestingly enough this isn’t all that different than how I learned web development. I started by viewing the source of a web page, then creating my own .html file, followed by applying CSS and JavaScript to build what I wanted.

However, neural networks and machine learning require far more discipline than web development. While I would argue the tools, frameworks and blogs are equally available, the actual pieces are often just assumed or are given without any explanation or application.

What are Embedding Layers

One of these key components are Embedding Layers. Rutger Ruizendaal wrote a fantastic article explaining Embedding layers in far more detail than I will here. My goal is to hopefully help others who struggle on how to make them applicable.

An embedding can be thought of a way to provide dimension to anything. Commonly it is applied with word embeddings such as GloVe and Word2Vec. I found this difficult to grasp as those two projects provide dictionaries with 100, 200 and 300 dimensions.

I read and understood that those large dimensions were designed to provide context to each word in the vocabulary. This is done to provide similarities and distinctions to better understand the word within the context of a sentence or phrase. For instance, the company “Apple” and fruit “apple” are the same word but depending on the context are completely different entities.

How to Apply Embedding Layers

When discussed you’ll often read that embeddings can be applied to almost everything. But what? It makes sense with vocabulary but can they also be applied to something such as food or an ingredient? Yes!

An embedding can be thought of as a list of floats that align to a set of features

Let’s say we wanted to classify a recipe as Breakfast, Lunch, Dinner or Desert. One way we might solve this is by using embeddings on each of the ingredients of a recipe. This would allow us to extrapolate the type of recipe beyond looking at the frequency of shared ingredients.

We could look at an ingredient by classifying features such as Citrus, Sweet, Sour, Tangy, Solidity, and Water Saturation. Most ingredients are not just one of these things, rather, they are a combination of all of them.

The radar chart to the left visualizes how I might embed the details of a tomato with the above features.

In code you would represent this as the following. The values themselves are relative to the feature they represent.

ingredients = {
  'tomato': [
    0.35, # Citrus
    0.54, # Sweet
    0.44, # Sour
    0.64, # Tangy
    0.62, # Solidity
    0.73 # Water Saturation
  ],
  'broccoli': [
    0.02, # Citrus
    0.09, # Sweet
    0.13, # Sour
    0.14, # Tangy
    0.87, # Solidity
    0.32 # Water Saturation
  ]
}

Why is this necessary

After giving this some thought I realized this is necessary since the neural network we are training (in my case at least) has no prior knowledge. It can only learn from the data we provide. This is a polar opposite to us. We have the ability to create ‘embeddings’ on everything we come into contact with. Whether it’s food, cars, software, television shows or even other people.

This is an important factor that I think can be easily overlooked. By understanding that everything is built up by many, small features we can learn how to represent our data to assist with training a successful neural network.

Off topic…

In my opinion as of 7/30/2018, the step to successful machine learning is trying to replicate the our own natural ability to collect information and make informed decisions. Neural networks never have 100% accuracy since we are modeling them after ourselves and we can expect nothing more considering our own limitations. I find this to be an interesting dynamic that generally isn’t shared with software development.