Sapir-Whorf on Programming Languages

Or: How I Got Blindsided By Syntax

In my previous blog post I had a brief mention about the Sapir-Whorf hypothesis, and how it blind-sided me to an obvious design. In this blog post I’ll retell yet another story of how Sapir-Whorf blind-sided me yet again - just moments, actually. I’m getting sick of this shit, yo. Sapir-Whorf, Briefly Briefly, the Sapir-Whorf hypothesis states that the language you speak influences the way you think. The proper term for it is “linguistic relativity”. [Read More]

Tensor Refactor: A Go Experience Report

May Contain Some Thoughts on Generics in Go

I recently finished a major refactor of `tensor`, which is a package for generic multidimensional arrays in Go. In this blog post I will recount the refactoring process, and why certain decisions were made. Further more I will also share some thoughts with regards to generics in Go while trying not to sound like a complete prat.

There has been major refactors done to the tensor subpackage in Gorgonia - a Go library for deep learning purposes (think of it as TensorFlow or PyTorch for Golang). It’s part of a list of fairly major changes to the library as it matures more. While I’ve used it for a number of production ready projects, an informal survey of found that the library was still a little difficult to use (plus, it’s not used by any famous papers so people are generally more resistant to learning it than say, Tensorflow or PyTorch).

Along the way in the process of refactoring this library, there were plenty of hairy stuff (like creating channels of negative length), and I learned a lot more about building generic data structures that I needed to.

[Read More]

21 Bits Ought to Be Enough for (Everyday) English

Or, Bitpacking Shennanigans

I was working on some additional work on lingo, and preparing it to be moved to go-nlp. One of the things I was working on improving is the corpus package. The goal of package corpus is to provide a map of word to ID and vice versa. Along the way package lingo also exposes a Corpus interface, as there may be other data structures which showcases corpus-like behaviour (things like word embeddings come to mind).

When optimizing and possibly refactoring the package(s), I like to take stock of all the things the corpus package and the Corpus interface is useful for, as this would clearly affect some of the type signatures. This practice usually involves me reaching back to the annals of history and looking at the programs and utilities I had written, and consolidate them.

One of the things that I would eventually have a use for again is n-grams. The current n-gram data structure that I have in my projects is not very nice and I wish to change that.

[Read More]

The Double Blind Monty Hall Problem

Last night as I was preparing today's lunch, I ran into an interesting real life scenario that is a variant of the Monty Hall problem. As I thought more about the subject I became more and more convinced that the probability of choosing the right door by switching was 0.5 instead of 0.6667. I even sketched out a Bayes theorem proof of why that is. Then I realized an assumption that the original Monty Hall problem had. And so in this post, I'll sketch out two variations of the Monty Hall problem.


The picture above is of my lunch today: three muffins baked with MyProtein’s muffin mix. Two of them contain raisins, and one of them contains chocolate chips. I had forgotten which is which. I personally prefer raisins, as the chocolate chips had sunk to the bottom of the pan making a gooey mess that sticks to the muffin papers during the baking process. An initial thought that I had was concerning the probability of choosing a subsequent raisin muffin after I had eaten one. Naturally, in scenarios where there are 3 unknowns and one was revealed, my thoughts get pulled towards the Monty Hall problem.

[Read More]

The Handmaid's Tale

Or, How to Successfully Oppress Women

Hulu just released three episodes of The Handmaid’s Tale, an adaptation of Margaret Atwood’s book of the same name. I binged watched it over the weekend, and I had some difficulty being immersed into it. There was something about the world that didn’t sit quite right with me in this adaptation, but I couldn’t quite put a finger on it. Eventually of course, I figured it out.

The story is set after the fall of the United States into Gilead. Not much is known about the world at this point, and so much is still up for speculation. However we the audience are entreated to these details:

  • Pollution and toxicity is so bad that it caused fertility rates to drop.
  • Congress was slaughtered, martial law was enacted.
  • A theocracy hijacked the government under martial law.
  • Communications between the populace and government is reduced - it would appear that the Internet doesn’t exist.
  • Swift changes were made to the laws of the land, leading up to the scenarios we see in the show.

So what was it that made me unable to pay attention to the world building of the show? The speed at which things fell into place - it all happened within a few years of Offred’s lifetime.

[Read More]

Deep Learning from Scratch in Go

Part 1: Equations Are Graphs

Note: This post was originally published at GopherData. This blog post is the exact same copy.

Summary: In this post I talk about the basics of computations, then I go on to show parallels between Go programs and mathematical equations. Lastly I will attempt to persuade you that there is a point in using Go for doing deep learning.

Welcome to the first part of many about writing deep learning algorithms in Go. The goal of this series is to go from having no knowledge at all to implementing some of the latest developments in this area.

Deep learning is not new. In fact the idea of deep learning was spawned in the early 1980s. What’s changed since then is our computers - they have gotten much much more powerful. In this blog post we’ll start with something familiar, and edge towards building a conceptual model of deep learning. We won’t define deep learning for the first few posts, so don’t worry so much about the term.

There are a few terms of clarification to be made before we begin proper. In this series, the word “graph” refers to the concept of graph as used in graph theory. For the other kind of “graph” which is usually used for data visualization, I’ll use the term “chart”.

[Read More]

New Blog

For about a month now, this blog has been a static site generated by Hugo. There had been a few issues with my wordpress installation at my previous host, and it left me unable to publish blog posts - they end up in “limbo” - some weird combination of plugins and tables. I had to hunt down all the posts I had published since the problem occured, and manually restore them. [Read More]

What To Test

You’re in a rush. Your product demo was due three months ago. And you still don’t have good unit tests. What do you do? You prioritize.

You wrote a very complex piece of machinery with a lot of moving parts. You sorta know how they all fit together in your mind but there isn’t any good know if the program you wrote actually works the way you want it. The solution: unit testing. But how to test and where to start? What do you do? You prioritize.

This post is about how I prioritize what to test. I know how I write my programs, and I write them in a particular way. Feel free to adapt the following to your own process.

Since my program is made mostly of functions, I prioritize my tests by analyzing the function calls of the package/program. Some functions are called many times by different callers, some functions are top functions which calls other functions and outside of the package the top functions are never called. This forms a hierarchy of sorts - some calls are simply more important than others. Since I write mostly in Go nowadays, I’ll use Go as an example. The Go toolchain actually provide us with a lot of useful tools to analyze and prioritize tasks. The most important tool is the callgraph program. This is the invocation of the spell:

callgraph -algo=static -format=graphviz $(go list -tags=release -f '{{.GoFiles}}' | sed -ne 's/\[//p' | sed -ne 's/\]//p') | grep -P '\-\> "(PACKAGENAME|\(\*?PACKAGENAME\.)' | uniq >

[Read More]

A Direct Way of Understanding Backpropagation and Gradient Descent

Summary: I believe that there are better representations of neural networks that aid in faster understanding of backpropagation and gradient descent. I find representing neural networks as equation graphs combined with the value at run-time helps engineers who don't have the necessary background in machine learning gets them up to speed faster. In this post, I generate a few graphs with Gorgonia to illustrate the examples.

Backpropagation has been explained to death. It really is as simple as applying the chain rule to compute gradients. However, in my recent adventures, I have found that this explanation isn’t intuitive to people who want to just get shit done. As part of my consultancy (hire me!) job* really, I need to pay the bills to get my startup funded , I provide a brief 1-3 day machine learning course to engineers who will maintain the algorithms that I designed. Whilst most of the work I do don’t use neural networks* people who think deep learning can solve every problem are either people with deep pockets aiming to solve a very general problem, or people who don't understand the hype. I have found that most businesses do not have problems that involves a lot of non-linearities. In fact a large majority of problems can be solved with linear regressions. , recently there was a case where deep neural networks were involved.

This blog post documents what I found was useful to explain neural networks, backpropagation and gradient descent. It’s not meant to be super heavy with theory - think of it as an enabler for an engineer to hit the ground running when dealing with deep networks. I may elide over some details, so some basic understanding/familiarity of neural networks is recommended.

[Read More]