A Direct Way of Understanding Backpropagation and Gradient Descent

Summary: I believe that there are better representations of neural networks that aid in faster understanding of backpropagation and gradient descent. I find representing neural networks as equation graphs combined with the value at run-time helps engineers who don't have the necessary background in machine learning gets them up to speed faster. In this post, I generate a few graphs with Gorgonia to illustrate the examples.

Backpropagation has been explained to death. It really is as simple as applying the chain rule to compute gradients. However, in my recent adventures, I have found that this explanation isn’t intuitive to people who want to just get shit done. As part of my consultancy (hire me!) job* really, I need to pay the bills to get my startup funded , I provide a brief 1-3 day machine learning course to engineers who will maintain the algorithms that I designed. Whilst most of the work I do don’t use neural networks* people who think deep learning can solve every problem are either people with deep pockets aiming to solve a very general problem, or people who don't understand the hype. I have found that most businesses do not have problems that involves a lot of non-linearities. In fact a large majority of problems can be solved with linear regressions. , recently there was a case where deep neural networks were involved.

This blog post documents what I found was useful to explain neural networks, backpropagation and gradient descent. It’s not meant to be super heavy with theory - think of it as an enabler for an engineer to hit the ground running when dealing with deep networks. I may elide over some details, so some basic understanding/familiarity of neural networks is recommended.

[Read More]

Advent of Code/Go

Apparently today is the start of Advent of Code/Advent of Go. I’m not exactly sure what they are (AoC seems to be a bunch of puzzles, AoG seems to be a bunch of blog posts). Nonetheless I think this is a good excuse for me to get busy writing blog posts every day, if only to trigger my writers’ block away.

How To Make Money

Hey Chewxy, what do you think will happen if one day everyone decides to move their money onto a blockchain and no longer need banks?

That was a question that a friend asked me last week. I thought about the situation, gave some answers based on my what I understood of the world and the economy, while sketching out in broad strokes, what would happen. Essentially the conclusion was “civil unrest and war breaks out”* There were other conclusions too, I give the alternatives at the end of the blog post .

Then came time to organize Sydney Python. Due to clashing meetup dates with Data Science Sydney, Girl Geek Sydney and other groups, there was a dearth of speakers. So I stepped up and gave a talk based on the hypothetical question. Here are the slides:

The code can be found in this the economics simulation github repository.

[Read More]


I released Gorgonia on Thursday. Gorgonia is a library like Theano or TensorFlow, but mainly written in Go. It provides the necessary primitives for creating and executing neural networks and machine learning algorithms.

According to cloc, these are the stats:

[email protected]:~/workspace/goworkspace7/src/github.com/chewxy/gorgonia$ cloc .
     357 text files.
     321 unique files.                                          
     604 files ignored.

http://cloc.sourceforge.net v 1.60  T=0.83 s (296.5 files/s, 55471.5 lines/s)
Language                     files          blank        comment           code
Go                             219           6308           3924          30858
Assembly                        22            585            740           2128
C/C++ Header                     2             55             57            666
C                                2             17             39            458
SUM:                           245           6965           4760          34110

So, it’s a pretty huge library. But the original version is about 80,000 LoC (though most of the lines of codes were different experimental variations of assembly code). I managed to cut down 50,000 LoC to something more manageable. In this post I want to outline the release of Gorgonia, and share some of the reasoning regarding the design of the library, as well as go thru some of the weirdness found in the library.

If you’re interested, here’s the video (otherwise, skip to the meat):

And here are the slides:

[Read More]

Yes and No

I was teaching my partner some mandarin recently and I came to the conclusion that "yes" and "no" are very weird constructs of language. We were practicing one day, where I'd ask her questions in English and she'd reply in Mandarin. I asked her a yes/no question and she replied 不, to which I surprised myself by pointing out that 不 is ever only used in a negatory manner. People who know some Mandarin would interject and say, but there is 不(bù), 没(méi), and 无(wú) that can be used in stead of " [Read More]

On the memory alignment of Go slice values

TL;DR and Meta – I was playing around with some AVX instructions and I discovered that there were some problems. I then described the investigation process of the issue and discovered that this was because Go’s slices are not aligned to a 32 byte boundary. I proceed to describe the alignment issue and devised two solutions, of which I implemented one.

On Thursday I decided to do some additional optimization to my Go code. This meant writing some assembly to get some of the AVX goodness into my program (I once gave a talk on the topic of deep learning in Go, where I touched on this issue). I am no stranger to writing assembly in Go, but it’s not something I touch very often, so sometimes things can take longer to remember how to do them. This is one of them. So this blog post is mainly to remind myself of that.

The values in Go slices are 16-byte aligned. They are not 32 byte aligned.

[Read More]

On Binary Classification of Human Beings

Over the years I have come up with some fun ideas of binary classifying people. They say “those who can’t do, teach”. That’s a binary classification – teachers and doers. I once did something like that, with a longer elaboration: Hackers and Engineers Abstract Thinking Capabilities Some people have better abstract thinking capabilities than others. I’ll use an example that makes this a particularly dangerous thought. Consider two young girls, A and B, who are playing with Barbie dolls. [Read More]

Bloody Side Tracking Brain

This morning I woke up with Hungarian Dance No. 5 by Brahms stuck in my head. In my head, it’s a superiorly orchestrated, super high definition audio – much like sitting in the concert hall and being enveloped by the music of a live orchestra. I also have a very high quality copy of Hungarian Dance on my hard drive. I woke up at 5.45 am, went to gym, and returned at 7am and showered. [Read More]

Batman v Superman – A Quick Thought

I watched BvS today. I don’t know what to think about it. Overall, I think the movie was a bit of a mess. But I can’t seem to pinpoint why. Breaking it down by the standard things that people use to judge movies, there doesn’t seem to be anything wrong. Characters Character-wise, I liked it quite a bit. Superman is at his Superman-est. Batman is also amongst the most Batmanest Batman I’ve encountered. [Read More]

Not Enough

Do you sometimes feel like you’re not smart enough, not strong enough… not _ enough to do your pursuits?

At what point do you give up? I am so tired. The alternative – not pursuing what I want to do… is worse.