Jonathan Chung-Kuan Huang

machine learning, probabilistic inference, education, computer vision
Research Scientist
Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
P: (650) 248-4441

New Announcing the Auto-Arborist Dataset! The Auto Arborist dataset is a multiview fine-grained visual categorization dataset that contains over 2 million trees belonging to over 300 genus-level categories in 23 cities across the US and Canada built to foster the development of robust methods for large-scale urban forest monitoring. The dataset was initially released as part of a CVPR 2022 publication. Data use and access information here.

New I recently gave a series of lectures at UW's Computer Vision course (CSEP576) on deep learning for computer vision. Here are the lectures slides on Advanced CNNs, Object Detection, and Dense Prediction (e.g. Segmentation).

New Announcing the Tensorflow Object Detection API! I’m happy to publicly share the codebase that I’ve been developing at Google along with many amazing collaborators. Our codebase was also featured in Google blogs for Research, a Cloud Platform ML, and Open Source as well as Techcrunch and VentureBeat articles.

New Check out the recently released results of the 2016 COCO Visual Recognition Challenge! The (Google Research and Machine Intelligence) object detection team (of which I was the tech lead) won 1st place in the COCO detection challenge with 41.3% mAP. To give a comparison of how difficult it was, none of the other teams were able to beat last year's best result of 37.1% mAP. For more information, check out my talk slides. Huge thanks to an extraordinary team (Alireza Fathi, Ian Fischer, Sergio Guadarrama, Anoop Korattikara, Kevin Murphy, Vivek Rathod, Yang Song, Chen Sun, Zbigniew Wojna, Menglong Zhu) and Google infrastructure!

New Check out the Techcrunch article on some of my recent deep learning projects at Google Research.

New Astrophotography excursion: here's a panoramic photo that Avneesh Sud, Nathan Silberman and I took of the Milky Way arch off Hole-in-the-Wall trail in Mojave National Preserve. The galactic center lies to the right with thunderstorms covering the lightdomes of Vegas to the left and Bullhead city in the center, which were aglow with dramatic lightning flashes. I'm also quite proud of the timelapse that we created the same night :) Best viewed full screen at 1080p.

New Here's a panoramic photo that I took recently of Queenstown, New Zealand.

I am a staff research scientist at Google working on machine learning and computer vision. I am particularly interested in core scene and video understanding (e.g. object detection, instance segmentation, tracking), weak/self supervised learning and generative models. I live in Seattle and work from Google's Fremont office.

Prior to Google, I was a postdoctoral fellow working in the Computer Science Department at Stanford University and was supported by an NSF/CRA CI (Computing Innovations) fellowship. At Stanford I was a member of the Geometric Computation Group which is headed by Leonidas Guibas. I was also part of the Lytics Lab, a multidisciplinary group focused on Learning Analytics.

I received a Ph.D. in Robotics from the School of Computer Science at Carnegie Mellon University in 2011, where I worked with Carlos Guestrin. During graduate school, I was fortunate enough to spend two happy summers interning in Seattle, first with Intel Research working with Ali Rahimi, then at Microsoft Research working with Ashish Kapoor.

Before coming to CMU, I studied math (also) at Stanford University. And before Stanford, I attended Oakton High School in Vienna, Virginia, and for a time, also Lynbrook High School in San Jose, California.

Here is an "official" bio and photo.

My research interests in wordle form. The right wordle is generated from my most recent publications on online education and the left wordle is generated from my work on probabilistic inference and learning with combinatorially structured data. Note as of May 2018: these wordles are outdated and are not the best reflection of my research activities. And given that wordles themselves are quite outdated, I probably won't be refreshing these. Just keep an eye on my recent papers to get a sense of my recent work :)

I am interested in theoretical and applied problems in machine learning. My main interests lie in designing computationally efficient probabilistic reasoning and learning algorithms which allow computers to deal with the uncertainty and complexity inherent in real world data. My work has focused on tackling applications whose mathematical abstractions involve probabilistic reasoning with combinatorially structured objects such as matchings, rankings, and trees. These problems are challenging both statistically and computationally due to structural constraints (like mutual exclusivity) which cause interactions between objects that traditional techniques in machine learning have been ill-equipped to handle. Portions of my work thus address:

  • Compact, probabilistic formulations for reasoning jointly with large collections of structured data,
  • Efficient algorithms for reasoning and learning that exploit problem structure,
  • Theoretical analyses of computational and statistical complexity as well as approximation quality.

While being dedicated to pushing on core research problems, I am also committed to problems with real world applications and impact. My past work has contributed solutions to a variety of applications such as predicting preference over webpages and political elections, tracking with camera networks, and reconstructing temporal orderings of events (such as the onset of symptoms in neurodegenerative diseases) from noisy and incomplete data.

I now focus most of my energies on applications with educational impact. The recent surge in popularity of massive open online courses (MOOCs), with platforms such as Coursera and EdX, has made it possible for almost anyone to take free university courses. However while new technologies allow for scalable content delivery, we remain limited in our ability to scalably evaluate and give feedback for open-ended assignments. I approach these challenges fundamentally as machine learning (ML) problems, in which we can leverage the massive datasets now collected by online learning platforms. My work has thus focused on ML-driven education and has contributed algorithms for giving feedback in MOOCs via crowdsourcing or semi-automated methods.

Efficient inference in occlusion-aware generative models of images,
Jonathan Huang, Kevin Murphy.
In The 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Las Vegas, Nevada, June, 2016.
Generation and Comprehension of Unambiguous Object Descriptions,
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, Kevin Murphy.
In The 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Las Vegas, Nevada, June, 2016.
Detecting events and key actors in multi-person videos,
Vignesh Ramanathan, Jonathan Huang, Sami Abu-El-Haija, Alexander Gorban, Kevin Murphy, Li Fei-Fei.
In The 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Las Vegas, Nevada, June, 2016.
Multiple Orderings of Events in Disease Progression,
Alexandra Young, Neil Oxtoby, Jonathan Huang, Razvan Marinescu, Pankag Daga, David Cash, Nick Fox, Sebastien Ourselin, Daniel Alexander.
In Information Processing in Medical Imaging (IPMI),
Isle of Skye, Scotland, 2015.
A database of vocal tract resonance trajectories for research in speech processing,
Li Deng, Xiaodong Cui, Robert Pruvenok, Jonathan Huang, Safiyy Momen, Yanyi Chen, Abeer Alwan.
In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2006),
pp. 60--63, Toulouse, France, 2006.
Tensorflow Object Detection API
Jonathan Huang, Vivek Rathod, Derek Chow, Chen Sun, Menglong Zhu, Matthew Tang, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, Jasper Uijlings, Viacheslav Kovalevskyi, and Kevin Murphy.
Open source framework built on top of TensorFlow that makes it easy to construct, train and deploy object detection models.
Jonathan Huang
Python routines for fitting and simulating from a generalized Mallows model. Learning algorithms implemented for both full and partial rankings
PROPS: Probabilistic Reasoning on Permutations toolbox
Jonathan Leonard Long, Jonathan Huang,
C++/Python library for reasoning/learning with distributions on permutations.
Littlewood-Richardson rule
Jonathan Huang
A matlab implementation of the Littlewood-Richardson rule.

(LDA) Latent Dirichlet Allocation
Jonathan Huang and Tomasz Malisiewicz,
An implementation of the mean field inference/learning algorithms from Blei et al. (2003)

Sample output on 20 Newsgroups dataset: [Link]

Note: Every now and then, Tomasz and I get emails from people about this code. While we're always happy to help out, I would like to point out that we wrote this code many years ago. Nowadays it is much more popular (and effective) to use collapsed samplers or online algorithms over the mean field + variational EM algorithm that was proposed in the first LDA paper.

Jonathan Huang and Tomasz Malisiewicz,
Code for fitting a Hierarchical Logistic Normal distribution.
There is also a Romanian translation by Maxim Petrenko - Software blogger

Deep learning for structured image understanding

Taller Deep Learning
CIMAT, Guanajuato, Mexico


Adaptive Fourier-Domain Inference on the Symmetric Group

Algebraic Methods in Machine Learning Workshop, NIPS '08
Whistler, Canada


Probability Distributions on Permutations: Compact Representations and Inference

Machine Learning Lunch Seminar, 2008
Carnegie Mellon University


Exploiting Independence and Its Generalizations for Reasoning about Permutation Data

Machine Learning Lunch Seminar, 2010
Carnegie Mellon University


Politics, Preferences and Permutations: Probabilistic Reasoning with Rankings

Seminar, 2011
Microsoft Research Cambridge (UK)


Data Driven Student Feedback for Programming Intensive MOOCs

MSR Latin American Faculty Summit, 2014
Viña del Mar, Chile

talk slides
Check out our visualization of 40,000 Octave/Matlab implementations of linear regression! This is part of the Codewebs project for analyzing and providing detailed feedback to students in a programming based MOOC with Chris Piech, Andy Nguyen, and Leo Guibas. Data from Andrew Ng's course on Machine Learning offered through Coursera.
Also check out Ben Lorica's blog post, and Hal Hodson's article at New Scientist about our work!
ICML 2015 Workshop on Machine Learning for Education
I co-organized the ICML 2015 workshop on Machine Learning for Education with Richard Baraniuk, Emma Brunskill, Mihaela van der Schaar, Mike Mozer, Christoph Studer, Andrew Lan.
Those Chatty Seniors!
Read my post at Stanford Online's Signal blog (with Jane Manning and Marc Sanders) on: Those Chatty Seniors! in which we analyze and discuss the demographics of MOOC forum posters. (the tl;dr is that older people talk more). The Stanford Daily also covered our work in this article.
NIPS 2013 Workshop on Data Driven Education
I co-organized a NIPS 2013 workshop on Data Driven Education with Sumit Basu and Kalyan Veeramachaneni.
NIPS 2009 Learning with Orderings workshop
I co-organized a NIPS 2009 workshop on Learning with Orderings with Tiberio Caetano, Carlos Guestrin, Risi Kondor, Guy Lebanon, and Marina Meila.
"Bag of words" art installation at Gates-Hillman complex.
(joint work with Khalid El-Arini, Sue Ann Hong, Joseph Gonzalez)
Learning and Inference in Vision: from Features to Scene Understanding
Tomasz Malisiewicz and I gave a tutorial on vision at the MLD Student Research Symposium on November 13 (2011).
Probabilistic Reasoning with Permutations: A Fourier-Theoretic Approach ,
Jonathan Huang,
My thesis proposal document.
Hierarchical Logistic Normal parameter estimation,
Jonathan Huang, Tomasz Malisiewicz,
A project for Alyosha Efros's class on Learning based methods in computer vision. See our application to object recognition.
Maximum Likelihood Estimation of Dirichlet Distributions ,
Jonathan Huang,
Notes on several ways to numerically find the MLE of a Dirichlet Distribution. This was done for a Math Fundamentals for Robotics course taught by Mike Erdmann.
Sperner's Lemma ,
Jonathan Huang,
Some theorems/corollaries of Sperner's Lemma that I collected for a combinatorics class. Sperner is an easy combinatorial fact about labelings on a simplicial complex, but it has several surprising applications in topology and analysis. The famous Brouwer fixed point theorem, and the fundamental theorem of algebra are two of the examples that I discuss.
Notes on the Kalman Filter ,
Jonathan Huang,
A derivation of the Kalman filter updates. The notes try to mostly follow the development given by Drew Bagnell in the Statistical Techniques for Robotics class at CMU.
Cup Products in Computational Topology ,
Jonathan Huang,
Senior Honors Thesis (advisor: Gunnar Carlsson). We show an application of topological persistence to computing invariants related to the cohomology (cup product structure) of a finite simplicial complex.