I’ve just discovered an incredibly cool project on Github: DeOldify, which uses deep learning to automatically colorize old black & white photos. It’s not perfect, but what it’s able to do is pretty amazing, and improving rapidly.
In addition to ninja-level coding, author Jason Antic (@citnaj on Twitter) does a terrific job writing up how the algorithm works in the README file.
Essentially, his code uses a deep learning technique called a Generative Adversarial Network (GAN.) GANs consist of two components: a “Generator” and a “Discriminator.” In brief, the Generator (its own neural network) attempts to synthesize fakes or originals, and the Discriminator attempts to figure out if the submitted instance is “real” or fake. In this way, a Generator can be considered a “Counterfeiter” trying to fool the Discriminator, and the Discriminator can be thought of as “The Police” trying to catch the counterfeiter passing off a fake.
A zero-sum game then is played over and over, thousands of times, with each party trying to maximize its winnings, within some constraints.
Over time, the counterfeiter gets better and better at counterfeiting, until it can stand on its own and truly generate something that is pretty close to good output. To me, this is somewhat analogous to how children learn to better tell the truth when they experiment with fanciful lies in pre-adolescence. (Some, sadly, never learn the lesson fully.)
Back to DeOldify. The input/output is pretty impressive, with several examples provided at the link above.
Examples
I went through the Google Colab (Jupyter Notebook) powered harness developed by Matt Robinson, and input a few photos. Keep in mind that none of this involved any Photoshop work on my end:
Not perfect, but remarkable nonetheless!
GANs are perhaps the most interesting thing coming out of the work in Machine Learning these days.
I plan to spend some time with this project and see how well it does on some family photos from decades past, and perhaps build a web front-end to make it easy to try out. (It’s computationally expensive, so perhaps that front-end will need to rely upon donations.)
UPDATE: Jason Antic continues to make incredible progress on the model. The new images are much better than those above. Give it a whirl at https://deoldify.ai/!
Steve’s a Seattle-based entrepreneur and software leader, husband and father of three. He’s half Canadian, and east-coast born and raised. Steve has made the Pacific Northwest his home since 1991, when he moved here to work for Microsoft. He’s started and sold multiple Internet companies. Politically independent, he writes on occasion about city politics and national issues, and created Alignvote in the 2019 election cycle. He holds a BS in Applied Math (Computer Science) and Business from Carnegie Mellon University, a Masters in Computer Science from Stanford University, and an MBA from the Harvard Business School, where he graduated a George F. Baker Scholar. Steve volunteers when time allows with Habitat for Humanity, University District Food Bank, Technology Access Foundation (TAF) and other organizations in Seattle.