Apr 9, 2018
Jolley Hall, Room 309
On the Redundancy in Deep Learning
As one of the most popular techniques in artificial intelligence, deep learning has been widely used for computer vision, natural language processing, robotics, etc. On specific tasks, it has achieved or even surpassed human-level performance. However, the strong generalization ability of deep learning is still not well understood by the community. Why can such over-parameterized neural networks generalize well?
In this talk, I will show that deep networks have considerable redundancy - although they are parameterized by millions of parameters, they may not use them effectively. I will first introduce a simple algorithm that reveals the parameter redundancy in deep residual networks and show that redundancy indeed helps improving generalization. From the practical viewpoint, redundancy is problematic as it increases computational cost. I will then present a novel network architecture that has less redundancy yet with strong generalization ability. Finally, I will introduce an adaptive evaluation method that reduces redundant computation for each individual sample.
Gao Huang is a postdoctoral researcher at the Department of Computer Science at Cornell University, where he works on deep learning and computer vision with Professor Kilian Weinberger. Before joining Cornell, he did his PhD at Tsinghua University. His work on DenseNet won the Best Paper Award at CVPR 2017.