COV877: Special Module on Visual Computing: Differentiable Graphics For Vision and Learning

I Semester 2023-24


Differentiable graphics is an emerging family of techniques that enable graphics algorithms to be combined with machine learning for many cutting-edge applications in computer vision, robotics, fabrication, and other fields. While traditional computer graphics solves the forward problem of predicting the appearance and dynamics of objects given the description of a virtual scene, differentiable graphics also provides the derivatives of the output image or motion with respect to the scene parameters. This allows gradient-based techniques to be applied to optimize any desired objective, such as recovering the shape and material of a real object from one or more input photographs, or computing control parameters for robotic manipulation of soft materials. This course will cover the basic theoretical foundations and implementation techniques underlying differentiable algorithms for rendering and simulation, and survey some of the key recent advances in this emerging field.

Texts: We will begin by covering the material from a few recent courses presented at SIGGRAPH:

The rest of the reading material in this course will primarily consist of recent research papers. A list of such papers will be added shortly.

Prerequisites: Students are expected to have taken a course on at least one of the following topics: (i) computer graphics, (ii) computer vision, (iii) machine learning. If you are not sure whether you have adequate background, please contact the instructor.


The course is expected to consist of about 10-12 sessions of 1.5 hours each. They will start around mid-August and continue until late September.

The first few sessions will be lectures delivered by the instructor introducing the key ideas underlying differentiable rendering and simulation. After this, the remaining sessions will be conducted in seminar style, with student presentations and discussions of various recent research papers in this area. Students will also write a short report on one of the presented papers, summarizing its key ideas for a general computer science audience.

Lecture slides:

Paper presentations:

Paper Presenter Report
19 Sep
Liu et al., Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning, ICCV 2019 Siddhesh Kalekar Badrinath
Li et al., Differentiable Vector Graphics Rasterization for Editing and Learning, SIGGRAPH Asia 2020 Nikhil Kumar Rajeev Gupta
Gropp et al., IGR: Implicit Geometric Regularization for Learning Shapes, ICML 2020 Sajal Tyagi Lalit Meena
22 Sep
Mildenhall et al., Representing Scenes as Neural Radiance Fields for View Synthesis, ECCV 2020 Adarsh S. Menon Abhay P.S. Rathore
Fridovich-Keil et al., Plenoxels: Radiance Fields without Neural Networks, CVPR 2022 Avani Jain Roshan Raj
26 Sep
Müller et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, SIGGRAPH 2022 Badrinath Deepanshu
Li et al., Differentiable Monte Carlo Ray Tracing through Edge Sampling, SIGGRAPH Asia 2018 Rajeev Gupta Siddhesh Kalekar
Loubet et al., Reparameterizing Discontinuous Integrands for Differentiable Rendering, SIGGRAPH Asia 2019 Lalit Meena Nikhil Kumar Rajeev Gupta
29 Sep
Hu et al., DiffTaichi: Differentiable Programming for Physical Simulation, ICLR 2020 Abhay P.S. Rathore Sajal Tyagi
Jatavallabhula et al., gradSim: Differentiable Simulation for System Identification and Visuomotor Control, ICLR 2021 Roshan Raj Adarsh S. Menon
Du et al., DiffPD: Differentiable Projective Dynamics, SIGGRAPH 2022 Deepanshu Avani Jain Roshan Raj


Presentation format: Presenters should read their chosen paper in detail and prepare a 15-20 minute presentation on it. The presentation should clearly describe: the problem the paper is trying to solve, the fundamental challenges that make it a hard problem, the key ideas that make the paper work, and the tradeoffs and limitations inherent to the approach.

All other students are expected to have gone through the paper at least superficially, and prepared some pertinent questions related to it (these need not just be about how does the method work, but also why this approach was chosen, or why not use a different/simpler approach?). We will discuss these questions after the presentation.

Report format: Report writers are expected to read their assigned paper in detail, attend the associated presentation, take notes of the discussion, and finally write a 2-3 page report on the paper. The report may be based on the presentation but should not be restricted to it; it is a report on the paper, not the presentation. In particular, it should also incorporate any interesting points raised in the post-presentation discussion.

The report should be submitted within 1 week of the presentation. For uniformity, students should use this template based on the ACM publication format.

Grading: Following institute policy, a minimum of 80% marks are required for an A grade, and minimum 30% marks for D.

Audit policy: A minimum of 40% marks and 75% attendance is required for audit pass.

Attendance policy: Attendance lower than 75% will result in a one-grade penalty (e.g. A to A-).