Publications
Neural Holography with Camera-in-the-Loop Training
Holographic displays promise unprecedented capabilities for direct-view displays as well as virtual and augmented reality applications. However, one of the biggest challenges for computer-generated holography (CGH) is the fundamental tradeoff between algorithm runtime and achieved image quality, which has prevented high-quality holographic image synthesis at fast speeds. Moreover, the image quality achieved by most holographic displays is low, due to the mismatch between the optical wave propagation of the display and its simulated model. Here, we develop an algorithmic CGH framework that achieves unprecedented image fidelity and real-time framerates. Our framework comprises several parts, including a novel camera-in-the-loop optimization strategy that allows us to either optimize a hologram directly or train an interpretable model of the optical wave propagation and a neural network architecture that represents the first CGH algorithm capable of generating full-color high-quality holographic images at 1080p resolution in real time.
Citation
Peng, Y., Choi, S., Padmanaban, N., & Wetzstein, G. (2020). Neural Holography with Camera-in-the-Loop Training. ACM SIGGRAPH Asia (Transactions on Graphics), 39(6), 1–14.
BibTeX
@article{Peng:2020:Neural,
title={Neural Holography with Camera-in-the-Loop Training},
author={Peng, Yifan and Choi, Suyeon and Padmanaban, Nitish and Wetzstein, Gordon},
journal={ACM Transactions on Graphics (SIGGRAPH Asia)},
year={2020},
volume={39},
number={6},
pages={1--14}
}
Factored Occlusion: Single Spatial Light Modulator Occlusion-Capable Optical See-Through Augmented Reality Display
Occlusion is a powerful visual cue that is crucial for depth perception and realism in optical see-through augmented reality (OST-AR). However, existing OST-AR systems additively overlay physical and digital content with beam combiners — an approach that does not easily support mutual occlusion, resulting in virtual objects that appear semi-transparent and unrealistic. In this work, we propose a new type of occlusion-capable OST-AR system. Rather than additively combining the real and virtual worlds, we employ a single digital micromirror device (DMD) to merge the respective light paths in a multiplicative manner. This unique approach allows us to simultaneously block light incident from the physical scene on a pixel-by-pixel basis while also modulating the light emitted by a light-emitting diode (LED) to display digital content. Our technique builds on mixed binary/continuous factorization algorithms to optimize time-multiplexed binary DMD patterns and their corresponding LED colors to approximate a target augmented reality (AR) scene. In simulations and with a prototype benchtop display, we demonstrate hard-edge occlusions, plausible shadows, and also gaze-contingent optimization of this novel display mode, which only requires a single spatial light modulator.
Citation
Krajancich, B.*, Padmanaban, N.*, & Wetzstein, G. (2020). Factored Occlusion: Single Spatial Light Modulator Occlusion-Capable Optical See-Through Augmented Reality Display. IEEE Transactions on Visualization and Computer Graphics, 26(5), 1871–1879.
BibTeX
@article{Krajancich:2020:Occlusion,
title={Factored Occlusion: Single Spatial Light Modulator Occlusion-Capable Optical See-Through Augmented Reality Display},
author={Krajancich, Brooke and Padmanaban, Nitish and Wetzstein, Gordon},
journal={IEEE Transactions on Visualization and Computer Graphics},
year={2020},
volume={26},
number={5},
pages={1871--1879}
}
Holographic Near-Eye Displays Based on Overlap-Add Stereograms
Holographic near-eye displays are a key enabling technology for virtual and augmented reality (VR/AR) applications. Holographic stereograms (HS) are a method of encoding a light field into a hologram, which enables them to natively support view-dependent lighting effects. However, existing HS algorithms require the choice of a hogel size, forcing a tradeoff between spatial and angular resolution. Based on the fact that the short-time Fourier transform (STFT) connects a hologram to its observable light field, we develop the overlap-add stereogram (OLAS) as the correct method of “inverting” the light field into a hologram via the STFT. The OLAS makes more efficient use of the information contained within the light field than previous HS algorithms, exhibiting better image quality at a range of distances and hogel sizes. Most remarkably, the OLAS does not degrade spatial resolution with increasing hogel size, overcoming the spatio-angular resolution tradeoff that previous HS algorithms face. Importantly, the optimal hogel size of previous methods typically varies with the depth of every object in a scene, making the OLAS not only a hogel size–invariant method, but also nearly scene independent. We demonstrate the performance of the OLAS both in simulation and on a prototype near-eye display system, showing focusing capabilities and view-dependent effects.
Citation
Padmanaban, N., Peng, Y., & Wetzstein, G. (2019). Holographic Near-Eye Displays Based on Overlap-Add Stereograms. ACM SIGGRAPH Asia (Transactions on Graphics), 38(6), 214.
BibTeX
@article{Padmanaban:2019:OLAS,
title={Holographic Near-Eye Displays Based on Overlap-Add Stereograms},
author={Padmanaban, Nitish and Peng, Yifan and Wetzstein, Gordon},
journal={ACM Transactions on Graphics (SIGGRAPH Asia)},
year={2019},
volume={38},
number={6},
pages={214:1--214:13}
}
Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes
As humans age, they gradually lose the ability to accommodate, or refocus, to near distances because of the stiffening of the crystalline lens. This condition, known as presbyopia, affects nearly 20% of people worldwide. We design and build a new presbyopia correction, autofocals, to externally mimic the natural accommodation response, combining eye tracker and depth sensor data to automatically drive focus-tunable lenses. We evaluated 19 users on visual acuity, contrast sensitivity, and a refocusing task. Autofocals exhibit better visual acuity when compared to monovision and progressive lenses while maintaining similar contrast sensitivity. On the refocusing task, autofocals are faster and, compared to progressives, also significantly more accurate. In a separate study, a majority of 23 of 37 users ranked autofocals as the best correction in terms of ease of refocusing. Our work demonstrates the superiority of autofocals over current forms of presbyopia correction and could affect the lives of millions.
Citation
Padmanaban, N., Konrad, R., & Wetzstein, G. (2019). Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes. Science Advances, 5(6), eaav6187.
BibTeX
@article{Padmanaban:2019:Autofocals,
title={Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes},
author={Padmanaban, Nitish and Konrad, Robert and Wetzstein, Gordon},
journal={Science Advances},
year={2019},
volume={5},
number={6},
pages={eaav6187}
}
Towards a Machine-Learning Approach for Sickness Prediction in 360° Stereoscopic Videos
Virtual reality systems are widely believed to be the next major computing platform. There are, however, some barriers to adoption that must be addressed, such as that of motion sickness — which can lead to undesirable symptoms including postural instability, headaches, and nausea. Motion sickness in virtual reality occurs as a result of moving visual stimuli that cause users to perceive self-motion while they remain stationary in the real world. There are several contributing factors to both this perception of motion and the subsequent onset of sickness, including field of view, motion velocity, and stimulus depth. We verify first that differences in vection due to relative stimulus depth remain correlated with sickness. Then, we build a dataset of stereoscopic 3D videos and their corresponding sickness ratings in order to quantify their nauseogenicity, which we make available for future use. Using this dataset, we train a machine learning algorithm on hand-crafted features (quantifying speed, direction, and depth as functions of time) from each video, learning the contributions of these various features to the sickness ratings. Our predictor generally outperforms a naïve estimate, but is ultimately limited by the size of the dataset. However, our result is promising and opens the door to future work with more extensive datasets. This and further advances in this space have the potential to alleviate developer and end user concerns about motion sickness in the increasingly commonplace virtual world.
Citation
Padmanaban, N.*, Ruban, T.*, Sitzmann, V., Norcia, A. M., & Wetzstein, G. (2018). Towards a Machine-Learning Approach for Sickness Prediction in 360° Stereoscopic Videos. IEEE Transactions on Visualization and Computer Graphics, 24(4), 1594–1603.
BibTeX
@article{Padmanaban:2018:Sickness,
title={Towards a Machine-Learning Approach for Sickness Prediction in 360$^\circ$ Stereoscopic Videos},
author={Padmanaban, Nitish and Ruban, Timon and Sitzmann, Vincent and Norcia, Anthony M and Wetzstein, Gordon},
journal={IEEE Transactions on Visualization and Computer Graphics},
year={2018},
volume={24},
number={4},
pages={1594--1603}
}
Accommodation-Invariant Computational Near-Eye Displays
Although emerging virtual and augmented reality (VR/AR) systems can produce highly immersive experiences, they can also cause visual discomfort, eyestrain, and nausea. One of the sources of these symptoms is a mismatch between vergence and focus cues. In current VR/AR near-eye displays, a stereoscopic image pair drives the vergence state of the human visual system to arbitrary distances, but the accommodation, or focus, state of the eyes is optically driven towards a fixed distance. In this work, we introduce a new display technology, dubbed accommodation-invariant (AI) near-eye displays, to improve the consistency of depth cues in near-eye displays. Rather than producing correct focus cues, AI displays are optically engineered to produce visual stimuli that are invariant to the accommodation state of the eye. The accommodation system can then be driven by stereoscopic cues, and the mismatch between vergence and accommodation state of the eyes is significantly reduced. We validate the principle of operation of AI displays using a prototype display that allows for the accommodation state of users to be measured while they view visual stimuli using multiple different display modes.
Citation
Konrad, R., Padmanaban, N., Molner, K., Cooper, E. A., & Wetzstein, G. (2017). Accommodation-Invariant Computational Near-Eye Displays. ACM SIGGRAPH (Transactions on Graphics), 36(4), 88.
BibTeX
@article{Konrad:2017:Accommodation,
title={Accommodation-Invariant Computational Near-Eye Displays},
author={Konrad, Robert and Padmanaban, Nitish and Molner, Keenan and Cooper, Emily A and Wetzstein, Gordon},
journal={ACM Transactions on Graphics (TOG)},
volume={36},
number={4},
pages={88:1--88:12},
year={2017}
}
Optimizing Virtual Reality for All Users through Gaze-Contingent and Adaptive Focus Displays
From the desktop to the laptop to the mobile device, personal computing platforms evolve over time. Moving forward, wearable computing is widely expected to be integral to consumer electronics and beyond. The primary interface between a wearable computer and a user is often a near-eye display. However, current generation near-eye displays suffer from multiple limitations: they are unable to provide fully natural visual cues and comfortable viewing experiences for all users. At their core, many of the issues with near-eye displays are caused by limitations in conventional optics. Current displays cannot reproduce the changes in focus that accompany natural vision, and they cannot support users with uncorrected refractive errors. With two prototype near-eye displays, we show how these issues can be overcome using display modes that adapt to the user via computational optics. By using focus-tunable lenses, mechanically actuated displays, and mobile gaze-tracking technology, these displays can be tailored to correct common refractive errors and provide natural focus cues by dynamically updating the system based on where a user looks in a virtual scene. Indeed, the opportunities afforded by recent advances in computational optics open up the possibility of creating a computing platform in which some users may experience better quality vision in the virtual world than in the real one.
Citation
Padmanaban, N., Konrad, R., Stramer, T., Cooper, E. A., & Wetzstein, G. (2017). Optimizing Virtual Reality for All Users through Gaze-Contingent and Adaptive Focus Displays. Proceedings of the National Academy of Sciences of the United States of America, 114(9), 2183–2188.
BibTeX
@article{Padmanaban:2017:Optimizing,
title={Optimizing Virtual Reality for All Users through Gaze-Contingent and Adaptive Focus Displays},
author={Padmanaban, Nitish and Konrad, Robert and Stramer, Tal and Cooper, Emily A and Wetzstein, Gordon},
journal={Proceedings of the National Academy of Sciences},
year={2017},
volume={114},
number={9},
pages={2183--2188}
}
Abstracts
Neural Holography
Holographic displays promise unprecedented capabilities for direct-view displays as well as virtual and augmented reality (VR/AR) applications. However, one of the biggest challenges for computer-generated holography (CGH) is the fundamental tradeoff between algorithm runtime and achieved image quality, which has prevented high-quality holographic image synthesis at fast speeds. Moreover, the image quality achieved by most holographic displays is low, due to the mismatch between physical light transport of the display and its simulated model. Here, we develop an algorithmic CGH framework that achieves unprecedented image fidelity and real-time framerates. Our framework comprises several parts, including a novel camera-in-the-loop optimization strategy that allows us to either optimize a hologram directly or train an interpretable model of the physical light transport and a neural network architecture that represents the first CGH algorithm capable of generating full-color holographic images at 1080p resolution in real time.
Citation
Peng, Y., Choi, S., Padmanaban, N., Kim, J., & Wetzstein, G. (2020, August). Neural Holography. In ACM SIGGRAPH 2020 Emerging Technologies (p. 8). ACM.
BibTeX
@inproceedings{Peng:2020:NeuralETech,
title={Neural Holography},
author={Peng, Yifan and Choi, Suyeon and Padmanaban, Nitish and Kim, Jonghyun and Wetzstein, Gordon},
booktitle={ACM SIGGRAPH 2020 Emerging Technologies},
year={2020},
month={August},
pages={8:1--8:2},
organization={ACM}
}
Autofocals: Gaze-Contingent Eyeglasses for Presbyopes
Presbyopia, the loss of accommodation due to the stiffening of the crystalline lens, affects nearly 20% of the population worldwide. Traditional forms of presbyopia correction use fixed focal elements that inherently trade off field of view or stereo vision for a greater range of distances at which the wearer can see clearly. However, none of these offer the same natural refocusing enjoyed in youth. In this work, we built a new presbyopia correction, dubbed Autofocals, which externally mimics the natural accommodation response by combining data from eye trackers and a depth sensor, and then automatically drives focus-tunable lenses. In our testing, wearers generally reported that the Autofocals compare favorably with their own current corrective eyewear.
Citation
Padmanaban, N., Konrad, R., & Wetzstein, G. (2018, August). Autofocals: Gaze-Contingent Eyeglasses for Presbyopes. In ACM SIGGRAPH 2018 Emerging Technologies (p. 3). ACM.
BibTeX
@inproceedings{Padmanaban:2018:Autofocals,
title={Computational Focus-Tunable Near-Eye Displays},
author={Padmanaban, Nitish and Konrad, Robert and Wetzstein, Gordon},
booktitle={ACM SIGGRAPH 2018 Emerging Technologies},
year={2018},
month={August},
pages={3:1--3:2},
organization={ACM}
}
Evaluation of Accommodation Responses to Monovision for Virtual Reality
Emerging virtual and augmented reality (VR/AR) systems can produce highly immersive experiences, but also induce visual discomfort, eyestrain, and nausea for some users. One of the sources of these symptoms is the lack of natural focus cues in all current VR/AR near-eye displays. These displays project stereoscopic image pairs, driving the vergence state of the eyes to arbitrary distances. However, the accommodation, or focus state of the eyes, is optically driven to a fixed distance. In this work, we empirically evalaute monovision: a simple, yet unconventional method for potentially driving the accommodation state of the eyes to two distances by allowing each eye to drive focus to a different distance.
Citation
Padmanaban, N., Konrad, R., & Wetzstein, G. (2017, June). Evaluation of Accommodation Response to Monovision for Virtual Reality. In 3D Image Acquisition and Display: Technology, Perception and Applications (pp. DM2F.3). Optical Society of America.
BibTeX
@inproceedings{Padmanaban:2017:Evaluation,
title={Evaluation of Accommodation Response to Monovision for Virtual Reality},
author={Padmanaban, Nitish and Konrad, Robert and Wetzstein, Gordon},
booktitle={3D Image Acquisition and Display: Technology, Perception and Applications},
year={2017},
month={June},
pages={DM2F.3},
organization={Optical Society of America}
}
Computational Focus-Tunable Near-Eye Displays
Immersive virtual and augmented reality systems (VR/AR) are entering the consumer market and have the potential to profoundly impact our society. Applications of these systems range from communication, entertainment, education, collaborative work, simulation and training to telesurgery, phobia treatment, and basic vision research. In every immersive experience, the primary interface between the user and the digital world is the near-eye display. Thus, developing near-eye display systems that provide a high-quality user experience is of the utmost importance. Many characteristics of near-eye displays that define the quality of an experience, such as resolution, refresh rate, contrast, and field of view, have been significantly improved in recent years. However, a significant source of visual discomfort prevails: the vergence-accommodation conflict (VAC). This visual conflict results from the fact that vergence cues, but not focus cues, are simulated in near-eye display systems. Indeed, natural focus cues are not supported by any existing near-eye display. Afforded by focus-tunable optics, we explore unprecedented display modes that tackle this issue in multiple ways with the goal of increasing visual comfort and providing more realistic visual experiences.
Citation
Konrad, R., Padmanaban, N., Cooper, E., & Wetzstein, G. (2016, July). Computational Focus-Tunable Near-Eye Displays. In ACM SIGGRAPH 2016 Emerging Technologies (p. 3). ACM.
BibTeX
@inproceedings{Konrad:2016:Computational,
title={Computational Focus-Tunable Near-Eye Displays},
author={Konrad, Robert and Padmanaban, Nitish and Cooper, Emily A and Wetzstein, Gordon},
booktitle={ACM SIGGRAPH 2016 Emerging Technologies},
year={2016},
month={July},
pages={3:1--3:2},
organization={ACM}
}
Active Feedback Real Time MPI Control Software
Real-time MPI has the potential to serve as a noninvasive alternative to X-ray angiography. In order to achieve real time MPI, we must (a.) generate vector-drive field waveforms at a location governed in real time by the physician, (b.) acquire the MPI image data in real time, (c.) reconstruct the MPI images in real time. We have designed our MPI data acquisition and control (DAQ) to enable all these steps in real time.
Citation
Padmanaban, N., Orendorff, R. D., Konkle, J. J., Goodwill, P. W., & Conolly, S. M. (2015, March). Active Feedback Real Time MPI Control Software. In Magnetic Particle Imaging (IWMPI), 2015 5th International Workshop on (pp. 1–1). IEEE.
BibTeX
@inproceedings{Padmanaban:2015:Active,
title={Active Feedback Real Time MPI Control Software},
author={Padmanaban, Nitish and Orendorff, Ryan D and Konkle, Justin J and Goodwill, Patrick W and Conolly, Steven M},
booktitle={Magnetic Particle Imaging (IWMPI), 2015 5th International Workshop on},
year={2015},
month={March},
pages={1--1},
organization={IEEE}
}
Presentations
Automatically Refocusing Reading Glasses
Autofocals: Evaluating Gaze-Contingent Eyeglasses for Presbyopes
ACM SIGGRAPH Talks, August 2019
OSA Frontiers in Optics, September 2019
Autofocal Correction for Presbyopes and Its Application to AR/VR
Fundamentals of Virtual- and Augmented-Reality Technologies
SID Display Week Short Courses, May 2019
Build Your Own VR Display: An Introduction to VR Display Systems for Hobbyists and Educators
Electronic Imaging Short Courses, January 2018
Electronic Imaging Short Courses, January 2019
Optimizing VR for All Users Through Adaptive Focus Displays
ACM SIGGRAPH Talks, July 2017
Build Your Own VR System: An Introduction to VR Displays and Cameras for Hobbyists and Educators
ACM SIGGRAPH Courses, July 2017
Gaze-Contingent Adaptive Focus Near-Eye Displays
SID Display Week Invited Talk, May 2017
Computational Focus Tunable Near-Eye Displays
NVIDIA GPU Technology Conference, May 2017
Panel: Frontiers in Technology
Stanford mediaX – Sensing and Tracking for 3D Narratives, October 2016