HAPTIC CINEMATOGRAPHY : AUTHORING HAPTIC FEEDBACK FOR MOVIES Creative Haptic Expression : Trends and Tools April 8th , 2016
Fabien DANIEAU Technicolor R&I
[email protected]
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Content 1. Introduction 2. H-Studio 3. Haptic Cinematography 4. Principles for Haptic Cinematography 5. Conclusion 6. References
2
INTRODUCTION
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Audiovisual Experience ◮
Evolution of the audiovisual experience ◮ ◮ ◮
Improving the Quality of Experience Focused on images and sound Next step : sense of touch ? (Haptics)
4
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Haptics and Audiovisuals ◮ ◮
Haptics already used in VR Early results on haptics for AV ◮
Haptics may enrich AV experience
[O’Modhrain and Oakley, 2003] ◮
Virtual Reality Setup [Cirio et al., 2011]
New medium to express content
[Magnenat and Bonanni, 2006] ◮
New eld of study : Haptic-AudioVisuals [El Saddik et al., 2011]
4D Cinema Setup
5
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Haptics and Audiovisuals ◮ ◮
Haptics already used in VR Early results on haptics for AV ◮
Haptics may enrich AV experience
[O’Modhrain and Oakley, 2003] ◮
Virtual Reality Setup [Cirio et al., 2011]
New medium to express content
[Magnenat and Bonanni, 2006] ◮
New eld of study : Haptic-AudioVisuals [El Saddik et al., 2011]
4D Cinema Setup
Challenge
How to augment audiovisuals with haptic feedback ? 5
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Work ow ◮ ◮
Based on architecture for video streaming Three steps ◮ ◮ ◮
Production Distribution Rendering
Capturing
Audiovisual media
Automatic extraction
Synthesizing
Haptic effects
E N C O D E R
Network
D E C O D E R
Audiovisual renderer
Screen & Speakers
Automatic extraction
Haptic renderer
Haptic devices
[Danieau et al., 2013]
6
H-STUDIO
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Overview of the Authoring Tool
8
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Overview of the Authoring tool ◮
Edition of motion e ects ◮ ◮ ◮ ◮
Capture of motion data Manual edition thanks to force-feedback device Automatic extraction of vibration e ects Preview of haptic e ects
9
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Overview of the Authoring tool ◮
Edition of motion e ects ◮ ◮ ◮ ◮
Capture of motion data Manual edition thanks to force-feedback device Automatic extraction of vibration e ects Preview of haptic effects
9
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Acquisition ◮
Capture device ◮ ◮
◮
Camera IMU : linear acceleration
Content
10
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Preview of Haptic E ects ◮
Signal processing ◮ ◮ ◮
◮
Synchro with AV (haptic clap) Low pass ltering Gravity removal
User feedback ◮
Realistic movement 11
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
H-Studio 2 ◮ ◮
Ported on Android device Use embedded sensors
12
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
H-Studio 2 ◮ ◮ ◮
Ported on Android device Use embedded sensors Edition with touchscreen
[Mollet et al., 2015]
12
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
H-Studio 2 ◮ ◮ ◮ ◮
Ported on Android device Use embedded sensors Edition with touchscreen Full body stimulation
[Mollet et al., 2015]
12
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
H-Studio 2 ◮ ◮ ◮ ◮ ◮
Ported on Android device Use embedded sensors Edition with touchscreen Full body stimulation Multiple haptic e ects (vibration, temperature, motion, etc.)
[Mollet et al., 2015]
12
HAPTIC CINEMATOGRAPHY
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Haptic Cinematography ◮
How to associate haptic feedback to movies ?
14
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Haptic Cinematography ◮ ◮
How to associate haptic feedback to movies ? Mainly used with action movies ◮ ◮
Physical events : explosions, car chases, gun shots, etc. 4D cinemas
14
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Haptic Cinematography ◮ ◮
How to associate haptic feedback to movies ? Mainly used with action movies ◮ ◮
◮
Physical events : explosions, car chases, gun shots, etc. 4D cinemas
Drama, comedy, romantic movies ?
14
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Haptic Cinematography ◮ ◮
How to associate haptic feedback to movies ? Mainly used with action movies ◮ ◮
◮
Physical events : explosions, car chases, gun shots, etc. 4D cinemas
Drama, comedy, romantic movies ?
Non-diegetic e ects
Move beyond enhancement of physical events only 14
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Diegesis : de nition ◮
Non-diegetic e ect ◮
◮
E ect whose source is neither visible on the screen nor has been implied to be present in the action
Example : non-diegetic sound ◮ ◮
Narrator’s commentary Mood music
15
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Taxonomy of Haptic E ects Global
Diegetic Local Haptic Effects
Camera Editing NonDiegetic Music Context
[Danieau et al., 2014]
16
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Taxonomy of Haptic E ects Global
Diegetic Local Haptic Effects
Camera Editing NonDiegetic Music Context
[Danieau et al., 2014]
Focus
Study of cinematographic camera e ects 16
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Cinematographic Camera E ects ◮
Typical cameras e ects ◮ ◮ ◮ ◮ ◮ ◮ ◮
Arcing Traveling Tilting Zoom-in Vertigo Crane shot Dutch angle
17
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Cinematographic Camera E ects ◮
Typical cameras e ects
◮
Two types of haptic feedback ◮ ◮
Cinematic→direct mapping Semantic→haptic metaphor
17
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Cinematographic Camera E ects
◮
Typical cameras e ects
◮
Two types of haptic feedback ◮ ◮
◮
Cinematic→direct mapping Semantic→haptic metaphor
Proof-of-Concept ◮
7 video sequences
◮
HapSeat
17
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Cinematic Model ◮ ◮
Follows camera movement Example with Arcing, Tilting and Zoom-in
18
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Semantic Model ◮
◮
Based on the semantic of the Camera E ect ◮
Crane shot→Flying away
◮
Dutch angle→Instability
◮
Arcing→Intensi cation
◮
Traveling→Crab walk
◮
Tilting→Inferiority
◮
Zoom-in→Walk forward
◮
Vertigo→Vertigo
Designed with H-Studio 19
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
User Study
◮
◮ ◮
Objective : study of the in uence of the haptic e ects based on camera motions on the QoE Hypothesis : New haptic e ects increases QoE Variables ◮ ◮
◮
Measure ◮ ◮
◮
7 video sequences 4 haptic conditions (cinematic, semantic, random, none) Pairwise comparison (78 couples) Score computed for each model
38 participants (¯ x = 36.3, σx = 10.4)
20
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
User Study Haptic e ects enhance QoE
◮
◮
Cinematic > Semantic ≈ None > Random Semantic > None ◮
Vertigo
◮
Arcing
◮
Tilting
0,8
0,6
Score
◮
QoE for all sequences 1
0,4
0,2
0
Cinematic
Discussion ◮
◮
None
Semantic
QoE per sequence
Metaphors well understood Dynamics of haptic feedback should be similar to dynamics AV content
Random
1
0,8
0,6
Score
◮
0,4
0,2
0 Crane Shot
Dutch Angle
Traveling
Cinematic
Zoom
Semantic
Vertigo
Random
None
Arcing
Tilting
21
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Application
22
PRINCIPLES FOR HAPTIC CINEMATOGRAPHY
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Inspiration ◮ ◮
Guidelines for designing haptic e ects ? Inspired from Disney’s 12 principles of animation ◮ ◮
realistic animations law of physics, emotional timing, character appeal, etc.
[Thomas and Johnston, 1995]
24
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Principles for haptic cinematography ? ◮ ◮ ◮ ◮
Lot of tests, demonstrations, discussion with professionals Work with student in cinematography H-Studio Blender Foundation Movies
25
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Principles for haptic cinematography ? ◮ ◮ ◮ ◮
Lot of tests, demonstrations, discussion with professionals Work with student in cinematography H-Studio Blender Foundation Movies
25
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Principles for haptic cinematography ? ◮ ◮ ◮ ◮
Lot of tests, demonstrations, discussion with professionals Work with student in cinematography H-Studio Blender Foundation Movies
9 Principles for the Haptic Cinematography
[Guillotel et al., 2016] 25
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles
1. Staging
2. Continuity 3. Realism
4. Perception
5. Anticipation
6. Exaggeration 7. Synchronism
8. Kinetic Interpolation 9. Diegesis
26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. Staging
2. 3. 4. 5. 6. 7. 8. 9.
Continuity Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. Staging 2. Continuity
3. 4. 5. 6. 7. 8. 9.
Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. Staging 2. Continuity 3. Realism
4. 5. 6. 7. 8. 9.
Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. 2. 3. 4.
Staging Continuity Realism Perception
5. 6. 7. 8. 9.
Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. 2. 3. 4. 5.
Staging Continuity Realism Perception Anticipation
6. 7. 8. 9.
Exaggeration Synchronism Kinetic Interpolation Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. 2. 3. 4. 5. 6.
Staging Continuity Realism Perception Anticipation Exaggeration
7. Synchronism 8. Kinetic Interpolation 9. Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. 2. 3. 4. 5. 6. 7.
Staging Continuity Realism Perception Anticipation Exaggeration Synchronism
8. Kinetic Interpolation 9. Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. 2. 3. 4. 5. 6. 7. 8.
Staging Continuity Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation
9. Diegesis 26
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
9 Principles 1. 2. 3. 4. 5. 6. 7. 8. 9.
Staging Continuity Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis
26
CONCLUSION
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Conclusion
◮ ◮
Haptics→strong potential for enhancing AV Authoring tool : H-Studio ◮ ◮ ◮ ◮
◮
Capture Manual edition Preview with force-feedback device Full body
Haptic Cinematography ◮ ◮
Taxonomy of haptic e ects 9 principles
28
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
Thanks ! Questions ?
29
REFERENCES
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
References I Cirio, G., Marchal, M., Hillaire, S., and Lecuyer, A. (2011). Six degrees-of-freedom haptic interaction with uids. Visualization and Computer Graphics, IEEE Transactions on, 17(11) :1714–1727. Danieau, F., Fleureau, J., Guillotel, P., Mollet, N., Christie, M., and Lecuyer, A. (2014). Toward haptic cinematography : enhancing movie experiences with camera-based haptic e ects. MultiMedia, IEEE, 21(2) :11–21. Danieau, F., Lécuyer, A., Guillotel, P., Fleureau, J., Mollet, N., and Christie, M. (2013). Enhancing audiovisual experience with haptic feedback : a survey on HAV. volume 6, pages 193–205. El Saddik, A., Orozco, M., Eid, M., and Cha, J. (2011). Haptic technologies - bringing Touch to multimedia. Springer. Guillotel, P., Danieau, F., Fleureau, J., and Rouxel, I. (2016). Introducing basic principles of haptic cinematography editing. Eurographics Workshop on Intelligent Cinematography and Editing. 31
Content
Introduction
H-Studio
Haptic Cinematography
Principles for Haptic Cinematography
Conclusion
References
References II
Magnenat, N. and Bonanni, U. (2006). Haptics in virtual reality and multimedia. IEEE Multimedia, 13(3) :6–11. Mollet, N., Dillon, T., Danieau, F., and Le Clerc, F. (2015). A work ow for next generation immersive content. IBC. O’Modhrain, S. and Oakley, I. (2003). Touch TV : adding feeling to broadcast media. In European Conference on Interactive Television : from Viewers to Actors. Thomas, F. and Johnston, O. (1995). The Illusion of Life : Disney Animation. Hyperion, Disney edition.
32