haptic cinematography : authoring haptic ... - Dr. Fabien Danieau

Apr 8, 2016 - Principles for Haptic Cinematography. Conclusion. References. H-Studio 2. ▷ Ported on Android device. ▷ Use embedded sensors. 12 ...
2MB taille 1 téléchargements 364 vues
HAPTIC CINEMATOGRAPHY : AUTHORING HAPTIC FEEDBACK FOR MOVIES Creative Haptic Expression : Trends and Tools April 8th , 2016

Fabien DANIEAU Technicolor R&I [email protected]

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Content 1. Introduction 2. H-Studio 3. Haptic Cinematography 4. Principles for Haptic Cinematography 5. Conclusion 6. References

2

INTRODUCTION

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Audiovisual Experience ◮

Evolution of the audiovisual experience ◮ ◮ ◮

Improving the Quality of Experience Focused on images and sound Next step : sense of touch ? (Haptics)

4

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Haptics and Audiovisuals ◮ ◮

Haptics already used in VR Early results on haptics for AV ◮

Haptics may enrich AV experience

[O’Modhrain and Oakley, 2003] ◮

Virtual Reality Setup [Cirio et al., 2011]

New medium to express content

[Magnenat and Bonanni, 2006] ◮

New eld of study : Haptic-AudioVisuals [El Saddik et al., 2011]

4D Cinema Setup

5

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Haptics and Audiovisuals ◮ ◮

Haptics already used in VR Early results on haptics for AV ◮

Haptics may enrich AV experience

[O’Modhrain and Oakley, 2003] ◮

Virtual Reality Setup [Cirio et al., 2011]

New medium to express content

[Magnenat and Bonanni, 2006] ◮

New eld of study : Haptic-AudioVisuals [El Saddik et al., 2011]

4D Cinema Setup

Challenge

How to augment audiovisuals with haptic feedback ? 5

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Work ow ◮ ◮

Based on architecture for video streaming Three steps ◮ ◮ ◮

Production Distribution Rendering

Capturing

Audiovisual media

Automatic extraction

Synthesizing

Haptic effects

E N C O D E R

Network

D E C O D E R

Audiovisual renderer

Screen & Speakers

Automatic extraction

Haptic renderer

Haptic devices

[Danieau et al., 2013]

6

H-STUDIO

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Overview of the Authoring Tool

8

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Overview of the Authoring tool ◮

Edition of motion e ects ◮ ◮ ◮ ◮

Capture of motion data Manual edition thanks to force-feedback device Automatic extraction of vibration e ects Preview of haptic e ects

9

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Overview of the Authoring tool ◮

Edition of motion e ects ◮ ◮ ◮ ◮

Capture of motion data Manual edition thanks to force-feedback device Automatic extraction of vibration e ects Preview of haptic effects

9

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Acquisition ◮

Capture device ◮ ◮



Camera IMU : linear acceleration

Content

10

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Preview of Haptic E ects ◮

Signal processing ◮ ◮ ◮



Synchro with AV (haptic clap) Low pass ltering Gravity removal

User feedback ◮

Realistic movement 11

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

H-Studio 2 ◮ ◮

Ported on Android device Use embedded sensors

12

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

H-Studio 2 ◮ ◮ ◮

Ported on Android device Use embedded sensors Edition with touchscreen

[Mollet et al., 2015]

12

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

H-Studio 2 ◮ ◮ ◮ ◮

Ported on Android device Use embedded sensors Edition with touchscreen Full body stimulation

[Mollet et al., 2015]

12

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

H-Studio 2 ◮ ◮ ◮ ◮ ◮

Ported on Android device Use embedded sensors Edition with touchscreen Full body stimulation Multiple haptic e ects (vibration, temperature, motion, etc.)

[Mollet et al., 2015]

12

HAPTIC CINEMATOGRAPHY

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Haptic Cinematography ◮

How to associate haptic feedback to movies ?

14

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Haptic Cinematography ◮ ◮

How to associate haptic feedback to movies ? Mainly used with action movies ◮ ◮

Physical events : explosions, car chases, gun shots, etc. 4D cinemas

14

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Haptic Cinematography ◮ ◮

How to associate haptic feedback to movies ? Mainly used with action movies ◮ ◮



Physical events : explosions, car chases, gun shots, etc. 4D cinemas

Drama, comedy, romantic movies ?

14

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Haptic Cinematography ◮ ◮

How to associate haptic feedback to movies ? Mainly used with action movies ◮ ◮



Physical events : explosions, car chases, gun shots, etc. 4D cinemas

Drama, comedy, romantic movies ?

Non-diegetic e ects

Move beyond enhancement of physical events only 14

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Diegesis : de nition ◮

Non-diegetic e ect ◮



E ect whose source is neither visible on the screen nor has been implied to be present in the action

Example : non-diegetic sound ◮ ◮

Narrator’s commentary Mood music

15

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Taxonomy of Haptic E ects Global

Diegetic Local Haptic Effects

Camera Editing NonDiegetic Music Context

[Danieau et al., 2014]

16

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Taxonomy of Haptic E ects Global

Diegetic Local Haptic Effects

Camera Editing NonDiegetic Music Context

[Danieau et al., 2014]

Focus

Study of cinematographic camera e ects 16

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Cinematographic Camera E ects ◮

Typical cameras e ects ◮ ◮ ◮ ◮ ◮ ◮ ◮

Arcing Traveling Tilting Zoom-in Vertigo Crane shot Dutch angle

17

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Cinematographic Camera E ects ◮

Typical cameras e ects



Two types of haptic feedback ◮ ◮

Cinematic→direct mapping Semantic→haptic metaphor

17

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Cinematographic Camera E ects



Typical cameras e ects



Two types of haptic feedback ◮ ◮



Cinematic→direct mapping Semantic→haptic metaphor

Proof-of-Concept ◮

7 video sequences



HapSeat

17

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Cinematic Model ◮ ◮

Follows camera movement Example with Arcing, Tilting and Zoom-in

18

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Semantic Model ◮



Based on the semantic of the Camera E ect ◮

Crane shot→Flying away



Dutch angle→Instability



Arcing→Intensi cation



Traveling→Crab walk



Tilting→Inferiority



Zoom-in→Walk forward



Vertigo→Vertigo

Designed with H-Studio 19

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

User Study



◮ ◮

Objective : study of the in uence of the haptic e ects based on camera motions on the QoE Hypothesis : New haptic e ects increases QoE Variables ◮ ◮



Measure ◮ ◮



7 video sequences 4 haptic conditions (cinematic, semantic, random, none) Pairwise comparison (78 couples) Score computed for each model

38 participants (¯ x = 36.3, σx = 10.4)

20

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

User Study Haptic e ects enhance QoE





Cinematic > Semantic ≈ None > Random Semantic > None ◮

Vertigo



Arcing



Tilting

0,8

0,6

Score



QoE for all sequences 1

0,4

0,2

0

Cinematic

Discussion ◮



None

Semantic

QoE per sequence

Metaphors well understood Dynamics of haptic feedback should be similar to dynamics AV content

Random

1

0,8

0,6

Score



0,4

0,2

0 Crane Shot

Dutch Angle

Traveling

Cinematic

Zoom

Semantic

Vertigo

Random

None

Arcing

Tilting

21

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Application

22

PRINCIPLES FOR HAPTIC CINEMATOGRAPHY

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Inspiration ◮ ◮

Guidelines for designing haptic e ects ? Inspired from Disney’s 12 principles of animation ◮ ◮

realistic animations law of physics, emotional timing, character appeal, etc.

[Thomas and Johnston, 1995]

24

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Principles for haptic cinematography ? ◮ ◮ ◮ ◮

Lot of tests, demonstrations, discussion with professionals Work with student in cinematography H-Studio Blender Foundation Movies

25

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Principles for haptic cinematography ? ◮ ◮ ◮ ◮

Lot of tests, demonstrations, discussion with professionals Work with student in cinematography H-Studio Blender Foundation Movies

25

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Principles for haptic cinematography ? ◮ ◮ ◮ ◮

Lot of tests, demonstrations, discussion with professionals Work with student in cinematography H-Studio Blender Foundation Movies

9 Principles for the Haptic Cinematography

[Guillotel et al., 2016] 25

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles

1. Staging

2. Continuity 3. Realism

4. Perception

5. Anticipation

6. Exaggeration 7. Synchronism

8. Kinetic Interpolation 9. Diegesis

26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. Staging

2. 3. 4. 5. 6. 7. 8. 9.

Continuity Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. Staging 2. Continuity

3. 4. 5. 6. 7. 8. 9.

Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. Staging 2. Continuity 3. Realism

4. 5. 6. 7. 8. 9.

Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. 2. 3. 4.

Staging Continuity Realism Perception

5. 6. 7. 8. 9.

Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. 2. 3. 4. 5.

Staging Continuity Realism Perception Anticipation

6. 7. 8. 9.

Exaggeration Synchronism Kinetic Interpolation Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. 2. 3. 4. 5. 6.

Staging Continuity Realism Perception Anticipation Exaggeration

7. Synchronism 8. Kinetic Interpolation 9. Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. 2. 3. 4. 5. 6. 7.

Staging Continuity Realism Perception Anticipation Exaggeration Synchronism

8. Kinetic Interpolation 9. Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. 2. 3. 4. 5. 6. 7. 8.

Staging Continuity Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation

9. Diegesis 26

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

9 Principles 1. 2. 3. 4. 5. 6. 7. 8. 9.

Staging Continuity Realism Perception Anticipation Exaggeration Synchronism Kinetic Interpolation Diegesis

26

CONCLUSION

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Conclusion

◮ ◮

Haptics→strong potential for enhancing AV Authoring tool : H-Studio ◮ ◮ ◮ ◮



Capture Manual edition Preview with force-feedback device Full body

Haptic Cinematography ◮ ◮

Taxonomy of haptic e ects 9 principles

28

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

Thanks ! Questions ?

29

REFERENCES

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

References I Cirio, G., Marchal, M., Hillaire, S., and Lecuyer, A. (2011). Six degrees-of-freedom haptic interaction with uids. Visualization and Computer Graphics, IEEE Transactions on, 17(11) :1714–1727. Danieau, F., Fleureau, J., Guillotel, P., Mollet, N., Christie, M., and Lecuyer, A. (2014). Toward haptic cinematography : enhancing movie experiences with camera-based haptic e ects. MultiMedia, IEEE, 21(2) :11–21. Danieau, F., Lécuyer, A., Guillotel, P., Fleureau, J., Mollet, N., and Christie, M. (2013). Enhancing audiovisual experience with haptic feedback : a survey on HAV. volume 6, pages 193–205. El Saddik, A., Orozco, M., Eid, M., and Cha, J. (2011). Haptic technologies - bringing Touch to multimedia. Springer. Guillotel, P., Danieau, F., Fleureau, J., and Rouxel, I. (2016). Introducing basic principles of haptic cinematography editing. Eurographics Workshop on Intelligent Cinematography and Editing. 31

Content

Introduction

H-Studio

Haptic Cinematography

Principles for Haptic Cinematography

Conclusion

References

References II

Magnenat, N. and Bonanni, U. (2006). Haptics in virtual reality and multimedia. IEEE Multimedia, 13(3) :6–11. Mollet, N., Dillon, T., Danieau, F., and Le Clerc, F. (2015). A work ow for next generation immersive content. IBC. O’Modhrain, S. and Oakley, I. (2003). Touch TV : adding feeling to broadcast media. In European Conference on Interactive Television : from Viewers to Actors. Thomas, F. and Johnston, O. (1995). The Illusion of Life : Disney Animation. Hyperion, Disney edition.

32