Title Details

List of Contributors XV

Foreword XIX

Part I Fundamentals 1

1 Introduction 3
Gabriel Cristóbal, Laurent U. Perrinet, and Matthias S. Keil

1.1 Why ShouldWe Be Inspired by Biology? 4

1.2 Organization of Chapters in the Book 6

1.3 Conclusion 9

Acknowledgments 9

References 9

2 Bioinspired Vision Sensing 11
Christoph Posch

2.1 Introduction 11

2.1.1 Neuromorphic Engineering 12

2.1.2 Implementing Neuromorphic Systems 13

2.2 Fundamentals and Motivation: Bioinspired Artificial Vision 13

2.2.1 Limitations in Vision Engineering 14

2.2.2 The Human Retina from an Engineering Viewpoint 14

2.2.3 Modeling the Retina in Silicon 17

2.3 From Biological Models to Practical Vision Devices 18

2.3.1 TheWiring Problem 18

2.3.2 Where and What 20

2.3.3 Temporal Contrast: The DVS 21

2.3.4 Event-Driven Time-Domain Imaging: The ATIS 22

2.4 Conclusions and Outlook 25

References 26

3 Retinal Processing: From Biology to Models and Applications 29
David Alleysson and Nathalie Guyader

3.1 Introduction 29

3.2 Anatomy and Physiology of the Retina 30

3.2.1 Overview of the Retina 30

3.2.2 Photoreceptors 31

3.2.3 Outer and Inner Plexiform Layers (OPL and IPL) 33

3.2.4 Summary 34

3.3 Models of Vision 34

3.3.1 Overview of the Retina Models 34

3.3.2 Biological Models 35

3.3.2.1 Cellular and Molecular Models 35

3.3.2.2 Network Models 36

3.3.2.3 Parallel and Descriptive Models 38

3.3.3 Information Models 39

3.3.4 Geometry Models 40

3.4 Application to Digital Photography 42

3.4.1 Color Demosaicing 43

3.4.2 Color Constancy – Chromatic Adaptation – White Balance – Tone Mapping 44

3.5 Conclusion 45

References 46

4 Modeling Natural Image Statistics 53
Holly E. Gerhard, Lucas Theis, and Matthias Bethge

4.1 Introduction 53

4.2 Why Model Natural Images? 53

4.3 Natural Image Models 55

4.3.1 Model Evaluation 65

4.4 Computer Vision Applications 69

4.5 Biological Adaptations to Natural Images 71

4.6 Conclusions 75

References 76

5 Perceptual Psychophysics 81
C. Alejandro Parraga

5.1 Introduction 81

5.1.1 What Is Psychophysics andWhy DoWe Need It? 81

5.2 Laboratory Methods 82

5.2.1 Accuracy and Precision 83

5.2.2 Error Propagation 84

5.3 PsychophysicalThreshold Measurement 85

5.3.1 Weber’s Law 85

5.3.2 Sensitivity Functions 86

5.4 Classic Psychophysics:Theory and Methods 86

5.4.1 Theory 87

5.4.2 Method of Constant Stimuli 88

5.4.3 Method of Limits 90

5.4.3.1 Forced-Choice Methods 92

5.4.4 Method of Adjustments 93

5.4.5 Estimating Psychometric Function Parameters 94

5.5 Signal Detection Theory 94

5.5.1 Signal and Noise 94

5.5.2 The Receiver Operating Characteristic 96

5.6 Psychophysical Scaling Methods 98

5.6.1 Discrimination Scales 99

5.6.2 Rating Scales 100

5.6.2.1 Equipartition Scales 100

5.6.2.2 Paired Comparison Scales 101

5.7 Conclusions 105

References 106

Part II Sensing 109

6 Bioinspired Optical Imaging 111
Mukul Sarkar

6.1 Visual Perception 111

6.1.1 Natural Single-Aperture and Multiple-Aperture Eyes 112

6.1.1.1 Human Eyes 113

6.1.1.2 Compound Eyes 114

6.1.1.3 Resolution 114

6.1.1.4 Visual Acuity 115

6.1.1.5 Depth Perception 115

6.1.1.6 Color Vision 116

6.1.1.7 Speed of Imaging and Motion Detection 117

6.1.1.8 Polarization Vision 117

6.2 Polarization Vision - Object Differentiation/Recognition 119

6.2.1 Polarization of Light 121

6.2.2 Polarization Imaging 125

6.2.2.1 Detection of Transparent and Opaque Objects 126

6.2.2.2 Shape Detection Using Polarized Light 131

6.3 High-Speed Motion Detection 133

6.3.1 Temporal Differentiation 135

6.4 Conclusion 138

References 139

7 Biomimetic Vision Systems 143
Reinhard Voelkel

7.1 Introduction 143

7.2 Scaling Laws in Optics 144

7.2.1 Optical Properties of Imaging Lens Systems 144

7.2.2 Space Bandwidth Product (SW) 146

7.3 The Evolution of Vision Systems 147

7.3.1 Single-Aperture Eyes 148

7.3.2 Compound Eyes 149

7.3.3 The Array Optics Concept 150

7.3.4 Jumping Spiders: Perfect Eyes for Small Animals 151

7.3.5 Nocturnal Spiders: Night Vision 153

7.4 Manufacturing of Optics for Miniaturized Vision Systems 154

7.4.1 Optics Industry 154

7.4.2 Planar Array Optics for Stereoscopic Vision 155

7.4.3 The Lack of a Suitable Fabrication Technology Hinders Innovation 155

7.4.4 Semiconductor Industry PromotesWafer-Based Manufacturing 156

7.4.5 Image Sensors 156

7.4.6 Wafer-Based Manufacturing of Optics 158

7.4.7 Manufacturing of Microlens Arrays onWafer Level 159

7.4.8 Diffractive Optical Elements and Subwavelength Structures for Antireflection Coatings 160

7.4.9 Microlens Imprint and Replication Processes 162

7.4.10 Wafer-Level Stacking or Packaging (WLP) 162

7.4.11 Wafer-Level Camera (WLC) 163

7.5 Examples for Biomimetic Compound Vision Systems 164

7.5.1 Ultraflat Cameras 165

7.5.2 Biologically Inspired Vision Systems for Smartphone Cameras 166

7.5.3 PiCam Cluster Camera 167

7.5.4 Panoramic Motion Camera for Flying Robots 169

7.5.5 Conclusion 170

References 172

8 Plenoptic Cameras 175
Fernando Pérez Nava, Alejandro Pérez Nava, Manuel Rodríguez Valido, and Eduardo Magdaleno Castellò

8.1 Introduction 175

8.2 Light Field Representation of the Plenoptic Function 177

8.2.1 The Plenoptic Function 177

8.2.2 Light Field Parameterizations 178

8.2.3 Light Field Reparameterization 179

8.2.4 Image Formation 180

8.2.5 Light Field Sampling 180

8.2.5.1 Pinhole and Thin Lens Camera 180

8.2.5.2 Multiple Devices 181

8.2.5.3 Temporal Multiplexing 181

8.2.5.4 Frequency Multiplexing 182

8.2.5.5 Spatial Multiplexing 182

8.2.5.6 Simulation 182

8.2.6 Light Field Sampling Analysis 183

8.2.7 Light Field Visualization 183

8.3 The Plenoptic Camera 185

8.4 Applications of the Plenoptic Camera 188

8.4.1 Refocusing 188

8.4.2 Perspective Shift 190

8.4.3 Depth Estimation 190

8.4.4 Extended Depth of Field Images 192

8.4.5 Superresolution 192

8.5 Generalizations of the Plenoptic Camera 193

8.6 High-Performance Computing with Plenoptic Cameras 195

8.7 Conclusions 196

References 197

Part III Modelling 201

9 Probabilistic Inference and Bayesian Priors in Visual Perception 203
Grigorios Sotiropoulos and Peggy Seriès

9.1 Introduction 203

9.2 Perception as Bayesian Inference 204

9.2.1 Deciding on a Single Percept 205

9.3 Perceptual Priors 207

9.3.1 Types of Prior Expectations 207

9.3.2 Impact of Expectations 208

9.3.3 The Slow Speed Prior 210

9.3.4 Expectations and Environmental Statistics 212

9.4 Outstanding Questions 214

9.4.1 Are Long-Term Priors Plastic? 214

9.4.2 How Specific are Priors? 215

9.4.3 Inference in Biological and Computer Vision 215

9.4.4 Conclusions 217

References 219

10 From NeuronalModels to Neuronal Dynamics and Image Processing 221
Matthias S. Keil

10.1 Introduction 221

10.2 The Membrane Equation as a Neuron Model 222

10.2.1 Synaptic Inputs 224

10.2.2 Firing Spikes 227

10.3 Application 1: A Dynamical Retinal Model 230

10.4 Application 2: Texture Segregation 234

10.5 Application 3: Detection of Collision Threats 236

10.6 Conclusions 239

Acknowledgments 240

References 240

11 ComputationalModels of Visual Attention and Applications 245
Olivier LeMeur and Matei Mancas

11.1 Introduction 245

11.2 Models of Visual Attention 246

11.2.1 Taxonomy 246

11.3 A Closer Look at Cognitive Models 250

11.3.1 Itti et al.’s Model [3] 250

11.3.2 Le Meur et al.’s Model [8] 251

11.3.2.1 Motivations 251

11.3.2.2 Global Architecture 252

11.3.3 Limitations 255

11.4 Applications 256

11.4.1 Saliency-Based Applications: A Brief Review 256

11.4.2 Predicting Memorability of Pictures 257

11.4.2.1 Memorability Definition 257

11.4.2.2 Memorability and Eye Movement 257

11.4.2.3 Computational Models 259

11.4.3 Quality Metric 260

11.4.3.1 Introduction 260

11.4.3.2 Eye Movement During a Quality Task 260

11.4.3.3 Saliency-Based Quality Metrics 261

11.5 Conclusion 262

References 262

12 Visual Motion Processing and Human Tracking Behavior 267
Anna Montagnini, Laurent U. Perrinet, and Guillaume S. Masson

12.1 Introduction 267

12.2 Pursuit Initiation: Facing Uncertainties 269

12.2.1 Where Is the Noise? Motion-Tracking Precision and Accuracy 269

12.2.2 Where Is the Target Really Going? 270

12.2.3 Human Smooth Pursuit as Dynamic Readout of the Neural Solution to the Aperture Problem 272

12.3 Predicting Future and On-Going Target Motion 273

12.3.1 Anticipatory Smooth Tracking 273

12.3.2 If You Don’t See It, You Can Still Predict (and Track) It 274

12.4 Dynamic Integration of Retinal and Extra-Retinal Motion Information: Computational Models 276

12.4.1 A Bayesian Approach for Open-Loop Motion Tracking 276

12.4.2 Bayesian (or Kalman-Filtering) Approach for Smooth Pursuit: Hierarchical Models 278

12.4.3 A Bayesian Approach for Smooth Pursuit: Dealing with Delays 279

12.5 Reacting, Inferring, Predicting: A NeuralWorkspace 282

12.6 Conclusion 286

12.6.1 Interest for Computer Vision 287

Acknowledgments 288

References 288

13 Cortical Networks of Visual Recognition 295
Christian Thériault, Nicolas Thome, and Matthieu Cord

13.1 Introduction 295

13.2 Global Organization of the Visual Cortex 296

13.3 Local Operations: Receptive Fields 297

13.4 Local Operations in V1 298

13.5 Multilayer Models 301

13.6 A Basic IntroductoryModel 302

13.7 Idealized Mathematical Model of V1: Fiber Bundle 307

13.8 Horizontal Connections and the Association Field 311

13.9 Feedback and Attentional Mechanisms 312

13.10 Temporal Considerations, Transformations and Invariance 312

13.11 Conclusion 314

References 315

14 Sparse Models for Computer Vision 319
Laurent U. Perrinet

14.1 Motivation 319

14.1.1 Efficiency and Sparseness in Biological Representations of Natural Images 319

14.1.2 Sparseness Induces Neural Organization 320

14.1.3 Outline: Sparse Models for Computer Vision 322

14.2 What Is Sparseness? Application to Image Patches 323

14.2.1 Definitions of Sparseness 323

14.2.2 Learning to Be Sparse:The SparseNet Algorithm 325

14.2.3 Results: Efficiency of Different Learning Strategies 326

14.3 SparseLets: A Multiscale, Sparse, Biologically Inspired Representation of Natural Images 328

14.3.1 Motivation: Architecture of the Primary Visual Cortex 328

14.3.2 The SparseLets Framework 330

14.3.3 Efficiency of the SparseLets Framework 333

14.4 SparseEdges: Introducing Prior Information 336

14.4.1 Using the Prior in First-Order Statistics of Edges 336

14.4.2 Using the Prior Statistics of Edge Co-Occurrences 338

14.5 Conclusion 341

Acknowledgments 341

References 342

15 Biologically Inspired Keypoints 347
Alexandre Alahi, Georges Goetz, and Emmanuel D’Angelo

15.1 Introduction 347

15.2 Definitions 349

15.3 What Does the Frond-End of the Visual System Tell Us? 350

15.3.1 The Retina 350

15.3.2 From Photoreceptors to Pixels 350

15.3.3 Visual Compression 351

15.3.4 Retinal Sampling Pattern 351

15.3.5 Scale-Space Representation 352

15.3.6 Difference of Gaussians as a Model for RGC-Receptive Fields 353

15.3.7 A Linear Nonlinear Model 354

15.3.8 Gabor-Like Filters 354

15.4 Bioplausible Keypoint Extraction 355

15.4.1 Scale-Invariant Feature Transform 355

15.4.2 Speeded-Up Robust Features 356

15.4.3 Center Surround Extrema 356

15.4.4 Features from Accelerated Segment Test 357

15.5 Biologically Inspired Keypoint Representation 357

15.5.1 Motivations 357

15.5.2 Dense Gabor-Like Descriptors 358

15.5.2.1 Scale-Invariant Feature Transform Descriptor 358

15.5.2.2 Speeded-Up Robust Features Descriptor 359

15.5.3 Sparse Gaussian Kernels 360

15.5.3.1 Local Binary Descriptors 360

15.5.3.2 Fast Retina Keypoint Descriptor 360

15.5.3.3 Fast Retina Keypoint Saccadic Matching 362

15.5.4 Fast Retina Keypoint versus Other Local Binary Descriptors 363

15.6 Qualitative Analysis: Visualizing Keypoint Information 363

15.6.1 Motivations 363

15.6.2 Binary Feature Reconstruction: From Bits to Image Patches 364

15.6.2.1 Feature Inversion as an Inverse Problem 364

15.6.2.2 Interest of the Retinal Descriptor 367

15.6.3 From Feature Visualization to Crowd-Sourced Object Recognition 368

15.7 Conclusions 370

References 371

Part IV Applications 375

16 Nightvision Based on a Biological Model 377
Magnus Oskarsson, Henrik Malm, and EricWarrant

16.1 Introduction 377

16.1.1 RelatedWork 378

16.2 Why Is Vision Difficult in Dim Light? 380

16.3 Why Is Digital Imaging Difficult in Dim Light? 382

16.4 Solving the Problem of Imaging in Dim Light 383

16.4.1 Enhancing the Image 385

16.4.1.1 Visual Image Enhancement in the Retina 385

16.4.1.2 Digital Image Enhancement 387

16.4.2 Filtering the Image 388

16.4.2.1 Spatial and Temporal Summation in Higher Visual Processing 388

16.4.2.2 Structure Tensor Filtering of Digital Images 391

16.5 Implementation and Evaluation of the Night-Vision Algorithm 393

16.5.1 Adaptation of Parameters to Noise Levels 394

16.5.2 Parallelization and Computational Aspects 395

16.5.3 Considerations for Color 396

16.5.4 Experimental Results 397

16.6 Conclusions 399

Acknowledgment 400

References 401

17 Bioinspired Motion Detection Based on an FPGA Platform 405
Tim Köhler

17.1 Introduction 405

17.2 A Motion Detection Module for Robotics and Biology 406

17.3 Insect Motion Detection Models 407

17.4 Overview of Robotic Implementations of Bioinspired Motion Detection 412

17.4.1 Field-Programmable Gate Arrays (FPGA) 413

17.5 An FPGA-Based Implementation 414

17.5.1 FPGA-Camera-Module 414

17.5.2 A Configurable Array of EMDs 416

17.6 Experimental Results 419

17.7 Discussion 421

17.8 Conclusion 422

Acknowledgments 423

References 423

18 Visual Navigation in a ClutteredWorld 425
N. Andrew Browning and Florian Raudies

18.1 Introduction 425

18.2 Cues from Optic Flow: Visually Guided Navigation 426

18.3 Estimation of Self-Motion: Knowing Where You Are Going 429

18.4 Object Detection: Understanding What Is in YourWay 434

18.5 Estimation of TTC: Time Constraints from the Expansion Rate 439

18.6 Steering Control: The Importance of Representation 442

18.7 Conclusions 444

Acknowledgments 445

References 445

Index 447