Introduction to Machine Vision

so uh welcome to machine vision 68016866and i'm not sure how we got so lucky butwe have the classroom that's the firstfrom my office so i guess i'm going toget a lot of exercise and i think we'regoing to havea lot of stragglers coming inwhat's there to know just abouteverything's on the website so i canprobablyeliminate a lot of the administriviaplease make surethatyou're actually registered for thecourse on the website andtake a lookat the assignmentsandhopefullyyou've eitherhad a chance to look at chapters one andtwo or you're about tothat's the assignment for this weekand there's a homework problem andyou're probably saying you knowgod i've just arrived here how can therebe a homework problem well i'm sorry butyou knowthe term is getting shorter and shorterand if i work backwards fromwhen the faculty rules say the lastassignment can be duewe have to start now now the good newsin return is there's no final so yesthere's a homework problem startingright away but there's no finaland there's a homework problem onlyevery second week so it's not a hugeburdenandthere are some take-home quizzes sotwo of thetimes where you'd normally have ahomework are going to be sort ofglorified homeworksthat count more than the others andthey are called be called quizzes sototal there i think they're fivehomework problems and two quizzesuh collaborationum collaboration is okay on the homeworkproblems but please make a note of whoyou worked with it's not okay on on thetake-home quizzes6866 so those of you in 686 thedifference is thatthere's a term project so you'll beimplementingsome machine visionmethodpreferably one that we cover in thecourseand there'll be a proposal dueabout a month from now i'll let you knowas we go alonguhtell me what you're planning to doand you know preferenceis going to be given tosort of dynamic problems rather thansingle image static analysisuh image motion that that kind of thingand umif there's enough interestwe'llhave a session on how to do this on anandroid phoneand i'm a little reluctant to do thatbecause some of you don't have anandroid phoneand um i have some lowness but you knowwhat it's like these dawn things go outof fashion in two years and so umall of the interesting new stuff havingto do with the camera on android is notin some of thebox full of old smartphones i have butuh that is an option so one of the waysof doing your term project is to do anandroid studio uh project andand to help you with that we have like acanned ready-made project that you canmodify rather than starting from fromscratch okay what else um[Music]grades sofor 6801 it's just splithalf for your homework problems and halffor your take-home quizzes so clearlythe take-home quizzes count morefor six eight six six it split threewaysa third fortake homehomework problems a third for quizzesand a third for the projectand again collaboration on the projectsumi actually favor it because you knowthere's just a finite length of time inthe term you've got other courses todeal with oftentimes people end uppostponing it near the end so if you'reworking with someone else that can oftenencourage you to start early and alsomake sure that you're making someprogresstextbookthere's no textbookas you sawif you have robot vision that could beusefulwe're not going to coverum all the robot vision we cover maybea third to a halfand quite a lot of the materialwe cover is referenced through paperswhich we will put up on the stellarwebsiteso in fact if you look at the websiteyou'll see there's a lot of materialand don't don't be scared i mean a lotof that is just for your reference likeif you're working on your project andyou need to know how to do i don't knowsiftuh then it's there so you're notexpected to read all of that so it's therobot vision book uh it should be on thewebsite if it is not the under materialsso when you get to the seller websitethere's a tab at the two tabs and thesecond one isi forget what but that's the one whereall the good stuff isand then when you get to that page oneof thewindows saysmaterial and unfortunately it only showsyou a little bit of it you have to clickon it to see all the materials so itshould be there andwe'll be doing this with some of theother chapters and some of the papers asi mentionedokayalso of course there are errors in thetextbook and sothe errata for the textbookare onlineso if you have the book you could gothrough andread mark all of the bad spotsso reading read chapters one and twodon't worry about all the referencematerial you won't be reading all of itso what are we doing todaywell mostly i need to tell you enough soyou can do the homework problemthat's one function and the other one isto give you an idea of what the courseis about these two things kind ofconflict so i'll try and i'll try and dobothin terms of the course i am supposed totell you what the objectives are so i'lli made up something uh learn how torecover information about environmentfrom the imagesand so we're going to take this sort ofinverse graphics view wherethere's a 3d world out there we get 2dimagesand we're trying to interpret what'shappening in the world you know visionis an amazing uh sense because it'snon-contact and it provides so muchinformationbut it's in a kind of coded form becausewe're not gettingall the information that's possible wedon't get 3d for example so that'sthe topic that we're going to discussand hopefullyyou willthen understand image formation andunderstand how to reverse that to tryandget a description of the environmentfrom from the imagesoutcomeswell you understand what's now calledphysics based machine vision so theapproach we're going to take is prettymuch you know they're light rays theybounce off surfaces they form an imageand that's physics you know rays lensesuh power per unit area that kind ofstuffand from that we can write downequations we can see you know how howmuch energy gets into this pixel in thecamera based onthe object out there how it'silluminated how it reflects light and soonand from the equations we then try toinvert this so the equations depend onparameters we're interested in likespeedtime until we run into a wallthe type of surface cover and so on sothat'sphysics-based machine visionand it's a preparation for more advancedmachine vision courses so there's somebasic material that everyone should knowabout how images are formed that's goingto be useful for other coursesand if you're going into learningapproaches one of the advantages oftaking this course is it'll teach youhow to extract useful featuresso you know you can learn with raw datalike just the gray levels at every pixeland that's not a particularly goodapproach it's much betterif you can already extract informationlike texture distance shape size and soon and do the more advanced work on thatandwell alsoone of the things some people enjoy isto seereal applications of some interestingbut relatively simple math and physicsyou know it's likesometimes we forget about this whenwe're so immersed in programming in javaor something but there's alot of math we learned and sometimespresented learning because like why am ilearning this well it's neat to find outthat it's actually really useful and sothat brings me to the next topic whichis that yes there will be math and butnothing sophisticated is you knowengineering math calculusthat kind of thing derivatives uhvectors matricesmaybe a little bit of linear algebraummaybe some ordinary differentialequationyou know that that kind of stuff nothingtoo advanced no number theory oranything like thatand there'll be some geometryand a little bit of linear system so yousaw the prerequisite with six level r3and that's because we'll talk a littlebit about convolution when we talk aboutimage formation but we're not going togo very deep into any of thatfirst of all because it's covered in6003 nowsince i changed the material to includeimagesand then we have other things to worryaboutso so that's what the course is about ishould also tell you uh what is notso it's not image processing so what'sthe difference well image processing iswhere you you take an image you dosomething to it you have a new imageperhaps improved in some way enhancededgesuh reduce the noise smooth things out orwhatever and that'sthat provides useful tools for some ofthe things we're doing it but that's notthe focus of the coursethere arecourses that do that i mean 6w3 doessome of it alreadyone 63446341 used to be six three fourtwo so there's a slew of imageprocessing coursesthat tell you how to program your dsp todo some transformation on an image andthat's not what we're doingthis is not about pattern recognition soi think of pattern recognition asyou know you give me an image and i'lltell you whether it's a poodle or a catwe're not going to be doing that andthere of coursethere's some courses on that that touchon that in course 9particularly with respect to humanvision and how you might implement thosecapabilities in hardwareand of course machine learning is intothatand that brings me to machine learningthis is not a machine learning courseand there are603 6 6 8 6 9 6 8 6 2 6 8 6 7 et ceteraet cetera so there are plenty of machinelearning courses and we don't have totouch on that here and also i want tokind of show howfar you can getjust understanding the physics of thesituation and modeling itwithout any black box that you feedexamples into in other wordswe're going to be very interested inso-called direct computations wherethere's some simple computation that youperform all over the image and it givesyou some result like okay my opticalmouse is moving to the right by 0.1 uhcentimeter or something like thatit's also not about computationalimagingand that what is that about socomputational imaging is whereimage formation isnot through a physical apparatus butthrough computing that sounds obviouswell we have lenses lenses areincredible lenses areanalog computers that take light raysthat come in and reprogram them to go indifferent directions to form an imageand they've been around a few hundredyears and we don't really appreciatethem because they do it at the speed oflight i mean it's you know if you try todo that in a digital computer it wouldbe very very hard and we've perfectedthem to where you knowi just saw an ad for a camera that had a125 to one zoom ratio i mean if thepeople that started using lenses likegalileoand people inthe netherlands you know they'd be justamazed at what we can do with lenses soumso we have this physical apparatus thatwill do this kind of computation butthere are certain cases where we can'tuse that so for example in commutedcomputed tomography we're shootingx-rays through a body we get an imagebut it's hard to interpret i mean youcan sometimes seetissue of very strong high contrast likebones will stand out but if you want the3d picture of what's inside you have totake lots of these pictures and combinethem computationally we don't have aphysical apparatus like an x-ray lensmirrorgadgetinterferometer that will you know thefinal result is the image here the finalresult is computed even more so an mriyou know we have a big magnet with agradient field we have little magnetsthat modulate it we have rf some signalcomes out it gets processed and today wehave a image of a cross section of thebody so that's computational imaging andwe won't be doing thatthere is a course 6870 which is notoffered this term but itin goes into that and we're also notgoing to say much about human visionagain of course 9 will do thatnow i wasin the interest of getting far enough todo the homework problem i was going tonot do a slideshow but i think it's justtraditional todo a slideshow show soi will try and get this to work it's notalways successfulbecausemy computer has some uh interfaceproblems but let's see what we can dookay so um let's talk about uh machinevision and some of the examples you'llsee in this uh set ofslides not all of it will be clear withmy brief introduction but we'll go backto this later on in the termso what are the sorts of things we mightbe interestedin doing well one is to recoveringrecover image motionand you could imagine variousapplications in say autonomous vehiclesand what have youanother thing we might want to do isestimate surface shape as we saidwe don't get 3d from our cameras wellnot most cameras and if we do get 3dthen it's usually not very great qualitybut we know that humans find it prettystraightforward to seethree-dimensional shapes that aredepicted in photos and photos are flatso where's the 3d come from so that'ssomething we'll look atthen they're really simple questionslike uhoh forgot my optical mouse how dooptical mice work well it's a motionvision problem it's a very simple motionvision problem but it's a good place tostart talking about motion visionsoas i mentioned we will take aphysics-based approachto the problem andwe'll do things like recover observermotion from time varying images again wecan think of autonomous carswe can recover the time to collisionfrom a monocular image sequencethat's sort of interesting becauseyou think that to get depth we might usetwo cameras and binocular vision like wehave two eyes and a certain baseline andwe can triangulate and figure out howfar things are awayand so uh it's kind of uh surprisingthat it's relatively straightforward tofigure outumthe time to contact which isthe ratio of thespeed to the distance right so if i'vegot you know 10 meters to that wall andi'm going 10 meters per second i'll hitit in a second so i need to do twothings i need to estimate the distanceand i need to estimate the speed both ofthese are machine vision problems thatwe can attackand it turns out that there's a verydirect method that doesn't involve anyhigher level reasoningthat gives us that ratio and it's veryusefuland it's also umsuggestive of biological mechanismsbecauseanimals use time to contact for variouspurposes like not running into eachotherand they use them for you know fliespretty small nervous system use time tocontact uh to landso they know what what to do when theyget close enough to the surfaceand so uh you know it's interesting thatwe canhave some idea about how a biologicalsystem might do thatuh contour maps from aerial photographsyou know that's how all maps are madethese days and we'll talk aboutsome industrial machine vision uh workand that's partly becauseactuallythose machines those systems really haveto work very very well not like you know99 of the timeand so they actually poo poo some of thethings us academics talk about becausethey're just not ready for that kind ofenvironment and they've come up withsome very good methods of their own andso it'll be interesting to talk aboutthatso at a higher levelwe want to develop a description of theenvironment just based on on imagesone of theafter we've done some preliminary workand put together some methodswe'lluse them to solvewhat was at one point thought to be aimportant problem which is picking anobject out of a pile of objects so inmanufacturing often uh parts arepalletized or you know arranged uhresistors come on the tapeand so by the time they get to themachine that's supposed to insert themin the circuit boardyou know it's orientation and so that'suhvery makes it very simple to build uhadvanced automation systems but when youlook at humans building thingsyou know there's a box of this andthere's a box of that and there's a boxof these other types of parts andthey're all jumbled and they don't liein a fixed orientationso that you can just uh grab them usingfixedrobotic motions and sowe will put together some machine visionmethods that allow us to find outwhere a part is and how to control themanipulator to pick it upwe'll talk a lot about ill pose problemsso according to hadamard ill postproblems are problemsthat either do not have a solutionhave an infinite number of solutionsor from our point of view mostimportantly have solutions that dependsensitively on the initial conditions soif you have a machine vision method thatsay determines the position andorientation of your cameraand it works with terrific measurementsthat's great but in the real worldthey're always small errors inmeasurementssometimes you're lucky to get thingsaccurate to within a pixel and what youwant is not to have a methodwhere a small change in the measurementsis going to produce a huge error in theresult and unfortunatelyyou know the field has quite a few ofthose and we'll discuss some of them avery famous one is the so-calledeight-point algorithmwhich works beautifully on perfect datalike you know your double precisionnumbers and even if you put in a smallamount of error it gives you absurdresults and yet you know many papershave been published on it sookaywe can recover surface shape frommonocular images uh let's look at that alittle bitso what do you see there think aboutwhat that could beokay so if you don't know what it isdo you see it as a flat surface let'sstart thereokay so no you don't see it as a flatsurface so that's where i was reallygoing with this i i promise you thescreen is perfectly flatthere's no trickery here but you areable to perceive some three-dimensionalshape even though you're unfamiliar withthis surfacewith this picture it happens to begravel braids in a river north of denaliin alaskain winter covered in snow and so onbut the important thing is that we canall agree that there's some sort ofgroove here and you know there's adownward slope on this side and so on sothat shows thateven though images provide onlytwo-dimensional information directly wecan infer three-dimensional informationand that's one of the things we're goingto exploresoyou know how is it that even though theimage is flatwe see a three-dimensional shape and ofcourse it's very common and veryimportant you know you look at a pictureof some politician in the newspaper uhwell the paper's flat but you can seethat face as some sort of shape in 3dprobably not with you know very precisemetric precision but you can recognizethat person based not just on you knowwhether they have a mustache orthey're wearing earrings or somethingbut you have some idea of what the shapeof their nose is and so onso here for exampleis richard feitman's noseand on the right is an algorithmexploring it to determine its shape soyou can see that even though presumablyyou know he washed his face and it'spretty much uniform and properties allover uh where it's curved down it'sdarker where it's facing the lightsource which in this case is near thecamera uh it's bright and soyou you have some idea of slopethat the brightness is somehow relatedto slopewhat makes it interesting is that wellslopeis not a simple thing it's notone number it's two right because we canhave a slope in x and we can have aslope in ybut we only get one constraint we onlyget one brightness measurement so that'sthe kind of problem we're going to befaced with all the time where we're sortof counting constraints versus unknownsyou know how much information do we needto solve for these variablesandhow sensitive is it going to be toerrors in those measurements as wementionedand you know there's a contour map ofhis nose and once you've got the 3dshape you can do all sorts of things youcan put it in a 3d printer and give itto him as a birthday present and thenwhenandhere's asomewhat later result werewe're looking at an image of ahemispherewell actually an oblate ellipsoidandwe're asked to recover its shape andthese are iterations of an algorithmthat works on a gridand finally achievesthe correct shape and we'll talk aboutthe interesting intermediate cases wherethere's sort of ridgeswhere the solution is not satisfied andthe isolated points that are sort ofconical andit's interesting in this case to look atjust how the evolution how the solutionevolvesso here's aoverall picture ofmachine vision in context so first wehave a scene where world out thereand the illumination of that scene isimportant and that's why that's shownalthough it's shown with dotted marksbecause we're not putting that muchemphasis on it there's an imaging deviceuh typically with a lens or mirrors orsomethingandwe get an image and then our job of themachine vision system is to build adescriptionand when it becomes interesting is whenyou then use that description to go backand do something in that world and so inin my view some of the more interestingthings arerobotics applications whereyou know the theproof of the pudding is when youactually go out and the robot grabssomething and it's grabbing it thecorrect waythat's one way you can knowthat's one constraint on your machinevision program if your machine visionprogram is not working that probablywon't happenso in many other cases you know if thefinal output is a description of theenvironmentumwho's to say whether it's correct youknow may it depends on the application imean if it's there for purposes ofwriting a poem about the environmentthat's one thing uh if there's purposeis toassemble an enginethen it'sthis type of situation where we havesome feedback if it works then probablythe machine vision part worked correctlyhere's the time to contact problem thati was talking about andas you can imagine of course as you movetowards the surfacethe image seems to expandand that's the that's the cue but youknow how do you measure that expansionbecause all you've got are these greylevels this array of numbers how do youmeasure that and how do you do itaccurately and fastandand also we noted that you know somehowthere are interesting aspects like onecamera the only twothe other one is that for many of thethings we do we need to know thingsabout the camera like the focal lengthand we need to know where the opticalaxis strikes the image plane so we'vegot this array of pixelsbut you know where's the center well youcan just divide the number of columnsand the number of rows by two but uhthat's totally arbitrarywhat you really want to know is if youput the axis through the lens where doesit hit that image plane and of coursethe manufacturer typically tries to makethat be exactly the center of your imagesensor but it's all you knowit's always going to be a little bit offand in fact in many cases they don'tparticularly care because if my cameraputs the center of the image 100 pixelsto the right i probably won't notice youknow in normal use if i'm going to poston facebook it doesn't really make anydifference if i'm going to use it inindustrial machine vision it does make adifference and so that kind ofcalibration is something we'll talkabout as welland what's interesting is that in thisparticular case we don't even need thatwe don't even need to know the focallength which seems really strangebecause you know if you have a longerfocal length that means the image isgoing to be expanded so it would seemthat that would affect this process butwhat's interesting is that at the sametime as the image is expanded the imagemotion is expanded and so the ratio ofthe two is maintained soso it's you know from that point of viewit's a very interesting problem becauseunlike many others we don't need thatinformationso here's an example of you know we'reapproaching this truckand over here is a plot time horizontaland vertical is the computedtime to contact the red curve is thecomputed and the barely visible greendotted line is is the true valuein the process by the way we exposeanother concept which is the focus ofexpansion so as we approach this truckuh you'll notice that weend up on the doorwhich is not the center of the firstimageso we're actually moving at an anglewe're not movingstraight along the optical axis of thecamera but we're moving at an angle andthe focus of expansion is very importantbecause it tells us the in 3d what themotion vector is so in addition tofinding the time to contact we want tofind thefocus of expansionand there's another one this one wasdone using uh time lapse uh you knowmoving thecar a little bit every time and uhwell i'm not very good at moving thingsexactly 10 millimeters so it's it's abit more noisy than the previous oneuh so yeah we'll be talking a little bitabout coordinate systems andtransformations between coordinatesystemsfor example in the case of robotapplications we'll want to have atransformation betweenacoordinate system that'snative to the camerato a coordinate system that you knowwhen you get the robot it has kinematicsprogrammed into it so that you can tellit in xyz where to go and in angle howto orient the gripper but that's interms of its defined coordinate systemwhich is probably you know the originsin the base where it's bolted in the inthe ground whereas your camera up hereit probably likes a coordinate systemwhereits center ofprojection is the origin so we'll haveto talk aboutthose kinds of thingsand i won't go into thatwe'lltalk about this later soi mentioned analog computingand um you know now we justautomatically everything is digital butuh there are some things that are kindof tedious you know if you have toprocess uh 10 million pixels and docomplicated things with themthat can since digital computing isn'tgetting any fasterthat can be a problemokay so you can use parallelism and youknow soso there's still an interest in analogand so herethis this is the output of a chip thatwe built to find the focus of expansionso and it's basically instantaneousunlike the digital calculationand the plot is so hard to see butlet's see thecirclesthe circle um they're determined by twodifferent algorithmsand you can see that you know there'ssome error but overall the uh the crossthex and the o are sort of on top of eachother sothis was a fun project becauseyou know tohave a chip fabricated is expensive andso you can't afford to screw up too manytimes and of course with an algorithmthat's complicated you know what's thechance you'll get it right the firsttimeso the studentfinally reached the point where operawouldn't pay for any more fabsand the last problem was there was alarge current to the substrate whichcausedit to get warm and of course once itgets hot it doesn't work anymore so he'dcome in every morning with acooler full of ice cubes and a littleaquarium pump and cool his focus ofexpansion chip to make sure that itwouldn't overheat sookay so we talked a little bit aboutprojection and motionlet's talk about brightness so as you'llseeyou can kind of split down the middlewhat we'll have to say about imageformation so the first half is the onethat's you knowcovered in physicsprojectionit answers the question whereso what is the relationship betweenpoints in the environment and points inthe image and well raised you know youconnect it with a straight line throughthe center of projection and you'repretty much done that's calledperspective projection and we'll talkabout that but then the other part ofhalf of the question is how bright youknow what is the gray level at a pointor in color terms rgb values at a pointand so that'sless often addressed in some othercourses and we'll spend some time onthat and obviously we'll need to do thatif we're going to solve that shape fromshading problem for exampleso this is what is this so we've gotthree pictures heretaken from pretty much the same cameraorientation and position of downtownmontreal and obviously if you go to aparticular pixel in the three imagesthey're going to have different valuesright because the lighting's changed sowhat this illustrates right away is thatillumination plays an important role andobviously we'd like to be insensitive tothatin fact you know if you showed anyoneany one of these three picturesseparately they say oh yeah okay that'splus sound ville marie and you know theywouldn't even think about the fact thatthe gray levels are totally differentbecause we automatically accommodatethat that differenceso we'll be looking at diagrams likethiswhere we have a light sourceshown as the sun andan image device shown as an eyeand atiny piece of the surfaceand the three angles that controlreflectionand so what we seefrom thatdirection is the function ofwhere that light comes from what type ofa material it is and how it's orientedand we will particularly focus on thatorientation question because if we canfigure out what the surface orientationis at lots of points we can kind of tryand reconstruct the surfaceand you know there's that business ofcounting constraints again because thewhat's the surface orientation it's twovariables right because you can tilt itin x and you can tilt it in y that'ssort of the crude way to see why that isand what are we getting we're gettingone brightness measurement so we're kindofit's not clear you can do it it might beunder constraintand the image you get of an objectdepends on its orientation and the wayi've shown it here is to show the sameobject basically in many differentorientations and not only does itoutline change but you can see thebrightness within the outlinedepends a lot on that as welland things depend a lot onsurface reflecting properties so on theleft we have a matte surfaceyou know white matte paint out of aspray can and on the right we have ametallic surface and so even though it'sthe same shape we have a very differentappearance and so we'll have to takethat into account and try and understandum how do you describe that you knowwhat whatequation or whatterminology shall we use for thatso we'll sort of jump ahead here toone approach to this question which isyou know suppose we had suppose we livedin a solar system with three suns thathave different colorsnothis is what we get there's a cubeand it will make things very easy rightbecausethere's a relationship between the colorand the orientationso if i have that particular type ofblue up there i know that the surface isoriented in that particular way so thatwould make the problem very easy and sothat leads us to an ideaof how to solve this problemsoas i mentioned there's the so-called binof parts problems which we werefoolishly enough to believewhat the mechanical engineers wrotein the annual report so what they saidwas here the 10 most important problemto solve in mechanical engineering andthis was i forget number two you knowhow to pick parts when they're notpalletized when they're not perfectlyarranged and so here the task is to takeone after another of these rings off thepile of rings and of course becausethey're lying if they were just lying onthe surface it would be easy becausethere are only that many stablepositions well for this object there areonly twoand so it would be prettystraightforward but since they can lieon top of each other they can take onany orientation in spaceand also they obscure each other andalsoshadows of one full on the other so itgets more interestingand you can see that it took manyexperiments to get this right so theseobjects got a little bit hammered so youhave to be insensitive to the noise dueto thatand we need a calibration so we need toknow the relationship between surfaceorientation and what we get in the imageand so how best to calibrate well youwant an object of known shape and youknow nothing better than a sphere forthatum don't it's very cheap you just go tothe store and buy onethat don't have to manufacture aparaboloid you know or somethingand this may be a little odd picture butnow this is looking up into the ceilingso in the ceiling there are three setsof fluorescent lights and in this casethey're all three turned onbut in the experiment they're used oneat a time so we have three differentilluminating conditions and we get aconstraint at each pixel out of each oneso ta-da we have enough constraintsright we've got the three constraints atevery pixel we need two for surfaceorientationand we have an extra one well the extraone allows us to cope withalbedo changes in reflectanceso we can actually recover both thesurface orientation and the reflectanceof the surfaceif we do this with threelights so here's our calibration objectilluminated by one of those lights andnow we repeat it with the other twoand just for human consumption we cancombine the results into an rgb pictureso this is actually three separatepictures and we've used them as the redgreen and blueplanes of a color pictureand you can see thatdifferent surface orientation producedifferent colors meaning differentresults under the three illuminatingconditions and so conversely if i havethe three imagesi can go to a pixel read off the threevalues and figure out what theorientation isand you know you might see a few thingsone of them is that there's certainareas where the color is not changingvery rapidly well that's bad rightbecause that means that if there's somesmall error in your measurement youcan't be sure exactly where you are andthe other areas where the color ischanging pretty dramatically and that'sgreat because any tiny change in surfaceorientation will have an effect and soone of the things we'll talk about isthat kind of noise gain that sensitivityto measurement errorwhy worry about it wellimages are noisy so first of alla lot of the images you're looking atthe 8-bit images that's one part in 256.you know that's really crudequantization and you can't even trustthe bottom one or two bits of thoseif you're lucky and you get raw imagesout of a fancy dslryou might have 10 bits or 12.another way to look at it is that apixel is smallhow big is the pixel in a typicalcameraso we can sort of figure it out so thechip is you know a few millimeters by afew millimeters and we got a fewthousand by a few thousand umcolumns and rows so it's sort of a fewmicrons and you know they're hugetrade-offs like the one in your in yourphone has smaller pixels the one in thedsvslr has larger pixels but in any casethey're tiny now imagine that there'syou know light bouncing around the rooma little bit of that light goes throughthe lensand a tiny tiny part of that gets ontothat one pixel so the number of photonsthat actually hit a pixel isrelatively small it's like a million orlessand so that means that now you have toworry about statistics of counting youknowas you as you can imagine if you if youhave 10 photons is it 9 is it 10 is it11 you know that's a huge error so ifyou're a million it's already betterit's like one in a thousand but so thenumber of uhphotons that cango into a single pixel is small but notonly is there a little light coming inbut actually the pixel itself can'tstore that much the photons areconverted to electrons each pixel islike a tiny capacitor that can take acertain charge before it's full soanyway images are noisy so we have to beuhcognizant of thatso that was the calibration now we go tothe real object and again differentsurface orientations produce differentcolorsfrom that we can construct thisso-called needle diagram so imagine thatwe divide the surface up into littlepatches and at each point we erect thesurface normaland then these tiny littles may be hardto see but they're tiny little bluishspikes that are the projections of thosesurface normals so in some areas likehere they're pretty much pointingstraight out at you so here you'relooking perpendicular onto the surfacewhereas over here the surface is curvingdown and you're looking sideways sothat's a description of the surface andwe could use that to reconstruct theshapebut if we're doing recognitionand finding out orientation we we mightdo something elseuh so here you see it's actuallyslightly more complicated because you'vegot shadowsand it's harder to see but there's alsointerflexionthat is with these white objectslight bounces off each of them in amatte way goes everywhere and it spillsonto the other surfaces so it's notquite as simple as i explainedso what do we do with those surfacenormals well what we want a compactconvenient description of shapeand for this purpose one suchdescription is something called extendedgaussian image which we'll discuss inclass where you take all of thoseneedles and you throw them outonto a sphere and so for example forthis objectwe have a flat surface at the topall of those patches of that surfacehave the same orientation so they'regoing to contribute that big pile ofdots at the north pole sojustcut that short it's a representation in3d that's very convenient if the objectif we need to know the orientation ofthe object because if we rotate thisobject that representation just rotatesyou can think of many otherrepresentations that don't have thatpropertyokay so here it is[Music]you could imagine that it wasn't easy toget the sponsor of the project to payfor these parts herei i think they were concerned they werenot for experimental purposesso umthis is a single camera system sothere's no depth so the way this worksis thatyou do all this image processing youfigure out which object to pick up andhow it's oriented and then youreach down with a hand untila beam is interruptedthen you know the depth so here the beamis interrupted and now the robot backsup and reorients the hand for graspingand then it comes back and grasp thatobject andand so onand i show this because another thinganother calibration i left out was whati previously mentioned the relationshipbetween therobot coordinate system and the visionsystem coordinate system andone way of dealing with that is to havethe robot carry around something that'seasy to see and accurately locatablethis is something called the surveyor'smark because surveyors have used thattrick for a very long timeit's easy to process the image and youcan find the location of theintersection of these two lines veryaccurately with subpixel accuracy so youmove that around in the workspace andthen find the you know fit thetransformation to it and then you canuse that to[Music]okayback tomore serious stuffso umso that should give you a sort of tasteofthe kind of thing thatwe'll be doingand what i'm going to do now iswork towardsumwhat you need for the homework problemso so first um are there any questionsaboutwhat you saw i mean a lot of that'sgoing to get filled in as we go throughthe termokaysoi mentioned this idea ofinverse graphicsso if we havea world modelwe can make an imageand you know the uhpeople who are into graphics will hateme saying that but that's the easy partthat's the forward problem right it'swell defined and and the interestingpart is how do you do it well how do youdo it fast how how do you do it when thescene has only changed slightly and youdon't want to have to recomputeeverything and so onbut what we're trying to do is kind ofinvert thatprocessso we take the image and we're trying tolearn something about the world now wecan't actually you know reconstruct theworld we typically don't end up with a3d printer you know doing that umusually this is this ends at a kind ofdescription there might be a shape oridentity of some object or itsorientation in spaceumwhatever is required for the task thatwe have it might be some industrialassembly task or it might be you knowreading uh the print on a pharmaceuticalbottle to make sure that it's readableand so on so but but that's the kind ofloop and that's why uhwe like to talk about it as as inversegraphicsnow to uh to do thatwe need to understand the imageformationand that you know sounds prettystraightforwardbut it has uh two partsboth of which will you know exploreuh in detail as we go alongthen umwith inverse problems like here we'retrying to invert thatwe often find that they're ill-posedand as i mentionedthat means that they you know don't havea solution have an infinite number ofsolutions or have solutions that dependsensitively on the dataand that doesn't mean it's hopeless butit does mean that we need methods thatcan deal with that and often we'll endup with some sort of optimization methodand in this coursethe optimization method of choiceis least squareswhy is that well you know the fancy uhprobability people will tell you thatthis is not a robust method if you haveoutliers it won't work very well andthat's that's great but in manypractical cases least squares is easy toimplement and leads to a closed formsolution wherever we can get a closedform solution we're happy because wedon't have iteration we don't have thechance of getting stuck in a localminimum or something so so we'll bedoing a lot of least squares but we haveto be aware of i already mentionednoise gainsonot only do we want to have a method forsolving the problem but we'd like to beable to say howhow robust it is you know if my imagemeasurements are off by one percent doesthat mean that the answers arecompletely meaningless or does it meanthat they're just off by one percent sothat kind of thing okaydiving right inuh we're going to address this problemand it's kind of straightforwardand we'll start off with somethingcalled the pinhole modelnow we know that you know real camerasuselenses or in some cases mirrorswhy pinholes well that's because theprojectionin the camera with a lens is the same asit's trying to be exactly the same as apinhole cameraby the way there's a great example of apinhole camerain santa monicait's a camera obscurayou walk into this uh small buildingthat's completely windowless it'sdark inside and there's a single hole inthe wall and on the other sideon the other wall painted white you seean inverted image of the world and yousee people walking by and so on sothat's a nice example of a pinholecameraso here's a box to keep the light outand then we have a hole in it and on theopposite side of the boxwe see projected a world of view of theworld solet's just try and figure outwhat that projection is so we've gotso there's a point in the world uhuppercase pand there's a littlelittle p point in the image plane so theback of the box is uh going to be ourimage plane and you know our retina isnot flat we're just going to deal withflat image sensors because you know allthe semiconductor sensors are flatand if it's not flat we cantransform sobut we'll just work with thatokay so what we want to know is what'sthe relationshipuh between these two and so this is a 3dpictureand now let me draw a 2d pictureokay so we're going to call this fand f isalluding to focal length although inthis case there's no lens there's nofocal length but we'll just call thatdistance f and we call this distancelittle xand we call this distance big xand this distance big zsoum in the real world we have a big x bigy big z and in the image plane we havelittle x and we're going to have alittle yand and fandwell there's similar trianglesso we can immediately writeokay and now umyou know although this isn't completelykosheri can do the same thing in the y planesoi can draw the same diagram just slicethe world in a different way and i getthe companion equationand that's it that's uh perspectiveprojectionnow why is it so simplewell it's because we pickeda particular coordinate system so wedidn't just you knowuhhave an arbitrary coordinate system inthe worldwe picked a camera centric coordinatesystem and that's made the equationsjust about trivial so what did we dowell this point here is called thecenter of projectionand we put that at the originwe just made that 0 0 0 in thecoordinate system and so this is alsouh copand then here's the image planeip image frameokay so we did two things the one was weput the origin at the center ofprojection and the other one is we linedup the axis with the optical axisso what's the optical axis well in alens you know a lens has a cylindricalsymmetry so it has a the cylinder has anaxis but there's no lens here but whatwe can do is we can look atwherea perpendicular drop from the center ofprojection onto the image planestrikes the the image planeso we'll use thatas a referenceand so that's are going to be ouroptical axis it's a perpendicular fromthe center of projection onto the imageplaneand we line up the z-axis with thatthat's going to be our z-axisso it's a very special coordinate systembut it makes the whole thing very easyand then if we do have a differentcoordinate system on our robot orwhateverwe just need to deal with thetransformation between this specialcamera centric coordinate system andthat coordinate systemnow one of the things that's veryconvenientwell not only are they going to make mewalk all across campusbut i'm going to get upper body strengthas well this is greatokay so what we do is um we flip theimage plane forward soyou know the image on your retina isupside down and in many cases that'ssort ofinconvenient so what we can do is we canjustpretendthat the world actually looks like thisright that's pretty much the samediagram we've just flipped 180 degreeswhat was behind the camera and in frontand it makes the equations even moreobvious right the ratio of this to thatis the ratio of this to thatnow that's you know sounds kind ofstraightforward and somewhat boringbut it has a number of implications thefirst one is it's non-linearright so you know we know that thingsare linear our math becomes easy and soonbut here we're dividing by zso on the one handthat's a inconveniencebecause you know like you takederivatives and stuff for the ratiothat's not so nicebut on the other hand it gives us somehope because if if the result depends onz we can turn that on its head and sayoh maybe then we can find z right sowe canmake uh take get an advantage out ofwhat seems like a disadvantageand thenthe next thing iswe won't do it today but we'll be doingit soon is to talk about motion so whathappens well we just differentiate thatequationwith respect to time right and what willthat give us right now we have arelationship between points in 3d andpoints in the imageand when we differentiate we can get arelationship between motion in 3d andmotion in the imageandwhy is that interesting well it meansthat if i can measure motion in theimage which i cani can try and guess what the motion isin 3dnow the relationship's not that simplefor example if the motion in 3d is isstraight towards me you know thebaseball bat is going to hit me in theheadthen the motion in the image is veryvery smallsoyou know i have to take into accountthat transformation buti do want to know that relationshipbetween motion and 3d and motion and 2dand i get it just by differentiatingthattheni want to sort of introduce severalthings that we use a lot in the coursethe next one is vectorsso we're in 3d why am i talking aboutcomponents i should be just usingvectorsso first of all notationum you know in publicationsuh in engineering publication not mathpublicationvectors are usually denoted with boldlettersuh and so if you look at robot vision orsome paper on the subject uh you'll seevectors involved now i can't do bold onthe blackboard and so uh we useunderlineand actuallyyou know there was a time where youdidn't typeset your own papers just asecondbut you somebody at the publishers typesit your paper so how did you tell themto touch it involved you underlined iti mean the camera actually works the waythat works up there in most cases uhsome of them will have mirrors to foldthe optical paththis is like a conceptual conveniencejust tomake it easier not to have to i meanmaybe some people don't have a problemwith minus signs but to me it'sconfusing having that one upside down soso i prefer to do it this way but theactual apparatus works that wayokay so the other bit of notation weneed is is a hatfor unit vectorbecause we'll be dealing with unitvectors quite a bit for example you sawthat we talked about the surfaceorientation on that doughnut in terms ofunit vectors it's a direction so we usea hat on top of a vectorandokay so let's turn that into a vectornotationwell umso i claim thatthis is basically the same as that upthererightbecause if you go component by componentthe first component is little x over fis big x of a big zuh the second component is little y overf is big y of a big z and the thirdcomponent is f over f is z over z sothat that doesn't do anything for usthat so that's the um equivalentandnow i can justyou know define a vector rso this is our little rokayand umnow i've got sort of a mixed notationright because i got a big z in herewell that's the third component of big rvectorso i justdot product withunit vector zright so let me write that out in fullso that's x y z transport dotright so the unit vector in the zdirection along the optical axis is just0 0 1 right and soso i finally have the equivalentof the equations up there in componentformi have it here invector form so that's perspectiveprojection in vector form now usually atthis point you say look how easy it gotby using vector notationwell you know it isn't really any easierlooking this is one of those rare caseswhere it didn't buy you a whole lot interms of number of symbols you have towrite down and so on neverthelessthe compactness of that notation comesout when we start manipulating it youknow if you have to carry around allthese individual components all the timethat can get pretty tedious whereas ifyou use the vectorit's uhmore interesting and as i've mentionedone of the things we're going to do soonis differentiate that with respect totime and then on the left we'll haveimage motion and on the right we'll havereal world motion and the equation weget will give the relationship betweenthe twoumokayso uh this may sound you know a littlebit uhhaphazard and chopped up to what we'redoing today and that's only because iwant to coveryou know stuff in chapter one and twoand um material you need foruh the homework problem so we're gonnarather than pursue perspectiveprojection more we're going to jump tobrightness uh in a second but first letme say something else uh which is thatumi'm thinking of these vectors as columnvectorsyou know that's sort of arbitrarybecausewe can establish a relationship betweenskinny matrices and vectors either wayright i can think ofyou know x y and z stacked up verticallyabove each other as a three by onematrix or i can write them horizontallyx y z and it's a one by three matrix andjustfor consistency i'm always going tothink of them as column vectorsand that's why sometimes i need atransposeand that's what the symbol t is for so ididn't say it here but we can now goback to this so if i write it this wayit's a row vector but actually all myvectors are supposed to be columnvectors so istuck in the transpose soso another bit of notation it's allpretty straightforwardokay let's talk about the brightnessso brightness depends on a bunch ofdifferent thingsit depends on the illuminationand in a linear way uh in that if youthrow more illumination on an objectit's going to be brighterandyou know the the few laws that arereally really linear this is linear overmany many many orders of magnitude imean when does it stop being linear wellwhen you put so much energy on thesurface that you're melting it you knowyou have to actuallyhave enough energy to fry it and it's alittle bit like ohm's law which is alsoone of those remarkable things thatfor some materials is linear over manymany orders of magnitude anyway sobrightness depends on the illuminationand then it depends on howthe surface reflects lightumandso we'll have to talk about that nowobviously there's a difference in termsof amount so like you know my my laptopreflects relatively little lightwhereas uh you know my shirt reflectsmore lightumanyone want to guess uhwhat the percentage of incident solarradiationthat the moon reflectsit's a trick questiondo you happen to knowbut it sort of looks white in the skyso it's got to be 90 or somethingit's 11.it's as black as coaland so why don't you know that wellbecause you have no comparison now if iwent up there with a sheet of whitepaper and held it next to the moon you'dsay oh yeah god it's really dark but noone does that it just hangs up there andyou have no reference sothis business about brightness is trickyyou know you got to be careful aboutthatand why by the way why is it as dark ascoal it's because of solarradiate solar windimpinging on the surfaceand you also probably know that wherethere were quote recent craters likeonly in the last few million yearsthere's bright streaks that's becausewhere the underlying material is exposedand the sun hasn't yet done its work onthem so anyway so brightness depends onthe reflectance how about distanceright if i have a light bulbit gets you know it's less intense wheni go further away right so there's ainverse square lawso does that apply toimage formationin in a more normal sense like you knowif i walk away from that wallif i stand on this side of the roomuh that wallis only a quarter as brightas when i stand over here rightrightdo you believe thatokay i can sell you a bridge in brooklynand thenno of course it's the same brightnessand and you know thatso what's going on why is it not gofollow the inverse square law well thereason is thatat the same time as i'm getting closerthe area that's imaged on one of myreceptorsis larger on the wallor if you want to think of it in termsof the little light bulb so the littleledimagine now the wall is covered withlots of ledsand each of them does in fact follow theone over r squared lawbut if you think about how many leds areimaged on one of my pixelsthat goes as a square of the distance sothe two exactly cancel out and so infact uhwe can x that one outsoand umsowhatelse does it depend on well it doesn'tdepend on the distance itself but itdepends on the rate of change ofdistance or orientationand and you know not not in a terriblysimple way but we can start with asimple exampleso uh here's a surfaceelement some little patch of a surfaceand here's a light sourceandwhat we find is thatthere's foreshortening that is thepowerthathits the surfaceper unit area is less so i can measurethe power in this planeyou know so many watts per square meteruh which in the case of the sun is abouta kilowatt per square meterbut obviously that same energy is spreadout over a larger area right this thislength is bigger thanthan that lengthand so theillumination of this surface is lessand how muchwell we can express it in terms of thisangle which is the incident angle thetaof iand that is the same angle as that anglei thinkand umthere's a cosine relationship rightbetweenthis red length and this lengthso we find out thatin this caseum the illumination on the surfacevaries as the cosine of the angle andthis issomething that we'll see again and againnow it doesn't necessarily mean that itsbrightness the amount of light itreflects goes as the cosine of theincident anglethat is the simplest caseand so here's an examplewherewe could use an image brightnessmeasurement to learn something about thesurfacerightbecause we can look at different partsof the surface and they'll havedifferent brightnesses depending on thisangle now does it tell us theorientation of every little facet of theobjectokay some people are shaking their headsno right because again it's onemeasurement two unknownswhy are they two unknownswell one way to see itis tothink of a surface normal a vectorthat's perpendicular to the surfaceand the way i can talk about theorientation of the surface is just totell you what that unit normal is rightso how many degrees of freedom are therehow many numbers do i well i need threenumbers to define a vector so it soundslike three except uh i have a constraintrightthree componentslike this isn't just any old vector thisis a unit vector so x squared plus ysquared plus z squared equals 1. so ihave one constraintso actually surface orientation has twodegrees of freedomand and since this is such an importantpoint let's look at it another way soanother way ofspecifying surface orientation is totake this unit normal and put it itstail at the center of a unit sphereand see where it hits thesphereand so every surface orientation thencorresponds to a point on the sphereand i can talk about points on thesphereusingvarious ways but one islatitude and longitudeand that's two variables so that tellsus again in another way thata unit normal has two degrees of freedomand if i want to pin it down i betterhave two constraints so that's the waythat was goingand that that makes it interesting imean if if if we could just say okay imeasure the brightness and it's you know0.6 and the orientation is such and suchuh the course would be over it'd bepretty boring but it isn't it's it's noteasy we need more constraint and we'llsee the different ways of solving thisproblem one of them you saw in theslides was a kind of brute force onesaying well we just get more constraintwe illuminate it with a different lightsource we get a different constraintbecause the other light source will havea different angle and then i can solveat every pointso that'syou know from an industrialimplementation point of view that'sgreat you can do that you can either usemultiple light sourcesput on at different times or you can usecolored light sources and so on but youknow suppose you were interested in howcome people can do this they don'tplay tricks with light sources and theydon't live in a world with three sons ofdifferent colorsthen you know we'll have to do somethingmore sophisticated and we'll we'll studythatokay how are we doing how we're gettingthereokay so um the foreshortening comes upin two places the one is here wherewe're talking about incident lightbut actually for shortening also plays arole[Applause]in uh in the other direction whoa highfriction blackboardalso hard to eraseokay so umso it's really the same geometryexcept now the rays are going in theother directionandsoand i have a foreshortening on thereceiving end as wellso in a real imaging situation in 3dwe'll see both of these phenomenathere's the foreshorteningthat affects the incident illuminationas up there and then there's this effectandfor example i can illustrate to youright away the stupidity of sometextbooks so some textbooks say thatthere's a type of surfacecalled inversionwhich emits energy equally in alldirectionsthat's what they literally saywell if that's truethen that energyisimaged in a certain areathat changes as i change the tilt of thesurface and as i tilt the surface moreand more and more that image areabecomes smaller and smallerbut it's receiving the same powersupposedly according to these guysand what does that mean that meansyou're going to fly your retina right atthe occluding boundary because all thatenergy is nowyou know focused in the tiny tiny areasothis is an important idea and it comesin when we talk about thefrequency of surfaces and we need to beaware of itsonow something i want to end up on iswe're solving a tough problem with youknow 3d we only got 2d images okay somaybe we're lucky in we have several butit's it's you know you've got a functionof three variables that's got so muchmore flexibility thana few functions of two variables so whydoes this work at allwell the reason it works is that we'renot living in a world of color jelloso we're living in a very special visualworld so if i'm looking at you know someperson back therethe ray coming from the surface of hisskin to my pupilis not interrupted and it's a straightlinewhy well because we're going through airand air is a refractive index almostexactly one and at least it doesn't varyfromthat position to hereand there's nothing in between there'sno smoking allowed in this room so itcan't be absorbedand and that's very unusualuh nowand the other thing is that that personhas a solid surface i'm not looking intosome sort ofuhsemi-translucent complicated thing uh sothe straight-line rays and there's asolid surface therefore there's a 2d to2d correspondence the surface of thatperson sorry i keep on looking at thesame person who's getting embarrassedbut2dwe can talk about points on the cheek ofthis person using you two vectors u andvand that's mapped in some curvy linearway into the 2d in my image and that'swhy that's one reason why this worksright it's not really 3d to 2d it's sortof a curvy linear 2d to 2d and what'sthe contrast wellsuppose i fill the room up with jelloand then somebody goes in and with thehypodermic injects colored dye all overthe show and then i come in the door andi'm not allowed to move around i canjust stand at the door and i can look inthe room can i figure out thedistribution of colored dyesno because in every directioneverything is superimposed from the backof the room to the front and soyou can't disentangle it from one viewcan you do it yeah if you have lots ofviews and that's tomographysowe're sort of in an interesting worldyou know tomography in a way is morecomplicated but it's also in a way muchsimplerthe math is very simpleand we have a world where there's amatch of dimensions but the equationsare complicated so it's not so easy todo that inversionsee if we need i think we need to stopokay any uh questionsso about the homework problem you shouldbe able to doat least the first three the fivequestions probably the fourth and thenon tuesday we'll cover what you need todo the last oneyou

Image Formation, Perspective Projection, Time Derivative, Motion Field

we'll start off by talking a little bitaboutreview of things we've done last timeperspective projectionanda little bit aboutthings that are relevant to the homeworkproblemparticularly the last two questionsand we'llgo from perspective projection tomotion so in perspective projection wehave a relationship between points inthe environment and points in the imageand we found that we hadthis simple relationshipbetween the 3d world and the 2d worldas long as we pick a suitable coordinatesystemwith the origin at the center ofprojection and the z-axis along theoptical axisand so onbut then we're going togo from that to motion by uhdifferentiating that equationand then we'lltalk aboutmotion of brightness patterns in theimageitselfokay so we had that and then we hadavector versionuh let's seewhichis only slightly more compactso it's not uh in this case a great helpbut we'll use both of them we'll switchback and forth and use whatever issuitable for the problem that we'redealing withso let's umlet's start with motionsothat's a staticsetup and we can easily relate points inthe environment to points in the imagenow what what if there's motion what ifthat point in the environment movesobviously there will be some motion inthe image and if we can find out whatthat relationship is then maybe in somecircumstances we can invert that so thatwe can measure the motion of brightnesspatterns in the image as you do in thehomework problem and somehow use that tofigure out what's happening out in theworld so we just differentiatewe take ourperspectiveequation and we differentiate so we getnow on the right hand side[Music]we have a ratioand so there are two parts to thederivativeandum we just apply the rule fordifferentiation of a ratioand what i'm going to do a lot isreduce theintimidation level of the equationbysubstituting more easily digestedsymbols so in this case we've got allthese derivatives well what are theythey're really velocities so there'sgoing to be a velocity in the image inthe x direction which i'll call uandthere's a velocity in the real worldin the x direction which i'll call big uand then we havea velocity in the real world in the zdirection which i'll hold wso that looks a little bit less scaryand of course i can do the same in the ydirection so apply the same idea overhereso that's the forward direction that isif you tell me what the motion vector isin 3dhere's the formula to tell me what themotion vector is in 2dand of course we might want to invertthat but for the moment let's go withthatnow we can rewrite this in various formsand since we'll use this a lot let's dothat so one thing we can do iswe can split upthese terms into1 over zx over zand why is that useful well because weknow that x over c is little x over f sowe can rewrite thisusing image coordinatesand we end up withandone thing this allows us to do is to askthe question uh where in the image isthere no motionso ifumwe're looking at themovement of brightness patterns in theimage there may be some places wherethere isn't any and those are ofparticular interestfirst of all because we can find themusing image processing techniques andthen because they tell us somethingabout the environment or the motion sosoyou cansee if we set thisequal to zerouh we find that's true at a point xnaught such that x naught over f is abig u of a big w and similarly for thisone equal to zero and so the point xnaught y naught is that uh sort afterpoint where there is no image motion andwe call it the focus of expansion and inthe case of a simple situation like i'mapproaching the wallthe focus of expansion is the pointtowards which i'm headingand sowe can just express it directly in termsof the 3d motion vector and you'llnotice that this is actually just aprojectionof the 3d motion vectorinto the image planeso i take that same diagram over thereand now instead of mappingthe vector to the point p i'm mappingthe velocityand if i do thatthenthat defines this point x naught ynaught in the image where there is nomotion and that's called the focus ofexpansionokay uh that's a useful thingbecause if you can find the focus ofexpansionyou all you need to do is connect thatpoint x naught y naught to this to theorigin and you have a vector that tellsyou the direction of motion soa nice exampleof inverting that processokay now once we have the focus ofexpansion we can rewrite those equationsyet one more timeand so there are a couple of thingsthere that are of interestone of them is thatthe f's cancelsowe can do thatand then we actually can get an idea ofwhat the image is going to look likesolet's suppose this is the image planenow we're looking at it straight on andsuppose thatthis is our focus of expansionand you know we're moving with respecttothe environment is moving with respectto us so we're moving with respect toenvironment you know einstein said thatit's all relative it doesn't matter theonly thing we can determine is thedifference between the two motionsthenthere will be no image motion here andwhat about other places well this uhtells youexactly what the pattern will be likeright so if wego to the right such that x naught minusx is[Music]uh positive while we keep this one zerowhat happens is well u will becomelarger the further we go away the largerso we'll have a little vector herethat tells us the image motion at thatpoint and another vector over here whichis perhaps largerand then we can go to where this isnegative so the other sidedraw a vector hereokayso i'm starting to draw a little vectordiagramum of themotion field if you like how things areexpanding from that pointand i can do the same in the y directionso i suppose i keep x naught equals xbut i change y naughtso i'm going along this linei'm going to get vectors that point thiswayandlet me be adventurous and go off atright angles wherex naught minus x is the same as y naughtminus y well that means that u and v aregoing to be the samethat means i'm going to get a vector at45 degrees right soand opposite direction like thatand i think you can seethat we're going to fill in this thiswhole diagramthis little vector diagram with arrowsthat are pointing outwards from thefocus of expansion that that's what thisequation says that all these vectors areradiating outward and that's why it'scalled the focus of expansion so i'mmoving towards that walland this is the place i'm actually goingto hityou know maybe going at an anglebut my wholeimage is looming it's uhzooming outwards whichis a useful clueuh useful cue formeasuring distance and velocitywellas you saw in the homework problemthere's this scale factor ambiguity thatbecause of perspective projectionwe can't actually tell absolutedistancesand we have a similar phenomena herebut what wecan tell is is this ratioand so this whole field depends uhcritically on that ratio so so what isthatw over zumand if i make that larger then you knowthe vectors will be larger that meansi'm sort of getting closer to thesurface i'm about to hit itconversely if i make it smallerthat will be smaller if i make itnegativewhat happens well all of these vectorsare reversed so it's a focus ofcompressionthere's no there's no such term but wecould call it that um it's just theinverse of the focus of expansion andthat's what happens if i'm moving awayfrom the surface so i'm taking off fromthe lunar surface and the image is sortof compressing onas i leave sookay well umso what is this well let's turn it onits headuh what is z over wand w of course is the rate of change ofzso we got z over dz dtwhat are the units of that so z is insay meters and dz dt is in meters persecondso the units of that ratio areseconds and so uh can you think of whatthat quantity issorrytime to impactgreat sothat's going to tell mehow long it's going to take before icrash into this surfaceand so that's uh a quantity that we'regoing to spend some time thinking aboutuh both because it's important if you'resay landing anasacraft on europaumor if you're a fly landing on theceilinguh and also it turns out to berelatively easy to compute compared tosome of the other things that we'll belooking atby the way thelanding on europa is a jplrfpthey wanted to haveindustry give them ideas of you know howthey're going to do that and how muchit's going to cost andwell we told them that this is the rightway to do it and we never heard fromthem again so i don't know soso if uh that craft crashes on europadon't blame me i told them what to do sookay so umthis is beginning tolook kind of promising because we canactuallysee our way to inverting this imagingprocess and getting some usefulinformationjust from the way the brightness patternin the image changesbut i need totorture you a little bit moreand do this in vector form andwe'll see later why this is useful rightnow it's not saving us a whole lot interms of writing going from the scalethe component form to the vector formbut uhwe'll see that it does eventually okaysoinstead of differentiating the componentversions we're going to nowdifferentiate the vector form and ofcourse it's very similarso what's the time derivative of avector wellyou know it's just the vector whereyou've differentiated each of thecomponents with respect to time so it'snothing very fancyso we have that term again thatcorresponds to this first part here butbecause it's a ratio and because r canbe changing big r can be changing weneed to take account of the fact thatit's a ratio and we getandokay soto make it look slightly lessintimidating we'll switch fromleibniz to newtonian notationagain the underline always means thatit's avectorand i'm splitting this up a bitokay so the dot here denotesdifferentiation with respect to timeand souh we have two terms one of which issort of obvious scalingthat there's a certain uh motion in 3dand it's going to be magnified orde-magnified by the ratio ofof these two distances sothere's a big motion out here it's asmaller motion in there by the ratio ofthosebut then there's this other term thatcorresponds to motion in depth that wehave to take into accountand i rewrote it this waybecause again we can make use of thefact that we have this perspectiveprojection equation and so we canintroduce the the image coordinatesokay[Music]so focus of expansion again would bewhere r dot is zeroand so that corresponds tozone oh let me put the one over f on theother side that's easierand so umwe basically get the same equations imean if we just write out thecomponents of that equation we getwhat we had beforethenjust wanted to do one more little trickwhich isin the appendix of the book we have someusefulresults aboutall sorts of simple math that's neededone of which has to do with uh vectorequations and withmanipulatingyou know cross products of crossproducts that kind of thingwhichi typically don't remember how to do soi'll go to the appendix of the book andthe appendix isunder materials on the seller websiteokay solet me first transform this one moretime so we've gotokay that's just rewriting thisand now we usea resultand the reason we do this is becauseit allows us to make some generalstatements about the flowokay so cross product of cross productscan be expressed as a difference of dotproductsnow you can match upthese two linesand you can see where we're going withthis which isokay so finally there we have theimage motion expressed in terms of worldmotionusing thissomewhat odd looking expression so whyis this of interest well the first thingyou can say is thatthe result is perpendicular to z rightbecause if you take the cross product oftwo vectors the result is perpendicularto both of the vectorsokay sor dot equals zeroumr dot z equals zero excuseme so is is that surprising that theimage motion is perpendicular to thez-axissoimage motionsin that planeright z-axis is there no no of courseit's obvious i mean it couldn't be anyother way if we got it this is a goodway to check the result if we gotanything else there'd be a problembecause soum in the image we only have 2d we havex yandwe have as velocities u and v we don'thave a velocity in the z direction thatwould be popping out of the image sothat'sinteresting and then something else wecan look at is[Music]what ifthe image motion is radially outward orinward that is it's uhthe motion is along the radius vector tothe point in the scenewhat do you expect will happen to theimage motion in that caseso you have the baseball coming straightfor you its image is doing whatit's expanding but it's not moving rightsothis is the case where we would expectimage motionto bezeroand now we check this formulaokay if r dot and r are parallel when wetake their cross product now the crossproducts length is proportional to thesign of the angle between the twovectors well if they're the same vectorthen that angle is zero and so the signis zero and soso this formula while it you know it uhtook a little bit of work to get thereit's helpful for answering certainquestionsand we can right away check things likethis okay ifumif the thing is coming straight at youor you're moving directly towards itthere'll be no image motion well that'sour focus of expansion right so if i'mmoving towards the wallthat is the point wherethemotion vector lines up with thedirection to that point and so i get noimage motionokaywhile we're herewe can look at some flow fields[Applause]so we already sawa flow field up therefor a general motion towards some pointand we could look atjust a couple of other cases so let'ssuppose that umyou know u v and w could be anything butlet's make itokay sowe're assuming now that two of thosecomponents are zero so there's onlymotion in the world in in the xdirectionand then what do we expect uh in theimagewell from our formulas uhwe see that uh the the only component ofthe image motionis a little little usoif the worldis moving in x or i'm moving in xrelative to the worldthenthe vector fieldofhow brightness patterns move in theimage looks like this[Music]one thing that to note though is thatthe length of those vectorsis not the same thatthe length depends on z or if you likethe inverse of zand so on so if you look at all of theseformulas you know we've got r dot zwhich is big z and look at thoseformulas up there you've got one over zso umyes this is a very simple vector fieldbut the vectors aren't all the sameso that kind ofis unpleasant at the same timeum when something's unpleasant you canoffer you can sometimes take advantageof that so in this caseif we had a vector field like this onething we could try and do isrecover depthright because these things are allinversely proportional to depthand so again anillustration of how once we understandthe forward process uh we can trim thataroundto try andsolve the inverse problemumokay u is zero andlet's try this onewell you know that's just turningeverything 90 degrees so that's not veryinterestingthe same ideanow i guess to exploit this idea ofrecovering uh depth from these fields itwe would need to umknow z we need to know the velocity sothere's sort ofuh two things affecting what thesepictures look likeone is the object the shape the distanceand the other one is the motion and ifwe know either one of the two then wecan calculate the other one of course inpractice often we don't know either oneand we'd like to recover bothand then often we end up with the illpose problemhow about u is zero v is zerow is not equal to zerowell we already drew that diagram upthere kind ofright if we look at the equations we'rejust going to getf o e isso that's the simplest uh case of ofthat picture where the focus ofexpansion is right in the center of theimageokaywellthis is starting to hint at some of thethings we're going to pursuewe you know develop the equations forsome transformation like this betweensomething in the world and something inthe image and then we try to invert itand the forward one is alwayssimple you know there's the equation upthere just plug in whateverthe inverse isn't because often theremay be more than one solution or theremay be no solution or there may be aninfinite number of solutions orit's imposed in the sense that if youmake some small measurement errorthe answer is going to be perturbed by alarge amount so we'll have to be carefulabout thatbuttalking ofmotion of image brightness patternslet's talk about thatthat wasyou know that's in the homework problemso let'stalk a little bit about thatso we'll ignore for the moment wherethese images come fromuh what what is an image well an imageis a 2dpattern of brightness values so it's youknow e as a function of x and y if youlikeandwe already said something about that interms of brightness being power per unitarea we will make that moreclear later onright so his question was you know whatabout color and yes we will be talkingabout color at some pointwhich is kind of in the way is thisrepeated three timesumanda couple of reasons not to bring it inat this point one iswe'll we'll do itmore simply just using gray levelsand thencolor ispeculiarly human-centric in that theonly reason that we use three numbers torepresent color is because ourparticular color sensing system hasthree sensorsbut we'll talk about all of that uhlater on for for nowumyou knowassume that we have uh taken rgb andturned it intobrightnesswhich is uh you know depending on whoyou readsome combination of the three componentsmostly gbecause humanvision is most sensitive tog and judgesabsolute brightnessbased on some combination of r g and bokaynow a couple of things i'm going to dorepeatedly so i want to kind ofpreemptively talk about them one of themis i'm going to be switching back andforth between continuous and discrete sothese days with everything being digitalof course the images we get arediscrete they're quantized they'requantized in spaceand typically on a rectangular gridwhich is kind of not the best grid touse but that's what we useand then in brightness so we also don'tget continuous values for brightnesswe get them quantized often to as few aseight bitsbutit turns out thata lot of what we do is easier tounderstand in the continuous domain soi've talked here about you know e of x ythat's mybrightness pattern for any x and y it'lltell me what the brightness is the powerper unit area but in practice you knowit's going to be more like i have anarray of numbers with two indicesand i'm dealing withe sub whatevercorrespondinglyin the continuous domain we'll often betaking integrals you know for examplewithin some measurement that's local ata pixel but we want to extend it overthe whole image and we take either youknow a single integral or doubleintegral well of course in the discreteworld we'll be taking you know sumsbut you know those are verystraightforward transformationsbut they're very useful because on theone hand we want to implement this asactual code and on the other hand we'dlike to make iteasier to develop the math and it'salmostinevitable thatit's easier in the continuous worldthen we get the derivative soof coursein the continuous world i can look atthe x and y derivative of brightness andthe combination of these two is calledthe brightness gradient which is goingto be very important inmany of the things we do and inpractice i'm going to approximate thosebyyou know making some sort ofdifference first differenceand we'll see that you know that's oneway of doing itit's not a particularly good way ofapproximating the first derivative butuh we will talk about others andi don't know just in terms of uh writingthis is easier to write than that soi know that's not a big argument butwe'll find thatsome of these things have a closed formsolution in the continuous domain thatyou can easily obtain and it's harder todo the same in the discrete domain soi'll be switching back and forth okaywith that in mindlet's look at our 1dimagesoof course images aretypically 2d they are 1d sensors andthey have some benefits one of them isthatyou can build a 1d sensor witha much larger number of pixels in onedirection than you can a 2d sensor so a2d sensor although these days you know2d sensor could be i don't know2k by 4k some huge number uh but linearsensors have been around with manythousands of pixelsand the disadvantage of a linear sensoris to to get a real image you need toscan itand this is done quite a lot in industrywhere things are moving along on aconveyor belt and you can use arelatively cheap very high resolutionlinear array sensor and the conveyorbelt motion provides the other dimensionof the image it's also used in satelliteimaging wherevery high quality 1dsensorsareused and then the motion of thesatellite provides the scanning toproduce an imageanyway suppose we have a 1d image andsuppose things movesoumso this is at timetand this is at time t plus delta tand you know think think of your opticalmouse you're holding it down on thetable it has a short focal length lensthat's imaging the surface of the tableand if there's some texture to it asthere is on on the wood over herethen that texture is mapped onto theimage and if you move the mouse then theimage of that surface will move and yourjob is to accurately estimatehow fast it's movingand sohere's the picture before we you knowtwo time steps andand so we're trying to figure out youknow how do idetermine what happens so let's supposewe have avelocity of uh uand soum we've movedby that amount so let me indicate thatover here so that'sdelta xnow we're making some assumptions herethat we'll get back to in a secondone of them is that the brightnessdoesn't changeand so you know if your optical mouse islooking at the surface of the tablethenpresumably between one frame and thenext the illumination isn't changingit's using an led with a constantcurrent and it's taking 2 000 frames asecondso it's unlikely to change the geometryis changing a little bit because you'vemoved and now you're looking at thesurface from a slightly different anglebut for most surfaces the change inbrightness due to that is very smallsoright here the way i've drawn thesecurves i've assumed constant brightnessso let's get it right out frontconstant brightness assumptionand we'll see that there arecircumstances where that doesn't applybut there are lots of cases where thatis the case for example as i'm walkingaround the room you know your shirtstill looks maroonish red and it doesn'treally changeand so we areused to most of the world being somewhatconstant incolorconstant in brightness as welleven though we know that there'scircumstances where that's not the casefor example if i look at the reflectionof the lights up there in my cell phonethey of course move relative to the cellphone as i move around so and we'llwe'll talk uh some more about thatokayso uhlet'ssort of enlarge a little piece of thatokay so we we're blowing upthis area and why am i doing that welli'd like to use a linear approximationso i'm assuming that this curve sosecond assumption i'm assuming for themoment this curve is relatively smoothsuch that if i look at a small enoughpart of it i can approximate it as alineokay and then there's a change inbrightnessand there'ssome slope here and the sloperelates themotionto the change in brightness you can seewhere this is going i'd like to use thechange in brightness to determine themotionbecause you knowhow else are we going to measure themotion in the imagethe pixels stay in the same placeall we've got is the brightness at thepixel is changing and somehow we have toinvert thatuh relationshipokay so uhso in terms of e of x and tuh what is this sloperight so this is in the x directionand that's the rate of change of e withx and so it'sd e d x okayslopeand i'm going to write it that way we'llas a partial derivativeandyou know i might as well use the correctnotation right away we're going to needpartial derivatives because we have bothx and y and tand so we need to be clear aboutwhich type of differentiation we'retalking aboutokay[Music]sothen that's the slope so that means thatdelta e must be the slopetimesdelta xand so that's going to be and i'm goingto write this as e sub xum so again in order tosimplify notation i will oftenuse subscripts to denote partialderivativesand you know e sub x and e sub y we useso much that it would be a pain to haveto write them out in full every timeas we indicated those are the componentsof the brightness gradient which is usedfor all sorts of things including edgedetectionokay so nowlet me divide through bydelta tandthis i guess in the limit as i takesmaller and smaller timesteps is going to bethat partial derivativesand if you like i can rewrite it onemore timeokay soyou can see here whyyou know using the switch to thecontinuous domain is helpful becauseuh i can take the limit here as delta tbecomes uhapproaches zero and get the partialderivative uh of course in practice i'mgetting frames at a fixed rate and theyhave a certain interval i can't makethat interval infinitely smallso i end up with a much messierexpressionbut the two are obviously relatedokay so that's the result for 1d imageandthat allows us to recover the motionumthatso one thing i probably glossed over isthatas we go fromt to t plus delta t uh the brightnessdecreases so this delta e over here isactually in for positive slope the deltae will be negative so thatmay have gotten lost in here somewhereokay soremarkablyfrom a single pixeli can get the velocity in this 1d casenow importantly that's not true in the2d case and after all that's the case wewant to solve so we're going tohave to work a little bit harderand now the couple of things you'll seefirst of allthe faster things are movingthelargeret will be well that's not a surprise ifif the brightness is changing uh slowlythen you speed things up then thebrightness will change more rapidly it'slike you know taking your youtube videoand changing the playback rate the umwhen you change the playback rate youchange all the velocities by the samefactorand there and then youhave more rapid changes of brightnessyou know intuitively obvious anotherthing you'll notice is thatthere's a problemif e x is zeroright we can't do thisif e x is zero so if we're for exampleat uhsuppose we are up herewell then if the thing moves thebrightness change doesn't change orchanges only a tiny bit and so from thatyou can't really tell how far it movedyou pretty much have the same result ifit moved that much or whatever soand you know just in terms ofuh implementation you're dividing byzero so that'suh not going to be good and it's notjust zero it's when it's very smallwhy is that well because when it's verysmall it probably means that you don'tknow itaccurately as a result of the fact thatyou obtain it by subtracting two pixelvalues so so let's just againyou know go over this we'reapproximating this by saying it's oneoverdelta x e of x plus delta xright well you know that's fancynotation but basicallythe two pixels we take this value and wesubtract that value so we read out thegray level there read out the gray valuethere subtract themandthat's how we estimate ourbrightness derivatives and similarly sowhat do we do with et well et is derivedthe same way except thatwe have two framesand we pick the same pixelout of the two framesand we subtract their gray valuesright so it's just in a differentdirection that we're approximating thederivativeandwhy is that important well if they're umis a small brightness gradient e x issmall then those two values will be verysimilarand since they are not known preciselythey have error measurement error inthem we're now subtracting twoquantities that are similar in size sowe get a smaller quantity and it hasit's mostly noise at some pointright sothat tells us that this is something toavoidwe we don't want to be and that makessense i mean if the image is uniform andbrightness e x is zero um you know youcan move it and you can't tell that itmoved i mean you'd have to have somereference point like some texture on itto tell that it moved and you know ifyou had texture on it e x wouldn't bezero souhi i know i'm sort of belaboring thisover and over again but it's veryimportant because it means that thistype of measurement is a very noisy andbnot trustworthy unless certain imageconditions are satisfiedokay and then the next thing you can doand say well well we can estimatevelocity from a single pixel but we gotlots of pixels and that you know that'spart of what we're going to be usingheavily that okay the single pixelresult is not greatbut we got a million or 10 million andso we can dothings like least squaresto improve the resultdramatically so it may be that theresult from a single pixel is reallyflaky but that's okay because we dividethe error by a thousanduh if we uh think about the statisticsof having a million of these uh pixelsso um i keep on saying they're noisy andat the same time i'm saying we can dothis and the reason we can do it isbecause we got lots of them sookay so we talked about how toapproximatehow to switch back and forth between thecontinuous world and the approximatethe discrete world andnow i don't want to give away thesolution to the homework problembut the next step i guess would be toget a bunch of these and combine them sowe might do something like thisyou know if we have n pixelsthen we might try something like thisright so now instead of just using a twopixels we use those two pixels to do thecalculation then we use these two pixelsto do the calculation then we use thosetwo pixels to the calculation and so onand so we add them up so we have a lotof noisy measurements and we take theaverage and magicallythings improve and so just without goinginto you know hairy statistics orassumptions about probabilitydistributions roughly speakingwhen you average n values you you reducethe standard deviation by this squareroot of n soumand you know when you have thesepolitical surveys and they say they haveafive percent margin of error what theymean is that theythey talk to 400 peopleright because one of uh square root of400 is 20 1 over 20 is 0.05 sosame principleso um it doesn't improve linearly with nunfortunately that would be even betterbut it doesn't improve with the squareroot of n so if we have a million pixelimage it improves by a factor of athousand which is you know prettysignificantokaynow if we just did this in this simpleway we'd still run into troublefirst of all you know if e x is reallyzero you'd be dividing by zero so youcan't do that i guess you could leavethose pixels outand then if e x is small you still havean answer which is relatively bad sothere are places where the slope islarge where you get good information andnow you're polluting it by adding in asan equal you know information fromplaces where the slope is low andand that's why in the homework problemwe talk about weighting so don't justtake an average and treat each of themequally butyou know multiply by some weight factorand umthen of course you what happens to theone over n well you have to compensatefor the fact that you've now switched touh using uh weightsokay umas we mentioneduh images are really 2d so let's extendthis analysis to 2d and see what happensnow some of this may you know seemtedious and repetitive to some of youbut we're all on different wavelengthsso i'm going to get the next resultin several different waysand you know different approaches mayappeal todifferent people solet me do it this way so let's start bytalking about the image volumeso think aboutyou know videoand we're stitching together the framesso each of these cross sectionsis a frameandwe end up withthis three-dimensional thingwhich is brightness as a function ofx y and tand typically in practice we slice itacross this way but we could slice thisvolume any way you like and in somecasesthere are advantages to slicing it in indifferent waysokay so that's part one and sometimesit's useful tovisualize things this waythen we're going to be using partialderivatives so what are theywell it's just the derivatives in theaxis directions here so we'll be dealingwith you know d e d xthe e d y and the e d tand just as we've been discussingwe approximate them by takingdifferences first differences ofneighboring pixelsin either the x the y or the t directionfrom frame to frame so you know thatthat'sall that isokay so what's you know some people findpartial derivatives a little bit morescary thanuh ordinary derivatives but that's allit is we're just takingthe derivatives in those threedirectionswhereas we could be doing something morecomplicatedso let's think about that so supposethatum you know this video is for me movingthrough an environmentand there's some object in theenvironmentthat um i'm tracking and at time t 0 youknow it's there in the image and then inthe next frame it's over here and so onso it follows some sort of pathin this three dimensional volumeandyou know i could do this for otherpointsand develop this wholeset of curves but let me let's justfocus on one now one of the things imight want to do isumsee how this changes so when i'm lookingat uh different frames of an imageoften what i want to do is not take thederivative at a pixel and see how itchanges to the next time frame but iwant to follow an object and see how howit changes and in many casesi'm going to make the constantbrightness assumption that it's notchanging in brightnessand so i would like to express someconstraint on the derivative along thiscurveand so i could have a curvehow do i define that well one way is togive x and y as a function of tso this green curvefor each time t i can give an x and a ythat's how i define that curveandthen what i'd like to do is look atfor exampleumthe total derivativeuh along that curveright keeping in mind that along thatcurve not only is t changing but x and yare changing in some defined way so sothis is the total derivativeandin what we're going to be doing we'llset that to zero because we're assumingthat this point maintains its brightnessand then i can use uh the chain rule tosplit this up so i'm going to getdx dt times the edxplusso i can take this total derivative andexpress it in terms of partialderivativesand again that looks kind ofintimidating but i can easily rewritethisso that's equivalent to what we had for1d motionright so this is showing um arelationship between the brightnessgradient and and the image motionandif i know the image motion and i knowthe image i can predict how it's goingto changeand you know very simple i just use thisformula and what we're more interestedin doing is we've got images imagesequences and we want to find u and bokay so that's where we're going nowover here we could just solve for ubecause we had a single equation in asingle unknownrightand and that's another thing we're goingto do a lot is equation countingyou know how many degrees of freedom howmany numbers are there that we don'tknowand how many constraints do we haveover hereyou knowwe have one constraint onethat equation we have one unknown uperfect match souh we know that uh we are likely to geta finite number of solutionsand because it's linear we get onesolutionbut when you look at this you'll seethere's one equation one constraintbut we've got two unknowns right we'vegot uhu and vso if we're trying to recover theoptical flow which is that vector fieldthat we're just discussingum it's like we don't have enough uhinformation here to do thatwhich is dramatically different from the1d case where you know we have a matchbetween the number of equations and thenumber of unknownsokay so uh what what do we know so arewe lost at c is is it hopeless and solet's look at just what thatconstraint provides usso again the overall objective here iswe have a time varying image or in thediscrete casea sequence of image framesand we're trying to recover the motionandwe find thatassuming that things don't change inbrightness the images don't change asthey movewe end up with that constraint equationand what is it towards well one way tothink about it isto plot it in velocity spaceso you know we're used to plottingimages with x and y as coordinateaxes but actually um for some purposesuh it's useful to have a different kindof representation and this is one andwhat it means is thatany point in here is a particularvelocitythat velocityand you know for example this is zerovelocity it's not going anywherevelocity spaceumwas used at one time in physics muchmore than it is now and it's actually uhkind of neatfor example you know if you look atplanetary orbits they're ellipses withthe sun at one focusand ellipses are sort of complicatedif you plot it in velocity space they'recircles which was exploitedin the early days before people couldjust you know feed equations into acomputer and have them solvewhere they have to reason about thingsgeometrically anyway velocity space issort of neat and here we have aconstraint in velocity space so this issaying thatokay before we open our eyes and lookedat the image we didn't know anything thevelocity could be anywhere in this planebut now we have one constraint on it sowhat does that tell us that mustsomehowlimitthe solution spaceandyes it's a linear equation in u and vand so linear equation in 2d worldcorresponds to whatline yes y equals mx plus c for exampleso it's going to be a lineand that's great because it means thatokay we haven't solved the problem butwe've come a long way so beforeit could be anywhere now it's going tobe on the line and what we really likeis to pin it down to a point so we'vecome part way and so what line is itwell we can rewrite that equationlike that andrewrite it a bit moreby normalizing this vector so i'mturning this vector into a unit vectorso first of all again areminder that this is the brightnessgradient and the brightness gradient isvery important for all sorts of thingsif it's zero nothing much is happeningbecause the brightness is constantand more likelyit has a high value at a transitionbetween now if i'm looking at thetransition between the wall and theblackboard there's a big change inbrightness and therefore the derivativeswill be large and therefore thebrightness gradient will be largeand not only that but the brightnessgradient as a vector will be pointingperpendicular to the transitionright because e x is very large here e yis zero soso so this brightness gradient is veryimportantand this isa unit vector as you can easily tell bytaking the sum of squares of the twocomponentsand why am i doing this well because i'minterested in the component of u and vin this direction the directionspecified by the unit vector and it'sthis constantand so what this is saying is thatyou know u and v are perpendicular tosome lineandand this is what it looks likeyou can check that i mean basicallyyou need to find this pointand then calculate the distance from theorigin and it's going to going to bethat sonot too interested right now in thedetails of that other thanthat we've shown that it's a line andthat the line depends uh on thebrightness gradientsoit tells us a lot of things already oneof them is thatwhen you make this localized measurementand you are under constraint you don'thave enoughequationsyou do know somethingyou know for example in this diagram idon't know what u and b isbut suppose that igive you a different coordinate systemokaylocally you're going to see linearpatterns you won't see that if if i keepon magnifying thisyou know it's going to be more and morelike alinear relationshipand so the argument is that if you lookthrough an aperture you cannot determinethe motion you can you can determine themotion in the direction of thebrightness gradient so that'sthat's what wewhat we did over hereokay[Music]soin the 1d case that we were done herein the 2d case we're not done because wehave a mismatchof constraints and unknownsand so um we need more constraints so weneed to know something elseand sometimes you know there's priorknowledge of the environment that helpsyousometimes you can look again at anothertime and get extra informationum now let's suppose that we have theoptical mouse problemwhere the whole image is moving as oneyou know you're over a flat surface at aconstant distance from the lensandto a very high degree of approximationthe image is just moving as one so wedon't just have one pixel we we can dothe same thing at two pixelsokay that's a pixel oneand that's at pixel 2.and our job is to recover u and vwellit's a two linear equationsuh we can write them that wayand then solve now because we've got uhtwo constraints nowuhwe have enough constraint to solve foruh the two unknowns and you know it'sjust a linear equation so it's verysimpleand ifwe can actually write it write theanswer out explicitlyuhso it's very mechanical i mean we justinvert this two by two matrixand you know multiply the result bythat vector and we're doneandwhat is thiswell this is the determinant of thatmatrixso you know for two by two and three bythreewe can explicitly write it downotherwise we'll just use gaussianelimination to solve the set of linearequationsokay so umin a way we're done we we need twopixels we can solve this and now youknow it's noisy so we can improve theresult by taking more than two pixelsbut before we do thatit's always important tocheck the edge conditionslike you know this can failand it fails when the determinant iszero right so we um when does thathappenwell uhcan that happen well uh sure let'srewrite it one more timeuh twookay so we have this nice linear methodthat if we make these measurements attwo pixels we can solve for the motionbutuh it won't workif this is the case so what is thatwell this tells you something about thebrightness gradient right this is thedirection of the brightness gradientwell the tangent of it right sowhat this is saying is thatyou have a problemif they'rerelatedso this is where thedon't have to be the same i meanobviously ifthe brightness gradients are the sameand you subtractthose two quantities you get zero andthings blow upbutthey don't have to be the same they justneed to be proportional to each otherrightwe're just saying that the ratio of e yto e x is the sameand that makes sense again becausethat means thatthe brightness gradient is the same inthe two places that means that theconstraint we get out of the equation isin the same direction it's not providingnew informationorgo back to this diagramwe now have two parallel lineswell if there's no measurement noisethey'll be on top of each otherand then you intersect them and what youget well the same line so that's notvery interesting uh or if you have noiseit'll be even worse because they won'teven intersect soso um this tells you a number of thingsone of them is okaythis isn't going to work if thebrightness gradient is the sameeverywhere and that makes perfect sensethat's the aperture problemuh it also tells you thatmaybe you need tosomehow weight contributions fromdifferent image regionsbased on this maybe you don't want toyou knowhave contributions from a lot of imageregions with the same brightnessgradient because they're not reallyproviding good constraint they're allsort of doing the same thingso that tells you that rather than takethis result for two pixels and thenapply it to other pairs of pixels andjust take the average you'll want toweight it in some way sookayum and you know that's kind of kuji imean that'ssort ofuhtrying to fix a problemin a simple waybut we can dowe can do this much bettersowe have this equationthat's supposedly true everywhere in theimageright this is our magic equation whichrelates image motion to brightnessgradientsand um that's supposed to be zero now ifthere's a this measurement noise itwon't be zero it hopefully will be smallif you plug in the wrong values of u andb it won't be zeroso maybe one way of solving this problemis tosearchfor u and vthatmakes that zero or if you can't make itzero make it as small as possibleand so that motivates this approachokay so what's going on herewell the integrand is just thisexpression and if everything was perfectand you had the correct values of u andb the integrand would be zero and youintegrate it over the whole image youget zeroand that's the smallest you can getbecause it's quadratic it can't get lessthan zero so that's itif you plug in the wrong values of u andvwellit will not be zero and soyou can base a strategy for finding uand v on on thatumnow uh there are cases where this isgoing to failyou know one is if uh i don't know youuh without a light in acoal mine uh if e is zero you know allhope is lost obviouslyif e x and e y are zero uh all hope islost becausethat means you're looking at a wallthat's you know constant brightness andyou won't be able to see it movingbecause there's nothing on it to catchyour attention and to trackif there was some texture on it then e xand e y would not be zero in some placessookay and soumthis is sort of approach we're going touse a lot so we have this constraintwhich would applyin ideal casewhere we know the answer u and vandwe're going totake contributions of that and now wecan't just add them because some of themmight be positive and some might benegative so it doesn't make sense tominimize the the integral of themwe turn them into something that'salways positive by squaring it and sowe're going to find these answers byminimizing that now in this caseyou know this is pretty straightforwardwe now take thatand this is a calculus problemright so so this is a function of u andvand uh we're going toset the derivatives with respect to uand v equal to zero and in this casewe're lucky enough that the equationsare linear and we can solve it andactually the answer is very similar toour uhvery similar to thiswe have we're going to end up with twoequations and two unknowns right uh oneequation from uh the derivative withrespect to u is zero and one with thederivative of v is zero sookayum so andthe so two equations two unknowns uhcool but that can fail right becausethey might not actually be differentequations they might be linearlydependent so we'd want to worry aboutthatand in this case the failure mode isthat turns out to be the determinant ofthat two by two matrix so thatcorresponds to to this thing over hereand in in this case uh that's uh whenthings fail so so when can that happenumwell uh certainly if e is zeroor e is a constantright because then e x and e y are zeroum ormaybe if just e x is zero everywhereright because then this is this integralis zero and that integral is zeroso let's see so if e x is zero and e yis not zerouh what is that well that means that ihave one of those pictures that onlyvaries in one directionright e x is zero meaning it's constantin the horizontal directione y is non-zero it's varying inbrightness in the vertical direction andif i were to draw isofoads they'd looklike thisrightright so heree x is zero e y is not zeroand we know that the problem is aproblem if i move in this directionthe image doesn't change soconsoleokay so e x equals 0 is a problemsimilarly e y equals 0 is a problemjust turned 90 degreesso those are kind of obvious i mean weknow those aren't going to workis there anything else wellhow about e x equals e ywell if that's the case then these twointegrals are the sameand actually that integral is the samealsoso this you knowwe're going to getintegral e x squared all squared minusthis thing all squared zero soso that's bad so what is that wellthat's where our isophotes uh arerunning at 45 degreesright because the brightness gradienthas the same x component as y componentthe brightness gradient is perpendicularto the isophoteand soe x e y means it's a 45 degree angle forthe gradient so the isophoto 45 degreesdown so so that's the picture for thatand you know of coursethatis really the same thing as that justrotate it and one of the things we'regoing to do issay that wellsomething shouldn't depend on our choiceof x and y in the image coordinatesystemi mean the answer would be different butin the new coordinate system it shouldmean the same thing so in this casei shouldn't get some result that'sparticular to the x or y axis it shouldjustgive me the same result if i rotate sothat's a 45 degree rotationi can get any rotationby doing something like thatright because then e y over e x is someconstant and that's the tangent of theangle of rotation or something like thatplug that in here and of course you'llsee thatyou get you know k squared times theintegral over here and the k squaredappears over there they cancel eachother out so these are so this isactually the most general case all ofthese others are just special cases ofthat oneso thatthatsummarizes all of the cases we foundthey're allyou know with isophotes at some angleparallel linesum and then you know important questionisis that itand umit turns out yes uh that's itand it's not easy to prove youyou need toat least use the triangle inequality orsomething profound like that soit's actually not that hard but i ithink it'snot uh really useful togo there anyway so umwhen can we do it well as soon asthere's curvature in the isofoads rightbecause if the isofoat looks like thisif i move itsort of tangent to the isophotethere are still changesso the only problem is where theisophotes are all straight linesparallel straight linesandin another way of thinking about it ifyou have areas in the image wherethere's a quote corneryou know something where the isophotemakes a sharp turnso here the brightness gradientsif there's some small area where thedirections of the brightness gradientschange a lot that's good right becausewhat we're worried about is that thebrightness gradients are all parallel toeach other that's the bad situationand sofor many purposes like image alignmentand recognitionwe look for places that are quoteinteresting and one interesting way isto look at the isophobic curvature orequivalently the rapid turning of thebrightness gradientsbecause in other areas of the imagethere's less constraint you know if i ifi have a part of the image that lookslike thison its own it doesn't provide enoughconstraintnow if next to it is another area of theimagewhere things you know look like thisif i have both of these then that's finebutif everything has the same gradientdirection that's not not satisfactoryokayokay sohomework problemtheoretically due and uh on thursdayand umyou knowi know we haven't done a whole lot onthis so if you have problems with itsend me emailand i'm i may say something that doesn'tgive away the answer but helps youand we'll go from therewhat we're going to do next istalk a little bit about this concept ofnoise gainso the idea is thatall this time we in the back we've beenthinking about oh we only have pixelswith eight bits and they're noisy andour calculations are going to be flakyunless this determinant is large and soon so there's a lot of attention to notjust you know a formula for computingthe answer but making sure that theanswer is actually meaningful as opposedto you know you've taken two noisynumbers and divided them by each otherand the result is notverypredictable so in noise gainthat's a very precise way of talkingabout it which is saying if i make thismuch of a change in the imagewhat is the change in the result so inthis casei'm trying to get the velocity of motionhow much is the velocity of motion goingto change if i change the brightnessgradient somewhere i have the wrongmeasurement it's slightly off and ofcourse we want that to be not sensitivewe don't want it to be very sensitive tonoise and so we'llmake that clearinin terms of a one-dimensionaltransformation where we know the forwardfunction and we're trying to invert itand we want to know when that inversionis sensible and when it's notokayoh questions

Time to Contact, Focus of Expansion, Direct Motion Vision Methods, Noise Gain

we'll start off today by talking alittle bit aboutnoise gain in other words therelationship between errors inmeasurement and errors in estimation ofquantities you're interested in aboutthe environmentand i just happen to have this exampleon my computer it's notvisionrelatedbutillustrates some of the pointssowhat is this[Music]this is a indoorequivalent of gpswhere instead ofusing the timing of signals fromsatellites we use the timing of signalsfromwi-fi access pointsand it'sbeen in the works for several years butit doesn't really exist yetfor example at this point you need theandroid nine and the only phone thatruns android 9 is pixeland most of the wi-fi access point don'twell all of the access points don'tsupport it i've been driving aroundlooking for them i've registeredthousands and i found one anyway theidea is thatin the future we'll be able to measuredistances to access points and of courseyou can imagine that the accuracy issomewhat limited becauseelectromagnetic radiation travels ata nanosecond per foot so you have toreally measure theseround trip times fromphone to access point and back with withvery high precision but suppose you doso you have a bunch of access points imean we've got you knowfour right here in this roomandthenyour job is to determine the accuracy ofyour locationandjust as in gpsyou may be able to measure thedistance to the satellite to let's say10 metersthat doesn't mean that the accuracy youcan expect of your xyz coordinateslatitude longitude and altitudeis 10 meters in fact is typically a lotworseand in gps terminology that's calleddilution of precisionand in gps it's different for horizontalthan for vertical that's anotherinteresting point that uh you candetermine your horizontal position withhigher accuracy than your verticalposition so like this morning my phone'sgps said i was at minus 36 meters and iknow boston is sort of at sea level andthe water level's rising but that'sprobably not accuratesoyou know point of that is thatwhen we do this noise gainanalysis of some machine vision processis probably going to be different indifferent directions so it's not just asingle number that says your accuracy isone meter so back to this one here sothe green points are four wi-fi accesspoints nicely symmetrically placedthe red points are possible places whereyour cell phone can beand the circles or ellipses are areas ofourcontours of constant error so in otherwords if you move out to thatcurveyou will be[Music]in error with respect to the distancesyou can measureand so turning that around it tells youuh the accuracy you can expectand it looks very nice in the middlethere they're they're small circlesmeaning a that you will be able todetermine your location prettyaccurately and b that there isn't sortof a difference in different directionsit's not like you can determine x betterthan ybut then when you go outoutside the convex hull of therespondersyou'll see that things becomeelongated we have these ellipsesand they get larger down here what doesthat mean well it'sthat you won't be able to measure youraccuracy very well in this directionbut you can still determine it prettywell in that direction and why is thatwell if you think about the distances tothe responders and how it changes as youmove around if you move in thisdirectionyou're moving at right angles prettymuch to those vectors and so you're notchanging the length of those vectors bymuch and so correspondingly you won'tsee a big change in the signal and socorrespondingly you won't be able toaccurately determine the positionwhereas if you move radiallyyou know if i move in this direction idirectly have an impact on the distanceto that one and the one up there and youknow with this one i'm at an angle butit's like 45 degrees so that's squareroot of 2one over square root of two so that'sseventy percent yeahright it's the ladder so the idea isthat the the red is a possible positionfor my sensing equipmentand the circle around the red is how fari have to go before the error is acertain threshold valuesoin other wordsif i'm hereand i measure these distances andthere's no measurement errorthen the sum of squares of errors willbe zero if i move a little bit away mydistance from the responders is wrongand i canand in this case i have four of thosenumbersand i look at how big they are and i addup the sum of squares as a overallmeasure of errorso when i'mdrawing this circle that's the locus ofall the points that have the samesize errorandout here you can see that i have to gofurther awaybefore i see that error and converselyi'm less sensitive tothe error and i will be able to notdetermine the position as accuratelyso that one's kind of uh pretty uh cleanbutlet'suh suppose thatsuppose that we have three transpondersnow you can see it gets a bit messierandslightly surprising results one is thatout here these things that look likeellipses becomenon-elliptical becausetheir ellipses in the limit of verysmall displacements butfor larger displacements the curve canhave any shapebut in most cases we'll be focusing onthe you know like the infinitesimal casemake life simpleso what's interesting about this wellone of the things is that you know it'spretty accurate down here which is awayfromthe respondus and it's away from theircentroid you'd sort of think well maybeit should be pretty good at theircentroid or at theintersection of the bisectors of theangle or the intersection of theperpendiculars dropped on the other sideof the triangle you know something likethat but that's not the case and so thisis sort of interesting because it meansthat you may be able to do thingsaway from where the responders are sothe responders could be like down ahallway and it'll still give you goodaccuracy away from thatat the same time we can show that if youput them in a line that's a really badidea and of course that's what they arehere everywhere they're just anywayagainthe idea here is that if i'm over hereand i move a little bit to the side i'mnot changing the distance to those threeresponders much because i'm movingpretty much at right angles to thosethree vectors and therefore i won'treally notice that i've moved andtherefore conversely i can't determinemy accuracy in that direction as wellso that's with three and let's go onemoreuh so that's with two two uh respondersand you can see that there's some areaswhere it's working just fine over herebut in between them not so good why wellbecause if i move horizontally i'm notreally changing the distance very muchit'll be second it'll be quadratic it'llbemoving changing as x squared and so forsmall x it's a very small change andeven worse out hereandthen you sort of start to seesomething maybe you can see a locus ofplaces where it's working pretty wellandtheir property is that from each ofthose positions where it's working wellthe two responders seem to be at rightangles to each otherand that sort of makes sense becausebasically you're sayingokay i have a constraintin a direction and a direction at rightangles if i rotate the coordinate systemi can make that x and y i have aconstraint in x and i have a constraintin y that's like the ideal condition ifthetwo directions are very similarthen i can moveat right angles to the average and notchange things very muchsoit turns out thatpoints where the respondersappear 90 degrees apart particularlygood and what is the locus of all suchpointsgeometry theorems aboutcirclessothe points on theperiphery of a circlehavethe diameterthe endpoints of the diameterright angle apartthat's not the way to say the theorembut i think you know what i mean so if idraw a circlewith theline connecting the two responders asdiameter points on that circle will havethat property and so not surprisinglythose are the points where i'm gettinggood accuracy even in thisprimitive in this simple caseokayenough of thatlet's go back toso the idea really is very simple it'sthatwe have some sort of forwardtransformation that takes us from aquantity in the environment we want tomeasure i don't know distance velocitywhatever to something that we canobserve in our instrument the camera sayand so the forward problemis that we let's just take the simplecase of a scalarthere's an xand we don't observe x directly we canmeasure f of xand of course uhenvision our problem is to go the otherwayi've measured if f of x you know what itwhat is xand you know if f isinvertible a nice smooth monotonicfunction we can do that but then thesecond question is okay my measurementsaren't perfectso i don't really know f of x perfectlywell what does that mean about x andokay that means x won't be accuratewhich isn't catastrophic butyou know how much bigger is the errorand of course the answer is very simpleso here's our function andlet's say we have a certain xthen the forward direction is we takethe x we go up to the graphwe read off a certain valuey andwhat we're doing is we're getting y fromour camera or whatever and we go in theopposite directionyou know sowe invert that functionas you know of course there's beproblems if it's multi-valued then therewon't be a unique answer but for themoment let's assume that we can uhinvert it okay so then the next questionis what if there's some errorwell in the forward direction it'spretty straightforwardsuppose thati take a second value herethen there'll be some error up heredelta yand what's what's the relationshipbetween delta x and delta ycalculus heard of derivativesokay you know what the answer is it'syou know delta y over delta x isbasically uh the derivative of f of x orat least in the limit as we make thosevery small right so so therelationship between the noise inthe quantity we're interested in and thenoise in the measurement is just thederivative of that transfer function andso converselywe're trying to go the other way aroundright we'll just turn this on its headand now we see thatthe noise in our resultis related to the noise in themeasurement by this one over f primeso clearly uh f prime equals zerois badand that's not too surprising you knowif we if this continues and becomes flatthat means that the thing we'remeasuring is not responding to thequantity we're interested in so nosurprise we can't recover it but also iff prime of x is smallthat's not so good because you know ifwe're up hereand we make some change in xthat's going to be an almostimperceptibly small change in yand converselythat if there's sort of any noise in themeasurementi don't really know where i am rightbecause i you know suppose the noise andmeasurement is thatwell that means they have an uncertaintyof that sizein the quantity i'm trying to estimateand so and that's it it's just this isthis is the noise gainumand you know a lot of situations wedon't worry about it too much we have toworry about it in machine vision as imentioned because of the noise in themeasurementsokay so that's a real simple case wejust got a scalar quantitylet's extend this a little bitum we're we're recovering a vector solet's supposeso this is the forward directionright there's something we're trying tomeasure like the location of a robot uhmanipulator armendpoint in space xyzand uh we're using acamera or a couple of cameras uh toimage it and so we're measuring umsomething in the image that we'll call xwhich is a transformation of thequantity we wantuh now in that case itwon't often be that linear but let'ssuppose we'd have a simple case wherethere's a linear transformation wellthen of courseyou know we just do thatif we can invert that matrix that's ouranswerwe've estimated where theend point of the robot manipulator isbut of course we're also interested inyou know how good is that answerand so we'd want to knowif i change x a little bit how much doesb change and so you know crude way oftalking about gain would bejust to take themagnitude of the changein the result and divided by themagnitude of the changein themeasurement so but that's not umit doesn't take into account uhanisotropy like we saw in those diagramsthe error may be very low if you go in acertain direction but it might be quitelarge in another direction so the answeris a little bit more nuanced weshould be a little bit more carefulso um let's say a little bit more aboutthisyou know how do we solve linearequations we use gaussian eliminationand um if you do that you are you comeup with some sort of formula whichincludes uh the inverse of thedeterminantand so the conclusion is thatdeterminant equals zero is really badbecause then you can't do thisand actually the magnitude of thedeterminant being smallis also not that goodwhy well because you're going to takeone over some small number gives you alarge number and then multiply yourexperimental measurements by that largenumber and so any tiny little deviationin that is going to be magnified bythe inverse of the determinantandlet's uhjust go back to an example that we diddowhich was 2d so our matrix in this caseis a two by twoso as a startjust to refresh your memory and ii did this last time but i think i got alot of blank looks and so i'll do itagainso what's the inverse of this two by twomatrixit's it's that and how do i know that orhow can i verify that well you just needtomultiplyand what you geta d minus b cand thenabminus a b plus a band here we get c dminus c dand then finally we getminus b c uh plus a dright so that so these of course arezero anduhthese arethe same as thatand soso for two by two matrices we think itcan explicitly get an inverse and ofcourse uh we've got a lot of interest inthat quantitybecause that quantity is small then ourcalculation is subject to amplificationofof errorand this came up becausewe were looking at umcomputing image motion so we hadat two pixelswe had a constraint equationwhich looked like thatso we had two of theseand now we're solving for the motion uhu and vand so of course we just getjust using the formula up hereand so on so we justplug that in so there's an example whereumwe can apply this idea of noise gaindirectly and we know thatif this quantity is small we're going tobe amplifying noise a lotandwe already went through the argumentthat that means that the brightnessgradients at those two pixels aresimilarthey're oriented in the same or almostthe same directionand sothat means they're not providing verydifferent informationit's almost as if you'd only made onemeasurement so no wonder the answer isflakyand you know we can go into the velocityspace diagrameach of these constraints this is astraight line why well because it's alinear function of u and vequal to zeroso that's going to give us a straightline this is going to give us a straightlinewe're basically looking for a point inthe plane that's on both lines so we'relooking for the intersection of the twolines if the two lines are at rightangles that's a very well defined pointif the two lines are almost parallelthen you can imagine that any smallshiftit can move thepoint of intersection a lot so this isthe case that's not so goodandif i move this one a little bit say imoveit over herethere's a huge change in theintersection pointso that corresponds tothe case wherethe two gradients are almost paralleland that makes this quantity smallnotice thatnot all hope is lostit's true that the component in thisdirection we don't really know very wellbut we got very good constraint in thatdirection so that's another lesson whichisyou know we may have a situation wherethe noise gain is high but it may not beequally high in all directions as we sawin the diagrams i showed you and it'sgood to know you know which componentcan you trust if i'm going to have myrobot move over and pick up a partthat's a useful thing to knowokay[Music]so let's uh review a little bit andi want to go on to umsomething called time to contactyou know we're kind ofcutting across the material in adiagonal way i mean iit would be nice to sort of present allof this stuff first and all of thatstuff second but you'd probably fallasleep i want to motivate you by showingyou that if you put these piecestogether you can get some prettypowerful results so so let's go therelet's see if we can first of all reviewuhwhat we've done so farso we hadthis ideathe constantbrightness assumptionand we had that that image solidwhich was afunction of x y and t souhand we followed some sort of curvethrough here which is perhaps the imageof some particular thing in theenvironmentthat moves or the camera moves anduh but we're assuming that as it movesits physical properties don't change soit's stillgoing to be imaged with the samebrightness right so along this curveuhthis holds true the total derivative andfrom that we got uh our constraintequationuh you're gonna get probably prettytired of seeing that oneand so this isthe constant brightness assumption andfrom it by just chain rule we getthis constraintwhich is also called the brightnesschange constraint equationand we use this in many different ways imean it's fundamental to what'shappening in an imagewhen there's motionand then we looked inparticular at the optical mouse problemwhich is uh a simplified version ofthings we'll be looking at later it'ssimplified in the sense that we'reassuming that the optical mouse isworking on a flat surface and that thewhole image is moving as one in otherwords u and v is the same for the wholeimage then that's obviouslya very nice and simple caseand we said that one way of dealing withthatis toturn it into a least squares problemso we're going toadd up over the whole imagein x and y direction we're going tointegrate or in the discrete case takesums of a rows and columnsof this quantity which supposedly iszero so we should get zerowhy might it not be zero well if youplug in the wrong values for thevelocity u and v you won't get zero andso one way of trying to find the rightvelocity is to find the minimum of thatintegralnow in practice your measurements e x ey and e twill be corrupted by noise and soyou'll never actuallyget this integral to be zero so theanswer isn't we're going to find theplace where it's zero the answer iswe're going to make it as small aspossible and that's our estimate of thecorrect value for you and bandyou know we could justify that by somehairy probabilistic statistical argumentbut i think it uh probablyis notbeneficial to go there we'll just itseems like an intuitivelyright thing to dookay so that's what we're going tominimizeandwe only have two parameters so clearlythis is a calculus problemwe just take thederivativeand set the result equal to zeroum so we're assuming that this isvaryinguh smoothly with u and v as it obviouslyis uh yeah well okay so let's put itthis way so suppose that i have measurede x e y e t and now uh you tell meu is one and v is twoand i can plug it inand i get some large number i'm likelyto say i don't think you're right andthenhe says well u is 1 and v is 3. and iplug it into this equation i get asmaller numberwhich one would you trustso it's you knowuh ideally this should be uh very smallor even zeroum andsorrywhy do you want a smaller number oh wellbecause uhif there was no measurement error ifthere was no error either in themeasurement or in your knowledge of uand b then it would be zero at everypixel and so then um integrating overthe whole imageokayall rightandi'm not going to do that because that'stoo close to what you did in thehomework problem so we'll we'll stopthere sookaywhat are we going to douh wellthe a very simple caseis where the velocity is the sameeverywhere optical flow case and we didthatthere are other simple cases where u andv are not constant but they vary in avery predictable way in in the image andand we saw that uh last timewhen we were talking about uh focus ofexpansionand so let's again review and go back toall the way toperspective projectionright sowe had a relationship betweenworldcoordinates and image coordinates so thecapital letters arecoordinates in the world and the smallones are corresponding in the imageso that was perspective projectionagain in a camera-centric coordinatesystemand then we said well now suppose uhthings are movingbig x big y b z are changinghow a little x and little y changing andwell we just differentiate with respectto timeand we get 1 over fdx dtis1 over z d big x dtand then unfortunatelywe have this one as wellall written written in a slightlynicer formum[Music]right because umdx dt is the velocity in the x directionand that's what we've been uh using thesymbol u forandsimilarly in 3d we'll use the big u tostand for the velocity in the xdirection in theworldand similarlyuh so so we had thatand then we looked at the situationwhereu is zero and v is zero and we call thatthe[Music]focus of expansion for that particularmotionand soif we plug in little u is zerowe get a relationship herethat isumoh where's my x zero so let's do it sowe got zero is uhone over z u minus and now thethe big x over big z we know is x overfand so umwe and this is x zero this is where thevelocity is zeroand similarly for the y directionokay and so we get x naught isf of z[Music]u over wsowe have arelationship between the velocities thedistances andwhere the focus of expansion isand then we talked about that quantitythat occurs therewhich is w of a zand you know what is that well uh easiermaybe if we turn it on its head as wedid last timeand of course big w is the uhcomponent of motion in the z directionso it's dz over here so this is adistanceof a speedright and so you know welast time said the unitswould be meter of a meter per second orsecond so this is actually the time tocontacthow long it's going to take before wecrashinto that object if if nothing changesand that's obviously a useful quantityto try and findand sowe'll try and find it from a sequence ofimage measurementsnow actually we're going to call thiswe're going to give this the name c andc is actuallyone over the time to contactand the reason for that is mostly tomake the algebra simpler after we find cof course we can just find its inverseandbut it has the advantage that if themotion is very uh slightthen the time to contact can behugewhereas c just becomes close to zeroyeah sorryno i hesitated because it didn't lookright thank you okayokay so umlet's start real simple let's supposethat there's only motion in the zdirection so i'm moving towards the walland of course the image is expanding asi as i approach the wallandwhat is the focus of expansionwell if i'm moving straight to the wallthe focus of expansion will be rightdown the barrel it'll be atzero zero right so let's take a specialcaseand then according to the formulas forx naught and y naughtand if i drawthe diagram for themotion fieldit's going to look like thatand umso a couple of things one iswell if i can measure those vectors thenyou know i'm donewhy do i need to do all this other stuffwell the thing is we don't know thosevectors all we have are imagesimages are brightness patterns imagesare brightness as a function of x and ythere are no vectors in thereso you know one approach might be wellfind the vectors and then we canintersect them to find where that pointisbutso for us this vector field is is auseful tool to visualize what's going onbut the actualexperimentaldata is there's an image and then wetake another image and the other imageis either expanded or shrunk and our jobis to to solve this problemtheni'vehad the arrows pointing inwardswhich means that that's actually a focusof compression rather than the focus ofexpansionbut of course they're just you knowdepending on the direction of motionif if i'm standing hereand and the wall is receding with apositive velocitythen it's going to compress in the imageand i get thison the other hand and there's no dangerthe time to contact is negativei left the surface a while agobutthe case we're going to be moreinterested in is where the velocity ofthe wall relative to me is negative andthen this diagram is reversed and thereis a focus of expansionokay[Music]so what can we do with thisso in this caseyou know we go back to the same oldequationand now in this particular caseu and v[Music]we've madewe've made the capital u and the capitalv zero so all that's left is that secondterm and so um uh u and v take uh takethis formokay so uh stick that in there we getlet's see u[Music]we get c xe x plus ye y plus e t is zeroright so i've just combined these twotermsand so if i wanted tojust asyou know what we did beforewe can measure one-dimensional motionfrom a pixelsimilarly here we can measurethe inverse of the time to contact justfrom brightness derivativesat one point in in the image of courseit's not likely to be very good becauseall these quantities are subject tonoise and in fact of courseestimating derivatives makes thingsworse right so we've got two two numbersthat are subject to noise we subtractthem if they're similar in magnitudeabout all you've got left is the noiseso um these are not this is not going tobe a very accurate method before we goon to let's look atthis term over hereand i'm going to call this the radialgradientand i put it in quotation marks becauseit's not the greatest notation butwe we use it so much we need somenotationand we can think of it as that dotproduct right it'swe've got two vectorsand what are those wellthis vector is the brightness gradientand we we keep on seeing the brightnessgradient and it's just pointing in thedirection of the most rapid change ofbrightness in the imageyou know one way to think about imagesyou know e of x and yis uhas a topographic map so if you think oftranslating brightness into heightthen you can visualize thethree-dimensional thing and theimage is some surface in thatthree-dimensional thing and thebrightness gradient is just the gradientof the surfaceit's a vector in the direction ofsteepest ascent you know if i want toget to this mountain as fast as possiblei go up the gradientand if i want to get down as fast aspossible or go in the opposite directionumso that's e x e y it's the gradient ofthe brightnessand then what's x and y well that's alike radial vector it's like in thepolar coordinate system i could imagineerecting a polar coordinate system inthe image with the origin at the centerand that's uh the that vector and sowhat this is doing is it's taking thedot product of those two so for exampleif they're at right anglesthen this isthis is going to be zeroandif they're parallel then they'll be aslarge as possiblenow we could uhlet's see which way do i want to golet's try thiswe could normalize thisand turnthis one into a unit vectorso first of all you recognize that bydividing by the square root of x squaredplus y squared i turn it into a unitvector because now if you take it dotproduct with itself you get one okaythat that clear so there's a unit vectorunit vectors are useful for indicatingdirectionsoum this doesn't have a well magnitude isone so magnitude doesn't tell meanything useful but it's a way oftalking about different directions andthen taking the dot product with anothervector does what well it gives you thecomponent of that vector in thatdirectionright sohere's a vector and i'm interested inhow much of it is going in thisdirectioni take the dot product and i get thatcomponentso what this is really computing ishow much of the gradientis in this radial directionand that's why we use that term radialgradientand the reason i am putting it inquotation mark is because it isn't quiteright because there's this factor thati've left outthat it's not just a radial gradient butit's multiplied by the radius but butit's you know basically it's measuringhow much of thebrightness variation is in the outwarddirection from the center of the imageokayandnow i'm ready to solve the problemuhbecause i have i have this equation hereand i'vesolved it for a single point except idon't really trust that so what i'mgoing to do is as before use leastsquaresso we're going to minimizeand this is over over the whole imageand again you know keep in mind thatwe're dealing with very simple caseswhere the motion in the image is notsome arbitrary sort of vector field butit's defined by a small number ofparameters yeah this here oh so thatcomes out of the equation up there sothere's an equation for[Music]is now assumed to be zerouh we're assuming that there's nomotion inx or y direction so those terms drop outand the other termwe've defined c to be w overz and then the big x over z islittle x of f because of perspectiveprojection equation sookay so yeah there's probably more thanone stepgoing from up there to down there butit's applying theperspective projection equationin the special case that um there's onlymotion along the optical axis there'sno motion in other directionsokay so again what's going on here isif we had the correct value of c andthere was no measurement noise thiswould be zero at every pixel and so ifwe add it up over all the pixels itshould still be zeroif we plug in the wrong value of cit ought to be non-zero grow with youknow our error in cand sofindingthe c that makes us as small as possibleis our way of estimating uh what it isand and yeah it can be justifieduh in terms of you knowif you assume the noise is gaussian anddo all that statistical probabilisticstuff but i don't want to do that ithink it's sort ofintuitive thatif this is supposed to be zero when ihave the correct informationthen making it as small as possible inthe presence of noise is a sensiblething to donow i uhjustyou know calculus there's only only oneunknown in there only one one knob i cantweakso i'm just going to see where thathas a zero derivativeand so well what do i get i get twotimesright there's a square in there so i'mgoing to get twice uh whatever is underthe squareand then i have to differentiate thatterm as well so multiplied by and nowthe derivative of this with respect to cis of course just that part so multiplybyand if it's equal to zero i don't reallycare about the 2 so let's get rid ofthatand now i can split this up into twointegralsand you know i'm going to leave out thedxdypretty soon because it'sit's implied before i have the doubleintegral over the imageokay and so i can solvefor c is the integral ofuh let's call this g squared and this isgoh sorry minus ge tg squaredwhere g is this radial gradientwhich will occur so often thatiget tired of writing it out soso um[Music]there's a way of estimating[Music]time to contact right because c is theinverse of the time to contactand it's kind of interesting becauseyou know there's no high level stuff inhere we're not detecting edges ortracking points ordoing anything sophisticated recognizingpoodles behind palm trees we're justdoing this brute forcenumber crunchingand it'sit's very effectiveyou know we gota million or 10 million points each ofwhich is lousy but we combine them thisway and they're good to one part in athousandso um what whatwhen you implement this what do youactually dowell you have to take uh the image andestimate the brightness gradient whichis trivial we just take neighboringpixels in x direction subtract themthat's our e x and then take neighboringpixels in the y direction subtract themthat's e yand we need e t so we we take two framesone after the other and we takecorresponding pixels and subtract themthat's our e toh there may be some scaling but that'sthe basic ideafrom those we then compute thisradial gradientwhich is easy to do x and y are theposition in the image relative tothe center of the image we'll talk aboutthat some more laterand thenwe just compute these two sums so sothese double integrals are just sumsover all of the pixels we we just runthrough all of the pixels adding upthese these things and so really uhwell if you want to do it sequentiallyyou need two accumulators you set themto zero then you run through the imagerow by row pixel by pixel and at eachpixel you compute gyou compute e x e y from it you computeg and then one accumulator you multiplyg by e t and add that to thataccumulator the other accumulator youtake g squared and you dump it into thatone and then when you've come all theway through the image you've got thesetwo sums you divide them and you're doneso it's you know totally brainlessthere's nointelligence there artificial orotherwiseand i call this aquote direct computation ofbecause there are lots of other ways ofapproaching this problemyou know for example you mightmeasure the size of like you coming outof a parking lot and there's an mitbus in front of youuh then you can measure in the image howmany pixels is the image of the busand uh how many pixels per framein time is it changing and so on you cando that but it means you have to detectthe bus you have to find the edges youhave to estimate the beginning and endof the bus and you better measure itwith very high precision like ahundredth of a pixeland it's not too hard to find out wherethe edge of the bus is in terms ofpixels but you know 100 and why do youhave to measure that accurately becauseit doesn't change much from one frame tothe next so um it there are other waysof doing this but this isit's brute forcemindlessand elegant that's my view of it okaynow of course this is very specializedwe're only dealing with a case wherewe're running straight into the wall andso we'lllook at more interesting cases in aminute but what i'm hoping to do if ican get the projector going againis to demonstrate some of thissowhat i'll show you isa little program calledmontevision whichallows you tocreateimage processingapplicationsby a graphical process of basicallyplugging togetherquote filtersand so let me bring this over here andshow that to you for a secondso this is what it looks like and i'veplugged togetherthe leftmost thing is a file sourcewhich in this case isa avi filevideothen it goes into a splitter that cutsit up into streamsincludinggoing fromcolor to gray levelthere's a decompressor because all videois compressed otherwise it would beridiculously largeand then it goes into the time tocontact boxand it comes outthe video rendererover there and this time to contact boxhasa set of parameters that show upsomewhere off screen right now okay soso i'll put this back where i can see iton my screenand let's hope this worksohokay so we'll i'll run this severaltimes because this goes by pretty fastso here we are running into a truckthat's uh used in the mining industryit's kind of a large truckand umwhat we're seeing is are a whole numberof different things let me also try andget rid of thiswelllet me just run it againso we'll you'll be seeing a circle redcircle that denotes the focus ofexpansion the estimate of the focus ofexpansionand you can see that we're going to hitthat tire or that wheeland then[Music]then you will see[Music]the three bars on the right so the thirdone is c the thing we just calculatedand you can see that it's uh red meaningbad and it's growing upwardsas we approach it's the inverse of thetime to contact and there are othersthere are things like the time tocontact is over there in framesumand we can plot to see how accurate itis compared to when we actually hit thattargetso so this is slightly different fromwhat we did because this allows uh somemotion in the x and the y direction andwe'll do that in a second and as aresult it has three quantities it'scomputing which we'll call a b and cand those are the three bars you see onthe right and if we were going straightdown the barrel the first two would bezero they wouldn't be either up or downbut they're not the x comp there's an xcomponent and a small y component thatand of course you know this is noisy soit's not uh not perfectokay let's look at the different onethenuh newmantripperum well i guess i'm obsessed withrunning into trucks one of my nightmaresso here's another truck we're runninginto the circle again indicates thefocus of expansionthe three bars are a b and cand the rightmost one is the one that wenow know how to calculateand i'll show this a couple more timesand it's growing as we go because thetime to contact is getting shorter soone over the time to contact is gettinglargerthe time to contact computed is down onthe right hand sidenow it turns out that if we were to dothis frame by frameyou would notice thatnear the end it just is wrongand so why is that well there's a wholebunch of different reasons one of themis thatthe imagebecomes out of focus and sothe information is uh changingwe're kind of assuming that the image isjustzooming but in a system that has a lensuh unless we adjust the lens to stay infocus we're going to go out of focusanother point is thatinitially the image motion is very smallit's the fraction of a pixel betweenframes but then of course as we getclose to the object you know things arereally exploding and so our wholeassumption about estimating derivativesby taking neighboring pixels andsubtracting them and all of that stuffis not working very well and so how doyou deal with that well one way is to umcombine pixelsso that on the super pixels the motionis a small number of pixelsand we know that's when this works sowhat we could do is run this at multiplescales so you have the full resolutionimageand when things are far away that's theone that's going to give us the bestinformation then we for example averagetwo by two blocks of pixelsnow we have an image which has a quarteras many pixels and it can cope withtwice the image motion right because acertain amount of image motion in theoriginal image now corresponds to halfas many pixels in this reduced oneand then we do that again we take thatreduced picture we againtwo by two average and now we're down touh a sixteenth so so this gets verycheap to computecompared to the original image and sothere's no realcost additional cost to thiswell someso what is 1 plus a quarter plus a 16plus a 64 plusokay well you know how to sum geometricseries and the answer is i don't knowfour thirds or something so yes it costsmore but it's not a huge cost you canafford to do this at multiple scales andthat allows you then to cope with umall of the velocities starting off witha very slowsub pixel motion to where things arereally uh blowing upokay well let me do one more thing andhook this up to the camera hopefullythat'll workohdon't do that to meokay so umhere's a a web cameraand umi guess it's relatively dark in here letme try the paperout of focusumtrying to find something that will giveit a nice texture to work with i guessthere's a chairwhich has lots of holes in itlet's try thatokay so uh i'll try and hold it as stillas possibleso again uh the three bars are a b and cand they'reall over the show but relatively smallnow if i move the camera up that thirdbar should go down and be green for safemeaning thetime to contact is negative if i move itdownthat third barwill go up and be read meaning i'm aboutto crash into somethingand i think the reason it's got thisjerky motion is becauseit doesn't like this projector so it'sit's got a very slow cycle time itshould be running at 28 frames persecond and it's obviously notokaysonow if itry to move in x let me see if i canmove in x that first bar should go upand i get oh i'm not holding it straightsoso i'm getting the crosstalk between xand yokay right now i'm moving in yand now move in the opposite direction yso that second bar goes up and thesecond bar goes down and here i can movein the x direction the first bar goes upand the first bar goes downand i mean you know to me this isfascinating becauseit's such a neat instrument and itdoesn't um have any magic in it uh andwe can understand its limitations we cancalculate what what the error is and soon it's not like something that's veryelaborate andhas some some hiddencode in itso let meshow you[Music]that version let's seeohokayoh i see it's closedumso one of the features of this code isit can show you some intermediateresultslet me i need to look at the controlpanel for this to see what it'swhat it's showing right now so right nowit's subsampling two by 2and let me show e x let's get rid ofthisokay so e x is just the derivative inthe x directionandandit'spositive is shown green negative inblack andand it's fairly weak in most parts ofthe image except where this black lineis they're on one side of the black linethere's a rapid change in brightness upand on the other side of the black linethere's a rapid change in brightnessdown so that gives you the the green andthe red fringes in the x direction nowif i turn it so that that black lineruns horizontally there'll be lessbecause the x derivative now is verysmalli see that um because it'smore or less constant in the x directionwhereas if i look at the y derivativeoh i can actually show x and y togetheruh in a slightly different presentationso hereumx is control the x derivative iscontrolling red and the y derivative iscontrolling umgreenand you get this funny kind of almostthree-dimensional feeling about theresultumbut you know it's the gradient isshowing the direction of most rapidvariationso think again of what we said about theanalogy with a surface a topographicmapand that's why this looks a little bitto us like it's three-dimensionalokay so that's you know those are thebasic things we compute i can also showinstead e tnow uh e t if i was uhsolid as a rock would be zero everywhereand obviously i'm not let's see if i canhold on to something to make it moreconstantso you can see it's sort of decreasingin magnitude although i can't see okayso it'sso that's a time direction we'resubtracting successive imagesandlet's seelet's look atgso g is that radial derivativexandwhat can you say well it goes outwardsyou see that[Music]that squareboxum it's y it's green all the way aroundwhich is not what e x would do because ex would be opposite on the two sides butbecause we're multiplying by the vectorfrom the center of the image it actuallyends up beingyou know green all the way aroundnow i can do one more thing which is tomultiply it by the time derivative e tso in that formula you may remember thatthere's a product of e t and thisbrightness gradient and umobviously if that is zero it will saythere's no motionuh if if there is motionthen it'll be non-zero and in particularit's unfortunate that it's notoh okay so here you can see that in mostplacesi wish it was running faster it it'svery hard toin most placesit's now greenbecause i'm moving away from the surfacewhen i do this in most places it'll bered the reason it the other areas isbecause it's jiggling back and forth i'mnota very good manipulator if i had arobotic manipulator we could do thisbetterokay so anyway um[Music]is there anything else we want to see onthis so before i put that away umare there any questions here we've got avery uh limited very constrained problemwhich is the problem ofyou know the fly landing on the surfaceor someone running into a wall if thereare multiple motionsthis is going to work and we're going todo thatsouh you knowwe're sort of slowly expanding westarted off with uh one one unknownwhich was just uh the inverse of thetime to contact and now we're going todo a little bit more but ultimately whatwe'd like to do is deal witharbitrary motionswhere u and vare not constrained by just a fewparameters butthey could bedifferent all over the image so u and vright noware some simple function of someconstantumwhere's the equation over hereand what we'd like to eventually get toafter a few more stages is where u and vcan vary arbitrarily over the wholeimage and you can see how that's goingto be problematic right becauseat every pixelwe have our magic equation up there thebrightness change constraint equation atevery pixel we get one constraint so wehave a million pixels we have a millionequationsbutwe have two million unknowns rightbecause at every pixel we'd like to knowu and vso that doesn't sound uh very goodsowe'll have to introduce some additionalassumptions to solve that problembecauseright now it looks like we havea million equations in two millionunknowns we know what the answer to thatthat one isbut fortunatelyin most cases there is additionalinformation for a startwhen i'm moving around the room andyou're moving in your seatsin most places in the image neighboringpoints are moving not the same but verysimilar and soif they were moving the same then wecould easily integrate that with ourapproachif they're similar it's a little bitharder but that's what we're going to dowe're going to say thatwe're trying to recover this vectorfieldin the situation wherethings can move independentlybut there's some it's not like you'relooking atoh you don't remember this but it usedto be that late at night tv stationswould go off the airand there was no signal and then theradio receiver would be basicallyputting white noise on your screen soyou just see this sort of every pixel istotally unrelated to every other pixelso that that's a situation we're notgoing to be dealing with because that'snot a natural situation that as i'mdoing my daily business of eating andwhateveri'm dealing with a situation whereneighboring pixels typically have almostthe same velocityokay well let's uh go back to this andtry and generalize it a bitso the the next most uh general thing todois to say okaylet's have motion in u and v as in the xand y direction as wellso so no longer will uh u and v big uand big v up there be zerobut they'll be umwe'll allow them to to vary as well solet's seebefore i go there let me justumsay something abouti totally missed this but everything wedid so far we got the answer and then isaid oh but how is this going to failwe haven't done that here so youremember that this was c equals theratio of two integralsand of course it's gonna fail if theintegral of g squared is zero forexample that's one way it can failso what does that mean well that meansthatthere's zero radial gradient everywherein the image and one way for that is tohappen is you're you know you're in acoal mine without lights e is zerothat's not a particularly interestingcasethe other one is thatthe brightness the radial derivative iszero everywhere that means that thecomponent of the gradientin the radial direction is zeroright so umwe have this expression over here thisis zeroand that means thatlet's see that means that the radialdirection and the gradient are rightangles to one another that's how we getthe dot product to be zeroso let's try and draw a picture likethat so here's an imageand now wesay we're there then the radialdirection is that and we're saying thatthe gradient has to be at right anglesto itso thatthe brightness can change in thatdirectionbut it can't change in this directionso here so in this direction we canchange we can't change in that area andsimilarly for you know any otherdirectionwe can have the brightness change uhthis waybut not uh radiallyso what sort of uh patternum would do thathow about a bull's-eyeis thatit will have variation of radialdirection so that that doesn't work soif it's not a bull's-eyeturn it 90 degrees everywherean x will douh sorry what did you sayif we rotate itif weif we draw a slice and rotate itpie charts right soif we have a pie chartthen everywherethe gradient isin therotating directionuh there's no variation in brightness uhinwards and outward so you know thiswhole pisector is one color this is anothercolor and so onand so that's a case where this willfailand does that make sense i mean wouldyou expect it to failyou know why why could you do betterso you're looking atyou see nothing else in the world exceptthis pie chart and you're moving towardsitright well theuh shaking of heads uh the image doesn'tchange i mean we're making assumptionslike you can't see fine littlespecks on the surface that will give youa clue assuming is perfectly smooth andyou just have these pi if you magnify itby a factor of two it looks the same soso that's perfectly consistent thismethod isn't going to work and it can'tit doesn't there's no information therefor it to work on so we will come backto this issue when we uh generalize itagain looking at the case where it'llfail and and you know is it reasonablefor it to failcould we do betternow in a lot of cases we can do betterjust becauseyou know if i look at a piece of paperit's not perfectly smooththey're tiny fluctuations they're littlefibers and soi might thisoverall pattern might look like this butthey're tiny little specks in there andi i can pay attention to those andsimilarly this algorithm could ifyou know if the contrast is high enoughof those little specks it would pick upa non-zero e x e y and it wouldcalculatethe time to contactokayso more general slightly more generalcaseso now we're allowingu and v to be non-zeroand so we've got uh u over f isi'm just copying the equation from overthereorandfor convenience i'm going to multiplythrough by ifokayso i'm just defining two new variablesso we've already had cwas w ofz which is the inverse of the time tocontract and now we're defining uh twomoreuh which are uh f u ofzwhich is f times x zeroandf v of zwhich is f over y zerouh why is thatwhat does that mean wellwhat we're going to find is time tocontact an foebut it'smore convenient to define thesevariables that are functions of time tocontact and foebecause then the equations are simple weonly have a few terms in themso this onehere is just f times the x component ofthe focus of expansion and this one isjust f times the y component of so soonce we know a a and bwe can calculate where the focus ofexpansion isassuming that we know fumone thing that i didn't mention is inour formula here for cf doesn't show upso that's kind of interesting because itmeans thatwe don't need to know certain propertiesof the camera in order to do thatcomputationwhich is surprising because for manypurposesyou do need to know for example if youtake the approach okay time to contactis distance divided by velocity well tocompute the distance you need f tocompute the velocity you need f so soit's kind of avery pleasant surprise that to computethe time to contact we don't need it andintuitively what's going on is that if iapproach a wallit's going to loom outwardumand if i approach it with a telephotolens it's still going to loom outwardand if you actually go through theperspective projection equation it'sgoing to increase in size at the samerateand uh you're right that that that isbeing excelled with bbcright so i've applied the perspectiveprojection equation hereokay umso now we have u and vas a function of three parameterunknownsand we plug them into our favoriteequationand let's see we get uhso that'sour brightness change constraintequationandand i'm going to again ifcall this g to cut down on the amount ofwriting the radial gradientandi'll use the same methodto try and find the answerokay sowe have this expression which if we hadthe correct values for a b and c is zerowe integrate over the image it shouldstill be zeroif we have the wrong values of a b and cit won't be zero and sowe formulate the problem as you knowmaking this as small as possiblenow in the absence of measurement noisewe could actually make it zero inpractice e x e y and e t will be subjectto measurement noise so we won't be ableto quite make it zero but it's still uhyou know a valid approach to[Music]pose it as this kind of least squaresproblemokayand of courseonly finite number of parameters a b andcso it's a calculus problemi keep on saying that because later onwe're going to be looking forfunctions not parametersand then we can't use calculus so can'tdifferentiate with respect to a functionthis is a short version of that story sookay sowhat happens if we differentiatethiswellthe derivative of thatintegralis the integral of the derivative of theintegrand the integrand is this squarething so we get 2 times thatso we're going to get2intoso we have 2 times that and then we haveto take the derivative of thatterm in herewith respect to aand the only thing here that depends ona is this part so that's just e xand then we repeat thatright and again we take the derivativeof the integrand so you get twice thisthing and the derivative of the termhere with respect to b is obviously juste y soand then the third stepis i'm not going to write this out againbutand the only thing that depends on cisthe cgand so we take the derivative and wejust get g so we get three equationsand uh magically we can actually solvethemso these these simple cases have awonderful feature that there's a closedform solution i can write out what theanswer isand that's you know invaluable becauseyeah you can always numericallycalculate stuffbut it's very hard to uhsay things about the solutionlike suppose i want to tell you thatthere's only one answerwell if you have some numerical way offinding the minimumthere could be another minimum somewhereelseand if i have an analytic solution aclosed form solution i can say thingslike how sensitive is it to noise well ican just differentiate with respect tothe thing that has noise in itif if i do it numerically i'll have torecomputeand then you say okay that's fine that'sthat answer is okay for this set ofparameters what about different set ofparameters you have to recompute whereasif it's analytic form you know there'sthe formula just there it is sounfortunately that doesn't happen allthe timeonce we make things complicated enoughwe typically can't find a closed formsolutionokay so what do we do with this well theintegral of a sumis the sum of the integrals so we canrewrite this[Music]like thisi'm leaving out the 2 because thatdoesn't make any difference to anythingright so i'm justmultiplying outthis term by e x and then splitting itup into several integrals becausethe integral of the sum is the sum ofthe integralsso that's going to be g e xokay so that's that's one equation andyou'll notice it's an equation that'slinear in a b and c so that right awaygives us hope that you know we can solvethis problem pretty easilyand if i repeat for the second equationso three three linear equations andthree unknownsand before we go on couple of things oneof them is you should recognizeparts of thisthat's what we had beforewhen we were going straight down theoptical axis where a and b are zerothen we're just left with with this partwe just had one equation and one unknownso that was nice and easyanother thing is that the coefficientmatrix is symmetricalso this coefficient is the same as thatone and that coefficient same as thatone and this one's the same as that oneand often having a symmetrical matrix uhgives some advantage in terms ofunderstanding the stability of thesolutionso what what are these things how do wedo thisbefore we even start to solve theequations so let's look at uhthe coefficient of a up herewell it's just the integral of e xsquared over the whole image so we we gothrough the image and at every pixel welook at the neighboring pixel wesubtract the gray levels to estimate e esub xand square it and add it to anaccumulatorwhich there you know and we do this forall the pixelsthen we have another accumulatorwhere we estimate e x and e y by lookingat neighboring pixels in the y directionmultiplying those two differencesand then we have a third accumulatorand g we have to do x times e x plus ytimes e yand then we take the result multiply bye x we throw that into that accumulatornow we don't need to do this one becauseit's the same as that onebecause of the symmetry of thecoefficientwe do have to accumulate that one wehave to accumulate this oneand that one and again these two uhbecause of symmetry we don't need to dothoseso we run through the imageand wehave sixaccumulators that we just add to as wego and of course you could parallelizethis because they don't interact i meanthe operation i do on this pixelin this case is completely independentof the operation i do on that pixel soif you have a gpu you can do a wholebunch of pixels at once you candramatically speed this up although youknow you you saw it running ona sillythinkpad and i was not doing any thingfancy it's using software that's youknow five years old soso this even with that uhuh if it's not cooked up to thisprojector it gets 28 frames per secondum so it's not uhcritical that you paralyze it but if youwanted to you could and then you couldrun it at thousands of frames a secondwhich by the way is what optical miceare on it they're typically running at1800 frames per second orin a gaming mouse even higherbut the images tend to be smaller theimages are often only 32 by 32 butokayso we we accumulate those six noticethat these do not depend on any changesin time these only depend on thebrightness pattern the texture that's inthe imagethen we need three morewhich do depend on onchanges in timeand that's it so we have a total oflet's see nine accumulators we have tokeep track ofand then we get to the end and then wehave to solve three equations in threeunknownsand i guess i i ran over time so arethere any questions before we departyou

Fixed Optical Flow, Optical Mouse, Constant Brightness Assumption, Closed Form Solution

we started off talking about the twoaspects of image formation where and howbrightand in terms of wherewe talk about respective projectionsand in a camera-centric coordinatesystem it's very easyand extended that to be able to talkabout motionand so we just differentiated thatand we had a slightly different versionof this wherewe made use ofthe perspective equation perspectiveprojection equation in order to write itin and then we introduced the idea ofthe focus of expansionuh that that is the place where u iszeroand clearly that's wherethis part is zero and so if we solve forxwe[Music]sothe focus of expansion is the point inthe image towards which you're movingand then we introduced thecontentand talked about various ways ofestimating thatokay thenfrom a somewhat differentpoint of view[Music]we looked at theimage solid sowe're thinking about an image asuh brightness as a function ofx and yand sometimesx and y and tandsothere's an image solidvideoandin this case welooked at the possibility thatthebrightness of an image of some point inthe environment doesn't change with timeso we introduced the uhconstant brightness assumptionso as wefollow some point and its image insuccessive frameswe are saying thatin many circumstances the brightnesswon't change and we can exploit thatso if weit's quite interesting to look at thisimage solid and slice it in differentdirectionsandyou'll seekind of a very streaky nature because ofthis phenomenon so theslices of course are not independentandwe see that things arestreaky likeyou know extruding toothpaste withmultiple different colors in itas things move in the imageand then from thiswe got thebrightness change constraint equationwhich gives us a relationship betweenthe movement in the imageand the brightness gradient and the timerate of change of brightnessand we thenaddress the problem that this is notgiving usthe ability to solve locally forvelocitybecausethis is a linear equation in u and v soit just defines a line in velocity spacesolooking at a single pixelwe can'trecoverthe motion unlike the 1d case where wecouldandso we need more constraintwella very extreme form of constraint iswhere everything's moving at the same[Music]speedand soas i announced there's apaper under materials on stella thatgoes into thatand it's sort ofdoing uh the last part of the previoushomework problemandwhat it does isminimize some error soif this applies at every pixeland we have a constant u and v for thewhole imageas in the optical mouse casethen we can write theproblem this wayand again that integral should be smallor should be zero if there was noerror in u and v and if there was nonoiseuh but we'resatisfied just tominimize that and that's going to be ourbest estimate of u and vand it's uh highly over constrainedright we've got one of these equationsfor every pixeland we're only looking for two unknownsso this is a case where we have you knowmillions of equations and we only gottwo unknowns so that's very favorablethat means that the result is going tobemuch more accurate than reliable than itwould otherwise beso doing thatwe obtained theequationlinear equation in the unknownswith a symmetric two by twocoefficient matrixsowe just run through the image and weaccumu we estimate uh the gradient e x ey we estimate the rate of change ofbrightness e tand then we just accumulate these totalsand when we're done doing that we havetwo linear equations in u and v and weall know how to solve linear equationsparticularly if they're only two of themnowas usual we need to look at when thatfailsand so we did that we said that well itdepends on thedeterminantsothat the problem iswhen thatcoefficient matrix is singular so wehave a problem ifif that is zero or if that's equal tothat and we looked at various umconditions like you know e equals zeroof course willif if you have a black image then thatthat won't workand also if e x is zero or e y is zeroand so onand justlet's look at one moresuppose that we have an image like thisand what we're trying to do is figureout whether thismotion recovery is going to worksohow can we attack that well sort of twoways the one isthat you know what type of an image isthat can weintuitively see why that's going to workor not workand the other one is just to you knowbreak down andcompute the derivativesso e sub x is going to bethe derivative of this thingplus the derivative of the argument withrespect to x many timesand so the derivative of this withrespect to x of course is just arightand take out the x so it doesn't lookconfusingly like atimes and then i can look at the yderivativeright so so this is a very particulartype of imageand in this case i havee x and e ythey may you know f may be somecomplicated function but the importantthing is that e x's and e y are in thesame ratio everywhereand soin thisintegration i can replace e yby b over a times e x right and so thisis going to be true this condition willbe true and it'll fail right soso this is another way of saying theconditionunder which this method won't work andthen we can look at what you know whatkind of an image is thatwell umas usual i can't draw gray levels on theblackboard and not draw gradients but ican draw contours of constant brightnessisophodes which are perpendicular to thegradient and so what are the isophodeswell the isophoto where e x y isconstant that means where f of a x plusb y is constantthat means where a x plus b y is aconstant and that's the equation of whatit's a straight line right so theisophotos are straight lines uh alsothey're all parallel straight lines theyonly differ in cright the uh a and b are fixed ahead oftime and soyou knowso this is the kind of image that givesus troubleand yeah we know that right because uhif it slides in this direction we wecan't measure that uh there's no changein the imageif it slides in that direction we canbut uhit doesn't allow us to determine theother part of itokayso thenso that's sort of the optical mouseproblemso from thatwe went to time to contactand we looked atso we had thisthere are various ways of rewriting thatum let's see which way around do i wantto do thisso w is the z component of the motion inthe world so that's uhd big z dtandand i don't know that may or may not uhring a bellbut that'sthat's the derivative of log of z so ifiplot things on a logarithmic scale thisis just the slope ofthe graphon that logarithmic scale so you knowthat's sort of interesting so it's allumdependent on ratios incfractional parts rather than absolutevalues so what's important is you knowby what fractional part does z change ina certain time interval not by how manymeters and that's one reason whywe could do this without calibration wedidn't know have to know what the focallength of the camera is for examplebecause it's only theratio the fractional part that mattersand nowanother way tothink about time to contactis in terms of image sizeso suppose up here is some objectin the world of size s and here is itsimage of size little sthen i can write an equation relatingthose quantities based on thetriangles similar triangles this thistriangle on the outside and thistriangle in the camera so i've got sover little fis big s over big zright there's just umthe lateral magnification of the camerawhich in this case is much smaller thanoneso the image is much smaller than theobject butwe call it magnification anywayokaywell i can cross multiplyright so i have that relationship andwhy do i do that well because now i'mgoing to differentiate thatand see how it changes with timeand so s is changing you know if we'reapproaching the object then s will beincreasing so s will be changing sowe're going to have big z d s dtand it's a product so we also get big zohget the other onewe're going to gets times dz dtand then we need the derivative of thisproduct well the size of the objectpresumably is constantwe aren't changing the imagingparameters so this is the derivative ofthat is zero and the derivative of zeroof course is zero so we so we get thisrelationshipand so this tells us that uhwhich were around it we use it over hereuhthat umbig s d s dt overs isz ofdz dt over zsothe uh change in image size infractional change in image size isexactly the fractional change indistanceand so for exampleif the picture of the the image of thebus increases by one percentuh as you go from one frame to the nextin your video sequence then that impliesthat the ttc is one hundredokay and so that's um[Music]you know at say uh20 frames a second that means it's onlyfive seconds away soso one conclusion is thatin a lot of important practical casesuh the time to contact is uh notnot tens of frames but uh you knowhundreds thousands of frames otherwiseyou you know you'd have a problem you'dprobably bejust about ready to crash into somethingand may have trouble compensating so ifthe time to contact in many cases isthousands that means that the fractionalchange per frame isone over thousandthsand so the fractional change in theimagefrom frame to frame is is relativelysmalland so that means if we were to use amethod that's dependent on actuallymeasuring the image sizeyou know estimating how big the pictureof the bus is in your image uh it betterbe really accuratelike you know one part in severalthousandand that means it's going to need subpixel accuracy that it won't be goodenough to simply measure umyou know where the front and the back ofthe bus is in the imageand that's whyum that turns out to be not a good wayto estimate the time to contactas opposed to the method thatwe describedokay so there's one thing i wanted toget across this very simple relationshipthatyou know if there's a certain percentagechange in size between frames thattranslates directly into a certainpercentage change in the distance andthatdirectly translates into time to contextit's a very easy way tounderstand thatokaynow when wedid this or you did itwehad a very simple situation we startedoffmoving uh directly towards a wallso that we had constraints on both thedirection of motionand on the surface we're looking atso the the very you may remember thevery first thing we calculated was cwhich was the componentin thez directionand we had some simple ratio of twointegrals and that's the case wherewe're moving straight towards the walland umyou know see the wall is what does itmean that we're the optical axis isperpendicular to the wall uhz is constant on the wall uh thisdoesn't vary as we go left and rightso that's a very simple case so then wesaid welllet'sbe slightly more general let's assumethat we could also be movingsideways as we're going along and thenwe added the motion in x and a motion iny and we had a slightly more interestingproblem where we werelooking for three unknownsa b and c and we ended up withthree linear equations in in threeunknownsand right now i remember was going tosay againthis is all spelt outin the paper that's on the websiteon on time to contact i guess the fulltitle is the time to contact relative toa planar surfaceokay and now the paper discusses someother things as well it's just like theother paper you know the first half isexactly what we did in class and then itgoes off into some other directions thesame hereso is is this the most general we canget well we're stillmaking an assumption thatz is constantso we're approaching the wall andtheoptical axis is perpendicular to thewall well what if our camera is tiltedor conversely we're approaching a wallthat that's tilted in the world soa different generalization has ushave thesez be a tilted planeso that umit's no longer the case that um thedepth is constant as i scan left andright or up and down in the imageand well what's the equation of a planewell it's going to besome linear equationum inx and y soyou know one way i could write it isin that formso this could be amore complicated model to look atandwell you might expect at some point theequations get pretty complicatedand in fact there might not be a closedform solution andthat's fine you can do it numericallybut in terms of understanding what'sgoing on and the noiseeffectit's nice to focus first on cases wherethere is a closed form solutionso uh what you can do actually is sayfine so now we will generalize it byallowing the plane to be tilted butlet's go back to the case where we weremoving straight down the barrel straightdown the optical axisand in that case then we'll only havethese three unknowns soinstead of having big a b and c be theunknowns we we have these three or somefunctions on themand sowe won't do that because it's you knowpretty straightforwarduh more messy algebra we end up withthree linear equations in three unknownsand and we can solve for that so we cando the time to contact uh in the casewhere the surf surface is tiltedso we're no longer making assumptionthat you know we're driving into theside of the truck uh coming out of aparking lot but we could be coming at ananglesothat's an interesting case to considerwhat if we do both you know what if weallow the surface to be tilted as wellas our motion to be generalwell we can formulate that problempretty easily and we end up with sixunknownsunfortunately they are no longer linearequations and soyou know from the point of view of painand agony uh they're not much fun towrite down and also it's sort ofunsatisfying because we end up withthese uh mixed set of equations some ofthem are linear some of them arequadraticandit's very hard to say anything generalabout themwhereas in these special caseswe can do a full analysis of of noiseand so on sookay so we're not going to do thatum just you know that it's therethen we get to okay why is the planesurface planar well the surface isplanar because it's going to give uslinear equations right so that so westart with a planar surface now realsurfaces may not be planar um and thenthen what wellwe can approximate them by polynomialsyou know of some locally quadraticsurfaceand we go through the same process weset up a least squares problemandwe unfortunately won't find uh closedform solutions but we canset it up so that some numerical processuh gives us gives us a solutionnow the reason we're not doing that ismostly because actually doesn't buy youanything so in practice when youimplement this you find thatmodeling the surface is planargives you a very good estimate of theof the time to contactand if you model it as something morecomplicated now you have more unknownswhich is good in a way because it allowsyou to model the world more accuratelybut at the same timeyou lose that over constrainedness rightevery time you introduce more variablesthere's an opportunity for the solutionto sort ofsquiggle off in another direction soso there's some pluses and some minusesand overallthe only time you want to even thinkabout that is if the objecthas a shapein depth where the depth change is uhsimilar to the distance from the objectso if if the truck is over there i i'm50 meters away from the truck and theside of the truck is one side is maybetwo meters closer to me than the otherit makes no difference i mean it's youknowfour percent change in distance and uhthat won't affect anything if if i'mright in front of the truckyou know i'm two meters away and oneside is one meter and the other one isfour meters then yes then you may needthis but we found in practice that wedon't need that extra level ofsophisticationthese two things are sufficient uh inpracticeumokaythen umlet me just briefly talk aboutmulti-scale so when i showed you theimplementationwhen you got really close to impactthings kind of just fell apart so thegraph of time to contactestimated was uh quite similar to theactualtime to contactsoyou know this was acontrived situation like constantvelocityso thatthe time to contact just decreasedlinearly as time went on right becausewe got closer and closer to the surfaceand then when we looked at the computedresults there were you know somethinglike thatnoisyand that's partly because themeasurement of the position was not veryaccurate it was you know eyeballing downon a measuring tapethere was an interesting offsetvertically so there's a bias so it's notjust noise actually the uhestimated time to contact wasoverestimatedwhich in itself isn't good because youknow you you're about to crash intosomething you don't want tobe told that actually it'll take longerthan the true timeon the other handsince it's a systematic fixed bias youcan compensate for itdoesn't mean that you shouldn't try andfigure out where it comes from but it'sprettysimple to justyou know fit a different slope to thisokay but then at the end herewe hadyou know some spikes and were basicallythe results were not reliable and wealready mentioned uh some reasons forthat and one of them is that the imagemotion is large so earlier we said thatyou know when the bus is far away theimage motion is very small the imagethe image of the bus will tend toexpand and contract and moveby you know a fraction of a pixeland that's wherethese methods really excelwhere you know we as we go along we'vebeen making some assumptionsthat uh certain distance epsilons aresmall when we estimate e sub x e sub y esub t for examplewe're sort of taking a finite differenceand saying oh this is almost aderivative because epsilon is small wellthat won't work if we have a largejump in x y or tand sothat's one reason that this falls aparti mean the other reasons one of them wasthat the camera went out of focus and soyou didn't have a clear picture of theobject anymore butthis first part is easy to deal withuh as i already mentioned before butjust want to reiterate thatif wehave an image with less resolutionsay we have half the number of rows andhalf the number of columnsin the image well thenthe motion in terms of pixels per frameis half what it was beforeand so uh what was a large motionin the original raw image is now uh halfof that and so that means thatyou'll still have it falling apart butit'll fall apart uhlater so this part will still be okayand then it'll fall apart down there andof course then you can repeat thatprocess and say okay so now in thatimage suddenly the motion's gotten to bemore than a pixel per frame so let'sagain subsample average in subsampleandand then we can continue it and somultiscale just means that we workin thatset of images that become smaller andsmaller and we can handlemotions that become quite largeand also i mentioned that you know if wedo the simple two by two block averagingthat means the second image is onlyquarter of the size of the first one sothe amount of work isone plus a quarter times what it wouldhave taken just on the raw imageand then if we do it again so that'sgoing to be a sixteenth andand so so the total amount of workwell you know writing the code of courseuhtakes a bit more effort but in terms ofthe time uh it's not a big penalty towork at multiple scales and you get uhhugely improved resultsand we'll we'll talk a little bit laterabout how to do this subsamplingthose of you who've taken 603 of courserealize thatyou you can't sample without gettingaliasingunless you've low pass filtered first soactually what you want to do islow pass filter and then subsample andyou don't necessarily need to subsampleon a scale in x by 2 and y by 2 youcould subsample by i don't know squareroot of 2 which is less aggressive andintroduces fewer artifacts but for themoment we'll ignore that and just takethe very simple idea of two by two blockaveraging which is a crude form of lowpass filtering and it doesn't do exactlywhat you need to do but it removes itsuppresses some of the high frequencycontent and while they will be aliasingartifacts they'll be greatly reduced andthat you know that's such a simplemethod to implementand it works pretty wellokay well let's talk a little bit aboutwhat to do what do i do with time tocontactsothere's sort of a number of interestingapplicationsevery yearthere are several incidents withairplanesonthe runwaywhere uh wingtips are taken offso you know these planes have very longwingsand they typically slip back so they'renot terribly visible and sothere's an opportunity to bang them intoa building or into another planeand that's you know really expensive uhthing to deal with it's notlife-threatening in most casesum but it's something that you try andavoid and as you know as you approachuh theplace that youget on the plane there's often someperson down there with some sort of redlighted stick and they're called wingmenand the reason is they they walk underthe tip of the wing so that the pilotcan look back and seeyou know whether where's the groundprojection of the wing and i'll try notto hit that staircase with the wing soum it looks likeyou knowyou could implement time to contact imean you can implement it easily onandroid for example so you could build areally cheap little box the only purposein in life for it is to look out and seeif something is rapidly approaching andthen give you a warningand so we've you knowthought about uh suggesting thattoairplane manufacturers and such anddidn't get anywhere but boeing took theidea and they came up with a 150 000radar solution and you know that'sobviously going tobe much more fun for the corporation toimplement than something as silly asthis soanywaysour grapesokaynext projectisnasa landing on europaso umyou know europa is a long way away andwe have some idea of what's on there butnot not a whole lot so it's not like wehave detailed topographic maps andimageryand so theythey want something that reliably bringsdown a spacecraft and so one ideais to use time to contactand so in controlso let's look at how we would do thatokay so we havea typical control system we input somedesiredtime to contactthen we havean actual estimatedand we subtract the two and that givesus some kind of error signaland we multiply that by againand we use that tocontrol the jet the rocket engine tochange the acceleration soand then there's a dynamical systemwhich is second orderin the sense thatwe're controlling accelerationnot height directlyheight is two integrals down from thepart that we can controland thenthere's an imaging systemand so this system does something verysimple which may not beyou know the best you can do but it'seasy to analyzewhich is totry and maintain the time to contactuhthe samesoifumyour measurement says thatat the current rate of descentyou're going to have a shorter time tocontact thandesired then it will add some moreforceto the engine andconversely if it looks likeyou're kind of hovering too much youshould be dropping downthen the time to contact will appearlargeandthen the error signal will cause you toreduce theengine output sovery simple system and we saw that youknowtime to contact is very easy toimplementand it doesn't care what theimagery is to large extent so okay wedon't really know what the surface ofeuropa looks like except from far awaybutwe're not depending on some particulartexture or calibration oror you know topographic map of thesurface or somethingthis method works uhwith any texture well except we saw thatthere were certain special textureswhere it would fail but presumablyeuropa hasn't been painted in one ofthese unique stripy patternsbutokaysoumwhat are the dynamics of thisbecause this idea oftime to contact control can be used inother situations as well uh such as inautonomous carsbut let's focus on the descent hereokay so if we umif we uhnow why constant time to contact wellum we don't really knowhow to uh accurately reliably easilycompute uh height from a monocularcamera imageunless we have some target like you knowthere's a walmart and we know what sizewalmarts are so we cancompute how high we are well therearen't any on europa i hope so thatwon't work for us um soif we could separately compute heightand speed then we could do much moresophisticated thingsbut we know that we can very robustlyget their ratio in a very simple way soso that's the attraction thereokay so we've gotz of a w is t and now we're assumingthat t is constantuh it's very curious that umyou know when dappa had the grandchallenge because mit was involved inthat and we instrumented the car andthere was one sequence that uhwe thought would be interesting tocompute you knowafter the fact uh okay they've recordedall this video let's do something withit uh let's compute time to contact andso there's the vehiclecoming out of a parking lot and it'sapproaching themit transport little busand if you plot the time to contactum it's almost constant so the thevehicle is slowing down the closer itgets to the bus the more it slows downand so it'si have no idea what in the controlalgorithm of that autonomous vehicledid that but it was interesting toobserve that it used aconstant time to contact control toapproach the bus without running into itsookay well here's aequation that we can solve so it goesz over dz dtis t and sodz dtis one of a t into zandwell there's a ordinary differentialequationthatyou should know the solution forkeeping in mind that big t is a constantso dz dt is proportional to zwhat sort of function does thatquadraticexponential anyoneokayso we get uh z is z naught e to theminus t over t something like thatright we differentiate that and thederivative is proportional touh the function itself and soif we implement this uh constant uh timeto contact uh control systemwhat we'll find is thatyou know we're going to geta descent that looks like thisso here's our z naught times zeroso it's going to be nice and gradual andsmooth and uh and it will never getthere so that's the downsideum so what do you do there well uh wecan do what flies do they use constanttime to contact when landing on theceilingand when the legs touch the ceiling theystopso we can havewire hanging down fromour spacecraft and when it touches theground we shut off the engines and thenit will just fall those last meter ortwo and that method of course has beenused in planetary spacecraft before butwe can just combine it here with thetime to contact so at a certain pointwe we do have a distance measureactual wire and then it just uh dropsunder gravitational controluntil it hits the surface i guessokay so that's one aspect of thisand we can you know go into an erroranalysis ofwhat will happen thereumjust for fun we can compare this with amore traditional approachso a more traditional approach would beyou know we run the we run the engine atits uh rated speedand we decelerate and we only turn it onat the timewhere weneed to turn it on so that we don'tcrash into the surfacesonow that method requires that we knowthe distance to the surface and we knowthe velocity so we have to know a lot ofstuff where withthe time to contact control we don't youjust keep the time to contact constantokay soso one advantage of this alternatemethod is it's more energy efficient souh hereyou know we'll be kind of going almostinto a hovering mode so we're kind ofwasting fuel in a wayso there's a trade-offokay soum sowe we're ignoring the fact that thespacecraft is getting lighter as isusing fuel and so the acceleration willchange we'll just assume that it'sconstant and so we integrate that onceand we get uma constant of integrationand we can apply the boundary conditionswhich are that at some point we reachthe surfacesookay and then we integrate a second timewe integrate that and we get z is a halfa t squaredplus some other constant of integrationand again we use the boundary conditionand we end up with z as a halfaoh of course you you know you can dothis the other way around it might beeasier to start off with that and thenjust differentiate to get theto the constant accelerationokay[Music]so why did i even botherdoing this well because sort ofinteresting to askunder that type of control what is thetime to contactand so how do we compute that well weneed z and dz dtso we just take the ratio of these twoso t isand so that's going to be a halfso that's sayingwhat the time to contact is during anypart of this maneuverit's fascinating that it's a half ofwhat you get over heresoyou knowover here of course the time to contactwould be t minus t 0 but because of thisconstant acceleration control we getget that result anywayit's justa way of comparingconstantto contact control with a more moretraditional approachthe more traditional approach issomewhat more fuel efficientbut it's much harder to implement itrequiresyou know accurate estimations ofdistance and velocities anywayapparently nasa decided they canaccurately estimate distance andvelocity soand you know these uh crazy machinevision programs who knows whetherthey'll work so they're not going to dothatbut um you can imagine that thereyou know the other applications for thisbecause this is a sensor that's reallydumbi mean it's simply um a matter ofbrute force estimating gradientsand accumulating totals multiplying andthen solving a bunch of equations soit's very straightforwardnow before we go on to another topic iwant topoint out another generalization thatwe'll take up later and that's opticalflowso the paper on stella that talks aboutthe optical mouse problem primarilyiswhat's the title fixed fixedcomputational fixed flow determiningfixed flow fixed flow what does thatmean well it means thatthe motion of all parts of the image arethe sameand as i explained for an optical mousethat's a very good model that's veryaccurate but as i'm walking around theroom uh the motion of different parts ofthe image are not the same becauseyou're at different distances and we sawyou know how we can get that out of theperspective projection equation bydifferentiating and we get some termsthat uh are affected by you knowdivision by z and so onso and then also you know some of youmay be moving around independent motionit's not thatso these problems are particularly easyto solve when they're only a fewparameters so for the optical mousethey're two motion in x motion and ywellyou might be turning the mouse so itcould be threeand we'll probably take that up in ahomework problem so it'll be a slightgeneralization of what we did with theadded feature that you're not onlytracking the position of the mouse butif the user is turning it you also wantto recover thatmaybe not because that's an interestinginput to your gui but because it mightscrew up the estimation of the x and ymotion if you do rotate soanyway so but those are all cases wherewe have a fairly small number ofunknowns and we have a hugely overdetermined system we have somemeasurement at every pixelwhat if we don't have that well that'sgoing to be a problem becausebecause at every pixel we have anequation like thatso we have a constraint so if we haveyou know 10 million pixels we've got 10million constraintsgreat except at every pixel now we alsohave an unknown velocitywhoacheap chalkum sothat means that we have umtwice as many unknowns as there areequations well that's you know aprescription for disasterso umhighly under constrainedand it's it's the uh it's an ill-posedproblemand in this case it's ill-posed in thesense thatthere's an infinite number of solutionsand in fact if you give me a solution ican easily constructanother solution because all i need todo is add every pixel i need to obeythis constraintso at every pixeli'm somewhere along this line right andand now you give me the quote correctsolutions let's say it's herefor that pixel well the thing is thati cango thereit doesn't changeanythingand so umthat's pretty dramatic that means that ican systematically go through the imageand at every pixel i can give you aninfinite number of other values that'llworkso we'll need some heavy constraintand the one constraint that we can useand we'll use later is thatneighboring points in the image do notmove independently they may not move atthe same velocity but they tend to moveat the similar velocity so you knowthat's a good thing and a bad thing imean it's good in that oh here we've gotsome sort of constraint and it's a badthing because it's not like an equationthat says you know u plus 3v is 15 orsomething it's more vague it's like ohit's varying smoothly you know what whatdoes that mean what smoothly whatepsilonuh change can you allow so that'sthat makes that problem uh veryinterestingand um uh you know non-trivial and we'llwe'll get to that later we don't havethe tools at this point uh to do thatbut it's just to alert you to the factthatthe fixed flow isn't the the be-all andend-all uh there's more to comenow suppose thatyou don'thave the tools to solve that problemwell you can do somethingwhich isdivide the image up into piecesandand then apply fixed flow to each pieceright um and the idea is that well if wemake the pieces small enough there won'tbe much variation in velocity withinthat piece and so the uh assumption thatu and v are constant in that little areaisn't such a bad oneand sonow they're all kinds of trade-offsright becauseif we make these boxesvery large we get only a very coarseimage of the flow we only have you knowa flow vector for each of these boxeson the other hand if we make thearea sub image areas very smallwe have much less constraint and we havemuch less of that wonderfulyou knownoise suppression property of an overdetermined systemand alsowhen we look at small areas of an imagethey are much more likely to be similartothe type of image that we talked aboutthat doesn't workyou know if i look at a very small areaof the image and it just has this edgein it well that's exactly the kind ofthing where the aperture problem comesin and i can't determine what the motionis in that direction so the so yes youcan do this you can use what we have thefixed flow methodon a gr on a grid of a chopped up imageum but they're they're trade-offs andthey're somewhat unpleasant you know ifyou make these areas fairly small someof them may be even more or less uniformin brightness like you know if i'mlooking at that gray wall it's uniformin brightness and if it if it moves ican't tell it's just the same so anywaythat's something that uhhas been doneandsort of worksbut it's not umyou know the solution that we'll we'llbe looking forokay um[Music]we're sort of moving towards talkingabout brightness more than uhperspective projectionbut i want to just do uh one more thingwith perspective projectionand that has to do with vanishing pointsandthese play a role incamera calibration orsometimesfinding the relative orientation of twocoordinate systemsand just to make that sound lessmysterious if we have man-made objectsthey're often they often have planarsurfaces and they often have rightangles between their planar surfacesyou know unless you go over to starterand so when we look in the images we'regoing to find a lot of straight edgesoften and often a lot of them areparallelsoand so uhwe can actually exploit that so if forexample you're hovering above somebuilding that like the main buildings atmit is all on a rectangular gridyou can determine from the image certainvanishing pointsand from that you can determine yourrotation relative to the coordinatesystem of that rectangular blockand so that's you know something thereorif you look at a cube or some othercalibration objectyou may be able to recover parameters ofthe imaging system so it's important incamera calibration which is of courseimportant in robotics and otherapplicationsso let's just see what uhwhat this is about so in the firsthomework problem one of the questionsyou were asked is you know what's theprojection of a lineand there are various ways ofapproaching that algebraically orgeometrically you know one geometric wayis just to say okay here's my linei'm going to connect itto the center of projectionand what do i get well i get a planeright if i connectuhif i look at the locus of all thoselines that connect that point to theline it forms a plane and then what isthe image of it going to be well i'mgoing to intersect that with the imageplaneso what's the intersection of one planewith another planeit's a straight linesothere's a simple way of seeing that aline in 3d projects into a line in 2dit's a funny sort of projection in thatif you were to mark this you know like ameasuring tapewith equal intervals and then you lookat the projection of those marks theywon't be equally spaced right becausethe part where the measuring tape isclose to youwill image with larger magnification sothe marks are further apartthanbut you know so so the fact that a linegoes to a line is a little bit umuhover confident it's it's making us thinkthat we understand the problem very wellwhen actually it's a little subtle sothe other way is algebraic and so oneway we can do this is tosaythat a line in 3d can be defined invarious ways one of them isyou know we have a point on the lineand then we have a directionand we might as well use a unit vectorto define the directionso so that's one way of talking about aline in 3dumwhat what are other ways can you thinkof some other ways ofumright we can do a parametricrepresentation wherewe have you know umfor example x is some function of someparameter t and y is some function ofparameter t and z as some function ofparameter tor we can have some implicitrepresentationorwe can intersect two planesrightand why is that convenient well theequation of a plan is a linear equationand so when you intersect two planeswe're dealing with two linear equationsand we've already seen that you knowhaving linear equations can be a plusbut but let's stay with this and youknow in component form of course thatjust means it's uhx naught plus s what did i call it alphaor somethingumalpha isokay so then we're going to now projectthatin into the imageusingperspective projection andwe can of course use the component formthat we use up there or we can usethe vector formand so we getso unfortunatelythis doesn't you know lead to some veryelegantnice resultor rather it it doesn't without uh sorryi guess z z is dependent on gammaum unless we go quite a long way furtherthan i want towhen it becomesnice and elegant againokayso that's our transformationand so s is a parameter that variesalong the along the lineand so you knowdifferent points on the line havedifferent values of sand if that's a unit vector then s isactually a measure of uhlength along there soumso that's what because of that divisionby z it's it's kind of messy but onething that's sort of interesting to dois to look at what happens when we govery far along the lineright thenso we go very so we make s very big wellthat means that we can ignore x naughtand we can ignore z naught and we getalpha over gammaoh gammaokay and that is called the vanishingpointso as youmove along this lineum youstart to go more and more slowly in theimage and you approach but never reachthis this point and so actually theimage ofa infinitely long line is not aninfinitely long line in the image planeit starts somewhere and and this iswhere it startsumthenas we move along the line in 3dwe don't move along the line in 2din auniform way right because when we'revery far outwe can move a long wayin 3d and it only has a tiny effect inthe image so that'swhat makes itkind of awkwardokaynow one thing that we should immediatelyrecognize is thatparallel lines have the same vanishingpoint right becausebecause um the offset x naught y naughtz naughtyou know theorigin of our line there don't come intothis equation it's only the directionthat that mattersand so so that's interesting becausethat means that if we have a bunch ofparallel lines in 3dthey're all going to give rise to thesame vanishing point so if we're lookingat a rectangular buildingthere are three sets of edgeseachcontaining parallel linesand so we expect to see three vanishingpointsso let's see if i can i can do this thismight benow this is a bit extreme because ipicked the vanishing pointspretty closeokay so here's a cubeandwellnot that good but anywayif if it was for real there would be uhthree sets of parallel lines which eachgive rise to a vanishing pointand soso what so well the point is that if ican find those vanishing points in theimage i can use them to my advantage tolearn something about the geometry ofthe image shaking situationand just toyou knowlet you know that this isn't some vaguetheoreticalthing that nobody cares aboutumhere's an application so umevery year a few people are killed onthe side of the roadbecause of distracted drivers evenbeforetextingand so there's an interest intrying to warn whoever stopped a policeofficer construction worker or whateverthat there's a car on a trajectory thatmay impact them so how do you do thatwell you can stick a camera maybe uhandroid phone uh in the window of thecruiser and it's watching uh this ismostly night time it's watchingheadlights going by andit's monocular so it doesn't have depthinformationbutit can figure out whether trajectory isuhyou know possibly dangerous or not butone of the things it needs to do is tofigure out the geometry of itscoordinate system and the roadscoordinate system right now you don'twant the person who's using it you haveto come out with surveying equipment andyou know measure the angles and so thatso the camera has to on its owntry and determine therotation of itscamera-centric coordinate systemrelative to theline to the roadx y and z and soone way to do that is to look for avanishing point soin the case that the road section isstraightyou can use image processing message tofind those lines and then you canintersect them to find the vanishingpoints and you can use those to recoverat least two parameters of thetransformation one is you know how muchis the camera turned relative to thedirection of the road and the other oneis how much is the camera turnedrelative to the line to the horizon sowe've got pan and tilt and we can getpan and tilt using vanishing pointsokay[Music]i don't want to do thatokay so let's think about umusing this in camera calibration anotherapplicationso umin calibrating a camerathere are several parameters you'relooking for um but the the main ones arethe center of projectionright soyou say well we are the center ofprojections where the lens isyeah but you know where's the lens andin relative to whatso here's our uhintegrated circuit sensorand up here is the lens or pinhole forthe momentandpresumably whoever built this thingtries to put the center of projectionabove the middle of the chip but that'snot necessarily going to be accuratelydone i mean these things are microscopicuh in some cases uh you know like likethis thinghad a focal length of four millimetersandso with dealing with very smallquantities plus uh the pixels here maybe you know 10 microns or in the cellphone maybe only five soit's very unlikely that you wouldbe able to position the lens in such away that it was accurate to one sensorposition so so we need to recoverthat position and we also need torecover theheight of the center projection abovethe image planeandultimately we'll be using acamera-centric system that's origin atthe center of projection but to do thatwe need to understand howrow and column in the image sensortranslate into x and y in thatcamera center coordinate system sothe short answer isuh if you give me a coordinate system inthe chipwhich is uhlet's say columnand row counti want to knowumwhere that point isand you know typically the row andcolumn count won't be taken from thecenter of the chip but from one of thecorners andwhen you give me the position of thecenter of projectionwhat units do i wantthe size of the pixelit'd be nice to have it in microns butifif you don't know the size of the pixelsyou can't do that and converselyyou don't need to know the size of thepixels for projection we only uh we canexpress thatlike the focal length we can express itin terms of pixel sizeparticularly easy if the pixels square imean otherwise you've got todeal with the fact that the x and ydimensions aren't on the same scale butokay so that's the taskyou know tell me where that point isand it may need to be repeatedparticularly if it's acamera that has zoom capability becausethen all of that's going to change asyou zoom in and out and you'd want toperform this kind of calibration againso the various ways of doing this one isand we'll talk about this some morelaterbuthere's a very simple onewe use a calibration objectand so calibration object is somethingwith a known shapeand we already talked about that when iwas showing the slides where we use thesphere as a calibration objectsowould a sphere bea useful calibration object hereso the image of the sphere as you saw inhomework problem oneis a conic sectionand soif this for example if the sphere wasdirectlyabove on this line its projection wouldbe a circle right if the sphere is uphere and how do i know that well becausei connect the sphere to the center ofprojection and i get a right circularconeand i extend that down here andintersect it with this plane andintersect the right circular cone with aplane perpendicular to its axisand you get a circle sobut when i move the sphere to the sideit's going to become elliptical it'sgoing to become you know imagine take anextreme case where this is way down hereand now you project it into the image itbecomes a very elongated ellipseand if you move it down far enough it'llbe a hyperbolaso yeah you can you can do that but itwould then require that you accuratelydeterminethe position of that figure whether it'san ellipse or hyperbola or circle orparabolaand its parametersso that can be done but it um the noisegain is high it's not a very good methodi mean the big advantage of this is thea sphere iseasy to make easy to obtaina cube isn't so let's try a cubei know that a cube isn't easy becausewhen my father got hismasters inoperating equipment to make objectshis task was to make a one centimetercubeand it had to be accurate to a micronand it apparently took him quite a whileto do itbecause it had to be accurate to amicron and the sides had to be at rightangles and so on so making a so making asphere as a calibration object issomewhat easier than making a cube butthe cube has some huge advantagesand one way of exploiting it is is thisdiagram so if we take an image of thecubewe we candetect the edges uh we'll talk aboutthat laterand then we can extend them to find thevanishing pointsand by the way the vanishing pointsdon't have to be in the image you knowit could be that the image you actuallysee is is this thingthe thevanishing points are in that plane buttheyin many cases are not in the image andthat's why i struggled in partaside from having no drawing ability istruggled with that diagram because inthe real situation they tend to befurther out and so theperspective distortion as it's calledwouldn't be as extremeokay so so what so suppose so i have acalibration object that's a cubeand then i take a picture of it and ifind the vanishing points what then wellone thing i know is that thecube has three sets of parallel lineand those are at right angles to eachotherokay if they weren't then it would be aparallelism pit not a cubeso they're at right angles to each otherand so that's very important uh becauseit brings me to umyou know the equations for the vanishingpoints it means that the directions tothe vanishing points are at right anglesto each otherso here's my center of projectionand then i have three vectorscorresponding to the three sets of linessothere's you know one that's going up anddown and one from that side and one fromthat side and i can draw them uh throughthe center of projection so here's onehere's another one and here's the thirdoneand so so what are those lines wellthose are lines in the direction of theuhof the 3d lines which i guess we've lostnowumsowe said thatall of the lines in this parallel bundleproject into the same vanishing pointsuh but there's one that specialwhich is the one that goes through thecenter of projection so this is you knowthink of this whole slew of parallellines and um they all end upat the same vanishing point but they canbe represented by a singleline we can just pick one we pick theone that goes through the center ofprojectionand so that's what this is so this isokayand then because of the way ourprojection worksyou know there's a point down here and apoint down herein the image planeokayso if you tell methe three vectors that point in 3d alongthe linesi can tell you where they will be imagedjust by this construction right becausehere's the direction that's going alongone set you know here all these parallellines they're all going in thatdirectionand all i need to do is follow thisbackward into the image plane and that'sgoing to be its vanishing point sothat's vanishing point one vanishingpoint two uh call them something elsesorryi guess i call them a b and cokayokaynow the neat thing isi know that if my calibration object isa cube that these three vectors up herearen't just any all three vectors butthey are all at right angles to oneanotherhard to draw that but there are threerelationshipsbetween them so let's call the unknowncenter of projectionpthat's our taskfind pgiven a b and cwell then i can say thisuh so why is that wellp minus a is a vector in along this lineand p minus p is a vector along thatline and those are the same vectors asthese two and we know they are rightangles and the dot product of twovectors that are at right anglesis zero rightsowe get the calibration object we take animage we find the vanishing pointsand then we have those three equationsand we have three unknownsnamely the components ofpthe center of projection or another wayof looking at it iswe need to know this heightcall it fand we need to know where this is on theimage center so there are two degrees offreedom here one degree of freedom therethree total but you know simply speakingwe're trying to find where this isand that's a vector in 3d so it hasthree degrees of freedom we've got threeequations great you know sowellif you look at them you'll see that thesecond order in p p is the unknownand the second order they're quadraticrightsothere's a finite number of solutionsexcept for pathological cases but whatis thenumber of solutionsso you know with a single quadratic weknow that they're possibly twoso maybe they're more solutions sothere's a thing here calledthe zoot's theoremwhich we'll use quite a bit and it saysthat the maximum number of solutionsis the product of the order soof the equations so for example if ihave two quadraticsit is possible there might be foursolutionstwo times two here i got threequadratics so it's possible there couldbe uh eight solutionsso that's unpleasant so we want to talksome more about thatby the way it's called uh bizu's theoremafter bazoot whowrote this up and actually uhnewton used this resultin his principiabut he didn't reallyformalize it he just sort of used itlike you know you anyone would know thisthat kind of thing and when bazoot uhwrote it upuh he didn't actually rigorously proveit so there's a whole lot of it's the asusual someone ends up with their name onsomething and it's there's a wholeinteresting story behind it like okay hedidn't that he wasn't actually the firstor he he didn't actually get it right orsomething anywayit's an important theorem for us nowwith linear equations the product'salways one right you know one to thewhatever power is one so with linearequations as long as we can matchconstraints with unknowns we're doneunfortunatelyin other cases it's not quite thatsimple okay but we can dowe can do something which is notice thatthese aren't just any all quadraticequations they have very specialstructure and we can subtract thempairwise to get rid of the second ordertermso we can get umso let's see if we subtract the secondthe first and the secondwe end up with this this is zero andthen we canget a few moreokay so uh terrific we've reduced it tothree linear equations and we know thatthey have only one solutionwell not so fastare those three linear equationslinearly independentwe have to worry about thatedge case where the matrix is singularand so on well the truth is thatumif we add two of these we get the thirdone so so they're not independentequations so actuallywhen we get to the third one we shouldstopso yes we can reduce itto two linear equations but we're stillleft with one quadratic and so theanswer is that there are two solutionsnow umand we won't actually do the algebra butit's pretty straightforward forward[Music]okay umi'm not quite done with this i wanted tosay something more uh but who is sort ofout of time so one thing that i wantedto still say is thatthose linear equations what do theydefine in 3d spaceare planes right soeach of them defines a plane and whatwe're really doing is we're intersectingthose planesand it turns out that thethe planes umin two planes intersecting a line and itturns out in this case that that linecontains the third plane so the thirdplane doesn't get you anythingumand then umyou know just to summarize how thisworks we will finish this next time umoftenwe are in need of calibrating a cameratime to contact is one of the few placeswhere we didn't need a calibrated cameraand to calibrate the camera we often usecalibration objects i mean it doesn'thave to be a cube it could be forexample the corner of a room as long asthisyou know the geometry of itandthen in that case often vanishing pointsare helpfuland we can work out the geometry ofvanishing points which lead us toa set of equations and when we solvethem we find the position of the centerof projectionnow if lenses were perfect that would beit so this would be it for cameracalibrationand we'll talk more about cameracalibration laterlenses have radial distortionand there are reasons why people don'tcompletely get rid of thatso actually in practice when you do realrobotics camera calibration it's thisplus the the radial distortion parametersookaythat's it for today

TCC and FOR MontiVision Demos, Vanishing Point, Use of VPs in Camera Calibration

have another go at thisdemonstrationusing a direct hdmi connection and let'ssee what happens card play graphokay so let's first go back up herejust makeso that's uh what we've seen beforeand now let's try uhokay that's uh that's a bit betterum[Music]so this is a webcam looking down at thekeyboard and when i hold it still youcan see a b and c are near zero they'resmall on the right side thereand the time to contact bottom right issome large number that sometimesnegative sometimes positiveand now if i move it away from thekeyboard it should go greenon c the third one is the c componentone over time to contact as i approachthe keyboard it should gored meaning dangerand the you knowso there's thatumand of course it's independent of thetexture so i can do the same sort ofthing on any surfaceif i try and move it in x onlythen the first bar would be large andthat's sort of true i'mi haven't got it oriented exactly rightbut then if i move it in this directionthe second bar would that's bandokay so that's uh working a little bitbetter than it was last timeum start off with this correctionsomebody last time pointed outa sign errorso we had thatdiscussion about different ways ofthinking about time to contactone of which waswhat we were discussing and another onewas sizerate of change of sizeand so we hadfromperspective projectionwe had this equation and then we crossmultipliedand we gotthat and[Music]sorry equalsand then we took the derivativeto getand because this is a productandthis is a constant so we get 0. and thenand so umthere's there's a minus sign thereand that makes sense becausewhendzdt is positivethings are moving awayand in that case the imageof those things is shrinkingnegative dsdtokayall right umwe're busy talking aboutperspective projectionandhow we can use it in various waysand in particularwe were busy talking aboutvanishing pointssoagain toexplain what those are so here's ourimaging systemthere's an imageplane there's a center of projectionand out in the worldwe have a bundle of parallel raysvectors that are filling space they'reall paralleland one of them is special because itgoes through the center of projectionso out of all of these parallel raysand of courseinmuch ofhumanconstructionwe have parallel lineshappens to be a fairly efficient way ofdoing thingsand unless you're trying to be veryartistic and build the starter centeryou'll have a lot of parallel lines sothese could be edges of buildingscornersedges of windows anywayone of them is special in that it goesthrough the center projection and ithits the image plane at a particularpoint and we can use that point asrepresentative that's a way we can talkabout which set of parallel lines we'retalking aboutand that's called the vanishing pointand the reason is that first of all forthis particular lineyou're looking straight at that line andyou just see a pointandwhat about the other lines well if yougo far enough out on any of theseparallel linestheir projection into the image willcome closer and closer to the projectionof of this linewhy well because there's adecrease in magnificationfromdistance in the world to distance in theimage the further away we go we have youknow f over z and so if z becomes verylargethen the magnification becomes verysmall and so any difference betweenthese isreflected in the image by a smaller andsmaller distance so these other arraysactually are also image these otherlinesbutas linesand those lines have to go through thevanishing pointsoas i move outwardi guess i have the arrows reversed inthe two cases but as i go outward alongthese linesi'm coming closer and closer in theimage to that vanishing pointsothat's the idea of a vanishing point andwe can exploit that because it allows usto determine relationships betweencoordinate systems and it allows us tocalibrate the camera so those are twothings we'll briefly discuss today andwe started yesterdayso first let's talk about the cameracalibration problemsohere's my image planehere's the center of projectionand now suppose that umwe live in a world of rectangularobjectsand so each rectangular object has umsets of parallel lines uh three sets ofparallel linesi guess uh four of each of themand sothey define a coordinate systemthey define three directionsandi canpickparallel linesthat go through the center ofprojection sookay soout here is some objectand it has parallel lines like thatand i pick out ofthe family of parallel linesin that group uh one that goes throughthe center of projectionsolet's call that direction xsothat's a coordinate systemwhich is uh just a parallel translationof the coordinate system on the objectdoes it if so for the moment we'reignoring translation we just were goingto worry about uh orientationokay so theni project those three into the imageplaneum you know as we did over thereand well in my case you know some ofthemsome of them may go out of the formatand often uh the vanishing points arenot in the part of the image that you'reactually sensing but that doesn't matterwe're only interested in the position uhin the image planeso let's suppose thati don't know we can call these a b and cor we can call themr 2and r 3.soif i have my picture of the cubeas i did last timei can define three vanishing pointsjust by extending those parallel linesnow they you know how accurately do weknow them well that's another storywe'll need to know how accurately we candetermine lines but let's suppose forthe moment that we've got these threevanishing points so we can imagine alittle diagram in hereof some sort of rectangular objectand you know highly distortedbecause actually in practice thosevanishing points will tend to be furtheroutif i draw them this close then i'm goingto get a lot of quote unquoteperspective distortionnow of course the term distortion issort of odd because basically this iswhatperspective projection doesit'snot likethe effect ofradial distortion in an image which isan undesirable property that warps theimage point okay what do we do with thatwellone thing we can do with that istry and figure out where the center ofprojection is so that's where we'regoing last timeso we have a coordinate systemthat'sin theimagingplane in the image deviceand we're trying to find therelationship betweenthe coordinate system in the object andthe coordinate system in the image planeand then we're trying to find outwhere the center of projection is soa couple of uh terms so one thing isthe point that is perpendicularbelowuh the center of rejection so we we dropa perpendicular from the center ofprojectionuh down here and that's called theprincipal pointandas we indicated you'd like that to be atthe center of the array but it won't bevery accurately and to do accurate workyou need to know exactly where it isso that's two numbers uh row and columnuh in the image plane and then the thirdnumber is the height of the center ofprojection uh above and we'll call thatf and that is toremind us that in the lens system that'ssort of the focal length it's typicallyslightly larger than the focal lengthanyway so they're three degrees offreedom the three numbers we needand one way to think about it compactlyis we're trying to find out where thatpoint isand um you know that could bea difficult task to do by physicallydisassembling the camerafor a startina cell phone camera those distances arevery small so you'd have to measure themvery accurately anditprobably won't be the same after youreassemble the cameraso what how do we do this well[Music]if we connect the center of projectiontothe vanishing points in the image planewe have three vectorswhich arebasically these three vectors up herewell you know itry and draw it that way maybeand so therefore they're at right anglesto each otherand where's that come from well ourassumption is it's a rectangular objectif it was some other object and we knewwhat the angles were we could use thatas well it'd be slightly less convenientbutokay sowe now know that we're looking for apoint up heresuch that if you stand there and youlook down into the image planethe directions to these vanishing pointswill be uh right angles to each otherso that's that's the task we we movearound in this spaceto find that uh that place and so umlet's start start in 2d we alreadymentioned this uh last timethat umthe angle made by the diameter of acircle from points on its circumferenceis a right angle so converselythe locus of all the places you could befrom which those will be at right anglesto each other is a circlethis diagram will come up uh later whenwe're talking aboutphotogrammetry andambiguity in imaging of surface terrainbecause you know if you have twolandmarksand they appear at right angles to eachotheryou might think that that would tell youwhere you are in the airplane butactually no because you could beanywhere on this circleand you would see them at right anglesto each other okay so that's the 2dversionand the 3d version of course isyou know you just spin this around itsaxis and you get a spheresoso thatconstraint on the positionsuh on the position of thecenter of projection is just that itlies on a sphere so so what's fear wellfor start it lies on the sphere wherer1 is one end of the diameterand r2 is the other end of the diameterso you can imagine that wedraw a sphere withr1 and r2as the diameter and so it goes above andbelow the image planeand the center projection must lie onthatnow of course that's not enough to tellus where it isso we have a second spherewe connect say r2 to r3and that's the diameter of a secondsphere and now we intersect those twospheresso what's the intersection of twospheresit's a ringright sonow we have a ring that means we haven'treally solved the problem quite becausewe still haveinfinite number of possibilities so weuse a thirdwe use uh oursphere with diameterthere r3 and r1 and now we intersectthose and we're left with how manysolutionstwo right okay and uh so there's a tworemaining two-way ambiguity and if inthis simple case it's simply thatwhen we're up herewe get the sameright angle condition than if we'remirror image below the image plane wellin our case we know thatthere's a physical constraint which isthat the center projection has to beabove theimage sensor so that it's imaged so thesecond solution uh can can be eliminatedokay so umi want to just talk a little bit aboutyou know some of this is allvery simple and basic but it's good toremind ourselves of these things sowe'll start off uhtalking about linear equationsand you know whenever we can we'll tryand reduce things to linear equationsbecause we know how to solve thoseand so geometrically you know uh whatare they well they're straight linesand what sort of equations do theycorrespond towell there's there's this old chestnutwhichhas some real problems but that's oneway of writing the equation for astraight linethenwe canjust say well it's a linear equationlike thatand in this caseso here we've got two parameters m and cand what is that that tells us that thefamily of straight lines is a twoparameter familyif we look at this they're threeparametershow can that be well because there's ascaling we can dosotrue they're three numbers but actuallyif you divide through by theby any one of themyou get the same line and you're down totwo two degrees of freedom so it's a twodegree of freedom worldand then we can goprobably get the signs wrong butummake sure i got them the right wayaroundsine theta x minus okay so what's thatwellthat's another way of parameterizing astraight linewhich we'll use quite a bitand you can check that you know for sinetheta equals 0this says that y equals rhoso theta equals zero is is that line andfor theta equals pi over twoum we get x is minus rho so that's thisline and so onand umtwo parameters so here we gotthe world of lines is parameterizedthrough uhtheta and rho instead of over here whereit's parameterized in terms of m and cand why is that usefulwell the thing up there is that if yourline happens tobe parallel to the y axis then m isinfinite so there's a singularity whichwe avoiduh in this in this representation anywaystraight lines in 2dare equal linear equations in x and ybut now we're dealing with 3d so let'stalk about thatand so now we're talking about uh planesand of course planes are alsorepresented by linear equations exceptnow in the in three unknownsand umone reason i introduced this notationis because we can generalize that to 3dokay sowe can write the equation of the planeasas thatso it's a linear equation i mean if iexpand out this uhdot product herei get[Music]you know linear equations in x y and zand it's just convenient to write itthat way so butif i wanted toi can write thata b c d so that makes it look likethey'refour degrees of freedom but of coursethey aren't because of the same scalething if i multiply abc and d by two ihave the same plane so actually it'sthree degrees of freedomand i can also see that from thisrepresentation because n is a vector sothree numbers but it's a unit vector sothere's one constraint so they're onlytwo degrees of freedom and then i haverho so 2 plus 1 is 3. so the family ofplanes in the3d world arethree-dimensional which is aninteresting duality because it meansthat there's a mapping between planesand points in 3djust as there's a mapping betweenlines andpoints in2d case so i can either plot theta rhoor mcoh and by the way if ilike that representation thenyou can now see the similaritywhere in the 2d caseright this this vector herehas an x component that that increasesas xoh it becomes negative larger as x thetaincreases so that's the minus sine thetatwo and the y component is large whentheta is zero and then gets smaller andthat's the cosine component so you'llsee that this this equationis this same as uhn dot r is rho or somethingand of course that's the same equationwe have over there just in in 3d okay sothat's all all pretty obviousnow back to our camera calibrationproblemso one way uh to approach this is tothink of this as the intersection of uhthree spheresand so let's talk about that a littlebitsoi mentioned last time this problem ofmultilaterationwhich comes up in roboticswhere for examplewe have um the distances to a number ofwi-fi access pointsand let's say we have threeand you think that that should allow youto compute where you are and it it doesso let'suh solve that problem first sowe'retrying to intersect three spheresso how can wetalk about a sphereso um here's a sphere the i f sphere wewill have three of them so rather thanwrite three equations i'll write thatand it's just that the magnitude of thatvector difference is the radius of thatsphere and sowe'll get three equations like this andtry and combine them now if i want towrite this outi can write thisright because the definition of thatmagnitude is just the square root of thedot product of those two vectors okaythen i can multiply this outto get that equationand so for every sphere i'm going to geta second order equation like thisand we mentioned last time thattherefore by bazoot's theoremwe may have as many as eight solutionsunless the equations have specialstructure sowellwe can exploit that byconsideringa second sphereright same equation justj instead of iand by subtracting them we can get ridof that annoying second order termand so we end up with a linear equationand that's always preferable so we get2r dotrj minus rium then i need to say what that is sori squared isokay so these things on the right handside are just constantsthey're thedistances i measured andthedistance from the origin of the tworeference points of the centers of thosespheresbut and what's important is that on theleft hand side i've got r dot somethingwell that's a linear equation in thecomponents of rand you know any time i can reducequadratics to linears i'm i'm happy sookay now we've done it for uh one pairofspheresbut we actually have moreso we can repeat this exercisefor the other combinationsso when i put it all togetherokay and then i have you know all ofthese constant termsright uh becauseumyou know the the transpose says i'vetaken a column vector and turned it intoa row vector so the first row of thematrix is is this difference r2 minus r1and multiplying this matrix by thisvectoris taking the dot product of thisdifference and that vector so that justcorresponds to what i've got over herewell there's a factor of 2 i forgot afactor of 2. let'sdo thatokay and then similarlythe second row i've taken a columnvector turn it into a row vector and sothe second term in the result ofmultiplying this matrix by this vectoris to take the dot product of thisvector with with that vector so that'syou know the same equation just with iand j uh changed and i do that a thirdtime and i get that and hurrah i got youknow three linear equations and threeunknowns what could be better umokay there's some people shaking theirheads sowhy is that wonderful if it's true wellbecause i know how to solve linearequations and there's only one answer etcetera et cetera but wait we said therewere two answers so something's wrongso what's wrong with this whatwhy will this failso when do linear equationsnot have a unique solutionwhen there's redundancy when the matrixis the rows in the matrix are notindependent when the matrix is singularwhen the determinant is zero you knowall different ways of saying the samething now how can i be sure that's aproblem well if i just add up thesethreerows what do i geti i get zero right so the so the threerows are indeed not linearly independentum the third row doesn't tell meanything new because i could have got itfrom the first twoyou know just by if i addor subtractif i take this one and subtract that onei get this one and the same with theright hand side so so it's allconsistent so yes uh this this statementis true but it's not giving me asolution becausethis is a singular matrixso we always have to check that umnot only do we have enough equations andunknowns butthat we can actually solve the problemokayyeah right okay so his statement wasthatwe think there should be two solutionsbut if the matrix is singular there'sactually an infinite number of solutionsi can constructalong a lineuh any number of solutions i likeso so what what happened what's going onwell what we did was we manipulated theequations and we got some more equationsbut we threw away the original equationsand and you know that may or may not uhbe legitimate in this case uh we've lostuh we don'tsomething that satisfies these threeequations does not necessarily satisfyuh these three second second orderequationso satisfying this equation you knowdoesn't uh guarantee that we actuallyhave a solution so sothis is another thing another cautionarytaleand you'll see this sometimes in papersgreat we manipulate the equations we getsome equations we can solveand then ohactually we're not only getting thesolutions we're supposed to get butwe're getting other stuff and so in thiscaseit's perfectly legitimate to derivethese equations but you can't then throwaway the original equationsand in particular in this case we canuse two of thembut we need to keep one of the quadraticones right sowe can keep for example the first twoequations over there and keep the thirdone of theseand that'sperfectly legitimate now we've got twolinear equations one quadratic and bybazoot's theorem we would get one timesone times twosolution maximumie two and and that fits uh with whatwhat we're expectingokayuh so let's uh deal with this in anotherway so what we've got isi you know constraints uh like thisand uh we can you know write them outthe way we did over there and thensubtract them and what we're going toend up with isand we can derive you know two moreequations like this but let's just uhstop and look at this for a second whatwas this r2okay so all i've done here is i'vesubtracted these two and and reorganizedthe terms a little bit so what does thistell me well dot product beingequal to zero means that the two vectorsare perpendicularso that's one important thing thatr minus r two is perpendicular to rthree minus r1the other thing we can notice is thattheplanegoes through r2so first of all this is a linearequation so by our discussion over thereit represents a plane in 3dand passesthroughr2right because ifr equals r2 this is zero and so thatsatisfies this equationokay so we got two important propertiesand now if we go back to ourvanishing points in the image planei don't know do it again soum r oneokay so what this is saying is that um[Music]this particular equationis a planethat is perpendicular to r3 minus r1 sor3 minus r1 that that's this vectorso this planeit has this as a normal perpendicularand so for a start it tells you thatthat planeis perpendicular to the image plane sothe solutions are on that plane that'sperpendicular to the image plane and butwhich of these planes is it because justsaying that this is the normalperpendicular to the plane you know wecould have lots and lots of planes wellthat's the second statement it passesthrough r2 soso um you know we're popping back andforth between algebra and geometrybecauseyou can do all this just algebraicallybut you don't really get much insightand it's much more fun to look at itgeometrically now of course i pickedthis particular combination i could picktwo other combinations so what do i getfrom those wellone of them is going to give me a planethat is perpendicular to r3 minus r2and passes through r1 so that's that oneand then there's a third one which isgoing to be a plane perpendicular to r1minus r2 and passing throughr3 and sotadayou know i got a solutionso what what is that called by the waytriangulation okayi was well sowith triangles there are a lot ofspecial pointsthat have names and i don't know atleast six i'm sure people have come upwith other ones for example there's theuhthe out center which is the center of uha circumscribed circlethen there's the in center which is thecenter of a inscribed circlethen there's the centroid which is theaverage of the three sets of coordinatesand this one is the centeri don't know there are a few moreintersections of bisectors and whateveranyway this is the one we wantandit's obtainable easily by solving linearequationsso so we're partly donewe're not quite done because now we knowuh where it is in the image planeso this is the principal pointthe one we were talking aboutum that that uh you know theperpendicular we dropped from the centerof projection into the image planethat's that's where that isuh what are we still missing well we'remissing fso so we knowthat the solution is along a linethat's coming perpendicular out of theimage plane uh why well because we'reintersecting these planes that are eachindividually perpendicular to the imageplane so they'll produce a lineand by the way of course i don't needall three planes i just need tointersect two yeahokaylet's start with r so so the unknowncenter of projection is is out hereand um this is theperpendicular i dropped from from r intothe image planeandthe equations i'm solving are theseequations and the thing i'm exploitingis thatall of these have a zero z componentuh why is that well because in terms ofthein terms of the coordinate system i'musing which is row and column in theimagethe theheight is zero in in the image plane thecenter projections out here at somenon-zero z but the these points areactually in the image plane uh which hasheight zero relative to the image planesonone of these vectors have a thirdcomponent and so all of these equationshere i can think of really as equationsintwo vectors not just x and yand i only need two of those to solvefor x and ybut you knowthat's just algebra that represents thisuh geometric insight so one of thethings that what do you do with thiswellcamera calibration but you can also dosome fun fun things with it otherwisefor so for exampleif you um take an image and youfind this pointand you find that it'ssort of not in the middle of the imageyou know like uhyou uhuh here's here's your pictureand you do this construction and youfind the vanishing points are i don'tknowoutside the image but yeah and then youdiscover that oh the center ofprojection is i don't know hereuh the principle pointwell then whoever took this pictureeither had a very funny cameraorum they cropped you out of the pictureyou know souhvery commonly done when uh relationshipsbreak up uh you take this great picturewhere you look really good and you cutout the other person and and so that'sone of thekind of light-hearted or maybe not solight-hearted in that caseuses for this technologyanother one isto try andquestion whether an image is an originalor has been modifiedand and this came up inyou knowdid admiral piri get to the north poleor notand his proof was partly in the form ofphotographs andwhat you can do and we know what camerahe used so we know the focal length etcand well we also knowwhat altitude the sun should be thattime of year so you can do somephotogrammetry and uh you know discoverthat well for example one of theimportant pictures has been croppedsothis is the kind of technology that willlet you let you do that and i guess[Music]the photogram photogrammetricassociation which is into this sort ofstuffpicked that up and they published a bookthat kind of questionshisclaim to have reached thenorth polei'm interested in that partly becausethey're wonderful hoaxes in explorationa lot of which areyou know easy tounderstand you know someone spent yearsraising moneyyears finding people to work with themthey get within 150 miles of the northpoleand there's no one around are they goingto come back and say no i didn't make itwell if they're really really honestthat's what they say but if they sentback the only other guy that knew how tooperate a sextant to measure thealtitude of the sunthey might just be tempted to say yeah igot there wellanyway so uhdoing this kind of vanishing pointanalysis can sometimes alert you to[Music]problems withimage manipulationbut we're using it forcalibration insteadokay soyou know we can write down the twolinear equations for that point but it'suh not i mean you know how to solve twolinear equationsum so we still have to find f but that'sokay because we nowuh know x and yand all we need to know is the thirdcomponent of that vector and we end upwith a quadratic and we know how tosolve a quadratic so so i won't do thatbut here's another interesting thing ikept on saying that in typical cases thevanishing points will be outside theframeand and so one thing we might want to doisgiven thispositioning of the vanishing pointcan we say something very quickly aboutthe uh the f the focal lengthwell here's alet's take a really simple caseso here the vanishing points and theyhappen to beequally spaced in the image plane wejust turned the cubeso thatwe're looking at the three faces equallyand let's suppose that this distance isi don't know uh v for vanishing pointand then the question isuh what's if now you knowthe generalapproach to this involvesplugging in the x and y we get from thisand then solving a quadratic in fbutmaybe we can do this withoutall of that because it's a special casewe should be able to figure this outokay so uhwe can think of this in terms of acorner of a coordinate systemokay sowe haveone zero zero zero one zero and zerozero oneand umweneed to knowthe distance from the origin to thisplanewhat and of course it'll vary dependingon where we are but somewhere out hereis the point that is closest to theorigin and it's the one we indicated bydistance rho from the originso the question is you know how far isthatpoint from the originwellpresumably it's a point wherethat'swhat's another variable name aaawhereit's symmetric so you would imagine thatthat point should have the same x y andz coordinatesright and soumthedot product of that with the uh unitnormal to this planeshould should be oneright so the the unit normal to theperpendicular to this plane comessymmetrically straight out equally in xy and z and to make it a unit vector wehave to divide by the square root ofthree right because we've got one oneoneand so so okay so i think this is that ais one of the square root of threebecause it'sthree times yeah okaysoum[Music]this is also what we called rho beforeokay so that's the distance from theorigin to this point and then that'sgoing to beour f but what is v well in this diagramthis is v so so in this case v is squareroot of 2and f is 1 over square root of 4.sothere's a relationship which is that vissquare root of 6 times for f is v of a square root of 6.soso in this special case we can easilycalculate what the focal length or theprincipal distanceisand it's substantially largerthansorry i got this the wrong way around fshould beno it is rightand so uh v will typically besubstantially larger than f theprincipal distance and so often the uhvanishing points will be outside the theframe that we've actually captured animage ofandyou know this is a special case we cansolve it in a general case it's justjust algebra sookay uh so that'suhapplication of vanishing points tocamera calibration now i want to talkabout another applicationyou know i mentioned the case wherewe just you know slap a cell phonecamera onto a car's window and we wantto relate the images we're seeingtosome sort of three-dimensionalworld coordinate systemjust by identifying features in theimage so what sorts of things can we seein the image well if it's on a straightroad we'll seeyou knowthe curb and we'll see road markingsandthose are supposedly parallel and sothey will produce a vanishing point thatwe can detect in the imageif we're lucky there's also a horizon sowe get a second constraint out of thatand what we want to know isyou know how is this camera orientedrelative to the road and relative togravityso that's the kind of problemwe're trying to solveandi guess we got rid of that also sothis is really about orientationso the transformation between a worldcoordinate system that's lined up with aroad and gravityuh to the camera coordinate system istranslation there's some shift androtation and we're going to focus hereon just recovering the rotationsowhere do we start well same diagramyou know let's suppose we're lucky andwe actually have all three vanishingpointsin the application i mentioned we don'tbutlet's take the easy case wherewe have all three vanishing pointsokay so we got v1 v23 or r1r2 r3and um we now now that we have a camcalibrated camera we just connect thoseup to the center of projection we nowknow where the center of projection isthat is we've got the three numberswe've got two for the principal pointand one for the principal distanceor if you like just the coordinates ofthe center of projection okay i don'tknow let's call it p or something we'vecalled it r over hereokaythenwe know that theedges of this rectangular object thatwe're looking at to get the vanishingpoints have directions that are justdefined by these linesthat isyou know let's call this the x-axisand then there's another onewhich will be the y-axisandhere we go so they look a little bitfunny because of the way i picked thevanishing points but they're supposed tobe at right angles to each other and ofcourse the first thing i can do is checkyou know i've i've got some algorithmthat finds the vanishing pointsi've calibrated the camera supposedly iconnect these up and i take their dotproducts and they better be small rightthey're unlikely to be exactly zero butso that's the first thing to check thatthey are in fact at right angles to eachotherand theni have nowwhat do i have i have the unit vectorsin theobject coordinate systemuh measured in the camera coordinatesystem right so so my definition of x yand z is still in this coordinate systemover hereso umi'll do thist minus r1and of courseyou know once i know thatthey're paralleland their unit vectors i can justcompute them by normalizingetcokay soimportant to understand that thosex y and zunit vectors are in the cameracoordinate systemokay so nowsuppose i have some point in the objectandlet's see what i call these primesand it's going to beyeah they're vectors okayso umso that's um inin mycamera coordinate systemandwhatis that vector in the original objectcoordinate system so everything now ismeasured in thecamera coordinate system and oh ishouldn't beokayso[Music]so it's alpha in the direction of thex-axisand beta in the direction of the y-axisand gamma in the direction of the z-axisright so what arewhat is the vector in the objectcoordinate systemwhat are its componentsits x component isalphaand y component is betaso this vector here that i've written inthe camera coordinate system correspondsto this vectorin the uh object coordinate system rightso i mean that's the definitionwe have three axesum in thatworldand we express the position of a pointin terms of a weighted sum of thosethree directionsand so in so in my camera coordinatesystem this is what it looks like in theobject the rectangular block coordinatesystem it looks like thatokay so what's the transformationum so i got let's see r isso let's seethe t again means i'm transposing acolumn vector into a row vector so thefirst row of that matrix isyou know theunit vector in the x direction laid outas a row and so the first the firstcomponent of r over here is the dotproduct of thisxunit vector and r r prime uh this guyand andwell that's this you know that's whatyou see in that equation the secondcomponent is the dot product of y withthis guy and so onso umthat's my transformation vector betweenthe two coordinate systemsand we will be doing more of thissodon'tdon't panicso that's a very important uh matrixand it represents the orientation of onecoordinate system relative to the otherand[Music]i said i claim this matrix is alsonormal uh what does that mean that meansthat the rows are perpendicular to eachother if you take the dot product of thetwo rows you'll get zerotake the product of those two rowsyou'll get zeroand by construction that's true becauseyou know we made a unit vector soand then um no sorry we uh we'reassuming that they represent the axes ofthe coordinate system so they'reperpendicular and then they're also it'salso normal that each row is hasmagnitude one and we did that byconstruction because we constructed aunit vectorso umamongst other things we're now led tounderstand that rotation is representedby orthonormal matrixandquite a few photogrammetric tasksinvolving finding that matrix and inthis case we were able to do it ratherstraightforwardly because we explicitlycould determinethe direction of the coordinate axes inthe objectwritten out in terms of the coordinatesof the camera sookay and of course if we want to we caninvert thisso if we've gotyou know maybe we need to go the otherdirectionmaybe we know uh the coordinates are prarethe coordinate r and we want to find thecoordinate r primewelland it turns out that for rotationalmatriceswe can show that theuh they're actually just thetranspose sothat follows from the property oforthonormality so in this case goingback and forth between the twocoordinate systems is particularly easyokayon to something else finally soso we spent quite a bit of time talkingaboutwhereperspective projection and all of thestuff that's connected with itincluding the derivatives which gave usthe motion field and then we talked alittle bit aboutvanishing points and exploiting them forcamera calibrationand let's go back now to the other partof the puzzle which is brightnesssoumand what we can do with it how how canwe exploit measurements of brightnessso umwealready talked about foreshortening soandand umso here's asmallfacet of the surface of an objectand we can specify its orientation bytalking about theunit normaland then here's an observerand let's call this direction theviewing directionand there's also illuminationand in the case there's just you knowonesource of illumination and we'll callthats for sourceand well we can draw someangles here so this is theta i forincident angle and this is theta e foremitted angleand well that's not enough we need weneed another angleto fully describe that situationright because if weif we just specify the incident and theemergent angle we can spin one of thesearound so we need some sort of azimuthangleand we'll talk about this later but forthe moment just imagine we take this rayuh from the light source and we projectit down into the plane and we take thisray to the viewer and we project it downinto the plane and then we measure thisazimuth anglenowtheobserved brightness is going to dependon those parametersit's going to depend on the material ofthe surfaceone and it's going to depend on you knowthe light sourceand it's going to depend on those anglesyou know for the extreme cases a mirrorwhere there's only one direction whereyou see anythingyou know the the reflected ray goes offin a particular direction and unlessyour camera or your eye happens to bethereit's it'sdarkbut the more interesting cases were youknow we have likea piece of yellow note paperand pretty much any direction i look atit from it has the same brightnessokayand we'll talk about that more laterwe're going to greatly simplify thisstory right awayby talking about the illuminationand we've talked about foreshorteningand so we know thatif we have a patch of a certain areait's going to get less and lesspower from the light source the moreit's tilted relative to the light sourcedirection andthe foreshortening makes the apparentareabe the true area times the cosine ofthat angle and so just in terms of powergetting inthat'sthe magic number wellif cosine theta i is less than 0that's not true because that would meanyou get negative power so what doescosine theta i being negative meanthat means it's greater than[Music]the angle is greater than pi over twoand so that means thatyou've turned the surface to face awayfrom the sunright sowe should really be saying you know maxof cosine theta and zero but that's sotedious we're just going toimplicitlysay that okayand thenthe surface may or may not reflect thelightit may absorb some of itandand it may reflect differently indifferent directions but let's take areally simple casestart off with that not to say thatwe're going to be stuck with thatbut um[Music]okay so this is a model of amatte surfacethat's very unmirror-like it reflectslightin various directions and it has thespecial property thatno matter what direction you look at itit has the same brightness and we'lltalk about exactly how we measurebrightness but so for the moment allwe're going to make use of isyou knowlet's imagine there's a surface wherethe brightness only depends on how muchpower is going in and therefore itdepends on cosine theta iand so how can we how can we exploitthatwell we make a brightness measurementandit's going to be proportional to thatlet'snot worry about the proportionalityfactor for the moment so the brightnessis proportional to that dot product inin this diagram between thenormal and the right because n dot s iscosine theta iand so does that allow me to determinethe surface orientation socan i solve this for uhsee where i'm going with this is i liketo recover the shape of the surfaceand one way i can do that is to look atevery little facetand figure out its surface orientationwhich is a little different fromyou know you might think that well we'lljust construct a depth map we'll findsome way of estimating the depth atevery point in the image butthat's something we can't dofrom a monocular image we'd like to beable to recover shape from monocularimages as wellsowe're trying to recover ncan we do it from this well this is justone constraintand the surface normalhas two degrees of freedom right it's aunit vector with one constraint so threeminus one is two so we got two unknownsand we only have one equation so thatdoesn't workbut let'simagine that you know we're in anindustrial situation we have control oflight sources and so on and we cantake a second imagewithwith a secondposition for the light source and youknow you saw that in the slides that wedid at the beginningwell that's already looking betterbecause now we have two constraints andand twouh degrees of freedom we have a matchbetween the number of unknowns and thenumber of equationsso um but uh they're not linearand so umwhere's the non-linearity come fromwell the thing is thatif we're solving for n we need toenforce uh the constraint that it's aunit vectorright so so in one way of thinking aboutit is we're trying to solve these threeequationsfor nandthese are nice and linear and this oneisn't sothere may beby bizu's theorem there may be twosolutions another way to see that issuppose that you measure brightness andtherefore you can estimate cosine thetai and therefore you can estimate theta iuh what what do we knowwell let's suppose we know the directionto the light source and you know this isthe direction to the light source andthen i know that there's a certain anglebetween the surface normal and the lightsource well that doesn't give me theanswer because it could be you know ican rotate this around sand soi actually have a whole cone of possibledirectionsso this is sand this is theta iand my normalcould be any one of theseon thaton that conenow if i have a second measurementsecond light sourcethat gives me a different cone ofdirectionsand ifthe normal has to be on both of themwell that means that i look for theintersectionof those conesand there will betwoand of course if i intersect two cones iget two linesbut i have the additional constraintthat it has to be unit normal so it's ona unit sphere so i then take those twolines and intersect them with a unitsphere and i get two points okaywell there could be bad things happeninglike if your measurements are wrongthese two cones may not even intersectso but we'll we'll ignore that for themoment okay and you know we could writeout the algebra for thatinvolves solving a quadraticknow how to solve quadratics sobut we can actually turn this into alinear equation problemand make it somewhat more interestingi imagine thatamongst other things one of thethings that will affect how brightsomething is in the image planeisthe reflectance of the surface andreflectance unfortunately is a veryfuzzy term that means something else foreveryone so i'm going to call talk aboutsomething called albedoand so that's a quantitythat would be betweenzero and oneand it's simplytelling you howreflective the surface is how much ofthe energy going in comes out againversus how much is absorbed and lostand i put albedo in quotation marksbecauseit has a very well-defined meaning insometechnical area such as astronomyuh would mean something slightlydifferent in the case of astronomy itmeans you've got a spherical planetand what's the ratio of power in overpower out and so that means it's somesort of average of lots of differentdirectionshere i'm talking about just a particularorientation but it's a very simpleconceptjust you knowa piece of white paper presumably has analbedo that's close to oneand umyou knowblack coal has an albedo that's like4157.92 andcan you have aalbedo greater than oneno right otherwise you'd be violatingsome some law of physics but you can getsuper luminous surfaces by cheating soif you have a fluorescent spray paintfor exampleor if you start the collar in your whiteshirt which i know you all dothen when you illuminate it in withsunlightthey are brighter than bright theyshouldn't be as bright as they are andthat's because they're convertingultraviolet into visible so they'rewhile you know they're not violating uhsecond law of thermodynamics they're uhconverting energy outside the spectruminto energy in the spectrum and in thatcase the power total visible power outcan be larger than the total visiblepower in and so a surface with that kindof property you know if you put starchinyour white shirt they will appearbrighter than a99.99 reflectivemagnesium sulfate powder which is one ofthe standards that people useokay but generally speaking you knowzero less than rho less than oneand so what what does that do well nowwe have a slightly different situationwhere e1 is rho and dot s1and e2 is rho n dot s2andso now actually i also havethree degrees of freedomin terms of unknownsright because i've gotthe unit vector that's twoand i want to recover the albedo thatthat makes it threewell that means i can't do it with twoequationsso let me add a third oneokayso that seems to be a match i have threedegrees three unknowns and i have threemeasurementsand now i'm going to[Music]define n to be this quantityso i'm going to define a three vectorwhere all three components are actuallyindependent are actually variable and umthat vector will encapsulate thethings that i want to know rho and uhand unit vector n and obviously i canrecoververy easily you know once if i can findthis ni can easily find rho because it's justthe magnitude of n and i can easily findthe unit vectorby just dividingsoand the reason i do it is because thisis going to be more convenient so then ihavetransposevery similar to what we had earliertoday except this time it's going toworkso so what's going on here wellthe firstresult of multiplying this matrix bythis vector is the dot product of thefirst row with this column vectorand so that's uh that's this that's e1and so on so so this is just a compactway of writing those three equationsandi can then writei can write the solution that way andi'm done and and from that n i can thenrecover rho and the unit vector ofsurface normalsonumber of things to discuss about thatone of them is that this is assumingthats is invertibleum i can easily construct cases wherethat's not the case for example if s3 isactually just the same as s2right then the determinant of thismatrix two of the rows of the matrix arethe samethe determinant will be zero i can'tinvert itand it makes sense i'm not getting newinformation right if if s3 is if mythird light source position is same asthe second light source position imeasure the same brightness so it'sintuitively clear that that's not goingto workand then you can think of other thingslike suppose that s3 is half of s1 plushalf of s2 that's a little differentbecause it's sayingokay there's one light source herethere's one light source here and nowi'm going to put one right in the middlebetween themthat gives me a third measurement in theimagewell it turns out that you can predictit from the other twobecauseyou knowif e3 is n dot s3 and s3 is a half s1plus half of s2 you can calculate whatthis isand so so they can't be coplanar so thelight sources have to be spread out inyour firmament you can't put them in ina planeand this hasimportant implications for astronomersbecauseyou knowthe orbits of the planets and our moonare pretty much in the same plane andthereforeas theas the sun orbits around the earthyou don't get different picturesokay um so that's one thingthis isn't going to work unless we pickthree uh independent light sourcedirectionsanother thing is that we can pre-computethis right if we know where the lightsources are say you're doing someindustrial inspection you control wherethe light sources areyouyou just compute s take the inverse andyou store it and then at every pixelyou have three frames taken threeexposures and at every pixel you simplyform this vector and multiply thepre-computed s minus one and there'syour answerit means it'sincredibly simple and you know no compvery little computationuh very efficientthenanother thing is thatit's sort of fortuitous that we needthreeto do this we need three light sourcesto do this and one reason that's sort ofinteresting is becausecamerastypically havethreesets of sensorsrgband so we might be able to exploit thatso one one thing we could do is insteadof having three light sourcesthat come on sequentiallywe could usethree colored light sourcesand then separate outfrom r g and b and there's a little bitmore work to do because a particularcolor light source is going to not justexcite r or g or b but some linearcombination and but the important thingis we'll have three different linearcombinations and then we canuse our magic matrix algebra to to dealwith that so that's a possibility andthat is in a way more convenient andfaster because we don't need to turnlight sources on and offalthough turning leds on and off ispretty fastbut we can just illuminate them withcolored lights the only problem withthat is if the object is colored becausethen the objects you know unless it'suniformly colored if different parts ofit have different colors that's going toconfuse this algorithm and make itbelieve that the surface orientation issomething that it's notokay umsoso that's sort of one quick look atone quick look at what we're going to doin terms of measuring brightnessthis is a particularly simple case andit's a case that's uh in a way contrivedbecause we're saying that we'recontrolling the light sources which youknow has application to industrial usein fact if you look at some recentpatents from cognix they'vedecided to use something like thisbut it doesn'treally mesh with understandingbiological vision systemsexcept in the very very deep ocean whereangel angle fish havelight sources that they dangle in frontof themto attract preywe don't typically find animalsilluminating the environment withdifferent colored light sources in orderto figure out the shapes of their preyor something like that so but from thepoint of view ofrecoveringinformation from monocular images in anindustrial settingthis is an interesting approach nowdo real surfacesfollowthis simple rulecosine theta i uh noso you can't use this directlybut you can build a lookup tableso in fact you don't even have to modelmathematically how the surface reflectslight you just calibrate it using ashape that you knowso for everyuh suppose you take a spherefor every point on the image of thesphere you can calculate what thesurface orientation is you know what nis thereand then you measure brightness in threeimagesand you can build a table and of coursethe table is going the wrong way thetable's going from surface orientationto e1 e2and e3 what you really want is theinverseyou know measuring e1 e2 e3 you want toknow what the orientation is but youknow that's justnumerical inversion of a table sothat's something we can dobut we'llwe'll go in different directions nexttimestarting off with a different projectionso we know that real cameras performperspective projection and we spent alot of time dealing with thatbut in some cases we can approximate itby a projection that's much easier tohandle called the orthographicprojection and the the condition forthat is that thethe range in depth is very smallcompared to the depth itself and in thatcase you can kind of assume that thedepth is constant and if z is constantthen f over z is constantand there's a constant magnification andin that casewe can use a simplification calledorthographic projection which we'llexploitin in our efforts to reconstructsurfaces from imagesany questionsokay

Photometric Stereo, Noise Gain, Error Amplification, Eigenvalues and Eigenvectors Review

talk some about noise gainandin the 1d casewhere we have one unknown and onemeasurement it was pretty simple becausewe just looked atthe inverse of the slope so we looked atso we had some small increment here andthe relationship with the smallincrement over here so xin this case is the unknown quantity andy is the measurement we're making andwe're trying to estimate x and obviouslyif this curve is very uh low slopethen a small error here canbe amplified into a large area there sothat's verysimple in the 1d case but of coursewe'll be lookingat more complicated cases where we'retrying to estimate uh three-dimensionalquantities and soi want to take a little detour now andtalk about at least the the 2d casewhich comes upin the optical mouse andsome other things that we'll be talkingabout pretty soonso we're going to talk a little bitaboutum eigenvectors and eigenvalues and ifyou are familiar with them thenjust go to sleep for the next 10 minutesbut i think it's important to geteveryone up to speed on this so so whatare these well if we take a matrix mand we multiplysome vectorwe're going to get a new vector which isa different size typically and pointingin a different directionand sothe interesting question isdo we ever have a situation wherethe vector we get is pointing in thesame direction as the vector that wemultiply the matrix withand sothose are considered special and in somesense they're characteristic of thatmatrix and so sometimes they're calledcharacteristic vectors withcharacteristic valuesand i guess eigen is the german word forfor ownit's the vectors and values that thematrix owns and so we're looking at asituation wherethe vector we get is parallel to thevector that we pump into the matrixand it may be a different sizesothat's the definition and we're mostlygoing to focus on real symmetricmatrices we can generalize this but forthe moment that's all we needand so obviously this isn't going to betrue for typical vectors also noticethat the the scale doesn't matterbecauseif e is an eigenvector and i multiply itby kthis equation still holds it's still aneigenvector so for many purposes we'lleitherjust ignore the magnitude or make it aunit vector for simplicityso the size doesn't matter and by theway that includes multiplication byminus 1.so if b is an eigenvector then minus vis also an eigenvectorokay so then the question is how do wefind these things and how many are thereandthings of that nature well one thing wecan do iswe can try and solve this equationbybringing this over to the other side soi is just the identity matrix of theappropriate sizeum you know m is n by n then i will be nby nand sothis is exactly the same equation mtimes e minus lambda times e is 0.okay so there's aset of linear equationsandyou know we've been bragging about howlinear equations are so easyand obviously what we can say isis that you know we just invert thatmatrixand we multiply byand and what do we get uh zero right soso that's called the trivial solutionandumif if we can do this thenthat's not going to be helpful so whatwhat is it we're looking for well we'relooking for the case where actually thisinverse doesn't existso usually we're very keen to make surethat inverses exist in this case uhwe're not so first of allterminologyhere we have a set of equations that arehomogeneous that means thatthe right-hand side is zeroand amongst other things that means thatif you have a solution then any multipleof that solution is also a solutionwhich is not the case normally whenwe're solving uh linear equations youknow there's aunique solution and any multiple of itis not a solution soanynonnon one multipleokay so uh what we're looking for thenisthatwe have a singularmatrix that means that the determinantof that matrix is uh zeroand so umwhat what does that mean wellunfortunately the determinantis this complicated messy thingbut if you think about itwe can saysomething about the determinantbut if we write this thing outi don't know 3minus lambdawhat we've done is we've taken thematrix m which is somen by n real symmetric matrix and wesubtracted lambda off the diagonaland now we'retalking about the properties of thismatrix and we we actually want this oneto be singular because otherwise thatequation has nothing but the uhobviousuh solutionso what's the determinant of this well idon't know you might remember that thedeterminantinvolves taking this and multiplying itby the determinant of that matrixplusthis times the determinant of some othersub matrixor you may may remember it as you knowtake one from column a one from column band one from column c and depending onwhether the number of switches indirection is even or odd you add a minussignuh don't need to remember the exactformula the key thing is that we'retaking something from here and somethingfrom there and so onnever repeating columns or rowsand there you know large numbers of waysof doing that which is why it's actuallycomputationally expensivebut the key thing is that we can get aterm as large asthis one along the diagonal in terms oflambda so if we look at the product ofall of the terms on the diagonal that'llbe part of the determinant one of manypartsand it's gotlambda to the n in it right we're takingm one one minus lambda m two two minuslambda and so on so so what's thedeterminant well it's a polynomialinin lambdaand it's an nth order polynomial inlambda and polynomials of nth order havehow many rootsngood so so we're going to have n rootsand so that means thatnot only is there a solution to this butthey're going to be uh n solutionsand umthose are the things that that we'll beuh we'll be looking forso uh just to make this concrete let'slook at avery simple examplealso because uh that's the one thatwe'redealing with right nowwith say the optical mousesowe already mentioned that just sayingthat the solution is unstable that thenoise gain is highisn't enough when we're dealing withmulti-dimensional problems it's fine forthis this is it it's the the noise gainis a scalarwe're done with butin the case of the optical mass we'rerecovering u and vso it's a two dimensional problem and itmay very well be that the error incertain directions is very differentfrom the error in other directions so wewant to have a more nuanced picture imean we can have a kind of a grossstatement that says okay it's bad if thedeterminant is small that's a good startbut it's that's a scalar constraint andit doesn't tell you that oh actually ifyou move in this direction you have verygood knowledge of the motion but if youmove in some other direction you don'tokay so let's look at this wellfollowing what we did over therethat's the condition for that set ofhomogeneous equations to have anon-trivial solutionso you know homogeneous equations wedon't run across a whole lot we almostalways have inhomogeneous equations thenthey have asolution unless the determinant is zeroand homogeneous equations have some veryinteresting propertiesand we'll just look at them a little bitokay so what is the determinant of thatwell it's justright it's just that times that minusthat times thatand if i multiply that outi get a second-order polynomial inlambda that i can solveand uh using the usual formula forquadraticso well actually isimplified something here so this wasbefore i simplified it this wasa plus c squared minus 4 ac minusb squaredright soit's umb squared which is now this term minusfour times uh that termand then when i multiply that out andrearrange terms i i get thisuh so uh those are the eigenvalues so umthey'll will betwo vectorsthat have this property that when youmultiply that matrix by the vector youget a vector in the same directionand their length will be changed by thisquantitythat's how much they'll bemagnified or demagnified and we're veryinterested in that becausetypically what's on the right hand sideis the error the measurement includingthe error and sothese eigenvalues will determine howmuch the error will be will be magnifiedso just forcutting down on the writing let me givethis a name so that i canabbreviate thingsokayso uh those are the eigen values andi'll be interested in how big they arenow in our case you know with opticalmouse these were integrals of e xsquared and integrals of e x e y andstuff like that and then we can plugthat into this formula to find out whatthose eigenvalues actually arenow in practice you know if someone'sfinding eigenvalues and eigenvectorsthey just go to matlab or whatever sobuti i want you to get a feel for this atthis trivial level of two by two andthen yeah you know if you're going to doa 13 by 13 matrix of course you're notgoing to do it by hand but it's usefultodo this and get a clear idea of whatit's all aboutokay so those are the eigenvalues whatabout the eigenvectors so they're goingto bespecial directions in whichthis property holdsand so how do we solve for those wellnow we have to solve the homogeneousequations so we got this matrix a minuslambda bum so this is going to be oureigenvectorthe one that corresponds to the lambdawell we have two choices uh that we pickand so we're now assuming that we'll beplugging in a specific value of lambdanamely one of one of these twoso uhwhat does this mean well it means thatthere's arelationship betweenthis vector and that vector and betweenthis vector and that vector namely theirorthogonal so so why is that wellthis value this zero up here comes frommultiplying this row vector by thiscolumn vector so it'sa minus lambda times x plus b y is 0.that that's the first equation and thenthe other zero comes from multiplyingthis row vector by that vector so i getb xand soi canthink of this as a dot productand i can think of this then asvectors being orthogonal to one anotherright so this means thata minus lambda b is perpendicular to x yand by the wayb minus c lambdac minus lambda is also perpendicular tox andy so in solving these homogeneousequationsuh i can note that basically i'm sayingthatthe solution is perpendicularto the rows and so i cani can write down the answer very easilyparticularly in the 2x2 caseall i need to do is find somethingthat's perpendicular to a minus lambda band so how about thisright thatif i multiply that by a minus lambda bi just get two terms with opposite signsand they cancel out so uh there's athere's an eigenvectoruh how big is it well as we said in asense it doesn't matter becauseuh any pr any multiple of an eigenvectoris also an eigenvectorokay sothat's one of themnowactuallythat's two of them becausewe can plug in the two different valuesof lambda we get two different vectorsbut why focus on the first row you knowthis should be true of the other row aswell so let's try it on the other rowwell on the other rowhere's a perpendicular to the other rowif i take the dot product of that withthe second row i get two terms that areequal in magnitude and opposite and sineso they cancel out so that that's alsoan eigenvectorand now that's getting a littleconfusing becausenow i can plug in the two differentvalues of lambda so i get foureigenvectors sowell it turns outthose actually point in the samedirectionandthey're the sameif lambda isgiven by that expression up thereand you know i'm not going tobore you with algebra it's prettystraightforward to show thatsoso all together we can actually writethe resultif we want a unit eigenvector we cannormalize these so we just divide by thesum of squares the square root of thesum of squares of of these two termsand we get something like thisand the plus minus applies to the twodifferent solutionsthe top sign always corresponding toone caseminus a minus cso um oh i should mention thatthis isthis is all written upin in this little four page pamphletthat's on under materialssoyou cancheck on it there i'm leaving out somedetails butokaysoi mean you might you might think wellgee this is what it is for two by twothis must be pretty complicated you knowif i get toten by ten and it isand that's whyyou know typically you use uh somepre-packaged toolsbut it's good to see how this works sofor an n by n matrixreal symmetric matrix we're going tohave n of these typically and umthere'll becorresponding eigenvalues andeigenvectors andthey allow us to talk abouterror amplification and let's see howthat works whywhat's the relationship to erroramplificationum[Music]so let me justi think i mentioned this beforebut we're thinking of our vectors ascolumn vectorsand we can alternativelyor equivalently think of them as youknow skinny matricesand soi can on the one hand write a dotproduct this wayor i can write it like this and so whatwhat is that well you know a1 a2and you know obviously if i multiplythese two skinny matrices i get a scalarwhich is just the product of a1 and b1plus the product of a2 and b2 and so onso that's just the dot productand that's aconvenient notation and we extend thatto matrices as wellsoin in our context over herewhat we can show isandyou know there there's a line of algebrawhich i'm leaving out uh which is inthis pamphlet so you can you can verifyitso that's sort of surprising what thisis saying is okay i'm going to takeeigenvector 1multiply the matrix by thatand then i get something newand i take the dot product with theother eigenvectorand it's the same as flipping ityou know like i don't know it'sm could be a rotation so it's like i'mrotating e1 and then taking the dotproduct with e2 and that's the same asrotating e2 and taking the dot productwith e1well if that's the caseum thenwe can uhshow that these have to actually beorthogonalso that's where we're going we want toshow that theseeigenvectors are actually orthogonal solet's first look at this so this ishow do i know that well because thewhole point of these eigenvectoreigenvalues was that m e1 is going to bein the direction e1 but a differentlengthmultiplied byby the eigen eigenvalueand this one hereis going to beme2 is the vector e2 just multiplied bythe eigenvectorokayso the part that takes a few steps isproving that these are actually equalbut it's in the paper okay now what doesthis saywelli can gather this upandthat tells me thatyou know e1 dot e2 is zeroso that means they're perpendicularwell there's aanother thing that can happenright if lambda 2 is the same as lambda1 then that doesn't uh followso um when i take the roots of thispolynomial and the roots are alldifferentthat means that all of the eigenvectorsare orthogonalso that's what that says and if they'renotall different if there's a multiplicityit turns out i can pick the eigenvectorsto be orthogonal so the the answer theexample would bethe eigenvectors are in a planeand i can pick any two vectors in thatplanethey'll all work all of the vectors inthat plane are eigenvectorsbut i i canpick two of them that are orthogonalso yes if umif two of the roots happen to be thesame thenum this doesn't force the eigenvectorsto be orthogonal but i can just pick outof all the possible ones two that rand the idea is to construct a wholecoordinate system so i've got all ofthese orthogonal vectors and they ofcourse define a basis for for the vectorspaceso umso in the two by two case you know whathappens well i get a eigenvector hereand the other onebetter beperpendicular to itand of course i can use those to talkabout points in the plane just as i canuse my original x and y axesthey form a basis and so that means thati can write any vectorcan write any vector as a weighted sumof these eigenvectorsso because they they form a basisokayum[Music]so what are these what are these alphai's how do i find themwell that's pretty easy because i justdov dot e i don't know let's call it jwe don't want to get in conflict withthe i which is a dummy variable in thesumand we said that the eigenvectors areorthogonal to each other so that meansthat all of these dot products are zerowell except one which is e i dot e iright and so this is going to give mealpha i right because uh if i pick theunit vector version so so okay so thisis a sum but i don't care about most ofthe terms becausethey'remultiplied by zero this dot product theonly one i care about is where these twoare the same vectorsand in that case i can if i pick unitvectors that's just one so i just getalpha i soum so it's very easy tore-express any vector in terms of theeigenvectorsokay[Music]so then let's look at mvissigma alpha r mso now we get to the juicy part theconclusion herewhich is that if we take an arbitraryvector measurement and we multiply thematrix by that measurement to obtainour unknown variableswhat happens is thatdifferent components are magnified bydifferent amountsright so those directions are special inthat along those directions we know howmuch the error is magnified and soyou know in this case obviouslyif wehave a large eigen valuethe component in that direction will bemagnified a lotif we have a small eigenvalue it'll beminified it'll it'll be uh diminishedand and we'll be happy sookayso that's the connection to to the errorgain but we mostly are dealing withinverses so this is what happens if wemultiply by a matrix m but we'retypically solving inverse problems wherewe're dealing with the inverse matrixand so what are the eigenvalues andeigenvectors of the inverse matrixsoto see thati want to first introduce something elsewe're going to needso that looks like what we did for dotproductuh but not quitenow it's the other one that's the hasthe transpose on it and we'll use thisnotation quite a bitand so let's write this out a1 a2 a nand that's not a scalar it's not a dotproductbecause what we're going to do ismultiply the first term here by thefirst term here a1 b1 and then the firstterm here by the first term there a1 b2data that a1 bnand then we go down to the next row herewe get a2soagain if we treat these vectors asskinny matricesthis is what we getso all togetherthe dyadic product of n vectors isn by an n by n matrixandand that'll be handy for us solet's apply that idea so we got vwe expandedit right over there we said thatbecause these eigenvectors form a baseswe can express an arbitrary vector interms of itand then we found the actual weights andwhen we plug that in we we get thisformulaand sowe can now rewrite that in various wayssoone way to rewrite this is vtimesv transpose times e iuh or another way of writing it is let'ssee we want to do thise ie i transposevso this is just taking the dot productand rewriting itand how do we get here well in a dotproduct that it's commutative we can wecan flip the v and the e iand therefore we can get this expressionfor the dot product just as easily asthat oneand then taking a scalar times a vectoris the same asmultiplying the vector by a scalar so sowe get this and why is this interestingwell because when we do these matrixproductsuh they're associative so we can rewritethis ase i e i transpose vand up there we have a sumover iandv is not dependent on i sowe can actually factor this out likethisso soyou see thatthese terms all depend on i so in thesum every term has a different one butthe v is the same so i can separate thatoutand wait a minute i'm saying v equalssomething times vso so what is thisthat's the identity matrixso this is a very long way around towrite the identity matrixand the reason we're doing it is becausewith a slight change we can now writethe matrix musing the sameideaand so umwe can actuallyget to this version of mand why is this interesting well becausenow we can seethe properties of the eigenvalues thatwe're interested inand then how do we check thiswell we can perform operations of m onsome vector and see whether it producesthe same result so we can we can checkthisby doing for example m times e ichecking that this is true well let'smake iti guess we've got e i here so make thisjthat's very easy to check and if that'strue then then this is trueummore interestingly is this one so thisis the one we finally get to that's themost interestingand how do we check thiswell an easy way isto take this expression for m multipliedby this expression for m inverse andshow that you get the id you get thisexpression the identity matrixokay so um you know this may be going alittle bit fast but remember it's it'sall there and it all fits on four pagesit's you know like those uhfacebook come ons that say oh and it allis only three pages longokay so uh why are we interested in thishere's the keyumthismatrix we're using to solve our visionproblemwhich takes a measurement and turns itinto somequantity of interest to us displacementvelocity whateverit multipliescomponents of the signal by 1 overlambda isoit's badif lambda iis smallso so that's where we're going with thisso thatoften the way we can understand theperformance of one of these methods isto look at howwhat the noise gain is and when we getpat you know up from one dimension thisis the way to do itwe take that matrix we find itseigenvaluesand if some of them are small we knowthat it's ill an ill pose problemit's not going to have a stable solutionif you make a small change in themeasurement you'll get a big change inthe result soi know i'm saying this again and againand again and it's because it's you knowkey it's importantso for example in our optical mousesituation where we just have the two bytwo casewe end up with a diagram like thiswhere we have two directionsthat are eigenvectors andif one of them has a small eigenvaluethen it's going to be hard for us toaccurately compute the motion of themouse and it turns out that in that caseone of themis the direction of the isofoldand the other one is the direction ofthe gradientso so remember thatyou know isophotes are just lines ofbrightness is constantand they're very handy for drawingthings on the blackboard because i can'tdraw gray levelsand they have the property that they'reperpendicular to the gradientwhere the gradient is just thetwo vectorofderivatives with respect to x and yokay and it turns out that theeigenvalue that corresponds tothe uh isofoat directionis very smallin the ideal case zeroand the one in the gradient direction isnotso that invert that that means thatthe in the in the isoford direction anysort of small error will be magnifiedhugelywhereas in the other direction thegradient directionit's okayand so that corresponds with ourunderstanding thatif you move this image in the gradientdirection you know things change thisisophote is now down here and thebrightness here has changedwhereas if you move it in the directionof the isophoteyou know by definition isofoat meaningconstant brightness the brightness tendsnot to changeor not change by much and so that's thena motion that is not easy to detectaccurately soso this is kind of a little a storyabout eigenvectors and eigenvalues andwe will see thatthey don't they play a role in this notjust in error analysis butthis is one of theirmainusesokay back back toslightly less abstract stuffphotometric stereoand we we discussed that uh somewhatlast timeso the idea is thata single brightness measurementdoesn't give us enough informationto recoversurface orientation because surfaceorientation has two degrees of freedomand so we need more constraints and andone way to get more constraint is totake more pictures but of course ifthey're under the same conditionsthey'll be just the same picture with alittle bit of different noisebut you know the difference in noiseisn't going to buy you much other thanmaybe you can average them to reduce thenoise butso we ended up with a system where wetook three picturesum to make things simpleunder three different lightingconditionsokay so and we started off with realsimple casewhere we said that the brightness wasproportionalthe cosine of the incident angleand that of course is the dot product ofthe surface normal and the direction tothe light sourceokayand we'll in a minute we'll talk aboutwell what if the surface doesn't satisfythat constraint because obviously realsurfaces don't they may approximate thissome surfaces approximate this ratherwell but we want to deal with arbitrarysurfacesokayso we might have for exampleuh one brightness measurement so we'renot talking about a particular pixelright so we're gonna focus ondoing this at every pixel and solooking at a particular pixel and thisis the brightness the gray level wemeasure thereand i've put in this rowwhich i call the albedoas a way of describing how much lightthe surface reflects so a white surfacethe row would be one and a black surfacerow would be zero and you know realsurface would be somewhere in betweenand one reason i do that isbecause i canand because it makes the problemactually easier because now i've gotthree unknowns andif i have three constraints that arelinear i can use my great linearequation solving methods to solve themin a waytwo two images would be enoughbecause byyou know if we have twoequations in two unknowns there'susually a finite number of solutionsthe trouble is the finite number may notbe oneand in this case the finite number istwo and so you know i've disambiguatedhugely before i didn't know what theorientation was at all now i know it'sone of these two possibilities but it'snotit's easier to deal first with a casewhere we can completely disambiguate itby introducinga problem with three unknowns and threemeasurementsokay so theni use a different light sourceand i get a second measurementand then i use a third light source andi get a third measurementand you know we did this last timeand this is where we use that notationfor dot productand we use it to talk about uhlet's do thatright so what what's this this is athree by three matrixuh s1 transpose is the vector to thelight source one just flipped from beinga column vector into a row vector so thefirst row of this matrix is s1 justturned on its side and this is studentson and so when you multiply this matrixby that vector the first thing you getis the dot product of s1 and nand that of course is e1okay[Music]and here i'm using the shorthandnotationso i'm absorbing that lambda into the nand that's convenient because nowinstead of having to deal with a unitvector i have a arbitrary 3 vector andthat means i don't need to deal with thenasty uhnon-linear constraintsee the the thing that leads totwo solutions is a quadratic and why isit why do we get a quadratic wellbecause we have this second orderconstraint on n butby doing this weavoidhaving to do thatokay so that i can just write as amatrix s times vector nisvector e so the vector e is just i'mstacking up my three measurementsso again to to be clear so i take onepicture i look at this pixel i get e1then i turn on a different light sourcei take a picture at that same pixel iget e2and then i turn on the third lightsource i look at that particular pixeland i get e3 and i stack them togetherto make this vector e1and so you know the solution is verysimpleand extremely cheap to computein particulari can pre-compute this matrix assumingthat the light sources are in fixedpositions so if i know where the lightsources are i know s1 s2 and s3 i canjust construct this matrix andinvert it ahead of time and then this isjust a multiplication of a three bythree matrix by a 3 vector and you knowwe talked about this last timeand we also said that wellthat's assuming that this matrix doesn'tgive us problemsso the problem would be where the matrixis singularthen we can't invert itor if it's nearly singular we can'tinvert it so when does that happen wellit happens when the rows are notlinearly independentso that's when this blows up and forexample if let's try this s3is some combinationof s1 some linear combination of s2so that'sthat's how we create problems right ifthe third row is just a combination ofthe first two rows for orin general if there's a linearcombination of the three rows that givesus zeroso okay um sowhy is this bad well because this meansthat e3 is alpha e1 plusbeta e2right i just need to take the dotproduct of this equation with nand i get this result and that tells methat the third measurement is redundantit's not telling me anything new so nowonder it blows up right it's likeyou're cheatingyou know if you made for example if youmade the third row the same as the firstrowit's pretty clear the matrix is singularand it's also pretty clear that you'renot getting any new information so so itall makes intuitive senseumsoi canthink about this as a sort of picturethis is a case where s1 plus s2 plus s3is 0 and that's you know clearly a badcaseand i can be fancier and i can you knowput multipliers on these i can put alphaon this andbeta on thatand if you like gamma on that so if somemultiple of thosethree vectorsadds up to zerouh this won't work and what is thatconditionhow can i geometricallysayuh what the condition is for this to gowrong so the vectors s1 s2 and s3arefor this loop to close remember this isnow in 3d so we've got a vector going acertain waythen of a certain length then anothervectoryou start at the tip of that firstvector and you put the tail of thesecond vector thereand then when you get to the thirdvector it closesso what is the condition on those threevectors geometrically that makes thatpossiblethey're in the same plane right if i hadyou know x y and z axes obviously youcan't do that you you know the one goesoff in the x direction one goes off inthe y direction one goes off in the zdirection it's never going to close sothe problem is if they're coplanarso so that's uh bad that's when thismethod fails and so obviously umyou should when you do this you shouldplace them so that they'rekind of as far as possible from beingco-planarfor exampleyou could put them[Music]at you know x y z axes so you could havean arrangement where the object is downhereandthen you can erect somerectangular coordinate system and s1goes there and s2 goes there and s3 goesthere and now they're not coplanar sothis won't happen they won't fall apartand then they're questions of well youknow which is bestobviously if i make them almostcoplanar i'm going to get uh unstableresultsthe some eigenvalue will be small and ifi take the inversethe inverse of the eigenvalue will belarge and so the noise amplificationwill be largeokaysoyou know if you're doing this in anindustrial settingyou're in control of where the lightsources go so that that's all you needto know thatyou know don't don't make them co-planarum and uhprobably if the closer you can get tothis arrangement the better wherethey're actually at right angles to eachotherso this is wherei want to talk briefly aboutthe earth moon and the sun soso let's suppose that the moon is madeof green cheese and green cheese has alamborghini reflecting propertyand so we're on earth and we're tryingtoget a topographic map of the moonwhich would be useful thing to do beforepeople land there and of course now wewe did a long time ago but before peoplelanded there there's a lot ofuncertainty you know people didn't knowfor example whetherthe solar wind had pummeled the surfaceto such an extent that there was youknow 10 meters of dust and if you'relanding on it you'll just disappear intothat dust so anyway so there was a lotof interest in trying to figure out youknow how how tall are these craters imean we can see these nice craters butwhat's the slope we don't want to landon something that has a very large slopeso it would be really great if we coulduse photometric stereoand uh because the sun does illuminatethe moon in different ways differentparts of the cycle so you know hereas youuh knowthe moon is alwaysshowing the same face uh to the earthand the other side which is stupidlycalled the dark side of the moonis not visible from the earthand so if you think about let's take aparticular point here uhwhile it's in the cycle here the the sunis uh over there uh but here uh the sunis in that direction and here the sun isin that directionso we could easily arrangeyou know 3 10however many measurements at differentpositions in the orbitand we can use this methodwell there's some assumptions one ofthem is that it'shas lambertian reflectance which whichit doesn't but we'llwe'll try and fix that laterumandannoyinglyit doesn't workand the reason is thatum[Music]this plane herethatcontains the moon's orbit is the same ispretty much the same plane as the planethat contains the orbit of the eartharound the sunsoyou can see what's going to happen thatthese three vectors or moreuh are coplanar or almost coplanar imean the the moon's orbit is a couple ofdegrees off theearth orbit around the sun so you get atiny change but it's not enough to makea useful measurementand so umyou know it's amazing we've just donesomething very simple here and we'vealready reached a very profoundconclusion which is thatyou can't get the moon's topographyfrom earth measurementsas it goes through its orbit anddifferent parts are illuminateddifferently as it does sowhich is you know apretty amazing thing umokayso umlet's talk a little bit about thelamborghini assumptionso lambert was his monkand uh he did these experimentsnow why was he a monk well because therewas a time in our historyin the western world where the onlypeople who could learn anything were inin the religious domainyou knowordinary peopleuh couldn't read or write and theyweren't allowed to read or write and andmonks did all these interesting thingslike you know figure out how to brew rumand whatnot andand this particular guyheyou know had wrapped up hislunch in paperand it contained some oily fish and so apiece of the paperwas infused with oil and i think you'veseen this wise piece of paper with someoil on it the oil looks kind of oilyarea looks kind of darkerand then if you hold it up and move itaround you see that it has differentreflecting and transmitting propertiesand so what he discovered was that hecould make measurements of lightusing this tool this you knowtoday of course we'd use a you know 100000 computer andpin photodiodes andhe used the piece of paperand so the idea the idea is this sohere's our piece of paper and here's thefatty spotand what happens is that the paper isnot absorbing any light it's it's awhite paper i mean ideally it couldabsorb a little bit but let's assume itabsorbs none so it only has sort of twochoices the one is sunlightarrives and is reflected backand a little bit of it goes through tothe other sidenow in the fatty partsome light arrivesand a little bit of it is reflected backbut a lot goes throughokay so that's the difference about thetwo parts the fat the fattysubstance basically fills in the airvoids and so the surface is no longer asreflective as before but it's not anabsorbing material so what happens isthe light just goes through okay so whywould this be of interest well it allowsyou to comparetwoilluminationintensitiesand you do it basically by illuminatingthis sidewith one sauceand this side with the other sauceand then you balance it so that youcan't see the fatty spotright becausewhen can that happen well we could writeout equations but i think you can seewhat's going to happen is if the sameamount of lights coming in from thisside as from that sidethen um these will balance that thisthis will appear equally bright whenlooked at from here as thatokayso that's a very powerful idea and hedidn't really do this with his lunchpaper but you knowthat waswhere he went he had white paper whichwas pretty precious at the time you mayremember that people would sometimesyou know write something and thenbecause paper was expensive they wouldwrite something at 45 degrees on top ofit and maybe more so anyway he had thisnice piece of paper and this way hecouldcompare brightness and one of the thingsyou can do then is get the inversesquare lawso he might put four candles on thissideand oneattwo at twice the distance as one candleon this side rightand so it should match right so so hewas able to do all of these amazingexperiments um with this very simpleapparatusum and um get the inverse square lawand of course you know people likenewton would say well it's obvious youdon't need to do an experiment it'sjust the energy is going over thesurface of a sphere and the surface ofthe sphere is 4 pi r squared ah rsquared so anywayso then the next thing he did was hewondered how surfaces reflect light whenthey illuminated from differentdirectionsand using methods like this he came upwith uhwhat's now called lambert's lawwhich is that it's a cosine of incidentanglethe brightness is proportional cosine ofincident angle now of coursepart of that is again not something thatyou need to do an experiment about wealready talked about foreshortening andwe know that the amount of light fallingon the surface varies as the cosine ofthe incident anglebut what's what he uh talked about washow bright does it look in other wordshow much light does it reflect and inwhat directionokaynow to umto talk about this in more detail weneed to have a way of talking aboutsurface orientationbecause the brightness is going todepend on surface orientationandand it may do so in a more complicatedway than lamberti mean i put law in quotation marksbecause it's not a law it'sa phenomenological modelwhat does that mean that means that youkind of postulate a particular waysomething behavesand then give it a name and it's notlikeyou know there's a real surface thatdoes exactly this many real surfaceslike paper are good approximations butthey don't do exactly thatokay we already talked a little bitabout talking about surface orientationwe said thatwe can erect a unit normalsoso here's a little patch of the surfaceand we've erected something that'sperpendicular to the surface and that'sa way of talking about orientation andwe mentioned that it has two degrees offreedom because it's a three vector butthere's a constraint that is a unitvector so two minus one three minus oneis twoso that's one way of talking about itand for an extended surface you canimagine thatyou know we do this for little facetsall over the surface and they'll all bepointing different waysbut if you pick a small enough areaas long as the surface is reasonablysmooththenwe we canreduce it to this caseand you know we have to exclude thingsthatmathematicians will construct like asurface thatyou know onewhere x is a rational number and 0 wherex is an irrational number we can't dothat there but but for real surfaces wecan facet it and for different facets wecan do thatthen we also mentioned since it's a unitvectorwe can talk about orientation in termsof point on the unit sphere and we'llfind that pretty handy later on you knowfor example if you want to talk aboutall possible orientationswell that's the whole surface of thesphereor you know there's certain operationsthat we'll be doing on the surface ofthe sphere so those are representationsbut we we need something else solet's look at this so here's a taylorseries expansionright wherethe dots indicate higher order termswhich we can ignore as long as we maketheinfinitesimals small enoughokaynow toso the difference of these twowe can write as delta z soso that's the same equationand thensojust asso i'm introducing p and q as shorthandsforthesederivativesand this is just analogous to what we'vedone before whereyou know we introduced u and b asshorthand for dx dt and dydtpartly because it's too much botherwriting all that stuff and partlybecause it makes it look toointimidating when it's actually verysimpleso p and q are slopesand in fact p and qis the gradient on the surfaceso remember how i said that one way tothink about an image is that itheight above someground level so we have x and y uh inthe image plane and then we can plot thebrightness as height above that and thenwhen we talked about the brightnessgradient e x e y i was saying that it'slike the great gradient of that surfacewell here's a case where we actually aretalking about a surface in 3dand this is its gradient it's you know zd x d z d yokay uh one way tosort ofmake a picture of that is thisokay so this is our surface with uniwith normal vectorand this edge is delta x and this edgeis delta yand this part here is qdelta yand this part is p delta xso this is a diagram that basicallyillustrates this ideai take a small step in the x directiondelta xand the surface goes upby you know dz dx times the delta xthen i take a small step in the ydirection and the surface goes up by d zd y times delta y because it might godown justin this particular picture it goes up sothis is another way of understandingwhat this equation is sayingnowwe'll find thatthey're places wherethe unit normal notation works for usand they're places where the gradientnotation works for us and so we need tohave ways of switching back and forthand this is very commonuh in you know computational problemswhere you know some problems are easilydone in one domain and some are easilydone in another domain so it ends upbeing a problem of conversionyou know likecartesian coordinates to polarcoordinates some things are easy to doin polar coordinates uh some are easy todo in cartesian coordinates so you justwant to be able to convert back andforthand so same here so how are theserelated how isnrelated to p and qwellone thing we can do islook at this surface which has a normalnany line in the surface has to beperpendicular to n right that's thewhole idea ofn is perpendicular to the surfacemeaning it's perpendicular to any linein that surfaceandif i have any two lines in the surfacethat are not the samethen i'm done because right i got if nis perpendicular to two lines i can justtake the cross product soright because the cross product of twovectors is perpendicular to both ofthose vectors sookay so i need to find some tangents inthis surfacewell here's one this is an edge whichlies in that surfaceandwhat is its uh directionwell in in the x the x component of thatvectoris delta x it has no y component sothat's zeroand thez component is p delta xso that's that's a vector in the reddirection and i can take the delta xoutsidebecause it's you know arbitrary lengthand i getthat that vector so that's a tangentwell i can get another one i can getthis one hereandthat oneif i move along that edge there's nochange in x so this is zerothere's a change in y right i'm movingalong here so delta yand there's a change in height which isqdelta yand so i can take the delta y ofokay so what have i goti have two directions that lie in thesurfaceand you know i could have picked someother directions those just happen to bevery uh convenient to calculateand so what i need to do is take thecross productand that should beparallel to ni mean it'll have some sizewhich we don't care about we're onlyworried about direction of that unitvectorokay so what's that cross producti think it's uh minus pminus q1so that's the connection between the tworepresentationsand now i can you know go a little bitfurther i can say the unit vector isjust normalize itso if you give me p and q i can computethe unit normal nand i guess we also want to be able togo the other directionso p is minus n dot xso if i have to ever go the otherdirection i can do thisand you know this looks sort ofintimidating but it's just saying takethe take the first component of nnamely this minus pand divided by the last component of nwhich is oneso why did i do that well because whatif n isn't the unit vectorthen this will take care of itif it is a unit vector i don't need tobe this fancy butokay so i can go back and forth betweenthe two notations representing surfaceand what do i do with this well thegreat thing now is i have a way ofmapping in the planeall possible surface orientationsi already kind of had that because i hadthe sphere i said that you know allpossible surface orientation correspondto points on the sphere trouble is youknow i don't have spherical paper and idon't have a spherical blackboard sothis is a projectionof the surface of that sphere into theplane that's particularly easy tounderstand because you know this is justthe zdxand this is just d z d yokay soas withvelocity space this is a very usefulconstruct but you know takes somegetting used toso points in this planeare not points in the imagepoints in this plane correspond todifferent p and qi.e different surface orientations sofor example let's consider that pointso that's the point where p is zero andq is zero and that'sa planenow if suppose that the floor here hasan x-y coordinate system and z comes upout of the flooruh what type of a surfacewould have this point in thisrepresentationyesright the floor for example anythingthat's levelso right why well because dz dx is zeroand the zdy is zero so if it had anytilt at all then dz dx or dz dy would benon-zero so it's the floor has thispropertyuh but also actually any surface abovethe floor that has the same orientationso there's the there's an ambiguity hereit's not telling us where this thing isit's only telling us how it's you knoworiented in spaceand so that's going to be a secondaryproblem suppose that we come up with amachine vision method which for everypixel allows us to recoversurface orientationeither this way or in p and qwe still have to patch it together tomake a complete surface but that turnsout to be easy because it's overdetermined unlike most of the problemswe deal with which are underdeterminedokay sothat's that surface now if i go overhereto p equals one let's suppose that xgoes to the right and y goes forwardthat corresponds to a surface where theslope going to the right is one you know45 degree upthat that would be pretty steep i wouldslide off it with these shoes so thatthen if i go over here uh what's thatwell that's the surface where the slopein the y direction is one so it's youknow the kind of thing youfind at ems to check out your rockclimbing boots uh make sure that theyare fitting well and you stand with yourtoes up like this and then this one ofcourse is a combination where we havethe slope to the right and the slope toto the forward and so on and the furtherout i go uh the steeper it getsso that's what this plane is every pointin this planecorresponds to a particular surfaceorientationnow inapplication to machine visionwe find that the brightness depends onsurface orientationso this is a wonderful tool to plotbrightnessright so i could you know justexperimentally i could take a patch ofthis materiali could orient it flatparallel to the ground i measure howbright it appears and i put that numberhere you know e1then i tilt it up 45 degrees and i putthat number over here and i tilt it upuh what's the slope of 2the inverse tangent of 2and i plot it there so i can plotmy brightness valuesas a function of surface orientationso this becomes kind of an image becauseat every point there's a brightness butit's not it's not a transformation of animage that you take with an opticalsystem at all it'sthat's perhaps confusing but it's notso everything in herecorresponds to orientation and then wecan plot whatever we wantsuch as brightnessokay um[Music]so where is this going well one idea iswe can then invert this suppose thatwe've made this mapthen you measure the certain brightnessand then you go back to it and say ohthat means the surface orientation issuch and suchso that'sthat's the ideaand you're probably saying wellreallyum because you know maybe brightnessdown here is the same as thereand maybe the brightness down here isthe same as thereand in fact maybe there's a whole lineofpoints that have the same brightness andthe whole line of points that have thesame brightnessandumyou know just counting equations andconstraints tells you what the problemisif we make one brightness measurement wecan't recover two unknownssouh just as withour velocity determinationa single measurement uh won't be enoughit will be a dramatic improvement overno measurement so if we don't take ameasurement we don't know where we arein this plane at all we could have anyorientation for a surface element if wetake one measurement we'll beconstrained to some curveand then we need more constraint toactually pin it down to a particularorientationso justlet's relate that to what wedid in our example of photometric stereoand also our discussion of lambertlet's suppose thatwe have a lamborghini surfaceso there's a common confusionwhich is that oh this stuff only appliesto lamborghini surfaces why are we doinglamborghini surfaces because forlamborghini surfaces i can show you nicediagrams i can solve equations in thereal worldnothing is exactly lamborghini so if youwant accurate result you will have tomeasure calibrate and we'll see how todo that but for the moment let's justassume that magically we're dealing witha a lamborghso then we have that surface normalproportional to brightness so brightnessis proportional to the cosine of theincident angle and that's the dotproduct of the surface normal andand the incident direction and nowuh we want to translate thatinto this notation p and qbutone thing that's important is we'relooking for isophotes in here we'relooking for these curvesand so we'll be looking at places wherethis is a constant but let's firstreplace so the unit vector n is minus pminus q oneand we can take the dot product of thatwith somelight source directionand then we can you know plot thatbutthereit'll be handy to introduce anotherlittle shortcutwhich isa different way of writing the directionto the light sourceso we sort of that equation there's kindof a mixture we've sort of halfway gonefrom unit vectors to pq spacelet's go fully and so the full way is tosay wellto perform the same transformation onthat unit vector that we did on n it'sjust the same equation just this timewe're talking about the vector to thelight source rather than the unit normaland so to to make that uh cleareruhthat point p s q sit's it's in the in that planeand and what is itwell it's the orientationwhere the incident light rays areparallel to the surface normalright so this is the point where[Music]and so for the lamborghini surface umthat's going to be the brightest spotright becausethe there's no the angle between thosetwo vectors is zero so the cosine of theincident angle is one and that's as bigas cosine can get so um we picked thisparticular point psqs because it's uh inthat plane because it's the one thatgives us the brightest surface so it hassome real meaningother than geometrically it just meansthat we're[Music]we're illuminating the surface rightdownthenormalanother way think about it is theforeshortening there's no foreshorteningwe haven't tilted the surface relativeto the light source so we don't have thesame powerspread over a larger area here here it'sconcentrated in the smallest possiblearea it's the most efficient it's whatyou do with your solar collectorsokayuh so that means we can rewrite the ndot s in this formso this is starting to look a little bitmessyand what are we doing well we want wewant isophodes so we want to know whereis this quantity constant what are thecurves in thatpq space in gradient spacewherethat quantity is constant welli can square this and move things arounda bitnowthis quantity herethis is just some constantright because i'm noti'm keeping the light source in thefixed positionandthenthe question is you know what kind ofcurve does this definein pq spacewell if you multiply it out you're goingto get some constant terms some termsproportional to p some proportional to qthe highest order terms you're going toget are second orderso when you multiply it all out which issort of messyuh you'll have something that's secondorder in p and qand so the question is what kind of acurve does does that defineand it may be hard to think about it interms of p and q i mean we're not justtalking about geometry in the planeso imagine you have uh in x and y youhave an equation that's got x squared ysquared x times y x y and a constant initand all added up uh what sort of uhcurve would that correspond yesit could be a parabola yeahanything elseellipse yes okaygreat uh generalize it a little bit moresorryit's a conic section okay so yes thoseare uh great examples overall uh they'recomic sections and so yes uh we can havea parabolawe can have an ellipsewe can even have a circlewe can have a line a specialdegenerative casewe can have a point even more specialgeneraldegenerate caseand we can even have a hyperbolaokay so so that's uhi want to plot this thing andthis is like a previewyou know whatwhat it's going to look likeso uhif c is zerolet's let's look at that special caseif z is zero then this is zero and thatmeans that uh one plus pspplus q as q is zeronow what kind of an equation in p and qis thatit's a line it's a linear equation soit's just a straight line so that's thatone special caseso there's going to be some sort of linein here and and it's where thebrightness is zeroand um okay and then another specialcase is where p equals ps and q equalsqsand that's a special case where we therewe talked about herewhere the normal is pointing straight atthe light sourceandwe're getting the maximum brightness sothere's some point over herewheree is oneif we normalize itappropriatelyokay and then the rest you know you canplot from using some sort of program ifyou likeokay so that's a very handy diagrama very handy diagram for graphicsbecauseif i have a surface that i'm plotting ican easily determine theunit normal from it i can get p and q ori can get p and q directly and then ijust go to this diagram and i read offyou know whatever the brightness is hereand i use that as a gray level or coloruh in the image that i'm plottingwhat we're doing is kind of the otherway around what we want to do is umsay okay i've measuredyou know e is4962.72 what's the orientationwellin this case that'sthat curve soi don't get a unique answerbut it's heavily constrained it's it hasto be uh on that curveokaynow if i have more constraint i canimprove on this for example suppose nowi move the light source then this wholediagram changes rightbecause this remember this point hereisone that basically depends on theposition of the light sourceright it's psqsis thefrom this equation it's related to thedirection to the light sourceokay so imaginei don't want to mess up this diagram butimagine that i've mirror-imaged this bymoving the light source over here soi'll have aa second set of isofoads that nowintersect with theseand if i make a measurement under thoseother light conditionsthen the answer has to be on both curvesand[Music]then i get the solutionfrom that sojust for fun let's suppose that theother curve was i don't know like thisi mean they're they're curves they'renot lines so it is quite possible forthem to uhintersect in uh in two placesso i have a finite number of solutionsin general umnot oneso and that's whywe focus more on the case where we usethree light sources instead of twoso just a note on why are they conicsectionswell that actually also has a easyanswer which is you know suppose i takea brightness measurement of alamborghini surface so here's my lightsourcehere's the surfaceand from the brightnessi can calculate this anglebutof course i can spin this vector aroundthis line to the light source i can spinthat aroundand what do i get i get a coneif i measure a different brightnessit'll be different anglei'll get a different coneand so on so i can you know imaginesome third surface element i measure yetanother angle and i get say this thischordso they're these uh nested conesand now imagine that you cut this with aplane sothis is our pq planeandtada conic sectionsand yesyou won't just get ellipsisyou may get a hyperbola you know as longas this bottom edge of the cone isactually below this planeyou will not get a closed curve soparabolas are possible yeahokaylet me let me first address that oneother side of the line so wesaid that it's cosine theta except whenit's negativethis is where uh cosine theta i goesnegativeand i haven't purposefully drawn thispart of the diagram because in practicebrightness doesn't go negative it's ameasure of power so it can't be negativesoif i were to just plot cosine theta itwould continue but we're having max of 0and cosine theta so so we have this partand so where does it turnuh the other part of the question is youknow where does it turn from being aclosed curve to being uh openum i'll leave that as a puzzle forfuture homework problems sowhy because i don't know the answer solet you figure it out okay uh thatthat's it for today so um[Music]i guess you all know there's a homeworkproblem out that will be dueand please make sure youare signed up on piazza because a lot ofannouncements are on there about officehourshomework problems and stuff like that

Gradient Space, Reflectance Map, Image Irradiance Equation, Gnomonic Projection

we're starting to talk aboutwhat determinesbrightness in an imageand how we canexploit thatand we introducedthe idea of a gradient spaceuhwhy well becausebrightness is going to depend on theillumination obviouslyand it's going to depend on the geometryof the situationincludingsurface orientationobviously the amount of light falling onthe surface willper unit area will depend on itsorientation and then different types ofsurfaces will reflect that light indifferent waysin any casewe expect thatthe brightness we observe in an image isgoing to depend on the surfaceorientationof the corresponding patch on an on anobjectand so we need to talk about theorientation of the patch and we had unitnormals and then we had p and qwell these are just convenientshorthandsfor thoseslopes in in the imageslopes on the surface so those arederivatives of height rather thanderivatives of brightnessand then umsince we were busy with photometricstereo and we justtalked a bit aboutlambertian surfaceswhich have the property that theirbrightnessdepends on the cosine of the incidentangleand it does not depend on the viewingdirection soa surface of that typewill appear pretty much equally brightfromuh all viewing points that you mighthavewhich is fairly commonfor material uh on on our human scaleso what what determines how surfacesreflect lightwell we get into that but largely it's amicrostructure you knowphotons get into the fibers in my paperthey bounce around they come back outagain in a different directionand that's what determines how brightthe surface will appear from a certaindirectionandso a lot depends on the imagingsituation if i'm looking at the moonwhat constitutemicrostructureare you know craters uh not fibers ofpaperso as we'll discuss laterlambertian surfaces or near lambertiansurfaces are fairly common in our worlda lot of matte surfaces are pretty goodapproximationssnow and whateverbut they don't necessarilyapply when we go to microscopic scale ortoa cosmic scale soanyway solambertian is a handyapproximation for some surfaces andwe canaddress it in different ways when if weexpand out that n dot syou may remember we gotsomething like this which is linear in pand q so that's the good partbut then unfortunately we divide by thisterm here which is uhnot linear in p and qand there's alsouh this term although we don'tworry too much about that because that'sa constant if the light source issomewhere as defined by psqsthen that's a constant and i'm going toget tired of writing that so i'm justgoing tointroducea constant rsto represent itand then we went through the business ofwhat are the isophotes well that's wherethis expression is constant and we canget rid of the square root by squaringand if we do that we end up with anexpression which is uh second order in pand q so it's gotp s p squared q squared p q p and q andconstants and plotting thosegives us conic sections and inparticularwe end up with a diagram like thisand what is thiswell this says if you tell me what thesurface orientation is i can tell youhow bright it's supposed to lookso umagain uh imagine a terrainbuilt above the floorand i don't know x-axis to the right andy-axis forwardthen i cantake the derivative in the x-directionthat's pand i can take the derivative in the ydirection that's qand those define the orientation locallyof that surface of course it might bedifferent p and q somewhere elseand that defines a point in this planeand i can go to that point and sayyou know what is the value of thisfunction there and thatwill be the brightnessand this is you know obviously handy ingraphics because we canhavesurface models we can determine not justtheirz position their depth but also thesederivativessurface orientationand then we go to this diagram whichcould be perhaps a lookup table in thecomputer and we just look up theappropriate gray level to paint at thatpoint in the imagethat's the forward problem and we'redealing with the inverse problem so ourproblem is okay i have measured thebrightness ofe equals e1what can you tell me about the surfaceorientationwell it's confined tothat curveit's unfortunately not going to tell meuniquely what the surface orientation isbut it's restricted now toa great dealand so i need additional informationand there are various ways of gettingadditional information one is to saywell most objects in the world aren'tsort of haphazardcollections of blobs in three space butthey hang together you know objects aresolid they have surfacesneighboring points tend to often havesimilar propertiesyou know different parts of this tablehave pretty much the same surfaceorientation until you get to the edgebut that's a hardconstraint to implement we'll get tothat latera much easier idea iswell if i illuminate the surfacedifferently i'll get a different mapandi'll get a constraint on that number sofor example suppose i move the lightsource from there to over hereand then i draw the same diagramand now i measure the brightness in thatsecond image under differentilluminationand let's suppose it comes out to be idon't know this this one herewell then i know that the surfaceorientation is where those um two curvesintersectso that's phenomena exterior done in inthe graphical way we we've done itbefore in an algebraic wayof course that algebraic wayonly worked if we had lambertiansurfacesthis is going to work for any surface aslong as you can drawthis diagram which is called areflectance mapso the reflectance map basicallyis a diagram that shows you for everyorientation how bright that surface willlook for that orientationokay nowwe've talked a lot about lambertiansurfaces and that's mostly becausethey allow us to write down theequations and solve them but realsurfaces aren't perfectly lumberjaneand some are dramatically non-version sowhat do we do in that casewell it's pretty easy toimagine we can create a diagram likethis and use thatandwe canperhaps think about building a lookuptablenow this is sort of going the wrong wayhereif you've got p and qyou can look up what the brightnessshould be we need a table that goes inthe opposite direction solet's think about how to do that so onewayis tolet's firstsay non-line versionuh one one wayis to take a surface elementandlook at it under different lightingconditions and recordwhat you see and then repeat that fordifferent orientationsand you can do that it's going to getpretty tedious because you have tobasically explore this whole spaceand everycell in the space in your lookup tableis going torequire that you reorient thatpiecebecause you can automate that and havesomerobot do the calibration for youbut it's uh an alternative that's ofteneasier is to use a calibration object ofknown shape and you know what betterthan a spherepartly because it's very easy to make asphereand what do we do well umwe take an imagefirst we'll have it lit up from allsidesandthe sphere of course will image as acircleand presumablyuh the brightness inside the circle isgoing to be much larger than outside sowe should be able to uh distinguish thetwo and and fit acircle to thisso we take this image and then we fit acircle and what does that mean thatmeans we find estimates for the centerand the radiussuch that all points within that arebright and all points out arenot and you know you can do somesubtle clever thingsto make that accurate to sub pixels andso on but we won't talk about that nowokay so what good is that wellfor a sphere we have a very convenientrelationshipif we draw a point from the center ofthe sphereto the surfaceand then we at that point on the surfacedraw a unit vectoruh guess what those are parallel so andyou know that's a unique property of thesphere so it makes it really easy toknow what the surface orientation isbecause we just connect the center ofthe sphere to the point on the surfaceyou're interested in and there's yourand of course this doesn't quite applyto the earth because it's not a spheresoneed to modify that slightlyand that's why there are severaldifferent definitions of latitudedepending on whether you're talkingabout the local surface normal or thevector from the center of the earthand usually we take the local surfacenormalas the definition forthe angle that's latitude okay so nowwhat well you know we we could be alittle bit more precise hereum so we have umfor every point in this image nowuh let's let's call this point x yuh we can calculate what the surfaceorientation is and how do we do thiswell let's start with a cross sectionso this isyou know here we're looking down on thespherenow we're lookingsideways across it and i don't know thecamera isway up thereokayso the surface normal isparallel to thisright it's just well this is the point xnaught y naught z naughtandthis is the point x y zand so umthat's the surface normal so all wereally need is a formula forz minus xenoswe we're measuringx and y in the image plane so we knowthosewe can't measure the depthso we have to calculate that and ofcourseit's just going to ber 0 squaredwhere's that come from well it justcomes from the formula for sphere whichis that theradius vector has a fixed sizeuh x minus x naught squared plus y minusy naught squared plus z minus z naughtsquared is r squared okay so having thatuh we can then compute uh p and qandthat comes from the formula we had lasttime last time we showed how to computen from p and q and we also said you cango the other wayif you're given p and q you can computen and so that's where that comes fromand you'll notice that[Music]oh sorry i think this is plus let's seeif we go to the right of the center thethe vector tilts to the right so it'splus sorryokay so for everypixelwe canwe know the orientation of the surfacep and q and then we take a pictureandwe get a mapunder different lighting conditions sounder one lighting conditions we get onee1 and another one we get e2 and let'ssuppose we take threeso we we developed this numericalmapping from surface orientation uh tobrightness in three pictures taken underdifferent lighting conditions so we'retreating pixels completely independentlyyou know just think of a single pointthat we're imaging and a single pixelthat's imaging because we repeat that atall the othersokay so nowthat's going the wrong waywe really want to go the other waywe wantto use this informationso that when we then later takethree images of the of an objectunder the three lighting conditionswe can go to some sort of table or somecalculation that gives us p and q so sowe want togo that wayand in terms of implementationyou knowwe left lambertian behind lambertian'snice and analytic you can invert it anddo all sorts of stuff but once we'vedecided that we want to be completelygeneral then we can't depend onanalytically inverting things so let'suse a numerical tableand so this is a three-dimensional arrayin the computerand you know each each little voxel hereis one entryand what is what is in that box welleach of these little boxes hasuh p and qso what i would do is i would make themeasurementin three image situation situations usethem i quantize them tothe discreteintervals of thislookup tablei go to that place and it tells me whatthe surface orientation isso so that's what i'd like but how dohow do i build thatright so so is that clear i mean you cansee how computationally trivial this isit costs you nothing just about tointerpret these images there's nocomplicated iteration or anything it'sjust a table lookup ishard to imagine anything cheaper thanthat so if i can build this table i'llhave a very efficient method forgetting the shape well not quite we willget local surface orientations we stillhave to talk about how to patch thattogetherinto shape but for the momentwe're just going to get surfaceorientationokay now what i'm doing then is i'mrunning over this calibration imageand at every pixeli'm looking i'm computing uh p and qand i'm uh measuring e1 e2 in e3 and iuse that to put something into thistableso i can quantize my e1 e2 e3that gives me the index into this 3darray and i write the p and q there andand i do this for every pixeland you know there might besome overlap because uh i can'trepresent the table with infiniteprecisionum i'll have to make some sort ofcompromise where i say okay i'm gonnaquantize quantizelet's suppose i quantized 200 differentvalues that means the lookup table willbe uh what a million entries so thatthat'syou knowit's a lot in terms of cash size and soon butit's a reasonable value whereas let'ssaya petabyte lookup table probablywouldn't be very satisfactorybut in any case that means we need toquantize and we might need to quantizeum fairly costly like 100and that means that some of these pointsmay produce the samethey happen to produce the same roundedoff values and so they'll be writing ontop of each otherso that's one issue and how do you dealwith that well one way is to averagethembecause presumably they're slightlydifferent p and q's and by combiningthem in some weighted fashion we can gethigher accuracybut a more serious problem isthere could be cells here that never gettouched that never get filled inand there are a couple of reasons forthat one is just the nature of you knowthese numerical quantization effects butthere's a more basic one which isp and qgradient space is a two-dimensionalspaceand we're mapping it here into athree-dimensional spacesowe're not actually filling that space atallrightso what do we get well we get a surfacein that spacesoif we ignore the quantization for themomentyou knowwe're not filling that space we'regettingsome surface and you know if you likeyou canaddress points on that surface using pand qso that'spoint number one so that means thatlater onif i find some combination of e1 e2 ande3 it's quite possible that it won'thappen to be on that surface and youknow then that means the lookup table'snot giving me an answerso so what's what's with that well ifyou remember when we went from twoimages to threeuh we introduced the albedoso that was one case we're making aproblem more complicated made it easierright we before we only had two unknownsp and q and we ended up with a quadraticso we had two solutions and we said ohyou know ambiguity let's add a thirdunknown and then suddenly we get linearequations unique answerwellthis would apply here as well so ourcalibration object of course is madewith one particular albedo perhaps wepainted it white and so its albedo isbasically onebutsuppose we also want to deal with calwith similar objects where not all thelight is reflectedand so how do we do that well it's veryeasy because rho albedolinearly scales e1 e2 e3and so that means thatanything on array out hereis connected and you know rhorow is zero on this end and one on thatendso suppose we've painted our sphere niceand white and we made these measurementsthen we've defined that uh that shapethatsurface in the spaceand nowthat's for rho equal to one and we canbecause it's perfectly linearfor other rows we can justgenerate thosesowhen we placean entry in the lookup table up herewe can just say okay for now we scanalong this line andwe fill inall of those cubes and now instead ofjust writing uhp and qwe write p q and row in thereso it's a three to three lookup tablenow three dimensions to three dimensionsand you know in a in a real situationmaybe you know that the object issupposed to be a certain color and sohaving avalue different value of rowis uh not really acceptable it indicatesthere's something wrong and that can bepretty useful like you know there's asmudge on the surfaceor it's not actually the surface youwere told it wasso uh it's like a checkan error condition that you can checkforor for example suppose thatthere's something blocking one of thethree light sourcescasting a shadow well that means thatone of these e1 e2 e2e3s is going to berelatively smalland so you'll beaway from the surfaceandif if you're using this method you canpick that up and say oh thisi don't know what p and q is here butthere's something weird going onori know if you remember the slides butone of the other things that can happenis thatifhighly reflective surfaces are close toeach other there'll be interflection andwe will have brightnesses that areabnormally highand will actually be outside thissurfaceand again we can say okaymy method doesn't tell me what thesurface orientation is but there'ssomething going on here and so um i'mnot going to make that part of thesurface i'm going to use that to breakup uh the image into parts that hangtogether and parts that don't so in thecase of those overlapping donuts this isa way to segment itintoyou know you start off the image is onething it's just e of x and ybut you know that it's an image ofmultiple objects and so segmentation isa big problemand if we use these methods we cansegment on on cast shadows where onedonut casts a shadow on another and wecan segment on areas of highinterflection which is where they touchand sowhat looks like a drawback can actuallybehelpfulnow this still doesn't guarantee thatwe'll have filled in all of the voxelsin this three-dimensional spaceso actuallyumthere's a customary way of solving thiskind of problem which isto go the other direction so what we didwas we went to the image and we saidokayfor that pixel we get p and q and wemeasure e1 e2 3 then we put that in thistablethe other way is to systematically stepthrough the table rather than stepthrough the pixels and then for each ofthese voxels find the correspondingthing over thereand that way you can be assured that youfilled in the tablethis this way it's sort of you'reprojecting in a non-linear way fromlattice or cubic lattice onto thiscurvilinear space and there's noguarantee that you won't have overlapsand yet you will fill everything inso but that's you know fairly boring soi won't talk about thatnow i was going to talk about lambert'slunch paperand explain exactly whywhen you cannot see the fatty spot theillumination from the left and the rightis equalbut i'm hoping you'll believe me it'sit's just a page of uh fairly boringalgebra somaybe we'll turn it into a homeworkproblem or something soanyway that was uhlambert's uh instrument for discoveringa lot of things about photometry solet's uh get a little more serious aboutphotometryyou know wewe use these terms like brightness andintensityrather loosely and it's good tomake them preciseso the first term is irradianceand it's a you knowthe most trivial concept you canimagine here's a patch of the surface ofarea delta aand here's some light sourceandthere's abit of the power emitted by the lightsource whichis intercepted by the surfaceandthe irradiance is just the power perunit areaand in terms of you know it's watts persquare meter if you likeand just for referencenoonday sun in washington dc issupposedlyone kilowatt per square meterand you know what what does that meanwell it's it's not a very precise valuebecause it'll depend on the state of theatmosphere and the time of year andwhat is what are you measuring are youmeasuring only visible light uh are youmeasuring near infrared as well and soon but roughly speaking that's a usefulnumber to know because from that we cancalculate things like uh suppose youhave an image sensor that's four micronsby four microns what energy falls onthatand then you put that image sensorbehind the lens which attenuates evenmore you know how much comes out so it'sa very simple conceptand unfortunately it's not terriblyuseful for us because we have an imagingsystem we're not exposing the sensordirectly to the illumination well if wedid we wouldn't learn anything aboutthe environmentnowyou might say well what we're interestedin is how much light comes off so wecould haveperhaps acomplementaryidea whereyou know we have a give a name to thisquantityuh wherewe take the power that's emitted dividedby the areatrouble isyou know that could be going anywhereand and there is terminology for thisquantity but since it's useless to uswe're not going to bother with it so umi mean this is obviously useful ifyou're worrying about you know heatexchange likehow do i keep my satellite cool enoughgiven that the sun is illuminating it onone side and there'sheat going off in intoblack space in another direction and soon but it's not useful for us becausewe're we're not intercepting all of thiswe're over hereuh intercepting a tiny part of that andtherefore it matters that this radiationis notisotropic it's notgoing in all directions equallysowe don't want thatso thenyou know many textbooks will use thistermintensity and we often talk aboutwell i try not to talk about imageintensity because it's really wrongthe intensity has a technical it's atechnical term that has a meaning andwhat is it well it'suseful for point source to talk abouthow much radiation is going off into acertain direction so here's a pointsource i don't know star a light bulbandwe're measuring how much power is goingin a certain directionuh well we need to normalize thatright we need totake this cone of possible directionsand somehow measure how big it isso it'sso we have todefine thismeasure which is called the solid angleand the units of which arestair radiansand in in case you haven't come acrossit beforeit's very simpleso in the plane in 2dyou know the preferred way of talkingabout anglesis in terms of radians andandyou know it's what you get by cuttingthatcirclewith that angle and looking at thelength of the arc of the circle anddividing by the radius so we all knowhow to do thatwell this is very similarin thatin that wetake this cone of possible directions inthree space now instead of in two spaceand we cut it with a circleand we imagine we're at the center of asphere we're at the center of the spherewe're cutting it and we get a certainareaand now to normalize it we need todivide by r squared because this areawill grow with radius square you knowradius squared rather than radius and sothat's the definition of solid angle sothis allows us to talk abouta set of directions it doesn't have tobe a right circular cone could be anyshape and i could have something likethisall that matters is what this area is onthe sphereokaysoyou know radians we go from zero to twopiso what's the corresponding thing forstair radiansso if i want to talk about all possibledirections around mehow many stir radians is thatwhat's the surface area of a sphereyeahfour pi r squared and sothank you and so uh it's four pi rightso this uh you know there we go here wego zero to two pi and this goes up tofour pi so the theset of all possible directions around meso if i'm radiating energy it could gointo a solid angle of four piuh if i'm for example only worryingabout light coming from the skythat's a hemisphere so that's obviouslytwo pi still radians sothere's just one more little subtlething that's kind of handy which is[Music]if that surface area is inclinedrelative to thedirection to the center of the spherein that caseso this is just another manifestation ofthatum foreshortening phenomenon and this issometimes handy because we will havecases where there is an inclination forexample now we're going to be talkingabout cameras and the lens is going tobe tilted relative to a subject that'soff-center and so we'll need to accountfor that and and that's obviously andwhy is that well becausethis area at an angle is equivalentto an area that's at right angle to theaxis and the ratio of these two lines toeach other is cosine thetaand that also is the ratio of the twoareas sookay so now we know how to calculate uhsolid anglesvery handy concept we go back tointensity and intensityiis defined as a powerfor solid angle and you can see thatit's independent of distance so if i gofurther out you know it will cover alarger area but we're assuming thatthere's no loss so the power going intothat cone is the same the further out igoand sothat's a useful quantity and it's a wayof describing you know how um distantpoint source mightum act and usually in machine visionthat's not the case we're dealing withwe're dealing mostly with continuoussurfacesand you know we could imagine breakingthem up into an array of point sourcesbutthatit isn't usually done and it's not veryhelpfulokay so intensitysoif you use that word in your homeworkproblems you willflunk the coursejust just kiddingsince all the textbooks use intensitiesunfortunatelywe have to accept thatthat is an alternate term for somethingelse which we'll talk about next so thethe true meaning of intensity is thisand it's all and it's never somethingthat people in machine vision considerunless they're doing things like tryingto reconstruct the center of our galaxyand finding you know what that blackhole looks likeokayso what is it that we want well uh wekind of had it herewe we don't want to know the wholepower coming off the surface we're onlyinterested in what reaches the observeror the camera and so we introduced thisidea of radianswhich is powerper unit area per unit solid angleright so we have a little part of thesurface herewith an areaand we have a person or a cameraover here and this is a solid angledelta omega and there's some powerdelta p going that way and that'sobviously much more what what we wantbecause that's that's what we'remeasuringit's uhin in the image this power will beprojected onto a certain area and uhour senses are measuring uh powerbasicallywhich which by the way is is unfortunateit'd be great if they measure theelectric fieldbutinstead measuring the absolute valuesquared of the electric fieldand so amongst other things we know thatquantity can never be negativeand so we can't do any phase imagingwith our usualcamerasand there's some there would be somereal advantages to that but we we can'tdo that okayumso that's radiance and it's a power perunit area per unit solid angle orin terms of uhunits it's wattsper square meter per uh solid anglestill radiantnow if you're a mechanical engineer andyou do dimensional analysisthat's kind of a nuisance because theradians like radiance doesn't have youknow units like meters per kilogram persecond or something they're just ratiosand so that clever trick mechanicalengineers use to guess at the answer byjust matching dimensions it doesn'tquite work because you have thesequantities thatare dimensionless sookayso what to do next well what we what iwant to do next is torelatebrightness out there to brightness inthe cameraand so i'm using this term brightness ina very loose wayand we'll justify in a little whilewhile we why that's acceptable so in away we can talk abouta brightness of a surfacein terms of radiancethis is how bright it's going to appearthen in the image planewe can talk about the brightness wemeasure as as irradiance so notice thatbefore i talked about light falling onan object in the world but the sameconcept of course applies in the imageplane i've got these little areas thatare light sensitive and what are theymeasuring they're measuring irradiancewell they're actually measuring energyover a certain time but of course energyof a certain time is the same as powerso they're measuring measuring powerokay so i want to relate those two andwhat we're going to end up showing isthat they'reproportional to each other and thereforewe can be sloppy we can call both ofthem brightness as people actuallyusually do and so we we you know oncewe've defined the being careful aboutthe terminologywe can justify being sloppy about theterminologyokay so uh for this to bemeaningful we need uh finite aperture soso far we've talked about pinhole modeland the pinhole modelgives usperspective projection and it's veryuseful for that but we know that umin terms of image themeasurement is problematic you know ifwe make the pinhole too large the imageis going to be blurredand if we make the pinhole too small wedon't have enough photons to count andalso there'll be fractions so if youmake it small enough it'll appear justlike a isotropic source that's radiatinginto two pi still radians okay so idon't know if anyone else uh thoughtabout this butinnotation of infinitesimals it is sort ofcustomary to do thatright because you're dividing by twoinfinite decimals so this should bei don't know two infinite decimals aproduct of two infinitesimalsnow fortunately we're not going touh get too perturbed by that but that'sa good observationand and you know i guessin textbooksthat'll be the notationand then and then you have someone elsesayingoh what do you mean it's del squared pthat's the power going there what isthat why is it squared so i i didn'twant to go there sookayuhlensesso what we're going to do isinvent this devicethat has the property that it providesthe same projection as the pinhole buthas a huge advantage that it actuallygives you a finite number of photonsand there there'll be you know a penaltywhich is it only works at a certainfocal uh length it only works at acertain distancebut let's uh so let's talk about lensesideal uh thin lensessogauss already showed thatthere's no such thingyou cannot make a perfect lensand butyou can become you can come incrediblyclose to making this ideal object so letme umthe three rulesso uh of course you know that you knowlenses are made of glass transparentmaterial they have a refractive indexthat's different from air and solight rays are deflected when they hitthe surface and that simple lenses aremade with spherical surfacesparticularly easy to grind uh you canplonk uh you know epoxy 100 of them ontoa sphere and grind them all at oncebecause you they all have sphericalsurfaces fancier lenses are asphericaland they're obviously much harder tomake and more expensive but the simpleones are there but for us what'simportant is the following so rulenumber oneso rule number one is a central rayundeflectedso what does that mean it means that ifyou have a ray of light going throughthe center of the lens it comes outtraveling in the same direction on theother sideand not just that ray but any ray thatgoes through the centerandsince time is reversible it works theother way around as well sowe'll just show arrays going in onedirectionand so that's a pretty remarkableproperty and why is that important wellbecause that gives us perspectiveprojection that means that in terms ofprojection this is acting just like ourpinhole sothennumber two is aray fromfocal centeremergesparallelto optical axisso so what is thatwell uh lenses have a focal lengthand let's call that f0so i can define a point here that's onefocal length away from the lens and whatthis rule is saying that ifi take any raycoming from thatfocal centerand i see what it does after it comesthrough the lens is going to be parallelto the optical axis and again of coursenot just that ray butyou know anyany arrayand then rule number three iswell it's really the same thingparallel arraygoesthroughfocal centerso if i have a parallel array coming infrom the rightit's going to go throughit's starting to look a bit messy but soyou can see that if you have a bundle ofparallel rays coming in from the rightthey're all going to go through thatpointthat's what that's where they arein focusandfrom that diagramwe can usesimilar trianglestoget the lens formula andi'm not going to go through it's kind ofboring algebraagain possibly suitable for homeworkproblembut at least i'll draw the diagram soso we have these two special pointsone of which is one focal length aheadand one is one focal length behind andwe have a central arrayand let's see so thenthat's paralleland it's going to go through herenotice how that didn't hit the lensthat's because the simple model doesn'thave an idea of the diameter of the lensit'sit's what happens in this plane itdoesn't it doesn'tyou knowi can get a larger lens to do that if iwant tookay and thenum and sowhat this lens doesis kind of remarkable namely it takesall of the rays that come fromthis point up hereand it magically brings them backtogether again over hereso so that's the good news the bad newsis that if i move this in depth if i goover here it'll no longer be focused inthis plane it'll be focused in adifferent plane we didn't have that witha pinhole we didn't say anything aboutyou know focus or length or whateverso the the upside is that we are nowbringing a significant number ofphotonstogether in order to make a goodmeasurement the downside iswe have this penalty that things can bein and out of focusokaythen whatwhat you can do with this is uhdraw similar triangles and you get threeequationsand if you put them together just theright wayand againit's you knowa page of fairly boring algebra i don'treally want to do thatand also you know you should know thatfrom physics butokay umwhat elsesoas i think i mentioned before we canthink of the lens as this amazing analogcomputerbecause you know rays come in from thisside going in every which direction andit magically figures out where to sendthem toum you know i don't think we reallyappreciate how amazing this is becauseyou know we all use lenses since we werevery small and we just accept that youknow you cando things with them that depend on thispropertynow this is a planar diagrambut this applies in 3d as well whichmakes it even more amazing because nowuh the rays coming from this pointoccupy a whole solid angle you know oneof them is going to come out of theboard and it's going to hit the lensover here and magically that part of thelens will redirect it to to be overthere soand uhi mentioned that actually you can't makea perfect ideal lensandthat's connected to this property thatumeach part of the surface of the lenshas to somehowdeal with rays coming from a whole solidangle of possible directions and so youcan't optimize it for one particulardirectionnevertheless by combining differentlensesyou can getyou know very good performance veryaccurate performanceand they're trade-offs between differentkinds of defectsone of which is radial distortionso umif youthink of an imageand particularly you know think of afisheye lens image wheremore than 90 degrees of the world isimage perhaps as much as 180 degreeswell in order to bring that into theplane you have to kind of squash theouter parts and so in terms of the imagewhat's happening is that the distancefrom the center of the image isdistorted and is made smaller than itwould be with perspective projectionand that's the only way you can get youknow a large anglein a wide angle lens or fisheye lensand that type ofdistortion is actually inherent in inlenses and we're not very sensitive toit if it's relatively small and sooftentimes lenses are designedsuch thatthey will suppress certain defects atthe cost of increasingradial distortion and that's why when wetalked aboutcalibration i mentioned thatunfortunately with real lenses we needto not just find theprincipal point principal distance andthe principal point but we may also needto talk aboutradial distortionokaysonow we're ready to put it together toseewhatthe irradiance in the image isgiven aobject radiance out in the worldso we'll draw a diagram of a simpleimaging system that includes a lensnow you know biological vision systemsdon't have flat image planesbutall of our cameras do and it'd be kindofpeople people have built cameras withcurved uhretinas but it's hard and it doesn'tseem to serve any particular purpose solet's assume as usual that we have aplanarimageplaneokay so thenthere's a lensand there's an objectlet's pick somethingokaysoi'm calling the distance from the lensto the image plane fas opposed to f0why is that well i've i'm using f0 todenote the focal length which is a fixedproperty of the lens and because of thisformula over here we know that forthings to be in focus they have toactually be further away from the lensthan fand sothen f0 and so this is going to be alittle bit larger than f0 depending onthe magnification well the way i'vedrawn it actually it's going to be a lotlarger than f0 butokay so thenwe said that the central arrays areundeflected so the rays going into thelens here come out in the same directionand thereforea couple of things one of them is thatsuppose there is a very narrow cone ofdirections i can assign an angle to thisthat it makes with respect to theoptical axis and that angle is going tobe the same on the two sidesand the other thing i can say is thatthese cones of directions have the samesolid angle because they're the samerays just turned aroundokay[Music]there's a small patch in the imagethat's being illuminated i'll call thatdelta iand there's a small patch on the objectthat's being illuminated andwe're going to see how much power comingoffthat patch ends up in this patch thatthat'sthat's the kind of thing we'll belooking atokaybut firstlet's equatethose umtwo solid anglesi'm trying to relate these two areasthen in order to do that i need to takeinto account the foreshorteningso there's a unit vector on the surfaceof the objectand what's well this also isnotimmune to that effect because the lightis not coming in perpendicular to theimage sensor the light is coming in atan angleright if i draw a surface normal to theimage planethenit's not the center of this cone of b ofraysand what is the anglebetween the surface normal to the imageplane and the incomingrays heresoremember that diagramuh whereyou know put these angles are the sameand these angles are the same so overherethis angle and that angle are the sameso that alpha affects theincident light onto the image sensor aswellokay so now i can write down i can usethat formula i had for solid angleso on on this side the area is delta iand then there's a foreshortening effectcosine alphaand then i have to divide by thedistance squaredso that'sf squaredormaybenotit's bigger than f squared rightsoit'sf secant alpha squaredhere's theso this part is fthis is alpha and i'm measuring uh thislengthright soyou can imagine if i draw this furtherout then that's going to become evenmore obviousokay and now thatsince we said that the central rays areundeflected this cone of directions isthe same as that cone of direction so ican just equate that toand it's the same thing on the othersideokay and fortunately those secantscancel outsolet's see i want delta o over delta i ordo i want the other way aroundokay so that's half of the story thatwe're making good progress hereuhand that's important becauseyou know the the total energy coming offthat patch depends on how big it is andthen that energy gets concentrated intothat smaller patch in the image andso the irradiance in the image is goingto bewhatever power ends up here divided bythis area delta iand so we'll be able to relatethe radians over there to the irradianceover here and this is what we measureit's the irradiance there okay so nowwe have to think abouthow much of the light from that patch onthe surfaceactually is going to be concentratedinto thatimageright sothe good news is that becausethe lens focuses the raysall i need to know really isthe solid angle that the lens occupieswhen viewed from the object right so ineed to know what the solid angle isbecausethe rays that come out the other sideall get concentrated into thecorresponding patchif things are in focusthe light that comes off this patch andgoes through the lens is allconcentrated into this area in the imageand converselylight coming from anywhere elsehas no effect on on this it's imagesomewhere else so so there's a directmatch betweenthepower coming off here that makes itthrough the lensand the power that's delivered to thatsmall area in the image okay so what isthis wellit's the area of the lens let's supposethe lens has a diameter dso it's a four pi squared and thenthere's a for shortening effect there'san anglewhich we drew as alphaand then we have to divide by thedistance squaredand that again isfsecant alpha squaredso so that's the solid angleandtypically that's actually quite smalland so wetypically only gather up a relativelysmall fraction of the uhoh sorry not f zand now this time the cosines andsecants don't cancel out unfortunatelyand and we get thatokay so almost doneso the uhpower delivered to that small area inthe imageis the radiance of the surface times itsareatimes the[Music]solid angle times cosine theta andthat's going to be ldeltao pi over 4d over z squaredso that's the total power we'redelivering and it's concentrated into anarea delta i so the power per unit areais we just divide through bydelta i and that's that's what weactually measureso that's theconclusion of thatso let's let's study that a little bitcarefullylet's look at this d over d over fwhat is thatormaybe f over d looks more familiari guess theno amateur photographers herethat's the f-stopthat tells you howopen your aperture isisf l dand soyou know typicallyslrs will have a maximumopening of maybe i don't know 1.8 forthat ratio and you can stop it down to idon't know 22 let's sayand so obviously the square of thatcontrols how much light you get andthat's one way of controlling theexposure the other one is is timeand so often there are trade-offsbetweenusing theaperture openingversus using time you know for exampleif you wanta lot of depth of fieldthenyou can achieve that bymaking the aperture very small nowapproaching a pinholebut the cost is you need a longerexposure so if things are moving that'snot going to work conversely if you wanttoyou know have a great portrait and youwant to wash out the background out offocusthen you go the other direction you openthe lens very wide so thatit only has a narrowfield of depthand uh then use a very short exposureanyway so so that anyway that quantityis one that's well known to peopleworking with cameras and it's uhintuitive thattheimage irradiance goes as the inversesquare of the f-stopand that's why the f-stops are usuallydone in square roots of two so it goes 11.4 23875.92 45.6 8 11 16 22 32and those are steps of square root of 2in size of the aperture which give youuh steps of two in uh exposureokay so that's that so that's you knowkind of intuitive and not particularlyinterestinguh the pi over four uh who cares it'sjust a constant we're not too excitedabout thatwhat's what's really exciting though isthat the thing we measureeimage irradianceis proportional to l the thing we'reinterested in out in the world theradianceso that's why we can be sloppy abouttalking about brightness because thebrightness of the surfaceradiance is proportional to thebrightness in the image irradiance soyou know we're measuring brightness inthe image and that has a meaning beyondjust power per unit area in the imageplane but it has a meaning out there interms of how much that object isradiatingand the remaining part is this uhannoying thing hereum cosine to the fourth of alphawhat does that do well it means that thebrightness is dropping off as we go offaxis so if you have apart of the image that's way out on inthe corner hereit's going to receive lesspower per unit areathan something in the middle of theimageand that means thatyou need to take that into account nowunfortunately for small alpha cosine tothe fourth alpha is as close to one asyou can getand soas long as alpha is small you cancompletely ignore thisso when does alpha get big well it onlyreally gets big when you have a wideangle lenswhen you're taking in a largesolid angle of the worldand thenyou know alpha maybe20 30 40 degreesand then it starts tomatter because cosineof 40 degrees is not any more close toonefortunately it's a fixed thing so youcan compensate for itand in factyou know if you buy adslrpart of the magic that happens thatyou're not allowed to see or understandis to compensate for this[Music]also we're not very sensitive to thiseffect so if you have an imagethat sort of slowly gets darker towardsthe edgeyou really have to focus on it to tonotice it sowhen i was much younger i was very keento have a telephoto lens but i couldn'tafford a telephoto lens and then i sawuh advertisement fora russian uh casa dioptic uh telephotolensand you knowhaving worked in telescopes that rang abelland i could afford that once i got itand it was quite nice and then i lookedatthose days we used slides and projectedthem on the wall so i'm looking at thisslide ofpredatory bird that i took looking up inthe tree and the sky background behindit and this guy behind the bird is whiteand i'm like okay this is oddand strangely in the corners the sky isblueand you know so what's going on therewell what's going on is that this lenswas cheap in part because it had a rapiddrop offin brightness with the angle perhapseven worse than thisandso the corners were not illuminated aswell as the centerand in the center there was enough lightso that these um the colors were oversaturated soyou know not onlydid i get the umthe green and the red channelsoverexposed but i even got the bluechannel overexposed so it just lookedwhite it you know it hadmore than the maximum intensity that thefilm could handlein all three channels anyway sothis comes up a lot in other situationsin x-ray imagingit's slightly different it's cosinecubedbut it's something that people oftenforget about and as i said you can kindof forget about it because it'ssomething you can compensate foryou just in your image processing chainit doesn't change from image to imageunless you change the lens andand the uhyeah okaysoso that this formula is uh centralum not because we're going to play a lotwith it but just becauseit justifies uh this whole idea oftalking about brightnessand measuring it using gray levels inthe image andthinking that that has something to dowithwhat's out in the real worldokay uhso that thatthat's ourdescription of what the camera does interms of brightness so this is thecounterpart towhat it does in terms of position whichwas the perspective projection equationwhich of course was trivial incomparison but those are the two keyuh you know cornerstones ofunderstanding of camerasokay so now that we know what it iswe're measuringyeahwell we already knew it was the powerper unit area we're measuring but now wealso know that that corresponds toradians in the worldwe need to try and understand whatdetermines the radians in the world andwe already mentioned it depends on theillumination it depends on the materialit depends on orientation so let's tryand make that a little bit clearerandand talk about thebi-directional reflectance distributionfunctionso let me tell you that untili don't know the the 80sthis field was a complete mess and therewere dozens of different terms allcompeting for attention some of whom hadfamous people's names on them and somedidn't andand thenthe national bureau of standards steppedin and a brilliant man namednicodemuscleaned it upandwho knows maybe the last thing thenational bureau standard did that wasvery interesting but it was a verypowerful effect on imagingnot just optical but x-ray as well andso what is this wellthe idea is thatwe have light coming inand we have light going outandyou know crudely speaking when we talkabout reflectance the ratio of those twothings is reflectance right so socruelly speakingsomething that's white reflects all ofthe light coming in something that'sblack reflects none of itbut you know where is that light goingso it's not as simple as just saying ohreflectance is4322.80 it's more nuanced and and that's whatthis is about so this basicallyis a quantitythat depends on the incident and theemitted direction and it tells youhow bright the surface will appear so solet's uh look at it so we have a patchof the surface a usual diagramshould have a rubber stamp for thatandwe have light coming inand we have light going outandhow how bright the surface will appearits radiance will depend on those anglesbut moreuh because this is in 3di've drawn it on a plane but it's ofcourse really in 3d so i really need totalk about the directions of these tworaysin more detail than just saying theymake an angle with the surface normaland so how do i talk aboutdirections well we've already beenthrough this unit vectorspoints on the spherelatitude longitudeso in this field it's customary to useumthese anglesso let's suppose thatthis line down this curve down here isis in the surface and i've constructedthe hemisphere above itand this is called the polar anglealso called cold latitudewhy because it's 90 degrees minuslatitudeand this is called azimuthazimuth anglesoto specify the direction of light comingin or light going out i need two anglespolar angle and azimuthand so here i've only shownthe polar angleand to draw the azimuth i'd have toproject this down into the plane of thesurfaceand then look atthe direction of those linesandthensince the brightnessis going to depend on all of thosei can write it as some function likethis andthe official terminology is that's thebi-directional reflectance distributionfunctionand as we said uh reflectance should beyou know power going out divided bypower going inand so that's that's what this is youknow clean and formalized so what we'vegot is it's the radiance you know howbright the object will appear whenviewed from this positiondivided by how much energy i'm puttingin from the source directionso so this isfinallya sense uh definition of reflectancethat actually worksright good because the other definitionof just saying oh it's 0.3 or somethingdoesn't unfortunately this is much morecomplicated butbut any other definition of reflectancecan be based on this basically it's anintegral of thisokayso umwell there are lots of questions thatimmediately come up one of them is youknow how on earthdo you measure this thingwellone way you can do it isyou know put the light source in acertain position put your camera in acertain positiontake a measurement then move your camerablah blah blahmove your light sourcebut you're exploring a four-dimensionalspace so that's pretty expensivenevertheless it's done and one onemethod is using goniometersbut just a fancy way of sayingangle measurement devicessosowhat i'm trying to draw here issomething that will rotate about thisvertical axisand then has aan arcalong which the apparatus can moveokay sothe first rotation corresponds to theazimuth angleand the second movement corresponds tothe polar angle so the pole would be uphere i canand so we can mount the light sensor onone of thesethen we get a second one and we mountthe light source on it as welland uhthen we make measurements well you'llget you know tired of it pretty soonbecause you're exploring this fourdimensional space and even if you sampleit pretty coarsely like you know one inten you're not talking about 10 000measurementsbut of course you can automate it youcan you know build arobotic device thatmechanically moves this and you know goaway for the weekend and it'll do thesemeasurements for youif you're in a hurry as people inmovie making areyou can have many light sources that youcan turn on and off so you couldconstructmaybe a whole sphere or maybe ahemisphereand you distribute light sources allover thesurface that can be individuallycontrolledand then if you want to so that takescare of two of the anglesso right so by pickingby turning on one of these light sourcesyou've controlled one of the goniometersto position the light source and thenyou can have light sensors or perhapseven camerasinterspersedand now you can do all of yourmeasurements very quicklybecause you justflash one source at a timeand take pictures and thenprocess the images you gettothen why would you want to do this wellsuppose you want to realisticallymodel how someone's skin reflects lightwell you could try and build somemathematical model but who knows howgood that is the better way is just tomeasure it andso that's something people do you don'twant to approximate it with i don't knowlambertian or some other well-knownmodel you you want the real thing and souh you you can do thisumso that's part of the story of thisfour-dimensional space the next part isthat well in most cases it's notfour-dimensionalokay and i think if youlook at this diagramyou can see that what really mattersisthe angle between these two azimuthlinesso this is a general formula thatapplies to any surfacebut for many surfaces what really onlymatters is the difference between thesetwo angles and what type of surfaces arethose well those are surfaces where ifyou rotate them in the planeyou know perpend if you rotate themabout their surface normal they don'tchange brightnesssothat you know it's pretty pretty muchtrue of this thing and it's pretty muchtrue of of lots of things like you knowwood andum this floor and so on so that's adramatic improvement because now you'vegot a 3d lookup table to fill in insteadof 4dsowhat kind of materials are there that donotsatisfy thiswhat materialsrequire that we take the fullfour dimensions into account whereyou know and think about it is there asurface where if you look at it and yourotate it in the plane of the surface itchanges appearance can you think ofsomething like thatsomething that's iridescentwhy because an iridescent material hasmicrostructure that's orientedand you know like a hummingbird'sneckthe feathers are lined upit produces the color not by pigment butbyinterference and so if themicrostructure you know with my fingersand then i rotate that microstructureit'll diffract light differently so thethe uh our ruby hummingbirds if you lookat the neck of the male from the wrongangle it looks blackwhy because it's reflecting all of thelight in one direction and it's and it'sonly the red so so that's an example ofsomething where unfortunately you needthe fullthe full model you can'treduceand you uh use this so and the otherexamples like thesemi-precious stones called tiger eye orvarious other things they're basicallyvery fancy forms of asbestos and youpeople don't people don't call them thatbecause you know as soon as you sayasbestoslawyers come and so on but it'sbasicallyjust as asbestos it has a microstructurethat's very linear and very tightlypacked on the scale of the wavelength oflight and so it has a very differentappearance as you rotate it which givesit its appeal as a piece of jewelrythen you know some people havevery straight black hair that's veryparalleland that will have the same effect wherebecause they're all parallel they'llreflect light in a certain way and so astheir head rotates the sheen on the onthat hair will will move and so ityou need the full four-dimensional modelfor that whereas for a lot of thingslike paper and snow and you knowstrawberriesthe three-dimensionalmodel is issowhat else do we know about thisbi-directional reflectance function wellthere's an important propertydue to helmholtzcalled the helmholtz reciprocitynow helmholtz of course lived a longlong time ago but long before nicodemusof the national bureau of standards sohow could he have come up with aproperty of the bi-directionalreflectance distribution function wellhe didn't call it that but he had thebasic idea which was you know basicallysecond law of thermodynamicsso suppose we have two objects atdifferent temperaturesand there's an object patched down hereand so there's radiation coming from oneand going to the otherand reciprocally there'll be radiationcoming from t the object at temperaturet2 andarriving at t1andit takes a little bit of hand waving butbasically what it's saying is if it'snot a reciprocalthenthen there will be energy transfer fromthe colder object to the hotter objectwhich we know doesn't happenso what this is saying if youinterchangeif you interchange incident and emittedyou should get the same value for yourbi-directionalso there's a symmetrywhich in a way helps you in the datacollection because there's half of thedata you don't have to collectby the way it reminds me ofwhen i was a student herethere was a famous professor calledletrin and he jerome litwincame up with a papercalled what the frog's eye tells thefrog's brain which was one of the earlyattempts to try and understand howneurons work and how image processingmight work and so onand you know everyone wanted to hear histalk and he was talking about color andi was way back in the room and asas a student i was really intimidated bythese people but i just had to ask andsoi stuck up my hand and i asked him somequestion iat this point don't even remember whatit was and he stared at me for aboutfive seconds and then he saidwell if you had read the original bookby helmholtzthebook by helmholtz in the original germanversion you would know blah blah blahand i'm like oh my god i reallykilled my career here becauseand then years later i was thinkingabout that same problem and i realizedthat he was just using a wonderfuldebating techniquebecause he had no idea what the answerwas but he certainly put me in my placeand you know and some of these peoplewent to schools where they taught youhow to debate i didn't so i i was i wasjust flabbergasted anyway helmholtz sothat was how i'm all thereokayso next time what we're going to do isum apply this to[Music]look at different types ofsurface material models of course usinglambertian againand also some new ones that apply tothe moon and rocky planets in our solarsystem and that uh enable us todetermine their uh surface shadeokay and i guess there you there was anextension rightso make sure you keep up to date onwhat's on piazza because there was anextension on the homework problem andother good stuff happening there

Shading, Special Cases, Lunar Surface, Scanning Electron Microscope, Green's Theorem

let's have a quick review of what welearned about photometryso there are a number of conceptsone of which was irradianceand we use the symbol e for it and itwaspower per unit areaand it's a way of talking aboutlight falling on the surfaceand it's what we measure in the imageplaneand convert towhat's commonly called a gray levelso thequantity of interest hereis directly used when we'reimaging but it's also of course ameasure oflight falling on the objects that thatwe're imagingthenwe talked about intensitywhich applies toa point sourceand itdescribes the power per unit solid angleand sowe had to define the solid angleand it's a quantity that typically uhvaries with the direction soif you have a good old incandescentlight bulb it's very low intensity inthe direction of the bass because that'sblockedby the bass and some higher intensity inother directionsandthat's a quantity that isn't of a wholelot of interest to us here it's justinteresting because a it's simple todefine and bit's the terminology incorrectly used totalk about the quantities that we'rereally wanting to talk aboutso theimportant oneis radiancewhich is basically a measure ofhow bright a surface appearsso again we have a littlefacet on the surfaceandand we're looking athow much power is emitted per unit areaand per a unit solid angle and that's ofinterest to us becausethat's what we actually measure uh withour instrumentscamerasand that's alsoobviously relevant to uh what we seethat small solid angleis the perhaps the entrance pupil toyour eyeokaysowe then looked at cameras and anythingwith a lens in itand we came up withwith this relationshipbetween the radiance of a surface thatwe're imagingthat's landthe irradiancee of the corresponding part of the imageand so it gives us a direct relationshipbetween something out thereloosely called brightness and somethinginside the camera loosely calledbrightness and the reason we can beloose about it is because they'reproportional to each other soumand then there's the pi over four whichis just a constant factorand then there's thisone over f stop squaredwhichis kind of obvious because we'relimiting the solid angle the d omegaover there by uh opening or closing theaperture on the lensandthe area of that goes as the squareofthat ratio and it's the area of coursethatwe need when talking about the solidangleokaysothen the next question is okay we'remeasuring e and it's proportional to lbut where's l come from what whatdeterminestheradiance of a surface and wealready indicated that well illuminationit's going to be directly proportionalto the amount of illuminationand it's also going to depend on thegeometry so how is the surface orientedand it depends on the materialand that's where webi-directional bdrfandthat's where we introduce thebi-directional reflectance distributionfunctionwhich is a function of the incidentdirection and the emitted directionso we have light coming into a surfaceandwe have light re-emitted from thatsurface and that's obviously the idea ofreflectance how much of that light goingin is reflected except it's not quite assimple as that it's not simply a ratioof you know what percentage of theincoming light is reflected but we'reinterested only in the light that'sgoing to hit the camera or the eye sowe're actuallyusing thisterminology so it's going to bedelta elet's see delta lof theta emittedso this is the radianceof the surfaceand this is the irradianceand so so that's what you imagine somedefinition of reflectance to be and it'stheyou know detailed fine graindefinition of reflectancefrom which we can derive otherquote reflectances for examplealbedo which is the total output powerdivided by the total input power well inorder to compute that we just take thisquantity and we integrate overall possibleoutput directionsbecause in this case we're interested inthe total power going out not justwhat's going to a particularlight sensorand in the process we may need to thinkaboutspherical geometriesokaythen we said thatthis quantity this brdfhas to satisfya constraintwhich basically says if you interchangethe directions to the source and thedirection to the to the viewerthebrdf should come out the same and that'sbecause if it wasn't then we'd beviolating the second law ofthermodynamicswhich periodically people try and do butgenerally don't have too much successwithokay and so um so we can't just have anyold function thereby the way uh in computer graphicsobviously they uh use models of surfacereflectance and uh quite a number ofthose models violate this constraintand yet you know we we don't seem tocare we like the picturesum which suggests thatthis isn't something critical to youknowhuman or machine vision other thanit's kind of a shortcut if you'vemeasured one of them then you've got theother one soitit cuts the number of measurementsyou need to takein twoyou know because you can just bysymmetry find the other oneokayso examplewell we've been talking about lambertiansurfacesandthe lambertian surfacehas the property thatit appears equally bright from whateverviewing directionyou haveand if it's an ideal lamborghini surfaceit also reflects all the incident lightsoproperty number oneand this is thecondition that's usually misstatedin terms of umemitted energy so it's usually statedincorrectly as it's emitting lightequally in all directionsso that's going to greatly simplifywhatever formula we come up with for theliberation surface because it's notgoing to depend on two of the fourparametersand then the other conditions is that ifit's an ideallambertian surfaceit reflects all lightand and doesn't generate any of its ownsookay soumas i indicated a lot of work with thebrdf the brdf is sort of the atomicthingit's the low level detail and in manycases we're interested in integrals ofthat so for exampleif i don't have a point sourcei have a distributed source like thelights in this roomhow can i deal with that well i cansimply integrateovera hemisphere of incident directionsright so i'd umintegrate over that quantityuh taking into account you know how muchlightis coming from each directionsosimilarly herewe need tointegrate over a hemisphereto get all of theenergy that's coming off the surfaceso we hadthisway of dividing upusingso we use the polar angleand thenthere was also an azimuth angleso this is one way we can talk about thepossible directions two parametersand if we perform the integral we needtotake into account the area of this patchwhich is obviously going to involvedelta theta and delta phibut it's also going to get smaller thecloser we get to the poleand since we're measuring theta from thepolethis would be sine thetait's kind of unfortunate they didn'tpick latitude but they pick a colatitude but whatever we can we can dothatokay sonowwe're dealing withwe're trying to integrate over allemitted directions so in this particularcase we're talking aboutthose quantitiesokay so[Music]um azimuth well that ranges of atwo pi rangeso we're going to be integratingfrom minus pi to plus pi for exampleuh the polar angle uh well we're notinterested inpoints below the horizon because theobject itself is blocking it's not it'sonly emitting above the surfaceso we only have todeal with zero to pi over 2 for thepolar angleand thenwe're going tohave to includethis term hereand you know but that'sobviously the jacobian the determinantof the jacobian of the coordinatetransformation but i find it easier justto draw the the diagramokay and then what what's in here wellthere's uh there's fand now we've decided that f isa constantsolet's justdo thatjust write fum and thenthe light that's falling on the surfacedepends onthe incoming radiation and the angleandthatwe're saying that all of that getsreflected soso the light's coming in at a certainangle there's foreshorteningso the power deposited on the surfaceis e cosine theta itimes the area of the surfaceand we're saying thatthat's all going to be reflected so whenwe integrate the reflected light whichis uh the brdftimes this quantity then that shouldequal the incoming light so we can justcancel it out convenientlyso what we're looking for is you knowwhat iswhat is this constant value of f for theinversion surface uhis it one you know or some otherconvenient quantityso first of all um theazimuth angle doesn't appear anywhere inthe integrand sowe aregoing to evaluate this quantity and thenjust integrate over phi2 pi so that's just 2 pi times thatinner integral so and i have 2 pi times0 to pi over 2intono we already dealt with thatum ifwell actually we can take the f outsidebecause it's a constantokaynow this issine2 theta eand so if we integrate that we getminusminus one half cosinetwo theta eand the limits are zero to pi over twoso we plug in pi over 2 we get cosine ofpi which is minus 1 minus 1 times minusa half is a halfand then we subtract what we get byplugging 0 in herecosine of 0 is 1and so we're going to subtract minus 1which is like addinga half and so this whole thing comes outto be 1.and so the result is that f is 1 over pisothat's it for lambertian surface that'sthe brdf for the inversion surface andthat'sas easy as it can getand um[Music]there's some question about why it's oneover pi and one over two not one overtwo piumso let's think about thatso if you think about the spherehemisphere of possible directionsso here's our surface element and it'sradiating into all these directionsandwhat is the solid anglethat's occupied by that hemisphere welltwo pi of courseso the object is radiating into thehemisphere that's above its level abovethe plane through the surfaceand that's two pi and if we wereradiating energy equally as aisotropically in that into thathemisphere then f should be one over twopiand you know sothose people who say it's uhradiating equally in all directionswould end up with one over two pi uhfour for thatso what's what's wrong with thatwell what's wrong with that is thatum it appears equally bright from alldirectionsdoesn't mean it's radiating equally inall directions so imagine that you're onthe surface of the sphere and you'relooking in at this objectthere's going to be foreshortening so ifyou're straight above it you see itsfull areaif you're off at an angle you see anarea that's reduced by the cosine of thepolar angleand so what does that mean well thatmeans that if you emitted the same powerthen the power per unit area would begrowing and when you get tobe on the horizonyou know now the area of that surfaceelement is pretty much shrunk to nothingbut you're still radiating the samepower supposedlywell that means that the power per unitarea is infinite and it will fry yourretina so you don't want thatand that's exactly that's not what itdoesit is radiatinglessin this direction than in the in thatdirectionso but it's in proportion to the areaso that the power per area staysconstant so it appears equally bright soso that's condition number one uh itappears equally bright and so that meansthat actually it's uhradiating more up hereand less down hereand in just such a way thatwe end up with half we end up with oneover pi instead of uh one over 2 piso so again the idea that the lambertiansurface radiates equally in alldirections is wrong and it will give youthe wrong answer hereokaynow how do we use thiswell let'syou know simple casenotice that there's no cosine theta i inhereso what's with that you know we've madea big fuss about lambert showing that itdepends on the cosine of the incidentangle well that's becausethat controls the foreshortening of theincoming radiationsosuppose thatwe havea distant source of radiation and thatit has an irradiance perpendicular toits rays ofe naught watts per square meternowwe're illuminating that surface and ofcourse that surface is has an areathat's larger so if we call this aand we call this area a primethena primea is a primecosine theta iright soso that means that this capturesa smalleramount of the incident radiation than itwould if it was oriented perpendicularto the surfaceso we find thatsoif we measure ourincoming light in terms of surface areathenl is 1 over pi times thatpower per unit surface areajust as you know you'd expect there's aone other there's a brdfif insteadwemeasure it relative to theincoming radiationperpendicular to the direction of thatradiation we have to take into accountthe foreshortening and then we get thefamiliar uh expression for for lambert'slawand so umso that's alittle thing that you have to keep inmindand avoid confusionhere's an example of how you might getconfusedhelmholtz reciprocityokay you look at this formula and yousayoops there's no cosine theta e so itdoesn't satisfy helmholtz forreciprocity so it's not a physicallypossible surfacebut the helmholtz reciprocity applies tothe brdf not not thatand here this isuh obviously if i interchange theta iand theta e it's the same it's one overone over piso uh we have to be a little bit carefulwhen we ask questions about helmholtzreciprocity for exampleyou know this is this is a perfectlyvalid formula but that's not the onethat you want to apply helmholtzreciprocity toit's insteadthebrdf the underlying brdnokaysothat's lambertian which is really simpleand we should havesome other examples so let's seeso let's try thisso this is another exampleso i'm not picking this you know totallyat random we're going to use thisparticular type of surface quite a bitso might as well introduce it at thispointsofor this one the brdfumisn't the constantit'ssomething like that it's aone over the square root of cosine thetai cosine theta eandin that form we can immediately answerthe question you know does this type ofsurfacesatisfy um helmholtz reciprocitywell yeah if you interchangetheta i with theta e you get the sameanswerokaynow when weusethis model in practicewe're adding illumination and looking athow bright the surface will appear undercertain illuminationand so we we do what we did over thereright sothe radiance is going to be theirradiance times the brdf and it's theirradiance in terms of power per unitarea on the surface patch and that'sgoing to be affected by theforeshortening because when it's tiltedit's going to receive less power okay sothat's interesting because that's nowgoing to beso here we have a surface thatacts quite differently from a lambertiansurfaceinstead of having cosine theta iwe have this this funny ratioand soit turns out thatthis type of behavioris what we find on the lunar surfacewellmore specificallythearea the dark areas the mars wherevolcanic eruptions have occurred to fillin the basinsbut actually uh rocky planets in generaland you know asteroids sosome asteroidsum and it's not a bad model for them andit's um significantly different from umyou know lambertianand by the way this isthis was the basis of the first methodsfor recovering shape fromvariations in brightnessokaynowlet's see if we can learn somethingabout this type of surfaceso one question is if we look at themoonwhat are the isophotesnow of course we know that you know thelunar surface has some texture on itand you know rays ejecting from cratersthat are brighter than the backgroundand so on but let's pretend that thelunar surface was pretty much uniform inits uh reflecting properties and whatwe'd like to know is you knowwhat is thesort of contour map of brightness now ifthe if it was a long versionwe know thatumall of the points that are the sameangle from the sunthat where the surface normal has thesame angle with respect to the sun havethe same brightness you know cosinetheta iand soif wewere tolook at the isophotes of a sphere on thesphere uh they'd all bethey'd be nested circlesand then if i project them into theimage planeum those circles are at an angleso the circle gets uh turned into anellipsedepend and eccentricity depending on youknow just how much of an angle so thisis what i'd expect to see and this iswhat i would see if say i took thatcalibration object that you know thatsphere painted white in the lab uh ifyou plot its isophots they pretty muchuh look like thatsothat's for the inversion so what aboutthis other materialwell that's a little bitmore trickyso let's uh see how we can do thatnow with a laboration we could just saidcosine theta i is a constant and thatmeans theta i is a constant and then youfind all the places wherethe angle between the direction to thelight source and the local surfacenormal is the sameand you just you know spin that aroundto get a cone and you're done so thisone's a little bit harderso uhsowe're doing it for this one and sohere we havel is constantforso we're now looking forall of the points on the surfacethathavea certain ratio of cosine theta i overcosine theta ewell we can write this in terms ofunit normals that might make iteasier to see what's going on so cosinetheta i is the unit normal dot productwith the light source direction andcosine theta e is the unit vector withrespect to the viewing directionso we're now looking forall values of nthat make this a constantand sofor the constant c that's what we getand nowand so we have a dot product that'sequal to zero that means we have twovectors that are perpendicular to eachotherand soso let's fix uh the constant c for themomentthen then this is some fixed fixedvectorand this is saying that uh all of theends that satisfy that all of the endsthat have thesame brightness uhmust be perpendicular to that vectorsowhat is theset of vectors that are perpendicular toa particular vector what what does thatlook like what's the locus ofthe endpoints of those unit vectorsso we havesome some vectorand then we're saying that n isperpendicular to thatthat's one n well here's another onerightso we actually get a planeso thatif we think of the unit vectors of allthe points on the surface that have thesame brightnessthey all lie in a plane so that'salready uh useful informationwhich isn't the case hereyou know because the unit vectors thisunit vector points up towards the pole alittle bit and this one pointsin in a different direction so so that'salready very uh non-lamb versionwell umwe're not going to do all the detailsbut we canbenefit somewhat fromthinking about spherical coordinatesto figure this one outnow again it's sad that they picked apolar angle instead of latitude so itlooks a little different from your usualformula but uh of course it's reallyjust the same thingwithnine you know subtract from 90.soi can always write a unit vector in thisformyou know unit vector has only twodegrees of freedomandi've picked thepolar angle and the azimuth hereas those two convenient uh parametersokay so what i'm going to try and do ismake someadvance in understanding this bysubstituting forall of themin that form now i'm going to end upwithyou know kind of a algebraic messunless i pick my coordinate systems welland so i happen to knowthathere's a good way to pick the coordinatesystemso here's the direction to the sunhere's the direction to the earthwhere the viewer isandand usuallywe think ofthe north pole as beingyou know up above this plane rightangles above that plane and this isn'tgoing to be perfect becausetheplane in which the moon orbits aroundthe earth is not exactly the same planeas the plane that theearth moves around the sun but let'spretend it's the same okay so we'regoing to pickthis preferred coordinate system whereyou know this is the z directionand so the sun and the earth is at zequals zeroandthat means that when we write thevectors for themwe can leave out the third partsoso their position depends only on theazimuthumsince we've picked this coordinateconvenient coordinate system so it's alittle bit simpler it's only that oneunknownand the third component is zeroand okay so now we go back toour expression for these normals andi guess i can write itandi suppose i do need the other boardso before we do that by the way there'ssomething you can alreadyascertain right now which is whathappens at full moonwell at full moonumthe earthfrom earth you're looking at the moon inthe same direction as the incident lightfrom the sun right so that means thattheta i is the same as theta eso that means that it'sconstantwhat does that mean that well that meansthat thedisk ucshould be uniformly brightaside from the surface markings that wediscussed and that's completely normalinversionright if we hadlambertian surface sphere illuminatedfrom the same direction as the vieweryou know you're holding the flashlightnext to your camerabut then we expect to see isopods thatlook like this because the incidentangle you know here at zero and then itincreases to 90 degrees so the cosine ofthe incident angle goes down and so ifwe're looking at a sphere and it hasisophotes like this we recognize it hasa three dimensional shape and it's kindof sphericalokayand all of a sudden here that's not thecaseand so umthat's pretty interesting because nexttime you look at a full moonyou realize that it doesn't look like aballnot i mean you know intellectually thatit's well unless you belong to the flatearth society you probably believe themoon is flat as well but uhleaving that out uhyou know it doesn't really look roundokay you can see that it must be sort ofround because the outline of it is acirclebutit doesn't look quite right and this iswhy because it in factin opposition at that timeispretty much uniformly bright and youknow this is it it's because it's notlambertian it's ait's a different uh microstructureand the hakka model is a pretty good oneuh for predicting that that kind ofthingokay well umlet mekind of jump from that uh so we we'regoing to haven dot suh which way around did i have it uhn dot s overn dot v is a constantand now i can plug inum the dot you know thespherical coordinate versions of thosevectorsandleave out a couple of steps what i'mgoing to get isa bunch of terms cancel and we end upwith thatnowunlessokay of course if theta s is the same astheta d i e opposition then we just getone andbut suppose that we're notuh then um this can only be true if thatthat is trueso so what does that mean well it meansthat all of the points on the surfacethat have the same brightness have thesame azimuthright soin in our coordinate system that meansthathere'shere's a great circlenot drawn very well butthat has a fixed azimuth fixed angle youknow if you go into the centerand we look atthis direction that's that's the samefor all of them so that's one isofoatand here's another oneso the so the lines ofso the lines of constant longitude areisophodesand that'sagain very different from lambertianand um[Music]it makes themoon look look oddjust thinking of something thatwhen i was a little kid we went for awalk in the black forest in germany andthe moon was just rising and the adultsyou know thought they'd have a littlejoke at my part and i said well how faraway do you think the moon isand i'm like okay if they ask me that itmust be much further away than i thinkso i said i don't know a hundred metersand they all loft soanyway so it's hard toestimate uh properties of celestialbodiesfor example i already mentioned that wewould besurprised to know that most people aresurprised to know that the reflectancethe albedo of the moon is about2555.44 which is the albedo of coal and yet itlooks so bright in the sky and that'sbecause we don't have any comparison wedon't have anything near it and all wecan measureis the product of incident light timesreflectanceand then we try to separate that intothose two components now in our ownworld often we have that theillumination is more or less constant ofan areaand so we canseparate changes in reflectance fromchanges in brightness particularly if wehave some calibration objects like apiece of white paperyou recognize that you say okay that'sone and everything else can be measuredin relation to that but if weonly see the product of illumination andreflectanceit is totally ambiguous we don't knowwhether it's dark becausethe illumination is weak or because thereflectance is low and so onokaysoum[Music]we can go a little bit further with thisyou know for example we might say wellsuppose we take a pictureof the surface under two differentillumination conditionscan we find the orientation the surfaceorientationandyes we canand it's just photometric stereo the waywe've we've done it beforeandbut then the remaining question iscan weget the shape of the surfacesolet's uhlet's talk about that a little bitsothis hopkin modelwe've looked at it here in terms of theangles but we also know and the unitvector we also know that we can use thegradient as a way of talking aboutsurface orientation so let's look atwhat this is like in terms ofsurface orientationby the way you may wonder why there's asquare root i mean it would still be[Music]um well it's it's partly because we wantto make sure thatwe satisfy helmholtzright because if it wasn't the squareroot then when you divide by cosinetheta i you get i don't know one overcosine theta e which is not symmetricalso that that wouldn't workand the other reason is that you wantthe integral of all the outgoingradiation to equal theincoming radiation and thatdoesn't work if you don't have thesquare root it blows it becomes infinitesookaysonow we can plug infor the various unit vectorsokay so this was our way of convertingfromgradient to unit vectorand thenwe canuse the same notationto talk about the position of the lightsourceandwe we've usually chosen the coordinatesystem so that z is the direction that'scoming straight up at me so it's alongthe optical axis so z is along the up isthe viewing direction so z isv is just zokay so now i'm going to take those dotproductsand you know this was the messy part ofthe inversion we had this non-linearterm so if we're trying to plot isophotsand so on this this would create asecond ordercomponent so we ended up with conicsectionsbutwe if we now take the ratio of these twowe get something that's linearand rs is just the shorthand forthis thing so rs is square root of 1plus ps squared plus qs squared and it'sconstant if the light source is in afixed positionand it's just a nuisance to have towrite that out all the timeso umwe're not quite done because actually wewant the square root of this thingbutwhat do the isopods look like ingradient space sowell if the square root of this is aconstant thenthis quantity itself is a constantsquared so we can look at isophotos interms of this formula so what are theisophotes well it's when 1 plus psp plusq as q is a constant and that's a linearequation p and q so it'swhat is thecurve in pq spaceif it's a linear equation in p and q[Music]it's a line right sookay soi canwhich is going to be great compared tolambertian which had these uh conicsectionsso that's one line now suppose iplot another isophote well it's going tobe a line again just with a differentconstant right because this this will bedifferent but the psp plus qsq will bethe same so it will have the same uhorientation so the other isophote mightlook like thatandso there's a whole bunch of parallellines that are the isophotos now they'renot equally spaced because i'm takingthe square root of this thingbut other than thatso i'm going to have i don't knowsomething like thatso this is myplot in gradient spaceandthere's one particular linewhich is where brightness is zero wherei've turned 90 degrees away from thelight sourceand uh just as with our lambertiancosine theta we need to be aware of thefact that brightness can't be negativeso actuallythis part of the diagram is is zero sookay umwhy is this exciting well because it'sit's linear and it's going to make itvery easy to solve all sorts of problemsand so first of allsince we're coming from photometricstereosuppose that we haveone lighting conditionand then we have a different lightingconditionwell then we'll get uh straight linesbutdifferent straight lines so i don't knowmaybe like thisand then obviously if i have the twomeasurementsi canfind the intersection of thecorresponding lines so suppose that themeasurement in theonelining condition was that and under theother lighting condition it was you knowthis linethenthere's there's the intersection so thatthat's the answer that's the surfaceorientationso photometric stereo is very easy andof course i can this is geometrically ican do this with equationsyou know just we have two linearequationsof this form 1 plus psp plus q as qequals somethingand of course we know how to solvelinear equationsso so that'sthat's kind of neat and there's noambiguity with lambertianwe had two conic sections intersectingand they could intersect inup to two placesnow actually by bazoot's theoremumwe gottwo second order equationsand by bizu's theorem there might be asmany as four solutionsand soit turns out that well that's forarbitrary second order equationsbut what if you havethe particular equations we have well itturns out that in that case there can beonly two anyway here there's only one sothat that's an advantagethenanother thing that we can read right offthis diagram is thatso we don't knowfrom one measurementwe can't determine the surfaceorientation as usualbutwe do have something pretty powerfulwhich is the surface orientation in aparticular directionand so let meoopsokay so this is for a particularorientation of the coordinate systemi've you know picked some xyz coordinatesystem and this is what i get in thediagramwhat if ipick a different coordinate system whatif i turn thisby some anglecall it alphawell itit turns out that then i turn thisand if i turn it the right the correctwayi get a pretty neat resulti'm just stating this i'm not proving itbut you can prove it you know keeping inmind that p is dz dx and q is dcdyand p prime is dzdx prime and q prime is d z d y primeand umthen use chain ruleand you basically get a rotationthrough an angle alphaso it's sort of surprising that thefirst derivatives rotate the same way asthe coordinate system but that they dookay why is that great well because um[Music]too bad i messed up that diagram welllet me do it againso i'm just copying thatfirst diagramand now suppose that i pickthis as alphawell thenwhen i measure a particular brightnesslet's say thisi as usual i don't know the surfaceorientationbut i know one component of it i knowthat the component in this x primedirection has a certain value becauseall of the points on this linehave that sameslope in that directioni don't know anything about the slope inthe direction at right angles but i knowthe slope in that direction and that'sdifferent from lambertian because in thelambertian case we had this curve and sothe orientation was different along thecurvehere we've got a line and all points onthat line have the same distance fromfrom this origin from from hereso uhcanfindp primeuh and then you know what is this anglealphawellit's obviously some function of ps andqstan alpha isand how do i know that well because iwant this straight line the 1 plus pspluspsp plus qsqi want that to become the vertical axisand soi have to find the angle that will makeone of the two terms disappear after therotation so anyway thatit's not a particularly important pointbut there that's the angle i actuallywant to use so this is you know this ismore exciting than you might thinkbecause what this means is i can look atthe surface if i pick the coordinatesystem right and measure the brightnessand i immediately knowhow steep the surface is in a certaindirectionso i could then sort of you knowtraverse i could say okay it's going upby uh one meter in ten so i'm going totake a one meter step and that's a tenthof a meter up in z and then i look atthe brightness thereand you know now i again calculate theslope umand it's whatever the slope is it allowsme to calculate how much height i get igain or lose in the next stepand so following that idea i canactually get a profile of the surface ican actuallyyou know keep on going and measuringbrightnessandcomputing the slopeand taking a small stepso the idea is i'm hereand i measure the brightnessand it gives me a slopeso now i can take a small stepin that directionif i go at right anglesyou know i could fall off a cliff that iknow nothing about the slope in thedirection at right anglesso now i'm at this point i measure thebrightness it gives me a different slopeand i take another small stepand i'm at that pointand so on so you can see howi can get a profileon the surfacenow of course you know the issues ofaccuracy becausewe already mentioned that it's hard tomeasure brightness accurately and alsothe surface may not be perfectly uniformthere may besome variation in reflecting propertiesand so on but conceptually i can do thisand by the way i can go in the otherdirection as wellright because if at this point the slopehas a certain valuei cango in the other direction by minus theslope times the step size and so i cancontinue thisprofile on this sideumwellthat requires thati need some sort of initial conditionso i need to know uh zat my starting point so that i canincrementally change the do i know z nowhen i measure brightnessbrightness gives me information aboutorientation not about absolute depthremember the formula l equals pi overfour e blah blah blahz doesn't appear in there that's veryimportant right that uhwhen i walk towards the wall it has thesame brightness and we went through thatargument you know changeuh twochanges that are proportional to rsquared which cancel each other so zdoesn't appear in there and converselythat means that i mean in a way it'snice it means that things don't changeyou know you you don't burn your eyeswhen you get too close to somebodywell most people and so umit's you know pretty much the samebrightnessso that's the good part thatyou can recognize things because theircolor doesn't change as you move aroundthe bad part is you can't get distancethat way you can't invert that processto get distanceso we don't know the distance so that'sthat's an important consideration here ican get this profile if i have theinitial valueandbut so what happens if i don't know theinitial value well the profile might beup hereit'd be it'd be the same shape so i canget the shape of the profile i justcan't get its absolute vertical positionokayso that's pretty exciting because itmeans that for example i can look at themoonother than that full moonwhen everything isindependent of surface orientationand i canrun a profile like this and it sohappens that in the case of theyou know we had the coordinate system upthere and not too surprisingly thedirection is going to be parallel to theequator soof course you typically do this on avery small scale butsuppose i start here well i can get aprofile that wayokaywell why don't i start somewhere elsewhy don't i start herewell i can get the profile thereget a profile thereso you can see how i can explore thewhole surfacei can get lots of profiles and you knowthey may not be very accurate andwhatever but not worry about that forthe momentand there i've got the shapeand as i said typically you'd be doingthis in a small area like you know insome craters where you might think oflanding or something you wouldn't do itfor the whole moon but this gives thedirection of the profilesit's perpend you know it's parallel tothe equator it's a longlines of latitudeokay so that's the good news the badnews isi don't know how these relate to eachother right because when i'm standinghere i have no idea what the slope isperpendicular to this profileand the same with all the other profilesso the good news is i can get theprofiles the bad news is they're allindependent and you know now you canstart imagining various heuristics likesaying ohwellthere aren't any gigantic cliffs sotypically neighboring profiles will besimilarand you know maybe the average heightalong one profile should be the same asthe average heightalong a profile next to it and so on andand then stitch them together into a 3dsurface attempt to stitch them togetherinto a 3d surfacebut it's you know that's uhgoing to depend on prior informationlike you knowwhat are the topographic properties ofthe lunar surface because ifyou had a surface with those reflectinglight properties and these were allyou know you could shift each of theseindependently in the vertical directionyou get the same imageso you don't know which of those are itsoanother idea isif i have a craterthat has some sort of rotationalsymmetry not not perfect perhaps theni'm you know scanningacross like thiswellif i'm lucky this cross sectionwill be very similar to this centralcross section so once i've got thishorizontal cross section i can pretendthat i know the vertical cross sectionand that will tie them all together andof course that makes an assumption aboutthe symmetry of the creator and so onanyway that that's what that's whatpeople didokay now um for a momenti want to have a complete uh change ofuh topic uh we'll get back to this laterit's the very beginnings of uh shapefrom shadingand this was the first shape fromshading problem solvedbecause it's so easyand at the time there was a strongincentiveto solve itokayi want to get back a little bit tolensesand and there's a reason because we'llbe switching to orthographic projectionand i'm going to try and justify uhthat sowe talked about thin lensesso a thin lens has the property that ithas exactly the same projection as apinhole perspective projectionandthe advantage is that it actuallygathers a certain amount of lightnow real lensesaren't thinand so you know if you actually look ata catalog of fancy expensive highquality lensesthere'll be all sorts of diagramsof many different elementsi don't knowjust keep on going you know lots ofindividual elementssymmetrically arranged around someoptical axis and so how do those workand why do they do that well as ialready mentioned um it's impossible tobuild a perfect lens there will alwaysbe trade-offs between different kinds ofaberrations and but by compounding byadding different lenses you cancompensate so for exampleglass has a refractive index that varieswith wavelengths and so that means thatthe focal length will depend onwavelength and that means that red lightwill be brought into focus at a slightlydifferent place from blue light and soyou get chromatic aberrations you getfringes color fringes around things wellyou don't see that in your camera that'sbecausethey've then put in a second lens of adifferent material that has a differentwavelength dependence and carefullydesigned to compensate for that anddepending on how fancy they are itcompensates exactly at two wavelength orif you uh more fussy at threewavelengths anyway so uh there's a needfor compound lenses and they then havedifferent properties but thoseproperties can beapproximated very well as followssoi don't know if you rememberyou should rememberthat for the thin lens we had thisnotion thatthe central ray was undeflected so a raycoming into the center of the lens at anangle alpha would beemittedat an angle alphawell the thick lens can be approximatedthis waywhich is very similarso umthese points are called nodal points andanything coming into the front nodalpoint at a certain direction will leavethe back nodal point in the samedirectionuh the planes through those points areoften called principal planes now in thethin lensuh the two nodal points on top of eachother and the two principal planes areon top of each other and not toosurprisingly the distance between themis called thicknessandusually the notation is tso that makes it actually quite simpleto deal with thick lensesbecause it doesn't change things a wholelot i mean it does mess a little bitwith our lens formularight because now a and b are notmeasured relative to one place butthey're measured relative tothose those pointsso it's umand you know justhow do you compound the lenses to createthis effect well but that's not our jobthepeople that zeiss and such know how todo that sookaywhy are we even talking about this wellbecause now there's a neat trick you cando it turns out thatt doesn't have to be positivethat is the nodal point front nodalpoint can be actually behind the rearnodal point so so who careswell if you make this pretty large youcan make a short telephoto lens right sonormally a telephoto lens is one thathas along focal length obviouslyandsmall field of viewand the lens with a long focal lengthmeans you need a tube a long tubewell if you make t negativeyou can compress thatand you can geta significant reduction you know if youtypically buy a telephoto lens fromnikon or canon and you look athow many millimeters its focal length isif you actually go measure the lensyou'll find that in many cases the lensis shorter than the focal length and sothat's uh one trick to play with thisuh thenanother one isto move uh one of these points far awayuh off to infinity in factright soand this is used quite a bit in machinevisionso umuh why well there are a couple ofreasonsone is thatwhen we have perspective projectionthe magnification changes with distanceso if you say looking down at a conveyorbelt and reading labels and whateveror trying to make a precise measurementof some dimensionwell the image size will depend on focallength and the distance to the objectand if there's any variation in thedistance to the object that'sproblematic or say you're doing i don'tknow printed circuit board inspectionsomething like that andyou want to be insensitive to smallchanges in in distancewellif you canget rid of perspective projection thatwould be good sohow do you do that well what you need isa very far distantcenter of projectionif you move the center of projection faraway thenthat effect of varying magnificationwith distance gets less and less becausethat cone of directions gets more andmore parallel and and in fact if youcould move the uh nodal point to uhminus infinity then there would be nochange in magnification andit's amazing but yeah bybuilding a compound lens you can do thatso that'sobject spaceit's telecentricityand as i mentioneda lot ofmachine vision systems commercially youknow used in industryuse this they're not cheappartly because they need a lot of glassandthe reason they need a lot of glass isthatyou knownormally a lens image is a cone of theworlda telecentric lens because the center ofprojection isway backuh actually images a cylindersoif you think about it you know here'sthe center of projectionhere's the lensnow normallyyou're imaging this whole area with amagnification that changes with distancethat gets smaller and smaller you knowthe image of an object gets smaller andsmaller the further the object is backwell now imagine that you move thispoint uh way back there then this conebecomes shallower and shallower until iteventuallyyou have a cylindrical volumesoan object spacetelecentric lens will image acylindrical volume and that means thatthe lens has to be as big as the objectotherwise it won't be imagedum and actually has to be a little bitbigger uhso so that means that uh you know ifyou're trying to read the circuit boardor something you may need a lens with asubstantial[Music]bit of glass and accurately made so andthat gets expensive but but it's uhit's doneokaynowthat's moving one of these nodesnow you can actually move the other oneas well so let's keep this one in thesame place but move the other one soimage spacesowe have the same kind of diagram with acone of rayson the other sidehitting the imageso we have here's the image plane andhere's our center of projectionandwe[Music]we know a number of things one of themis thatif your image plane isn't in exactly theright place the magnification changesright so if i move my image plane therethe magnification is different nowin order to achieve a sharp image i'mgoing to focus the lens whichta-da that means i'm either changing thefocal length of the glass which is notpossible or i'm moving the lens relativeto the image plane and therefore i'mchanging the magnificationmaybe by a small amount but if you'remaking accurate measurements that that'sa drawback so that's one issueand the other one is that cosine to thefourth lawwe really don't want thatnow if we move this center of projectionoff to plus infinitythen this colon becomes more and morelike a cylinderand first of all that means that as imove the if if the image plane is movedif i got in the wrong place it doesn'tchange the size of the imageuh it may make it more blurry you knowthat's another issue but they want andso that's very useful in the metricsituation where you're actually tryingto measure something so it turns outthat umwhat's the terminology for thisthere are lenses which aretelecentric on both sidesdouble telecentric now that makes senseokay double telecentricsowhy does this cosine to the fourth goaway well becausethat came from the inclinationof the rays coming into the sensorsand so by moving thenodal point way outnow the radiation reaching a particularsensor is coming in perpendicular to thesensorand so that actually hasother effects so here's ourour little sensing element and theradiation is coming in this wayand before it might come in at an angleso particularly near the edge of thesensorthe light is coming from the center ofthe lens and it's coming in at an angleand so we get look at that effect wellthere's another reasonnot to want that and here's onewhich is that oftenthe sensor has right in front ofit a set of little lens lidswhich concentratethe incoming light into a smaller arearight so thatthey don't create your image or focusyour image or something what they'redoing isthey're taking uh the light that iscovering a certain area andconcentrating it into a smaller area andand this is very commonand uh why well because umthere's circuitryso the surface of the sensor isn't allsensitive to light if we look down on itfrom above you know it might look likethisand then there's a lot ofswitching circuitry and stuff around itso here's the area that's actuallysensitive to light and then there's thisother thing and there's something calledthe fill factorsobecause there are many different designsbutthat's kind ofuh something that happens in manydesignsokay so the the different issues one ofthem is you're throwingif you imagewithout the lenses you're throwing awaylight not measuring itwhat's worseyou have to protect the circuitry fromthe light becauseyou knowlight goes into the semiconductorcreates electron hole pairs and oh ifthat's right in the middle of yourmosfet that's that's not such a goodthing so that means you have to put ametal layer on top of itso that's uh one reason whypeople add the little lens lid arraysthe other reason is that is aliasingsothose of you may remember in six6003or some other equivalent signal andsystem course that when you samplediscretely you have to be sure thatthere aren't high frequency componentsin the signaland in our case we could have like sharptransitions in brightness from one areato another and those will createeffects where some high frequencycomponent is it looks like a lowfrequency componentmaybeyou know aliased alias down to a lowerfrequency and how do you avoid that wellyou low-pass filter first and thenthere's a wonderful theorem that says ifyou have a low pass signal you only needto sample it twice the bandwidth of thesignal and you can reconstruct itperfectlyandwhat's the relevance here well uh if weare notuh if we're measuring over the wholearea we're performing a crude form oflow pass filtering with sort of blockaveraging which isn't you knowyou know the low pass real low passfilter is a sync function but you knowwe can't build thatso if we have the large pixelwe get a certain amount of low passfiltering that's advantageousand you know fancy cameras haveadditional mechanisms for this but if wehave these smaller areas it's more likepoint sampling it's more like we didn'tlow-pass filter which is sampling andthat has you knowvery bad aliasing effects so by usingthis lens arrayuh we're actually using the light fromthat whole area and measuring it becausewe're projecting it on to the sensitivearea and so we reduce the aliasingproblem weokaynowuh this works great if light is comingin more or less uh perpendicular to thesurface it's uh not so good if the lightis coming in at an angleright because thenyou knowyou'd have to somehow change the scaleof the lenslet array and even then youcan't make it workcorrectly because there's a spreadthe lens has areas thatare in different directions so anywaylong kind of long story short there'sseveral reasons why uh people like lightto come in perpendicular to the sensoryou know starting with the cosine to thefourth and soin high quality digital slrsthe lenses tend to beimage space telecentric or at leastpartially i meanthey don't actually move the center ofprojection all the way to infinity butthey move it far enough out thatthat cosine to the fourth is is becomesnegligible and we have those effectsokay so that's uh telecentric lenses andthese used to not be availableand it took a while for people to figureout how to design them but now they'reyou know all the rage sookay double telecentric so where are wegoing with this uh orthographicprojectionso we said thatwe no longer have a dependence ondistance that um in the object spacetelecentric devicean object of a certain size will beimaged the same sizeuh independent of its distance uh thethe sharpness of the image will changejust as in the normal lens but the thesize will be the sameand similarly you know for the imagespace telecentric sowhat we're reallydoing istaking a perspective projection equationand making the focal lengthhugeso thatyou know our center of projection is faraway and in effectwe can then pretend that umwe're dealing with orthographicprojection rather thanperspective projectionsso we saw thatperspective projectionwas quiteuseful in a wayso this is where we startedbecause we had this dependence on ondepth and particularly in terms ofmotion that was helpful but now supposethatthe changes in depth in the scenearemuch smaller than the depth itselfthen we can write x is f of z naughttimes xso you know if z is approximately con sothis is another way to get toorthographic projection uh we can make zpretty much constantand how can we do that well one way isto add a very large number to zand that's essentially what happens whenwe move the center of projection we'readding a very large number to z and sosome small variations in z aren't goingto make any differencethe projection is in is pretty muchindependent of the position and so wehave a linear relationship betweenuh x and y in the world and x and y inthe imageand for amongst other things this meanswe can measure distancessizes of objects independent of uh howfar away they arenowin many casesit's convenient to just pretend thatthat scale factor is oneandoften we'll just use use that uh versionof ituhokay i guess it's 12 30.so um that's where we're going to goorthographic projectionuh is uh useful in practice withtelecentric lenses and it's also goingto beuh greatly simplifying some of theproblems we're going to work onthat's not so this is a little bit likethe lambertian thing you know a lot ofpeople say oh these methods only workfor lamborghini no they don't they workfor everything it's just that foranything butler and version theequations get messy and it's the samething herethe kind of reconstruction we're goingto address next can be done underperspective projection it's justcomplicated and not very insightful youknow if you if the math gets verycomplicated you sort of lose track ofwhat you're doingwhen we change the orthographicprojection it turns out thatmany of these problems become quiteclear sookay there'll be a newhomework problem as usual on thursday so

Shape from Shading, General Case - From First Order Nonlinear PDE to Five ODEs

recoveringthe shapes of objectsusing image brightness measurementsand we talked at some length aboutphotometric exterior which wasa method thatgets us the result but it's kind ofunnatural in that it requires multipleexposuresand we're about to transition intoshape from shading which is the methodfor doing this with a single image andwe've already talked about a fewdifferent types of surface materialsand their reflecting properties inparticular we talked about thelambertian surfacesand we started to talk a little bitabout uhhapkewhich is a model for the reflection fromrocky rocky planetsand i want to introduce a third onewhichhas to do with microscopy sothat's a scanning electron microscopebut first for comparison uh here's somepictures of transmission electronmicroscopesas you probably know they allow you toachieve ahugemagnification much larger than withvisible light because you're limited bythe wavelength of light and thewavelength ofelectrons at kilowatt energies is verymuch shorterpotentially allowing you toimage large moleculesandthey haven't yet achieved their limitthey theoretically would be able toresolve individual molecules and atomsand in some some cases they do anyway ifyou do a search electron transmissionelectron microscope you click on imagesyou get all of these pictures ofthese types of microscopesand then eventually you get to somepictures of things that might have beenimaged with an electron microscope[Music]like this thing on the rightand this is you know a cell with anucleus andsome vacuoles and and so on umand it's you know extremely useful uhtechnique forseeing fine detailbut it has its limitations in terms ofinterpretationso you know you look atwellwhat we will do in a momentis look at the comparison with otherkinds of microscopy so let's see there iwas going to click on this thing[Music]so you know i guess you can sort of makesense of it but it'sit's a very thin slice of somethingand you need to knowthe details of of what that isnow if in comparison we do a search onthescanning electron microscopes and clickon imageswe get thisand you have to scroll down before youget a picture of a microscopewhy is that well because scanningelectron microscopesmake these great pictures that peoplelovetransmission electron microscope makegreat pictures that you know scientistsinvolved in the research love butthey're not uh easily interpretablewhereas these scanning electronmicroscope picturesyou know they're very popular they'rejust fabulous i mean you look at um[Music]you know look at that i mean you canimmediately seethe head of this uh jumping spider anduhit's six of its eight eyes and so on soand umwhy why is thatyou know beautiful pictures that we findvery easy to interpret know exactly whatthat is and this is a surface of whoknows what but you can immediately tellthe shape and where the crevices are andso on sothe point isthatthese scanning electron microscopesproduce images that we find easy tointerpretsoi don't know can you think of a reasonwhywe don't find the transmission electronmicroscope image is so easy to interpretbutthese these we cangiven where we are in the coursewell if you look at these pictures yousee that there's a variation inbrightness okay so that's pretty obviousbut it's a variation that's sort ofspecialas you approach the edgesthings get brighterwhereas if you're looking at thefrontal surfacethey're darkerso right so there's a variationin brightness depending on surfaceorientationta-da that's what we're talking about sothe reason we find these images soappealing and easy to interpret isbecause they have shadingthey have a dependence of surfaceof brightness on surface orientationand um so you know this is obviously thehead of some moth you can see thecompound eyes you can see the schnozzlethere all rolled up so uh you know it'suh it's so easy to see what's going onthere's an alien landing onand sothe only thing that's a little bit oddis if you look at theseshapes so this is obviously some sort ofovoid footballlike shapeit's bright it's darkest in the middleand it gets bright towards the edgesso that's sort ofanti-lambertianright with lambertian we get thebrightest uh surface reflection wherethe surface is perpendicular to theincident light and so we expect that asyou approach the occluding contour it itgets darker so if we look at theisophotes on say the image of a sphereit will be concentric curves that sortof drop off towards the edge and thesego the other wayand so that's kind of interestingbecauseuh if we grew up in a world where mostthings were sort of lambertian we'd getvery good at interpreting shape ofthingsuh that have that type of surfaceand yet we're able to interpret theshape of of this very welland you know at one pointi had a europetake a bunch of these pictures andreverse the contrast because you knowthey should be much easier to interpretwhy didn't anyone think of thatand well it was not a very successfuleurope assignment becausewith the reverse contrast the picturesdidn't look any better in fact they weremarginally harder to interpret and andwe had some uh explanations for thatso it shows a couple of things first ofalluh this modality of microscopic imagingis uh verynice from our point of view because wecan understand these images we don'tneed some complicated calculationand the other one is that the humanvisual system apparently can do this aswell and it's not hardwired to have anyparticularidea about what the surface material isit can adapt and deal i meanwe can see that this is not a longversion surfaceat the same time we can have a prettygood idea of what the shape is i meanmaybe not metrically accurate butthat that's another storyokay so that's what we're going totalk about next so let meget rid of thatyou know how do we how do we do this howdo we implement thisyes i want to shut downso first a little note on how thesescanning electron microscopes workyou're all probably too young toremember cathode ray tubesbutpeople used tocreate imagesby deflecting electron beams andhaving them impinge on phosphorus whichwould then glowand those devices had a source ofelectronsbasically you're sort of boilingelectrons off some surface so that's alittle heated coil so that's the sourceof electronsandthey just sort of bubble off the surfaceand then you accelerate them by havingsome electrodewhich we'll call the anodeandwell a lot of the electrons end up therebut other electrons are acceleratedthroughand then we have uh lenses and thesetypically aremagneticand they they can also beelectrostatic lenses and the idea isthat we're going to focus this beamonto some object somake some space for some object downhereum sowith suitable cleverness and enoughmoneyyou can focus that beam down toa very smallpointandthenwhat do you do well the electrons hitthat object and some of them bounce offso we havebackscatterbackscattered electronsbutmost of them penetrateand in the process they lose energy andcreate secondary electronsso you can sort of think of them asyou know bumping into molecules andlosing some energyand creating uhbumping electrons off ionizing thingsand some of those secondary electronscome out of the object and they'regathered by some kind of electrode whichis alsothere are other effects for examplethere'll be some fluorescenceat x-ray energiesso some people measure the x-rays comingoffand there may also be visible light andso on but we're going to focus on on thesecondary electrons because that'swhat's uh used in imagingand creating those images that i showedyou so whywhy does the secondary electron currentwhich we measure herevary with the surface well first of allthe way i've drawn itit's not a whole lot of use because wejust get you know one measurementso what we really want to do is scanthis so some of these electrodes welltypically it's done magneticallyas you know electron in the magneticfield will turnand so using two orthogonal magneticfields that you can control from thecomputer you can direct that beamanywhere you want and you can scan theobject in a raster like fashionif you for example sothis always reminds me of mr farnboroughwhohas uh his claim to fame is that he hasa dirt road named after him a few milesfrom where i livehe's the guy who invented television andyou know nobody's giving me any creditfor ituh and that's becausehe wasa very clever person but not a goodbusinessman and he got shafted by thepeople who ran rca etc anyway probablynever heard of himbut when he was asked by a reporter youknow how did you come up with this ideai mean to us now it all seems prettyobvious butat the time what was available wellthere was a telegraphbeginnings of telephoneand the big question wasyou knowaudio is a one-dimensional signal okayso we understand how touh how to measure that how to transmitit how to reproduce it but but thisimage is 2d you know how do you do thatand there were all kinds of crazy hairbrain schemesandsupposedlyhis inspiration were the furrows in hisfather's farm because i guess he had towalk behind the oxen andholdthe plow and he decided that well justas i can plow this whole area by turningit into a 1d problem i can do the samewith imagesand you know to be honesti think that's an explanation he came upwith when a reporter askedbecause a lot of times you know peopleneed someway of understanding how on earth youever got to the solution and then youhave to dream up some reason so anywaythat's what he did and that's appliedhere so we scan in a raster fashion andthen what do we do we use the currentmeasured here to modulate a light beamin a display either another crt orread it into a computer and display iton on an lcdand then you can control themagnificationhow do you do that well you just controlthe deflection soif youdeflect the beam a lot you don't getmuch magnification and as you reduce thedeflection you get more and moremagnification and so you can getyou know thousands tens of thousandsum compared to optical microscopy whichmaxes out uh below a thousand prettymuch despite all sorts of incrediblyclever tricksokay umwhy do we create shaded images so wehave to look at howthe surface interacts with thebeam so here's the beamcoming inelectrons at several kilo electron voltsand they hit this material and theystart interacting they bump into thingsand they create secondary electronsand they lose energy as they go so thefirst few are rather high energy andthen they get lower energy and thoseguys bump into things andandmost of those electrons just disappearin in this object but some of them thatare near the surface escapeandare measured by our secondary electroncollectorand so basically what we're displayingis what fraction of the incoming beamactually makes it back out againby the wayone of the limitations is this thingbetter be conductive because you'repumping all these electrons in thereit's not a huge currentmicroamp or less but still you've got ayou know this insect is in there andyou're umputting a microamp of current in prettysoon it'll charge up to be somesignificant fraction of a coulomb andit'll explode soin order to do this you typicallygold plate these objects so that they'reconductivewhy gold well because it doesn't outgasthis whole thing has to be done invacuumokay so that's what we have in thevertical situation now imagine that wehave a highly inclinedbeamsurface elementwell nowmany moreof the electrons are going to escaperight because they're generated closerto the surface so that means the currentwe measure will be higher whenwhen we're dealing with a surface that'sinclined and so that uh that's it that'sthe mechanism for turning surfaceorientation into brightness and our jobis to to invertthat so one way we can understand thisis to plot the reflectance mapwhich we've already done for lamborghinisurfacesandhopkins surfacesbrightness versus orientationso in this casecoming straight down means that we get arelatively dark image because a lot ofthe secondary electrons are justnever going to make it out of thesurfacethen as we go to higher surfaceinclinationit gets brighter because more of thesecondary electrons escapeand in typical arrangementsthe thing is axially symmetricso whether these electrons come out inthis direction or that direction if theyget out of the surface they're going tobe counted and so that means that thisis going to be symmetric and so if weplot the isofoatswe're going to get uh something likethatwhere you knowput numbers on it'swhere it gets uhbigger the furtherout we get the more inclined the surfaceisand yes you canmodify this you can you know remove someof the collectors or wire this collectorto a different sensor and so on and makethis asymmetrical but the standard wayof using it is just to measure the totalsecondary electron currentandthat gives us that type of reflectancemapokay so now if i measure the brightnessat a particular point in the imagei get the slope of the surfaceso i can directly translate that youknow it's 0.7 then the slope is whateverthat distance from the origin isbuti don't get the gradientright what i'd like to know is what'sthe surface orientation and as usualwe've got two unknowns p and q we onlyhave one constraint we measure thebrightness and so yes we've reduced thenumber of possibilities but we stillhave a one-dimensionalinfinite set of possibilitiesandsoyou know slopetypically is thought of as a scalarwhereasthis is a vector so it's a little bitlikespeed versus velocityi mean not everyone agrees on thosedefinitions but to mespeed is a scalar quantity you knowgoing 50 miles an hour whereas velocityis a directed vectorsookayso umwhat to do well it's just anotherexample ofthis shape from shading problem we'regoing to have some distribution of somepattern of brightness and our job is toestimate what the surface shape isand we havea reflectance map to help us with thatbecause it allows us to look up for anyparticular brightness we measureuh what the slope is unfortunately notthe gradient if it was the gradient thenwe would be in better shapeand we're going to do thisfor various special casesand then for the general case wherewe can use any reflectance map we likeand and that's important because wedon't want this to only work for saylambertian surfaces or the lunar surfaceor whatever sookaysowhen you get a chanceyou know look upscanning electron microscope images andjust marvel at thewonderful worldthat exists there and how we're able tounderstand it uh so welland umyou know puzzleslike why is it that if you reverse thecontrast it's not easier to interpretthe image which is sort of what you'dexpect you'd expect thata surface that's brightest where it'sfacing you would be easier to understandthat than one that's darkestokaysoin preparation for thisi want to talk a little bit abouthow to go froma needle diagramto shapenow what wewhere we're going with this is we'regoing directlyto a methodthat takes us from a single image to ashape but meantime we have some toolsand i want toexploit those tools so for examplewhat's a needle diagram well it's justsurface orientation that every pixelso yeah and what are the needles wellyou can think of each facet of thesurface as having a normal that'ssticking out and you're looking down atthat normal that's the needleand theview of ityou may remember from the very firstclassif you're looking straight downperpendicular to the surface then thatneedle is just a point and if it's notthen it'll be pointing in a directionthat tells you what the surface gradientdirection is and the length of it willtell you how steep the surface is sowhere do we get these well we're notgoing to get them from there so there'smore work to be done butfor example we we get this fromphotomatrix stereo so in photometricstereowe're not computing z as a function of xand ywhat we're getting instead isp and q as a function of x and yof course remembering that you know p isd z dxand q is the z d ywell it's umit's our estimate based on ourimage brightness measurementssookayso that seems actually like a verysimple problemwhy wellin the discrete case we have a number ofunknowns equal to the number of pixelswe're trying to recover z at every pixellet's sayso there are you know millions ofunknownsbut at every pixel we also havetwo pieces of informationpmqsoif we have a million pixel image we havetwo million pieces of information soit's uh it's over determined we actuallyhave more information than we needand as we sawthat's always handy because that meansthatwe will be able to reduce noise and geta better result than if it wasn'toverdetermined so in fact we have twiceas many constraints as the unknownokay so what can we do well one thing wecan do is suppose we start hereand just integrate outpwhich is d z dxalong this axis and what we'll get isthe change in height so z of x is z oflet's say zero suppose we start at theoriginplusfrom 0to xright because dz dx is the derivativeyou integrate the derivativeyou know you know all that so okay so ican get the height anywhere along thex-axis and similarly i can get theheight anywhere along the y-axisby integrating uh the other thing i knowso uhthen from there i can combine them and ican go part way up on the y-axis andacross on the x-axis or i can getpartway and the x so i can fill in thewhole areain various ways like this and actuallyyou know i could take a curve so let'stake that curveand so along that curve what we'relooking at isa combinationright sowhat is this well that's just the dotproduct of the gradient p q dotd x d yso that just tells me the change inheightso this is actuallydelta z if you likethat's a change in height if i take asmall step in the x direction then thesmall step in the y directionokay pretty obviousand so i can do that for all thesepoints i just pick contours andintegrate upwell umthat's okaybut what if somebody elsecomes along and says oh i don't like thecontour you chose i'm gonna go this wayandwhat you'd hope of course is they getthe same answerbut there's no guarantee of thatright because these p and q's aredetermined experimentally they'resubject to measurement noise so they'renot perfect they're not they're notactually the derivatives of z withrespect to x and the derivative of zwith respect to y there are estimateswhich include some noise sookay wellwhat we'd hope is that they come out thesameand what that means is that actuallywhat we're hoping is thatif you go around the loopright if we go this wayup there and then we come backwhat what should this equalso that should be zero yes thank you andevery every jogger knows that that's nottrue if you jog in a loop you seem to begoing uphill all the way andokay i guess we don't have many joggershere sooh it's like um well okay so umthat should be truesothere's no guarantee that when wemeasure p and q experimentally weestimate themthat this is going to be true sowhat do we do with thatwell uh one thing we can think about isturning this into some sort of conditionon p and qthat is p and q have to satisfy someconstraint for this to be true and thishas to be true for any any loop not notjust that oneand i can decompose a large loop likethat into a small loop so supposesuppose it's true for for that loopuh well then i canput another loop next to itand another loopand nowrather than go around these four loops ican eliminate the inner partsand they cancel each other see if i'mgoing from here to there and then i'mgoing from there to here i'm back at thesame place soso this is a crude way of proving thatif it's true for a small loopfor any small loopi can decompose any large loop into lotsof small loops and then it's going to betrue for for the large loop as well sookay so let's see whatsolet'shave a small loop of size delta x inthis direction and delta y in thatdirection and let's suppose that this isthe point x yand let's see how the height changes aswe go around this loop so start at startdown here saywellthen we need to know what the slope isin thex direction so that's pand let's take the slope at the centerof this stretchwhich is going to be p of x andy minus delta y over 2.and that's the slope and we're goingdelta xwith that slopeso we make this loop small enough sothat we can use this linearapproximation thatthe slope is pretty much constantover that stretch then we go upokay so that's going to be we need theslope in the y direction which is qand we'll take the slope and estimate itbased on the center of this line sothat's at x plus delta x over 2and deltayand ysorryand then we go across the topminus px y plus delta y over 2.oh this is times the other yand then we go down so minus qand that should be zero we should bebackat ground zerosowhat we get here isif we do the taylor series expansionof that we getand down here we get p of x commay plus higher order termsand then we expand this we get q of xcomma yplus delta x over 2 q xlet's seeplusokay and of coursethese terms cancelso we end up with[Music]p ydelta x delta yminus qx delta x delta y is zero or inother words these two are the sameandcancel out thethe size of this littleboxand so we get py is qxsoif this is true ofvery smallareas then that condition has has to betrueandthat makes sense becauseyou know p was supposed to beour measurement of dz dx and q wassupposed to be a measurement ofd z d yand so so that's z sub x yand that's z suby xuh now of it's slightly confusingbecause that's obviously true of theoriginal surface as long as it's youknow smooth enough uh that the mixedpartial derivatives in the don't dependon the orderokay so if p and q really were thederivatives of the surface z of x y thenthis would be truebut we're measuring them fromexperimental data so this typicallywon't be true and if we take thisintegralwe will get different results dependingon different directionsand so so this is this is likeyou knowover determined linear equations there'sno solutionall we can do is find some least squaresapproximationso same herethere is no surfacez of x y that will give usexactly the p and qthat we measured fromphotometric exterioror some other vision methodokaynow actuallywe can get there a more elegant waywhich is handy because we'll use thisagain now later solike many important theorems this onehas many namesuh i guess it's a attribute to gauss andit's a tribute it's a special case ofmore general theorems used in fluiddynamics but what it does is it relatesa contour integral to an area integralwhere uh let's draw some sort of shapeso l is this uh contour and d is is thatareaso you know why do we carewell in machine vision often we havecomputations thattake us over all pixels and there are alot of pixelsanytime you can reduce that computationto a computation about the boundary of aregion aloneyou've got a huge win you're down fromyou know millions of calculations tothousands of calculationsso there are several places in machinevision wherewe use a trick like thistoturn an integral over an area into anintegral over a just a curvewith a huge benefit of courseyou know there's a very specializedthing there are only that many thingsthat have this special property sonamely things that can be written inthis formbutyou know suppose for example you want tocompute the area of a blobwell you can count the pixels you canjust go row by row by row see okay thispixel is in the blob that one's not andso onor you can just trace the outlineturns out you can compute the area justby tracing the outlinein fact there used to be a wonderfulinstrument which isfiendishly difficult to understandbut it allowed people tocompute integralsin an analog fashionokay it's uh you should look it up it'san instrument that often was made out ofprecious metals because it was only forspecial people like surveyorsand you would basicallyhold one part of it and trace theoutlineandcompute the areaonce you complete the loop you'vecomputed the area you can read it offthe instrumentand that's an example wherewe're turning an area integralyou know the integral of one over thatareainto a contour integralas you go around the edge there's alittle wheel that slips on the paper injust the right way and so on soso this isn't just you know somethinguseful in digital domain it's been usedin the analog world as well so area isan example of something where this is auseful formula because we can computearea by doing something on the boundaryinstead of doing something in theinterior but it turns out thatthere are lots of other things we wantto do for example in vision often wewant to know where something is and in asimple case you know it's a bloband where is it wellthe centroid is a very good way oftalking about the position of a blob inan imageand so how do you compute the centroidwell again you can go pixel by pixel andaccumulate x times something throw it inthe accumulator and when you're done youdivide and you get the centroidor you can go around the boundary whichis much fasterandcontinuing along that line of reasoningthe things called momentsof which the centroid computation is thefirst and the area componentcomputation is the zeroth but generallyyou can compute moments by just goingaround the boundary and moments areuseful in describing shapes you know soso far we talked about area and positioncentroidbut you might also want to be able tosay something like oh it's it'selongatedand sothese moments are useful for that andthey can all be computed in thisclever wayso let's apply uh green's theoremthat you know there's a 3d version ofthatuhturning 3d integral volume integralsinto surface area integrals but sinceour images are 2d we typically don'tneed that okay so we we apply this toour problemso we're trying to matchyou know something in this formula withwhat we were doing and i guess we had uhpdx plus qdy up here so let's try thatlet's tryusinggreen's theoremso m is qdq dxminusl is pand we're saying that that is zero rightwe're going around the loopokay and this has to be true forallnot just you know particular loop butany loop small loop large loop whateverand that can only be trueif dq dxequals dpdyright because suppose that dpsuppose that that quantity thatintegrand was non-zero at some point alli need to do is construct a loop thatincludes that point and i'd have acontradictionrightso um that again confirms the conclusionwe arrived thatpainfully here by making a small squareand then using taylor series uh thisslightly more uh elegant way of gettingto the same thingokayso umwhat what to do our p and q do notsatisfy that constraint so we could tryand introduce that as a constraint wecould say okay find the z of x and ythat has these p's and q's asderivatives approximatelyand also satisfy that constraint well wehaven't talked about how tosolve constraint optimization problemsthat's a little harder than the leastsquares we've done so farso let's uhtry a different tagso let's uhbasically you know do just brute forcethese squaresso we're trying to look for a surface zsuch thatsuch that that is the smallest possibleright so ideally if there was nomeasurement error and we had the correctsurfacethenthere was a z such that its x derivativematched the p we computed from the imageand its y derivative uh match thesorry the q the p that we got from theimage and this one match the q that wegot from the image but because ofmeasurement noise we expect that willnot be possible so let's at least tryand make it as small as possibleokay so that's theleast squares problemin a nutshell andwell uh you know we solve itas usual wewe just differentiate with respect to zyes can we do thatwhat is zis z a parameterright so no unfortunately we can't dothatright if we had a finite number of knobswe could turn parameters variablesthen we just differentiate with respectto each of them and set the result equalto zero and when we're done that's whatwe've been doingunfortunatelyz here is a functionand so you know there's no sort ofsensible way of talking about thederivative with respect to a functionso it it has not a finite number ofdegrees of freedom it has an infinitenumber of degrees of freedomand there's asubject in mathematicsthat deals with thatcalculus of variationsand it's called that because basicallyyou say something like suppose that z isthe solution then if i make any smallvariation on zthat integral should go upandbased on thatvery sensible idea you can come up withsome equations for solving that problembutwe're going toyou know remain in the trenchesand work with the discrete version for amomentbecauseand this particularproblem is pretty simplethe reason to work through it in detailis because it hides a lot of thepitfalls that you might run into whenyou solve a more complicated problemokay so the discrete in the discretecasewe don't have a function of x and y whatwe have is a bunch of discrete values ona gridright so a finite number of unknownsmaybe huge maybe a millionand so butwe can use calculus ordinary calculusjust differentiate with respect to eachof them set the result equal to zero sotrue we're going to get a lot ofequations but you know our methods applywe don't need to invent something newso this is what we're looking for andwhat we're given isagain aset of gradients on a gridwith measurement noise in them and we'retrying to findavalue of z at every grid pointthat willmake the error as small as possiblewhere the error is the discreteequivalent of of this thing so let'slook at thatso i and j arerow and column numbersso you know slight slight uhannoyance here for computer scientistsand mathematicians which is thatyou would like the first discrete indexto correspond to x and the seconddiscrete index to correspond to yand you know as a student i had a realhard time with that but of course inmathematicswecountrows down and columns across and wewrite them i j in the other order andand also with i reverse butokay so this is what is this we'retrying to estimate the derivative in thex direction and that's why we're varyingy rather than i rightand so this is our estimate of thederivative in the x direction and thatshould be smallexcuse me it should match what weobservedfrom the imageand we also need to add inthe other term we also want to match thederivative in the y direction and nowtypicallythere aren't the set of values of z thatwill make both of those terms zero so wekind of compromise we just want to makethem as small as possibleso we're going to minimize this and whatcan we minimize well we have the set ofzijsokaysowe then differentiateandset the result equal to zero for allpossible values of i jso if the image has a million pixelsthere'll be a million equations uhcoming from that fortunately because wepicked least squaresthe equations are all going to be linearso so as much as thedisadvantages to using least squaressuch as not being robust againstoutlierswe can solve these equations becausethey're just going to be linearequationsokay so herepay attention big problem if youdifferentiate with respect to z i j youget the wrong answerwhy is thatwell i j up here these are dummyvariablesright you can'tif i replace i j with alpha and betait's the same sumbut then if you differentiate withrespect to z i jso another way of thinking about it inprogramming language terms is you knowyou've got an identifier collisions youyou're using i and j to mean one thingin hereand now you're trying to make it meansomething elseso so here we have a sum of all possibleijs and now you're going todifferentiate solong story shortpick some other identifier namesandyou know you might think okay that'skind of obvious yes but there have beenpapers published that got this wrong soso it's possible it can happen okay soum well this sounds kind of messybecause we have a sum over a millionterms you know and now we're going todifferentiate but the good thing is thatyes we have a lot of equations but allthey're all very sparsebecause you know if you think about okayanything here having to dowith rho a thousand when youdifferentiate with respect to z inrow 990is not going to have any effect so allof those derivatives are zero except asmall numberokay the small number where k lmatches up with an index in here i jor k lmatches i j plus 1 or k l matches i plus1 j that's it three termsso we're going to have a lot ofequations each of which has three termsin this caseokay so let's try that solet's first try where we matchkl is i comma jsothere's a square so differentiate thatwe get 2 timesthattimes the derivative of the thing insidewith respect toz kl which is going to be minus 1 overepsilonright so it's a square we get twice thatterm and then we have to differentiatewhat's inside with respect to zkl and wesaid that it matches this term so it'sgoing to be minus 1 over epsiloni'm sorry if i'm going over this in avery pedestrian way but it'swe you'll need to do this in a morecomplicated example and it's it'seasy to not do it right sookay so that's that one but there's alsothis match so we have to add to this twotimes this termso get twice this term and thendifferentiate this with respect to z klwhich is matching that so that gives meminus 1 over epsilon againokay so that'sthere are two termsthen we also have to consider where k lisi j plus onein other words k is il is j plus one or conversely j is lminus oneright so we differentiate with respectto f we get two times thisokay okayso we get twice that term and we have todifferentiate what's inside with respectto zkl which matches that one so it'sgoing to be one plus one over epsilonand we're almost done just one more termwe this one herenow we have k comma l isi plus 1 j where in other words kmatches i plus 1 meaning i is k minus 1and l is just jso we get that one that's going to be zk l minus zokay i isl k minus 1l over epsilon minusandplus one of absence soand all of that supposed to be zerorightthat's the uh least squares methodand this is going to be true foryou know all points in the image notso uhfortunately this simplifiesso first of all of course i can ignorethe two because if twice something iszero then something is zero so if i getthe twouh i could also cancel out one of theepsilonsbut i'll keep them just uh for reasonsthat will become apparentokay so now i'm going to gather up theterms let me first gather up all theterms in p well the only two of them sothat's pretty straightforwardso i get p k lminus p k l minus 1divided by epsilonso those two termsand let me gather up all the terms in qso i get umminus qk minus 1 l over epsilonand then let me gather up all the termsin zuh well there are a bunch more of thoseum and they're they're one over epsilonsquared so let me write that out frontso we're gonna have minus z k l plus oneminus z kplus one lminus e k l minus 1 minus e k minus 1 land thenz k l occurs four timeswell let me justkeep the epsilon squaredand all of that supposed to be zerouh so what what are those termswell this this looks awfully likeuh the x derivative of of p rightbecause wetake the value of p at two positionsseparated in x and we divide by epsilonso you know we canthink of this as p sub xand this looks awfully like a yderivative of qand then i i claim that this thing herecorresponds to the laplacian of zuh let me let me see why okay so i candraw a little diagram hereand gather up all these terms so i get 1over epsilon squaredand i have 4for kl and then minus 1 for all of theneighborsand nowz sub x xokay so those things i'm drawing downthereare variously called you knowcomputational molecules or stencils orsomething like thatandthere are there a graphic way of showinghow to computer to estimate a derivativenow i didn't botheruh for these but i could so i could forexamplethat's the computational molecule forthe x derivative and then here's thecomputational molecule forfor the y derivative you know but it'sso obvious we don't even bother withthat but when we get to higherderivatives it's handy to draw thesediagrams soso if we convolve this with itselfremember 603then you get you get thatand if you convolve this with itself youget this second derivativeright sojust how do you convolve youflip and shiftanywayyou could also get this from taylorseries or any otherany number of other methods forestimating derivativesokay so then if we combine these two ifwe add these twowe get that diagram well except for thesignso this is actually minus the laplacianbutit occurs on the same side of the equalsign so when you move it to the othersideit becomes plus the laplacianso the laplacian is actually heavilyused inmachine vision particularlypre-processing so you know we shouldbecome familiar with itsoby the way another way of writing it isthat and you know engineers have adifferentpoint of view on that as mathematiciansi i gather mathematicians prefer thefirstway of writing the laplacian andyou know fluid dynamics people like likethe other one but whatever it's alaplacian so the laplacian is aninteresting operator from many points ofviewyou know it's a derivative operatorwe've already seen the brightnessgradient plays an important rolefirstderivativesfor some operationson images we would like things to berotationally invariant in other words ifyou are performing some image operationenhancing edge detectionyou don't want to depend on your choiceof x y coordinate system so that whenyou turn the coordinates you wouldexpect things to work out pretty muchthe sameof course they'll have differentcoordinates but you know they'll be inthe corresponding placeso one of the big questions is are therederivative operatorswhich are very useful in edge detectionthat are rotationally symmetric so thatwhen you rotate the coordinate systemthey do the same thingwell uh the laplacian is the firstthe lowest order linear uhcombination derivative combination thatdoes that it doesn't look like it doesit because we're adding a second partialderivative with respect to x and thesecond partial derivative with respectto yum butif you go through the painful pain andagony of changing coordinate systems andcomputing the first derivative and thencomputing the second derivatives and youadd them up and magicallyin the new coordinate system you getexactly the same so suppose we have acoordinate system like this and then wego to a rotated coordinate system itturns out that z x primethat in that rotated coordinate systemthe laplacian is the same i'll spare youthe details it's pretty boringbutthis was this makes the laplacian of alot of interest in all sorts of imageprocessing operations uh because it isrotationally symmetric of course on adiscrete grid we're going to be somewhatbiased anyway like you know does thislook rotationally symmetric well uh notterribly so they're actually betterapproximations if you're on a squaregrid and and we'll talk about them andif you not on a square grid you can doeven better for example if you're on ahexagonal grid you can get a betterapproximation anyway back to this sowhat we've done over here iswe'vebypassed the calculus of variation forthe momentand wesomewhat painfully went into thediscrete world of discrete pixels and wefound a equation that has to be trueat every pixel in order for this to be aleast square solutionthat is this is a solution thatminimizes that error it's it's the bestfit it's as good as we can get and thensort of looking at this formula herewe decided ohmaybe that's the discrete version ofthat continuous equationand that's exactly right because if weuse the calculus of variationwe go directly to that in one stepso why didn't we do that well becausethere's this huge learning curveso this one step after you do a lot ofstuff and i didn't want to do that rightnow soanyway what do we do with theseequations how do we solve themof course we can they're linearequations so we can usegaussian eliminationyou know or some perhaps slightly moresophisticated algorithmall of which take a long time forlots of equationsright typically they go as somethinglike n cubed or if you you knowcomplexity theory personn to the 2.76 or whateverwith a very large multiplier in front sobut whatever once n is a million or 10million that's a large number and itit's not likely you're actually going tobe doing thatso umwhat to do instead well the great thingis that we havea sparse set of equationsso we got a million equations but eachof them only has a handful of terms init right so they they only involve umeach equation only involves fivevalues of zso we got you know you think of writingthem out there's amillion rows of these but each row onlyhas five non-zero elements well it turnsoutwe can then solve this iterativelypretty simplyandof courseit's easy to propose an iterativesolution it's hard to show that itconverges so we leave out that partand we'll justappeal to the textbooks that say that itdoes converge sowhat can we do here well we can pull outone of these termsand pretend that we we know we have aninitial guess so we know what thesevalues areat the momentand we're going to compute a new valuefor for this oneso um let's see if i got it written downor notso i'm going to use a subscript inparenthesis as a way of indicating theiteration step so this is step n plusoneand that step inumokay um i almost wrote uh subsuperscript uh parenthesis n here forthe n step but no these are from theimage they're fixed once we've processedthe images uh that's uh givenuh it's the only thing that might changeis this lotwhat what is that well that's a localaverageand in our case with that computationalmoleculethat looks like thatsothe iterative step isvery simple we go to particular pixeland we get a new value by taking theaverage of the neighbor neighbors infrom the previous stepand then we add in this correction herewhich is based on the image informationandwhat does this do well it brings uscloser to satisfying the laplacianconditionthat you know because this is theestimate of p sub x and this is theestimate of q sub yand so uh and the difference betweenthis and that is our estimate of thelaplacian so as we go alongwe adjust that pixel value using thatvery simple equation so here's where thesparseness comes inyou know it would this wouldn't be veryuseful if we had a million terms in thisbut we've only got uhyou know four four neighbors to considerand you can't show that this convergesum we won't do thatand umit's going to be very much faster thantrying to solve it with gaussianelimination particularly sinceyou know you don't need infiniteprecision you can stop after a certainpointandand relaxso a couple of things point out first ofallthis is closely related to solving theheat equationthis this type of iteration because theheat equation is a second order pde justlike the equation we haveit's alsothe diffusion equationand so um so there are loads oftechniques available from other fieldsthat tell us how to do this efficientlyhow to do it in parallel and you knowwhateverso again the idea is you step throughthe pixels and at each pixel you computethe average of the neighborsand then add in a correction term andthat's the new value and you just keepon doing that now for startyou don't have to do this sequentiallyas uh farnborough didyou know the rows of his fieldyou can do this in any order you wantand in fact the advantages in terms ofconvergence if you do thatalsoyou can parallelize thisyou can do as many of these you likethatin parallel as long as they don't toucheach otherright soif you are making a change to a pixelbased on some neighboryou do not want the computation thatchanges the neighbor at the same timestepbut that that's uh huge becauseyou know we we just we can divide theimage into i don't know nine sub imagesuh three little three by three blocksand while we're operating on this onewe're not allowed to touchthese othersbut that's fine that means we can havehuge parallelism you know a millionpixels divided by nine well we probablydon't have 110 000 processors butso we can do something slightly moreintelligent anyway the methods numer fornumerically solving these equations arewell known and it's a very stableproblem in the sense that if there's anoise in your measurements it it'llstill converge it'll still work andeverything will be coolokay soumokay we have a bit of time so what whathave we done well what we've done rightnow is uhlearned some techniques because in a waythis isn't directly usefulexcept for the case of photometricstereo and we now want to move on fromphotometric stereo but the the kind oftoolslike this discretization andyou know avoiding the conflict ofidentifiersand then looking at the terms and seeingoh this is an approximation of thederivativethose will reoccur so it was good thatwe did this butbeyond thatthis shows you if you have photomatrixstereo datayou can reconstruct the surface in aleast squares way and get a reasonablesolution that matches the experimentaldataokaynow now we're going to go back tothe the somewhat more challengingproblemof uhsingle image mono uhsingle image reconstructionsandwe've sort of solved one case alreadywhich is if the surface had these veryspecial propertiesright so we hadlet's justgo over that a little bitso this waswhere we hadbrightness was a function of that whichwasincidentand in this case the reflectance map wasvery simpleall rightwe had these parallel lines that theywon't be equally spaced because of thesquare root but they're parallel linesand each of them has the one plus pspplus qsq equals a constantso the isophotesin the reflectance mapare these parallel straight linesand that's actually what allowed us tosolve the problem because then we saidwellsuppose i rotateto a somewhat more useful coordinatesystemin that coordinate systemi can just determine theslopein that special direction so if i readoffyou know brightness value like thisin the original coordinate system youknow it's a linear relationship betweenp and qbut in the new coordinate system it's uhconveniently giving me an exact value ofp primeand it tells me nothing about the qprime so i've sort ofmaximized the information in onedirection and made it worse in anotherdirectionandi i'm just going to write this down i'mnot going to do this carefully but itturns out that we will need to deal withthese coordinate transformations at somepoint so oops sooh and what is this anglelet's call it theta isumisincethat raythat axis p prime is perpendicular toall of these linesright i'm going to have that p comma qdotp sq sis going to allow me to determine thatthat angle is given by this triangleq isso i can[Music]well might as well write it downso whatever the light source position isi can figure out what that angle isso and so actually the light source issomewhere along this line it'slet's say it's up hereokay umnow what i want to do is rotate theimageahead of timesouh tomake this umunnecessary later sowhat's the relationship well you knowyou i'm sure you know thisand we can call it theta s or just thetafor simplicityokay uh but what we need is arelationship between p and q and p primeand q prime so we need dz dx primeis dz dxdx dx prime plusdz dy dy dx primeand of course this isp prime that i want to go for this is puh this is qand so i'm left with thatwell annoyingly that's going the wrongwayright so i actually need the inversetransform of thisso what's the inverse of rotatingthrough thetarotating through minus theta right sox is x prime cos theta minus y primesine theta and y isx prime sine theta plusokay so then this isdx dx prime is cosine thetaand this is the y the x prime so that'ssine thetaso actually i got p prime isp cosine theta plus q sine thetaand similarly i'm going to get q primeis minus p soso this is umniceand you might say welli could have guessed that right becauseit has the same formwell think again because when you get tothe second derivative this doesn't workyou can't just guess the answer andthat may be relevant because we will betalking about we're talking about thelaplacian so right thereif you were trying to prove that theoremi said about rotational symmetry you'dneed to get second derivatives and thenthis wouldn't work anywayso i'm going to do that i'm going torotatethese coordinate systems and then i'mgoing to end up with with something likewhich relates the image measurements tothe slopeand you know the details of the equationisn't so important what's important isthat i can be at a particular point inthe image i read off the brightness andicompute this quantity the other thingsare all known i know where the lightsource isand i've got the slope which is amazingi have the slope of the surface in a ina particular direction and you knowthat'suhenough i we cannow reconstruct the surfacebut but this did requirethat we[Music]rotate the coordinate system that wehave a particular direction and that'sgoing to be true in the general case sothis is only for hapkawhere we have nicestraight line isophotes in thereflections mapbut we'll generalize that and umokay so what do we do with that well wewent over this a little bit last timebecause uhwe can just integrate out uhyou know z of x isright so we have the slopewe just integrate it out or you know inin terms of discrete implementationwe're hereuh we go a small distance delta x like idon't know pixelwe know the slopeso we know how much z is changingright so we have a new value of z andthen welook at the image and say what's thebrightness here and using that formulawe find a new value for p primeand we you know continue we take anotherlittle step delta x and we get there andso onand so we can build up a shape in thediscrete world and we can go the otherdirection as well we can go minus deltaxand build up a profile this wayso that'skind of amazing that you can do thatwell that's a single profile but now inthe imagewe havethe y direction to deal with as well butwe can do this for any yright webut in each case we were confined torunning along a row there's nointeraction between the rowswe just treat it as a 1d problem alongalong each rowwe read off the brightness at each pixelwe convert it to the slope using thatformulaand then use that slope to see uh wherewe goso there are a couple of things aboutthis that are interesting so one is thatif i add a constant to zuh does anything changeso in fact let's let's go back to tothis one up here this over determinedproblem uh which has now sort ofdisappearedif you chan if you add a constant to zdoes the laplacian of z changeno right souh that's interesting because that meansthat okay you can recover z but actuallythe absolute height of z you don't knowand that's reasonable because ifbrightness depends on surfaceorientation then moving the surface upand down shouldn't have any effectand in particular in the case oforthographic projection it doesn'tchange in any way it's it's not evenmagnifying or minifying or something andso similarly here uh you know theshading information tells us nothingdirectly about depth only about relativedepth you know what's the slope of thesurface and so hereto to actually get a reconstruction ineed an initial valuefor for zandwell uhthat's a problem now if it was justuh an over like in the case of thelaplacian solution if it's just anoverall a change in in height that wouldbe easy to deal with that's just oneunknown but here we potentially have adifferent initial value for every rowthat we're integrating along soso you can imagine that you know we'recomputing these solutionsandwe have have an initial valuesay hereand we integrate it out but we can pickthose independentlyor somebody could go there and measureitbut but that sort of defeats the purposeyou know we're trying to find thesurface shape of the moon before anyonegoes there and so having someone gothere and survey it kind of defeats thepurpose although it does mean that youonly have to survey one curve and theneverything else sprouts off from thatand we already talked about you knowvarious heuristics to try andbuild something even though we don'tknow thatsowe don't need to have these initialpoints along the y-axis though we couldhave initial points along any any curveso that means what we actually need isan initialcurve doesn't really really matter whatthe curve is isas long as it's not parallel to the xprime axisokayand then now we can map this back intoour originalthe unrotated worldandthese profiles now will run at an angleand we have some sort of initial curveherenow let me introducecoordinateslet's callthe initial curve is some function ofeta soinitial curveis excavator y of etaz of etaright so we'll assume that's givenand then we goaway from that using some otherparameterchaiand that's our x prime in this case butwe want to generalize that so this isgoing to be very similar to the[Music]general method and sowewe take a step delta x along this curvewhich would be you know a step in xprimeandbecause of the rotationso this is the the scheme we'refollowing sowe're exploring the surface by movingalong these curves and you know fromstep to stepthis is what we do we have delta x anddelta y changein a waythat's at an angle theta sin this original xy coordinate systemi've gone back to that system because ingeneral we won't be able to use thistrickit's only for hapka surfaces that itworksand so you can see the numericalcalculations very straightforwardwe pick up the image brightness we plugit into this formula and we get a stepin the z directionand so we just continue doing thatand explore that curve and then we dothe same for the next curve and they canbe computed independently so you can dothem in parallel if you wantjust one little noteyou can change the speed that youexplore the solution inright because if iif this tells me to take a step delta xdelta y delta zmaybe i'll take2 delta x 2 delta y 2 delta z ora half and how would i decide wellsuppose that you only move a hundredthof a pixel that's probably a bad choicefor incrementor suppose you move 10 pixels that's abad choice for increment you want toreduce itand the convenient thing is we caneasily change the speed we just multiplyall three of them by the same constantand and souh you know if you want simple equationsuh we can just multiply by that constantand you get a somewhat simpler lookingexpressionright so we're moving in the x directionproportional to q s we're moving the ydirection proportion of ps and those arefixed we're just driving along and thenin the z direction we have this simpleformulaandsowhat we'll do next time is generalizethis to arbitrary reflectance map notjusthopkin type surfacesand please do have a look online forpictures ofscanning electron microscopeimages taken by scanning electronmicroscopethey're really neat and theyprovide nice examples of shading and ourability to interpret shading

Characteristic Strip Expansion, Shape from Shading, Iterative Solutions

let's start with some announcementswe've had three homework problems andwe'reready for the firsttake-home quizand the rules again are that it'll counttwice as much as the homework problemand there's no collaboration unlikehomework problemsso that's uhnext weekand thenthose of you in 686there's a proposal due on the 22ndand that's meant to beshortand that's where you tell me uh whatyou'll be doing for the term projectand sothe idea of the term project is to takesome machine visionproblem uh preferably something wediscussed in classand implement it in some wayand your choice it could bei don't know windows matlabandroidwhateverit could also be somethingmore theoretical it could besome mathematical solution to somemachine vision problem most people optto do some kind of implementationand it's very flexible i mean if youfind thatyou can use opencvto implement part of what you want to dogo ahead and do that but of course thenyour contribution has to be on top ofthat not just using opencv butdoing something useful with itandif you have problems coming up withideas send me emailwhat will happen with the proposal isi'll take a look at them and if i canpoint you to sources that might behelpful in implementing that projectthen then i will do sookay andwe'rejust about to finish our discussion ofhow to extract stuff from imagebrightnessandin particular shape from shadingand it's a little bit abstract a littlebit mathematical and would soon have abig change of pacewhen we start talking about industrialmachine visionand of course we can't cover everythingand we'll take a different approach tocovering it so rather than use publishedpapersor textbookswe'll look at patents and part of thereason for that is that you know in ourworld you publish papers that's what youget credit forin their world you don't publish that'swhat you get credit for sowhen you do see what they're doingit's in the patents where they're tryingto cover themselvesuhuhprotect themselves from somebody elseusing that same idea so that'll be a bigchange of pace and in the process we'lllearn a little bit aboutpatents and patent languagesince you know that's aimportant topic if you'rean entrepreneurinvolved in a startup or something ofthat sort so obviously that's going tobe a little different from partialdifferential equationsand some of you may be looking forwardto that but but let's finish with thepartial differential equations so whereare we oh okaywellfor example you could implement time tocontact on your android phoneandin that caseyou'd use android studioand[Music]i could supply you with a dummy projectthat's just a shellso that you don't have to uh write allof those files for android studio sothat's an example umyou could uh implement some of thesubpixelmethods that we'll talk aboutforedge detectionand usewhatever matlabwhatever is convenientum sothat's an example projectanother example more theoretical wouldbeyou know we've talked aboutshape from shading in the context ofparticular types of reflectance mapslike hapkiand you could implement you could workout the details for a different type ofreflectance map there would be a moreabstract mathematical projectandi think what i'll do is i'll pulltogether some of those andmaybe say something about them on on thestella websitesookay so uhthe first part of the term we werefocusing mostly onimage projection perspective projectionequation and derivatives motionmotion in the world motion in the imageand then we switch toone one thing that we can do with imagebrightness measurements which is theother half of image formationand in particularwe're looking at shape from shadingand so far we'vesolved the problem for very particularcasewhich is thehapka model of surface reflectanceandat the core of all of it isthe image irradiance equationwhich basically says thebrightness at some point in the imageis the reflectance mapcorresponding to that surfaceorientation so here we're focusing onthedependence of brightness on surfaceorientation and as we mentioned itdepends on illuminationit depends onsurfacematerial that's where the reflectancemap comes into itand it depends onthe geometry in particularin particular surface orientation andit's a local thing so that's sort ofgood it means that the brightnessmeasurement at a particular point in theimage typically depends on what'shappening at the corresponding point onon the objectand the reflectance map was our way ofsort of summarizinguh the detailed reflecting propertieswhich in turn atomicallyarereflected in the brdfso we had thebi-directional reflectance distributionfunctionbut we pretty muchbuilt up on that to where we have asomewhat easier to managereflectance map so the brdf depends onfour parameters the reflectance map onlydepends on two but it's it's derivedit's built on top of the the brdfokay umsowe looked at a particular casewhich is that ofreflecting properties of themoon the mara of mario of the moon andtherockyother rocky planetsand in that casewe found that we could solve thisproblem in a particular directionso in incase of the moonumif we take the ecliptic planethen that's the direction along which wecan actually determine the surfaceorientationsowe can integrate outin certain direction and we can'tintegrate at allin the direction at right angles and imean typically we'd be looking at somesmall patchbut this gives us the idea of whatdirections we can perform this in andwhatnotokay andwhat we ended up with is a set ofequationsthattake us from point to point on thesurface and first of all umx and y are varying in this way and ihave to apologize i think maybe lasttime i ended up with i was reversedi had the first one being qs and psand this corresponds to that angle ofrotation where we rotated one coordinatesystem to the otherand then what's uhthe change in heightuh well according to you know chain rulewhich sort of looks intimidating butit's just uhp timesthe derivative up here so we got umjust uhp times ps or ps plusqsqokay so thisis the rule we can use to take a smallstepuh and we we take a small stepin the imagebased on that and that corresponds to asmall step in heightandforgot to mention it here butwe're now assuming orthographicprojectionsothat's sort of a pointthat's important which is thatonce we switch to dealing withbrightness we switch to orthographicprojectionwhy well because it makes everythingmuch easier all of this can be done withperspective projection and it originallywas but the math gets messyand so uh the thing to remember is thatyou know pretending we have atelecentric lens we talked abouttelecentric lenses and uh using aorthographic projection is itcorresponds to being very far awayas we are when we're looking at the moonand the light source is very far awayand we have a very small visual angleand it simplifies things it's just likelambertian you knowit's not that these methods arerestricted to lambertian well in factwe're talking about non-laboration hereum it's just that uh you can do someinteresting things if you make that uhassumptionand then generalize from thereokay sothat's the ruleand basically we have three ordinarydifferential equationsthat we're going to solve numericallyand we don't need very sophisticatedmethods like you know eighth order rangecutter or something we're just going todoforward euler or in other wordsif you have the slope and you have astep size you just multiply the slope bythe step size to see how much higheryou've gone come sothat that's the method we're going touse and of course it's not terriblyaccurate butif we make the steps small enough it'sit's good enoughand the measurements we're dealing withare themselves noisy so it doesn't makesense to you know apply a method that'sgood to12 decimal places when we start off withthings based on image brightnessmeasurementsokaynowhow do we employ this well we need totie it to brightnessso somehow brightness has to feed intothis this equationwellfor the hapka type surface we havewe have this kind of dependence oractually it was cosinethetaandin ourarrangement for orthographic projectionit's very convenient to have the viewerup along the z axis sookayand then we can plug inin terms of p and qso that's in terms of angles and interms ofunit normalsbutwe we would like to express it in termsof p and qand then you remember that n dot s cameout to bethis thingand let's see we're going to take thesquare root of thatand then we're going to divide it byn dot v or where v is the same as z sothat's n dot z so that's one overand that conveniently cancels outso so we're in the end up with that sothis is our r of p and qand by the um our assumption here aboutthe image of radiance equation that's eof x and yso we can writee of x and y is this thingandyou can see the term we're looking foris right in here p s p plus q qsqso we need to squarethe whole thing get rid of the squareroot and then multiply through by rs andsubtract the 1. so we end up with umokay so for this particular surfacethere's a direct relationship betweenthe quantity we can measure uh eand the thing we needto continue the solutionright sowe are what's r s well r s is just umdependent on the source position it's aconstantand it just avoids having to write thatall the timeso we just take the brightness we squareit we multiply it by rs we subtract oneand there's the derivative in the zdirection and that's it we just marchalongyou know from one point to the nextadjustingx y and z as we go and at each point wedo we know the surface orientationwells a bit of itwe know the slope in in that direction imean that's what we're exploitingbut as we indicated we don't knowanything about the slope in the otherdirection so so no we don't don't knowthesurface orientationbased on this and we need something elseto do thatnowsince each of these profiles is going tobe independentto actually get z as a function of x andy in a real description of the surfacewe need to somehow have initialconditionfor these differential equationsso you know x and ywell we pick some point in the image tostartbut what about zwell under our assumption ofthisimage formation modelthere is no dependence on z thedependence is on the slope of zright soactually if we movedthisobject in the z direction its imagewouldn't change well under perspectiveprojection it would change in sizebut we're notdealing with perspective projectionwe're dealing with orthographicprojectionand so its size doesn't change so sothere's an ambiguityso for each of thoseuh curves that we're computingwe need an initial condition so actuallywe need an initial curveand so in 3d how do we do that wellhere's a way ofdefining a curvewe have some parameter that varies alongthe curve could be arc length or somearbitrary parameteretaand for each eta we give a position inspace x y and zand that's a curve and so that's ourlet's assume we have that initial curvesosome sort of curve like this and then wecan start at any point on that curveand integrate out those equationsnumericallyandthere's our surfaceand as we mentionedwe can actually go in both directionsfrom the initial curveand sowe end up withz of x y or actually z ofeta and psiright because the way we'veparameterized it isone parametergoes along thecurvethe other parameter goes along theinitial curve soso it's the surface in 3d and it takestwo parameters to parameterize that soso that'sstraightforward i hopethat in that particular case wehave some very special properties one ofthem is that we can locally determinethe slope in a particular direction andthat means of course we can go in thatdirection and and build up a curveand that'snot going to be true in the general casesowhat do we do aboutabout the general caseso we'll still start off withour image irradiance equationwhich says that the brightness at aparticular point in the imageis dependent on the surface orientationat the corresponding point point on theobject and we'll sort of try and followthis model here soyou know suppose we're at someparticular point x y zand then we take a small stepand in the imagelet's suppose the stepssize is delta x delta y andfor the moment we won't say whichdirection we're going we'll justleave thatunknownandto construct the solution what we needto know is you know what's z how is zgoing to changeand so of courseum we have that relationship the changein z is dz dx times delta x plus dz dytimes delta yand sowe can calculate the change in heightif we know p and qand suppose we know p and q then we'reat a new point on the surface and we canrepeat wetake a small step in x and y andnow over here we kept on going in in acertain direction thatin this case we may need tochoose a direction in some particularway butwe need to know p and qokay well uh we can assume that we startoff not only knowing x y and z but alsothe surface orientationso we could haveyou know x y and z and p and qbut then how do we update every step weneed toupdate p and qso here we have updates rules for x yand z for x and y you know we're theones controlling the stepand then zthe change in z is given by thisequationso um well we can use the same chainrule trick we can say that delta p is psub x delta x plusp sub y delta yand delta q is q sub x delta x plusq sub y delta yso thatwe can update p and q as we go along sowe're not only updating x y and z butwe're also updating p and q so this iskind of interesting before we had acurve in space we were tracing out x yand zas we construct the solution now we'vegot more because at every point we knowp and qwhich means we know the surfaceorientationso what we're really constructing now isa stripwell notnot very elegantand this is called a characteristicstrip characteristic of thatdifferential equation andthat means thatyou know we're carrying a long surfaceorientation so if i wanted to i coulderectsurface normals as as i go along sothat's obviously more information thanjust a curveokay so that's what we'll be doing we'llupdate not just x y and z but p and qwhich we didn't have to do over herebecause of the particularproperties of of the hapka type modelokay buthow do we do this well in order toupdate we need to know p x p yq x and q y andwe can write this another wayin matrix form so the two two linearequations two unknowns we can write itwith a two by two matrixand so what are these are are s and t soris p sub x which is really uhyou know the second derivative of zs is p sub ywhich is q sub x which is this onesothequantities we need in order to use thisupdate rule are the second partialderivatives ofheightand those are interesting because theycorrespond to curvature so the firstderivative have to do with surfaceorientation and the second derivativeshave to do with how quickly theorientation is changing and that ofcourse is curvature and fora 3d surface curvature is a little bitmore complicated than it is for say acurve in the planeand you needthree numbers to describe it so for acurve in the plane you can just give theradius of curvatureor the inverse of that which is calledcurvature just one number but for 3dsurface it's a little bit morecomplicated and you need this wholematrix ofsecond order derivatives and it's calledthe hessian matrixand in here i assume thatyou know the order of differentiationdoesn't matter that z of x y is z of y xand that will be true for somereasonable surfacewon't specify the exact conditions forthatand of course you can constructpathological thingsthat don't satisfy that butyou know those are mathematicalcuriosities rather thanreal surfaces that willmeet in machine visionokay so that's thecurvature matrix and souh we if we know the steps and we knowthat matrix we cancalculate the change in p and qand we can continue the solutionwell that kind of means that we shouldadd r s and tto our menagerie of variables so we'regoing to carry along x y and zp and qr s and tand yeah we can do that now how do weupdate our s and tsecond derivatives well we use the thirdderivativesso i think you can see where this isgoing this is kind ofcontinuing ad nauseam using higher andhigher higher order derivativesand so that's probably uh not going towork in factwe end up with more unknownsyou know here we've got before we didn'tknow p and qtwo unknowns now we don't know r s and tthree unknowns so it's it's not going inin a good directionbut what's sort of neat is thatwe haven't yet even used ou