Yes, it’s another episode of Can’t Google My Way Out, with your host Jason Hughes!

While working on our editor, we just recently got to the point where we needed to do mouse selection of objects.  There were two methods I was considering for accomplishing this: 1) the obvious method that involves ray casting from the near-plane out to the far-plane and selecting the nearest object  the ray hits, and 2) rendering all objects to a render target that has object IDs stored per-pixel, and simply scrape through the render target to figure out what was clicked on.

The obvious method seemed simplest, so I went with that, for now.  Long-term, I’d really like to do method #2, since you can trivially do lasso selection and additive/subtractive regions in a single swipe.  Very cool.  The second method also has the extremely nice property that, if you can’t see it, you won’t select it by accident.  I hate editors that let you pick things you can’t see, since it’s very rarely what you wanted in the first place.

Well, the way I approached casting a ray in 3D based on a 2D point is not extremely complicated and can be found in bits and pieces on the web if you look.  The key is that I wanted to generate a ray in homogeneous clip space, not in camera or world space.  Why?  It’s simpler and works whether you’re using perspective OR orthographic projectiong matrices.  Of course, you could calculate things directly, but there’s at least two cases involved, and it may prevent computing wacky projection matrices if you do it the forward route.  So don’t.

The idea is your mouse cursor is floating along the near plane in clip space, which goes from -1 to 1 along X and -1 to 1 along Y and fixed at 0 or -1 on Z (depending on your graphics system, OpenGl and D3D disagree).  So take your 2D XY coordinate and scale and bias them based on the width of the viewport to be in that range.  Once you have that, your far plane point is the exact same XY position, but at a different fixed Z (again, depends on your graphics system).

Since this is in homogeneous  clip space, you now have a Vector4(xn,yn,zn,1) that represents your near point, and Vector4(xf,yf,zf,1) that represents your far point.  Multiply these vectors against the (4×4) inverse projection matrix.  Now you have camera-space points, but their XYZ are all weird and the W component is not 1 anymore.  That’s because it’s still a homogeneous vector.  You have to convert it to cartesian space, which is simply dividing each vector by its W component… ta-da!  Camera-space near and far plane points.

Oh, and one more thing: if for whatever reason you are modifying your projection matrix by negating certain axes, I found that I needed to perform the same negations when forming the near/far clip space vector, so it corresponds with my projection matrix, and not the graphics hardware’s clip space description. This is because you’re using the projection matrix you created, not the graphics driver itself.  Thus, take care that your orthographic and perspective matrices have the same clip space coordinates, or you’ll have to take that into account here.

Why post this? There are plenty of explanations for how to do some parts of this, but I haven’t found a place where all of the steps were explained in simple terms.

JH

spread the word:

You must be logged in to post a comment.