Methods To Sense The 3D Surface/Structure Of A Book

[spamsickle]

andrewgreendf, thanks for the recommendation of Learning OpenCV. The discussion of epipolar geometry was reassuring; I guess my completely naive thoughts weren't completely ridiculous. It looks like two non-coplanar-image-plane cameras can be mathematically aligned, and the fact that we're using static cameras to photograph a static scene may even mean that we can use a series of images to refine the matrix used to perform this transformation.

I learn elsewhere that OpenCV should have CUDA support in spring 2011.

What I'm wondering now is how easy it will be to go from two camera images and 3D surface structure (I assume we'll be able to calculate this with the precision we'll need) to an "unwarped" page image. We won't just be back-projecting the image from the page onto a 2D plane, we'll be flattening the page itself. I assume some kind of inverse texture mapping operation will do it, but since I haven't written a texture mapping routine either, I'm expecting surprises.

Sorry for the software talk in the hardware section. Man with a hammer, and all that...

[vitorio]

What I'm wondering now is how easy it will be to go from two camera images and 3D surface structure (I assume we'll be able to calculate this with the precision we'll need) to an "unwarped" page image.

Did you see the paper I just linked, right before your post?

Also, take a look at OpenVIDIA if you're looking for OpenCV with GPU acceleration.

[spamsickle]

What I'm wondering now is how easy it will be to go from two camera images and 3D surface structure (I assume we'll be able to calculate this with the precision we'll need) to an "unwarped" page image.

Did you see the paper I just linked, right before your post?

The board told me that you'd posted while I was composing, but I didn't follow your links until after my post went through. To me, it looks harder than computing the 3D structure in the first place, but doable.

[daniel_reetz]

Sorry for the software talk in the hardware section. Man with a hammer, and all that...

That's my fault, not yours - since this is going to be a mix of hardware and software in any case, I couldn't think of a better place to put the thread.

[spamsickle]

To make it up, let me suggest an easy "maybe good enough" hardware solution.

See your method #2 - "Use the edges of the book"?

See the illustration right below it for method #3 (I forget the method) where the schematic shows the book edge on?

My suggestion would be to combine the two, and use the edges of the book with an image taken edge on. This could either be done with a third camera (or actually, the second camera, which it now occurs to me is no longer needed for stereoscopic matching), or by simply taking a single image which shows the curvature of the pages clearly. Now that I think about it, since we have two cameras, and one "above" the book can get both pages, I think it makes most sense to use the second camera to take the edge-on image which can provide the 3D information.

I think this should work pretty well, because the cross-section that's available from the side should be pretty consistent from the top edge of the book to the bottom edge. If the pages are "crinkled" by water damage or age, this won't work, but that's a tiny fraction of the books I expect to scan. I wouldn't be surprised if the results are as precise or even more precise than stereoscopic methods. It seems to me that the big problem to solve would be to detect and match the edges of the page, but I'd suggest going with the edge of the book (cover and all) instead -- it's easier to find, and the "map" should still be valid across the part we care about.

We're dealing with a very constrained problem, and while there is something to be said for tackling the challenge of implementing a more general solution just for the intellectual exercise, there's also something to be said for cutting as many corners as possible. We don't need to be able to find a general depth map to unwarp a book -- we may only need the view from the side of the book.

If someone has "dewarping when given a depth map" code handy, I'd be willing to give it a try this week.

[iwenttoofast]

Howdy Folks:

A couple of data points from the commercial side of the house:

We use i2S' software for this called Book Restorer.

http://www.i2s-bookscanner.com/produits ... 1_EDI_1003

It is a proprietary and rather expensive piece of software which we bought solely for two reasons: this algorithm (which is generally referred to as Geometric Curve Correction) and its Lighting Correction algorithm, which essentially evens out hotspotting, gutter shadows, and other illumination flaws on scanned pages. Both algorithms do an excellent job. The rest of the Book Restorer suite, IMHO opinion, isn't that great, and Photoshop does all of the same stuff, but better. But we thought it was worth the price just to have access to these two algorithms. We have used the Curve Correction with great success on actual book scans shot with our CopiBook as well as with tens of thousands of page images digitized from microfilm using non-i2S equipment. One drawback is that the text blocks, once straightened, sometimes become slightly trapezoidal in shape, but our customers feel that's much better than having to deal with the curve sloping into the gutter and poor OCR accuracy.

Note that this is a software-only solution, with no special hardware required. You tell it what page / text components to key off of and the software does the rest. The biggest problem with the software is that it's single-threaded -- processing only one page at a time -- so a single roll of microfilm yielding a few thousand grayscale TIFFs will take hours to process. But we just let it run all night / weekend on a batch, and stuff gets done faster than you'd think.

Out of respect for our dealer, I can't comment on Book Restorer pricing, but it's more expensive than something like Photoshop etc.

Zeutschel has a similar product, but I believe it's only compatible with their scanners -- best to check with them or one of their dealers. i2S' software is great because we can run any old image files from any scanner / camera through it and it does its thing.

Hope this is of interest . . .

-- Jim

[daniel_reetz]

Very interesting, thanks Jim!

[daniel_reetz]

I am going to prototype three methods so we can start playing with them.

The first method is the laser method. I believe that this method is the simplest and has the highest chance of success. Additionally, it can be combined with other methods to produce better results. As such, I made a frame today to test out different laser positions and stuff. I still need to make a mount for this frame, but I ran out of time today so for tonight it rests on the floor:

CIMG0061.JPG (119.47 KiB) Viewed 14506 times

The images look very promising.

CIMG0064.JPG (108.62 KiB) Viewed 14506 times

Obviously the hardware is large and bulky but need not be for the final version. In fact, it could be just a couple of rods that could go in a messenger bag, no problem.

The second method will be one similar to this, but looking at the book-edge as described by Spamsickle. I am less enthused about that idea, but the thing I like about both of them is that we can use the millions of pixels already available to us to get the work done. It's not like using the Kinect where the resolution is fixed. Depending on how we position the cameras/lasers we can use thousands or millions of pixels to represent a few centimeters of depth, resulting in very high accuracy.

Next up, I will be prototyping the lighting-based system. I like the lighting-based idea best, because it doesn't necessarily rely on exact positioning, and we need to light the book anyway so it isn't an extra component. I managed to fabricate two arrays today before running out of time. I am now waiting for two BuckPucks to come in the mail (hopefully tomorrow) to get this idea prototyped.

CIMG0065.JPG (136.84 KiB) Viewed 14506 times

Wish I'd got more done this weekend. I also worked on PatHall's BOX scanner a bit, but just don't feel good about the design yet.

[kasslloyd]

I like the idea of a laser grid.. using a few $8 or so (about what a line generator module is on ebay) to make a grid that is projected from above... using Arduino to trigger the cameras, shoudn't be hard to write a script that does everything with the press of a button with an Arduino in the following order for each:

2 main high rez cameras, positioned directly above the book, one for left one for right page, or I suppose one MUCH higher rez DSLR (20+ mp probably) directly above...
1 lower rez cheap digital camera that will be used to just take a picture of the laser grid.
lights on a relay that can be turned on and off
laser diodes directly above that project a grid on the book surface.

When you press the button it takes a picture with the high rez camera(s) with the bright lights on, then turns the lights off and the lasers on, then takes a picture of the grid over the book in the dark so the laser lines are very visible with the lower rez cheaper camera.

Provided everything is numbered properly each page should have the high rez images and a laser grid image paird with it.

Software could be written, this is the big part, to analyze the laser grids to extrapolate the 3d structure of the page, then dewarp it, maybe with some additional refinement with other software techniques like ScanTalior is working on now..

Provided there was actual software that would dewarp based on laser grid lines, this could be accomplished very cheaply.. a $30-40 camera and maybe $40 of lasers at most...

Tack on the aduino stuff... heh. Could be a little expensive in the rage of $100-200 of equipment.. but it has the possibility of being quite accurate. You may not even need another second cheaper camera, just take two pictures with the high rez camera(s) one with the lights on and lasers off and one with the lights off and lasers on, that might even be better... donno.

[spamsickle]

Oh, you boys and your lasers...

Dan, I could tell you weren't that keen on my suggestion when it didn't elicit an immediate "Great!", so I'm just happy it made the first cut. Naturally, in keeping with the "not invented here" principle, I still prefer it, but putting together systems to test should give us a more objective basis for comparison.

The thing that bothers me about both the laser and the lighting solutions is that you're going to have to calculate and infer the depth map. The thing I like about the laser solution is it may be more accurate if the cross-section isn't really uniform from the top of the book to the bottom. It seems to me that if uniformity is a problem, both the edge-on solution and the lighting solution will suffer.

From your pictures, it looks like there's a lot of scatter on those lasers, and I'm wondering if that won't make automated detection difficult, especially if you're implementing a grid instead of two lines. Will two lines be enough?

On the lighting solution, I'm wondering about how much diffraction is going to fuzz your edges, and add to the problem of inferring "book edges" from "shadow edges."

Just some random early-morning thoughts; if there's anything I can do to help in the prototyping or testing, don't hesitate to ask.