Image Recognition (scary)

[Takteek]

Hi everyone,
I'm working on a programming project that involves image recognition. Given an image similar to this:

the program would need to read the numbers from the boxes into variables.

I was hoping that I could get some help with how I should go about programming this. Maybe some helpful links, etc.
Thanks in advance.

 

[JohnH]

- Aspress to try (and buy) http://asprise.com/product/ocr/index.php?lang=vbnet
- with Office lib http://www.codeproject.com/office/modi.asp

 

[Takteek]

Hm.. very interesting. I'll look into the Office one. I can't use the Asprise one because I can't pay for it.

Anyone else?

 

[shroomy]

Are the images/fonts of the numbers going to always be the same??

If so scrape (print screen or whatever) them, and put them into variables

Get a bitmap of the screen you are trying to read .. via the window handle or whatever ..

create a rectangle that could contain each number and grab that image fro the larger screen image in the previous step to get a bitmap of each number.

compare those with the images you save that you know represent each number.


some notes .....
if you can capture exactly the right area you can just compare the two bitmaps. If not you will have to move the image of the known bitmap around and check to see if any of them match.

If speed is not an issue you can use the getpixel method on each bitmap and just compare them. If it is you should read them into an byte array and compare the arrays.

Here is some code that should save you a few hours (or days )

[SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Private[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Function[/COLOR][/SIZE][SIZE=2] BitMapsAreEqual([/SIZE][SIZE=2][COLOR=#0000ff]ByVal[/COLOR][/SIZE][SIZE=2] BitMap1 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Bitmap, [/SIZE][SIZE=2][COLOR=#0000ff]ByVal[/COLOR][/SIZE][SIZE=2] BitMap2 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Bitmap) [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Boolean[/COLOR][/SIZE]
[SIZE=2][COLOR=#0000ff]'This is the slow but easy way and is more understandable
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] bRet [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Boolean[/COLOR][/SIZE][SIZE=2] = [/SIZE][SIZE=2][COLOR=#0000ff]True
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]If[/COLOR][/SIZE][SIZE=2] BitMap1.Size = BitMap2.Size [/SIZE][SIZE=2][COLOR=#0000ff]Then
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]For[/COLOR][/SIZE][SIZE=2] x [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Integer[/COLOR][/SIZE][SIZE=2] = 0 [/SIZE][SIZE=2][COLOR=#0000ff]To[/COLOR][/SIZE][SIZE=2] BitMap1.Width - 1
[/SIZE][SIZE=2][COLOR=#0000ff]For[/COLOR][/SIZE][SIZE=2] y [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Integer[/COLOR][/SIZE][SIZE=2] = 0 [/SIZE][SIZE=2][COLOR=#0000ff]To[/COLOR][/SIZE][SIZE=2] BitMap1.Height - 1
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] pix1 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Color = BitMap1.GetPixel(x, y)
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] pix2 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Color = BitMap2.GetPixel(x, y)
[/SIZE][SIZE=2][COLOR=#0000ff]If[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Not[/COLOR][/SIZE][SIZE=2] pix1 = pix2 [/SIZE][SIZE=2][COLOR=#0000ff]Then
[/COLOR][/SIZE][SIZE=2]bRet = [/SIZE][SIZE=2][COLOR=#0000ff]False
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]GoTo[/COLOR][/SIZE][SIZE=2] DoneCompare
[/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]If
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Next
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Next
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Else
[/COLOR][/SIZE][SIZE=2]bRet = [/SIZE][SIZE=2][COLOR=#0000ff]False
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]If
[/COLOR][/SIZE][SIZE=2]DoneCompare:
[/SIZE][SIZE=2][COLOR=#0000ff]Return[/COLOR][/SIZE][SIZE=2] bRet
[/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Function
[/COLOR][/SIZE] 
 
[SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Private[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Function[/COLOR][/SIZE][SIZE=2] BitMapsAreEqual3([/SIZE][SIZE=2][COLOR=#0000ff]ByVal[/COLOR][/SIZE][SIZE=2] BitMap1 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Bitmap, [/SIZE][SIZE=2][COLOR=#0000ff]ByVal[/COLOR][/SIZE][SIZE=2] BitMap2 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Bitmap) [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Boolean[/COLOR][/SIZE]
[SIZE=2][COLOR=#0000ff]'Faster Way
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] bRet [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Boolean[/COLOR][/SIZE][SIZE=2] = [/SIZE][SIZE=2][COLOR=#0000ff]True
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] bm_bytes1 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]New[/COLOR][/SIZE][SIZE=2] BitmapBytesRGB24(BitMap1)
bm_bytes1.LockBitmap()
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] bm_bytes2 [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]New[/COLOR][/SIZE][SIZE=2] BitmapBytesRGB24(BitMap2)
bm_bytes2.LockBitmap()
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] b1() [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Byte[/COLOR][/SIZE][SIZE=2] = bm_bytes1.ImageBytes
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] b2() [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Byte[/COLOR][/SIZE][SIZE=2] = bm_bytes2.ImageBytes
[/SIZE][SIZE=2][COLOR=#0000ff]If[/COLOR][/SIZE][SIZE=2] b1.GetUpperBound(0) <> b2.GetUpperBound(0) [/SIZE][SIZE=2][COLOR=#0000ff]Then
[/COLOR][/SIZE][SIZE=2]bRet = [/SIZE][SIZE=2][COLOR=#0000ff]False
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Else
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]For[/COLOR][/SIZE][SIZE=2] i [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Integer[/COLOR][/SIZE][SIZE=2] = 0 [/SIZE][SIZE=2][COLOR=#0000ff]To[/COLOR][/SIZE][SIZE=2] b1.GetUpperBound(0)
[/SIZE][SIZE=2][COLOR=#0000ff]If[/COLOR][/SIZE][SIZE=2] b1(i) <> b2(i) [/SIZE][SIZE=2][COLOR=#0000ff]Then
[/COLOR][/SIZE][SIZE=2]bRet = [/SIZE][SIZE=2][COLOR=#0000ff]False
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Exit[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]For
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]If
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Next
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]If
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Return[/COLOR][/SIZE][SIZE=2] bRet
[/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Function
[/COLOR][/SIZE] 
[SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Private[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Function[/COLOR][/SIZE][SIZE=2] GetBmpBytes([/SIZE][SIZE=2][COLOR=#0000ff]ByVal[/COLOR][/SIZE][SIZE=2] bmp [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Bitmap) [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Byte[/COLOR][/SIZE][SIZE=2]()[/SIZE]
[SIZE=2]'use this to convert a bmp to a byte array[/SIZE]
[SIZE=2]'comparing the arrays of two bitmaps is much quicker[/SIZE]
[SIZE=2] 
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] bitmap [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] Bitmap = [/SIZE][SIZE=2][COLOR=#0000ff]New[/COLOR][/SIZE][SIZE=2] Bitmap(bmp)
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] data [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] BitmapData = bitmap.LockBits([/SIZE][SIZE=2][COLOR=#0000ff]New[/COLOR][/SIZE][SIZE=2] Rectangle(0, 0, bitmap.Width, bitmap.Height), ImageLockMode.ReadOnly, bmp.PixelFormat)
[/SIZE][SIZE=2][COLOR=#0000ff]Dim[/COLOR][/SIZE][SIZE=2] rgbValues(data.Stride * data.Height) [/SIZE][SIZE=2][COLOR=#0000ff]As[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Byte
[/COLOR][/SIZE][SIZE=2]System.Runtime.InteropServices.Marshal.Copy(data.Scan0, rgbValues, 0, data.Stride * data.Height)
bitmap.UnlockBits(data)
data = [/SIZE][SIZE=2][COLOR=#0000ff]Nothing
[/COLOR][/SIZE][SIZE=2][/SIZE][SIZE=2][COLOR=#0000ff]Return[/COLOR][/SIZE][SIZE=2] rgbValues
[/SIZE][SIZE=2][COLOR=#0000ff]End[/COLOR][/SIZE][SIZE=2] [/SIZE][SIZE=2][COLOR=#0000ff]Function
[/COLOR][/SIZE] 
 
Public Function CaptureWindow(ByVal handle As IntPtr) As Image
Dim SRCCOPY As Integer = &HCC0020
' get te hDC of the target window
Dim hdcSrc As IntPtr = User32.GetWindowDC(handle)
' get the size
Dim windowRect As New User32.RECT
User32.GetWindowRect(handle, windowRect)
Dim width As Integer = windowRect.right - windowRect.left
Dim height As Integer = windowRect.bottom - windowRect.top
' create a device context we can copy to
Dim hdcDest As IntPtr = GDI32.CreateCompatibleDC(hdcSrc)
' create a bitmap we can copy it to,
' using GetDeviceCaps to get the width/height
Dim hBitmap As IntPtr = GDI32.CreateCompatibleBitmap(hdcSrc, width, height)
' select the bitmap object
Dim hOld As IntPtr = GDI32.SelectObject(hdcDest, hBitmap)
' bitblt over
GDI32.BitBlt(hdcDest, 0, 0, width, height, hdcSrc, 0, 0, SRCCOPY)
' restore selection
GDI32.SelectObject(hdcDest, hOld)
' clean up 
GDI32.DeleteDC(hdcDest)
User32.ReleaseDC(handle, hdcSrc)
' get a .NET image object for it
Dim img As Image = Image.FromHbitmap(hBitmap)
' free up the Bitmap object
GDI32.DeleteObject(hBitmap)
Return img
End Function 'CaptureWindow

 

[tyonuts]

First of all shroomy, you're WAY off the point! what takteek needs is a software similar to, let's say, HP ReadIris. That's a piece of soft which uses a scanner to get an image (if you don't specify a local one) and then converts the bitmap text into rtf. It's a little bit simpler than full handwriting recognition, but just as challenging. Comparing two bitmaps bit by bit is, let's say, less difficult.
So takteek, unless you can get the sources for ReadIris, i suggest trying a way to convert that C++ code into vb.net (from the office article). That, or learn how to create and use genetic algorhytms (took me 2 years)