Screen Recognition -> finding out what colour something is

[Wibble]

What I want to know is: is it possible to detect things on the screen with VB.NET? If so, how!? (I dont imagine it will be easy but anything useful would be good)
I want to be able to find out what colour a number on screen is (well a range of numbers) and perform an action when a certain combinations of colours occur.

Im a bit of a novice but Im quite ambitious.
I mean, the fact that say red is displayed on the screen means that information for displaying red must be stored somewhere right?
Anything at all would be useful, even if you can just tell me if its possible or not.

-Wj

 

[JohnH]

It is not very difficult to get the color of a specific pixel pointed to by mouse anywhere on screen, but virtually impossible to get the color of a range of numbers because of the difficulty to calculate exactly what pixels to examine to get a specific but unknown digit number (also of unknown font).

 

[Wibble]

Hmm ok thanks.
Basically there are going to be 6 numbers continuously scrolling at random from a range of about 7 to 20-ish. They are all red but when they reach a specific number they will turn yellow. I want to know when they turn yellow. Can I set up something that says when pixel X turns yellow do this or that - but for four different pixels at once? Does it have to be on the point of a mouse?

-Wj

 

[JohnH]

It don't have to be the point of a mouse, but without human visual targeting you'll be having a lot of trouble getting reliable result, and talking about scrolling text you should be in for a treat, for instance if this is a window of another application will it be located at the very same pixel coordinates every time..? You can only get one pixel at a time, but it wont take you a millisecond to get four if you know what I mean. The code is very simple and no secret, here it is:

'Win32 API declarations
Private Declare Function GetPixel Lib "gdi32.dll" (ByVal hdc As Integer, ByVal nXPos As Integer, ByVal nYPos As Integer) As Integer
Private Declare Function GetDC Lib "user32.dll" (ByVal hWnd As Integer) As Integer
 
Private Function getScreenColor(ByVal x As Integer, ByVal y As Integer) As Color
Return ColorTranslator.FromWin32(GetPixel(GetDC(0), x, y))
End Function

Feed the function x/y screen coordinates and it will return Color.

 

[Wibble]

Ah thank you very much for that peice of code, very useful!
Ive been doing some playing around and have found out that the yellow isn't actually pure yellow. Each pixel is randomly one of every shade of yellow that is possible as far as Im aware. Ive been converting the results of the getScreenColor to a string and Ive so far gotten all these back:

Color [A=255, R=175, G=205, B=57]
Color [A=255, R=175, G=249, B=63]
Color [A=255, R=175, G=63, B=9]
Color [A=255, R=177, G=178, B=30]
Color [A=255, R=179, G=233, B=4]
Color [A=255, R=180, G=198, B=42]
Color [A=255, R=180, G=240, B=61]
Color [A=255, R=181, G=231, B=54]
Color [A=255, R=181, G=236, B=22]
Color [A=255, R=181, G=248, B=36]
Color [A=255, R=182, G=203, B=10]
Color [A=255, R=183, G=235, B=13]
Color [A=255, R=183, G=248, B=18]
Color [A=255, R=184, G=184, B=54]
Color [A=255, R=184, G=223, B=21]
Color [A=255, R=184, G=229, B=62]
Color [A=255, R=186, G=232, B=59]
Color [A=255, R=187, G=195, B=37]
Color [A=255, R=188, G=233, B=9]
Color [A=255, R=189, G=205, B=44]
Color [A=255, R=191, G=224, B=73]
Color [A=255, R=194, G=227, B=7]
Color [A=255, R=196, G=188, B=25]
Color [A=255, R=198, G=222, B=67]
Color [A=255, R=199, G=208, B=6]
Color [A=255, R=199, G=227, B=41]
Color [A=255, R=202, G=210, B=63]
Color [A=255, R=202, G=218, B=41]
Color [A=255, R=202, G=245, B=70]
Color [A=255, R=204, G=237, B=49]
Color [A=255, R=205, G=179, B=45]
Color [A=255, R=205, G=181, B=71]
Color [A=255, R=209, G=212, B=30]
Color [A=255, R=210, G=220, B=8]
Color [A=255, R=210, G=234, B=58]
Color [A=255, R=212, G=208, B=200]
Color [A=255, R=212, G=217, B=59]
Color [A=255, R=218, G=196, B=22]
Color [A=255, R=218, G=200, B=44]
Color [A=255, R=219, G=225, B=36]
Color [A=255, R=222, G=192, B=47]
Color [A=255, R=223, G=185, B=58]
Color [A=255, R=223, G=215, B=59]
Color [A=255, R=223, G=225, B=9]
Color [A=255, R=224, G=186, B=17]
Color [A=255, R=224, G=197, B=72]
Color [A=255, R=225, G=245, B=51]
Color [A=255, R=226, G=233, B=49]
Color [A=255, R=230, G=246, B=17]
Color [A=255, R=231, G=185, B=1]
Color [A=255, R=232, G=184, B=65]
Color [A=255, R=232, G=210, B=54]
Color [A=255, R=232, G=235, B=61]
Color [A=255, R=234, G=211, B=34]
Color [A=255, R=236, G=209, B=23]
Color [A=255, R=236, G=213, B=42]
Color [A=255, R=236, G=213, B=42]
Color [A=255, R=237, G=176, B=44]
Color [A=255, R=238, G=213, B=61]
Color [A=255, R=239, G=212, B=72]
Color [A=255, R=240, G=247, B=24]
Color [A=255, R=241, G=199, B=6]
Color [A=255, R=241, G=200, B=17]
Color [A=255, R=243, G=183, B=11]
Color [A=255, R=244, G=214, B=13]
Color [A=255, R=245, G=223, B=41]
Color [A=255, R=247, G=233, B=14]
Color [A=255, R=248, G=188, B=33]
Color [A=255, R=248, G=239, B=16]
Color [A=255, R=248, G=241, B=60]

And there has only been one duplication so far! Basically its going to be a LOT harder than I thought.

As for your two questions, the text scrolls once when enter is pused (using sendkeys) and the number isnt quite the same location so I would have to do 3 or 4 pixel reads per number for enter click.

My original idea was that, because there are different shades of yellow, Ill find them all out and put them into a string array to compair against the result of the getScreenColor. But seeing as there are clearly way more colors that I anticipated, I tried something like this:

strCurrentColour = x.ToString 'sets the pixel colour to strCurrentColour
For aaa = 174 To 248 'lowest to highest Red shade.
For bbb = 175 To 249 'lowest number is infact 63 but that is more of an anomaly and just adds extra search time
For ccc = 0 To 73 'again highest is 200 but thats too much extra searching for 1 colour
If strCurrentColour = "Color [A=255, R=" & aaa & ", G=" & bbb & ", B=" & ccc & "]" Then
Label1.Text = "Number is yellow"
ExitSub
Else
Label1.Text = "Number is not yellow"
EndIf
Next
Next
Next
(sorry, couldnt get the code display thing to work)

which does work, only it takes about 20 seconds to search all possibilities and with say 9 - 12 pixel reads per set of numbers this may not be worth the hard work! Sorry, Im mostly rambling and what Im asking isn't really anything related to graphis/GDI, more just wondering if you have any thoughts on it.

-Wj

 

[JohnH]

I have investigated that color range, most of it you got there is green actually. This was done by checking all possible colors within those RGB ranges and analyzing the ranges related to the HSL color space. To get a best guess if color is within the Yellow ranges limit Hue to something around 35-45, Red translates best to Hue 0-15 and 230-239 (from Hue 45 way up into the 100s the color is mostly very green). If you bring up any standard color dialog (from MSPaint for instance) and view the custom colors, you may point to any color with the mouse and see the values in the HSL color space change, it is easy to see how unique Hue values behave for colors the human eye perceive as distinct red, green, blue etc for any Luminance value (=brightness). So I would just check the range of Hue like this:

Dim c As Color = getScreenColor(x, y)
If c.GetHue > 35 and c.GetHue < 45 Then
  Label1.Text = "color is yellow" 
Else
  Label1.Text = "color is not yellow" 
End If

Also, your color comparison above using strings is highly inefficient, instead do the same thing as the Hue check in my example but with the R G and B properties of the Color instance.
Attached is an image of the color dialog. Hue around 40 with high Saturation and mid-high Luminance gives good yellows.

 

[Wibble]

That is actually a really good idea! After my last post I scrapped the for loops and used this:

TextBox1.Text = w.ToString
Try
TextBox1.Select(16, 3)
pixelResults(1) = TextBox1.SelectedText
TextBox1.Select(23, 3)
pixelResults(2) = TextBox1.SelectedText
Catch
label1.Text = "Number is NOT yellow"
EndTry


If pixelResults(1) > 174 And pixelResults(1) < 251 And pixelResults(2) > 174 And pixelResults(2) < 251 Then
label1.Text = "Number is yellow"
blStr = True
Else
label1.Text = "Number is NOT yellow"
EndIf

which does actually work perfectly because it seems all yellows are within the range of 174-200 on R and G. Ill try your method because it may be a bit shorter and will avoid the try catch and see which works best!
Thanks though, you've been extremely helpful!

 

[JohnH]

What on earth are you doing with that "pixelResults(1)=TextBox1.SelectedText" code?? You can't treat color as text, you must use the color structure and properties. There is absolutely no point in first getting a color structure, get some string interpretation of it, then trying to parse out the R/G/B values again. Trying to read out parts of those strings will in some cases also not work because pure colors are just displayed as "[white]" for instance.

Your Red-Green asumptions are also wrong, pure yellow is R=255,G=255,B=0 (that's a Hue 40, for the record).

Reading "my first VB.Net book" or something in that vein is highly recommended to get your first basic understanding of programming and common coding convensions.

 

[Wibble]

Perhaps, but like I said, it does work flawlessly! Ive changed the try/catch to the right exception and it really does work just as I need it to! (I couldn't get the w.gethue to work properly, it missed out some yellows and i tried some different ranges).
For this particular problem it seems all yellow colours are inbetween the range of 175 and 250 on both R/G. Im working on the logic that if

TextBox1.Select(16, 3)
pixelResults(1) = TextBox1.SelectedText
TextBox1.Select(23, 3)
pixelResults(2) = TextBox1.SelectedText

fails because the R/G numbers are shorter then that means its not a yellow colour (So thats why its in the try/catch). When the pure colours do come up (namly Color[Black]) it faills the try and is caught and then set to not yellow because if it was a yellow it would not produce an error with the try/catch .Yeah I know my methods are a bit dodgy but still, it works!!
Ill try and do some extra reading all the same though, cheers!
-Wj