Question Remote Window mouse control and screen recording to image file

[Dirk Pitt]

Hi All. I am an old boy from Italy (sorry in advance for my bad english )that love programming. Now i am exercising to improve my skills in .net (actually Vb.net, maybe in the future c# or other) . My main problem actually is API. There are lots of documents but i have not understood exactly how to use them in correct way. I spent an hour on microsoft docs searching for api to control processes and mouse movements, but nothing. I have found some codes but they were older. So please help me in the basics. After i will walk by myself as usual. So i am doing various things to have a problem, try to solve it, use commands, libs and so on. Visual Studio is nice, powerful but for me sometimes complicated. This time is the time of windows & processes. I have a window (any window of any kind, maybe a vb app, a browser or other. My idea was to scan the processes into my pc, search for the correct window, get the HANDLE (correct?) or ID (?) and after that try (trough API i think), control the process of the other window making various things, like move mouse, click somewhere and so on, maybe very interesting screen recording to file of the process. I will do that trough a Vb .net app Windows forms that i have done to do tests.
So... how to prepare the filed for using API, how to find the correct one and use it.
Thanks in advance for you cooperation.

 

[jdelano]

This is a super simple example using FindWindow, FindWindowEX, and SendMessage Windows APIs to click a button on another VB.Net app.

Public Class Form1

    Private Const BM_CLICK As Integer = &HF5
    Private Declare Auto Function FindWindow Lib "user32.dll" (ByVal lpClassName As String, ByVal lpWindowName As String) As IntPtr
    Private Declare Function SendMessage Lib "user32.dll" Alias "SendMessageA" (ByVal hWnd As IntPtr, ByVal wMsg As Integer, ByVal wParam As Integer, ByVal lParam As IntPtr) As IntPtr
    Private Declare Auto Function FindWindowEx Lib "user32.dll" (ByVal hwndParent As IntPtr, ByVal hwndChildAfter As IntPtr, ByVal lpszClass As String, ByVal lpszWindow As String) As IntPtr

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim hWndWinApp, hWndButton As IntPtr

        hWndWinApp = FindWindowLike("Another")

        If hWndWinApp <> IntPtr.Zero Then
            hWndButton = FindWindowEx(hWndWinApp, IntPtr.Zero, "WindowsForms10.Button.app.0.33c0d9d_r3_ad1", "Fill Chart") ' Replace with the button class and text
            If hWndButton <> IntPtr.Zero Then
                SendMessage(hWndButton, BM_CLICK, 0, 0)
                ListBox1.Items.Add("Sent the button click message")
            Else
                ListBox1.Items.Add("Didn't find the button")
            End If
        Else
            ListBox1.Items.Add("Process not found")
        End If

    End Sub

    Private Function FindWindowLike(ByVal sCaption As String) As IntPtr

        ListBox1.Items.Clear()
        Dim procList As Process() = Process.GetProcesses()
        For Each p As Process In procList
            If p.MainWindowTitle.Length > 0 Then
                ' fill the listbox with the processes that are available
                ListBox1.Items.Add($"PID: {p.Id} Window Title: {p.MainWindowTitle}")
                Application.DoEvents()

                If p.MainWindowTitle.Contains(sCaption) Then
                    ListBox1.Items.Add("Found it")
                    Return p.MainWindowHandle  ' window found, return window handle
                End If
            End If
        Next
        Return IntPtr.Zero ' window not found
    End Function
End Class

I used the Spy++ in the Tools menu of VS to find what the class of the Fill Chart button is in a vb.net app (that is what is on the right-hand side)

Here is a video of it running, clicking Button1 on the top app causes Fill Chart button to be clicked on the bottom app

 

[Dirk Pitt]

Thank you very much. What a nice example. Trough it i obtained the handle.

Now i have the Handle of the browser window, but trying to communicating with it nothing returns. here one example.

Public Sub MoveMouseToProcess(handleBrowser As IntPtr, x As Integer, y As Integer)

Dim rect As New NativeMethods.RECT()

NativeMethods.GetWindowRect(handlebrowser rect)

If Not NativeMethods.GetWindowRect(handlebrowser, rect) Then
MessageBox.Show("GetWindowRect in move mouse failed")
Return
End If

Dim absoluteX As Integer = rect.Left + x
Dim absoluteY As Integer = rect.Top + y

NativeMethods.SetCursorPos(absoluteX, absoluteY)

End Sub

I heard that permission are important sendind messages to processes. So i moved my program to a trusted location "c:\Program Files" and launched it as admin. Nothing returns.

 

[Dirk Pitt]

Maybe we'have another trouble, instead of the permission : i used spy++ and the handle shown bi it is different than mine. I am obtaining something not correct ?
here you are the routine to obtain handle:

Private Function FindWindowByPartialTitle(partialTitle As String) As IntPtr

Dim handle As IntPtr = IntPtr.Zero

NativeMethods.EnumWindows(New NativeMethods.EnumWindowsProc(Function(hWnd, lParam)

Dim title As New StringBuilder(256)

NativeMethods.GetWindowText(hWnd, title, title.Capacity)

If title.ToString().StartsWith(partialTitle, StringComparison.OrdinalIgnoreCase) Then
handle = hWnd
Return False ' stop enumerating
End If
Return True ' go on enumerating
End Function), IntPtr.Zero)
Return handle
End Function

 

[jdelano]

If you need to automate a webpage, take a look at Selenium WebDriver Most Complete Selenium WebDriver VB.NET Cheat Sheet