Question Remote Window mouse control and screen recording to image file

[Dirk Pitt]

Hi All. I am an old boy from Italy (sorry in advance for my bad english )that love programming. Now i am exercising to improve my skills in .net (actually Vb.net, maybe in the future c# or other) . My main problem actually is API. There are lots of documents but i have not understood exactly how to use them in correct way. I spent an hour on microsoft docs searching for api to control processes and mouse movements, but nothing. I have found some codes but they were older. So please help me in the basics. After i will walk by myself as usual. So i am doing various things to have a problem, try to solve it, use commands, libs and so on. Visual Studio is nice, powerful but for me sometimes complicated. This time is the time of windows & processes. I have a window (any window of any kind, maybe a vb app, a browser or other. My idea was to scan the processes into my pc, search for the correct window, get the HANDLE (correct?) or ID (?) and after that try (trough API i think), control the process of the other window making various things, like move mouse, click somewhere and so on, maybe very interesting screen recording to file of the process. I will do that trough a Vb .net app Windows forms that i have done to do tests.
So... how to prepare the filed for using API, how to find the correct one and use it.
Thanks in advance for you cooperation.

 

[jdelano]

This is a super simple example using FindWindow, FindWindowEX, and SendMessage Windows APIs to click a button on another VB.Net app.

Public Class Form1

    Private Const BM_CLICK As Integer = &HF5
    Private Declare Auto Function FindWindow Lib "user32.dll" (ByVal lpClassName As String, ByVal lpWindowName As String) As IntPtr
    Private Declare Function SendMessage Lib "user32.dll" Alias "SendMessageA" (ByVal hWnd As IntPtr, ByVal wMsg As Integer, ByVal wParam As Integer, ByVal lParam As IntPtr) As IntPtr
    Private Declare Auto Function FindWindowEx Lib "user32.dll" (ByVal hwndParent As IntPtr, ByVal hwndChildAfter As IntPtr, ByVal lpszClass As String, ByVal lpszWindow As String) As IntPtr

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim hWndWinApp, hWndButton As IntPtr

        hWndWinApp = FindWindowLike("Another")

        If hWndWinApp <> IntPtr.Zero Then
            hWndButton = FindWindowEx(hWndWinApp, IntPtr.Zero, "WindowsForms10.Button.app.0.33c0d9d_r3_ad1", "Fill Chart") ' Replace with the button class and text
            If hWndButton <> IntPtr.Zero Then
                SendMessage(hWndButton, BM_CLICK, 0, 0)
                ListBox1.Items.Add("Sent the button click message")
            Else
                ListBox1.Items.Add("Didn't find the button")
            End If
        Else
            ListBox1.Items.Add("Process not found")
        End If

    End Sub

    Private Function FindWindowLike(ByVal sCaption As String) As IntPtr

        ListBox1.Items.Clear()
        Dim procList As Process() = Process.GetProcesses()
        For Each p As Process In procList
            If p.MainWindowTitle.Length > 0 Then
                ' fill the listbox with the processes that are available
                ListBox1.Items.Add($"PID: {p.Id} Window Title: {p.MainWindowTitle}")
                Application.DoEvents()

                If p.MainWindowTitle.Contains(sCaption) Then
                    ListBox1.Items.Add("Found it")
                    Return p.MainWindowHandle  ' window found, return window handle
                End If
            End If
        Next
        Return IntPtr.Zero ' window not found
    End Function
End Class

I used the Spy++ in the Tools menu of VS to find what the class of the Fill Chart button is in a vb.net app (that is what is on the right-hand side)

Here is a video of it running, clicking Button1 on the top app causes Fill Chart button to be clicked on the bottom app

 

[Dirk Pitt]

Thank you very much. What a nice example. Trough it i obtained the handle.

Now i have the Handle of the browser window, but trying to communicating with it nothing returns. here one example.

Public Sub MoveMouseToProcess(handleBrowser As IntPtr, x As Integer, y As Integer)

Dim rect As New NativeMethods.RECT()

NativeMethods.GetWindowRect(handlebrowser rect)

If Not NativeMethods.GetWindowRect(handlebrowser, rect) Then
MessageBox.Show("GetWindowRect in move mouse failed")
Return
End If

Dim absoluteX As Integer = rect.Left + x
Dim absoluteY As Integer = rect.Top + y

NativeMethods.SetCursorPos(absoluteX, absoluteY)

End Sub

I heard that permission are important sendind messages to processes. So i moved my program to a trusted location "c:\Program Files" and launched it as admin. Nothing returns.

 

[Dirk Pitt]

Maybe we'have another trouble, instead of the permission : i used spy++ and the handle shown bi it is different than mine. I am obtaining something not correct ?
here you are the routine to obtain handle:

Private Function FindWindowByPartialTitle(partialTitle As String) As IntPtr

Dim handle As IntPtr = IntPtr.Zero

NativeMethods.EnumWindows(New NativeMethods.EnumWindowsProc(Function(hWnd, lParam)

Dim title As New StringBuilder(256)

NativeMethods.GetWindowText(hWnd, title, title.Capacity)

If title.ToString().StartsWith(partialTitle, StringComparison.OrdinalIgnoreCase) Then
handle = hWnd
Return False ' stop enumerating
End If
Return True ' go on enumerating
End Function), IntPtr.Zero)
Return handle
End Function

 

[jdelano]

If you need to automate a webpage, take a look at Selenium WebDriver Most Complete Selenium WebDriver VB.NET Cheat Sheet

 

[Rythorian]

This effectively demonstrates how to interact with other applications by bringing a window to the foreground and simulating mouse clicks. By leveraging the Windows API through P/Invoke, developers can create powerful applications that enhance user interaction and automate tasks. This approach can be particularly useful in scenarios where user input is required in a different application, allowing for seamless integration and improved workflow.
Here's another example:

Imports System.Runtime.InteropServices
Imports System.Diagnostics

Public Class Form1
    ' Import necessary user32.dll functions
    <DllImport("user32.dll", SetLastError:=True)>
    Private Shared Function FindWindow(lpClassName As String, lpWindowName As String) As IntPtr
    End Function

    <DllImport("user32.dll", SetLastError:=True)>
    Private Shared Function SetForegroundWindow(hWnd As IntPtr) As Boolean
    End Function

    <DllImport("user32.dll", SetLastError:=True)>
    Private Shared Function SendMessage(hWnd As IntPtr, Msg As Integer, wParam As IntPtr, lParam As IntPtr) As IntPtr
    End Function

    <DllImport("user32.dll", SetLastError:=True)>
    Private Shared Function PostMessage(hWnd As IntPtr, Msg As Integer, wParam As IntPtr, lParam As IntPtr) As IntPtr
    End Function

    ' Constants for mouse events
    Private Const WM_LBUTTONDOWN As Integer = &H201
    Private Const WM_LBUTTONUP As Integer = &H202

    ' Function to find a window by its title
    Private Function GetWindowHandle(windowTitle As String) As IntPtr
        Dim hWnd As IntPtr = FindWindow(Nothing, windowTitle)
        If hWnd = IntPtr.Zero Then
            Throw New InvalidOperationException($"Window '{windowTitle}' not found.")
        End If
        Return hWnd
    End Function

    ' Function to bring a window to the foreground
    Private Sub BringWindowToFront(windowTitle As String)
        Try
            Dim hWnd As IntPtr = GetWindowHandle(windowTitle)
            If Not SetForegroundWindow(hWnd) Then
                Throw New InvalidOperationException("Failed to bring the window to the foreground.")
            End If
        Catch ex As Exception
            MessageBox.Show(ex.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
        End Try
    End Sub

    ' Function to simulate a mouse click
    Private Sub ClickMouse(windowTitle As String, x As Integer, y As Integer)
        Try
            Dim hWnd As IntPtr = GetWindowHandle(windowTitle)
            PostMessage(hWnd, WM_LBUTTONDOWN, 1, (y << 16) Or x)
            PostMessage(hWnd, WM_LBUTTONUP, 0, (y << 16) Or x)
        Catch ex As Exception
            MessageBox.Show(ex.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
        End Try
    End Sub

    ' Example usage
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        BringWindowToFront("Untitled - Notepad") ' Change to your window title
        ClickMouse("Untitled - Notepad", 100, 100) ' Change coordinates as needed
    End Sub
End Class