How do I figure out the anchor and active ends of the current selection of a TextBox or RichTextBox?

[RobertAlanGustafsonII]

WHAT I HAVE:
Visual Basic 2019, .NET Framework, WinForms

MY ISSUE:
I'd like to be able to programmatically figure out which end of a selection is the anchor and which is the active end when the user changes the selection of a TextBox or RichTextBox. The SelectionStart property is always the left end of a selection, and apparently EM_GETSEL always returns left/right, whereas EM_SETSEL sets anchor/active end. Left always comes before right, but the anchor can be after the active end. This matters if one has to temporarily change, then restore the selection programatically--as the user can thus find oneself extending the selection on the left, then suddenly contracting it on the right for no apparent reason!

The following StackOverflow page provided a supposed "kludge" fix in C++ and Delphi: "EM_GETSEL swaps parameters", but I haven't been able to successfully adapt the code there to VB.NET. The following code below always sets the Active end--and the caret (with no subsequent selection)--to the end of the full text:

<DllImport("User32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)> _
Public Shared Function SendMessage(hWnd As IntPtr, msg As Integer, ByRef wParam As Integer, _
        ByRef lParam As Integer) As IntPtr
End Function

Public Sub GetEnds(ByRef Anchor As Integer, ByRef Active As Integer)
        '   get anchor and active end of selection
        Const EM_GETSEL = &HB0, EM_SETSEL = &HB1
        Anchor = 0 : Active = 0
        Dim Start As Integer = 0, Finish As Integer = 0
        With TextBoxOrRichTextBox
            '    get ends of selection
            SendMessage(.Handle, EM_GETSEL, Start, Finish)
            '   temporarily kill selection to get active end
            SendMessage(.Handle, EM_SETSEL, -1, 0)
            SendMessage(.Handle, EM_GETSEL, Active, 0)
            '   restore selection and return values
            If Active = Start Then
                Anchor = Finish : SendMessage(.Handle, EM_SETSEL, Finish, Start)
             Else
                Anchor = Start : SendMessage(.Handle, EM_SETSEL, Start, Finish)
            End If
        End With
End Sub

I need a solution quickly. Please respond ASAP--and keep it in VB.NET (that goes without saying here), and as simple as possible.

 

[Rythorian]

THIS IS THE BEST I CAN DO FOR YOU, IT GIVES YOU AN EXAMPLE TO WORK WITH:

# Determine Anchor and Active End of Selection

Dim textBox As New RichTextBox()
Dim anchor As Integer
Dim active As Integer

' Assuming the user has made a selection in the RichTextBox
anchor = textBox.SelectionStart
active = textBox.SelectionStart + textBox.SelectionLength

If anchor > active Then
    ' Swap values if anchor is greater than active
    Dim temp As Integer = anchor
    anchor = active
    active = temp
End If

' Now anchor and active represent the correct ends of the selection

 

[RobertAlanGustafsonII]

THIS IS THE BEST I CAN DO FOR YOU, IT GIVES YOU AN EXAMPLE TO WORK WITH:

# Determine Anchor and Active End of Selection

Dim textBox As New RichTextBox()
Dim anchor As Integer
Dim active As Integer

' Assuming the user has made a selection in the RichTextBox
anchor = textBox.SelectionStart
active = textBox.SelectionStart + textBox.SelectionLength

If anchor > active Then
    ' Swap values if anchor is greater than active
    Dim temp As Integer = anchor
    anchor = active
    active = temp
End If

' Now anchor and active represent the correct ends of the selection

Anchor will never be greater than Active in the above example, because SelectionStart is always the left end of the selection, and SelectionLength is never negative; therefore, the swap will never take place! I provided the link to the StackOverflow article because apparently one has to use a WinAPI hack to figure out whether Anchor is < or > Active, and one needs to use the EM_SETSEL message if one wants to set the Anchor to be > Active. Check out the article and see if there's a flaw in my VB routine which tries to emulate the "kludge".

 

[Rythorian]

To determine the anchor and active ends of a selection in a TextBox or RichTextBox, access the SelectionStart and SelectionLength properties in control.

Key Changes:​

1. Initialization: The anchor and active variables are initialized directly upon declaration, which enhances readability and reduces the number of lines of code.
2. Swap Function: I introduced a Swap function to encapsulate the swapping logic. This not only makes the code cleaner but also adheres to the DRY (Don't Repeat Yourself) principle. By using a separate function, we can easily reuse the swapping logic if needed in the future.
3. Clarity: The code now clearly indicates its purpose and functionality, making it easier for other developers (or future you) to understand the logic at a glance.

' Determine Anchor and Active End of Selection

Dim textBox As New RichTextBox()
Dim anchor As Integer = textBox.SelectionStart
Dim active As Integer = textBox.SelectionStart + textBox.SelectionLength

' Ensure anchor is always less than or equal to active
If anchor > active Then
    Swap(anchor, active)
End If

' Now anchor and active represent the correct ends of the selection

' Function to swap two integers
Private Sub Swap(ByRef first As Integer, ByRef second As Integer)
    Dim temp As Integer = first
    first = second
    second = temp
End Sub

 

[RobertAlanGustafsonII]

This is what you've proposed--which provides only cosmetic changes to your "non-solution", sorry:

To determine the anchor and active ends of a selection in a TextBox or RichTextBox, access the SelectionStart and SelectionLength properties in control.

Key Changes:​

1. Initialization: The anchor and active variables are initialized directly upon declaration, which enhances readability and reduces the number of lines of code.
2. Swap Function: I introduced a Swap function to encapsulate the swapping logic. This not only makes the code cleaner but also adheres to the DRY (Don't Repeat Yourself) principle. By using a separate function, we can easily reuse the swapping logic if needed in the future.
3. Clarity: The code now clearly indicates its purpose and functionality, making it easier for other developers (or future you) to understand the logic at a glance.

' Determine Anchor and Active End of Selection

Dim textBox As New RichTextBox()
Dim anchor As Integer = textBox.SelectionStart
Dim active As Integer = textBox.SelectionStart + textBox.SelectionLength

' Ensure anchor is always less than or equal to active
If anchor > active Then
    Swap(anchor, active)
End If

' Now anchor and active represent the correct ends of the selection

' Function to swap two integers
Private Sub Swap(ByRef first As Integer, ByRef second As Integer)
    Dim temp As Integer = first
    first = second
    second = temp
End Sub

You still don't get it. SelectionStart always returns the "left end" of the selection, and SelectionStart is always positive--indicating the distance to the "right end". The Swap function will never come into play, as anchor is never greater than active to begin with; even if it did, it would force anchor to always come before active, negating the whole exercise of determining the direction of the selection--as opposed to simply finding the ends!

Similarly, the Windows message GET_SEL always has the first value as the left end and the second as the right end, even if the user has expanded the selection leftwards, so that the right end is the "anchor" (fixed) and the left end is the "active end" (can move). The SET_SEL message, like the user, can set the selection forwards (left end anchor, right end active) or backwards (right end anchor, left end active)--however, the GET_SEL message all by itself cannot be used to discover the direction of the selection: it only returns the left and right ends without telling which is anchor and which is active.

Once again, the anchor is the "fixed" physical start from which a selection is extended, while the active end is the "movable" part; that's not respective to the left and right ends (as opposed to vice versa) unless it's a forward selection. Basically, some kind of kludge is needed to determine the direction, according to the web-page referred to in my original link (please read it)--when I adapt the code in the referred page to VB.NET, it doesn't work! Somebody please help me! There are times (like when one is temproarily modifying the selection) when one needs to know which direction the user has extended the selection (so that one can properly reset it, i.e., using SET_SEL).

P.S. Stop trying to re-tell me your previous "solution". I've already told you it won't work, so do the research I did, and look into the matter yourself;--that is, about programmatically finding out the direction of a selection. I await anyone's coming up with something.

 

[JohnH]

GetCaretPos seems to work for me, try this out:

Private Sub DebugSelection(tb As TextBoxBase)
    Dim pt As Point = Point.Empty
    If GetCaretPos(pt) Then
        Dim caret = tb.GetCharIndexFromPosition(pt)
        Dim selStart = tb.SelectionStart
        Dim selEnd = selStart + tb.SelectionLength
        Debug.WriteLine($"{selStart}-{selEnd} active: {caret}")
    End If
End Sub

<DllImport("user32.dll")>
Private Shared Function GetCaretPos(ByRef lpPoint As Point) As Boolean
End Function

 

[RobertAlanGustafsonII]

How does this work? I mean, how does getting the character index of an empty point derive the active (movable) end of a selection?

Also, I'm having difficulty with using SendMessage and EM_SETSEL to correctly set the sends when I want the active end to precede, not follow, the anchor end (to allow the user to extend the selection backwards). (Properly setting the SendMessage parameters with DllImport platform invoke is always tricky for me.) The whole point of my problem is to figure out the direction of the selection, and, when restoring the selection after changing it, preserving the original direction of it (via EM_SETSEL). (Otherwise, if a user is initially extending it on the left end, and I have to change-and-restore, he/she might suddenly then be contracting on the right end.)

 

[JohnH]

It is not empty after GetCaretPos return the point ByRef.

 

[JohnH]

You don't need EM_SETSEL to set selection with anchor/direction, use the Select(start, length) method where start is the anchor and positive or negative length.

(for EM_SETSEL selection is set as (start,end) where end can be lower value than start)

 

[RobertAlanGustafsonII]

It is not empty after GetCaretPos return the point ByRef.

The active-end value returned by this code can be off by +1, 0, or -1 compared to the actual active-end value; this often is the case when the active end is at the beginning or end of the full text. This means that when just 1 character is selected, this isn't 100% reliable in terms of figuring out the direction of the selection. Also, it gets set to 0 whenever another text control (but not another non-text control) gets the focus. Also, isn't necessarily reliable when the selection is set programmatically and/or when the control doesn't have the focus. Basically, it is most reliable when the control has the focus (set by the user), and, better yet, the selection is set by the user.
I need something a tad more fool-proof. (Good work though.)

 

[RobertAlanGustafsonII]

I think I've solved it! Do the following:

    <DllImport("User32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)> _
    Private Shared Function SendMessage(hWnd As IntPtr, msg As Integer, _
        ByVal wParam As Integer, ByVal lParam As Integer) As IntPtr
    End Function

    Private Function GetSelectionInfo(RichTextBox As RichTextBox, _
        Optional ByRef Anchor As Integer = -1, _
        Optional ByRef Active As Integer = -1) As Integer
    '   get anchor/active ends of selection and distance between
    Const EM_SETSEL = &HB1

    With RichTextBox
        Dim Start As Integer = .SelectionStart, _
            Distance As Integer = .SelectionLength
        If Distance = 0 Then
            '   no selection
            Anchor = Start : Active = Start
            Return 0
         Else
            '   temporarily deselect to get active end
            SendMessage(.Handle, EM_SETSEL, -1, 0) 'MUST use API!
            If .SelectionStart = Start Then
                '   backward selection
                Anchor = Start + Distance : Active = Start
                Distance = -Distance
             Else
                '   forward selection
                Anchor = Start : Active = Start + Distance
            End If
            '   restore selection
            .Select(Anchor, Distance)
            Return Distance
        End If
    End With
    End Function

Note that Anchor and Active are optional, but if included, then they must be passed as variables since they are ByRef in order to supply info to the calling code. In any case, whether included or not, the return value is equal to Active - Anchor.

Also note that the SendMessage API is only needed to temporarily kill the selection and set the caret to the active end as SelectionStart; everything else works using the standard properties.

To eliminate flicker, you can temporarily suspend redrawing the control, and save & restore the scroll position of the control as well as the selection while redrawing is suspended. (These also require Win32 APIs!)

BTW, this only works with RichTextBoxes, not standard TextBoxes! Why?! I don't know.

 

[JohnH]

Great. There is only one caret per message queue, and it is only set when a edit control has focus, which is why there is no GetCaretPos for specific edit control window, it can only return position for the last edit control that had focus.

I also noticed the selection is only visible for TextBox when the edit control had focus
(edit: scratch that, forgot to set HideSelection=False for the TextBox test)

I changed your method from RichTextBox As RichTextBox to edit as TextBoxBase and passed a TextBox after making a selection, and it worked for me, both when selection is visible and not.