Question ComPorts and Threading?

[JEofVA]

If this is not the right forum for this question, then please point me in the right direction.

I'm brand new to Visual Basic 2008 (just started with it 5-6 weeks ago) and I'm pretty stuck. I've managed to learn enough to develop a small program that reads the ASCII "sentences" from my GPS receiver by polling the port. I've also managed to parse the sentences into usable data, and feel pretty sure that I can manage to do what I want to with the data. However, polling the port takes too much time. The sentences that the GPS sends take about 1.4 seconds to arrive in the computers UART buffer, leaving 0.6 seconds to do what I want with everything, which is most probably not enough time.

What I'd like to do is let the Serial port's input buffer fill to a predetermined level (about 600 bytes), and then have the ComPort's "ReceivedBytesThreshold" method trigger, causing a subroutine to read the entire contents of the input buffer all at once. I'm hoping that will give me about 1.9 seconds to do what I want with the data.

In order to test that theory I wrote a little routine that determines the amount of time it will actually take to read everything at once. However, when I go to update a text box on the form with the "start" and "end" times I get a threading runtime error.

The best I can gather for what I've managed to read is that when the "ReceivedBytesThreshold" method is triggered it creates a new thread, which limits me from modifying anything in the original thread from data in the new thread. However, there is something called Invoke and BeginInvoke that may let me get the job done, but I have no idea what I am doing so I cannot get it to work.

Can anyone here help me with this problem? I will be happy to share the code, but I thought I would wait and see if anyone was willing to lend me a hand first.

Thanks in advance,

Jonathan

 

[jmcilhinney]

[ame="http://www.vbforums.com/showthread.php?t=498387"]This[/ame] provides a pattern you can follow to access controls from worker threads.

 

[JEofVA]

JM,

Thank you for the link to the explanation for getting data from a secondary thread to the primary thread. I was able to use that information to confirm my "theory" that there would be a substantial time savings by using the ComPort's "ReceivedBytesThreshold" method trigger to signal when to collect the data in the Com Port buffer. I was effectively able to cut the read time down from about 1600 ms to as little as 10ms (average ~13ms).

However, I'm not sure how to utilize the techniques you outlined to get the abstract data that I parse from the data "sentences" (e.g. latitude, longitude, time, depth, temperature, etc.) from the serial port buffer to the various display and storage objects (e.g., database record, graphic display, main form fields, secondary serial output). I've tried to hold the data in a Structure data type that I can pass in and out of the various routines that will handle the storing and displaying of the data.

Unfortunately, I can't seem to make that work using the Invoking techniques that you outlined. I suspect that I probably using an inefficient data structure. Can you help me figure out a better (more proper) way of moving the collected data from the secondary thread to the main thread so it can be disseminated?

Thanks again for previous answer,

Jonathan

 

[jmcilhinney]

Saving data to a database or the like can be done from your secondary thread. You only need to invoke a method on the UI thread in order to update the UI.

In that submission of mine the first step was to just write a method that does the job you want done as though there was just one thread. If you can do that and post it, then we can work from there. Most likely the first thing we'll do is break it up into at least two parts and separate the operations that can be performed on the current thread from those that require marshalling to the UI thread.

 

[JEofVA]

Well, I have structured the program as I believe you suggested, and here is the code:

Public Class MainForm
    Dim WithEvents ComPort_IN As New IO.Ports.SerialPort
    Dim ComPort_OUT As New IO.Ports.SerialPort
    Dim endprogram As Boolean = False


    Public Structure degree_type
        Public deg As Integer
        Public min As Single
        Public hemi As String
    End Structure
    Public Structure Fix
        Public update As Boolean ' Flag indicating whether the current set of data is new
        Public time As String
        Public lat As degree_type
        Public lon As degree_type
        Public depth As Single
        Public tempC As Single
        Public tempF As Single
        Public direction As Single
        Public warning As String
        Public qual As String   ' Indicates the quality of the GPS signal 
        Public diff_id As String
        Public diff_sec As String 'Age in seconds since last update from diff. reference station
    End Structure
    Public Fix_arr(2) As Fix    ' Holds the parsed data from one to three blocks of NMEA sentences
    Public isRunning As Boolean


    Private Sub MainForm_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load

        btnStop.Enabled = False

        'Fill ComPort_IN Selector with all available ports
        For i As Integer = 0 To My.Computer.Ports.SerialPortNames.Count - 1
            cbbCOM_In.Items.Add(My.Computer.Ports.SerialPortNames(i))
        Next

        cbbCOM_In.Text = My.Computer.Ports.SerialPortNames(0) 'preset to the lowest in the list
        cbbCOM_Out.Text = My.Computer.Ports.SerialPortNames(1) 'preset to the next lowest in the list


    End Sub

    Private Sub subPause(ByVal msec As Integer)
        Dim dtCheck As DateTime = Now

        Do
        Loop While Now < dtCheck.AddMilliseconds(msec)
    End Sub


    Private Sub btnStart_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnStart.Click
        'Dim arrDataline As Array
        'Dim testFixdata As Fix

        btnStart.Enabled = False
        btnStop.Enabled = True

        InitializeComPortOUT()
        ComPort_OUT.Open()
        PresetTitler()

        If ComPort_IN.IsOpen Then
            ComPort_IN.Close()
            Application.DoEvents()
        End If
        InitializeComPortIN()
        ComPort_IN.Open()

        ' Set up COM Trigger level
        ComPort_IN.ReceivedBytesThreshold = 600 ' set trigger to a few bytes less than a full block of sentences

        PresetComPortIN() ' Prime the COM port so that the first record is first in the queue


        isRunning = True

'  ===>  MOVED TO "bufferReady_Trigger" SUBROUTINE <====
        'If Fix_arr(0).update = True Then
        ' UpdateDataBase(testFixdata)
        'UpdateTitler(Fix_arr(0))
        'UpdateScreen(Fix_arr(0))
        ' Update the Map
        'UpdateScreen(Fix_arr(0))
        'Fix_arr(0).update = False ' Reset the 

        'End If


    End Sub

    Private Sub bufferReady_Trigger(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ComPort_IN.DataReceived
        Threading.Thread.CurrentThread.Name = "Trigger Thread"

        '  This routine is called when the ComPort's input buffer has at least 600 bytes in
        '    it.  First, it reads the entire block of sentences from the input buffer, one  
        '    at a time. After splitting each sentence into an array of it's parts, and 
        '    checking to see if the sentence had been transmitted intact, it checks to see 
        '    if it's one of the sentences of interest. If so, it parses the data according 
        '    to the type of sentence it is. When the last sentence in the block is parsed, it sets 
        '    an "" flag and exits the DO Loop. After exiting the Loop, if the Update has 
        '    new data (.update=True), then the four primary updating subroutines are executed.

        Dim arrDataline(20) As String

        Fix_arr(0).update = False ' Make sure if this data is bad that the data is ignored by presetting it
        Do
            arrDataline = getDataLine() ' Returns a two element array; [0] has sentence identifier, 
            '                                                          [1] has comma-delimited datastring
            If arrDataline(0) <> "Bad" Then
                ' Decide if it is a string you want to decode
                Select Case arrDataline(0)
                    Case "$GPGLL"
                        If arrDataline(6) = "A" Then ' "A" means the data is valid
                            Fix_arr(0) = Parse_GLL(arrDataline)
                        Else
                            Fix_arr(0).qual = "BAD"
                        End If
                    Case "$SDDPT"
                        Fix_arr(0).depth = Parse_DPT(arrDataline(1)) ' pull out the depth information
                        'Debug.WriteLine("arrDataline(0): >" & arrDataline(0) & "<  arrDataline(1): >" & arrDataline(1) & "<")
                    Case "$SDMTW"
                        Dim arr_Temp As Array
                        arr_Temp = Parse_MTW(arrDataline)
                        Fix_arr(0).tempC = arr_Temp(0)
                        Fix_arr(0).tempF = arr_Temp(1)
                        Fix_arr(0).update = True ' finished processing the all the NMEA sentences

                End Select

            Else
                ' The record is bad so indicate that by setting the time component to "BAD"
                Fix_arr(0).update = False
                Exit Do
            End If

        Loop Until Fix_arr(0).update = True

        If Fix_arr(0).update = True Then
            UpdateTitler(Fix_arr(0)) ' Send data out the second COM port to the video Titler
            UpdateDatabase(Fix_arr(0)) ' Store data in GPS Fix table - UNWRITTEN
            UpdateScreen(Fix_arr(0)) ' Display position, depth, temp data to screen
            UpdateMap(Fix_arr(0)) ' Redraw map with current position as the center - UNWRITTEN
        Else
            '  ???  Just hang out until the next valid update comes along???

        End If

    End Sub

    Private Sub btnStop_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnStop.Click

        isRunning = False
        ComPort_IN.Close()
        ComPort_OUT.Close()
        btnStop.Enabled = False
        btnStart.Enabled = True

    End Sub

    Private Sub InitializeComPortIN()
        Dim baud As String = "4800"
        Dim databits As Integer = 8
        Dim stopbit As String = "One"
        Dim parity As String = "None"

        ' Fill the COM In label fields
        lblBaudIn.Text = "Baud: " & baud
        lblDatabitsIn.Text = "Data: " & databits
        lblParityIn.Text = "Parity: " & parity
        lblStartbitIn.Text = "Start: 1"
        lblStopbitIn.Text = "Stop: 1"

        If ComPort_IN.IsOpen Then
            ComPort_IN.Close()
        End If

        With ComPort_IN
            .PortName = cbbCOM_In.Text
            .BaudRate = baud
            .Parity = IO.Ports.Parity.None
            .DataBits = databits
            .StopBits = IO.Ports.StopBits.One
            .Handshake = IO.Ports.Handshake.XOnXOff
        End With



    End Sub
    Private Sub InitializeComPortOUT()
        Dim baud As String = "9600"
        Dim databits As Integer = 8
        Dim stopbit As String = "One"
        Dim parity As String = "None"

        ' Fill the COM In label fields
        lblBaudOut.Text = "Baud: " & baud
        lblDataOut.Text = "Data: " & databits
        lblParityOut.Text = "Parity: " & parity
        lblStartOut.Text = "Start: 1"
        lblStopOut.Text = "Stop: 1"

        If ComPort_OUT.IsOpen Then
            ComPort_OUT.Close()
        End If

        With ComPort_OUT
            .PortName = cbbCOM_Out.Text
            .BaudRate = baud
            .Parity = IO.Ports.Parity.None
            .DataBits = databits
            .StopBits = IO.Ports.StopBits.One
            .Handshake = IO.Ports.Handshake.XOnXOff
        End With



    End Sub

    Private Sub PresetComPortIN()
        Dim dataline As String

        Do
            dataline = ComPort_IN.ReadLine() ' Read until get to start
        Loop Until (InStr(dataline, "$SDVHW") >= 1)

    End Sub

    Private Sub PresetTitler()
        Dim dlay20 As String = Space(20)
        Dim ESC As String = Chr(27) & "00"

        ComPort_OUT.Write(ESC & "25" & dlay20)          ' clear all lines 
        subPause(100)                                   ' PAUSE 100ms
        ComPort_OUT.Write(ESC & "07" & dlay20)          ' Video ON
        subPause(20)                                    ' PAUSE 20ms
        ComPort_OUT.Write(ESC & "10" & dlay20)          ' Background: 09=On, 10=Off
        ComPort_OUT.Write(ESC & "11" & dlay20)          ' Character color: 11=White, 12=Black
        ComPort_OUT.Write(ESC & "17" & "3" & dlay20)    ' Set Contrast: 1=Low, 2=Medium, 3=High
        ComPort_OUT.Write(ESC & "44" & "10" & dlay20)   ' Set Horizontal Position (just seems to work)
        ComPort_OUT.Write(ESC & "45" & "1" & dlay20)    ' Horizontal Letter Size = 1
        ComPort_OUT.Write(ESC & "46" & "19" & dlay20)   ' Set Vertical Position (just seems to work)
        ComPort_OUT.Write(ESC & "47" & "1" & dlay20)    ' Vertical Letter Size = 1


    End Sub

    Private Function getDataLine() As Array

        Dim strNMEAline As String
        Dim arrSplitString(2) As String
        Dim arrNMEAdata(20) As String


        strNMEAline = ComPort_IN.ReadLine()
        If InStr(1, strNMEAline, "?") = 0 Then

            ' Split the sentence from the checksum
            arrSplitString = Split(strNMEAline, "*") ' [0] = datastring; [1] = checksum
            If arrSplitString.Count = 2 Then

                ' Save NMEA string's checksum
                Dim NMEACheckSum As String
                NMEACheckSum = arrSplitString(1).Remove(2) ' This is the check sum string that came with the NMEA sentence

                ' Run Checksum to see if it matches
                Dim charCount As Integer
                Dim intsum As Integer = 0 'Initialize it to zero to be sure where we are starting 
                For charCount = 1 To (Len(arrSplitString(0)) - 1)     ' Loop through all chars to get a checksum; skip $
                    ' Is checksum currently being processed?
                    If intsum <> 0 Then
                        ' Yes. XOR the checksum with this character's value
                        intsum = intsum Xor Val(Asc(arrSplitString(0)(charCount)))
                    Else
                        ' No. This is the first time through so set the checksum to the value
                        intsum = Val(Asc(arrSplitString(0)(charCount)))
                    End If
                Next

                ' If the calculated checksum matches the two character string at the end of the NMEA sentence...
                If (Strings.Right("00" & Hex(intsum), 2) = NMEACheckSum) Then
                    arrNMEAdata = Split(arrSplitString(0), ",", -1) ' Create an array starting at '0'
                Else
                    arrNMEAdata(0) = "Bad"  ' otherwise, send message back that string had non-matching checksums

                End If
            Else
                arrNMEAdata(0) = "Bad"  ' Corrupted string: arrSplitString not split into two strings
            End If
        Else
            arrNMEAdata(0) = "Bad"  ' "?"'s in string indicating a corrupted string
        End If

        Return arrNMEAdata  ' Called by bufferReady_Trigger


    End Function
    Function Parse_GLL(ByVal arrNMEAdata As Array) As Fix
        Dim structFix As Fix

        ' Parse the sentence array
        With structFix
            .lat.deg = CInt(Strings.Left(arrNMEAdata(1), 2))
            .lat.min = CSng(Strings.Mid(arrNMEAdata(1), 3, Len(arrNMEAdata(1) - 2)))
            .lat.hemi = arrNMEAdata(2)
            .lon.deg = CInt(Strings.Left(arrNMEAdata(3), 3))
            .lon.min = CSng(Strings.Mid(arrNMEAdata(3), 3, Len(arrNMEAdata(3) - 3)))
            .lon.hemi = arrNMEAdata(4)
            .time = Strings.Left(arrNMEAdata(5), 2) & ":" & _
                    Strings.Mid(arrNMEAdata(5), 2, 2) & ":" & _
                    Strings.Right(arrNMEAdata(5), 2)
        End With

        Return structFix


    End Function
    Function Parse_DPT(ByVal str_depth As String) As Single
        Dim depth As Single
        If str_depth <> "" Then
            depth = CSng(str_depth)
        Else
            depth = 0.0
        End If

        Return depth
    End Function

    Function Parse_MTW(ByVal arrNMEAdata As Array) As Array
        Dim arr_Temp(2) As Integer
        ' Parse the sentence array
        If arrNMEAdata(1) <> "" Then
            If arrNMEAdata(2) = "C" Then
                arr_Temp(0) = CSng(arrNMEAdata(1))          ' Centigrade
                arr_Temp(1) = (arr_Temp(0) * 9 / 5) + 32    ' Fahrenheit
            Else
                arr_Temp(1) = CSng(arrNMEAdata(1))          ' Fahrenheit
                arr_Temp(0) = (arr_Temp(1) - 32) * 5 / 9    ' Centigrade
            End If


        Else
            arr_Temp(0) = 0.0
            arr_Temp(1) = 0.0
        End If

        Return arr_Temp
    End Function

    Sub UpdateTitler(ByRef structFix As Fix)
        ' This subroutine sends data to a video titler through a second COM Port
        Dim dlay20 As String = Space(20)
        Dim ESC As String = Chr(27) & "00"
        Dim strTime As String
        Dim arrTime As Array
        Dim strDate As String
        Dim strDepth As String
        Dim strOutput As String
        Dim today As Date = Now


        strDate = today.Year.ToString.Substring(2) ' Gets the last two digits of the Year string
        strDate &= today.Month.ToString.PadLeft(2, "0") ' Creates a two character string of the Month with "0" padding
        strDate &= today.Day.ToString.PadLeft(2, "0") ' Creates a two character string of the Day with "0" padding

        arrTime = structFix.time.Split(":")
        strTime = arrTime(0) & arrTime(1) & arrTime(2) ' Create a six character time string

        strDepth = Format("###.#", structFix.depth.ToString).PadLeft(5, " ") & "FT"

        strOutput = ESC & "28" & "001" & strDate & "." & strTime & strDepth
        ComPort_OUT.Write(strOutput)

    End Sub
    Sub UpdateScreen(ByRef testFixdata As Fix)
        ' Code that updates the Main UI

        lblLatStatus.Text = testFixdata.lat.hemi & "" _
                        & CStr(testFixdata.lat.deg) _
                        & "° " & CStr(testFixdata.lat.min) & "'"
        lblLongStatus.Text = testFixdata.lon.hemi & "" _
                       & CStr(testFixdata.lon.deg) _
                       & "° " & CStr(testFixdata.lon.min) & "'"
        lblDepthStatus.Text = testFixdata.depth & " Ft"
        lblTempFStatus.Text = CStr(testFixdata.tempF) & "° F"
        lblTempCStatus.Text = CStr(testFixdata.tempC) & "° C"

    End Sub

    Sub UpdateDatabase(ByRef testFixData As Fix)

        ' Code to update the PositionData table goes here

    End Sub

    Sub UpdateMap(ByRef testFixData As Fix)

        ' Code that updates and redraws a map showing all the datapoints that fall within the lat/long 
        ' boundaries of the map.

    End Sub
End Class

I indicated in the "btnStart_Click" subroutine where I originally had the 4 routines that were suppose to do all the updating. However, according my understanding of your instructions, I moved them to the bottom of the routine that gets called when the Input Com buffer gets triggered. Two of the "updating" routines have not been written yet because I felt I needed to get past the threading issue first. I tried to put in lots of comments, but it may not have been enough in spots.

Currently, the program chokes when it tries to execute the "UpdateScreen" (UI screen) subroutine.

I really appreciate the help.

Thanks,

Jonathan

 

[cjard]

Er.. I don't think you should be manipulating the data on the com port trigger thread. Really you ought to be reading the data from the com port and dumping it into a container, like a Queue(Of String) then starting a backgroundworker (if it is not already running) to consume the container. Doing a lot of work on the thread that processes the event is going to cause problems.

Essentially you can get away with an event handler that looks like:

'class level variable
Private _databuffer as New Queue(of String)

Sub X Handles ComportEvent

  _dataBuffer.Add(comport_buffer_read_returning_string())

  If Not _backgroundWorker.IsBusy() Then
    _backgroundWorker.RunWorkerAsync()
  End If
End Sub

Your background worker's dowork event handler will process the queue one item at a time then revert to a stopped stage (if it finishes first) or continue consuming messages as they arrive

Note that .net naming conventions for methods are like:

InitializeComPortOut
ParseGll
ParseDpt
Pause (not subPause)

 

[JEofVA]

Thanks for the feedback. As I said at the beginning of the post, I'm very new to Visual Basic 2008, just started in May, and am definitely NOT up-to-speed on OOP.

I don't think you should be manipulating the data on the com port trigger thread.

Originally, I wasn't. Not knowing much about Visual Basic, and needing some kind of "container" to hold the various bits of data, such as strings, integers, and reals, I stumbled across "structures" which seemed like they would hold the data that I collected in the triggered event handler nicely, and might allow me to send the collection back to the UI thread for dissemination to the various subroutines in one easy-to-access package. Unfortunately, "getting it back" wasn't so easy.

When I originally started working with worker threads I followed [ame="http://www.vbforums.com/showthread.php?t=498387"]jmcilhinney's "tutorials"[/ame] on the subject, which worked fine for sending text directly back to the UI thread. However, I couldn't figure out how to do that with the structure that I created.

Really you ought to be reading the data from the com port and dumping it into a container, like a Queue(Of String)

Until you mentioned it, I had never heard of the construct. I tried to find out more information about it from the MSDN Visual Studio Developer Center, which is pretty sparse. According to what I read, the "queue" is a FIFO structure and therefore I wouldn't think it is very conducive to random access of the data contained within it. The data does go into the structure in order, but its access needs are going to be different for each "updating" subroutine. Of course, I could be misunderstanding your intention.

How about hashtables? Could they be set up to handle my data, and pass it between the threads any better than the "structure" structures?

I will have to read up more on the backgroundworker method before I can really respond intelligently. I am not understanding if you are having that replace the triggered read from the serial port or if it is suppose to be another thread that runs parallel to the UI thread and the triggered ComPort thread.

As far as the naming conventions go, I will look into that and correct my routine names.

Thanks for your assistance,

Jonathan

 

[cjard]

Thanks for the feedback. As I said at the beginning of the post, I'm very new to Visual Basic 2008, just started in May, and am definitely NOT up-to-speed on OOP.

Originally, I wasn't. Not knowing much about Visual Basic, and needing some kind of "container" to hold the various bits of data, such as strings, integers, and reals, I stumbled across "structures" which seemed like they would hold the data that I collected in the triggered event handler nicely, and might allow me to send the collection back to the UI thread for dissemination to the various subroutines in one easy-to-access package. Unfortunately, "getting it back" wasn't so easy.

You defintiely wouldnt use a struct for this:
Value Type Usage Guidelines

For what youre after, it doesnt behave like a value type and have an instance size under 16 bytes

Until you mentioned it, I had never heard of the construct. I tried to find out more information about it from the MSDN Visual Studio Developer Center, which is pretty sparse. According to what I read, the "queue" is a FIFO structure and therefore I wouldn't think it is very conducive to random access of the data contained within it. The data does go into the structure in order, but its access needs are going to be different for each "updating" subroutine. Of course, I could be misunderstanding your intention.

The queue is a buffer used to store the GPS strings prior to parsing. I can't understand why you'd need random access to this, as youre essentially consuming sentences in order in repetitive fashion. You use the comport reading thread to offload the data into the buffer, and then another thread consumes it if there is anything to consume. That other thread can write the data to database, screen, bang it into a class (not a struct), whatever is necessary and the comport thread is free to return to wherver it came from as soon as possible.

How about hashtables? Could they be set up to handle my data, and pass it between the threads any better than the "structure" structures?

You havent really given much of a commentary on what youre doing with this data, and I don't fancy reading thousands of lines of code to work it out but use of a hash table is still kinda old school. If youre looking to make an object representation of the GPS data, you'd use a class, populate it on the background worker thread and put it into a container ready for processing by something else

I will have to read up more on the backgroundworker method before I can really respond intelligently. I am not understanding if you are having that replace the triggered read from the serial port or if it is suppose to be another thread that runs parallel to the UI thread and the triggered ComPort thread.

Think of the comport event handler as a baker, baking bread. Every time bread is ready, a hatch opens and out comes the loaf, then the hatch closes and the baker goes back to what he is doing. The bread goes on a conveyor (queue - fifo) to you, who is a background worker who consumes the bread. You work continuously, sometimes faster than the baker, sometimes slower. Bread builds up on the conveyor, or you wait for it. You turn the bread into sandwiches then put them in boxes, in other containers and store them. You can do all that because youre fast. and good. You don't want to bother the baker much because the process of opening the hatch to get the bread into your work zone is quite complicated, and you certainly don't want the baker to put the bread through the hatch, then come round and turn it into sandwiches himself because it's not his job.

So, decide what youre doing with this data, and I can help you work out where to put it. If I was writing an app that.. let's say, recorded everywhere my car had been then I'd push the sentances into a queue and consume them with a background worker. The worker would parse the data, possibly into a special object, or maybe just into something suitable for storing in a DB. I'd run an insert query on the data and insert it, keeping a log of everything the GPS had emitted. Once per 10 seconds I would update a display in the car by grabbing the most recent positional information that had been built, and showing it. This would require use of the same thread that built the UI, and it would definitely not be the thread from the backgroundworker that consumed all the sentences, so I'd use the backgroundworker's ReportProgress functionality to have the correct thread read the data and update the UI. Remember, in the interim 10 seconds, I might have inserted 5 records into the DB, but only shown one.

 

[JEofVA]

You havent really given much of a commentary on what youre doing with this data, and I don't fancy reading thousands of lines of code to work it out

I don't blame you for not wanting to read through all the amateur code, I put it up there because I thought that jmcilhinney was suggesting that I do that.

So, here's what I'm trying to do:
Every 2 seconds (2000ms +/- 20ms) my GPS unit puts out 14 formatted (NMEA 0183) "sentences" in a comma delimited form, complete with a checksum at the end. (The time from the start of the first sentence to the arrival of the last sentence is about 1350ms, with no transmission during the rest of the time - not critically important, but FYI). Of the 14 sentences, I am only interested in the data in three of them (for instance, I have no interest in how many satellites are above me, or how far above or below the geoid I happen to be). When a block of 14 sentences have been transmitted they constitute about 650-700 bytes of text. The 600 byte trigger level thus represents an almost completed cycle of text sitting in the GPS's buffer. Through experimentation I found out that once the trigger level has been reached, I can read down the entire buffer in 10-20 ms, therefore I should safely have about 1950 ms to "pull the bread out of the oven, and do with it as I will before the next 14 loaves are ready".

So, if I understand your analogy correctly, once the14 "loaves of bread" slide out of the "baker's oven", instead of the "baker" slicing up the "loaves" in front of the "oven" and putting only the "slices" that I want into a "single pan" on a "conveyor belt" that goes to my workstation, "the baker" should instead just "push" (enqueue) each of the uncut "loaves" into a "train of loaf pans" (queue), and conveyor that to my work area, where I can do with them as I will.

Okay, assuming that is the algorithm that I use, here is what has to happen. Once I have the queue of raw NMEA sentences where I can work on them, I then have to put each of them through a parser that determines whether they are a type of sentence that I want to mess with, and if so, if their checksum is valid. Once both of those conditions have been met, I can parse the sentence to get out the data that I want. That is already worked out. I'm still not clear what kind of "holder" I put the data in, but once the last sentence has been parsed into whatever data structure I end up using, I have to use the data in that structure to:

• Update a video titler through a second serial port using specially crafted input sentences,
• Insert a new record in a database table. (I have tried to create a Jet database table that all of the positional information will go in, but I'm not sure if I have that done correctly or not.)
• Update some labels and whatnot on the screen with the current GPS data,
• Redraw a map graphic (on the UI) depicting the current position in the center and the other positions plotted around it (to the extent that they are within the bounds of the map). Ideally, I'd like to color code each position dot according to the depth recorded at that position, and I'd like to be able to zoom the map. But those things aren't absolutely necessary, and I'm probably out of development time anyway.

Once those steps are accomplished, that's it. After that the program can sit and wait for the next set of "loaves" to come popping out of the "oven" (assuming that I can do all that in 1800 ms, or so).

I hope this clarifies what I'm trying to do a little,

Thanks,

Jonathan

 

[cjard]

Mmmm.. I really wouldnt do all that in one thread and definitely not the thread that invokes the com port event because that thread might actually be the one doing the underlying com port stuff too; as an example of how designing a program incorrectly can have an impact on other processes, i've managed to hang Windows Explorer in the past, by doing too much file processing in a drag drop handler..

The notion before would still stand, your com port event trigger just pushes the sentences into a queue. You shouldnt worry about setting your trigger at such a level that each time the trigger fires it has all the 14 sentences youre looking for; just push stuff into a queue continuously and pull them out in parts. You might find it easier to use a StringBuilder instead. Suppose the alphabet is arriving on the com port and the trigger is 10 bytes. The handler looks like:

sub whatever handles comport event
  _gpsDataBuffer.Append(comportDataString)

  If Not _backgroundworker.IsBusy() Then _backgroundWorker.RunWorkerAsync()

end sub

Pushing into a queue would leave you having called (over a period of time):


_gpsDataBuffer.Append("abcdefghij")
_gpsDataBuffer.Append("klmnopqstuv")
_gpsDataBuffer.Append("wxyzabcdef")

Note I deliberately put 11 bytes into entry 2

Now consume your queue in the background:

sub worker_dowork handles _backgroundWorker.DoWork

  Dim idx as Integer = _gpsDataBuffer.ToString().IndexOf("z")
  While idx >= 0

    Dim sentence as String = _gpsDataBuffer.ToString(0, idx + 1)
    _gpsDataBuffer.Remove(0, idx + 1)

    Dim gd as GpsData = ParseGpsData(sentence) 'GpsData is a class representing your data

    _mostRecentlyParsedGpsData = gd 'store it for use by the ui thread, maybe 10 parsings will take place before the next UI refresh. we don't care

    InsertToDB(gd) 'record the info in the db

    idx = _gpsDataBuffer.ToString().IndexOf("z") 'try to parse another sentence

  End While

i.e. just append everything you get from the gps on a regular basis and then on a different period (how often do you want to read the buffer and parse the data?), check the contents of the stringbuilder for the presence of the end-of-14-sentences-identifier. At that point, pull all the data up to the identifier and parse it on a background thread into a class (not structure) representing the data.
Insert the class data youre interested in, into a database. Keep a reference to the most recently parsed data.



On yet another different period (how often do you want the screen data to refresh), update your screen from the most recently parsed data. You do this in this way because the thread that built the UI should not be bogged down parsing or inserting data as it will cause the display to hang and not respond to input

ui_timer_handler handles someTimer.Tick

  'draw the data out of _mostRecentGpsData on screen/video overlay
  'remember to cache the _mostRecentGpsData object here:

  Dim tmpGd as GpsData = _mostRecentGpsData

  'that way it can be changed to a new instance by the parser routine and not affect your display

end sub

Rules:
Offload the data from the comport as quickly as possible and let the event finish
Parse the data on a background thread as often as you like, probably on a shorter time period to the display refresh.. You can use the com trigger to start the background worker, who will look for work to do and stop if there is none
Use a timer on your UI to read the most recently known data and redraw the display. The UI thread should finish the timer handler as quickly as possible, to avoid hanging the display. If data isnt available, don't wait for it.

Use variables at the class level, syncronizing them if necessary, or at least making a local pointer to them rather than using the class level pointers (a good way to come unstuck with multithreading)

 

[JEofVA]

Thanks for the reply. I haven't had the time to fully digest what you are saying, because I had to go on the road to attend an urgent business meeting that was originally scheduled for next week. I just arrived and have about a half hour to reply to this, so it will be short.

I think my attempt to run with your analogy has backfired.... What I was trying to say is that out of the 2000 milliseconds that it takes for an entire block of 14 NMEA sentences to cycle, it only takes 10 to 20 milliseconds (less than 1% of an entire cycle) to read all 14 sentences from the COM buffer and send them to the backgroundworker routine. Therefore, it I don't see the benefit of chopping the data up into such small bits and sending them piecemeal to the backgroundworker routine. Also, reading all 14 sentences at once seems to me to be the most expedient way to meet your first Rule:

Offload the data from the comport as quickly as possible and let the event finish

Am I missing something?

I haven't had a chance to absorb the rest of what you are suggesting, but will read it over carefully as I get a chance.

Thanks again for the input .... got to run,

Jonathan

 

[cjard]

What I was trying to say is that out of the 2000 milliseconds that it takes for an entire block of 14 NMEA sentences to cycle, it only takes 10 to 20 milliseconds (less than 1% of an entire cycle) to read all 14 sentences from the COM buffer and send them to the backgroundworker routine. Therefore, it I don't see the benefit of chopping the data up into such small bits and sending them piecemeal to the backgroundworker routine.

I don't recall saying that the com port reading thread had to do any chopping at all. Just push data continuously and in small chunks. What I'm trying to get across is that you dont have to have a trigger level of 600 because your 14 sentences are 650 bytes long and by the time the trigger fires there will be 650 bytes available. It can be a problematic assumption to make because if the offsets drift, you'll end up with 650 bytes being in your buffer, but it's half of the last 14 sentences and half of the next.
You could push in chunks of 5 bytes if you wanted to, you'd push 130 times and then there'd be enough data to parse.. Just push data regularly and empty 14 sentences out when there are 14 sentences available; this is the essence of a queue; you continually add more data to the stringbuilder and then remove a portion of it when that portion is a complete set of 14 sentences

 

[JEofVA]

Sorry it is taking me so long to reply, but I've been out of town and when I got back found that this laptop would not stay up for more than 10 minutes before spontaneously shutting down. It has taken me a couple of days to track the problem down to software I put on it over 2 months ago. At least, that seems to be the problem.

Anyway, back the the issue at hand. I see your point about "offset drift", which is certainly a problem to consider. I hate seeming so dense, but I'm still not clear about what you are proposing. Are you suggesting that the algorithm should be written so that the SerialPort.ReceivedBytesThreshold trigger should trigger every time there are 5 bytes in the buffer, and that the trigger routine would send the contents of the buffer to a stringbuilder object in another thread, which would get parsed when it had a complete last sentence in it? Or, are you saying that the the trigger would be set for a larger number of bytes (650 -700?), but that the buffer would only be read 5 bytes at a time until the buffer was empty, and that the reading routine would send the data to the stringbuilder routine in 5 byte bites?

Also, what I think I should have done long before now is give you a better sense of how the data sentences are structured. Each "sentence" has the following structure:

$ABCDE,d1,d2,d3,...,dn*XXcr/lf​

where:

$ABCDE - is the sentence identifier
d1 - dn - are comma-delimited ASCII data elements representing integers, reals, and short text stings. Their number and types vary from sentence to sentence (but remain the same for each type of sentence)
* - separator character between the sentence and the checksum
XX - the XORed hexadecimal checksum
cr/lf - sentence terminator​

So, the "last sentence" starts with a particular identifier string, but doesn't end with anything in particular since the checksum characters will change with the changing data in the sentence. The only way to know that you have read the complete sentence is when you encounter the cr/lf characters. Therefore, it seemed logical to use the Readline method to pull data from the COM port buffer, and to do that reading when the buffer most likely contained nearly a complete block of sentences.

The 600 byte trigger level was selected so that the COM port buffer would be emptied before it held a complete last sentence, requiring that it be polled for a few milliseconds until the cr/lf characters of the last sentence arrived in the buffer. However, like you say, if the identifier of the last sentence gets garbled for some reason then the reading routine would keep reading until it encountered the next a good "last sentence" identifier, which would be at least two seconds away. That would essentially lock up the computer for that time.

Perhaps one solution might be to set a 50 ms timer at the beginning of the buffer-reading routine so that, if I haven't gotten the 14 sentences read by then, I know there is a problem. I can then dump the data, go back to the UI thread, reinitialize the COM port by reading sentences until I encounter a "last sentence", then I allow the COM port buffer to fill again and the process starts anew. It would mean that all of the updating routines would be in limbo until things started again, but that would be okay if the recovery time was only a few seconds.

Still hanging in there,

Jonathan

 

[cjard]

Are you suggesting that the algorithm should be written so that the SerialPort.ReceivedBytesThreshold trigger should trigger every time there are 5 bytes in the buffer

I used the word "could", not "should"

, and that the trigger routine would send the contents of the buffer to a stringbuilder object in another thread, which would get parsed when it had a complete last sentence in it? Or, are you saying that the the trigger would be set for a larger number of bytes (650 -700?), but that the buffer would only be read 5 bytes at a time until the buffer was empty, and that the reading routine would send the data to the stringbuilder routine in 5 byte bites?

Neither; if you review the pseudocode I provided earlier you can see that all I'm proposing is using a stringbuilder like a bucket, pouring stuff in straight away, as soon as it is received, and then in a separate action, taking stuff out when it is right to do so. This is the point of a buffer; it fills at some rate, maybe uneven, sometimes fast, sometimes slow, and then a point is reached where an action can take place, some of the buffer is emptied, and an action performed on the stuff that was emptied.

I made 5 bytes an EXAMPLE of how you could have it work. If you chose 5 bytes as a trigger level,you'd have a lot of buffer writes before you take action. If you have a massive trigger, you'd have few buffer writes before an action. Typically you choose some value that is a performance trade off; 5 bytes is probably too small, but 650 bytes might be too large. Why? let's say your device produces a sentence that is 651 bytes long, and it produces 650 bytes one every 5 seconds. The buffer is empty, 650 bytes come in after 5 seconds, then another 650 bytes come in 5 seconds after. Thus, we are 10 seconds down the line and we finally have 651 bytes of a full sentence after 10 seconds.
Now say you'd set your trigger at 135 bytes, which drip in at a rate proprtional to the 650/5sec (650/5sec ~= 130/sec). How long will it take us to get 651 bytes in 135 byte drips? ABOUT 5 seconds (a little bit more, because 135 bytes come every 1.038 seconds)

Depending on the performance requirements of your app it might be perfectly OK to wait 10 seconds for a sentence, but if the trigger were smaller, it'd be closer to 5 seconds consistently

Therefore, it seemed logical to use the Readline method to pull data from the COM port buffer, and to do that reading when the buffer most likely contained nearly a complete block of sentences.

It makes sense, but doing so will probably see you doing most of your work on a thread that is tied to the native thread handling the com port; when your routine takes a long time and blocks it, it may cause a problem. This is why we are rolling our own buffered solution; we are getting data off the comport and buffering it on a regular basis, and then we can take our time in reading and doing something with it

The 600 byte trigger level was selected so that the COM port buffer would be emptied before it held a complete last sentence, requiring that it be polled for a few milliseconds until the cr/lf characters of the last sentence arrived in the buffer. However, like you say, if the identifier of the last sentence gets garbled for some reason then the reading routine would keep reading until it encountered the next a good "last sentence" identifier, which would be at least two seconds away. That would essentially lock up the computer for that time.

It's too complicated! You're asking your baker to notify you before he's baked a full loaf, and then stopping what you do while you pull the last slices out of the hatch, causing him a nuisance too.
The comport reader routine should just dump data into a buffer as and when. Then, let's say once a second (use a Timer), a routine on a separate thread (use BackgroundWorker) should look to see if there is anything worth processing. Maybe there will be 1, or 2, or 3, or 57 if your GPS suddenly went nuts and spat out a load of stuff.. Then you start processing the sentences.

Further; we don't mind if a sentence is damaged, just toss it and wait for the next one that will be put into the buffer at some point. Youre envisaging complicated ways of waiting for certain amounts of time (not guaranteeable) to make sure that something will have happened(not guaranteed)

You need to take a much simpler view on this. Think of it like a toilet with a big cistern. You told your baby brother "when the water level gets above this mark, flush the toilet until it is below, because it won't flush unless the water is above here", you didnt tell him "flush once it every 5 minutes exactly" because if the tap is slow, it might not flush, or if the tap is fast 5 minutes might not be quick enough and it will overflow.
The trigger condition of "if above the line flush it" ("if there is at least one sentence in the buffer, process it") is simple, and it works no matter what the fill rate; while there are sentences available, you consume them, all of them.