Have a bit of a dilemma and I'm wondering what type of opinoins are out there ...
I have a piece of equipment that I communicate with via a serial port. Data comes in at a pretty high rate of speed and the data is in "packets" of varying sizes.
I currently use the mscomm control as VB.Net 2002/2003 does not have built in comm functionality yet.
The data comes into the mscomm control and I have a thread that is waiting for for data to be in the input buffer. Once the thread sees data there, it grabs it and moves it out of the mscomm buffer area.
Sample incoming data looks like ...
------------
RX : 08 05 04 03 02 01 02 03 04 18
Where ....
- 1st byte on left is the "header" which states how many bytes are in the packet. (08)
- 1st byte on the right is the checksum which is a summation of the bytes that are neither header or checksum... (18)
I have no way to know WHEN a packet is starting (as no lines are set like DTR or anything). All I can do is ASSume when a packet would beging and then just do a checksum calc on it and if I don't get a good calc, drop the data; however, I haven't implemented that much effort yet. Currently every byte goes into a byte array.
In order to match up a response from the hardware, I have a function that you give a expected response and a count of characters to return such as :
Search : FF 05 04 03 , 10
Where :
FF = Match ANY byte
05 = 2nd byte of return data is 5
04 = 3rd byte of return data is 4
03 = 4th byte of return data is 3
10 = Return 10 characters of data starting at the location matched ...
Here's the inefficient part .....
In order for this to work, I have to do a pretty long and tedious match and iterate through the byte array (which could be a couple thousand characters depending on what is happening)....
Once it finds a matching "sub string" it returns it back to the caller.
Also, when a match is found, we need to remove it from the buffer of characters since we don't want to find one response twice ..... So after the search is completed, I also then have to delete those characters which is basically an array copy minus the range of bytes we just matched out....
The problems I am having are :
- Speed Inefficiency: Goes up a lot with amount of bytes returned from hardware
- Memory Useage : since its just a never ending byte array, it can get hefty if we have a lot of data coming in.
What I was thinking about doing is instead of the bytArray switch this over to a circular stack type implementation. i.e. define an array of bytarrays (say 50). Each of the 50 would be a pointer to a bytArray. That bytArray would contain on "packet" of data. As data comes in , you just increment the last record pointer.
The issue that comes to mind is that I may need to get data out in a different order than it came in ... for instance lets say we have 3 packets in the circular stack. IF i need the middle packet, it would screw up the begin and end pointers as the data would no longer be together.
I guess you could do a linked list to solve this problem; however, then everytime you add a node you need to update the links and the same when removing. Also, how can you readily do something like this in vb.net as I don't think you can just do a malloc ?
I also still have the dilemma of how do i determine when data come in is a valid packet ? The packet structure i have is pretty weak and without actually trying to construct everything into a packet and then validate the checksum, it will be hard to make each one into a packet ....
Has anyone here had to do an app with a lot of serial data acquisition ? I'd really love to talk about what would be the "best case" way to handle this.
Thanks !
I have a piece of equipment that I communicate with via a serial port. Data comes in at a pretty high rate of speed and the data is in "packets" of varying sizes.
I currently use the mscomm control as VB.Net 2002/2003 does not have built in comm functionality yet.
The data comes into the mscomm control and I have a thread that is waiting for for data to be in the input buffer. Once the thread sees data there, it grabs it and moves it out of the mscomm buffer area.
Sample incoming data looks like ...
------------
RX : 08 05 04 03 02 01 02 03 04 18
Where ....
- 1st byte on left is the "header" which states how many bytes are in the packet. (08)
- 1st byte on the right is the checksum which is a summation of the bytes that are neither header or checksum... (18)
I have no way to know WHEN a packet is starting (as no lines are set like DTR or anything). All I can do is ASSume when a packet would beging and then just do a checksum calc on it and if I don't get a good calc, drop the data; however, I haven't implemented that much effort yet. Currently every byte goes into a byte array.
In order to match up a response from the hardware, I have a function that you give a expected response and a count of characters to return such as :
Search : FF 05 04 03 , 10
Where :
FF = Match ANY byte
05 = 2nd byte of return data is 5
04 = 3rd byte of return data is 4
03 = 4th byte of return data is 3
10 = Return 10 characters of data starting at the location matched ...
Here's the inefficient part .....
In order for this to work, I have to do a pretty long and tedious match and iterate through the byte array (which could be a couple thousand characters depending on what is happening)....
Once it finds a matching "sub string" it returns it back to the caller.
Also, when a match is found, we need to remove it from the buffer of characters since we don't want to find one response twice ..... So after the search is completed, I also then have to delete those characters which is basically an array copy minus the range of bytes we just matched out....
The problems I am having are :
- Speed Inefficiency: Goes up a lot with amount of bytes returned from hardware
- Memory Useage : since its just a never ending byte array, it can get hefty if we have a lot of data coming in.
What I was thinking about doing is instead of the bytArray switch this over to a circular stack type implementation. i.e. define an array of bytarrays (say 50). Each of the 50 would be a pointer to a bytArray. That bytArray would contain on "packet" of data. As data comes in , you just increment the last record pointer.
The issue that comes to mind is that I may need to get data out in a different order than it came in ... for instance lets say we have 3 packets in the circular stack. IF i need the middle packet, it would screw up the begin and end pointers as the data would no longer be together.
I guess you could do a linked list to solve this problem; however, then everytime you add a node you need to update the links and the same when removing. Also, how can you readily do something like this in vb.net as I don't think you can just do a malloc ?
I also still have the dilemma of how do i determine when data come in is a valid packet ? The packet structure i have is pretty weak and without actually trying to construct everything into a packet and then validate the checksum, it will be hard to make each one into a packet ....
Has anyone here had to do an app with a lot of serial data acquisition ? I'd really love to talk about what would be the "best case" way to handle this.
Thanks !
