storing different things in one byte array

iosys

Member
Joined
Apr 6, 2006
Messages
7
Programming Experience
5-10
I'm writing some code to transfer files using system.net.sockets methods. It works fine for sending one file or string in a byte array because I know exactly what to expect on the receiving end.

What I want to to do is include multiple elements in the byte array, like a string containing information about the computer sending the file, and then the file itself. I could just convert the string to bytes and combine the string+file into one byte array and send it, but I won't know how to parse it on the receiving end.

The only way I've come up with so far is to insert bytes between the string/file so I know where to split the array on the receiving end. How do I then split it? Is there another way to do this better?
 
Last edited:
Serialization can help you with this, consider the example, also know that MemoryStream has got the Length property and ToArray method that returns the byte array, convenient to send with Socket class or NetworkStream.
VB.NET:
<Serializable()> Class clsTest
  Public name As String
  Public bytes() As Byte
End Class
 
Sub serializationExample(ByVal filename As String)
[COLOR=darkgreen]'new class instance & data[/COLOR]
  Dim x As New clsTest
  x.name = filename
[COLOR=darkgreen]'(x.bytes=byte array of file)[/COLOR]
  Dim bf As New System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
  Dim mem As New IO.MemoryStream
[COLOR=darkgreen]'serialize[/COLOR]
  bf.Serialize(mem, x)
[COLOR=darkgreen]'-here you could send the byte array from mem.ToArray()[/COLOR]
 
 
[COLOR=darkgreen]'-receiver gets the bytes/reads into memorystream.[/COLOR]
[COLOR=#006400]'then seek memorystream to start[/COLOR]
  mem.Seek(0, IO.SeekOrigin.Begin)
[COLOR=darkgreen]'deserialize from stream[/COLOR]
  x = DirectCast(bf.Deserialize(mem), clsTest)
  mem.Close()
[COLOR=darkgreen]'check 'transfer'[/COLOR]
  MsgBox(x.name)
End Sub
File to byte array to file operations not included because you already done this.
 
iosys said:
The only way I've come up with so far is to insert bytes between the string/file so I know where to split the array on the receiving end. How do I then split it? Is there another way to do this better?

on a base level.. that's the only way there is. adherence to a standard way of demarking data to be sent is said to be "obeying a protocol" :)


think about http... a request starts with GET, then the url to be got and the method in use.. some parameters are passed as newline-delimited-strings, then two newlines demarks the end of the request..
 
Well the serialize method appears to work, but only on smaller files.

When I try sending a file of about 100kb in size, I get an error "End of Stream encountered before parsing was completed" during the DeserializeData() on the line containing:

o = DirectCast(bf.Deserialize(mem), SendObject)

Could this be because the data is not arriving all at once in PollListen()? I tested the whole serialize/deserialize procedure directly in the form_load event instead of sockets, which does work with any size file.

[edit] Using a file with a size of 119,877 bytes, I checked the size of the byte array during the timeline of operations and came up with:

During the SerializeData() method: 240126
During the Send() method (code I did not post): 240126
m_ListenSocket.Available: 8192
After the m_ListenSocket.Receive() method: 8192
During the DeserializeData() method (before the error): 8192


In another test with a smaller sized file I came up with:

During the SerializeData() method: 13560
During the Send() method (code I did not post): 13560
m_ListenSocket.Available: 8192
After the m_ListenSocket.Receive() method: 8192
During the DeserializeData() method (before the error): 8192

A SendObject containing this file deserializes ok and I am able to get accurate data from the object.

VB.NET:
    ' PollListen
    ' Listens for a client and receives data.
    Sub PollListen()
        Try
            If m_Listener.Pending = True Then
                ' create the socket
                m_ListenSocket = m_Listener.AcceptSocket
                ' check if the socket contains data
                If m_ListenSocket.Poll(1, SelectMode.SelectRead) = True Then
                    ' create a byte array for the socket data
                    Dim bData(m_ListenSocket.Available - 1) As Byte
                    ' fill the byte array with the socket data
                    m_ListenSocket.Receive(bData)
                    ' recreate the object
                    Dim o As SendObject = DeserializeData(bData)
                End If
            End If
        Catch ex As Exception
            Console.WriteLine(ex.Message)
            RaiseEvent ErrorOccurred(ex)
        End Try
    End Sub
 
    ' SerializeData
    ' Serializes the passed object into a byte array.
    Function SerializeData(ByVal obj As SendObject) As Byte()
        Dim bf As New System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
        Dim mem As New IO.MemoryStream
        ' serialize this object
        bf.Serialize(mem, obj)
        ' return the serialized memory
        Dim b() As Byte = mem.GetBuffer
        mem.Close()
        ' return the serialized bytes
        Return b
    End Function
 
    ' DeserializeData
    ' Deserializes the passed Bytes() and casts it to a SendObject.
    Function DeserializeData(ByVal Bytes() As Byte) As SendObject
        Dim bf As New System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
        Dim mem As New IO.MemoryStream()
        ' write the passed Bytes() to the memory stream
        mem.Write(Bytes, 0, Bytes.Length - 1)
        ' seek to beginning of memory stream
        mem.Seek(0, IO.SeekOrigin.Begin)
        ' deserialize the memory stream and cast it to a SendObject
        Dim o As SendObject
        o = DirectCast(bf.Deserialize(mem), SendObject)
        mem.Close()
        ' return deserialized object
        Return o
    End Function
 
<Serializable()> _
Class SendObject
    Public Sender As String
    Public FileName As String
    Public Data() As Byte
    Sub SetFile(ByVal Path As String)
        Dim oFile As New IO.FileInfo(Path)
        FileName = oFile.Name
        Data = IO.File.ReadAllBytes(Path)
    End Sub
End Class
 
Last edited:
You're confusing the transfer buffer with the full transfer, the 8192 byte numbers you post is your system DefaultReceiveWindow setting, which you will also find in Socket.ReceiveBufferSize. When transferring synchronous as you do with Socket.Receive you have to loop to get all bytes, else with asynchronous you append the received buffers until transfer is complete.

When you eventually get all data you will run into another problem if your serializable class is just copied into both client/server projects, this is because deserialization have to be done with assembly qualified type, meaning "client.SendObject" is not the same as "server.SendObject". The solution is easy - put the class in a class library and reference it from both project, now both serialize/deserilize work on the same assembly qualified type "classlibrary.SendObject".

Your buffer is very low, are you on a 56k modem dialup connection? There is a connection optimizer I like called CableNut, also good for improving modem settings. http://www.cablenut.com/
 
I have created a simple client/server solution that could be interesting for you and many others. Attached zip contains two solutions, socketserver and socketclient, very barebone examples of client/server Tcp communication with no other features than transfer of multiple files/data and hardly any errorchecking. It works - and is intended as educational at beginner level.
1. Build and start socketserver solution, it includes ClassLib used for serialization/deserilization which also is referenced in socketclient.
2. Build and start socketclient solution. If there is complaint about missing ClassLib, then update the reference, the dll is found in /socketserver/ClassLib/Bin/Debug folder.
3. Click 'connect' button in socketclient window, click 'send file' button to browse for a file. File is sent to socketserver when you click 'ok' button in file dialog. Send more files if you wish. There is no waiting for send to complete at client or server because asynchronous threading methods are used for transfer.
4. Review transferred files, they are listed in listbox in socketserver window, the physical files are found when debugging in the /socketserver/socketserver/Bin/Debug folder.

Key coding techniques used are:
- Data transfer with System.Net.Sockets and TcpListener/TcpClient classes.
- Asynchronous send/receive with NetworkStream methods BeginWrite/EndWrite and BeginRead/EndRead.
- How to transfer complex data in one packet using (serialized) class instance.
- Serialization/deserialization of binary data. (The key to serialize/deserialize between different assemblies is to link both projects to the same class library containing the common class.)

edit: added also a more advanced client/server set in zip file, one that will not read full file into memory, and also allow to select multiple and large files at once and send them as a continuous stream.
 

Attachments

  • sockets.zip
    76.7 KB · Views: 112
Last edited:
On the subject of 'CableNut' i downloaded it after you mentioned it in a previous post JohnH. Great program, and also free!! It's good to know that there are people out there trying to improve peoples experiences with the internet etc... especially as there are so many out to destroy it.

P.s (JohnH, hope i'm not speaking out of turn but i don't suppose you had anything to do with the development of cablenut, did you?)
 
vis781 said:
P.s (JohnH, hope i'm not speaking out of turn but i don't suppose you had anything to do with the development of cablenut, did you?)
No, I just found it on the web many years ago and have always found a good reason to use with continuing new operating system versions and new network equipment.. Windows doesn't seem to optimize connections and default settings have to be 'one size fits all'. ;)
 
Good information...I'll check out the code later. My solution is to combine the client/server into one class, and allow switching on/off listen mode. Maybe it's not the best, but it's what I eventually came up with so far.
 
Back
Top