Question WHy do I run out of memory?

[pauldavey]

I have a need to upload about 1000MB to a webserver (it is a VMware ESX server). I have been told that I must open the file I want to upload, read it in to a byte array and then stream it to the target file. The targetfile already exists. The target server does not support Chunked Mode.

Below is the code that I am using:

  Dim fileToUpload As String = "C:\banana.iso"
            Dim fileLength As Long = My.Computer.FileSystem.GetFileInfo(fileToUpload).Length
            url = "https://" & vcenter_server & "/folder/" & appliance_name & "/" & appliance_name & "-flat.vmdk" & "?dcPath=" & selected_datacenter & "&dsName=" & selected_datastore
            Dim u As New Uri(url)

           Dim userName As String = vcenter_username
            Dim password As String = vcenter_password

            'Create the request
            Dim request As HttpWebRequest = DirectCast(System.Net.HttpWebRequest.Create(u), HttpWebRequest)
            request = DirectCast(System.Net.HttpWebRequest.Create(u), HttpWebRequest)
            request.Credentials = New NetworkCredential(vcenter_username, vcenter_password)

            request.KeepAlive = False
            request.Timeout = 6000000
            request.AllowWriteStreamBuffering = False    ' set to false so the outgoing data is not continually buffered when shipped
            request.Credentials = New NetworkCredential(userName, password)
            request.Method = WebRequestMethods.Http.Post
            request.SendChunked = False

            Dim s As IO.Stream = request.GetRequestStream()
            Dim fs As New IO.FileStream(fileToUpload, FileMode.Open, FileAccess.Read)
Dim s As IO.Stream = request.GetRequestStream()
            Dim fs As New IO.FileStream(fileToUpload, FileMode.Open, FileAccess.Read)

            'Create the buffer for storing the bytes read from the file
            Dim byteTransferRate As Integer = 10485760
            Dim bytes(byteTransferRate - 1) As Byte
            Dim bytesRead As Integer = 0
            Dim totalBytesRead As Long = 0
            Dim totalbyteswritten As Long = 0

            'Read from the file and write it to the server's stream.
            Do Until totalBytesRead = fileLength
                'Read from the file
                '  bytesRead = fs.Read("return byte array", "offset", "Max number of bytes to read")
                fs.Position = totalBytesRead
                bytesRead = fs.Read(bytes, 0, bytes.Length)
                If bytesRead > 0 Then
                    totalBytesRead += bytesRead
                    Try
                        s.Write(bytes, 0, bytesRead)
                       totalbyteswritten = totalbyteswritten + bytesRead
                        percentage = (totalbyteswritten * 100 / fileLength)
                        bgworker1.ReportProgress(percentage)
                    Catch webex As WebException
                    End Try
                End If
                bytesRead = 0
                fs.Flush()
                s.Flush()
                GC.Collect()
            Loop

            s.Close()
            fs.Close()

Running the code results in the app memory usage getting up to 700MB before it falls over. I though I was clearing out the memory, but obviously not. Can anyone help? I have been stuck on this for days now!

 

[newguy]

1) use code tags for your code - easier to read.
2) don't call GC.Collect - the framework will dispose it's object and you dispose the ones you create - like filestreams, pens, brushes, etc...

 

[pauldavey]

In response to the above:

1. Will do
2. Its the same whether it is included or not; I added in to try to force a collection.

It is definately the line that writes the data to the stream that causes the issue.

Any idea's?????

 

[alander]

which line in ur code was throwing System.OutOfMemoryException?

 

[pauldavey]

It is when the data is written. From the following snippet of code:

'Create the buffer for storing the bytes read from the file
            Dim byteTransferRate As Integer = 10485760
            Dim bytes(byteTransferRate - 1) As Byte
            Dim bytesRead As Integer = 0
            Dim totalBytesRead As Long = 0
            Dim totalbyteswritten As Long = 0

            'Read from the file and write it to the server's stream.
            Do Until totalBytesRead = fileLength
                'Read from the file
                '  bytesRead = fs.Read("return byte array", "offset", "Max number of bytes to read")
                fs.Position = totalBytesRead
                bytesRead = fs.Read(bytes, 0, bytes.Length)
                If bytesRead > 0 Then
                    totalBytesRead += bytesRead
                    Try
                        s.Write(bytes, 0, bytesRead)
                       totalbyteswritten = totalbyteswritten + bytesRead
                        percentage = (totalbyteswritten * 100 / fileLength)
                        bgworker1.ReportProgress(percentage)
                    Catch webex As WebException
                    End Try
                End If
                bytesRead = 0
                fs.Flush()
                s.Flush()
                GC.Collect()
            Loop

This individual line causes the issue

s.Write(bytes, 0, bytesRead)

I believe that the data is buffered in memory as the webrequest doesnt send the data until all of it has been written? Basically, my belief is that I need to write the data to the stream without buffering it in RAM. But, I don't know how to do this

 

[pauldavey]

Further to the above, there appears to be some people that have worked around the issue when using Java - confirming my suspicions that the answer is to not buffer the information. Can anyone tell me how to do this?

HttpURLConnection (Java 2 Platform SE 5.0)

setFixedLengthStreamingMode
public void setFixedLengthStreamingMode(int contentLength)This method is used to enable streaming of a HTTP request body without internal buffering, when the content length is known in advance.
An exception will be thrown if the application attempts to write more data than the indicated content-length, or if the application closes the OutputStream before writing the indicated amount.

When output streaming is enabled, authentication and redirection cannot be handled automatically. A HttpRetryException will be thrown when reading the response if authentication or redirection are required. This exception can be queried for the details of the error.

This method must be called before the URLConnection is connected.


Parameters:
contentLength - The number of bytes which will be written to the OutputStream.
Throws:
IllegalStateException - if URLConnection is already connected or if a different streaming mode is already enabled.
IllegalArgumentException - if a content length less than zero is specified.
See Also:
setChunkedStreamingMode(int)