Question Multiple Downloads?

[Senoska]

Hello,

This seems like a simple question, but I've been struggling with it for awhile now.

I'm writing something to download about 200 small (Around 200kb) files at a time. I pass the file urls off to a class which starts the downloads. The problem is that it'll start the download (Just creates a 0kb file) and then slowly downloads each file one by one, waiting for the next to complete despite this being a threaded class. This would be fine but after about a minute, windows removes all of the 0kb files and I'm stuck with only a few of the files being downloaded. Any ideas how to get multiple files to be downloaded concurrently?

Imports System.Net
Imports System.IO

Public Class DownloadClass
    Public downloadUrl As String
    Public folderURL As String
    Public Name As String

    Private InstancedThread As System.Threading.Thread

    Public Sub DownloadFile(ByVal strsource As String, ByVal folder As String, ByVal name As String)

        'Try
        Dim webClient As System.Net.WebClient = New System.Net.WebClient()
        webClient.DownloadFile(strsource, folder & "\" & name)
        'Catch ex As Exception
        '    'Nothing here.
        'End Try
    End Sub

    Public Sub Kickit()
        InstancedThread = New System.Threading.Thread(AddressOf startRunning)
        InstancedThread.Start()

    End Sub

    Private Sub startRunning()
        DownloadFile(downloadUrl, folderURL, Name)
    End Sub
End Class

 

[jmcilhinney]

You shouldn't be creating 200 threads. For something like this you want the system to manage the number of threads for optimum performance. That's what the ThreadPool class is for. You call ThreadPool.QueueUserWorkItem and specify the method to run. The system will decide how many threads to run simultaneously and the others will be queued.

Try making that change and see what happens. That's the way it should be done regardless, but if that doesn't solve your problem then we can look further afield.

 

[Senoska]

While I appreciate the input, the 0 byte and single-file download problem is still here. The problem wasn't the amount of threads. The amount varies from 1 to 200, with 200 being abnormally rare.

However, with that being said, even with the change:

ThreadPool.QueueUserWorkItem(AddressOf startRunning)

windows is still clearing the 0 byte files before they have a chance to download and one file is being downloaded at a time.

 

[jmcilhinney]

OK, I did say we could keep looking if that didn't solve the problem. What happens if you just try to download a single file. Does that work?

 

[Senoska]

Like I said, it starts the download of all the files. All files show up as 0kb. Each file downloads one after the next, although rather slowly. After so long windows (by default as far as I know) cleans 0kb files. It's not downloading the files concurrently, but rather, after the last finishes.

That being said, out of 200 files, maybe 20 will make it fully downloaded, the rest get cleaned.

 

[ProtekNickz]

Hi, just an idea?, whilst you D/L 200 files and you have said only 20 do actually D/L out of the total amount, isn't their a possibility that you need to do some for of memory management? or Caheing the data to the harddrive as trying to store this many files to The memory as such would be near impossible, unless their 20kb's each, i find that a program i used simler to this freezes up till it's finished the d/l so it was mentioned to me the same.

 

[JohnH]

If using Http protocol the server should limit you to 2 concurrent connections, other requests will be queued. Another thing is you have commented out Try-Catch, if the download fails you will get a WebException with information about what the problem is. When it comes to sockets I have good experience in using the dedicated asynchronous methods, even over good old common sense thread pooling, so in this case I would go for DownloadFileAsync method for async file download instead of DownloadFile. You whole class arrangement is fishy, as it limits usability and add nothing to the existing functionality of the WebClient class.