Question Multiple Downloads?

[Senoska]

Hello,

This seems like a simple question, but I've been struggling with it for awhile now.

I'm writing something to download about 200 small (Around 200kb) files at a time. I pass the file urls off to a class which starts the downloads. The problem is that it'll start the download (Just creates a 0kb file) and then slowly downloads each file one by one, waiting for the next to complete despite this being a threaded class. This would be fine but after about a minute, windows removes all of the 0kb files and I'm stuck with only a few of the files being downloaded. Any ideas how to get multiple files to be downloaded concurrently?

Imports System.Net
Imports System.IO

Public Class DownloadClass
    Public downloadUrl As String
    Public folderURL As String
    Public Name As String

    Private InstancedThread As System.Threading.Thread

    Public Sub DownloadFile(ByVal strsource As String, ByVal folder As String, ByVal name As String)

        'Try
        Dim webClient As System.Net.WebClient = New System.Net.WebClient()
        webClient.DownloadFile(strsource, folder & "\" & name)
        'Catch ex As Exception
        '    'Nothing here.
        'End Try
    End Sub

    Public Sub Kickit()
        InstancedThread = New System.Threading.Thread(AddressOf startRunning)
        InstancedThread.Start()

    End Sub

    Private Sub startRunning()
        DownloadFile(downloadUrl, folderURL, Name)
    End Sub
End Class

 

[jmcilhinney]

You shouldn't be creating 200 threads. For something like this you want the system to manage the number of threads for optimum performance. That's what the ThreadPool class is for. You call ThreadPool.QueueUserWorkItem and specify the method to run. The system will decide how many threads to run simultaneously and the others will be queued.

Try making that change and see what happens. That's the way it should be done regardless, but if that doesn't solve your problem then we can look further afield.

 

[Senoska]

While I appreciate the input, the 0 byte and single-file download problem is still here. The problem wasn't the amount of threads. The amount varies from 1 to 200, with 200 being abnormally rare.

However, with that being said, even with the change:

ThreadPool.QueueUserWorkItem(AddressOf startRunning)

windows is still clearing the 0 byte files before they have a chance to download and one file is being downloaded at a time.

 

[jmcilhinney]

OK, I did say we could keep looking if that didn't solve the problem. What happens if you just try to download a single file. Does that work?

 

[Senoska]

Like I said, it starts the download of all the files. All files show up as 0kb. Each file downloads one after the next, although rather slowly. After so long windows (by default as far as I know) cleans 0kb files. It's not downloading the files concurrently, but rather, after the last finishes.

That being said, out of 200 files, maybe 20 will make it fully downloaded, the rest get cleaned.

 

[ProtekNickz]

Hi, just an idea?, whilst you D/L 200 files and you have said only 20 do actually D/L out of the total amount, isn't their a possibility that you need to do some for of memory management? or Caheing the data to the harddrive as trying to store this many files to The memory as such would be near impossible, unless their 20kb's each, i find that a program i used simler to this freezes up till it's finished the d/l so it was mentioned to me the same.

 

[JohnH]

If using Http protocol the server should limit you to 2 concurrent connections, other requests will be queued. Another thing is you have commented out Try-Catch, if the download fails you will get a WebException with information about what the problem is. When it comes to sockets I have good experience in using the dedicated asynchronous methods, even over good old common sense thread pooling, so in this case I would go for DownloadFileAsync method for async file download instead of DownloadFile. You whole class arrangement is fishy, as it limits usability and add nothing to the existing functionality of the WebClient class.

 

[Rythorian]

I'm providing you my method and I've tested it. It's an app called BundleCrush that I wrote. You don't have to provide a Zip File name to extract, it targets the specialfolder recursively. I use it daily. A lot of people struggle with downloading and properly extracting a zip file fully. I ran into the same problems myself in the past and had to work around it.

'APPLICATION: BUNDLECRUSH (DOWNLOAD & ZIP EXTRACTION UTILITY)
'
Imports System.IO
Imports System.IO.Compression
Imports System.Net

Public Class Form1


    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load



    End Sub

    Public Sub Download_ProgressChanged(sender As Object, e As DownloadProgressChangedEventArgs)

        Try
            ProgressBar1.Value = CInt(Math.Round((e.BytesReceived / e.TotalBytesToReceive) * 100, 0, MidpointRounding.AwayFromZero))

            If ProgressBar1.Value = ProgressBar1.Maximum Then

                RecursiveMeasures()

            End If

        Catch ex As Exception

            Debug.Print(ex.ToString)

        End Try

    End Sub

    Public Sub RecursiveMeasures()

        'PATH TO SUB DIRECTORY, NO ZIP FILE NAME IS NEEDED
        Dim FolderStartPath As String =
        Environment.GetFolderPath(Environment.SpecialFolder.Desktop)

        Dim startPath As String = FolderStartPath

        'FOR EACH STATEMENTS SPEAK FOR THEMSELVES, IT WILL RECURSIVELY GO THROUGH EACH FILE IN A FOLDER
        Directory.GetFiles(startPath,
            "*.zip",
            SearchOption.AllDirectories).ToList().ForEach(Sub(zipFilePath)

                                                              'PATH TO DECOMPRESSED FILE
                                                              Dim FolderEndPath As String =
                                                                  Environment.GetFolderPath(Environment.SpecialFolder.Desktop)

                                                              'NAME OF FOLDER YOU'LL FIND YOUR EXTRACTION FILE IN
                                                              Dim extractPath As String = Path.Combine(FolderEndPath, "Extracted") '<<< FOLDER NAME


                                                              'https://learn.microsoft.com/en-us/dotnet/api/system.io.path.getfilenamewithoutextension?view=net-9.0
                                                              'Returns the file name of the specified path string without the extension.
                                                              '
                                                              Dim extractPathForCurrentZip As String = Path.Combine(extractPath, Path.GetFileNameWithoutExtension(zipFilePath))

                                                              If Not Directory.Exists(extractPathForCurrentZip) Then

                                                                  Directory.CreateDirectory(extractPathForCurrentZip)

                                                              End If

                                                              ZipFile.ExtractToDirectory(zipFilePath, extractPathForCurrentZip)

                                                          End Sub)



    End Sub

    Public Shared Sub ExtractFile(baseZipPath As String,
            extractPath As String)

        If Not Directory.Exists(extractPath) Then

            Directory.CreateDirectory(extractPath)

        End If

        ZipFile.ExtractToDirectory(baseZipPath, extractPath)

        Dim nestedZipDirectories As String() = Directory.GetFiles(extractPath, "*.zip")

        For Each nestedZipDirectory As String In nestedZipDirectories

            ZipFile.ExtractToDirectory(nestedZipDirectory, extractPath)

        Next

    End Sub

    Private ReadOnly files As New List(Of String)()
    Public Property Response As Object

    Private Sub RecursiveFiles(sDir As String)

        Try

            For Each d As String In Directory.GetDirectories(sDir)

                RecursiveFiles(d)

            Next

            For Each file As String In Directory.GetFiles(sDir)

                DoActionModifier(file)

            Next

        Catch e As Exception

            MessageBox.Show(e.Message)

        End Try

    End Sub

    Private Sub DoActionModifier(filepath As String)

        files.Add(filepath)

    End Sub

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        Dim saveFileDialog1 As New SaveFileDialog With {
            .Filter = "All files (*.*)|*.*",
            .FilterIndex = 1,
            .RestoreDirectory = True
        }

        If saveFileDialog1.ShowDialog() = DialogResult.OK Then

            Dim myStream As Stream = saveFileDialog1.Site

            Dim snl As String

            snl = TextBox1.Text

            'Dim url As String = "https://github.com/DeadReport77/Scarlett-Speech-Recognition-Final-Production/releases/download/2.0/Scarlett-Speech-Recognition-Final-Production.zip"

            ' Dim savePath As String = Environment.GetFolderPath(Environment.SpecialFolder.MyVideos) '"C:\Users\rytho\Videos\Scarlett-Speech-Recognition-Final-Production.zip"

            Dim download As New WebClient()

            AddHandler download.DownloadProgressChanged, AddressOf Download_ProgressChanged

            download.DownloadFileAsync(New Uri(snl), saveFileDialog1.FileName)
            ' Code to write the stream goes here.
            myStream?.Close()
        End If


    End Sub


End Class