Multi-part downloading

[mentalhard]

I was wondering how do i download a file in a separated threads (multi-part downloading) and then merge when all parts are downloaded?

I guess it's related with AddRange method but i am not sure about the details:

Dim theRequest As HttpWebRequest 

theRequest.AddRange(range)

Thanks for any further help/advice/suggestion

P.S. All i want to achieve is to accelerate the download speed by separating the file(s) in a few threads ... at least i believe that's the correct approach.
How actually torrent clients work? I've noticed that they create some temp files. Usually two files (probably one is the info and the second is data) and when all parts are downloaded it merge the parts and create the final file.

 

[JohnH]

Here is some info http://www.vbdotnetforums.com/showthread.php?t=15827
Attached is a ~finished~ implementation with a sample project using my Downloader class you can review.
While testing I've seen both much improvement and also worse performance using multiple asynchronous download parts rather than a single request. For HTTP 1.1 there is a recommandation not to employ more than 2 concurrent (persistent?) sockets to the same server from a client, the server may enforce this and put your other parts on hold. But two could be better than one. The server also doesn't have to support partial requests, and it doesn't have to tell you, so to do this you have to make two additional requests that adds a little overhead to find this out.

 

[mentalhard]

Thanks for the code sample John and sorry because i didn't answer earlier.
Actually you are right the HTTP protocol supports only 2 connections (althrough HTTP 1.0 supports 4).

I think i found exactly what i was looking for. Using Collection i am actually making as much threads as the user defines.

Then to avoid the TimeOut for the third connection (two is limit so the third always return timedout) i have something like:

If threadCount > 2 Then
'set up the connection limit to avoid HTTP Connection Timmed Out after the 2nd connection
   Net.ServicePointManager.DefaultConnectionLimit = threadCount + 1
Else
   If threadCount = 0 Then threadCount = 3 'Set Default if discarded =3 
End If

Hope it makes sense.

However, thanks ones again for the code

 

[JohnH]

The class was already multithreaded and managed DefaultConnectionLimit automatically I've had no problems running tests with 100 async parts also, but that is not very good conduct... hehe, it is not a limit for the protocol, it is a limit for good behaviour.

 

[mentalhard]

Actually i just checked your code and it seems like you are doing pretty same in your code:

'there should be no more than 2 concurrent http connections to a server from one application (rfc 2616)
Net.ServicePointManager.DefaultConnectionLimit = parts

Thanks

 

[JohnH]

I've also found that using the available async methods (BeginSomething/EndSomething) give much better performance than creating your own Threads.