littlebigman
Well-known member
- Joined
- Jan 5, 2010
- Messages
- 75
- Programming Experience
- Beginner
Hello
I need to download a bunch of pages from a web server, ie. spidering. I know that servers are typically configured to only allow a couple of concurrent connections from a given IP, but that would already halve the total time to run the script instead of downloading one page at a time.
At first sight, I guess there are two ways to do this:
- multi-threading
- non-blocking, async HTTP connections
Before I go ahead and investigate, has someone already done this and could share some code?
Thank you.
I need to download a bunch of pages from a web server, ie. spidering. I know that servers are typically configured to only allow a couple of concurrent connections from a given IP, but that would already halve the total time to run the script instead of downloading one page at a time.
At first sight, I guess there are two ways to do this:
- multi-threading
- non-blocking, async HTTP connections
Before I go ahead and investigate, has someone already done this and could share some code?
Thank you.