Question pause a thread, let other threads run

Ron Miel

Member
Joined
Apr 9, 2011
Messages
16
Programming Experience
10+
Hi there. In my application I have a loop that makes calls to a website to download data. I'm using a DownloadDataAsynch call with a handler.

I'm making many such calls in a loop. What I'd like to do is, if I have more than a certain number of replies pending, I should pause the loop and let the data handlers run.

How do I do that?

VB.NET:
     Dim DataReader As SqlDataReader = sp_call.ExecuteReader
     Dim count As Integer = 0
     [COLOR=DarkGreen]
     '---------------------------
     ' loop through results  
     ' get description for 
     ' each item on the list
     '---------------------------[/COLOR]
     While DataReader.Read()
            [COLOR=DarkGreen]' get the item ID[/COLOR]
            Dim itemID As String = DataReader(0).ToString
            Dim request_string As String

            [COLOR=DarkGreen]'get the data for that item[/COLOR]
            request_string =   [  web request constructed using item no ] 

            [COLOR=DarkGreen]' create an asynchronous call[/COLOR]
            Dim data_downloader As New WebClient
            AddHandler data_downloader.DownloadDataCompleted, AddressOf Data_Downloader_complete
            data_downloader.DownloadDataAsync(New Uri(request_string), count)
            items_pending = items_pending + 1
               
            If items_pending > 50 Then
                 [ pause until items_pending < 50 ]
            End If

      End While
 
You don't really need to. I'm fairly certain that DownloadDataAsync uses the ThreadPool, in which case it will only execute a certain number of operations at a time anyway. Others get queued until a thread becomes available.
 
Perhaps I didn't explain the problem clearly enough.

I don't WANT to have too many threads queued up. I might have a few thousand records to process. I have to loop through all the records, make a webrequest for each record.

I want to make about 50 webrequests, then pause the loop, and wait until I've received and processed the responses from most of them. Only when there are fewer than 10 responses queued will I submit the next 50.
 
Perhaps you have a reason but, if you start 50 tasks, why wait until 40 of them are finished before starting more? If it's OK to have 50 tasks running to begin with, why isn't it OK to start another task when each one of those 50 finishes? If 50 is OK to start with then why is it not OK to have 50 ongoing? I ask because what you want is possible but certainly more complex and there's no point adding complexity for no advantage.
 
I don't object to 50 tasks queued. I object to having several thousand queued, which is what happens at the moment. I want to ensure that no more than 50 are waiting.

Several reasons for this. One is that in case of my program or my computer crashing, it leave the records in an inconsistent state. Doing it 50 at a time will minimise the potential damage. Another reason is that it tends to slow down my whole computer. Maybe minimising the number of queued operations will keep it running smoothly. I don't know for certain that will work, but I want to try.

And 50 is just an example, I might use a larger number, or a smaller one. I'd have to experiment and see how it works.
 
Your attempts are misguided. The number of queued tasks is pretty much irrelevant. It's not going to affect anything. As for records being left in an inconsistent state, all that matters is whether a task is running or not. If it's queued but not started then it's not in an inconsistent state. If you want to minimise the risk of tasks being left in an inconsistent state then just process one at a time. That said, the fewer you run at a time, the greater the risk that some will be aborted because the whole thing will take longer. If you process more at a time then the overall risk that some will be aborted is less but, if something does go wrong, more tasks will have to be aborted. It's six of one, half a dozen of the other.
 
Your attempts are misguided. The number of queued tasks is pretty much irrelevant. It's not going to affect anything.

Maybe not, but I'd like to try it anyway and see if it makes a difference.

How can it not make a difference? If I have several thousand responses in the queue waiting to be processed, I presume they are stored in memory, or paged onto the hard disk. Wouldn't reducing the number make the memory management more efficient?

Or, if it's held in a buffer at my ISP, is that going to be unlimited size? Any risk that I might lose responses if I make too many requests at once?

As for records being left in an inconsistent state, all that matters is whether a task is running or not. If it's queued but not started then it's not in an inconsistent state.

It is in my program.

The problem is that the webresponses includes, among other things, a list of picture URLs. Processing each webresponse in turn I make further webrequests for each picture. The responses to those requests go on the end of the queue. So, that means that if my program fails, or is aborted, the main text of the item has been downloaded and marked as processed, but not the pictures. Hence the inconsistency.
 
Back
Top