One app we have written at work has to do a lot of data comparison stuff. We are looking at doing this on the SQL server but technical issues (with computed columns 
) are currently causing us greif.
In one instance, the following code gets iterated about half a billion times. Removing records from the dtAllowed set reduces this from about a billion iterations! One dataset starts with 16000 rows, the other 72000. Essentially one set is a list of 'allowed' records and the other set is filtered for matches.
In this case, the process takes about 10 minutes to run. Can anyone see a way of making this process more efficient? Even slight changes will make a difference with this many iterations.
Cheers guys
mafro
) are currently causing us greif.In one instance, the following code gets iterated about half a billion times. Removing records from the dtAllowed set reduces this from about a billion iterations! One dataset starts with 16000 rows, the other 72000. Essentially one set is a list of 'allowed' records and the other set is filtered for matches.
In this case, the process takes about 10 minutes to run. Can anyone see a way of making this process more efficient? Even slight changes will make a difference with this many iterations.
VB.NET:
For i = 0 To dtSource.Rows.Count - 1
For j = 0 To dtAllowed.Rows.Count - 1
'search for matches in the two sets
If dtSource.Rows(i)("path") = dtAllowed.Rows(j)("path") Then
'import a matching record
dtOverlap.ImportRow(dtSource.Rows(i))
'remove from the set for efficiency
dtAllowed.Rows.RemoveAt(j)
Exit For
End If
Next
'check for cancel flag
If bw.CancellationPending Then
Return Nothing
End If
Nextmafro
