Handling System.OutOfMemoryException

[c.vaibhav]

Hi,

I have a datagridview that has 65000 rows and 200 columns. I need to extract unique rows from the datagridview based on some sort of relevancy percentage.

While processing these rows it gives me an error "System.OutOfMemoryException" at around 47000 row.

Can I by any chance handle this exception?

I searched for this exception over internet and found that it is related to RAM. I have 4GB RAM and my OS is Vista. However it would be used by people having as low as 512 MB RAM. So I need to find out a method that can handle this exception well..

Please help..

Regards,
Vaibhav


P.S. - I have posted the same question on vbforums.

The link is [ame="http://www.vbforums.com/showthread.php?p=3565016#post3565016"]http://www.vbforums.com/showthread.php?p=3565016#post3565016[/ame]

 

[Robert_Zenz]

Hello.

I figured that this exception has nothing to do with RAM- or avalaible SWAP-Memory. Rather it seems that the program is running out of allocated memory, without the ability to reallocate new one. I had the same problem when drawing my own controls (and I sure WAS NOT out of memory).

Though, I can't remember what I did if I did something to fix this.

Maybe you can post the code where this exception is thrown...maybe there's some space to tune it so that you can avoid the exception. Also, if the rows are coming from a database, you should think about doing the processing there.

Bobby

 

[c.vaibhav]

Okay here is the code..

        array1.Add(relevantrow)

        For i = 1 To lastrow

            For j = array1.Count To 1 Step -1
                counter = 0
                flag = False
                For k = 1 To lastcolumn
                    abc = array1.Item(j)
                    If Me.DataGridView1.Item(k - 1, i - 1).Value = Me.DataGridView1.Item(k - 1, abc - 1).Value Then
                        counter = counter + 1
                        If ((counter * 100) / lastcolumn) >= Me.TextBox2.Text Then
                            flag = True
                            GoTo asdasd
                        End If
                    End If
                Next
            Next

asdasd:

            If flag = False Then
                array1.Add(i)
            End If
        Next

Here.. relevant row is the most relevant row.
lastrow = datagridview.rows.count
lastcolumn = datagridview.columns.count
me.textbox2.text = relevancy percentage

I have the raw data in MS Excel.. I cannot think of any query that can be executed to find out the desired results.. Also I had this running in MS Excel, but it takes awfully long time.. that is the reason why I need to shift to VB.NET..

Thanks for your help!!

Vaibhav

 

[Robert_Zenz]

Some notes:
* Stop using GoTo in .NET
* Arrays start by 0, not 1 (at least in .NET)

Consider using nested Foe Ech Loops:

        For Each row As DataGridViewRow In Me.DataGridView1.Rows
            For Each column As DataGridViewColumn In Me.DataGridView1.Columns
                If row.Cells(column.Index).Value Then 'blabla
            Next
        Next[CODE]

How exactly is this percentage thing working?

 

[c.vaibhav]

I have attached a simple zipped excel file with less amount of data..

In sheet1 raw data is present and in Uniques Sheet I have executed the macro and I have extracted relevant rows that are not more than 70% similar to each other..

 

[c.vaibhav]

I have made the modifications you have told..

Thanks very much Robert..

I have mistakenly named a collection as array1. Array1 is actually a collection and not an array..
Also, instead of 'goto' what else do I use..

Can you suggest something to handle System.OutofMemory Exception or any other alternative?

Regards,
Vaibhav

 

[JohnH]

You don't load that amount of data into a control. Do your processing with the DataTable, not the DataGridView. You also don't do processing like this in the UI thread, use a secondary worker thread, for example the BackgroundWorker. If you have no need for UI at all just use a Console application.

Having looked at your loops and the sample data I still have no idea what your algorithm is, what makes source list end up as result list? You mention something about dissimilar rows in other thread, but it was very diffuse. When you can explain in plain words what is supposed to happen it makes it easier to write the code to do it, or get help from people that understand what you are trying to do.