Free memory used by dataview

[ctiwari]

Free up memory used by dataview

Hi All

I am using VB.net and I have several csv files that I need to sort and manipulate in simple ways (add & sum columns, etc.). I am currently using a dataview to load these files into the memory and then perform some simple operations on the data before writing it out as a file.

I have multiple files (and these are quite big - 100mb or so each) and I'd like to clear up the contents of the dataview (or free up memory??) after I process each file. I couldn't seem to find any info on how to do this. I'm not really a programmer and any help in this regard will be greatly appreciated.

Thanks,

Chetan

FYI my code looks like this:

' create a data view to help sort and compute weights
Dim dvdistance AsNew DataView(inputDS.Tables("table1"))
dvdistance.Sort = "gridid, distance"
// do my manipulations here
// write out to file

---> I'd like to clear up the memory used by the dataview dvdistance at
---> this point!

Dim dvdistance AsNew DataView(inputDS.Tables("table2"))
dvdistance.Sort = "gridid, distance"
// manipulations for the second dataset here
// write out to file

 

[mjb3030]

As soon as you dereference the first dataview by setting dvdistance equal to a new instance of DataView, the first one will be marked for garbage collection. However, (someone correct me if I'm wrong here) I don't believe the DataView is actually holding any of the data. Your DataSet is holding all the data. The DataView is merely giving you a filtered view of the data. So, the only way you are going to free any memory is by killing the DataSet.

 

[mjb3030]

If you don't need to work with all tables at once, try loading them one at a time.

 

[ctiwari]

Thanks mjb

I tried doing what you suggested and it seems like the tables are being removed from the memory (I do a count to see how many were there originally and then if they are still in memory after i remove them). The dataview is being emptied when i use the dataview.dispose() method.

However, if I look at the memory that is being used by the application (through the task manager), it seems to increase as more tables are added, but it does not decrease after i remove the tables. I tried doing a gc.collect(), but that didn't work either. In fact, I even tried clearing the dataset that holds all these elements and that did not free up any memory either. Thanks for your suggestions though - they seem like the right thing to do, but the results are unexpected!

- Chetan

 

[mjb3030]

Are you removing the tables from the DataSet? That is where they are located.

inputDS.Tables.Remove("Table1")

 

[BadgerByte]

Hey Ctiwari,

Sorry to butt in here!!

I had exactly the same concerns, the problem is that using taskmanager to view memory usage is not a good indicator, especially for a .net app. It's not uncommon to see weird things happening. The framework allocates memory rather strangely, but rest assured that the garbage collection does work. From what I've gathered from various websites, the framework doesn't release the memory (as far as task manager is concerned) until another app needs it (or something, I'm no expert!!).

 

[ctiwari]

Thanks MJB and Badger,

Yes, I think you're right about the task manager not accurately reflecting memory usage. I noticed that as I load more tables into memory, the memory consumption by the program drops (rather suddenly infact!) as soon as it seems like my computer is going to break down!

Thanks again y'all,

Chetan