Efficient handling of large .txt files? / StreamWriter V's ??

[22-degrees]

Hi all,

I have data being exported to .txt/.csv files every 2 seconds.

Output at max expected load is c200Mb per hour.

I initially started exporting to individual sector files but when I checked on the output after a few hours, 80+ files being written to, 3 lines at a time using Filesystem.WriteAllText was taking almost 10 seconds..

I did a little research and decided to try StreamWriter in a 'Using' block and lump all the data into one file.. Hey Presto, almost 100 times faster according to stopwatch data..

However I have noticed a number of before and after differences.. Before, when 80+ files were being written to, the CPU and RAM gauge were low, but my PC was sluggish.. Now, with StreamWriter to one file, my CPU and RAM gauge are a lot higher, but PC seems to be responding a lot quicker than before to general operation.

My question is, what is the best or most efficient way to manage larger txt files such as 100Mb+ in vb.net? I am not 100% clear on how exactly 'Stream' manages the file and I am not a regular user of the "Using" block either. This is my first time exporting such a large amount of data so CPU/RAM efficiency has never been a concern of mine..

Hoping someone can point me in the right direction.

 

[Herman]

For this volume of data, using text files is retarded if you pardon my french. I understand you may be stuck working with an already developped application, but if not please for the love of god use a proper database.

Otherwise just use threads. StreamWriter is already doing it to some extend, which is why it can better use your CPU and RAM and leave your UI more responsive.

 

[22-degrees]

Thanks for the reply.. It's not only working with a developed app that's the problem, it's my level of experience, or inexperience i should say.. When I should have been utilizing databases in the past, I opted to go with Excel for historical data management instead as it suited non-vb projects at the time.. As a result I have never given databases too much time..

Based on your reply though, it would seem database would be a no-brainer.. As it happens, the implementation of my first database is scheduled at a later stage in this project. The current stage is testing concepts using temporary storage while formula's are perfected. Once everything works as it should, temporary storage will be replaced with permanent storage (database) which I have only just begun to learn (todo list tutorial).