Resources and Datasets Question about Handling Large Tables

[Hopworks]

I have been using datasets for my MYSQL database tables and everything is fast and reliable, but...

As I continue to put a fair amount of daily information into my database and its tables, I can't help wondering the resource hit the machine running my MDI application is taking, as my database grows.

At first, when I took the dataset course as opposed to just hammering my database one query at a time, I suspected or hoped that the data adapter would take into consideration of limited resources and channel database data in blocks based on record age (not sure what criteria would be used), but now that I have a better understanding of what goes on, I suspect the whole enchilada is loaded into the datasets. As my database grows, so does the resource hit.

I considered writing a class that loads only a range of data into the datasets, changing that range if queries call for older data, but before I set sail on those uncharted (for me) seas, I thought I would ask for some advice on this topic. The advice being how I would approach using datasets on my database when the database is expected to grow very large.

I sincerely thank you for your time!

Hop

VB 2008 Express Windows XP Pro / MYSQL (5.0.90) through an Apache Server (Apache/2.2.14 (Win32) PHP/5.2.11)

 

[jmcilhinney]

It really comes down to how the application is going to be used. Regardless of memory requirements, does it really make sense to display a very large number of records to the user at the same time? Probably not, because a user can only really deal with a certain amount of data at a time. If it only makes sense to show the user data from the last month then filter the data by date. If they need to see older data then let them specify a previous month. You should start by analysing the requirements of your application and work from there.

 

[Hopworks]

Since you put it that way, it makes sense to load some datasets for the work at hand. Shouldn't be too hard to modify my code to adjust to that. Thank you.