Background Load Datagridview with 1000s of records

Ultrawhack

Well-known member
Joined
Jul 5, 2006
Messages
164
Location
Canada
Programming Experience
3-5
Hi,

I need to "quickly" display 30,000+ records in my datagridview. Using a traditional dataset & fill like
http://www.codeproject.com/useritems/DataGridView__GridView.asp
it takes upwards of 20 seconds for the DGV to display the records on a Intel DualCore 3Ghz 2GB RAM system (and of course hangs till it's done)

Is there any more efficient way to populate a DGV with this kind of volume async using a backgroundworker and perhaps a progressbar?

Thanks !
 
hi,

i think the answer is to create another thread for data access, and run it separate the Ui, that way it will stay responsive, at the same time as filling your dgv

search for the thread "useful multithreading tutorial",

good luck :)

regards
adam
 
The real question here is, do you really need that amount of records loaded into a datatable? This is a question that i see quite regularly. I have never written a program that retrieves that kind of volume of data. Have a think about your app's design. I just can't invisage a situation where a user would want need to be able to look at 30'000+ rows of data.
 
not quite true vis. a program i am working on at the moment is going to be using that sort of data. it is for stock codes in a warehouse, and my boss wants to be able to see a complete list of every item, how many times its picked, how much is picked and a date range. so it is possible to need to view 30000 rows... its just a pain in the ass :p

regards
adam
 
I that was the case then i would arrange the program to be able to retrieve information by Date Range, items picked, How much is picked and/or a combination of each. Thats the point of a query engine. Select the data that is pertinant at the time, when done discard and get the next set of results. I can't even imagine how long it would take to trawl through 30'000 + rows of data all in one go, let alone be able to make sense of it. Applications should be designed to be user friendly, the idea of 'Dump it all out into one result set' gives me the shivers and i would strongly advise any one asking me to create such a program against the idea.
 
my program has functionality similar to that, but he still wants a complete, unfiltered list to be shown if he chooses. now initially, the list isnt gonna be bigger than 3 or 4 hundred, but it will get bigger as time goes on.

what you need to understand, is that sometimes.. just sometimes.. this amount of data is necessary, due to the demands of his company and how often they choose to audit him (or his section). it might not be nice... it might not be fun.. but it does happen. i know that in my company there are reports that span about 80 pages... of stock ids, descriptions and locations. i have seen them and wow... they are nasty, just like you say... but they are necessary.

but at the end of the day it all comes down to what the "client" (in this case my boss) wants. we can offer suggestions (as i did by offering to filter by dates, item codes etc) but when the client says "i want this, and thats that"... we have to do it.
 
I can accept your point, but only just. Every time i have been asked to do something like that i've always got my own way. When people hire me, they hire me to tell them what they need in terms of IT solutions. They give me some guidelines i'll admit. But i can't say i've ever had a situation where some hires me to do a job and then tells me how to do it.
 
dont get me wrong vis, i understand what your saying and i do agree with you, but when a company has its own way of doing things, sometimes there not much you can do except go with the flow and just deal with it in the best way possible.

regards

adam
 
Not wanting to make anyone feel like sides are being taken, but I support Vis' point here.. I'm currently doing work for a company with a large customer service department and all calls are logged. The Customer Services Manager said he wanted to see all the calls and information about them in a table. I pointed out that I didnt think he would get much out of my providing a facility that allowed him to scroll up and down a million rows - one pixel scroll of the scrollbar jumped more than a page of information anyway.

There comes a point where data is jsut data, and loses any relevant meaning as information purely through volume.

If I were in your warehouse scenario I would have declined to provide a list of all stock items for the same reason that I would decline to write a program that searched the entire public drive and combined all word documents into one.

Rich, your point about the bos being the client and hence must be provided with what he wants is valid, but generally management arent skilled or technical, and cannot envisage a good way to do things on a technical level. i.e. They might tell you what they want, but they dont have much of a clue what it is they actually want. As a consultant, your time is best spent easing your workload and reducing resource demand through user reeducation in this stage:)


now, back to the question:

Hi,

I need to "quickly" display 30,000+ records in my datagridview. Using a traditional dataset & fill like
http://www.codeproject.com/useritems/DataGridView__GridView.asp
it takes upwards of 20 seconds for the DGV to display the records on a Intel DualCore 3Ghz 2GB RAM system (and of course hangs till it's done)

Is there any more efficient way to populate a DGV with this kind of volume async using a backgroundworker and perhaps a progressbar?

Thanks !

20 seconds? you gotta admit, it's pretty good hey? 1500 records a second, over a 100 meg network link, from a server with a huge memory and raid array, shoed into a poxy client, stored in a structure not really intended for massive volumes. If one of those records takes up 2kb in overhead, that's 3 megs a second! I'm pretty sure I dont see that often, over our network, so your setup looks pretty hot..
Short of creating a fatter network pipe and ensuriong youre using a native driver, there isnt much you can do. Download less data. Vis' point about the more information you show to the user, the more it reverts to meaningless data, is a worthy one.

On a side, you should know that there is nothing that will speed things up. Putting something on a background thread usually achieves the task slower, because CPU cycles are dedicated to keeping the UI responsive and processing the thousands of windowing events per second. The most efficient way to populate a DGV is to remove extranous data, leaving only that to be downloaded that is relevant - the true peak of efficiency; waste nothing.
 
Rich, your point about the bos being the client and hence must be provided with what he wants is valid, but generally management arent skilled or technical, and cannot envisage a good way to do things on a technical level. i.e. They might tell you what they want, but they dont have much of a clue what it is they actually want. As a consultant, your time is best spent easing your workload and reducing resource demand through user reeducation in this stage

Cjard,

although i feel slightly annoyed at that comment (and the little bit above it as well), you are the better programmer than me so i will definitely store that advice away for future use... i appreciate the feedback, anything that will make me a better programmer in the future i value greatly. next time my boss wants something like this i will "reeducate" him :)

have a good one

regards
adam
 
Sorry.. I wasnt seeking to annoy you - I've just learned the hard way, not being paid because my project ran over schedule as a result of trying to bend over backwards too far to accommodate all user's wishes..

I now find that managers are very amenable to reasoned arguments, especially if you can point out that their requests will cost money directly (its going to take longer to write so I will charge you more) or indirectly (its going to put a huge strain on your resources so you'll have to upgrade your server) - make out that youre being sensitive to their needs while being honest that the extra effort required from you is not an economically viable one :)
Additionally, your programming brilliance compared to theirs may make it possible for you to suggest and implement another route that they hadnt considered - one that solves all sides!

Try never to get into the situation where youre working for free :D Even if it's fun, man's gotta eat :)

(oh, and never think that anyone is a better programmer than you - they're just different or have more experience in a certain field.. It may lead you follow without question, and that's bad! Sorry again, if I came across as a "hey, i'm saying I know more than you so jolly well do as I say" - I didnt intend to)
 
(oh, and never think that anyone is a better programmer than you - they're just different or have more experience in a certain field.. It may lead you follow without question, and that's bad! Sorry again, if I came across as a "hey, i'm saying I know more than you so jolly well do as I say" - I didnt intend to)

haha, its ok, dont worry about it. in regards to your first sentence... ouch! man thats gotta sting! ill definitely take on board everything that has been said (includin what vis said... i realise my comments might have offended him. ie. not caring about his opinion, which is wrong cause i do, so my bad :)), and i understand completely what you said about managers and them "saving money"... they are very eager when it comes to stuff like that, so i will probably do what you suggested.

cheers for your feedback mate, much appreciated.

have a good one :)
 
Thanks gentlemen, this discussion has made me rethink paging strategy and I've put reasonable bounds in place and balance functionality with performance.
 
Back
Top