Calling Analytical Guru's: Challenge - Getting fastest web services...

[Administrator]

Calling all analytical .NET gurus that like to get the most out of .NET performance!

I've been doing some tests to see what the fastest way is to get a "dataset" of data via web services. Take an example DataSet containing 1,000 rows (or more if you want) then try every way you can think of to transfer it from your server to your client. Use the StopWatch class to measure the elapsed milliseconds in your test (ms is sufficient) and see which delivery method gives you the best speed.

I've been experimenting with retrieving plain strongly typed datasets, byte arrays, compressed byte arrays (DeflateStream class), then also throwing in some encryption and compression again and seeing what that brought.

My initial tests with smaller datasets concluded that the strongly typed dataset was fastest! Faster even than a compressed byte array. Granted, all web services are using my custom IHTTPHandler also employing gzip/deflate compression on every call, so the DataSet effectively is getting HTTP compression to help it along. I haven't tried larger datasets yet.

So for those interested, let's see if we can determine and/or come up with a way to deliver the FASTEST datasets from .NET 2.0.

No restrictions, see what you come up with!

 

[Administrator]

aha, just got your point, you manually add the string "?cx=off" to the URL in the client application.. well, that is an option. But my suggestion is easier to maintain at service point

Just got your reply to your post, so maybe my idea will work? It would involve manually setting the URL though (I think) which could be a little bit of a pain, not sure. Can you give me a little more detail/example of your method?

 

[Administrator]

Just got your reply to your post, so maybe my idea will work? It would involve manually setting the URL though (I think) which could be a little bit of a pain, not sure. Can you give me a little more detail/example of your method?

The other thing to consider, and why I like my method, which is actually used by another system I use, so I'm familiar with this way of doing things, is we have to open our view that this is not just web services in this asp.net web site. We may need this IHTTPModule for the "web site" in which this web.config applies. So we need some trigger not only for web services but other normal .aspx pages that we may NOT want compressed, i.e. web PDF reporting which I've had a problem with in the past writing PDF to the response stream. I don't see a reason I really would NOT compress web services, I can't think of a conflict. But .aspx pages could be a conflict and the method I offered is universal, any caller can easily bypass compression with this method **IF** I coded it correctly!

 

[JohnH]

Can you give me a little more detail/example of your method?

I typically do like this:

Dim webserv As New localhost.Service
Dim ds As DataSet = webserv.getDS

I understand the 'webserv' instance here is type WebClientProtocol from namespace System.Web.Services.Protocols, and don't see any way of modifying the request call from this.

 

[JohnH]

Silly me wasn't thinking, the equivalent of this using strong code is simply to provide a parameter to the webmethod, this parameter will add itself to the request params collection by its name and can be handled in IHttpModule BeginRequest event the same. Example, "input" will be the key of this parameter in params collection:

<WebMethod()> _
Public Function getData(ByVal input As Integer) As Byte()
'...
End Function

It's still not all the same, since your suggestion would be a webservice parameter and not an actual method parameter. They could be coupled without trouble, this gives a clean interface for the service client. Hardcoding a querystring with hidden undocumented parameters is not particularily clean, but could in cases be useful.

Somebody find us more optimizations now!

 

[Administrator]

LOL! No reason you can't code both into your IHTTPModule. What you suggest works great for use of web services only, which is fine, to me, I see no reason NOT to allow all of them to run through the IHTTPModule. I just don't see it taking much time especially if already internally compressed in the byte array. It's like winzipping a zip file, if no compression is used, the time will be minimal.

However, what in my approach, using a query string parameter, it's universal to anything the IHTTPModule is handling, i.e. any page type.

But we digress, I think we've beaten this horse deeead! Back to the topic of the fastest web services possible, if anyone has anything to share on that, please do. I realize the binary byte array is the fastest processing, but the package is 4 times the size of the binary deflated test, which is why I selected that as best in my opinion for fastest overall, keeping in my transport time (bandwidth). If caching is used, the processing time factor could be removed from the equation if the cache is serving the data.

Also, I wonder if there is a way in .NET 2.0 to control the compression strenght of the deflate/gzip streams? Right now we just have a "compress" method but if we could optimize that such as a hi/med/lo we may optimize here by saving some of the processing time while balancing the payload out???

 

[jhomminga]

Serialization examples

Many many thanks for the thread. It was a great read about hard won test results.

Would anyone be willing to share some examples to get me started with VB2005 serialized dataset optimizations. Found stuff for MTOM and then stuff using DeflateStream, BinaryFormatter, and Byte() arrays.

I want to send and receive dataset via webservice.

Reading your thread says send byte arrays instead converting to/from datasets. Just need the compress/decompress steps for reliably maximizing byte array to/from dataset.

Also found Bromberg Self-Contained Compressed Remoteable Object but the code is intense and in C#. I am trying to keep this VB and without using xceedSoft components.

much appreciated,
jhomminga

 

[JohnH]

Actually typing the webservice function as Dataset and applying the IHttpModule compression (attached post 2) gives best overall performance, no "conversion" time and only the slow bzip2 can beat the transmission byte size. Just link that class to the webservice and off you go, no additional coding necessary.

If you want to look into the other methods I have attached the class I used to test all the different combinations of serialization methods and compression options (table post 10). If you don't want the bzip2 from SharpZipLib (link post 7) it can easily be removed from the "compression options" region (or adding others and tell us if they are better..). There are no code comments, but the class is cleanly coded and organized in regions. It works with only one public shared method to convert Dataset to Bytes and one the other way, just giving the relevant options.

 

[jhomminga]

thanks!!!

most excellent examples thanks!!!

I will work through them, googling all I can find to complement them, and will post with hopefully some beneficial findings.

Not waiting on the conversion is compelling. I feel a lot better about testing with your class in hand <vb grin>. Hope to repay the favor.
Some apps here really just talk server to server (cross-domain) on interval so the "compress time" may be worth the extra wait.

I agree on get all you can each communique and not tie up the server with round trips where avoidable. Prior i used base 64 encoding into xml node and xmlhttp upload to an awaiting ADODB stream object. The Ado.Net 2 serialization improvements and compression namespace are awesome. And I can't wait to try out the fiddler tool...another excellent find.

again thanks