Question AseDataReader timeout

Patti

Member
Joined
Jun 7, 2010
Messages
5
Programming Experience
Beginner
Hello. I am just starting out my career in programming, so please forgive my noobness.

In brief, my reader is timing out before all the data I need is loaded into a list of objects in my program. As I see it, I have two options (please let me know if there are more). I can:
a) Use a cursor to retrieve only 1000 rows at a time (works with the default timeout value)
or
b) Increase the timeout to x number of seconds

Increasing the timeout is simple and, as I understand it, more efficient than using a cursor. However, I am not sure this is a good solution to my problem for two reasons:
a) If the timeout is set to a value that will handle the amount of data in production at the present time, but that amount of data is constantly growing, the timeout may occur again, but this time in production some time in the future (don't want that happening in production)
b) If the timeout is set to a really large value, this may allow the program to keep running if there is a connectivity issue or some issue other than just the program handling the large amount of data (should this be a concern?)

So because of those two reasons, I am considering using a cursor to retrieve 1000 rows at a time, since 1000 rows can be retrieved in a few seconds, well under the timeout default.

My questions are:
Is there some other solution to my issue other than using cursors?
Should I just increase the timeout value for my reader? If so, is there some way I can handle the eventuality that the connection is hanging and not handling data?

Thank you in advance for any help and advice.
 
If you can't load enough data before your query times out, did you consider:

Your query is too slow and you should consider rewriting it so it operates more quickly
Youre trying to load too much data (how many objects are you trying to load? - there's little point loading 20,000 objects and expecting the user it interact with them, for example)
 
Hello cjard,

The task of the program is to read two different sets of data from the database and output each set to it's own file. The user has requested that every night, this process runs, so that each morning, an updated file is available to them. The user wants all records matching a certain specification.

There are currently 400,000 records being written to the second file (which is larger than the first). The amount of records being output has doubled in size over the past two years and will continue to grow. The size of the database itself will also continue to grow, as I have just found out there is no archiving in process yet for the database and no word of any archiving being planned. Thus, my concern for needing a larger timeout in the future.

Below are my SQL statements which are currently within stored procedures:
For the first file:
select

r.REMITTANCE_TYPE_TABLE_ID,
r.REMITTANCE_TYPE_CD,
gd.cd as REMITTANCE_TYPE_CD_TEXT,
r.REMITTANCE_ID,
r.REMITTANCE_SEQUENCE_NO,
r.REMITTANCE_STATUS_TABLE_ID,
r.REMITTANCE_STATUS_CD,
gc.SHORT_DESC AS REMITTANCE_STATUS_TEXT,
r.REMITTED_AMT,
r.COLLECTED_AMT,
r.REMITTANCE_USED_IND,
r.BANK_ERROR_IND,
r.REMITTANCE_ERROR_IND,
r.BANK_BATCH_ID,
r.EXPIRED_IND,
r.PAID_DT

from
REMITTANCE r
LEFT JOIN generic_code gc
ON r.REMITTANCE_STATUS_CD = gc.NUMERIC_CD
AND gc.TABLE_ID = 5
AND gc.language_cd = @LANGUAGE_CD

LEFT JOIN generic_code gd
ON r.REMITTANCE_TYPE_CD = gd.NUMERIC_CD
AND gd.TABLE_ID = 6
AND gd.language_cd = @LANGUAGE_CD

where
REMITTANCE_STATUS_CD = @RemittanceStatusCD

ORDER BY r.REMITTANCE_ID


For the second (longer) file:
SELECT

BRD.REMITTANCE_TYPE_TABLE_ID,
BRD.REMITTANCE_TYPE_CD,
BRD.REMITTANCE_ID,
BRD.REMITTANCE_SEQUENCE_NO,
BRD.LOB_TABLE_ID,
BRD.LINE_OF_BUSINESS_CD,
GC.SHORT_DESC as LINE_OF_BUSINESS_TEXT,
GC.CD as LOB_CODE,
LOB.LINE_OBJECT_CD as LINE_OBJECT_CODE,
BRD.REMITTANCE_UNIT_NO,
BRD.REMITTANCE_LINE_AMT,
BRD.CAPTURED_LINE_AMT,
BRD.DEFERRED_STATUS_TABLE_ID,
BRD.DEFERRED_STATUS_CD,
BRD.COUNTED_IND

FROM
BANK_REMITTANCE_DETAIL BRD

LEFT JOIN Line_of_Business LOB
ON BRD.LINE_OF_BUSINESS_CD = LOB.LINE_OF_BUSINESS_CD
AND LOB.LOB_TABLE_ID = 10

LEFT JOIN generic_code GC
ON BRD.LINE_OF_BUSINESS_CD = GC.NUMERIC_CD
AND GC.TABLE_ID = 10
AND GC.language_cd = 1

LEFT JOIN remittance rm
ON BRD.REMITTANCE_TYPE_TABLE_ID = RM.REMITTANCE_TYPE_TABLE_ID
AND BRD.REMITTANCE_TYPE_CD = RM.REMITTANCE_TYPE_CD
AND BRD.REMITTANCE_ID = RM.REMITTANCE_ID
AND BRD.REMITTANCE_SEQUENCE_NO = RM.REMITTANCE_SEQUENCE_NO

WHERE
RM.REMITTANCE_STATUS_CD = @RemittanceStatusCd

ORDER BY
BRD.REMITTANCE_TYPE_CD, BRD.REMITTANCE_ID
 
Sorry, what do you mean by "plans"?

The entired stored procedure is the following:
For first file:
USE hpmqa
go
IF OBJECT_ID('dbo.pr_get_RemitByStatus') IS NOT NULL
BEGIN
DROP PROCEDURE dbo.pr_get_RemitByStatus
IF OBJECT_ID('dbo.pr_get_RemitByStatus') IS NOT NULL
PRINT '<<< FAILED DROPPING PROCEDURE dbo.pr_get_RemitByStatus >>>'
ELSE
PRINT '<<< DROPPED PROCEDURE dbo.pr_get_RemitByStatus >>>'
END
go
create procedure dbo.pr_get_RemitByStatus

@RemittanceStatusCd smallint,
@app_object varchar(25),
@app_object_version smallint,
@LANGUAGE_CD smallint
as
declare @proc_name varchar(30), -- this procedure name
@section_ident varchar(30), -- identifier for a section of code - used for debugging
@errmsg varchar(255), -- message text
@lang char(8), -- language name NULL if English otherwise "French"
@procerr tinyint, -- used to trap return codes from called procs
@err int -- used to capture the current @@error value

-- ---------------// initialize variables //----------------------

select @proc_name = "pr_template"

------------------// validate parameters //-------------------------

select @section_ident = "validate parameters"
--
if @LANGUAGE_CD = 1
select @lang = NULL

else
if @LANGUAGE_CD = 2
select @lang = "French"
else
begin
select @lang = NULL

select @err = 50000
goto error
end
--
if @app_object = NULL

begin
select @err = 50001
goto error
end
--
if @app_object_version = NULL

begin
select @err = 50002
goto error
end
select @section_ident = "mainline"

begin

select

r.REMITTANCE_TYPE_TABLE_ID,
r.REMITTANCE_TYPE_CD,
gd.cd as REMITTANCE_TYPE_CD_TEXT,
r.REMITTANCE_ID,
r.REMITTANCE_SEQUENCE_NO,
r.REMITTANCE_STATUS_TABLE_ID,
r.REMITTANCE_STATUS_CD,
gc.SHORT_DESC AS REMITTANCE_STATUS_TEXT,
r.REMITTED_AMT,
r.COLLECTED_AMT,
r.REMITTANCE_USED_IND,
r.BANK_ERROR_IND,
r.REMITTANCE_ERROR_IND,
r.BANK_BATCH_ID,
r.EXPIRED_IND,
r.PAID_DT

from
REMITTANCE r
LEFT JOIN generic_code gc
ON r.REMITTANCE_STATUS_CD = gc.NUMERIC_CD
AND gc.TABLE_ID = 5
AND gc.language_cd = @LANGUAGE_CD

LEFT JOIN generic_code gd
ON r.REMITTANCE_TYPE_CD = gd.NUMERIC_CD
AND gd.TABLE_ID = 6
AND gd.language_cd = @LANGUAGE_CD

where
REMITTANCE_STATUS_CD = @RemittanceStatusCD

ORDER BY r.REMITTANCE_ID
end

if (@@error != 0)
begin
select @err = @@error
goto error
end

return
-- ---------------// error handling //------------------------
--
error:

--
-- get the text of the message in the correct language
--
exec @procerr = sp_getmessage @err, @errmsg output, @lang
--
if @procerr = 0
raiserror @err, @errmsg
else
raiserror
@err "unknown error: %1! occurred in proc: %2!, section: %3!",
@err, @proc_name, @section_ident
--
return
go
EXEC sp_procxmode 'dbo.pr_get_RemitByStatus', 'anymode'
go
IF OBJECT_ID('dbo.pr_get_RemitByStatus') IS NOT NULL
PRINT '<<< CREATED PROCEDURE dbo.pr_get_RemitByStatus >>>'
ELSE
PRINT '<<< FAILED CREATING PROCEDURE dbo.pr_get_RemitByStatus >>>'
go
GRANT EXECUTE ON dbo.pr_get_RemitByStatus TO gp_developer
go
GRANT EXECUTE ON dbo.pr_get_RemitByStatus TO gp_user
go

---------------------------------------------------------------
second file (longer processing time):
USE hpmqa
go
IF OBJECT_ID('dbo.pr_get_BankRemittanceDetailByStatus') IS NOT NULL
BEGIN
DROP PROCEDURE dbo.pr_get_BankRemittanceDetailByStatus
IF OBJECT_ID('dbo.pr_get_BankRemittanceDetailByStatus') IS NOT NULL
PRINT '<<< FAILED DROPPING PROCEDURE dbo.pr_get_BankRemittanceDetailByStatus >>>'
ELSE
PRINT '<<< DROPPED PROCEDURE dbo.pr_get_BankRemittanceDetailByStatus >>>'
END
go
create procedure dbo.pr_get_BankRemittanceDetailByStatus
@RemittanceStatusCd smallint,
@app_object varchar(12),
@app_object_version smallint,
@LANGUAGE_CD smallint
as
declare @proc_name varchar(30), -- this procedure name
@section_ident varchar(30), -- identifier for a section of code - used for debugging
@errmsg varchar(255), -- message text
@lang char(8), -- language name NULL if English otherwise "French"
@procerr tinyint, -- used to trap return codes from called procs
@err int -- used to capture the current @@error value

-- ---------------// initialize variables //----------------------

select @proc_name = "pr_template"

------------------// validate parameters //-------------------------

select @section_ident = "validate parameters"
--
if @LANGUAGE_CD = 1
select @lang = NULL

else
if @LANGUAGE_CD = 2
select @lang = "French"
else
begin
select @lang = NULL

select @err = 50000
goto error
end
--
if @app_object = NULL

begin
select @err = 50001
goto error
end
--
if @app_object_version = NULL

begin
select @err = 50002
goto error
end
select @section_ident = "mainline"

begin

SELECT

BRD.REMITTANCE_TYPE_TABLE_ID,
BRD.REMITTANCE_TYPE_CD,
BRD.REMITTANCE_ID,
BRD.REMITTANCE_SEQUENCE_NO,
BRD.LOB_TABLE_ID,
BRD.LINE_OF_BUSINESS_CD,
GC.SHORT_DESC as LINE_OF_BUSINESS_TEXT,
GC.CD as LOB_CODE,
LOB.LINE_OBJECT_CD as LINE_OBJECT_CODE,
BRD.REMITTANCE_UNIT_NO,
BRD.REMITTANCE_LINE_AMT,
BRD.CAPTURED_LINE_AMT,
BRD.DEFERRED_STATUS_TABLE_ID,
BRD.DEFERRED_STATUS_CD,
BRD.COUNTED_IND

FROM
BANK_REMITTANCE_DETAIL BRD

LEFT JOIN Line_of_Business LOB
ON BRD.LINE_OF_BUSINESS_CD = LOB.LINE_OF_BUSINESS_CD
AND LOB.LOB_TABLE_ID = 10

LEFT JOIN generic_code GC
ON BRD.LINE_OF_BUSINESS_CD = GC.NUMERIC_CD
AND GC.TABLE_ID = 10
AND GC.language_cd = 1

LEFT JOIN remittance rm
ON BRD.REMITTANCE_TYPE_TABLE_ID = RM.REMITTANCE_TYPE_TABLE_ID
AND BRD.REMITTANCE_TYPE_CD = RM.REMITTANCE_TYPE_CD
AND BRD.REMITTANCE_ID = RM.REMITTANCE_ID
AND BRD.REMITTANCE_SEQUENCE_NO = RM.REMITTANCE_SEQUENCE_NO

WHERE
RM.REMITTANCE_STATUS_CD = @RemittanceStatusCd

ORDER BY
BRD.REMITTANCE_TYPE_CD, BRD.REMITTANCE_ID

end

if (@@error != 0)
begin
select @err = @@error
goto error
end

return
-- ---------------// error handling //------------------------
--
-- some specific messages should be passed parameters
--
error:

--
-- get the text of the message in the correct language
--
exec @procerr = sp_getmessage @err, @errmsg output, @lang
--
if @procerr = 0
raiserror @err, @errmsg
else
raiserror
@err "unknown error: %1! occurred in proc: %2!, section: %3!",
@err, @proc_name, @section_ident
--
return
go
EXEC sp_procxmode 'dbo.pr_get_BankRemittanceDetailByStatus', 'anymode'
go
IF OBJECT_ID('dbo.pr_get_BankRemittanceDetailByStatus') IS NOT NULL
PRINT '<<< CREATED PROCEDURE dbo.pr_get_BankRemittanceDetailByStatus >>>'
ELSE
PRINT '<<< FAILED CREATING PROCEDURE dbo.pr_get_BankRemittanceDetailByStatus >>>'
go
GRANT EXECUTE ON dbo.pr_get_BankRemittanceDetailByStatus TO gp_developer
go
GRANT EXECUTE ON dbo.pr_get_BankRemittanceDetailByStatus TO gp_user
go
 
Sorry, what do you mean by "plans"?
Using a query tool you can ask a database to plan a query, and it shows you how it has decided to retrieve the data, in what order, how it joins it together, what indexes it will use etc

With this info you can then tune the query so that it is easier for the db to run. By example heres an oracle plan:
VB.NET:
IDPIDOperation                             Name                            RowsBytesCostCPU CostIO 
 0   SELECT STATEMENT                                                      1   153  930 4       929
 1  0  NESTED LOOPS OUTER                                                  1   153  930 4       929
 2  1    VIEW                                                              1   86   929            
 3  2      NESTED LOOPS                                                    1   147  929 3       928
 4  3        NESTED LOOPS                                                  1   123  928 2       927
 5  4          NESTED LOOPS                                                8   768  926 1       925
 6  5            VIEW                                                      132 5280 892            
 7  6              WINDOW SORT PUSHED RANK                                 132 9636 892         892
 8  7                HASH JOIN                                             132 9636 889         889
 9  8                  INDEX FAST FULL SCANBW3.CNSTR_CHT_136               13  52   1           1  
10  8                  TABLE ACCESS FULL   BW3.SVC_CARD_STATUS_CHANGE      132 9108 887         887
11  5            TABLE ACCESS BY INDEX ROWIBW3.SVC_CLIENT_CARDS            2   112  2   1       1  
12 11              INDEX UNIQUE SCAN       BW3.PK_SVC_CLIENT_CARDS         1                       
13  4          TABLE ACCESS BY INDEX ROWID BW3.CAS_CLIENT_ACCOUNT          2   54   2   1       1  
14 13            INDEX RANGE SCAN          BW3.IX3_CAS_CLIENT_ACCOUNT      2                       
15  3        TABLE ACCESS BY INDEX ROWID   BW3.CIS_CLIENT_DETAILS          1   24   2   1       1  
16 15          INDEX UNIQUE SCAN           BW3.PK_CIS_CLIENT_DETAILS       4                       
17  1    TABLE ACCESS BY INDEX ROWID       BOSTON_GROUP_CONTROL            1   67   2   1       1  
18 17      INDEX UNIQUE SCAN               BOSTON_PK_GROCON                1

When posting code, please use
VB.NET:
 tags
 
So basically the solution to this issue, is to fine-tune the query as opposed to fine-tuning the program code?

Is there a query planner that can be used with a Sybase database?
 
First, we take a look at the plan and see if there are optimisations to be made; youre talking about joining half a million rows and dumping them to disk every night.. If those half million rows must be joined to another half million rows, that's a big job, and the wrong kind of join can be a disaster. Get the join right, and it will be nearly as quick as just dumping the rows from one table, no join

After we have the query running quickly in a query tool, or the query tool saying that the query is running quickly, we can make a decision to increase timeout if there physically jsut isnt enough time to drag all this data over a 100 meg LAN or whatever, but typically I'd expect a datareader to wait until the first rows start coming through as its timeout i.e. if your query takes longer to generate its first row than the timeout that's the sticking point (and the question you have to ask is how long is reasonable for this query to take - i work in an oracle system where some queries take several hours and i've rewritten them to take a few seconds because of the impact they were having on general business).
Once you start reading rows, you shouldnt have a timeout problem.. and yep, this is one case where I'd use a datareader rather than a datatable.. persist to disk one row at a time through a buffered disk writing system rather than try and fit the entire resultset into memory before writing it (datatable route)

I don't use Sybase at all, sorry - never have; i cannot comment on how to find its query planner output. Try something like Sybase - Explain Plan or suitable google search terms
 
Hi cjard,

We took at look at the problematic query and found out the join was actually incorrect. We were getting data from the wrong table first, which resulted in the retrieval of extra rows we did not need. The query went down from two minutes to 30 seconds on our test server, which is slower than production. We agreed that a 10 minute timeout to allow for the growth of the production data was acceptable.

Also, we eliminated the need for two separate queries. By modifying the first query a bit, we can get the extra data that we need for the second query and run only one query instead of two, speeding up the program as a whole. We can now create our two files with only having to go to the database once, insted of twice.

Thank you for suggesting that we look into optimizing the query. This was the correct solution in our case.
 
Back
Top