Question Streamreader limits

[daz51]

I have a problem with a streamreader in visual basic 2008. I am updating a database from text files on an ftp server. The streamreader works fine until the file is more than 100,000 bytes long. It reads until it reaches this point and then just ends. No error message, it just reads to this point which happens to be 2 fields out of 6 in the database stream. My question is - does the streamreader have a limit? And if so what is a workaround?

Dim connString As String = "Data Source=" & pDataBase
        conn = New SqlCeConnection(connString)
        Dim strQuery As String = ""
        Dim pDate As Date
        Dim sDate As String

        Try
            conn.Open()
            
            Dim sr As New StreamReader("c:\NCPLink\files\" & sFile, Encoding.Default, 32000)
            sr.BaseStream.Seek(0, SeekOrigin.Begin)

            sCnt = 0

            strQuery = "delete from departments"
            cmd = New SqlCeCommand(strQuery, conn)
            cmd.ExecuteNonQuery()


            Do Until sr.EndOfStream

                Dim sLine As String = ""
                Dim sOut() As String

                sLine = sr.ReadLine()
                sOut = Split(sLine, ",")
                sCnt = sCnt + 1

                        strQuery = "INSERT INTO [departments](code, description, price, priceoverride, plusgst, extra) "
                        strQuery = strQuery & "VALUES ('" & Trim(sOut(0)) & "', '" & Trim(sOut(1)) & "', " & Trim(sOut(2)) & ", '" & Trim(sOut(3)) & "', '" & Trim(sOut(4)) & "', '" & Trim(sOut(5)) & "')"

                cmd = New SqlCeCommand(strQuery, conn)
                cmd.ExecuteNonQuery()
            Loop

            sr.Close()
        Catch ex As SqlCeException
            MessageBox.Show(ex.Message)
        Finally
            conn.Close()
        End Try

 

[ss7thirty]

Ive used the similar code to below on some really large files for streamreader and writer:

Sub ProduceTxtFromDat()
   Dim sr as new System.IO.StreamReader("LargeTextFile.dat")
   Dim line as string = sr.readline
   Do until line is nothing
      ProcessLine(Line)
      line = sr.readline
   Loop
   sr.close()
End Sub

Sub ProcessLine(Line)
   Dim sw as new System.IO.StreamWriter("LargeTextFile.txt", True) 'T=append
   sw.writeline(line)
End Sub

 

[daz51]

Thank you for the reply, but this code also does the same thing. 100,000 bytes are read in then the code ends. No error. I have looked at the import file (many times) and that is not the problem. If I remove the record that it stops on, it stops on the next, or the next......... I have looked at this problem over the last 2 weeks and I am tearing my hair out, not that I have any hair anyway.

 

[ss7thirty]

I created a 20MB text file and ran my code on it and it worked.

 

[daz51]

Thank you for doing that test. It gave me another clue. I was downloading the file from an ftp server and I had set my buffer to 100000!!! The file arriving was only 100000 bytes long! My program is over 50,000 lines of code, so it was a little hard to spot. The problems you perceive is not always where it seems. Thank you problem solved.