Question FTP to IBM Mainframe - How do I convert to EBCDIC?

[robertb_NZ]

I am trying to implement FTP to/from a zOS mainframe. My current code is below: this seems to transfer the data to the mainframe perfectly except that the results are gibberish, presumably because the data hasn't been converted from Unicode to EBCDIC. How do I do this?

The source file (path\TEST1.JCL, on my laptop) has data: -
//IBMUSERF JOB ,CLASS=A,MSGCLASS=H,NOTIFY=&SYSUID,COND=(8,LT)
//*** COPY SOURCE INTO SOURCE LIBRARY
//* Dummy Data for initial FTP test

After this has been FTP'd to the zOS system it looks like this: -
??(????? ?|? ?< ?? (???< ?? ? +|???? ?????? ?|+? <? ?
?? ?+?| ?|???? <??? ?? ??__` ?/?/ ??? ?>???/% ??& ????
(which replaced the previous contents, so I know that something has been transferred).

Here is my code: -
Dim JCLFile As String = My.Settings.UserCommonPath & "\" & My.Settings.Programs & "\TEST1.JCL"
Try
Dim FTPServer As String = "FTP://" & My.Settings.SubmitIP & ":" & My.Settings.SubmitPort & "/"
Dim TargetFile As String = "'IBMUSER.MANAJAZZ.SRCLIB(TEST1)'"
Dim FTPTarget As String = FTPServer & TargetFile
Dim Request As System.Net.FtpWebRequest =
DirectCast(System.Net.WebRequest.Create(FTPTarget), System.Net.FtpWebRequest)
Request.Credentials = New System.Net.NetworkCredential(My.Settings.Userid, My.Settings.Password)
Request.Method = System.Net.WebRequestMethods.Ftp.UploadFile

' read in file...
Dim bFile() As Byte = System.IO.File.ReadAllBytes(JCLFile) '<= Source file: see JCLFile above

' upload file...
Dim clsStream As System.IO.Stream = Request.GetRequestStream()
clsStream.Write(bFile, 0, bFile.Length)
clsStream.Close()
clsStream.Dispose()
Catch ex As Exception
MsgBox("Error with FTP:" & ex.Message)
End Try

Thank you for any feedback,
Regards, Robert.

 

[jmcilhinney]

Here's an example of converting text to EBCDIC:

VB.Net/C# - ASCII to EBCDIC

In your case, call ReadAllText instead of ReadAllBytes to get the file contents into a String, then create a StreamWriter on top of your request stream using the EBCDIC Encoding object, then write the String to that StreamWriter.

By the way, I just Googled .net EBCDIC, which is a pretty obvious search, and that was one of the first few matches. Assume that the information you need is already out there until proven otherwise.

 

[robertb_NZ]

Thank you for this, it looks very promising, although I'm having a bit of difficulty figuring out how to use ReadAllText. If I change
Dim bFile() As Byte = System.IO.File.ReadAllBytes(JCLFile)
to use ReadAllText then the target is no longer dimensioned: -
Dim bFile As String = System.IO.File.ReadAllText(JCLFile)
Perhaps I should be using ReadAllLines, in which case bFile would be defined Dim bFile() As String.

Whichever I use, the following statements
Dim clsStream As System.IO.Stream = Request.GetRequestStream()
clsStream.Write(bFile, 0, bFile.Length)
no longer compile, so they need to be changed somehow.

I'll try to figure this out tomorrow: it's after 1:00AM in the morning here in New Zealand, and my brain is too fuzzy to think straight

>By the way, I just Googled .net EBCDIC, which is a pretty obvious search ...
I'd done a very similar search, but hadn't found an answer that was obvious to me. I guess that's why you're an MVP and I'm not .

Regards, Robert.

 

[jmcilhinney]

You've ignored this part:

create a StreamWriter on top of your request stream using the EBCDIC Encoding object, then write the String to that StreamWriter.

 

[robertb_NZ]

No, I didn't ignore it, at 1:00 in the morning I didn't understand it. This morning I still don't, I'm sorry. I don't know what you mean by "Create a streamwriter on top of your request stream"
I have succeeded in sending EBCDIC to the zOS system, but it is ignoring my line breaks. I couldn't find any way to pass a string to clsStream.Write, the only options that would compile passed a byte array. So the best that I could do was: -
Dim JCLFile As String = My.Settings.UserCommonPath & "\" & My.Settings.Programs & "\TEST1.JCL"
Dim EBCDICFile As String = My.Settings.UserCommonPath & "\" & My.Settings.Programs & "\TEST1.ebc"
Try
Dim FTPServer As String = "FTP://" & My.Settings.SubmitIP & ":" & My.Settings.SubmitPort & "/"
Dim TargetFile As String = "'IBMUSER.MANAJAZZ.SRCLIB(TEST1)'"
Dim FTPTarget As String = FTPServer & TargetFile
Dim Request As System.Net.FtpWebRequest =
DirectCast(System.Net.WebRequest.Create(FTPTarget), System.Net.FtpWebRequest)
Request.Credentials = New System.Net.NetworkCredential(My.Settings.Userid, My.Settings.Password)
Request.Method = System.Net.WebRequestMethods.Ftp.UploadFile
'Request.UseBinary = False
' read in file...
ConvertAsciiToEbcdic(JCLFile, EBCDICFile)
Dim bytFile As Byte() = System.IO.File.ReadAllBytes(EBCDICFile)

' upload file...
Dim clsStream As System.IO.Stream = Request.GetRequestStream()
clsStream.Write(bytFile, 0, bytFile.Length)
clsStream.Close()
clsStream.Dispose()
Catch ex As Exception
MsgBox("Error with FTP:" & ex.Message)
End Try


This doesn't seem a very elegant method. Can you show me what you meant/what I should write?

BTW, note that I tried to use
'Request.UseBinary = False
hoping that that would solve the problem, but to my surprise that didn't work, the data not being passed in EBCDIC at all. I don't understand that.

Sorry if I'm a bit dense.

Regards, Robert.

 

[jmcilhinney]

No, I didn't ignore it, at 1:00 in the morning I didn't understand it. This morning I still don't, I'm sorry. I don't know what you mean by "Create a streamwriter on top of your request stream"

A StreamWriter is an object that writes text to a Stream. The most common way to create a StreamWriter is to specify the path of a file. Internally, the StreamWriter will create a FileStream and then, when you write text to the StreamWriter, it converts that text to bytes and writes those bytes to the FileStream. Rather than have the StreamWriter create a FileStream implicitly though, you can pass it an existing Stream; any type of Stream, e.g. a MemoryStream, a CryptoStream, a GZipStream or a NetworkStream. Look at this from your own code:

Dim clsStream As System.IO.Stream = Request.GetRequestStream()

Oh look, you have a Stream. Create the StreamWriter and pass it your Stream, specifying the appropriate Encoding. When you write your text to the StreamWriter, it will use that Encoding to convert the text to bytes and then write those bytes to the Stream.

For future reference, if someone says "use ClassX" then the first thing you do is open the MSDN Library documentation, which you can access via the Help menu in VS, and read about ClassX. Don't wait for information to come to you; go find it. If you'd read the documentation for the StreamWriter class then you'd have seen the constructor that takes a Stream and an Encoding as arguments.

I also don't see you calling ReadAllText.

 

[robertb_NZ]

Please don't assume that I haven't already done this. Before I first posted this query I'd Googled ".NET FTP EBCDIC" which showed me many of the articles that you've referred to, I had already read the MSDN library documentation on the relevant classes, including Stream, StreamReader, and StreamWriter, and I'd noted the point about a StreamWriter having a suitable-looking constructor. Also, in the last few years I have successfully used Stream, StreamWriter, StreamReader, and MemoryStream classes in ASP.Net and VB.Net needing only minimal help from the forums, However I'm struggling here, with the combination of FTP, Code Conversion, and Stream classes. Does it come as a surprise to you to hear that some people (= everybody I've spoke to who doesn't work for Microsoft) finds the MSDN documentation very obscure, and so they learn how to use VB, C# etc largely by trial and error, and by adapting examples that are close to what they want to do?

My major problem was interpreting your original advice "call ReadAllText instead of ReadAllBytes to get the file contents into a String, then create a StreamWriter on top of your request stream using the EBCDIC Encoding object, then write the String to that StreamWriter". Naturally I can't Google "What does he mean?" I was hoping that you'd simply show the the statement or two that I need to write and point me at the useful documentation (if it exists) to explain what is actually happening. I would have read your references. It seems to me that this would have been a lot easier for both of us than what seems to me to be unhelpful statements about things that I already know/have done, still leaving me to try to interpret what you originally meant.

I eventually figured out that you meant me to write this: -
Dim clsStream As New System.IO.StreamWriter(Request.GetRequestStream(), Encoding.GetEncoding(37))
(But why didn't you say so - it would have been simpler and easier for us both!)

EUREKA (almost). Much more elegant than my earlier klutzy "solution", but as before, line boundaries are being ignored: -
//IBMUSERH JOB ,CLASS=A,MSGCLASS=H,NOTIFY=&SYSUID,COND=(8,LT) //*** COPY SOUR
CE INTO SOURCE LIBRARY //COPY EXEC PGM=IEBGENER //SYSPRINT DD SYSOUT=* //SYSI
N DD DUMMY //SYSUT2 DD DSN=IBMUSER.MANAJAZZ.SRCLIB(CRDTA1),DISP=SHR //SYSUT1 D

Is the problem the code table? (Just to prove that I DO look up MDSN!), I Googled ".NET Encoding.GetEncoding" which took me to an MSDN page. Naturally this was pretty hopeless: it told me the inheritance hierarchy and gave an example, and referred me to a list of the code tables, but it didn't suggest any way of determining if my suspicion, that I'm using the wrong code table, is correct. IBM037 is "EBCDIC (USA and Canada)" which is appropriate for a computer in Dallas, TX, but I tried "IBM500" (=IBM International) anyway. Same result.

I confirmed with a Hex editor that the lines in JCLFile are separated by Hex 0D0A, which is vbCrLf

Two ideas: A: I should set Request.Usebinary = False
This screwed things up, causing a single 80-character record to be written. Here is the first 72 characters of it, in text and hex, as it appeared in the zOS system: -
aaIBTdbEYH@QVB@@kCSAbb~AkTbGCSAbb~HkUVcIFh~PbhbdIDkCVUD~MxkSc]@ %aa\\\@C
88CCE88CEC7DEC779CEC88AC9E8CCEC88AC9EE8CC8AD8888CC9CEECADA9E8B70688EEE7C
1192342588C852CC232122112327321221824539681728249423544147223DCDC11000C3

B: I should be using ReadAllLines rather than ReadAllText. I tried various options of this, but ended up with either the same results, or with nothing between one line and the next: either way I didn't end up with the original lines.

I Googled "FTP EBCDIC Line Breaks". Most of the references refer to IBM-side procedures. Perhaps this is where the answer is - I'll do some more research tomorrow.

This forum item should not have been closed: I don't yet have a working solution.

 

[jmcilhinney]

I assume that anyone posting here knows how to call a method, get or set a property and create an object. If they don't then they should be reading a beginners tutorial rather than posting here. If I say "call MethodX" then I assume that the person can call that method. If they have any doubts about the specifics then I expect that they can open the MSDN Library documentation and find out what parameters it has and what its return type is. If they post more code later that doesn't include a call to that method or any reference to their failed attempts to understand how to call that method in that particular context then I assume that the reference has been ignored. Similarly, I expect that when I say "create a TypeX object" that the person will either create the object or read the documentation for the type to see how an instance can be created.

Does it come as a surprise to you to hear that some people (= everybody I've spoke to who doesn't work for Microsoft) finds the MSDN documentation very obscure, and so they learn how to use VB, C# etc largely by trial and error, and by adapting examples that are close to what they want to do?

No, not at all. I've heard that numerous times but I'm afraid that I have little sympathy. I found the MSDN Library somewhat confusing at first too. I didn't just assume that it would be too hard every time and not read it though. When I first started posting on forums, I was able to answer numerous questions on subjects with which I had no prior experience, simply by reading the relevant documentation, which the poster obviously hadn't bothered to do.

If you've read the documentation but don't understand what you found then give some indication because I can only go by what I read.

As for posting code, I've posted code examples more times than you could poke a stick at and often had people copy and paste it with no understanding of what it does and still come back asking more questions. It creates issues as many times as it solves them.

 

[robertb_NZ]

Sorry if I was a bit grumpy before. I'd had very little sleep, and I simply couldn't understand what you were trying to tell me. You were trying to tell me some things from the Microsoft perspective, others were telling me things from the IBM perspective, it's as if Microsoft speak only French, IBM speak only German and my job is to put both together but I only speak English.

The current fault which I'm trying to remedy is that the file ends up as a block of text ignoring line breaks. This is actually only test #1, once this is sorted I need to go on to work out how to submit jobs, get output, and so on. To do this I seem to need to be able to give FTP commands like ?quote site filetype=jes?.

I can transfer files to/from the mainframe perfectly if I use a TN3270 emulator, log on to the mainframe, and use the emulator's "Transfer" function to upload (or download) the file, but there?s zillions (well, over 20!) of manual steps to get the point that you?ve run your ZOS job and are looking at the output. I could reduce this to a single click in my application if I can figure out how to do all of this.

I?ve found papers that show me exactly how to do what I want, butfrom OS2, from Java, and from a Windows command line, but nothing from .NET even though I'd Googled ?.NET FTP Submit Job MVS?.

Where/how do I give these commands from my VB.NET program? Clearly not as a value of FTPWebRequest.Method: MSDN says
?You set the Method by using the strings defined in the public field members of the WebRequestMethods.Ftp class. Note that the strings defined in the WebRequestMethods.Ftp class are the only supported options for the Method property. Setting the Method property to any other value will result in an ArgumentException exception.?

I went through the MSDN documentation for FTPWebRequest, but found nothing in Properties or Methods. So the only option seems to be to put these commands into the input file, but I tried this and they?re treated as part of the input. IBM?s documentation says: -
Before transferring files between your local host and a remote host, or using any other FTP functions, you must enter the FTP environment. Enter the FTP environment by doing one of the following:
? Code PGM=FTP in a batch job and pass parameters using the PARM keyword. See Submitting FTP requests in batch for more information.
? Enter the FTP command from TSO.
? Enter the FTP command from the z/OS? UNIX? shell.
? Pass the FTP command parameters to the FTP Client API. See FTP Client API information in the z/OS Communications Server: IP Programmer's Guide and Reference for complete details on the FTP Client API
Only the last of these could be relevant, but I?m clearly not using IBM?s FTP Client API. Am I?

So I'm still stuck. I don't know how to get around the problem of no line breaks, nor to give commands to the remote FTP system beyond those defined in WebRequestMethods.Ftp.

 

[jmcilhinney]

I?ve found papers that show me exactly how to do what I want, butfrom OS2, from Java, and from a Windows command line, but nothing from .NET even though I'd Googled ?.NET FTP Submit Job MVS?.

This might turn out to be the key. You can execute a commandline in .NET by calling Process.Start. If you need to execute multiple commands within the same process then you can execute cmd.exe, redirect the standard input stream and send commands to the console as though you were typing them. You can also redirect the standard output stream to get the responses. If that sounds like something that might be of use, check this out:

Automate Command Prompt Window (CMD), Redirect Output to Application

 

[robertb_NZ]

I am continuing to be frustrated with this.

We've managed to use System.Net.FtpWebRequest to send a file to the mainframe, converting it to EBCDIC. However it arrived as a block of code, not as lines, and was thus unusable. Also, we couldn't find a way to use commands like "QUOTE SITE FILETYPE=JES" which are available through TFTP from the command line.

I therefore tried to automate sending commands with the code in Automate Command Prompt Window (CMD), Redirect Output to Application [2003/2005]-VBForums. However this doesn't seem to work with FTP, or indeed any command that changes the directory (like "CD .."), although it works fine with the example command, "DIR".

I have been able to use FTP perfectly by entering these commands into a command prompt. (Items like <IP> are my values, not literally "<IP>"): -
FTP
OPEN <ip>
<Userid>
<Password>
PUT <local file> <Destination file>
Quit

And
OPEN <ip>
<Userid>
<Password>
QUOTE SITE FILETYPE=JES
PUT <local file>
Quit

So I'm considering which strategy to follow to automate this: -
A/ Continue trying to automate a command sequence, building my own version of the 381405-Automate-Command-Prompt-Window code.
B/ Follow the approach of How To Write Pluggable Protocol to Support FTP in Managed Classes by Using Visual Basic .NET, and derive my own equivalent of FTPWebRequest from WebRequest.

Can anybody please advise me about this? I feel that strategy B would be better as it would result in properly managed .NET code and properly handle all the errors like timeout, wrong destination etc, but it's annoying that I can't download VbFtpClient.EXE that the KB paper is supposed to link to. Either approach is going to require significant learning of new concepts for me, so I would like to choose a strategy that will work and give me a good result. I have spent too long on this problem already.

 

[jimmajsterski]

I may be way off base here, but - back in the late 90s, I was doing file transfers ( a few hundred ) from a Honeywell-Bull mainframe to a PC/server, then from PC to an IBM mainframe. I believe the host file transfer programs handled any conversions; I did not need to do any. Maybe you can get help from IBM ??

 

[robertb_NZ]

If I do the FTP using TFTP (which I assume stands for "TelNet FTP") everything is handled properly, without my having to worry about code conversion. Objects leave my Windows environment as ASCII, arrive in zOS as EBCDIC, and come back as ASCII, with line breaks handled correctly. Unfortunately so far I have only made this work from the Control Prompt by entering the FTP commands. To achieve what I want I need to automate this command sequence, and the "proper" way to do this from VB.NET is presumably to use a class like System.Net.FtpWebRequest. This handles the FTP protocol and seems to make everything easy, but it doesn't quite work properly. (See previous posts). So my current challenge is to find the right approach that will give me a robust solution that does what I want.

I've been seeking help from both IBM and through this forum. The challenge with IBM don't understand MS - their answers relate to Java or OS2 - while MS don't understand IBM. Not really surprising, but it creates challenges for me. MS are usually a lot more helpful, but neither side's answers are 100% for somebody like me.

 

[robertb_NZ]

This topic is being closed (forum admin - how do I do this) as this failed to find an answer. I've reframed the questions as
http://www.vbdotnetforums.com/vb-ne...ing-job-submission-post167805.html#post167805

 

[robertb_NZ]

Progress! Add this statement
Request.UseBinary = False
and you don't need to worry about code conversion or line breaks, it all happens automatically. Now, if only I can figure out how to handle job submission/output, for which I used the the commands: -
QUOTE SITE FILETYPE=YES
PUT <filename>
when running FTP from the Command Prompt. QUOTE SITE allows you to send a site-specific FTP command to the FTP Server. The PUT is unusual in that you don't need to name a destination.