Screen scraping a page behind HTTPS login page

[leedo2k]

Hi,

Can anyone help here please. I would like to authenticate myself to a website which uses HTTPS (Tomcat Server) and then after authentication, I would like to get the contents of the webpage. I bypassed the SSL certificate by using the servicepoint callback and validated the certificate.

I can do that without a problem for normal HTTP sites. But every time I try my code I get the log in page again and never get passed it. It is worth mentioning that when I try the URL I want, the server redirects my HttpWebRequest to the log in page.

Here is my code:

Imports System.IO 
Imports System.Net 
Imports System.Net.Security 
Imports System.Security.Cryptography.X509Certificates 
 
Public Class Form1 
 
    Private Function ValidateCertificate(ByVal sender As Object, ByVal certificate As X509Certificate, ByVal chain As X509Chain, ByVal sslPolicyErrors As SslPolicyErrors) As Boolean 
        If certificate.Issuer.Contains("OU=CISCO") Then 
            Return True 
        Else 
            Return False 
        End If 
    End Function 
 
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click 
 
        ServicePointManager.ServerCertificateValidationCallback = New RemoteCertificateValidationCallback(AddressOf ValidateCertificate) 
 
        Dim Req As HttpWebRequest 
        'Dim SourceStream As System.IO.Stream 
        Dim Response As HttpWebResponse 
  
        '-- get file 
        Dim URL2 As String = "https://mysite.com:8443/filelist/library.htm?o_id=2" 
        Req = HttpWebRequest.Create(URL2) 
        Req.Credentials = New NetworkCredential("admin", "password") 
        Req.AllowAutoRedirect = False 
        Response = Req.GetResponse() 
        Dim sr2 As IO.StreamReader = New IO.StreamReader(Req.GetResponse.GetResponseStream) 
        '-- check responser, and make sure you are logged in 
        Dim html_data As String = sr2.ReadToEnd 
        Console.WriteLine(sr2.ReadToEnd) 
        Response.Close() 
 
    End Sub 
 
 
  
End Class

 

[JohnH]

There could be cookies set that need to be transferred to future requests, see for example http://www.vbdotnetforums.com/net-sockets/28191-httpwebrequest-cookies.html