bypass a websites "enter" screen?

mushimushi84

Member
Joined
Jan 22, 2010
Messages
10
Programming Experience
1-3
Ok so ive been looking everywhere to try and bypass a websites "enter" screen using the POST method of webrequest(). I've successfully done this and recieve the HTML of the page after the login page. Problem is now i want to automatically do another webrequest on this page entering GET paramaters equal to doing a 'search' on the page. I figure i need cookies for this to happen which i think im doing correctly, but the second web request just boots me back to the original login screen... does this imply the cookies are not being sent, or they are wrong somehow?? I've been trying to get this to work all day... heres my code so far...

' Set the initial parameters
Dim domain As String = "......."
Dim encoding As New System.Text.ASCIIEncoding
Dim CookieC As New Net.CookieContainer

' Use the appropriate HTML field names to stuff into the post header
Dim PostData As String = "user_choice=Enter(or whatever postdata)"

Dim Data() As Byte = encoding.GetBytes(PostData)

' Initialise the request
Dim LoginReq As Net.HttpWebRequest = Net.WebRequest.Create(domain) ' Login location taken from the form action
With LoginReq
.KeepAlive = False
.Method = "POST"
' Note: if the page uses a redirect if will fail
.AllowAutoRedirect = False
.ContentType = "application/x-www-form-urlencoded"
.ContentLength = Data.Length
' Set empty container
.CookieContainer = CookieC
End With

' Add the POST data
Dim SendReq As IO.Stream = LoginReq.GetRequestStream
SendReq.Write(Data, 0, Data.Length)
SendReq.Close()

' Obtain the response
Dim LoginRes As Net.HttpWebResponse = LoginReq.GetResponse()
Dim sReader1 As IO.StreamReader = New IO.StreamReader(LoginRes.GetResponseStream)
Dim HTML1 As String = sReader1.ReadToEnd

' Retreive the headers from the request (e.g. the location header)


Dim Headers As WebHeaderCollection = LoginRes.Headers()
Dim Redirect As String = LoginRes.Headers("Set-Cookie")

' Add any returned cookies to the cookie collection
Dim tempCookies As Cookie
For Each tempCookies In LoginRes.Cookies
CookieC.Add(tempCookies)

Next

'CookieC.Add(LoginRes.Cookies)

' Move to the redirected page as a GET request...
LoginReq = Net.WebRequest.Create(domain & Redirect)
With LoginReq
'.KeepAlive = False
'.Method = "GET"
'.ContentType = "application/x-www-form-urlencoded"
.AllowAutoRedirect = False
.CookieContainer = CookieC


End With

' Perform the navigate and output the HTML
LoginRes = LoginReq.GetResponse()
Dim sReader As IO.StreamReader = New IO.StreamReader(LoginRes.GetResponseStream)
Dim HTML As String = sReader.ReadToEnd


Like I said, everything WORKS; I get no errors. BUT the second webrequest is returned with the HTML script of the login page, not the validated page.

I hope this makes sense


any help would be mucho awesome :)
 
Have you tried literally handling cookies exactly as I explained in that thread?

Using Fiddler Web Debugger - A free web debugging tool or similar tool to look into exactly what requests is sent when you login using a regular browser may also help you.
 
HEY! Thanks for the speedy reply!

Thanks for that amazing tool! now i can see all the cookies requested/responded with... ok so i figure i can manually set all of the headers/cookies for the second request. But how do i do that? all of the examples i have found are in c#, which i haven't really looked into.

I don't really know the difference between headers and cookies. From what i can gather cookies are a part of ONE header. If the second request needs all of the headers, not just the cookie collection i am adding, how do i add those headers to the request?

for each x in webheaderscollection
webreq.headers.add(NAME,VALUE)???
next

how do i retrieve each of these name value pairs from the first request?

If someone could post some code that would help alot!
 
Ok so I'm Trying to muck about with this again this morning. I'm trying to literally imply the headers like you said using the values im picking up in my browser from "Fiddler" but am getting an exception thrown at me...


'''''''
With LoginReq
'.Headers.Add(HttpRequestHeader.Cookie, Redirect)
.Headers.Add(HttpRequestHeader.Accept, value:="*/*")
.Headers.Add(HttpRequestHeader.AcceptEncoding, value:="gzip, deflate")
.Headers.Add(HttpRequestHeader.AcceptLanguage, value:="en-us")
.Headers.Add(HttpRequestHeader.UserAgent, value:="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; Hotbar 11.0.78.0)")
.Headers.Add(HttpRequestHeader.Connection, value:="keepalive")
.Headers.Add(HttpRequestHeader.Host, value:="http://x.com")

End With

''''

the exception:
This header must be modified using the appropriate property. Parameter name: name

any ideas?
 
Ok so IM TRYING TO F***IN understand this stuff...

after reading what you said in the previous thread and the MSDN library....

Looking at the request/response headers off fidler for the site im trying to log into i DONT UNDERSTAND how the response headers after the POST in the login page DONT MATCH AT ALL to the request headers in the next GET request to the website... how does the browser take these response headers(Some are the same sid for example) but there are many other cookie values not give in the Response headers of the previous request... so WTFFFF do they come from???

sorry im getting frustrated with this..

also looking at your past forum you stated

CookieJar = New CookieContainer

req1.create
req1.CookieContainer = CookieJar
req1.getresponse

req2.create
req2.CookieContainer = CookieJar
req2.getresponse

i tried this but it didnt work... aren't you just setting the second cookiecontainer to nothing this way?? what i mean is you are first setting CookieJar to the first REQUEST'S headers, not its RESPONSE headers, how do you know which one the website will need from here on??? Wouldnt you set CookieJar to the webheaderscollection recieved from the getresponse event??


As far as the headers go the MSDN library's examples

Dim myHttpWebResponse As HttpWebResponse = CType(myHttpWebRequest.GetResponse(), HttpWebResponse)
Console.WriteLine(ControlChars.Cr + "The HttpHeaders are " + ControlChars.Cr + ControlChars.Cr + ControlChars.Tab + "Name" + ControlChars.Tab + ControlChars.Tab + "Value" + ControlChars.Cr + "{0}", myHttpWebRequest.Headers)

I have no idea how to use this information.. i don't want to Console.Writeline, how do i convert this into headers i can then reuse in the nex webrequest?? Can you PLEASE just post some example code that relates to mine to get me on the right track!!?? id really appreciate it
 
The point being that regarding cookies you only need to set the object for first request, then use the same container for subsequent requests, it 'catches on' the cookies the response want you to set automatically. If you don't believe me you can try it for yourself, here is a Cookie Test test page that wants to set a "test cookie" and next page verifies it. Watch the requests and responses in Fiddler. Now do the same in code:
VB.NET:
Dim jar As New Net.CookieContainer
Dim req As Net.HttpWebRequest = CType(Net.WebRequest.Create("http://www.tempesttech.com/cookies/cookietest1.asp"), Net.HttpWebRequest)
req.CookieContainer = jar
req.GetResponse.Close() '(I don't bother reading the response here)
req = CType(Net.WebRequest.Create("http://www.tempesttech.com/cookies/cookietest2.asp"), Net.HttpWebRequest)
req.CookieContainer = jar
Dim res As Net.HttpWebResponse = CType(req.GetResponse, Net.HttpWebResponse)
Using reader As New IO.StreamReader(res.GetResponseStream)
    MsgBox(reader.ReadToEnd)
End Using
res.Close()
Three proofs are presented here; (1) you get the correct response "Cookies are enabled and functioning!". If cookies were not handled correctly the message would be "Cookies are not enabled!", (2) you can look into the cookies that exists in the jar after the request has been submitted, there is one cookie named "TestCookie" with value "Test" present, and (3) you can see the requests and responses made with Fiddler. Still not convinced? ;)

About the other headers, I thought what was explained in that article I linked to was rather simple, but apparently for you it is challenging. Let me highlight this part from the Remarks section:
help said:
The Headers collection contains the protocol headers associated with the request. The following table lists the HTTP headers that are not stored in the Headers collection but are either set by the system or set by properties or methods.
The table then lists the headers you have set using a property or method with link to the relevant help page. The first one is the Accept header, if you want to set the Accept header you have to set it with the Accept property:
VB.NET:
req.Accept = "*/*"
And so on.
 
Lol, well i literally started coding about a month ago picked up a VB book and i love it. But yes i am a NOOOOB! Thanks for ur patience man.. ok so im really trying here!

I Tried to add as many custom headers as i could...


Ok so here is the SECOND request to the site setting the headers manually and passing along the cookiescontainer from the first request.

<code>
LoginReq = Net.WebRequest.Create("http://www.x.com/(GETSTUFF)")
With LoginReq
.CookieContainer = CookieC
.Headers = HeadersC
.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1,utf-8;q=0.7,*;q=0.7")
.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us,en;q=0.5")
.KeepAlive = "300"
.KeepAlive = "true"



.IfModifiedSince = " Mon, 25 Jan 2010 23:09:40 GMT"

End With

' Perform the navigate and output the HTML
LoginRes = LoginReq.GetResponse()
</code>



But there are some there that i do not know how to set

ie:
The _qca=""
_utm[abc]=""
etc
age_check="1"
screen_width="1280"

these seem like they should be transferred with the rest of the cookies in the CookiesContainer. Why don't they? if not how do you set them.

The Failed Response states Cache-Control: no-cache, must-revalidate, max-age=0

is this referring to the IfModifiedSince header? I tried to set it manually but I dont think this is working, how do i retrieve this specific header from the previous request?

Thanks again for your help, I hope you can help me with this... I've never spent more than a day on a coding problem, and this is more a "WHY CANT I FIGURE THIS OUT" than i really need to be able to do this, it just bugs me lol :)
I have pictures of the resulting request/response headers in Fiddler, I'm going to try and send them to your profile(dont know if you can do that) but if not send me ur email and ill send them to you so you can see what all of the requests should look like when done in a browser.

thanks again
 
Nope couldn't send them.. problem is they're too big for the forum... if you could pm me your email address ill send them to you so you can get a complete picture of what im trying to do....

THAAAAAAANKS for humoring a noob :)
 
mushimushi84 said:
But there are some there that i do not know how to set
Those values you mentioned is probably part of the request data, you have to GetRequestStream and write them in there with a StreamWriter. This also indicates that the request sent is a POST request and not a GET request. HttpWebRequest.GetRequestStream Method (System.Net)
I have pictures of the resulting request/response headers in Fiddler
Image attachments is allowed when you post in forums. When getting data from Fiddler I find it easier to use the Raw view in Inspectors, here you can copy the plain text.
 
I do use the GetRequestStream for the first request because yes, the first request is a POST. That request is successful. However the second one I don't think is (atleast not according to fiddler when i navigate to it in firefox)

The following is my code snipet
Followed by
Fiddler 1: The Response Header i get after my first SUCCESSFUL Post to the site
Fiddler 2: The Request Header my code is sending to the site (Not Successful)
Fiddler 3: The Request Header firefox is sending to the site (Obviously what mine needs to be...? - Seems to be a GET)

MyCode:
VB.NET:
   Dim domain As String = "http://www.x.com/"
        Dim encoding As New System.Text.ASCIIEncoding
        Dim CookieC As New Net.CookieContainer
        Dim HeadersC As New WebHeaderCollection
        

        ' Use the appropriate HTML field names to stuff into the post header
        Dim PostData As String = "user_choice=Enter"

        Dim Data() As Byte = encoding.GetBytes(PostData)

        ' Initialise the request
        Dim LoginReq As Net.HttpWebRequest = Net.WebRequest.Create(domain) ' Login location taken from the form action
        With LoginReq
            .KeepAlive = False
            .Method = "POST"
            .AllowAutoRedirect = False
            .ContentType = "application/x-www-form-urlencoded"
            .ContentLength = Data.Length
            ' Set empty container
            .CookieContainer = CookieC
            .Headers = HeadersC
        End With

        ' Add the POST data
        Dim SendReq As IO.Stream = LoginReq.GetRequestStream
        SendReq.Write(Data, 0, Data.Length)
        SendReq.Close()

        ' Obtain the response
        Dim LoginRes As Net.HttpWebResponse = LoginReq.GetResponse()
        Dim sReader1 As IO.StreamReader = New IO.StreamReader(LoginRes.GetResponseStream)
        Dim HTML1 As String = sReader1.ReadToEnd




        ' Move to the redirected page as a GET request...

        LoginReq = Net.WebRequest.Create("http://www.x.com/search?query=x")
        With LoginReq
            .CookieContainer = CookieC
            .Headers = HeadersC
            .Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
            .Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1,utf-8;q=0.7,*;q=0.7")
            .Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
            .Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us,en;q=0.5")
            .KeepAlive = "300"
            ' This I manually put in to some future date, but i dont think the server likes it, how do I
            ' Manually retrieve this from the first request?
            .IfModifiedSince = " Mon, 25 Jan 2010 23:09:40 GMT"

        End With

        ' Perform the navigate and output the HTML
        LoginRes = LoginReq.GetResponse()
        Dim sReader As IO.StreamReader = New IO.StreamReader(LoginRes.GetResponseStream)
        Dim HTML As String = sReader.ReadToEnd

Fiddler 1: (My First POST Request's Response Headers)
VB.NET:
HTTP/1.1 200  OK
Server: nginx
Date: Tue, 26 Jan 2010 13:47:53 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Vary: Accept-Encoding
Cache-Control: max-age=49
Expires: Tue, 26 Jan 2010 13:48:21 GMT
Last-Modified: Tue, 26 Jan 2010 13:47:21 GMT
P3P: CP="CAO PSA OUR"
Age: 21
Content-Length: 48022
Rating: RTA-5042-1996-1400-1577-RTA
Set-Cookie: sid=Q+SBi0te8om8dx2yoesvAg==; expires=Thu, 26-Jan-12 13:47:53 GMT; domain=.x.com; path=/

Fiddler 2: The Request Header my code is sending to the site (failed)
VB.NET:
GET /search?query=x&type=straight HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
If-Modified-Since: Mon, 25 Jan 2010 23:09:40 GMT
Host: www.x.com
Cookie: sid=Q+SBi0te8om8dx2yoesvAg==
Connection: Keep-Alive

The Response headers i get on my failed request say basically:

Cache-Control: no-cache, must-revalidate, max-age=0
and boots me back to the login page

Fiddler 3: The Request Header FIREFOX sends to the site (successful- and what mine needs to look like im guessing. It seems to be a GET tho..?)


VB.NET:
GET / HTTP/1.1
Host: x.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: sid=C29WSknOLnMiDG0aCIr7Ag==; __utma=60671397.1454637416.1238249077.1264457217.1264461276.28; __qca=1238249076-12595045-97184853; __utmz=60671397.1264110798.20.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); age_check=1; screen_width=1152
If-Modified-Since: Mon, 25 Jan 2010 23:15:13 GMT


Thanks for your help again man, I've never been stuck on something for so long... I know you said the cookies like : (__utma=60671397.1454637416.1238249077.1264457217.1264461276.28; )

indicate it should be a POST but it says GET.. so i dunno
 
Ok, the __"utma" etc are other cookies (actually Google Analytics trackers) and not POST data as it seemed to be when you posted this info. These must have been present in your Firefox from other sessions or intermediate requests, the response from your first .Net request only asks to have have the 'sid' ('session id' presumably) cookie set. What you basically have to do is to reset your Firefox cookies (Tools, Delete history, make sure cookies are checked) and make a fresh manual login then inspect all page requests made and the reponses for these. Page requests are usually type text/html and you can see the url like index.php and login.php and similar. Normally when you fill in a form in a webpage with username/password and press login a POST request is made, the response then give the cookies needed to be set to identify the user for later requests.
 
FINALLY! SOLVED!

so thought i'd reply to help anyone who read through this debacle


I was actually just making the whole thing too complicated than it needed to be:

Never having done a webrequest before I learned how to post data, and to make further requests to the server you must forward the cookies headers etc to the second POST request.

Because my second request was a get, all i had to do was put the get data (http://x.com/q=""(GETINFO) with the FIRST POST request, responding HTML was returned with the search results... ugh so stupid

anyway thanks alot JohnH, appreciate the help
 
If you meant querystring, then that is a possibility, but it is rare login systems expose credentials in the request url.
 
Back
Top