Question CURL request in VB.net

[digitaldrew]

I'm trying to send a CURL request, but appear to be having some issues with my header and hope someone can help...

curl -u 'myusername-test:1815dff0d321430378567bc84963ecd06f71d10f' 'https://api.dev.thewebsite.com/v4/domains' -X POST -H 'Content-Type: application/json' --data '{"domain":{"domainName":"example.org"},"purchasePrice":12.99}'

Here is my code:

Try
Dim myReq As HttpWebRequest
Dim myResp As HttpWebResponse
Dim myUsername As String = "myusername-test:1815dff0d321430378567bc84963ecd06f71d10f"

myReq = HttpWebRequest.Create("https://api.dev.thewebsite.com/v4/domains")

myReq.Method = "POST"
myReq.ContentType = "application/json"
myReq.Headers.Add(myUsername)
Dim myData As String = "{""domain"":{""domainName"":""example.org""},""purchasePrice"":12.99}"
myReq.GetRequestStream.Write(System.Text.Encoding.UTF8.GetBytes(myData), 0, System.Text.Encoding.UTF8.GetBytes(myData).Count)
myResp = myReq.GetResponse
Dim myreader As New System.IO.StreamReader(myResp.GetResponseStream)
WebResponse = myreader.ReadToEnd
Catch ex As Exception
txtLog.AppendText("THREAD EXCEPTION at " & TimeOfDay & " with error: " & ex.Message & vbCrLf)
End Try

This looks to me like it should work, but I keep getting an exception:
THREAD EXCEPTION at 4:47:46 AM with error: The remote server returned an error: (401) Unauthorized.

=== ADDED ===
Even changing this line:

myReq.Headers.Add(myUsername)

to something like this:

myReq.Headers.Add("myusername-test", "1815dff0d321430378567bc84963ecd06f71d10f")

Also returns the same (401) Unauthorized message.

 

[digitaldrew]

I made some changes to the code, adding the header name "Authorization" and used the "Basic" authentication scheme to add the `myUsername` to the headers. I've also modified the way the request data is written to the request stream for better readability.

Here is my new code:

Try
Dim myReq As HttpWebRequest
Dim myResp As HttpWebResponse
Dim myUsername As String = "myusername-test:1815dff0d321430378567bc84963ecd06f71d10f"

myReq = HttpWebRequest.Create("https://api.dev.thewebsite.com/v4/domains")

myReq.Method = "POST"
myReq.ContentType = "application/json"
myReq.Headers.Add("Authorization", "Basic " & Convert.ToBase64String(Encoding.UTF8.GetBytes(myUsername)))

Dim myData As String = "{""domain"":{""domainName"":""example.org""},""purchasePrice"":12.99}"
Dim dataBytes As Byte() = Encoding.UTF8.GetBytes(myData)
myReq.GetRequestStream.Write(dataBytes, 0, dataBytes.Length)

myResp = myReq.GetResponse()
Dim myreader As New System.IO.StreamReader(myResp.GetResponseStream)
Dim WebResponse As String = myreader.ReadToEnd()

Catch ex As Exception
txtLog.AppendText("THREAD EXCEPTION at " & TimeOfDay & " with error: " & ex.Message & vbCrLf)
End Try

However, this also does not work and gives the following:
THREAD EXCEPTION at 5:12:24 AM with error: The remote server returned an error: (400) Bad Request.

 

[cjard]

This approach is intensely manual, quite hard work and most would consider unacceptably fragile.

Install a nuget package called Flurl.Http and its dependent Flurl. It adds extension methods for strings and Uri objects so that you can just call a bunch of http related methods directly on the string in chained fluent syntax, then have a code like:

Dim x =  New With (.domain = New With {.domainName = "example.org", .purchasePrice = 12.99 } }  'or a proper class if you have one

Dim response = Await "https://api.dev.thewebsite.com/v4/domains" _
  .WithBasicAuth("myusername-test", "1815dff0d321430378567bc84963ecd06f71d10f") _
  .PostJsonAsync(x) _
  .ReceiveJson(Of TypeOfYourReturnHere)()

You can also ReceiveString etc, but if the service sent you back JSON like { "domain": "example.org", "owner": "John Smith" } you'd have a class XX with two props, Domain and Owner, and you'd `ReceiveJson(Of XX)` - your `response` variable would be an instance of XX ready for working with:

  Console.WriteLine("The domain is owned by " + response.OwnerName)

 

[digitaldrew]

Thanks for your reply @cjard! Is there no way to pass this without something like Flurl? Shouldn't I just be able to pass it in the web request??

Here is something else I've tried.....
1) I took my CURL request and used the curl to C# converter to convert it to C#
2) I took the C# code and used the C# to VB.net converter to convert it to VB.net
3) That gave me this:

Using httpClient = New HttpClient()
    Using request = New HttpRequestMessage(New HttpMethod("POST"), "https://api.dev.thewebsite.com/v4/domains")
        Dim base64authorization = Convert.ToBase64String(Encoding.ASCII.GetBytes("myusername-test:1815dff0d321430378567bc84963ecd06f71d10f"))
        request.Headers.TryAddWithoutValidation("Authorization", $"Basic {base64authorization}")
        request.Content = New StringContent("{""domain"":{""domainName"":""example.org""},""purchasePrice"":12.99}")
        request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json")
        Dim response = Await httpClient.SendAsync(request)
    End Using
End Using

However, this code gives a lot of errors..

I've done some re-writing, but continue to have issues...

                        Dim myUsername As String = "myusername-test:1815dff0d321430378567bc84963ecd06f71d10f"
                        Dim EncodedString As String = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(myUsername))

                        Dim httpWebRequest = CType(WebRequest.Create("https://api.dev.thewebsite.com/v4/domains"), HttpWebRequest)
                        httpWebRequest.Method = "Post"

                        httpWebRequest.Headers.Add("Authorization: Basic " & EncodedString)
                        httpWebRequest.ContentType = "application/json"
                        Using streamWriter = New StreamWriter(httpWebRequest.GetRequestStream())
                            Dim Body As String = "{""domain"":{""domainName"":""example.org""},""purchasePrice"":12.99}"
                            streamWriter.Write(Body)
                        End Using

                        Dim httpResponse = CType(httpWebRequest.GetResponse(), HttpWebResponse)
                        Using streamReader = New StreamReader(httpResponse.GetResponseStream())
                            Dim result = streamReader.ReadToEnd()
                            WebResponse = result
                        End Using

Returns a (400) Bad Request. Since this is a "Bad Request" I can't really tell if that means I'm getting authenticated okay now and the error is maybe in my Body/data?

 

[cjard]

You can, of course, do anything without helper libraries. You can do all your database code in ADO.NET from 25 years ago, without entity framework. You can write your own JSON parser and spend weeks of your life cutting up strings and parsing their contents to other primitive data types. You can even throw away vb.net entirely and write all your code in assembler from 40 years ago, and spend months of your life writing the tens of thousands of instructions necessary to draw a grid on screen

The reason why we use helper libraries is because someone else finds some task that is tedious and repetitive, like downloading some json from a web service, with all its auths and parameters and different action verbs etc, and turning it into an object that is easy to work with..
..and they write library that makes it all as simple as a single line of code. That single line of code I wrote does all those things you wrote, inside Flurl. Flurl is well written, tested and adopts best practices for doing http, including using httpclient, instead of the deprecated httpwebrequest you've chosen..

Take care not to slip into the sink cost fallacy of "I've spent three days writing hundreds of lines of code, I must be close now.. if I can spend just one more day it'll work and surely it will be good because I've spent so long on getting it there.."

It isn't necessarily the case; mostly that approach spent 3 days reinventing the wheel

 

[digitaldrew]

Thanks for taking the time to write back @cjard! I get exactly what you're saying. The main reason I wasn't trying to import the helper library is because this really seems like a rather simple JSON request. The user, one header (content type) and a small amount of data. That's partly why I'm a little confused as to how the code isn't working. I'm converting the authorization details, including the content type in the header, and when I look at my POST data it matches the example perfectly. Seems like a small and simple WebRequest should be able to do this without importing the library. But, I also know that I can sit here and re-write some code 100 different ways and have it not work each and every way I try (reinventing the wheel).

I've never used Flurl, but have used Newtonsoft.JSON before. I had been considering possibly importing that and messing with it, but couldn't remember if it even had the capabilities of doing what I'm trying to do. I might just have to give Flurl a try. I just didn't think such little JSON would require me to go that route.

Just for some context, here is a link to the documentation for the exact API I'm working with. Here is basically my most up-to-date code, although it's pretty much what I've already shared in this thread:

Dim uri As New Uri("https://api.dev.name.com/v4/domains")
Dim myUsername As String = txtUsername.Text.Trim & ":" & txtApiKey.Text.Trim
Dim data As String = "{""domain"":{""domainName"":""example.org""},""purchasePrice"":12.99}"
Dim dataByte As Byte()

request = WebRequest.Create(uri)
request.Headers.Add(HttpRequestHeader.Authorization, "Basic " & Convert.ToBase64String(Encoding.UTF8.GetBytes(myUsername)))
request.ContentType = "application/json"
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"
request.Method = "POST"

dataByte = Encoding.UTF8.GetBytes(data)
request.GetRequestStream.Write(dataByte, 0, dataByte.Length)

Dim myResp As HttpWebResponse
myResp = request.GetResponse()
Dim myreader As New System.IO.StreamReader(myResp.GetResponseStream)
WebResponse = myreader.ReadToEnd()

MsgBox(WebResponse)

Just as with the others, this code always returns a (400) Bad Request. I tried adding a UserAgent to see if that would help, but it's the same with and without it. Maybe the helper library will be my only solution here, but I think if you look at how little data there is it just seems strange this little WebRequest wouldn't work.

Thanks again!

 

[cjard]

Seems like a small and simple WebRequest should be able to do this without importing the library

But you're already using an abstraction; HttpWebRequest (which Microsoft do not recommend using for new development) is a level above the tcp socket it communicates with - tcp socket transmits any bytes you like, httpwebrequest transmits particular bytes that are specified in the http protocol. You could say "this is so simple why not just use a tcp socket and send a formatted string that is exactly the request I want, with a couple of placeholders for the bits I want to vary"..

We also know you're happy with abstractions because you used curl in the first place; arguably it's equivalent to flurl, where a single line of code declares the url, user, password, data and dumps the result. To make DOS do that on its own "because it's such a simple request" would be huge, maybe impossible. We use abstractions all the time; there is no need to shy away from them.

For another perspective on it- say you get this working, and then you need to make another, different request. You might think to extract all this functionality you've written into a helper class that takes some parameters and makes two requests, or 3, or 10. And you keep adding features until, basically, you've implemented Flurl (or RestSharp or any of the other wheels that already exist to solve this problem), you publish it to nuget, all your colleagues use it, other people use it- reuse is a great idea. So you then wrote your own library; someone who was dead set against using libraries

I don't know if I'll ever understand the desire to sway away from using third party libs. I get that there are potentially security concerns; someone else might have put malicious, code in, accidentally or deliberately, to steal all your cookies/credit card/whatever, but these are open source libraries. You can go and read the code or if you want make sure it's not going hose your computer, and millions of other people use them. You might say "I don't want to import this 50 megabyte library when all I need is this one method call" but already .net has routines to cut out unused code and ship as small an exe as possible if you so desire. Libs like flurl and RestSharp don't add a lot of bulk, so saving a few kb of disk space isn't much of a plus.. You've got more interesting things to be doing with your dev life than spending days pushing bytes down a socket, finding all the edge case flaws in your code, trying to get it working and finding out at the end of it all you had a space in the wrong place or the character case of some data variable was wrong.

If your API publishes an OAS (swagger) spec you can go to the next level and use something like AutoRest or NSwag to generate you a big block of code (programmatically writes the code you're manually writing now) that calls the api and does all this, so your life is as simple as New MyApiClient(userName,password).GetStockPrice("MSFT") - dealing with APIs is tedious; make as much use of automated, existing solutions as possible

 

[cjard]

Incidentally, I did set up an account on name.com but ran into various errors when trying the examples in the API docs; even using their cURL suggestions I got HTTP 403 a lot. The one you're trying here does work, but seems to give 400 naturally as part of its operation:

Note that I did make a small typo in the code in my initial post and set up the anonymous type a bit wrong. Here's the full code I used:

Module Program
    Sub Main(args As String())
        Dim x = New With {.domain = New With {.domainName = "example.org"}, .purchasePrice = 12.99}  'or a proper class if you have one

        Try
            Dim response = "https://api.dev.name.com/v4/domains" _
                .WithBasicAuth("cjardd-test", "xxx") _
                .PostJsonAsync(x) _
                .ReceiveJson(Of CreateDomainRoot) _
                .Result 'do not use this, use await. I only used this because I wrote a scrappy console app and VB doesn't support async main

            Console.WriteLine($"Server says OK, and order number is '{response.Order}'")

        Catch ex As AggregateException

            Dim fex = DirectCast(ex.InnerException, FlurlHttpException)
            Dim err = fex.GetResponseJsonAsync(Of TError) _
                .Result 'do not use this, use await. I only used this because I wrote a scrappy console app and VB doesn't support async main

            Console.WriteLine($"Server says there was an error with HTTP {fex.StatusCode}: '{err.Message}' because '{err.Details}'")

        End Try

    End Sub
End Module

Class TError
    <JsonProperty("message")>
    Public Property Message As String

    <JsonProperty("details")>
    Public Property Details As String
End Class

    Partial Public Class CreateDomainRoot
        <JsonProperty("domain")>
        Public Property Domain As Domain

        <JsonProperty("order")>
        Public Property Order As Long

        <JsonProperty("totalPaid")>
        Public Property TotalPaid As Double


    End Class

    Partial Public Class Domain
        <JsonProperty("domainName")>
        Public Property DomainName As String

        <JsonProperty("nameservers")>
        Public Property Nameservers As String()

        <JsonProperty("contacts")>
        Public Property Contacts As Contacts

        <JsonProperty("locked")>
        Public Property Locked As Boolean

        <JsonProperty("autorenewEnabled")>
        Public Property AutorenewEnabled As Boolean

        <JsonProperty("expireDate")>
        Public Property ExpireDate As DateTimeOffset

        <JsonProperty("createDate")>
        Public Property CreateDate As DateTimeOffset

        <JsonProperty("renewalPrice")>
        Public Property RenewalPrice As Double
    End Class

    Partial Public Class Contacts
        <JsonProperty("registrant")>
        Public Property Registrant As Admin

        <JsonProperty("admin")>
        Public Property Admin As Admin

        <JsonProperty("tech")>
        Public Property Tech As Admin

        <JsonProperty("billing")>
        Public Property Billing As Admin
    End Class

    Partial Public Class Admin
        <JsonProperty("firstName")>
        Public Property FirstName As String

        <JsonProperty("lastName")>
        Public Property LastName As String

        <JsonProperty("address1")>
        Public Property Address1 As String

        <JsonProperty("city")>
        Public Property City As String

        <JsonProperty("state")>
        Public Property State As String

        <JsonProperty("country")>
        Public Property Country As String

        <JsonProperty("phone")>
        Public Property Phone As String
    End Class

The CreateDomainRoot and related classes were generated by pasting their API example into app.quicktype.io and then pasting the C# output of that into icsharpcode's converter. Use NewtonSoft option if you use app.quicktype.io

 

[digitaldrew]

Thanks @cjard!! I'm going to open this project up tonight and try the code you suggested. Looks like you got it working on your end, so it should work on mine. Thanks for taking the time to explain everything. Very helpful!