Adding empty HTTP headers via libcurl (pycurl)

This entry was posted in Linux on by

When testing for the correct behavior of the OWASP ModSecurity Core Rule Set, a popular Web Application Firewall rule set, I needed to send empty Acccept- and User-Agent headers. This is relatively simple on the command line with curl:

$> curl --header "User-Agent;" https://example.com

Pulling this off with libcurl (pycurl in my case), was way more difficult though. It is also not really documented properly, hence this little blog post.

The tricky bit is that the Accept- and the User-Agent header are already set by default. So when you add an instruction to create an empty header, it will just add a 2nd User-Agent header (which will then be concatenated with the first one by the webserver. At least on the one I tested.)

So what you need to do is removing the existing header (suffux “:”) and then adding it anew as an empty header (suffix “;”). The following worked for me on pycurl 7.43:

import pycurl

c = pycurl.Curl()
c.setopt(pycurl.URL, "https://example.com")
c.setopt(pycurl.HTTPHEADER, ["User-Agent:"] + ["User-Agent;"])

c.perform()

Here is a dump of the network traffic documenting the request (against localhost in cleartext). This proves that this is indeed what we wanted to get.

    0x0030:  b7a9 3516 4745 5420 2f20 4854 5450 2f31  ..5.GET./.HTTP/1
    0x0040:  2e31 0d0a 486f 7374 3a20 6c6f 6361 6c68  .1..Host:.localh
    0x0050:  6f73 740d 0a41 6363 6570 743a 202a 2f2a  ost..Accept:.*/*
    0x0060:  0d0a 5573 6572 2d41 6765 6e74 3a0d 0a0d  ..User-Agent:...
    0x0070:  0a



Christian Folini / @ChrFolini on Twitter