GopherProxy

#[1]daniel.haxx.se » Feed [2]daniel.haxx.se » Comments Feed
[3]daniel.haxx.se » Deviating from specs Comments Feed [4]alternate
[5]alternate [6]alternate

[7]daniel.haxx.se

[8]daniel.haxx.se

[9]Search

(BUTTON) Primary Menu [10]Skip to content
* [11]About
* [12]Contact

Search for: ____________________ Search

[13]cURL and libcurl

Deviating from specs

[14]October 18, 2022 [15]Daniel Stenberg [16]1 Comment

tldr: we do not particular keep track nor document curl's exact spec
compliance. I cannot fathom how we could.

Today, in October 2022, curl and libcurl combined consist of nearly
150,000 lines of source code (not counting blank lines). 19% of those
are comments.

This source code pile was carefully crafted with the sole purpose of
performing Internet transfers using one or more of the 28 separate
supported protocols. (There are 28 different supported URL schemes, it
can be discussed if they are also 28 protocols or not.)

Which specs does curl use

It was recently proposed to me that we should document which RFCs curl
adheres to and follows, and what deviants there are. In the name of
helping the users understand what to expect from curl and educating the
world how curl will behave.

This is indeed a noble idea and a worthy goal. We do not want to
surprise users. We want them to know.

It was suggested that it might have a security impact if curl would
deviate from a spec and if this is not documented clearly, users could
be mislead.

What specs

curl speaks TCP/IP (and UDP or QUIC at times), it does DNS and
DNS-over-HTTPS, it speaks over proxies and it speaks a range of various
application protocols to perform what asked of it. There are literally
hundreds of RFCs to read to catch up on all the details.

A while ago I collected the what I consider most important RFCs to read
to figure out how curl works and why. That is right now 149
specification documents at a total of over 300,000 lines of text. (It
was not done very scientifically.)

Counting the words in these 149 documents, they add up to a total of
many more words than the entire Harry Potter series, and the Lord of
the Rings series (including the Hobbit) is far behind together with War
and Peace: 1.6 megawords.

Luckily, specs are mostly reference literature and we rarely have to
read through them all to start our journey, but we often need to go
back to check details.

Everything changes over time

The origins of curl trace back to late 1996 and it has been in constant
development since then. curl, the Internet and the specifications have
all changed significantly over these years.

The specifications that were around when we started have generally been
updated multiple times, while we struggle to maintain behavior and
functionality for our users. It is hard to spot and react to minor
changes in specification updates. They might have been done to clarify
a situation, but sometimes such a clarification ends up triggering a
functionality change in our code.

Sometimes an update to a spec is even largely ignored by fellow
protocol implementers out there in the world, and for the sake of
interoperability we too then need to adjust our interpretations so that
we work similarly to our peers.

Expectations from users change as values and terms are established in
people's minds rather than in specs. For example: what exactly is the
"URL" you see in the browser's top bar?

Over time, other tools and programs that also work on URLs and on the
Internet, gradually change as they too development and slowly morph
into the new beings we did not foresee decades ago. This change
perceptions and expectations in the user base at large.

The always changing nature of the Internet creates interoperability
challenges ever so often: out of the blue a team of protocol
implementers can decide to interpret an existing term or a passage in a
specification differently one day. When the whole world takes a turn
like that, we are sometimes forced to follow along as that is then the
new world view.

Another complication is also that curl uses (several) third party
libraries for parts of its operations, and some of those details are of
course also covered by RFCs.

Guidelines

Our primary guidelines when performing Internet transfers are:
1. Follow established standard protocol specifications
2. Security is a first-tier property
3. Interop widely
4. Maintain behavior for existing features

As you can figure out for yourself, these four bullet points often
collide with each other. Checking off all four is not always possible.
They can be hard enough on their own.

Protocol specifications

There are conflicting specifications. Specifications vary over time.
They can be hard to interpret to figure out exactly what they say one
should do.

Security

Increasing security might at the same time break existing use cases for
existing users. It might violate what the specs say. It might add
friction in the ability to interoperate with others. It might not even
be allowed according to specifications.

Interop

This often mean to not follow specifications they way we want to read
them, because apparently others do not read them the same way or
sometimes they just disregard what the specs say. At times, it is hard
to increase security levels by default because it would hamper interop
with others.

Maintain behavior

The scripts written 15 years ago that use curl should continue working.
The applications written to use libcurl can upgrade libcurl and its
Internet transfers just continue. We do not break existing established
behaviors. This may very well conflict both with interop and protocol
updates, and sometimes it is hard to tighten the security because it
would hurt a certain share of existing users.

How does curl deviate from which specs?

I consider this question more or less impossible to answer to, to
document and to keep accurate over time. At least it would be a huge
and energy-consuming effort both to get the list done but it would also
be a monster task to maintain. And it would involve a lot of gray
zones.

What is important to me is not what RFCs curl follows nor what or how
it deviates from them. I have also basically never gotten that question
from a user.

Users want reliable Internet transfers that are secure and interoperate
correctly and conveniently with other "players" out there. They want
consistent behavior and backwards compatibility.

If you use curl to perform feature X over protocol Y version Z, does it
matter which set of RFCs that this would touch and does anyone care
about the struggles we have been through when we implemented this set?
How many users can even grasp or follow the implication of mentioning
that for RFC XYX section A.B we decided to disregard a SHOULD NOT at
times?

And how on earth would we keep that up-to-date when we do bugfixes and
RFCs are updated down the line?

No one else documents this

The browsers have several hundreds of paid engineers on staff involved
and they do not provide documentation like this. Neither does any curl
alternative or competitor to my knowledge.

I don't know of any tool or software anywhere that offer such a
deviance documentation and I can perfectly understand and sympathize
with why that is so.

[17]cURL and libcurl[18]documentation[19]protocols[20]RFC

Post navigation

[21]Previous PostRewriting curl in three days[22]Next PostFunded curl
improvements

One thought on "Deviating from specs"

1.
JH says:
[23]October 23, 2022 at 19:18
> I don't know of any tool or software anywhere that offer such a
deviance documentation
The only exception I know of is compilers for C or C++, which in my
experience often do have such documentation. That being said, this
is largely enabled by there being one reasonably-well-defined spec.
[24]Reply

Leave a Reply [25]Cancel reply

Your email address will not be published. Required fields are marked *

Comment *
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________

Name * ______________________________

Email * ______________________________

Website ______________________________

Time limit is exhausted. Please reload CAPTCHA. __ × three = 15

Post Comment

D
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________
_____________________________________________

This site uses Akismet to reduce spam. [26]Learn how your comment data
is processed.

Recent Posts

* [27]thehttpworkshop2022-day3.txt November 3, 2022
* [28]Workshop season 5 episode 2 November 3, 2022
* [29]HTTP Workshop 2022 - day 1 November 2, 2022
* [30]curl 7.86.0 with WebSocket October 26, 2022
* [31]Funded curl improvements October 19, 2022
* [32]Deviating from specs October 18, 2022

Recent Comments

* [33]Fagner Brack on [34]There is a tab in my cookie
* JH on [35]Deviating from specs
* [36]Daniel Stenberg on [37]There is a tab in my cookie
* Bruno on [38]There is a tab in my cookie
* Marcus Bointon on [39]curl's TLS fingerprint
* Bartek on [40]curl's TLS fingerprint
* [41]nikitosiusis on
[42]http://http://http://@http://http://?http://#http://
* [43]Michael on
[44]http://http://http://@http://http://?http://#http://
* [45]Daniel Stenberg on
[46]http://http://http://@http://http://?http://#http://
* ilmari on [47]http://http://http://@http://http://?http://#http://

tech, open source and networking

Daniel Stenberg

Swedish open source developer and curl maintainer.

Sponsor me:[48] on GitHub
Follow me: [49]@bagder
Keep up: [50]RSS-feed

CAPTION: October 2022

M T W T F S S
1 2
3 4 5 6 7 8 9
10 11 12 13 [51]14 15 16
17 [52]18 [53]19 20 21 22 23
24 25 [54]26 27 28 29 30
31

[55]« Sep [56]Nov »

[57]Proudly powered by WordPress

References

Visible links
1. https://daniel.haxx.se/blog/feed/
2. https://daniel.haxx.se/blog/comments/feed/
3. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/feed/
4. https://daniel.haxx.se/blog/wp-json/wp/v2/posts/20422
5. https://daniel.haxx.se/blog/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fdaniel.haxx.se%2Fblog%2F2022%2F10%2F18%2Fdeviating-from-specs%2F
6. https://daniel.haxx.se/blog/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fdaniel.haxx.se%2Fblog%2F2022%2F10%2F18%2Fdeviating-from-specs%2F&format=xml
7. https://daniel.haxx.se/blog/
8. https://daniel.haxx.se/blog/
9. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/#search-container
10. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/#content
11. https://daniel.haxx.se/blog/about/
12. https://daniel.haxx.se/blog/contact/
13. https://daniel.haxx.se/blog/category/floss/curl/
14. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/
15. https://daniel.haxx.se/blog/author/daniel/
16. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/#comments
17. https://daniel.haxx.se/blog/tag/curl-and-libcurl/
18. https://daniel.haxx.se/blog/tag/documentation/
19. https://daniel.haxx.se/blog/tag/protocols/
20. https://daniel.haxx.se/blog/tag/rfc/
21. https://daniel.haxx.se/blog/2022/10/14/rewriting-curl-in-three-days/
22. https://daniel.haxx.se/blog/2022/10/19/funded-curl-improvements/
23. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/#comment-26191
24. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/?replytocom=26191#respond
25. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/#respond
26. https://akismet.com/privacy/
27. https://daniel.haxx.se/blog/2022/11/03/thehttpworkshop2022-day3-txt/
28. https://daniel.haxx.se/blog/2022/11/03/workshop-season-5-episode-2/
29. https://daniel.haxx.se/blog/2022/11/02/http-workshop-2022-day-1/
30. https://daniel.haxx.se/blog/2022/10/26/7-86-0-with-websocket/
31. https://daniel.haxx.se/blog/2022/10/19/funded-curl-improvements/
32. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/
33. https://fagnerbrack.me/
34. https://daniel.haxx.se/blog/2022/10/14/there-is-a-tab-in-my-cookie/comment-page-1/#comment-26193
35. https://daniel.haxx.se/blog/2022/10/18/deviating-from-specs/comment-page-1/#comment-26191
36. https://daniel.haxx.se/
37. https://daniel.haxx.se/blog/2022/10/14/there-is-a-tab-in-my-cookie/comment-page-1/#comment-26188
38. https://daniel.haxx.se/blog/2022/10/14/there-is-a-tab-in-my-cookie/comment-page-1/#comment-26187
39. https://daniel.haxx.se/blog/2022/09/02/curls-tls-fingerprint/comment-page-1/#comment-26083
40. https://daniel.haxx.se/blog/2022/09/02/curls-tls-fingerprint/comment-page-1/#comment-26082
41. https://nikitosi.us/
42. https://daniel.haxx.se/blog/2022/09/08/http-http-http-http-http-http-http/comment-page-1/#comment-26068
43. http://citizen428.net/
44. https://daniel.haxx.se/blog/2022/09/08/http-http-http-http-http-http-http/comment-page-1/#comment-26067
45. https://daniel.haxx.se/
46. https://daniel.haxx.se/blog/2022/09/08/http-http-http-http-http-http-http/comment-page-1/#comment-26066
47. https://daniel.haxx.se/blog/2022/09/08/http-http-http-http-http-http-http/comment-page-1/#comment-26065
48. https://github.com/users/bagder/sponsorship
49. https://twitter.com/bagder
50. https://daniel.haxx.se/blog/feed/
51. https://daniel.haxx.se/blog/2022/10/14/
52. https://daniel.haxx.se/blog/2022/10/18/
53. https://daniel.haxx.se/blog/2022/10/19/
54. https://daniel.haxx.se/blog/2022/10/26/
55. https://daniel.haxx.se/blog/2022/09/
56. https://daniel.haxx.se/blog/2022/11/
57. https://wordpress.org/

Hidden links:
59. https://daniel.haxx.se/blog/wp-content/uploads/2022/10/important-rfcs-curl.jpg
60. https://daniel.haxx.se/blog/wp-content/uploads/2022/10/rfc-words-curl.jpg