Submit Search

UploadLoginSignup

mastering the curl command line.pdf

Sep 1, 2023•
3 likes•952 views

D
DanielStenberg7
Follow

Technology

Daniel Stemberg's presentation on how curl works. From the basic command line
use, to URLs, options, curl basics into HTTP specifics. YouTube:
https://youtu.be/V5vZWHP-RqU?si=IkGJdHqvguYLffeG
Read more
mastering the curl command line.pdf
4 of 154
Download Now
Save slide
Recommended
How to Prepare For a Successful Job Search for 2024 by

How to Prepare For a Successful Job Search for 2024Albert Qian

1.2K views•37 slides
Social Media Marketing Trends 2024 // The Global Indie Insights by

Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media
Age(ncy)
2.7K views•96 slides
Trends In Paid Search: Navigating The Digital Landscape In 2024 by

Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
1.8K views•31 slides
5 Public speaking tips from TED - Visualized summary by

5 Public speaking tips from TED - Visualized summarySpeakerHub

2.7K views•16 slides
ChatGPT and the Future of Work - Clark Boyd by

ChatGPT and the Future of Work - Clark Boyd Clark Boyd

55K views•69 slides
Getting into the tech field. what next by

Getting into the tech field. what next Tessa Mero

13.4K views•22 slides

More Related Content

Slideshows for you (20)
PROBLEM SOLVING POWERPOINT by

PROBLEM SOLVING POWERPOINT

Andrew Schwartz•289.4K views
Business intelligence ppt by
Business intelligence ppt
sujithkylm007•157.5K views
Enterprise Resource Planning(ERP) by

Enterprise Resource Planning(ERP)

Makan Singh•48.9K views
Business process re engineering by

Business process re engineering

Student Placement Coordinator,N.S.E.C Kolkata.•56.6K views
Problem Solving PowerPoint PPT Content Modern Sample by

Problem Solving PowerPoint PPT Content Modern Sample

Andrew Schwartz•88.8K views
Decision Support System - Management Information System by
Decision Support System - Management Information System
Nijaz N•77.9K views
Design thinking by

Design thinking
Manik Choudhary •5.2K views
Problem solving& Decision Making by

Problem solving& Decision Making

Sahar Andrade, MBBCh Diversity, Leadership, Reinvent Yourself•70.7K views
DECISION MAKING POWERPOINT by

DECISION MAKING POWERPOINT

Andrew Schwartz•297.6K views
Logical design vs physical design by
Logical design vs physical design
Md. Mahedi Mahfuj•136.2K views
system analysis and design Chap009 by

system analysis and design Chap009

Nderitu Muriithi•3.8K views
Problem Solving by

Problem Solving
Miles Berry•4.3K views
System Development Life Cycle (SDLC) by

System Development Life Cycle (SDLC)

fentrekin•96.3K views
CREATIVITY POWERPOINT by
CREATIVITY POWERPOINT
Andrew Schwartz•127.1K views
Information Systems Development and Acquisition by

Information Systems Development and Acquisition

Yonathan Hadiputra•4.2K views
Management Information System (Full Notes) by

Management Information System (Full Notes)

Harish Chand•176.8K views
Chapter 8 system analysis and design by

Chapter 8 system analysis and design

Pratik Gupta•7.3K views
Effective decision making by
Effective decision making
Madhusoothanan Venkataraman•29.3K views
Systems Analysis And Design Methodology And Supporting Processes by

Systems Analysis And Design Methodology And Supporting Processes

Alan McSweeney•5.2K views
Emerging Technologies Final Exam by

Emerging Technologies Final Exam

Stephanie Gobble•1.1K views
Recently uploaded (20)
Is the UK Cloud Market a Ponzi Scheme.pdf by

Is the UK Cloud Market a Ponzi Scheme.pdf

Jasper Colin•8 views
The (un)surprising truth about DevOps Culture, by Manuel Pais by
The (un)surprising truth about DevOps Culture, by Manuel Pais
Agile Connect®•11 views
Data_Structure_and_Algorithms_Using_C++ _ Nho Vĩnh Share.pdf by

Data_Structure_and_Algorithms_Using_C++ _ Nho Vĩnh Share.pdf

Nho Vĩnh•40 views
Molded Breakers.docx by

Molded Breakers.docx
Pitiakin•10 views
OpenText Cybersecurity Tabletop Exercise by

OpenText Cybersecurity Tabletop Exercise

Marc St-Pierre•8 views
Turning Your Volunteers Into Donors: Insights for Multi-Chapter Nonprofits by
Turning Your Volunteers Into Donors: Insights for Multi-Chapter Nonprofits
Bloomerang•26 views
Content-adaptive Video Coding for HTTP Adaptive Streaming by

Content-adaptive Video Coding for HTTP Adaptive Streaming

Alpen-Adria-Universität•26 views
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data by

Cloud Revolution: Exploring the New Wave of Serverless Spatial Data

Safe Software•310 views
We are among the first people who started deploying Open source Telephony sol... by

We are among the first people who started deploying Open source Telephony
sol...
dvcom2•13 views
Presentation on Algorithms and Flowcharts by
Presentation on Algorithms and Flowcharts
RAKSHITDOGRA1•5 views
TRP-369.pdf by

TRP-369.pdf
Friendly Technologies•9 views
ADI Program Information Webinar by

ADI Program Information Webinar

The Digital Insurer•12 views
Guide To Hiring Dedicated Python Developers In India.pdf by

Guide To Hiring Dedicated Python Developers In India.pdf

AIS Technolabs Pvt Ltd•8 views
Building a serverless AI powered translation service by
Building a serverless AI powered translation service
Jimmy Dahlqvist•21 views
Impact Hackathon Job Hunt by

Impact Hackathon Job Hunt

SmritiSharan4•14 views
Microsoft Visual Studio vs Visio: 2023 Showdown by

Microsoft Visual Studio vs Visio: 2023 Showdown

Direct Deals, LLC•12 views
Ave D Case Studies Q124 by

Ave D Case Studies Q124

ChadDyar1•45 views
Webinar: SpiraTeam - A Jira Alternative to Revolutionize Your Project Management by
Webinar: SpiraTeam - A Jira Alternative to Revolutionize Your Project
Management
Inflectra•446 views
3D PRINTER technology by sultana.pptx by

3D PRINTER technology by sultana.pptx

riyasathalikhan03•6 views
Transactions Start to Finish by

Transactions Start to Finish

Bloomerang•40 views
Featured (20)
Introduction to Data Science by

Introduction to Data Science

Christy Abraham Joy•84.2K views
Time Management & Productivity - Best Practices by
Time Management & Productivity - Best Practices
Vit Horky•170.4K views
The six step guide to practical project management by

The six step guide to practical project management

MindGenius•37K views
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright... by

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __

Bright...
RachelPearson36•13.4K views
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present... by

Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look,

present...
Applitools•55.9K views
12 Ways to Increase Your Influence at Work by
12 Ways to Increase Your Influence at Work
GetSmarter•402.1K views
ChatGPT webinar slides by

ChatGPT webinar slides

Alireza Esmikhani•31.5K views
More than Just Lines on a Map: Best Practices for U.S Bike Routes by

More than Just Lines on a Map: Best Practices for U.S Bike Routes
Project for Public Spaces & National Center for Biking and Walking•7.3K views
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G... by

Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko

(Belka G...
DevGAMM Conference•3.7K views
Barbie - Brand Strategy Presentation by
Barbie - Brand Strategy Presentation
Erica Santiago•25.3K views
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well by

Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them
well
Saba Software•25.3K views
Introduction to C Programming Language by

Introduction to C Programming Language

Simplilearn•8.6K views
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr... by

The Pixar Way: 37 Quotes on Developing and Maintaining a Creative

Company (fr...
Palo Alto Software•88.5K views
9 Tips for a Work-free Vacation by
9 Tips for a Work-free Vacation
Weekdone.com•7.2K views
I Rock Therefore I Am. 20 Legendary Quotes from Prince by

I Rock Therefore I Am. 20 Legendary Quotes from Prince

Empowered Presentations•142.9K views
How to Map Your Future by

How to Map Your Future

SlideShop.com•275.3K views
Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+ Inclusive -... by

Beyond Pride: Making Digital Marketing & SEO Authentically LGBTQ+

Inclusive -...
AccuraCast•3.5K views
Read with Pride | LGBTQ+ Reads by
Read with Pride | LGBTQ+ Reads
Kayla Martin-Gant•1.1K views
Exploring ChatGPT for Effective Teaching and Learning.pptx by

Exploring ChatGPT for Effective Teaching and Learning.pptx

Stan Skrabut, Ed.D.•58.2K views
How to train your robot (with Deep Reinforcement Learning) by

How to train your robot (with Deep Reinforcement Learning)

Lucas García, PhD•42.6K views
mastering the curl command line.pdf
 1. mastering the curl command line August 31, 2023 Daniel Stenberg
 2. Daniel Stenberg @bagder https://daniel.haxx.se @mastodon.social
 3. curl.se/support.html @bagder
 4. Setup - August 31 2023 Live-streamed Expected to last multiple hours Recorded Pause when
you want to - hydrate! Lots of material never previously presented Terminal overlays on/off
@bagder
 5. @bagder Just ask! @bagder Live questions from non-visible chat
 6. mastering the curl command line ❏ The project ❏ Internet transfers ❏ Command line ❏ curl
basics ❏ TLS ❏ Proxies ❏ HTTP ❏ FTP ❏ Future @bagder
 7. The project @bagder slide 7 of 154
 8. November 1996: httpget August 1997: urlget March 1998: curl August 2000: libcurl @bagder
 9. Name client for URLs “see URL” curl URL Request Library Capable Ubiquitous Reliable
Libre @bagder
 10. Main products curl - command line tool for client-side internet transfers with URLs libcurl -
library for client-side internet transfers with URLs ★ Always and only client-side ★ An internet
transfer: upload or download or both ★ Endpoint described with a URL @bagder
 11. Open Source Everything in the curl project is open source Every discussion and decision are
held and done in the open Open source means everyone can reshare and change curl is
(essentially) MIT licensed @bagder Free Open Gratis
 12. COPYRIGHT AND PERMISSION NOTICE Copyright © 1996 - 2023, Daniel Stenberg,
<[email protected]>, and many contributors, see the THANKS file. All rights reserved. Permission
to use, copy, modify, and distribute this software for any purpose with or without fee is hereby
granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO EVENT SHALL THE AUTHORS
OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE. Except as contained in this notice, the name of a
copyright holder shall not be used in advertising or otherwise to promote the sale, use or other
dealings in this Software without prior written authorization of the copyright holder. @bagder
 13. Development curl is developed by “everyone” only Daniel works on curl full-time everyone
can provide their proposed changes as “pull requests” No paperwork required Changes are
reviewed and tested thoroughly A small team of maintainers can accept and merge changes
@bagder
 14. Releases We do releases every 8 weeks (or sooner when necessary) At 250 releases We
release what is in the master branch at the time @bagder
 15. Releases @bagder
 16. Releases We never break existing functionality @bagder
 17. Issues and pull requests Problems are submitted as issues Changes are proposed as pull
requests https://github.com/curl/curl @bagder
 18. Learn more man curl (curl --manual) https://curl.se/docs/manpage.html Everything curl curl
--help @bagder
 19. Asking for help Mailing lists command line tool questions: curl-users
https://lists.haxx.se/mailman/listinfo/curl-users library/development/debugging questions: curl-
library https://lists.haxx.se/mailman/listinfo/curl-library Web “forum”
https://github.com/curl/curl/discussions @bagder
 20. Paying for curl help [email protected] @bagder
 21. DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP,
LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS,
TELNET, TFTP, WS and WSS TLS certificates, HTTP POST, HTTP PUT, FTP upload, HTTP
form based upload, proxies (SOCKS4, SOCKS5, HTTP and HTTPS), HTTP/2, HTTP/3, cookies,
user+password authentication (Basic, Plain, Digest, CRAM-MD5, SCRAM-SHA, NTLM,
Negotiate, Kerberos, Bearer tokens and AWS Sigv4), file transfer resume, proxy tunneling, HSTS,
Alt-Svc, unix domain sockets, HTTP compression (gzip, brotli and zstd), etags, parallel transfers,
DNS-over-HTTPS and much more @bagder
 22. Runs everywhere Linux Windows (by default) macOS (by default) FreeBSD OpenBSD
VMS … @bagder
 23. 92 operating systems @bagder Syllable OS TPF Tizen Symbian Tru64 SunOS tvOS
ucLinux Genode Hurd iOS Integrity Illumos HP-UX HardenedBSD Haiku z/OS Nintendo Switch
NonStop OS NetWare MorphOS MPE/iX MS-DOS NCR MP-RAS NetBSD RISC OS Redox
ReactOS Sailfish OS SCO Unix Serenity SINIX-Z Qubes OS UnixWare WebOS vxWorks VMS
Windows UNICOS Windows CE Wii System Software AmigaOS Blackberry 10 BeOS Android
Blackberry Tablet OS AIX Cell OS Aros IRIX RTEMS Mbed Micrium macOS Mac OS 9 Linux
Lua RTOS eCOS FreeRTOS FreeBSD FreeDOS Fuchsia DragonFly BSD Cygwin Cisco IOS
OpenBSD OS/2 OS/400 Ultrix ipadOS NuttX Solaris Xbox System Chrome OS MINIX Garmin
OS QNX PlayStation Portable Plan 9 OS21 OpenStep Orbis OS z/TPF z/VM z/VSE Operating
systems known to have run curl Atari FreeMiNT DR DOS OmniOS Zephyr watchOS Xenix
DG/UX
 24. Commit authors @bagder
 25. Contributors @bagder
 26. Lines of code @bagder
 27. >20,000,000,000 installations @bagder
 28. “No big deal” @bagder
 29. Internet transfers @bagder slide 29 of 154
 30. curl does internet transfers A server is a remote machine running server software curl acts as
a client on the network curl connects to a server in order to do Internet transfers A stream of data
from or to a server Data from a server to curl, is a download Data from curl to a server, is an
upload The data can be anything: text, images, maps, code, film, sound curl does not care curl
does not know @bagder server client Upload Download

 31. @bagder Protocol Download Upload HTTP(S) ✅ ✅ GOPHER(S) ✅ ❌ FTP(S) ✅ ✅ TELNET ❗ ❗

DICT ✅ ❌ LDAP(S) ✅ ❌ FILE ✅ ✅ TFTP ✅ ✅ SCP ✅ ✅ SFTP ✅ ✅ IMAP(S) ✅ ✅ POP3(S) ✅ ❌ SMTP(S) ❌ ✅

RTSP ✅ ❌ RTMP(S) ✅ ❌ SMB(S) ✅ ✅ MQTT ✅ ✅ WS(S) ✅ ✅

 32. authenticated vs unauthenticated protocols @bagder Always use authenticated protocols
Authenticated means they use TLS or SSH HTTPS, FTPS, LDAPS, IMAPS… end with S + SCP
and SFTP Never disable server verification (--insecure) in production Unauthenticated transfers
can be eavesdropped and tampered with Without curl or the user knowing Unauthenticated
transfers are easily attacked - avoid!
 33. Command line @bagder slide 33 of 154
 34. command line options short options: -V long options: --version boolean options: --path-as-is
options with arguments: --output store.html arguments with spaces: --write-out “received %{path-
as-is}” negative boolean options: --no-path-as-is combinatorial explosion availability depends on
version availability depends on build availability depends on 3rd party libraries (and their
versions) @bagder
 35. command line options @bagder
 36. URLs by RFC 3986 scheme://user:password@host:1234/path?query#fragment @bagder RFC
3986+ really no spaces, use %20 “schemeless” means guess name and password - remember URL
encode hostname can be name, IDN name, IPv4 address or IPv6 address https://example.com/
http://日本語.tw ftp://192.168.0.1/ imap://[2a04:4e42:800::347]/
 37. URLs @bagder Anything not an option is a URL
 38. URL port numbers A port number is from 0 to 65535 Each URL scheme has a default port
that curl uses Unless another is set in the URL curl https://example.com:8080/ curl
tftp://[fdea::1]:8080/ @bagder
 39. URLs and browsers browsers, URLs and their address bars @bagder
 40. URLs and output options curl accepts any amount of URLs every downloaded URL needs a
destination - stdout or a file curl -o file1 -o file2 https://example.com/file1 https://curl.se/file2 curl
-o file1 https://example.com/file1 https://curl.se/file2 -o file2 curl -O https://example.com/file1 -O
https://curl.se/file2 curl https://example.com/file1 https://curl.se/file2 > everything --remote-name-
all automatically sets -O for all URLs @bagder
 41. query Queries are often name=value pairs separated by amperands (&) name=daniel &
tool=curl & age=old Add query parts to the URL with --url-query [content] content is (for
example) “name=value” “value” gets URL encoded to keep the URL fine name@file reads the
content from file before encoding it … and more curl https://example.com --url-query
“name=Daniel Stenberg” @bagder scheme://user:password@host:1234/path?query#fragment
 42. trurl created in the spring of 2023 parses and manipulates URLs companion tool to curl $
trurl --url https://curl.se --set host=example.com $ trurl --url https://curl.se/we/are.html --redirect
here.html $ trurl --url https://curl.se/we/../are.html --set port=8080 $ trurl --url "https://curl.se?
name=hello" --append query=search=string $ trurl "https://fake.host/search?
q=answers&user=me#frag" --json $ trurl "https://example.com?a=home&here=now&thisthen" -g
'{query:a}' https://curl.se/trurl/ @bagder
 43. URL globbing “globbing” = ranges and lists [1-100] [001-100] [a-z] [001-100:10] [a-z:2]
{one,two,three} {mon,tue,wed,thu} $ curl https://{ftp,www,test}.example.com/img[1-22].jpg -o
“hey_#2_#1.jpg” Can do 9 * 1018 iterations - per URL --globoff turns it off @bagder
 44. Parallel transfers by default, URLs are transferred serially, one by one -Z (--parallel) By
default up to 50 simultaneous Change with --parallel‐max [num] Prefer speed to multiplexing with
--parallel‐immediate Works for downloads and uploads @bagder server client Upload Download
 45. list curl options --help --help [category] --help category --help all curl --manual @bagder
 46. config file “command lines in a file” one option (plus argument) per line $HOME/.curlrc is
used by default -K [file] or --config [file] can be read from stdin can be generated (and huge)
10MB line length limit @bagder
 47. p4ssw0rds -u name:password .netrc (more soon) config files local leakage network leakage
debug log leakage @bagder
 48. progress meters Unless -s, --silent or --no-progress-meter % Total % Received % Xferd
Average Speed Time Curr. Dload Upload Total Current Left Speed 0 151M 0 38608 0 0 9406 0
4:41:43 0:00:04 4:41:39 9287
###############################################################################
# 84.6% DL% UL% Dled Uled Xfers Live Total Current Left Speed 12% -- 34.5G 0 2 2 --:--:--
0:00:09 --:--:-- 3903M -#, --progress-bar Different again when doing parallel transfers @bagder
 49. --next do everything to the left of it then reset the state continue on the other side in
perpetuity $ curl -H “header: one” https://example.com/one -H “header: two”
https://example.com/two $ curl -H “header: one” https://example.com/one --next -H “header: two”
https://example.com/two @bagder
 50. curl basics @bagder slide 50 of 154
 51. curl version --version or -V $ curl -V curl 8.2.1 (x86_64-pc-linux-gnu) libcurl/8.2.1
OpenSSL/3.0.10 zlib/1.2.13 brotli/1.0.9 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.3)
libssh2/1.11.0 nghttp2/1.55.1 librtmp/2.3 OpenLDAP/2.5.13 Release-Date: 2023-07-26 Protocols:
dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp
sftp smb smbs smtp smtps telnet tftp Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2
HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL
threadsafe TLS-SRP UnixSockets zstd @bagder
 52. verbose --verbose or -v -v and -vv is the same --trace-ascii or --trace --trace-time --trace-ids
careful before you share debug logs with others @bagder
 53. trace @bagder $ curl -d moo --trace - https://curl.se/ == Info: processing: https://curl.se/ ==
Info: Trying [2a04:4e42:e00::347]:443... == Info: Connected to curl.se (2a04:4e42:e00::347) port
443 == Info: ALPN: offers h2,http/1.1 => Send SSL data, 5 bytes (0x5) 0000: 16 03 01 02 00 .....
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1): => Send SSL data, 512 bytes (0x200)
0000: 01 00 01 fc 03 03 82 d6 6e 54 af be fa d7 91 c1 ........nT...... 0010: 92 0a 4e bf dc f7 39 a4 53
4d ee 22 18 bc c1 86 ..N...9.SM.".... 0020: 92 96 9a a1 73 88 20 ee 52 63 66 65 8d 06 45
df ....s. .Rcfe..E. … == Info: CAfile: /etc/ssl/certs/ca-certificates.crt == Info: CApath: /etc/ssl/certs
<= Recv SSL data, 5 bytes (0x5) 0000: 16 03 03 00 7a ....z == Info: TLSv1.3 (IN), TLS
handshake, Server hello (2): <= Recv SSL data, 122 bytes (0x7a) 0000: 02 00 00 76 03 03 59 27
14 18 87 2b ec 23 19 ab ...v..Y'...+.#.. 0010: 30 a0 f5 e0 97 30 89 27 44 85 c1 0a c9 d0 9d 74
0....0.'D......t … == Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 ==
Info: ALPN: server accepted h2 == Info: Server certificate: == Info: subject: CN=curl.se == Info:
start date: Jun 22 08:07:48 2023 GMT == Info: expire date: Sep 20 08:07:47 2023 GMT == Info:
subjectAltName: host "curl.se" matched cert's "curl.se" == Info: issuer: C=US; O=Let's Encrypt;
CN=R3 == Info: SSL certificate verify ok. … == Info: using HTTP/2 == Info: h2 [:method:
POST] == Info: h2 [:scheme: https] == Info: h2 [:authority: curl.se] == Info: h2 [:path: /] == Info:
h2 [user-agent: curl/8.2.1] == Info: h2 [accept: */*] == Info: h2 [content-length: 3] == Info: h2
[content-type: application/x-www-form-urlencoded] == Info: Using Stream ID: 1 => Send SSL
data, 5 bytes (0x5) 0000: 17 03 03 00 5d ....] => Send SSL data, 1 bytes (0x1) 0000: 17 . => Send
header, 137 bytes (0x89) 0000: 50 4f 53 54 20 2f 20 48 54 54 50 2f 32 0d 0a 48 POST /
HTTP/2..H 0010: 6f 73 74 3a 20 63 75 72 6c 2e 73 65 0d 0a 55 73 ost: curl.se..Us 0020: 65 72 2d
41 67 65 6e 74 3a 20 63 75 72 6c 2f 38 er-Agent: curl/8 0030: 2e 32 2e 31 0d 0a 41 63 63 65 70
74 3a 20 2a 2f .2.1..Accept: */ 0040: 2a 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 *..Content-
Lengt 0050: 68 3a 20 33 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 h: 3..Content-Ty 0060: 70 65 3a 20
61 70 70 6c 69 63 61 74 69 6f 6e 2f pe: application/ 0070: 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 72
6c 65 6e x-www-form-urlen 0080: 63 6f 64 65 64 0d 0a 0d 0a coded.... => Send data, 3 bytes
(0x3) 0000: 6d 6f 6f moo … <= Recv header, 13 bytes (0xd) 0000: 48 54 54 50 2f 32 20 32 30 30
20 0d 0a HTTP/2 200 .. <= Recv header, 22 bytes (0x16) 0000: 73 65 72 76 65 72 3a 20 6e 67 69
 6e 78 2f 31 2e server: nginx/1. 0010: 32 31 2e 31 0d 0a 21.1.. <= Recv header, 25 bytes (0x19)
0000: 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 74 65 content-type: te 0010: 78 74 2f 68 74 6d 6c
0d 0a xt/html.. <= Recv header, 29 bytes (0x1d) 0000: 78 2d 66 72 61 6d 65 2d 6f 70 74 69 6f 6e
73 3a x-frame-options: 0010: 20 53 41 4d 45 4f 52 49 47 49 4e 0d 0a SAMEORIGIN.. <= Recv
header, 46 bytes (0x2e) 0000: 6c 61 73 74 2d 6d 6f 64 69 66 69 65 64 3a 20 54 last-modified: T
0010: 68 75 2c 20 31 30 20 41 75 67 20 32 30 32 33 20 hu, 10 Aug 2023 0020: 30 32 3a 30 35 3a
30 33 20 47 4d 54 0d 0a 02:05:03 GMT.. <= Recv header, 28 bytes (0x1c) 0000: 65 74 61 67 3a
20 22 32 30 64 39 2d 36 30 32 38 etag: "20d9-6028 0010: 38 30 36 33 33 31 35 34 64 22 0d 0a
80633154d".. <= Recv header, 22 bytes (0x16) 0000: 61 63 63 65 70 74 2d 72 61 6e 67 65 73 3a
20 62 accept-ranges: b 0010: 79 74 65 73 0d 0a ytes.. <= Recv header, 27 bytes (0x1b) 0000: 63
61 63 68 65 2d 63 6f 6e 74 72 6f 6c 3a 20 6d cache-control: m 0010: 61 78 2d 61 67 65 3d 36 30
0d 0a ax-age=60.. <= Recv header, 40 bytes (0x28) 0000: 65 78 70 69 72 65 73 3a 20 54 68 75 2c
20 31 30 expires: Thu, 10 0010: 20 41 75 67 20 32 30 32 33 20 30 37 3a 30 36 3a Aug 2023
07:06: 0020: 31 34 20 47 4d 54 0d 0a 14 GMT.. … … <= Recv data, 867 bytes (0x363) 0000: 3c
21 44 4f 43 54 59 50 45 20 48 54 4d 4c 20 50 <!DOCTYPE HTML P 0010: 55 42 4c 49 43 20 22
2d 2f 2f 57 33 43 2f 2f 44 UBLIC "-//W3C//D 0020: 54 44 20 48 54 4d 4c 20 34 2e 30 31 20 54
72 61 TD HTML 4.01 Tra 0030: 6e 73 69 74 69 6f 6e 61 6c 2f 2f 45 4e 22 20 22 nsitional//EN" "
0040: 68 74 74 70 3a 2f 2f 77 77 77 2e 77 33 2e 6f 72 http://www.w3.or 0050: 67 2f 54 52 2f 68
74 6d 6c 34 2f 6c 6f 6f 73 65 g/TR/html4/loose 0060: 2e 64 74 64 22 3e 0a 3c 68 74 6d 6c 20 6c
61 6e .dtd">.<html lan 0070: 67 3d 22 65 6e 22 3e 0a 3c 68 65 61 64 3e 0a 3c g="en">.<head>.<
0080: 74 69 74 6c 65 3e 63 75 72 6c 3c 2f 74 69 74 6c title>curl</titl 0090: 65 3e 0a 3c 6d 65 74
61 20 6e 61 6d 65 3d 22 76 e>.<meta name="v 00a0: 69 65 77 70 6f 72 74 22 20 63 6f 6e 74 65
6e 74 iewport" content 00b0: 3d 22 77 69 64 74 68 3d 64 65 76 69 63 65 2d 77 ="width=device-w
00c0: 69 64 74 68 2c 20 69 6e 69 74 69 61 6c 2d 73 63 idth, initial-sc 00d0: 61 6c 65 3d 31 2e 30
22 3e 0a 3c 6d 65 74 61 20 ale=1.0">.<meta 00e0: 63 6f 6e 74 65 6e 74 3d 22 74 65 78 74 2f 68
74 content="text/ht 00f0: 6d 6c 3b 20 63 68 61 72 73 65 74 3d 55 54 46 2d ml; charset=UTF-
0100: 38 22 20 68 74 74 70 2d 65 71 75 69 76 3d 22 43 8" http-equiv="C 0110: 6f 6e 74 65 6e 74
2d 54 79 70 65 22 3e 0a 3c 6c ontent-Type">.<l 0120: 69 6e 6b 20 72 65 6c 3d 22 73 74 79 6c 65
73 68 ink rel="stylesh curl -d moo --trace - https://curl.se/
 54. --write-out outputs text, information and HTTP headers after a transfer is completed curl -w
"formatted string" http://example.com/ curl -w @filename http://example.com/ curl -w @-
http://example.com/ Information from over 50 “variables” "Type: %{content_type}nCode: %
{response_code}n" Show HTTP response header contents "Server: %header{server}nDate:
%header{date}n" @bagder
 55. persistent connections connections are kept “alive” repeated transfers to the same host try to
reuse the connection reused connections is a key to speedier transfers reuse is done per scheme +
host name + port - not IP address all connections close when curl exits @bagder
 56. persistent connections illustrated @bagder curl.se client example.com $ curl
https://example.com/file1 https://curl.se/file1 https://example.com/file2 http://curl.se/file2 1 2 3 4
 57. downloads download to a file named by the URL: -O (--remote-name) use the Content-
Disposition name from the server: --remote‐header‐name danger! shell redirect works: curl
https://curl.se > output.txt curl https://curl.se https://example.com > output.txt maximum file size
accepted: --max‐filesize <bytes> file size often not known ahead of time! --output-dir saves the -O
data in another directory --create-dirs is useful in a combination @bagder
 58. retry If a transient error is returned when curl tries to perform a transfer Do a few retries: --
retry [num] Retry for this long: --retry‐max‐time <seconds> Wait this long between retries: --
retry‐delay <seconds> Consider “connection refused” to be transient: --retry‐connrefused
Consider all errors transient: --retry‐all-errors @bagder
 59. uploads upload is data sent to the server curl -T file $URL if URL has no file name part,
appends the -T name to the URL curl -T file ftp://example.com/path/ HTTP offers several
different uploads @bagder server client Upload Download
 60. transfer controls stop slow transfers --speed‐limit <speed> --speed‐time <seconds> transfer
rate limiting curl --limit-rate 100K https://example.com no more than this number of transfer starts
per time unit curl --rate 2/s https://example.com/[1-20].jpg curl --rate 3/h https://example.com/[1-
20].html curl --rate 14/m https://example.com/day/[1-365]/fun.html @bagder
 61. naming tricks Provide a name + port => address mapping curl --resolve
example.com:443:127.0.0.1 https://example.com/ curl --resolve example.com:80:
 [2a04:4e42:200::347] http://example.com/ Provide a name + port => name + port mapping curl --
connect-to example.com:80:curl.se:8080 http://example.com/ Talking HTTP, it is also sometimes
fun/useful to set Host: header: curl -H “host: curl.fake” http://example.com/ When TLS is used,
this might fail certificate checks @bagder
 62. connection race curl uses both IPv6 and IPv4 when possible - and races them against each
other “Happy eyeballs” Restrict to a fixed IP version with --ipv4 or --ipv6 @bagder curl.se client
curl.se has address 151.101.129.91 curl.se has address 151.101.193.91 curl.se has address
151.101.1.91 curl.se has address 151.101.65.91 curl.se has IPv6 address 2a04:4e42:800::347
curl.se has IPv6 address 2a04:4e42:a00::347 curl.se has IPv6 address 2a04:4e42:c00::347 curl.se
has IPv6 address 2a04:4e42:e00::347 curl.se has IPv6 address 2a04:4e42::347 curl.se has IPv6
address 2a04:4e42:200::347 curl.se has IPv6 address 2a04:4e42:400::347 curl.se has IPv6 address
2a04:4e42:600::347 DNS 2a04:4e42:800::347 2a04:4e42:a00::347 2a04:4e42:c00::347
2a04:4e42:e00::347 2a04:4e42::347 2a04:4e42:200::347 2a04:4e42:400::347 2a04:4e42:600::347
151.101.129.91 151.101.193.91 151.101.1.91 151.101.65.91 IPv6 IPv4
 63. connections Use a specific network interface curl --interface enp3s0 https://example.com
local port number range curl --local-port 1000-3000 https://example.com TCP keep alive curl --
keepalive‐time 23 https://example.com DNS servers (when c-ares is used) curl --dns-ipv4-addr
10.1.2.3 https://example.com @bagder
 64. timeouts Maximum total time allowed to spend curl --max-time 12.34 https://curl.se/ Never
spend more than this time to connect: curl --connect-timeout 3.14 https://remote.example.com/
“Connect time” implies DNS and everything else before transfer starts @bagder
 65. .netrc a file for users to store their credentials for remote FTP servers $HOME/.netrc since
1978 --netrc makes curl use it --netrc‐file [file] to use another file for all protocols beware: weakly
specified $ cat $HOME/.netrc machine example.com login daniel password qwerty @bagder
 66. exit status the numerical value curl returns back to the shell/prompt zero for success conveys
the reason for errors can be tested for in shell scripts #!/bin/sh curl https://example.com/page.html
-O res=$? if [ $res -ne 0 ]; then echo “curl command failed with $res” fi @bagder $ curl -o save
https://example.com/ … curl: (23) Failure writing output to destination $ echo $? 23
 67. exit status @bagder $ man curl … 0 Success. The operation completed successfully
according to the instructions. 1 Unsupported protocol. This build of curl has no support for this
protocol. 2 Failed to initialize. 3 URL malformed. The syntax was not correct. 4 A feature or
option that was needed to perform the desired request was not enabled or was explicitly disabled
at build-time. To make curl able to do this, you probably need another build of libcurl. 5 Could not
resolve proxy. The given proxy host could not be resolved. 6 Could not resolve host. The given
remote host could not be resolved. 7 Failed to connect to host. 8 Weird server reply. The server
sent data curl could not parse. 9 FTP access denied. The server denied login or denied access to
the particu‐ lar resource or directory you wanted to reach. Most often you tried to change to a
directory that does not exist on the server. 10 FTP accept failed. While waiting for the server to
connect back when an ac‐ tive FTP session is used, an error code was sent over the control
connection or similar. 11 FTP weird PASS reply. Curl could not parse the reply sent to the PASS
re‐ quest. 12 During an active FTP session while waiting for the server to connect back to curl, the
timeout expired. …
 68. Avoid --insecure SCP and SFTP SSH-based instead of TLS curl sftp://example.com/file.zip
-u user curl scp://example.com/file.zip -u user curl sftp://example.com/ -u user curl
sftp://example.com/~/todo.txt -u daniel ~/.ssh/known_hosts curl sftp://example.com -u user --
insecure @bagder
 69. Reading email POP3 / IMAP Reading is download curl pop3://example.com/ curl
imap://example.com/ usually with TLS (more on this a few slides later) @bagder server client
Upload Download
 70. Sending email SMTP Sending is uploading curl -T data smtp://example.com/ -u
user:password The file needs to have all the mail headers (To, From, Subject, …) The file needs to
be “correctly” formatted use this with TLS (more on this a few slides later) @bagder
 71. MQTT Subscribe to the bedroom temperature in the subject: curl
mqtt://example.com/home/bedroom/temp Set the kitchen dimmer: curl -d 75
mqtt://example.com/home/kitchen/dimmer @bagder
 72. TFTP Download a file from the TFTP server curl -O tftp://localserver/file.boot Upload a file
to your TFTP server curl -T file.boot tftp://localserver/ @bagder
 73. TELNET An odd child in the curl family Session, not really download/upload curl
telnet://example.com:80 Reads input on stdin @bagder
 74. DICT Dictionary lookups (downloads) curl dict://dict.org/m:curl curl
dict://dict.org/d:heisenbug:jargon curl dict://dict.org/d:daniel:gcide @bagder
 75. WebSocket Still experimental in 8.2.1 Still not too fancy in the curl tool curl
wss://example.com/wss @bagder
 76. curl vs browsers “I downloaded HTML and it looks different” Web Browsers supports
different charsets and fonts Web Browsers do (a lot of) JavaScript Servers might provide different
content depending on client Servers might act differently: different cookies, different redirects
Some servers try very hard to identify clients (fingerprinting) @bagder
 77. copy as curl Generates a curl command line that mimics a previous transfer Firefox, Chrome,
Edge, Safari all feature this other tools do as well Never perfect, use as starting point Live demo
@bagder
 78. figure out the browsers Additional techniques to mimic browsers: wireshark +
SSLKEYLOGFILE (more on this soon) nc -l -p 8080 (next slide) @bagder
 79. h2c - headers to curl command line nc -l -p 8080 Make your browser go to
http://localhost:8080/… See what the request looks like https://curl.se/h2c/ Demo @bagder
 80. --libcurl Generate libcurl source code from a command line Gets a bootstrap skeleton for
your program Generates C code but often easy enough to convert Demo @bagder
 81. TLS @bagder slide 81 of 154
 82. enable TLS also still referred to as SSL necessary for transport security (unless SSH is used)
TLS verifies the peer prevents eavesdropping and tampering typically indicated by the URL
scheme (HTTPS://) all schemes curl supports that end with S do TLS --ssl‐reqd for FTP IMAP
POP3 SMTP LDAP SCP and SFTP use SSH instead of TLS, similar but different @bagder Use
TLS!
 83. TLS versions curl uses latest version/suitable ciphers automatically Offers: --tlsv1.0 --tlsv1.1
--tlsv1.2 --tlsv1.3 (lowest version accepted) --tls‐max <VERSION> (highest version accepted) --
sslv2 --sslv3 don’t work anymore due to security problems @bagder
 84. verifying server certificates verifying certs is key to security CAs sign certificates curl trusts
CAs intermediate certificates CA store as file or “native” --cacert --insecure (-k)
https://curl.se/docs/caextract.html @bagder Avoid -k
 85. OCSP stapling certificate revocation is problematic a standard for checking the revocation
status curl --cert-status https://example.com/ @bagder
 86. client certificates often called “mutual authentication” prove to the server that the client is
“legitimate” curl --cert mycert:mypassword https://example.com curl --cert mycert:mypassword --
key mykey https://example.com @bagder
 87. ciphers TLS negotiates what ciphers to use --ciphers --tls13‐ciphers --proxy‐ciphers Post
Quantum https://curl.se/docs/ssl-ciphers.html @bagder AES128-SHA256 AES256-SHA256
AES128-GCM-SHA256 AES256-GCM-SHA384 DH-RSA-AES128-SHA256 DH-RSA-AES256-
SHA256 DH-RSA-AES128-GCM-SHA256 DH-RSA-AES256-GCM-SHA384 DH-DSS-
AES128-SHA256 DH-DSS-AES256-SHA256 DH-DSS-AES128-GCM-SHA256 DH-DSS-
AES256-GCM-SHA384 DHE-RSA-AES128-SHA256 DHE-RSA-AES256-SHA256 DHE-RSA-
AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-DSS-AES128-SHA256 DHE-
DSS-AES256-SHA256 DHE-DSS-AES128-GCM-SHA256 DHE-DSS-AES256-GCM-SHA384
ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES128-GCM-
SHA256 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-SHA256 ECDHE-
ECDSA-AES256-SHA384 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES256-
GCM-SHA384 ADH-AES128-SHA256 ADH-AES256-SHA256 ADH-AES128-GCM-SHA256
ADH-AES256-GCM-SHA384 AES128-CCM AES256-CCM DHE-RSA-AES128-CCM DHE-
RSA-AES256-CCM AES128-CCM8 AES256-CCM8 DHE-RSA-AES128-CCM8 DHE-RSA-
AES256-CCM8 ECDHE-ECDSA-AES128-CCM ECDHE-ECDSA-AES256-CCM ECDHE-
ECDSA-AES128-CCM8 ECDHE-ECDSA-AES256-CCM8 TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256
TLS_AES_128_CCM_8_SHA256 TLS_AES_128_CCM_SHA256
 88. TLS backends curl can get built with different TLS backends different backends might.. -
behave slightly different - have different CA stores - have different feature sets - work on different
platforms OpenSSL is the most commonly used TLS library for curl @bagder BearSSL AWS-LC
 GnuTLS mbedSSL OpenSSL Schannel wolfSSL Secure Transport rustls BoringSSL libressl
AmiSSL
 89. SSLKEYLOGFILE TLS transfers are encrypted encrypted transfers can’t be snooped upon
unless we can extract the secrets in run-time set the environment variable named
SSLKEYLOGFILE to a file name tell wireshark to read secrets from that file name then run curl
as normal also works with browsers @bagder
 90. SSLKEYLOGFILE @bagder
 91. SSLKEYLOGFILE @bagder
 92. Proxies @bagder slide 92 of 154
 93. a proxy is an intermediary a server application that acts as an intermediary between a client
requesting a resource and the server providing that resource proxy website Network A Network B
client @bagder
 94. discover your proxy a proxy is a type + host + port number curl does not auto-detect it check
your browser network settings check your system network settings environment variables like
http_proxy (more on this soon) @bagder
 95. PAC - Proxy Auto-Configuration JavaScript that determines which proxy to use for a given
URL curl does not support PAC real-life PACs are all from simple to super complicated when too
complicated for manual inspection: check behavior @bagder
 96. Captive portals Not actual proxies Network equipment normally blocking the whole
machine They also affect curl When unlocked, curl typically also works through @bagder
 97. Proxy types There are different proxy types HTTP vs SOCKS HTTP / HTTPS / HTTPS-H2 /
SOCKS4 / SOCKS4a / SOCKS5 / SOCKS5h use type as URL scheme when set with --proxy --
preproxy allows both SOCKS and HTTP(S) proxy TOR is SOCKS forward proxy and reverse
proxy @bagder
 98. HTTPS proxy HTTPS to the proxy, anything over it HTTPS authenticates the proxy: 🮱 curl
supports HTTP/1 and HTTP/2 over HTTPS proxy Prevents users on network A from
eavesdropping Features its own set of TLS options --proxy-*, like --proxy-insecure and --proxy-
tlsv1.3 proxy website Network A HTTPS Network B client @bagder
 99. MITM proxy Sometimes used for debugging Sometimes used for surveillance The proxy
terminates TLS and can snoop on data MITM proxies present a certificate from a CA normally not
trusted curl must be told to trust that CA to allow this stunt blindly trusting a middleman is a
recipe for disasters @bagder Avoid proxy website Network A Network B client
 100. Proxy authentication Proxies might require authentication curl supports numerous proxy
auth methods -U, --proxy-user <user:password> --socks5‐basic --socks5‐gssapi --proxy-basic --
proxy-digest --proxy-negotiate --proxy-ntlm @bagder
 101. Proxy environment variables [scheme]_proxy curl https://example.com/ HTTPS_PROXY
curl ftp://example.com/ FTP_PROXY curl http://example.com/ http_proxy ALL_PROXY
NO_PROXY @bagder
 102. Proxy headers --proxy-header vs --header proxy website Network A Network B client
@bagder
 103. HTTP @bagder slide 103 of 154
 104. HTTP protocol basics HTTP/1 - HTTP/2 - HTTP/3 HTTPS is HTTP + TLS Name resolve
+ TCP+TLS/QUIC Request - Response Method - verbs GET /index.html HTTP/1.1 User-agent:
curl/2000 Host: example.com HTTP/1.1 200 OK Server: example-server/1.1 Content-Length: 5
Content-Type: plain/text hello POST /index HTTP/1.1 Host: example.com User-agent: curl/2000
Content-Length: 5 hello HEAD /index.html HTTP/1.1 User-agent: curl/2000 Host: example.com
PUT /index HTTP/1.1 Host: example.com User-agent: curl/2000 Content-Length: 5 hello
@bagder
 105. HTTP Method Sometimes called “verb” GET HEAD (-I) POST (-d or -F) PUT (-T)
anything methods are sometimes abused -X [whatever] Avoid -X @bagder
 106. HTTP headers in terminal bold header names “linkified” Location: URLs @bagder $ curl -I
curl.se HTTP/1.1 301 Moved Permanently Connection: close Content-Length: 0 Server: Varnish
Retry-After: 0 Location: https://curl.se/ Accept-Ranges: bytes Date: Fri, 25 Aug 2023 18:33:12
GMT Via: 1.1 varnish X-Served-By: cache-qpg1235-QPG X-Cache: HIT X-Cache-Hits: 0 X-
Timer: S1692988392.312085,VS0,VE0 alt-svc: h3=":443";ma=86400,h3-
29=":443";ma=86400,h3-27=":443";ma=86400
 107. HTTP response code HTTP “success” 200 OK 404 File not found curl does not care -f
makes curl care -w ‘%{response_code}’ HTTP/1.1 200 OK Server: example-server/1.1 Content-
Length: 5 Content-Type: plain/text hello @bagder
 108. HTTP response headers -v or -I to see them or -i describes the body or the transfer save
them with -D [filename] looks like HTTP/1 even when other HTTP versions are used HTTP/1.1
200 OK Server: example-server/1.1 Content-Length: 5 Content-Type: plain/text hello @bagder
 109. HTTP response bodies The “payload” Content-Length HTTP/1.1 chunked-encoding
HTTP/2 and HTTP/3 have distinct end of data markers --compressed store to file with -o [file] or -
O HTTP/1.1 200 OK Server: example-server/1.1 Content-Length: 5 Content-Type: plain/text hello
@bagder
 110. HTTP authentication web logins are often done with cookies, we get to those later if HTTP
based, returns 401 for server auth needed returns 407 for proxy auth WWW-Authenticate: Basic,
Digest, NTLM, Negotiate, etc -u [user]:[password] --anyauth --basic, --digest, --ntlm, --negotiate
@bagder
 111. HTTP ranges Ask for a piece of a remote resource curl --range 500-999
https://example.com/file.txt server may ignore the ask curl’s -C [where] resumes the transfer at
that index @bagder
 112. HTTP versions curl supports HTTP/0.9, HTTP/1.0, HTTP/1.1, HTTP/2 and HTTP/3
Generally: you don’t need to care Different over the wire, made to look similar for users
HTTP/0.9 must be enabled with --http0.9 HTTP/1.0 with --http1.0 HTTP/1.1 is a general default
or --http1.1 HTTP/2 is default over HTTPS, or asked for with --http2 HTTP/3 is experimental,
asked for with --http3 @bagder
 113. HTTP versions What version does the server support? @bagder curl -sI https://curl.se
-o/dev/null -w '%{http_version}n' curl -sI --http3 https://curl.se -o/dev/null -w '%{http_version}n'
 114. HTTP time based conditions do the transfer only if… The remote resource is newer: curl --
time-cond “Wed 01 Sep 2021 12:18:00” https://example.com/file The remote resource is older:
curl --time-cond “-Wed 01 Sep 2021 12:18:00” https://example.com/file Newer than the local file:
curl --time-cond file https://example.com/file Set the remote date on the local file: curl -R -O
https://example.com/file @bagder
 115. HTTP etags do the transfer only if… The remote resource is “different” curl --etag-save
remember -O https://example.com/file curl --etag-compare remember -O https://example.com/file
Both can be used at once for convenient updates: curl --etag-save remember --etag-compare
remember -O https://example.com/file @bagder
 116. HTTPS HTTPS is HTTP with added TLS for security HTTPS:// instead of HTTP:// should
be the only difference curl negotiates the latest TLS version + suitable cipher HTTPS:// tries
HTTP/2 by default, HTTP:// does HTTP/1.1 HTTP/3 will only be done for HTTPS:// (more on
this soon) @bagder
 117. HTTP POST: simple pass any data to a HTTP(S) server curl -d 'name=admin&shoesize=12'
https://example.com/ curl -d name=admin -d shoesize=12 https://example.com/ curl -d @filename
http://example.com curl --data-raw '@string' https://example.com curl --data-binary @filename
https://example.com @bagder
 118. HTTP POST: content-type -d / –data defaults to Content-Type: application/x-www-form-
urlencoded curl -d dust -H 'Content-Type: stuff/dream' https://example.com @bagder
 119. HTTP POST: JSON curl --json ‘{“name”: “daniel”}’ https://example.com curl --json
@object.json https://example.com Sets Content-Type: application/json and Accept:
application/json Create JSON easily jo -p name=jo n=17 parser=false | curl --json @-
https://example.com/ Receive/parson JSON easily curl --json '{"tool": "curl"}'
https://example.com/ | jq curl + jo + jq jo -p name=jo n=17 | curl --json @- https://example.com/ |
jq @bagder
 120. HTTP POST: URL encoding --data-urlencode helps URL encode data to send curl --data-
urlencode "name=John Doe (Junior)" http://example.com sends name=John%20Doe
%20%28Junior%29 -data-urlencode [content] where content is… anything URL encode the
content =anything URL encode the content (leave out the ‘=’) any=thing Send as “any=[URL
encoded thing]” @anything Read content from file, URL encode and use any@thing Send as
“any=[URL encoded file contents]” @bagder
 121. HTTP POST: convert to GET (query) curl -d name=admin -d shoesize=12
https://example.com/ but do it as a GET instead curl -d name=admin -d shoesize=12
 https://example.com/ --get GET /?name=admin&shoesize=12 HTTP/1.1 User-agent: curl/8.2.1
Host: example.com we recommend using --url-query instead! @bagder
 122. HTTP POST: Expect 100-continue For HTTP/1.1 only A request header used by curl when
POST or PUT > 1 megabyte Meant to avoid sending lots of data if server does not want it Server

responding 100 means: 👍 - go ahead If it bothers you, disable with curl -H Expect:
https://example.com/ Commonly ignored by servers, leading to wasted waiting time @bagder
 123. HTTP POST: chunked Sends data without specifying the size up front For HTTP/1.1 only
curl -H "Transfer-Encoding: chunked" -d @file http://example.com @bagder
 124. HTTP POST: <form> The HTML <form> tag is “filled in” with a POST type=hidden fields
as well -d name1=var1 -d name2=var2 -d name3=var3 … The action=[here] identifies where to
send the POST “Copy as curl” is your friend @bagder
 125. HTTP multipart formpost This is a POST sending data in a special multipart format
Content-Type multipart/form-data The data is sent as a series of “parts”, one or more Each part
has a name, separate headers, file name and more Each part is separated by a “mime boundary”
@bagder
 126. HTTP multipart formpost curl -F [content] adds one part per instance Use as many -F as
you like Insert plain text content -F “name=Daniel Stenberg” Insert content from a file -F
name=<file.txt Insert a file as an “upload” -F [email protected] Insert a file, different file name -F
[email protected];filename=fake.jpg Set a custom content-type: -F
[email protected];type=image/myown <form action="submit" method="post"
enctype="multipart/form-data"> @bagder
 127. HTTP -d or -F Both sends HTTP POST Both works over every HTTP version Both can “fill
in” HTML <form> What data do you need to send? Very rarely can the client decide. Do what the
server expects! @bagder
 128. HTTP redirects The response you want is … over there! HTTP/1.1 301 Server: example-
server/1.1 Location: https://example.com/over-here.html hello A 30X response code + Location:
header tell curl to “follow” with --location (-L) curl -L curl.se curl -L curl.se --max-redirs 7 The
numeric code defines method in redirected-to request --location-trusted Avoid -X @bagder
 129. HTTP modify the request Sensible and basic by default You as a user add the bells and
whistles Modify the method with --request Add/change/remove/blank headers with --header curl -
H "curl-master: very-soon" http://example.com/ curl -H "Host: test.example" http://example.com/
curl -H "User-agent:" http://example.com/ curl -H "User-agent;" http://example.com/ @bagder
 130. HTTP modify the request The request “target” is made from the URL path + query GET
/user/profile?shoesize=12 HTTP/1.1 User-agent: curl/8.2.1 Host: example.com curl -X OPTIONS
--request-target "*" http://example.com/ Convenient shortcuts: --user-agent [string] --referer
[URL] (yes spelled wrong) Remember “copy as curl” @bagder
 131. HTTP PUT The “upload file” of HTTP (and others) curl -T localfile
https://example.com/destination/replacement curl -T -
https://example.com/destination/replacement curl -T file https://example.com curl -T "img[1-
1000].png" http://example.com/images/ curl --upload-file "{file1,file2}" https://example.com curl
-d "data to PUT" -X PUT http://example.com/new/resource/file @bagder
 132. HTTP cookies: an explainer key/value pairs that a client stores on the behalf of a server sent
back in subsequent requests only when the cookie properties match each cookie has an expiration
date - or end of “session” A session typically ends when user closes browser When running curl
command lines, when do you close the browser? @bagder
 133. HTTP cookies: send some you can tell curl to send specific cookies name + values curl -b
“name=daniel;talks=alot” https://example.com rarely what you want Often what “copy as curl”
will give you @bagder
 134. HTTP cookies: start the engine curl ignores cookies by default needs the cookie engine
enabled first specify a file to read from or use a blank string curl -b “” https://example.com More
practically combined with redirect follows curl -L -b “” https://example.com The cookie engine
keeps the cookies in memory it forgets cookies that expire only sends cookies according to the
rules @bagder
 135. HTTP cookies: cookie jar Maybe also combined with saving a cookie jar Writes the in-
memory cookies to the given file at exit curl -L -b “” -c cookies.txt https://example.com Read
from and write to the cookie jar (can be different files) curl -L -b cookies.txt -c cookies.txt
 https://example.com The cookie jar is a readable text file The cookie jar uses the “netscape cookie
format” Also includes “session cookies” because … sessions @bagder
 136. HTTP cookies: session curl does not know when a “session” ends you need to say when a
new cookie session starts --junk‐session‐cookies curl -J -b cookies.txt https://example.com
@bagder
 137. HTTP version 2 changed how data is sent over the wire curl hides those differences from
users curl tries to negotiate HTTP/2 for all HTTPS transfers with --http2 you can ask for HTTP/2
for HTTP:// transfers With HTTP/2, curl can do multiplexed transfers with -Z @bagder
 138. HTTP version 3 changed how data is sent over the wire - again HTTP/3 is done over QUIC,
a new transport protocol QUIC replaces TCP + TLS, and runs over UDP curl hides protocol
differences from users HTTP/3 is experimental in curl HTTP/3 is only for HTTPS, there is no
clear text version with --http3 you can ask curl to attempt HTTP/3 --http3 races HTTP/3 against
HTTP/1+2 and picks the winner With HTTP/3, curl can do multiplexed transfers with -Z @bagder
 139. @bagder racing curl.se client curl.se has address 151.101.129.91 curl.se has address
151.101.193.91 curl.se has address 151.101.1.91 curl.se has address 151.101.65.91 curl.se has
IPv6 address 2a04:4e42:800::347 curl.se has IPv6 address 2a04:4e42:a00::347 curl.se has IPv6
address 2a04:4e42:c00::347 curl.se has IPv6 address 2a04:4e42:e00::347 curl.se has IPv6 address
2a04:4e42::347 curl.se has IPv6 address 2a04:4e42:200::347 curl.se has IPv6 address
2a04:4e42:400::347 curl.se has IPv6 address 2a04:4e42:600::347 DNS 2a04:4e42:800::347
2a04:4e42:a00::347 2a04:4e42:c00::347 2a04:4e42:e00::347 2a04:4e42::347 2a04:4e42:200::347
2a04:4e42:400::347 2a04:4e42:600::347 151.101.129.91 151.101.193.91 151.101.1.91
151.101.65.91 h2 IPv6 h2 IPv4 h3 IPv4 h3 IPv6
 140. HTTP alt-svc server tells client: there is one or more alternatives at "another place"
(possibly using a different HTTP version) The Alt-Svc: response header Each entry has an expiry
time Only recognized over HTTPS curl can save alt-svc alternatives curl can use previously saved
alt-svc alternatives curl --alt-svc altcache.txt https://example.com/ The alt-svc cache is a text based
readable file @bagder
 141. HTTP HSTS HSTS - HTTP Strict Transport Security Lets an HTTPS server declare that
clients should automatically interact with this host name using only HTTPS going forward The
Strict-Transport-Security: response header Only recognized over HTTPS Each entry has an expiry
time curl can save HSTS data curl can use previously saved HSTS data curl --hsts hsts.txt
http://example.com/ The HSTS cache is a text based readable file @bagder
 142. FTP @bagder slide 142 of 154
 143. FTP(S) is not SFTP They are two completely different protocols, both supported by curl
@bagder
 144. FTP uses two connections With FTP, a second connection is setup for the data transfer This
second connection adds complications for firewalls and more The 2nd connection is server-to-
client (active) or client-to-server (passive) Passive is default (--ftp‐pasv) Enable active mode with
--ftp‐port (-P) @bagder ftp://curl.se client control connection passive active
 145. FTP authentication by default: user: anonymous password: [email protected] -u
user:password @bagder
 146. FTP directory listing FTP can list the contents of a remote directory curl does not know
what is a directory or not tell curl with a trailing slash curl ftp://example.com/pub/linux/ The list

format is not standardized 🙁 show only file names with --list-only (-l) curl --list-only
ftp://example.com/pub/linux/ @bagder
 147. FTP upload curl -T is for upload normally requires -u to be allowed curl -T localfile
ftp://example.com/pub/linux/newfilename curl -T localfile ftp://example.com/pub/linux/ append to
remote file curl --append -T localfile ftp://example.com/pub/linux/ Create dir on the server if
missing: curl --ftp-create-dirs -T localfile ftp://example.com/pub/linux/ @bagder
 148. FTPS is FTP with TLS add --ssl-reqd to the command line, keep ftp:// in the URL use
FTPS:// if using (rare) implicit TLS FTPS is even more problematic for stateful firewalls @bagder
 149. Future @bagder slide 149 of 154
 150. How to dig deeper curl is the accumulated results and experiences from 25 years of
improvements man curl Everything curl source code ask the community! @bagder
 151. https://everything.curl.dev/ @bagder
  152. Going next? curl is 25 years old curl has been growing and developed its entire lifetime curl
development speed is increasing the Internet does not stop or slow down protocols and new ways
of doing Internet transfers keep popping up new versions, new systems, new concepts and new
ideas keep coming there is now slowdown in sight reasonably, curl will keep develop curl will
keep expanding, get new features, get taught new things we, the community, make it do what we
think it should do you can affect what’s next for curl @bagder
 153. @bagder @bagder You can help!
 154. Daniel Stenberg @[email protected] https://daniel.haxx.se/ Thank you!
Questions? @bagder
AboutSupportTermsPrivacyCopyrightCookie PreferencesDo not sell or share my personal
informationEverand
English

© 2024 SlideShare from Scribd