cURL is a powerful command-line tool for transferring data with URLs. It supports numerous protocols including HTTP, HTTPS, FTP, and many more. Let’s explore 20 essential tips that will help you become proficient with cURL.
Table of Contents
1. Basic GET Request
The simplest way to use cURL is to fetch a webpage:
curl https://example.com
Use case: Quickly check if a website is up or view its HTML source.
Imagine you’re building a monitoring script and need to verify if your company website is responding. A simple curl command gives you an immediate answer!
Get Your Free Linux training!
Join our free Linux training and discover the power of open-source technology. Enhance your skills and boost your career! Start Learning Linux today - Free!2. Save Output to a File
Instead of displaying the output on the screen, save it to a file:
curl -o output.html https://example.com
Use case: Download files or save API responses for later analysis.
Think of this like taking notes in class – sometimes you want to review the material later rather than just seeing it once. This is perfect when you need to analyze API responses or save reference documentation.
3. Follow Redirects
Many websites redirect to different URLs. Use -L to follow these redirects:
curl -L https://github.com
Use case: Access websites that use redirects for authentication or content distribution.
This is like following signs when you enter a building – sometimes you need to go through multiple doors to reach your destination! Many modern websites will redirect you from HTTP to HTTPS or to different domains.
4. Display Request Headers
To see what’s happening behind the scenes:
curl -v https://example.com
Use case: Debugging connection issues or understanding how a server responds.
Think of this as looking under the hood of a car. When something isn’t working right, the verbose output shows you all the communication details between you and the server – extremely helpful for troubleshooting!
5. Send Custom Headers
Add your own headers to the request:
curl -H "User-Agent: My Browser" -H "Accept-Language: en-US" https://example.com
Use case: Setting authentication tokens or mimicking browser behavior.
This is like wearing the right ID badge to access different areas of a building. Many APIs require specific headers for authentication, and websites might show different content based on your User-Agent.
6. POST Requests
Send data to a server:
curl -X POST -d "name=John&age=25" https://example.com/submit
Use case: Submitting forms or sending data to APIs.
Just like filling out a paper form and submitting it, POST requests let you send information to a server. This is essential for creating resources or submitting data through APIs.
7. POST JSON Data
For APIs that expect JSON:
curl -X POST -H "Content-Type: application/json" -d '{"name":"John","age":25}' https://api.example.com/users
Use case: Interacting with RESTful APIs that require JSON payloads.
Modern web applications often speak JSON – think of this as knowing the right language for the conversation. When you’re creating a new user account via an API, you’ll often need to format your data as JSON.
8. Upload Files
Send files to a server:
curl -F "[email protected]" https://example.com/upload
Use case: Uploading images or documents to web services.
Just like attaching a file to an email, this lets you send files to a server. Perfect for uploading profile pictures, documents, or any file-based content to web applications.
9. Use Authentication
Basic authentication:
curl -u username:password https://example.com/secure
Use case: Accessing password-protected resources.
Think of this as using your library card to check out a book. Many APIs and websites require authentication, and this is the simplest way to provide your credentials.
10. Download Files with Progress Bar
See the download progress:
curl -# -O https://example.com/largefile.zip
Use case: Tracking progress when downloading large files.
Nobody likes staring at a blank screen wondering if things are working! The progress bar is like watching the status of your food delivery – it reassures you that your download is progressing and gives you an estimate of completion time.
11. Limit Bandwidth
Control how much bandwidth cURL uses:
curl --limit-rate 100K https://example.com/largefile.zip -O
Use case: Avoiding network saturation when downloading large files.
This is like driving at a controlled speed so you don’t use all the gas at once. When downloading large files during work hours, limiting bandwidth ensures you don’t slow down everyone else’s internet connection.
12. Use Cookies
Send cookies with your request:
curl -b "session=abc123" https://example.com/account
Use case: Maintaining sessions when interacting with websites.
Cookies are like your wristband at an amusement park – they tell the website who you are across multiple requests. This is crucial when accessing protected pages that require login sessions.
13. Save and Load Cookies
Save cookies from responses and use them in subsequent requests:
curl -c cookies.txt https://example.com/login
curl -b cookies.txt https://example.com/account
Use case: Automating login processes for web scraping.
This is like storing your concert ticket so you can re-enter the venue. First, you log in and save the session cookies, then you can access protected resources by presenting those same cookies in future requests.
14. Set Timeouts
Avoid hanging on slow connections:
curl --connect-timeout 10 --max-time 30 https://slow-server.example.com
Use case: Making your scripts more robust against network issues.
This is like deciding how long you’ll wait for a friend before giving up and going home. Setting timeouts prevents your scripts from hanging indefinitely when servers don’t respond properly.
15. Retry Failed Attempts
Automatically retry if the connection fails:
curl --retry 5 https://unstable-server.example.com
Use case: Dealing with unreliable network conditions or services.
Sometimes servers are like busy restaurants – if you can’t get a table the first time, try again! This option is perfect for unreliable connections or temporarily overloaded servers.
16. Send PUT/DELETE Requests
For RESTful API operations:
curl -X PUT -d '{"status":"active"}' https://api.example.com/users/123
curl -X DELETE https://api.example.com/users/123
Use case: Complete CRUD operations on REST APIs.
These are like the update and erase buttons on your document editor. PUT typically updates existing resources, while DELETE removes them – essential operations when working with modern APIs.
17. Use Proxy Servers
Route requests through a proxy:
curl -x http://proxy.example.com:8080 https://target-site.com
Use case: Testing geo-restricted content or enhancing privacy.
A proxy is like having a friend in another country mail you something you can’t get locally. This is useful for accessing region-restricted content or routing requests through corporate networks.
18. Ignore SSL Certificate Errors
For testing against self-signed certificates:
curl -k https://self-signed.example.com
Use case: Development environments with invalid SSL certificates.
Think of this as temporarily ignoring a “Staff Only” sign when you’re actually working with the staff. In development environments, you often use self-signed certificates that curl would normally reject.
19. Download Multiple Files
Download several files at once:
curl -O https://example.com/file1.txt -O https://example.com/file2.txt
Use case: Batch downloading resources for offline use.
This is like grabbing multiple handouts in one trip to the front of the classroom. When you need to download several related files, this saves you from writing multiple commands.
20. Create a Config File
Store your favorite options in a config file:
# In ~/.curlrc
user-agent = "My Custom User Agent"
connect-timeout = 30
max-time = 60
Use case: Standardizing your curl environment across sessions.
This is like setting up your desk just the way you like it. A config file saves your preferred settings so you don’t have to specify them every time you run a curl command.
Conclusion
cURL is an incredibly versatile tool that can handle almost any HTTP request scenario. These 20 tips should give you a solid foundation, but cURL has many more options for specific use cases. As you become more comfortable with these basics, explore curl --help or man curl to discover additional capabilities.
Remember, practice is key to mastering cURL. Try these examples in your terminal and experiment with different options to see how they work in real situations. Just like learning any new skill, start with the basics and gradually take on more complex tasks as your confidence grows!