Learn Python Series (#35) - Working with APIs Part 2: Beyond GET Requests

Repository

What will I learn?

You will learn the conceptual difference between GET, POST, PUT, PATCH, and DELETE requests;
how APIs use different HTTP verbs to represent different operations on the same resource;
how to think about pagination as a pattern for handling large datasets;
why rate limiting exists and how to work with it instead of against it;
the fundamental concept behind connection pooling.

Requirements

A working modern computer running macOS, Windows or Ubuntu;
An installed Python 3(.11+) distribution;
Completion of Learn Python Series #34 or familiarity with basic HTTP requests;
The ambition to learn Python programming.

Difficulty

Intermediate

Curriculum (of the `Learn Python Series`):

GitHub Account

https://github.com/realScipio

Learn Python Series (#35) - Working with APIs Part 2: Beyond GET Requests

In episode #34, we covered the basics of making GET requests to read data from APIs using the modern httpx library. That's perfect for web crawling (like we did in episodes #13-17), but real applications need to do more than just read. They need to create, update, and delete data as well.

Nota bene: The original Web Crawler mini-project only used GET requests because crawling is fundamentally about reading web pages. But when you're building applications that interact with APIs (managing users, posting content, updating records), you'll need the full set of HTTP operations.

The REST philosophy: resources and verbs

Before diving into the code, let's talk about how modern APIs are designed. Most APIs today follow a pattern called REST (Representational State Transfer). The core idea is simple: everything is a resource, and you perform operations on those resources using HTTP verbs.

Think of a resource as a "thing" your application cares about. A user is a resource. A blog post is a resource. An order in a webshop is a resource. Each resource lives at a specific URL (its address), and you tell the server what you want to do with that resource by using different HTTP verbs.

The standard verbs are:

GET: read the resource (what we've been doing)
POST: create a new resource
PUT: replace an entire resource
PATCH: update part of a resource
DELETE: remove a resource

Here's the pattern: same URL, different verb, different meaning. Let's say you have a user resource at https://api.example.com/users/42. Here's what each verb means:

import httpx

url = "https://api.example.com/users/42"

user_data = httpx.get(url)

This reads the user with ID 42. The server sends you back the current state of that user.

Now let's update that user's email address. We're changing part of the resource, so we use PATCH:

changes = {"email": "[email protected]"}
httpx.patch(url, json=changes)

Same URL. Different verb. The server now knows you want to modify the resource instead of reading it.

Nota bene: you might wonder why there are both PUT and PATCH for updates. PUT means "replace the entire resource with this new version" (you send all fields). PATCH means "just change these specific fields" (you send only what's changing). In practice, modern APIs mostly use PATCH because it's more efficient.

Creating new resources with POST

When you create something new, you don't yet have an ID for it. So instead of POSTing to /users/42, you POST to the collection endpoint /users:

new_user = {"name": "Scipio", "email": "[email protected]"}
response = httpx.post("https://api.example.com/users", json=new_user)

The server creates the user, assigns it an ID, and sends back the created user (usually with its new ID included):

created = response.json()
print(created["id"])

Notice the pattern: GET and DELETE act on a specific resource (/users/42), while POST acts on the collection (/users) to add a new item to it.

Deleting resources

Deleting follows the same URL pattern as GET:

httpx.delete("https://api.example.com/users/42")

Nota bene: many APIs don't actually delete data immediately. Instead they mark it as deleted (a "soft delete"), because completely removing data can break historical records or reports. The HTTP response will tell you if the deletion succeeded, usually with a 204 No Content status code.

Pagination: handling large datasets

Now let's talk about a problem you'll hit almost immediately when working with real APIs: what happens when a collection contains thousands of items?

Imagine you're fetching a list of blog posts from an API:

posts = httpx.get("https://api.example.com/posts").json()

If the blog has 50,000 posts, should the server send all 50,000 in one response? Absolutely not! That would be massive (megabytes of JSON), slow to transfer, and would consume tons of memory when you parse it.

Instead, APIs use pagination: they send you a "page" of results (say, 20 items), and tell you how to get the next page. There are different pagination styles, but the most common one looks like this:

response = httpx.get("https://api.example.com/posts?page=1&per_page=20")
data = response.json()

posts = data["items"]
total_pages = data["total_pages"]

The API sends back the first 20 posts, plus metadata telling you how many pages exist in total. To get page 2, you request it explicitly:

page2 = httpx.get("https://api.example.com/posts?page=2&per_page=20")

Some APIs use cursor-based pagination instead, where each response includes a "cursor" (a token) pointing to where the next page starts:

response = httpx.get("https://api.example.com/posts?limit=20")
data = response.json()

posts = data["items"]
next_cursor = data["next_cursor"]

Then you request the next page using that cursor:

next_page = httpx.get(f"https://api.example.com/posts?limit=20&cursor={next_cursor}")

Cursor-based pagination is more reliable for data that changes frequently, because it doesn't break if items are added or removed while you're paging through the results.

Nota bene: always check the API documentation to see which pagination style it uses. There's no universal standard, unfortunately!

Rate limiting: playing nice with the server

Almost every public API has rate limits: restrictions on how many requests you can make per minute (or hour, or day). Why? Because servers cost money to run, and if everyone could make unlimited requests, a single misbehaving script could bring down the entire service.

A typical rate limit might be "1000 requests per hour". The API usually tells you about limits in the response headers:

response = httpx.get("https://api.example.com/data")

limit = response.headers.get("X-RateLimit-Limit")
remaining = response.headers.get("X-RateLimit-Remaining")
reset_time = response.headers.get("X-RateLimit-Reset")

print(f"Limit: {limit}, Remaining: {remaining}, Resets at: {reset_time}")

The exact header names vary by API (some use X-RateLimit-*, others use RateLimit-*), but the concept is always the same: the server tells you how many requests you have left and when your limit resets.

If you exceed the limit, you'll get a 429 Too Many Requests response. The professional way to handle this is to wait before retrying:

if response.status_code == 429:
    retry_after = response.headers.get("Retry-After")
    print(f"Rate limited! Wait {retry_after} seconds")

The Retry-After header tells you how long to wait before trying again.

Nota bene: a good approach is to track your remaining quota and slow down before you hit the limit, rather than waiting for the 429 error. Your scripts become better citizens of the API ecosystem that way!

Connection pooling: reusing connections

Let's talk about performance. Every time you make an HTTP request, your computer needs to:

Open a TCP connection to the server
Perform the TLS handshake (for HTTPS)
Send your request
Receive the response
Close the connection

Steps 1 and 2 take time. If you're making hundreds of requests to the same server, opening a new connection for each one is wasteful. This is where connection pooling comes in.

The httpx library handles this automatically when you use a Client object:

client = httpx.Client()

response1 = client.get("https://api.example.com/data/1")
response2 = client.get("https://api.example.com/data/2")
response3 = client.get("https://api.example.com/data/3")

client.close()

With this pattern, httpx opens one connection to api.example.com and reuses it for all three requests. Much faster!

Even better, use the client as a context manager so it closes automatically:

with httpx.Client() as client:
    for i in range(100):
        response = client.get(f"https://api.example.com/data/{i}")

The connection pool handles everything behind the scenes. You write simpler code, and it runs faster.

Nota bene: connection pooling also works with async code using httpx.AsyncClient, which we'll cover in a future episode about concurrent requests.

What did we learn, hopefully?

In this episode we covered the conceptual foundations of working with modern APIs beyond simple GET requests:

How REST APIs use HTTP verbs (GET, POST, PUT, PATCH, DELETE) to represent operations on resources
Why the same URL can mean different things depending on which verb you use
How pagination solves the problem of large datasets by breaking them into manageable chunks
Why rate limits exist and how servers communicate them via response headers
What connection pooling is and why reusing connections makes your code faster

The key insight is this: APIs aren't just about fetching data. They're about managing resources over HTTP. Once you understand that pattern, every API you encounter will make more sense, because they're all variations on the same core ideas.

We've now covered both the basics (episode #34) and the deeper concepts of working with APIs. From here, the series continues into more advanced Python territory — stay tuned!

Thank you for your time!

@scipio

Sort:

Trending

[-]

femdev (65) 2 days ago

Scippie!!! I got your Discord DM about you posting on Hive again, love the tutorials! Connection pooling with httpx is something I actually use at work a lot, so nice to see you cover that here.

You also got me enthusiastic to (re-)start something like this on my own! Would you mind me doing that? I was thinking a similar format but on Zig, Lua, and JavaScript — languages I work with a lot these days. Could be fun. :-)

$0.01

1 vote

scipio (65) 2 days ago

Hi! Sure go ahead! Sounds like fun!
Just make sure you outpace me on my tutorial productivity ;-)

(...and don't forget to keep reminding yourself to keep blogging, that's what I kept forgetting myself as well, setting reminders now so I'll keep pushing it, hop e that works !)

Xxx

$0.00

femdev (65) yesterday

Zeg allez .. een paar wijntjes later op de bank en ik heb ineens het perfecte plan: een creative coding tutorial serie om mee te beginnen. Helemaal mijn goesting, da gaat 'm worden!

Sallukes! (en trusse eh)
x

scipio (65) yesterday

Oew da's een leuk idee! Ga ervoor Char! ;-)

stemsocial (64) yesterday

Thanks for your contribution to the STEMsocial community. Feel free to join us on discord to get to know the rest of us!

Please consider delegating to the @stemsocial account (85% of the curation rewards are returned).

Consider setting @stemsocial as a beneficiary of this post's rewards if you would like to support the community and contribute to its mission of promoting science and education on Hive.

Thanks!

Learn Python Series (#35) - Working with APIs Part 2: Beyond GET Requests

Learn Python Series (#35) - Working with APIs Part 2: Beyond GET Requests

Repository

What will I learn?

Requirements

Difficulty

Curriculum (of the Learn Python Series):

GitHub Account

Learn Python Series (#35) - Working with APIs Part 2: Beyond GET Requests

The REST philosophy: resources and verbs

Creating new resources with POST

Deleting resources

Pagination: handling large datasets

Rate limiting: playing nice with the server

Connection pooling: reusing connections

What did we learn, hopefully?

Thank you for your time!

Curriculum (of the `Learn Python Series`):