Python Requests: A Concise Tutorial Just An Hour – FREE

Last updated on 10th Jul 2020, Blog, Tutorials

Requests library is one of the integral part of Python for making HTTP requests to a specified URL. Whether it be REST APIs or Web Scrapping, requests is must to be learned for proceeding further with these technologies. When one makes a request to a URI, it returns a response. Python requests provides inbuilt functionalities for managing both the request and response.

Requests is a popular apache2 licensed module in Python that can be used to interact with HTTP servers such as world wide web servers to download content that can be used for parsing websites or automatically posting to web forms. You can make a GET request, a POST request, passing parameters in URLs, get response content and addition of custom headers.

In this article, we’ll look at the Requests module in python and its basic operation with some examples and then finally we will conclude.

Why learn Python requests module?

• Requests is an Apache2 Licensed HTTP library, that allows to send HTTP/1.1 requests using Python.
• To play with web, Python Requests is must. Whether it be hitting APIs, downloading entire facebook pages, and much more cool stuff, one will have to make a request to the URL.
• Requests play a major role is dealing with REST APIs, and Web Scrapping.
• Checkout an Example Python Script using Requests and Web Scrapping – Implementing Web Scraping in Python with Beautiful Soup.

Making a Request

Python requests module has several built-in methods to make Http requests to specified URI using GET, POST, PUT, PATCH or HEAD requests. A Http request is meant to either retrieve data from a specified URI or to push data to a server. It works as a request-response protocol between a client and a server. Let’s demonstrate how to make a GET request to an endpoint.

Http Request Methods

Methods	Description
GET	GET method is used to retrieve information from the given server using a given URI.
POST	POST request method requests that a web server accepts the data enclosed in the body of the request message, most likely for storing it
PUT	The PUT method requests that the enclosed entity be stored under the supplied URI. If the URI refers to an already existing resource, it is modified and if the URI does not point to an existing resource, then the server can create the resource with that URI.
DELETE	The DELETE method deletes the specified resource
HEAD	The HEAD method asks for a response identical to that of a GET request, but without the response body.
PATCH	It is used for modify capabilities. The PATCH request only needs to contain the changes to the resource, not the complete resource

What can Requests do?

Requests will allow you to send HTTP/1.1 requests using Python. With it, you can add content like headers, form data, multipart files, and parameters via simple Python libraries. It also allows you to access the response data of Python in the same way.

In programming, a library is a collection or pre-configured selection of routines, functions, and operations that a program can use. These elements are often referred to as modules, and stored in object format.

Response object

When one makes a request to a URI, it returns a response. This Response object in terms of python is returned by requests.method(), method being – get, post, put, etc. Response is a powerful object with lots of functions and attributes that assist in normalizing data or creating ideal portions of code. For example, response.status_code returns the status code from the headers itself, and one can check if the request was processed successfully or not.

Authentication using Python Requests

Authentication refers to giving a user permissions to access a particular resource. Since, everyone can’t be allowed to access data from every URL, one would require authentication primarily. To achieve this authentication, typically one provides authentication data through Authorization header or a custom header defined by server

SSL Certificate Verification

Requests verifies SSL certificates for HTTPS requests, just like a web browser. SSL Certificates are small data files that digitally bind a cryptographic key to an organization’s details. Often, a website with a SSL certificate is termed a secure website. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate.

Session Objects

Session object allows one to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance and will use urllib3’s connection pooling. So if several requests are being made to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase. A session object all the methods as of requests.

GET and POST Requests

Start off by importing requests. Now we are going to try to get a webpage using get request.

R_webpage is a response object. All the information about the web page can be extracted from this object.

Passing parameters in URLs

Parameters in URLs can be passed in a formal way. Requests allow us to give these arguments as a dictionary of strings. params is the keyword to use in the arguments for that purpose.

Response Content

Response of the server can be viewed completely as text. The complete text will be decoded after getting it from the server and displayed as text.

Custom Headers

Custom headers can be added to requests. Headers is the parameter which will have a dictionary passed by argument in order to specify the header.

In place of key and value, you can put your desired values throughout.

Directory listings

Python can be used to get the list of content from a directory. We can make a program to list the content of a directory which is in the same machine where python is running. We can also login to the remote system and list the content from the remote directory.

Listing Local Directory

In the below example we use the listdir() method to get the content of the current directory. To also indicate the type of the content like file or directory, we use more functions to evaluate the nature of the content.

Listing Remote Directory

We can list the content of the remote directory by using ftp to access the remote system. Once the connection is established we can use commands that will list the directory contents in a way similar to the listing of local directories.

Request Status Codes

After receiving and interpreting a request message, a server responds with an HTTP response message. The response message has a Status-Code. It is a 3-digit integer where first digit of the Status-Code defines the class of response and the last two digits do not have any categorization role. 

Successful Response

In the below example we access a file from a URL and the response is successful.

Unsuccessful Response

In the below example we access a file from a URL which does not exist. The response is unsuccessful. 

Performance

When using requests, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly.

Timeouts

When you make an inline request to an external service, your system will need to wait upon the response before moving on. If your application waits too long for that response, requests to your service could back up, your user experience could suffer, or your background jobs could hang.

By default, requests will wait indefinitely on the response, so you should almost always specify a timeout duration to prevent these things from happening. To set the request’s timeout, use the timeout parameter. timeout can be an integer or float representing the number of seconds to wait on a response before timing out:

The Session Object

Until now, you’ve been dealing with high level requests APIs such as get() and post(). These functions are abstractions of what’s going on when you make your requests. They hide implementation details such as how connections are managed so that you don’t have to worry about them.

Underneath those abstractions is a class called Session. If you need to fine-tune your control over how requests are being made or improve the performance of your requests, you may need to use a Session instance directly.

Max Retries

When a request fails, you may want your application to retry the same request. However, requests will not do this for you by default. To apply this functionality, you need to implement a custom Transport Adapter.

Transport Adapters let you define a set of configurations per service you’re interacting with. For example, let’s say you want all requests to https://api.github.com to retry three times before finally raising a Connection Error. You would build a Transport Adapter, set its max_retries parameter, and mount it to an existing Session.

Remote Procedure Call

Remote Procedure Call (RPC) system enables you to call a function available on a remote server using the same syntax which is used when calling a function in a local library. This is useful in two situations.

• You can utilize the processing power from multiple machines using rpc without changing the code for making the call to the programs located in the remote systems.
• The data needed for the processing is available only in the remote system.

So in python we can treat one machine as a server and another machine as a client which will make a call to the server to run the remote procedure. In our example we will take the localhost and use it as both a server and client.

Running a Server

The python language comes with an in-built server which we can run as a local server. The script to run this server is located under the bin folder of python installation and named as classic.py. We can run it in the python prompt and check its running as a local server.

Running a Client

Next we run the client using the rpyc module to execute a remote procedure call. In the below example we execute the print function in the remote server.

Expression Evaluation through RPC

Using the above code examples we can use python’s in-built functions for execution and evaluation of expressions through rpc.

Errors and Exceptions

There are a number of exceptions and error codes you need to be familiar with when using the Requests library in Python.

• If there is a network problem like a DNS failure, or refused connection the Requests library will raise a Connection Error exception.
• With invalid HTTP responses, Requests will also raise an HTTP Error exception, but these are rare.
• If a request times out, a Timeout exception will be raised.
• If and when a request exceeds the reconfigured number of maximum redirection, then a Too Many Redirects exception will be raised.

Web Server

Python is versatile enough to create many types of applications ans programs that drive the internet or other computer networks. One important aspect of internet is the web servers that are at the root of the client server model. In this chapter we will see few web servers which are created using pure python language.

G unicorn

G unicorn is a stand-alone web server which has a central master process tasked with managing the initiated worker processes of differing types. These worker processes then handle and deal with the requests directly. And all this can be configured and adapted to suit the diverse needs of production scenarios.

Important Features

• It supports WSGI and can be used with any WSGI running Python application and framework
• It can also be used as a drop-in replacement for Pasteur (ex: Pyramid), Django’s Development Server, web2py, etc
• Offers the choice of various worker types/configurations and automatic worker process management
• HTTP/1.0 and HTTP/1.1 (Keep-Alive) support through synchronous and asynchronous workers
• Comes with SSL support
• Extensible with hooks

CherryPy WSGI Server

CherryPy is a self contained web framework as it can run on its own without the need of additional software. It has its own WSGI, HTTP/1.1-compliant web server. As it is a WSGI server, it can be used to serve any other WSGI Python application as well, without being bound to CherryPy’s application development framework.

Important Features

• It can run any Python web applications running on WSGI.
• It can handle static files and it can just be used to serve files and folders alone.
• It is thread-pooled.
• It comes with support for SSL.
• It is an easy to adapt, easy to use pure-Python alternative which is robust and reliable.

Twisted Web

It is a web server that comes with the Twisted networking library. Whereas Twisted itself is “an event-driven networking engine”, the Twisted Web server runs on WSGI and it is capable of powering other Python web applications.

Important Features

• It runs WSGI Python applications
• It can act like a Python web server framework, allowing you to program it with the language for custom HTTP serving purposes
• It offers simple and fast prototyping ability through Python Scrips (.rpy) which are executed upon HTTP requests
• It comes with proxy and reverse-proxy capabilities
• It supports Virtual Hosts
• • It can even serve Perl, PHP ET terrace

Proxy servers

Proxy servers are used to browse to some website through another server so that the browsing remains anonymous. It can also be used to bypass the blocking of specific IP addresses.

We use the urlopen method from the urllib module to access the website by passing the proxy server address as a parameter.

Internet Protocol

The Internet Protocol is designed to implement a uniform system of addresses on all of the Internet-connected computers everywhere and to make it possible for packets to travel from one end of the Internet to the other. A program like the web browser should be able to connect to a host anywhere without ever knowing which maze of network devices each packet is traversing on its journey. There are various categories of internet protocols. The protocols are created to serve the needs of different types of data communication between different computers in the internet.

Python has several modules to handle each of these communication scenarios. The methods and functions in these modules can do the simplest job of just validating a URL or also the complex job of handling the cookies and sessions. In this chapter we will look at the most prominent python modules used for internet protocols.

DNS Looks-up

The IP addresses when translated to human readable formats or words become known as domain names. The translation of domain names to IP address is managed by the python module Python. This module also provides methods to find out CNAME and MX records.

Finding ‘A’ Record

In the below program we find the ip address for the domain using the resolver method. Usually this mapping between IP address and domain name is also known as ‘A’ record.

Finding CNAME Value

A CNAME record also known as Canonical Name Record is a type of record in the Domain Name System (DNS) used to map a domain name as an alias for another domain. CNAME records always point to another domain name and never directly to an IP address. In the query method below we specify the CNAME parameter to get the CNAME value.

Finding MX Record

A MX record also called mail exchange record is a resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient’s domain. It also sets the preference value used to prioritizing mail delivery if multiple mail servers are available. Similar to above programs we can find the value for MX record using the ‘MX’ parameter in the query method.

Routing

Routing is the mechanism of mapping the URL directly to the code that creates the web page. It helps in better management of the structure of the web page and increases the performance of the site considerably and further enhancements or modifications become really straight forward. In python routing is implemented in most of the web frame works.

Routing in Flask

The route() decorator in Flask is used to bind an URL to a function. As a result when the URL is mentioned in the browser, the function is executed to give the result. Here, URL ‘/hello’ rule is bound to the hello_world() function. As a result, if a user visits http://localhost:5000/ URL, the output of the hello_world() function will be rendered in the browser.

Using URL Variables

We can pass on URL variables using route to build URL on the fly. For this we use the URL_for() function which accepts name of the function as the first argument and the rest of the arguments as variable part of the URL rule.

In the below example we pass the function names as arguments to the URL_for function and print out the result when those lines are executed.

Redirects

We can use the redirect function to redirect the user to another URL using routing. We mention the new URL as a return value of the function which should redirect the user. This is helpful when we temporarily divert the users to a different page when we are modifying an existing web page.

JSON RPC call

JSON or JavaScript Object Notation is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. The RPC call made based on JSON is able to send data in a much compact and efficient manner than the normal XML based RPC call. The python module jsonrpclib is able to create a simple JSON based server and client.

SFTP

SFTP is also known as the SSH File Transfer Protocol. It is a network protocol that provides file access, file transfer, and file management over any reliable data stream. The program is run over a secure channel, such as SSH, that the server has already authenticated the client, and that the identity of the client user is available to the protocol.

Conclusion

You were given a basic introduction of the Python request module along with its working. Now, if you practice the above given examples on your own and add, eliminate and substitute things then you will get a better idea of its working. If you have made it here, congratulations because you have learned how to make basic requests to a server, passing parameters or arguments to the URLs, getting response content and showing it and passing custom headers. This will be very useful when you are trying to scrape web pages for information.

You’ve come a long way in learning about Python’s powerful requests library.

You’re now able to:

• Make requests using a variety of different HTTP methods such as GET, POST, and PUT
• Customize your requests by modifying headers, authentication, query strings, and message bodies
• Inspect the data you send to the server and the data the server sends back to you
• Work with SSL Certificate verification
• Use requests effectively using max_retries, timeout, Sessions, and Transport Adapters

Because you learned how to use requests, you’re equipped to explore the wide world of web services and build awesome applications using the fascinating data they provide.