Twisted is an framework event-driven Python application and web server builder. Find out all you need to know about this widely used tool: origins, history, how it works, training courses...
In the early 2000s, networked video game developers had no multiplatform programming language library at their disposal.
It was during this time that the American programmer Glyph Lefkowitz was working on a text-based multiplayer game titled “Twisted Reality.” The project quickly turned chaotic. For each connection, three Java threads were required.
One thread was used for input and would block on reads, another thread for output would block on certain types of writes, and a “logic” thread slept while waiting for timers to expire or events to be added to the queue.
As players navigated and interacted in this virtual environment, the threads would deadlock, caches would become corrupted, and the blocking logic was never appropriate.
In other words, the use of threads made the software complex, buggy, and very difficult to extend. Faced with these issues, Glyph realized the need for an extensible, multiplatform, event-driven networking framework.
The Twisted story
In seeking solutions to the problems encountered during the development of his game, Glyph discovered Python and its “select” module for I/O multiplexing from stream objects like sockets or pipes.
At that time, Java did not expose the operating system’s select interface or any other asynchronous I/O API. The java.nio package for non-blocking I/O was added in 2002 with J2SE 1.4.
As a result, a rapid prototype of the game in Python using the select module proved immediately less complex and more reliable than the threaded version.
Glyph then wrote a client and server for his game in Python using the select API. However, he wanted to go further and be able to transform network activity into method calls on objects in the game.
He envisioned, for example, the possibility of receiving an email in the game, similar to the mail daemon in the 1987 game NetHack. He also envisioned each player having a homepage.
To realize his ideas, Glyph needed Python HTTP and IMAP clients and servers using select. He first tried the Medusa platform, developed in the mid-90s for writing networking servers in Python based on the “asyncore” module.
This module is a socket manager that builds a dispatcher and callback interface on top of the operating system’s select API. However, Medusa had two weaknesses.
First, the project’s abandonment was planned for 2001. Additionally, the asyncore module is such a thin wrapper around sockets that application programmers still need to manipulate sockets directly.
Portability, therefore, remained the responsibility of the programmer. Moreover, asyncore support for Windows was still buggy, and Glyph knew he wanted to run a GUI client on Windows.
Faced with the prospect of implementing a networking platform himself, Glyph realized that Twisted Reality had opened the door to a problem as interesting as the game itself. Over time, his video game project evolved into a networking platform: Twisted.
What is Twisted?
Twisted is an event-driven networking engine written in the Python language. It is used to create servers for protocols such as SSH, proxy, SMTP, and HTTP.
This tool addresses several weaknesses of existing Python networking platforms. It utilizes event-driven programming rather than multi-threaded programming. It is also a cross-platform tool that provides a uniform interface for event notification systems exposed by major operating systems.
Additionally, Twisted directly offers client and server implementations of many application and transport layer protocols, including TCP, UDP, SSL/TLS, HTTP, IMAP, SSH, IRC, and FTP. Utilities also allow for easy configuration and deployment of applications from the command line, making it immediately useful for developers.
Twisted is compliant with Request for Comments (RFCs), demonstrating its compliance through a robust test suite. It is extensible and simplifies the use of multiple networking protocols.
from twisted.web import server, resource
from twisted.internet import reactor, endpoints
class Counter(resource.Resource):
isLeaf = True
numberRequests = 0
def render_GET(self, request):
self.numberRequests += 1
request.setHeader(b"content-type", b"text/plain")
content = u"I am request #{}\n".format(self.numberRequests)
return content.encode("ascii")
endpoints.serverFromString(reactor, "tcp:8080").listen(server.Site(Counter()))
reactor.run()
What is event-driven programming?
The main feature of Twisted is that it is an event-driven networking engine. This programming paradigm determines the flow of the program based on external events.
It is characterized by an event loop and the use of callbacks to trigger actions when events occur.
Among the most common programming paradigms are single-threaded synchronous programming and multi-threaded programming.
In single-threaded synchronous programming, tasks are executed sequentially. If a task blocks for some time on I/O (input/output), all other tasks must wait until it is completed before they can execute.
This defined order and serial processing simplify reasoning, but the program is needlessly slow if tasks are not dependent on each other because they still have to wait for each other.
On the other hand, multi-threaded programming executes tasks in separate control threads.
These threads are managed by the operating system and can run concurrently on multiple processors or interleaved on a single processor.
This allows threads to make progress even when others are blocked on resources. In general, this approach saves time compared to a similar synchronous program.
However, code needs to be written to protect shared resources that can be accessed simultaneously by multiple threads. Multi-threaded programs require more complex reasoning as it is essential to be concerned with thread safety through process serialization, reentrancy, thread-local storage, or other mechanisms. Subtle bugs can occur with poor implementation.
Finally, an event-driven program interleaves task execution in a single control thread. During I/O or other resource-intensive operations, a callback is registered with an event loop, and execution continues while the operation completes.
The callback describes how to handle an event after its completion. The event loop queries events and dispatches them to callbacks as they arrive.
This allows the program to make progress when it can without the need for additional threads. Event-driven programs simplify reasoning compared to multi-threaded programs because the programmer doesn’t need to worry about thread safety.
This model is generally the best choice when there are many independent tasks that don’t need to communicate or wait for each other, or when some of these tasks block while waiting for events.
It’s also a good choice when an application needs to share mutable data between tasks since no synchronization needs to be performed. Networking applications often have these properties, making them particularly suitable for event-driven programming.
In summary, Twisted offers the parallelism of multi-threaded programming combined with the ease of reasoning of single-threaded programming.
What is the Twisted reactor?
The concept of a reactor involves distributing events from multiple sources to their recipients within a single-threaded environment. The reactor’s event loop is at the core of Twisted’s operation.
The reactor is aware of network, file system, and timer events. It waits for events and then handles them, allowing for abstraction of platform-specific behavior and presenting interfaces to simplify event handling anywhere on the network stack.
By default, a reactor based on the poll API is provided on all platforms. Other high-volume multiplexing APIs specific to certain platforms are also supported.
For example, there’s a KQueue-based reactor using FreeBSD’s kqueue mechanism, an epoll-based reactor, and an IOCP-based reactor using Windows’ I/O completion ports, among others.
Examples of implementation-dependent details supported by Twisted include network and file system limits, buffering behavior, lost connection detection method, or error return values.
Twisted’s reactor implementation also supports the use of underlying non-blocking APIs. Since the Python language does not expose the IOCP API, Twisted maintains its own implementation for this purpose.
How do I test with Twisted?
Twisted provides a comprehensive test suite that can be executed using the “tox” command. For example, “tox -e list” allows you to see all test environments, “tox -e nocov” runs all tests without coverage, “tox -e withcov” runs tests with coverage, and “tox -e alldeps-withcov-posix” installs all dependencies and runs tests with coverage on the POSIX platform.
You can run the test suite on different reactors using the TWISTED_REACTOR environment variable, for example, “$ env TWISTED_REACTOR=epoll tox -e alldeps-withcov-posix”.
Some of these tests may fail if the required dependencies for a specific subsystem are not installed, if a firewall blocks certain ports, or if they are executed as root.
Additionally, you can check that the code adheres to Twisted’s standards. The “tox -e lint” command runs pre-commit, and the “tox -e mypy” command runs the MyPy checker to detect static code typing errors.
Comment installer Twisted ?
Before starting to develop applications with Twisted, you need to download and install it along with all its dependencies.
For a Linux installation, you can use the following commands:
– On a dpkg-based system (like Debian or Ubuntu): “apt-get install python-twisted”
– On an rpm-based system (like Red Hat or CentOS): “yum install python-twisted”
For a Windows installation, you can find MSI and EXE installers in both 32-bit and 64-bit versions. If you’re unsure about which version to use, the 32-bit MSI installer should work in most cases. Download and run the Twisted installer as well as the zope.interface installer from the Twisted homepage.
You can also install optional packages and dependencies to take advantage of additional features in Twisted. There’s an Ubuntu PPA available on the Twisted-dev team’s Launchpad page, which provides packages for the latest Twisted version on all active Ubuntu versions.
For SSL or SSH functionality, you can install the pyOpenSSL and PyCrypto packages, which are available as python-openssl and python-crypto.
La communauté de Twisted
If you need advice for a project, you can seek help from the Twisted community. Contributors are always willing to provide support.
There are two main mailing lists. The twisted-python list is a general discussion list for development-related questions and also serves for announcements related to Twisted and projects using it.
This list is also used for organizing sprints, discussing tickets, requesting feedback on projects, and discussing project maintenance and infrastructure. The twisted-web list is dedicated to discussions about web technologies related to Twisted.
IRC channels #twisted and #twisted.web are also available on the Freenode network. These channels are quite active, but if you don’t receive an immediate response, you can send a message to the appropriate mailing list.
Many Twisted-related questions and answers can also be found on Stack Overflow, and developers post sprint reports and release announcements on the official blog. Developers’ personal blogs are aggregated on Planet Twisted as well.
Conclusion
Twisted has long been a reference for creating servers or internet applications in Python. However, this open-source tool has struggled to adapt to the evolving landscape of the internet and its usage. Nevertheless, its many extensions make it a flexible and extensible tool.
To learn how to master Twisted and the Python language, you can consider DataScientest. All our training programs include a dedicated module for this programming language.
Our various courses provide you with the skills required for professions such as Data Analyst, Data Scientist, Data Engineer, Machine Learning Engineer, or Data Product Manager.
Our programs are fully accessible online, and our organization is recognized by the state, making it eligible for different financing options. Discover DataScientest!
