libtorrent blog

bittorrent development from the trenches

socket receive buffers

In an attempt to save memory copying, libtorrent attempts to receive payload bytes directly into page aligned, pool allocated disk buffers. These buffers can then be used to DMA directly to disk (either with blocking O_DIRECT files or via AIO operations, if run on a clever kernel). To do this for the bittorrent protocol, the . . .

requesting pieces

Deciding how many outstanding requests to keep to peers typically is based on the bandwidth delay product, or a simplified model thereof. The bandwidth delay product is the bandwidth capacity of a link multiplied by the latency of the link. It represents the number of bytes that fit in the wire and buffers along the . . .

block request time-outs

timing out requests, and requesting blocks from other peers, may seem like a straight forward and trivial problem. However, there’s a balance to be struck between timing out a block too early, causing certain situation to systematically request every block multiple times, or never timing blocks out causing partially downloaded pieces to linger for a . . .

smart-ban

banning peers sending corrupt data Bittorrent lets you verify data you receive from the swarm against the SHA-1 hashes in the .torrent file. This enable clients to ban peers that sends data that fails the hash check, and thus cannot be trusted. However, the integrity checking can only be done at piece level. A piece . . .

scalable interfaces

The typical bittorrent client user interface shows you a list of all torrents loaded in the client. Some of these torrents may be stopped and inactive, some of them may be seeding, but not having any peers interested in the torrent, and some (obviously) downloading. The simple (and probably most common) way to update the . . .

multi-threaded piece hashing

The typical design of bittorrent clients is to run SHA-1 hashing of piece data as it’s being written to disk (typically in a disk thread). Doing this helps keeping a lot of things simple. The disk cache and the disk operations are all synchronous, including the SHA-1 hashing. Whenever the disk cache decides to flush . . .

writing a fast piece picker

One of the key algorithms in bittorrent is the rarest-first piece picker. It is vital to bittorrent’s performance that the piece picker fulfills both of these requirements: The rarest piece is picked (from the client’s point of view of the swarm) If two or more pieces have the same rarity, pick one of them at . . .