libtorrent blog

bittorrent development from the trenches

windows’ disk cache

A long standing problem with bittorrent clients on windows is that if you’re seeding large files or downloading large files, windows may decide to essentially use all your physical RAM for disk cache. The disk cache grows to the point where running processes start having their working set swapped out, significantly slowing down the system . . .

bittorrent over SSL

Running bittorrent over SSL could make sense for several applications. Anything you want distributed to a closed group, but large enough to warrant bittorrent would do well being distributed over bittorrent/SSL. Currently closed group distributions either don’t use any peer-to-peer distribution at all, or they use poor-man’s privacy/security. I’m referring to the “private” flag of . . .

seeding a million torrents

There are two main architectures of peer-to-peer networks. There’s the peer-centric (limewire style) and content-centric (bittorrent style). In a peer-centric network each participant announces its existence to the network, and other peers looking for content go around asking peers if they have the content. This makes the peer-centric networks scale well with pieces of content. . . .

socket receive buffers

In an attempt to save memory copying, libtorrent attempts to receive payload bytes directly into page aligned, pool allocated disk buffers. These buffers can then be used to DMA directly to disk (either with blocking O_DIRECT files or via AIO operations, if run on a clever kernel). To do this for the bittorrent protocol, the . . .

requesting pieces

Deciding how many outstanding requests to keep to peers typically is based on the bandwidth delay product, or a simplified model thereof. The bandwidth delay product is the bandwidth capacity of a link multiplied by the latency of the link. It represents the number of bytes that fit in the wire and buffers along the . . .

block request time-outs

timing out requests, and requesting blocks from other peers, may seem like a straight forward and trivial problem. However, there’s a balance to be struck between timing out a block too early, causing certain situation to systematically request every block multiple times, or never timing blocks out causing partially downloaded pieces to linger for a . . .

smart-ban

banning peers sending corrupt data Bittorrent lets you verify data you receive from the swarm against the SHA-1 hashes in the .torrent file. This enable clients to ban peers that sends data that fails the hash check, and thus cannot be trusted. However, the integrity checking can only be done at piece level. A piece . . .

scalable interfaces

The typical bittorrent client user interface shows you a list of all torrents loaded in the client. Some of these torrents may be stopped and inactive, some of them may be seeding, but not having any peers interested in the torrent, and some (obviously) downloading. The simple (and probably most common) way to update the . . .

multi-threaded piece hashing

The typical design of bittorrent clients is to run SHA-1 hashing of piece data as it’s being written to disk (typically in a disk thread). Doing this helps keeping a lot of things simple. The disk cache and the disk operations are all synchronous, including the SHA-1 hashing. Whenever the disk cache decides to flush . . .

writing a fast piece picker

One of the key algorithms in bittorrent is the rarest-first piece picker. It is vital to bittorrent’s performance that the piece picker fulfills both of these requirements: The rarest piece is picked (from the client’s point of view of the swarm) If two or more pieces have the same rarity, pick one of them at . . .