Project README

Website: jiayuxuan123.github.io/TurboDL — documentation, plugin authoring guide, download page

Languages: English · 简体中文 · 繁體中文 · 日本語 · 한국어 · Deutsch

TurboDL is a download core written from scratch. It only draws on the architectural and algorithmic ideas of mature download managers (aria2, IDM/XDM, axel, Persepolis, Motrix, ab-download-manager) without copying any of their source code, and is therefore released under the permissive MIT license (with supplemental plugin-ecosystem terms), free to use in open-source or commercial projects.

Features#

• Multi-threaded segmented download: HTTP Range parallel segments written directly into the destination file at their offsets (positional writes), connection reuse (HTTP/2 multiplexing + keep-alive).
• No merge phase: segments are not separate temp files that get concatenated afterwards. Measured on one disk, 16 MB: creating 512 segment files costs 627 ms while writing the same bytes positionally into one file costs 11 ms — 57x. On loopback this took 128-connection throughput from 20 MB/s to 88 MB/s, and it no longer degrades as connection count rises. There is also no "stuck at 99 %" step, because nothing is left to assemble.
• Progress survives crashes without inflated claims: a small sidecar records each segment's written byte count, and it is always persisted after the data lands. A crash can only lose the seconds since the last flush, and re-downloading that range is idempotent.
• Failure attribution without collateral damage: every segment keeps its own last error, and a request cancelled because another segment exhausted its retries does not consume the cancelled segment's retry budget. A single flaky connection therefore neither fails the task nor produces a misleading "retry N times failed (Canceled)" message.
• Dynamic segmentation: fine-grained pre-splitting + work stealing, so slow connections don't drag down the whole transfer and the "last segment finishing on a single thread" long tail is eliminated (IDM/XDM idea).
• Host-controlled adaptive concurrency: the configured connection count is a ceiling. DOWNWARD starts at the caller's requested count and reduces only when repeated throughput measurements prove the connections are wasted; UPWARD starts conservatively and doubles only after aggregate throughput improves by at least 30%. The selected policy is a host decision, not a provider-specific rule baked into the engine.
• Endgame by ETA, not by segment size: tail assist now tracks each in-flight segment's EWMA throughput and takes the remaining tail from the connection with the worst estimated completion time. A shrinking segment's old writer also stops at the handoff boundary instead of writing into the portion assigned to the new connection.
• Global resource budgets: task slots are FIFO and atomic, and maxTotalConnections caps the aggregate connection count across every built-in HTTP task. Three 64-connection tasks can no longer silently create 192 sockets when the host or router should only see a smaller budget.
• Desktop I/O path: JVM hosts use direct ByteBuffers; Android/Dalvik falls back to heap buffers and RandomAccessFile. Sidecar replacement keeps a durable backup instead of deleting the old progress before the new one is visible.
• Caller-supplied SHA-256: DownloadRequest.expectedSha256 streams the finished file through SHA-256 before completion; a mismatch fails the task and drops the resume snapshot so bad bytes are never reused.
• Multi-source mirrors (0.4.0.0): DownloadRequest.mirrors adds alternate URLs for the same file. The engine samples each source, then routes segments toward the fastest one by measured throughput, drops a source into exponential back-off when it fails, and rejects any mirror whose Content-Range total disagrees with the task size — so a mislabelled mirror can never write wrong bytes into the file. CLI: --mirror <URL> (repeatable).
• Sequential segment priority: earlier segments are downloaded first, enabling progressive preview/playback (axel/Persepolis idea).
• Robust fallback & retry:
• Server does not support / ignores Range → automatic fallback to whole-file single-stream download;
• A bad/timed-out segment → only that segment is retried, the whole task is not discarded;
• Verifies that the actually returned bytes match the requested Range (guards against servers tampering with Range and returning the whole file).
• Restrained adaptivity: a task starts at the caller's connection ceiling. A lower tier is accepted only when a clean, post-switch window still retains at least 90% of aggregate throughput; stale requests from the previous tier are excluded. 429/503 backpressure always takes priority.
• Resumable downloads: segments are persisted; pause/resume continues from the real on-disk progress.
• Byte-level integrity check: total size is verified after merge, rejecting corrupted files.

0.4.1.0 — industrial reliability pass#

This patch builds on the 0.4.0.0 correctness overhaul:

• HTTP 416 no longer marks an incomplete segment as successful. A transient rejection is retried; a persistent one carries an actionable failure and a changed source size invalidates the learned redirect.
• Dynamic tail segments are persisted as an ordered, validated immutable snapshot. An incomplete transient grid cannot replace the last durable resume point.
• Mirrors no longer overwrite each other's learned redirect target, and cross-origin mirrors/redirects do not inherit Cookie, Authorization or API-key headers. Proxy credentials remain separate.
• Task events are forwarded by one ordered consumer. MERGING, COMPLETED and FAILED now have explicit state-change events before their terminal events, so a host cannot receive completion before the lifecycle transition.
• Pause cleanup can no longer overwrite a completed/canceled/failed terminal state.
• DownloadRequest.mirrors moved to the end of the constructor and JvmOverloads keeps the legacy JVM constructor available; component7/component8 retain the old field meanings.
• Concurrent tail-assist snapshots no longer race Kotlin's size()/iterator fast path at high concurrency; a c=256 stress run exposed and reproduced the NoSuchElementException.
• Adaptive down-probes now wait until in-flight work is actually at the new target. Previously, requests left over from the old tier inflated the probe window and could halve real throughput.
• The CLI's JSON completion event now reports the real file SHA-256 plus the resume stableKey; it previously put the 16-character stable key in the sha256 field.
• The benchmark runner now computes a local SHA-256 and supports --sha256 instead of trusting file size alone. Its adaptive scenario uses the production fast-start policy unless slow start is explicitly requested.

0.4.0.0 — correctness overhaul#

This release is mostly bug fixes: thirteen defects surfaced by an internal audit, three of which could silently corrupt the output file.

Each fix is covered by a regression test (RegressionFixesTest.kt) plus two end-to-end mirror tests (MirrorPoolTest.kt).

Nine engine capabilities#

Quick start#

import dev.turbodl.core.*
import java.io.File

val client = TurboClient(
    TurboConfig(
        maxConnectionsPerTask = 16,
        globalSpeedLimitBytesPerSec = 0,        // unlimited
        dynamicSegmentation = true,
        proxy = ProxyMode.Direct,
        dns = DnsMode.System,
    )
)

// Submit a task
val id = client.submit(
    DownloadRequest(
        url = "https://example.com/big.zip",
        destination = File("big.zip"),
    )
)

// Observe progress events
scope.launch {
    client.events.collect { event ->
        when (event) {
            is TurboEvent.Progress -> println("${event.progress.percent}%  ${event.progress.speedBytesPerSec} B/s")
            is TurboEvent.Completed -> println("done: ${event.file}")
            is TurboEvent.Failed -> println("failed: ${event.reason}")
            else -> {}
        }
    }
}

// Suspend until finished
val result = client.await(id)   // Result<File>

// Controls
client.pause(id)
client.resume(id)
client.cancel(id, deleteOutput = true)

client.shutdown()

Command line#

turbodl-cli is a full-featured, Agent/script-friendly downloader (multi-threaded, resumable, JSON output).

Build the standalone fat JAR:

./gradlew :turbodl-cli:fatJar
# => turbodl-cli/build/libs/turbodl-cli-all.jar
java -jar turbodl-cli-all.jar --help

Common usage:

# Basic (default file name from URL)
turbodl <url> -o out.bin -c 64

# Through a proxy, with encrypted DNS, speed limited to 10MB/s
turbodl <url> -p http://127.0.0.1:7890 --doh https://dns.alidns.com/dns-query -l 10MB

# Machine-readable NDJSON events (start/progress/completed/failed) for scripts & agents
turbodl <url> --json

# Batch download (each line: <URL> [output])
turbodl --batch tasks.txt -c 128

# Custom headers / UA / ignore TLS
turbodl <url> -H "Cookie: k=v" -A "MyAgent/1.0" --insecure

Notes:

• Interrupted downloads resume automatically when re-running the same command (segments kept in .turbodl-parts/ next to the output, cleaned up after merge).
• Exit codes: 0 success, 1 download failed, 2 usage error.
• --json emits one JSON object per line; the final completed line contains the absolute file path and size.

NPM wrapper (@turbodl/cli)#

A thin Node.js shell around the JVM CLI (no download logic in JS — it auto-downloads and caches the JAR on first run, then passes all args through):

# After publishing:
npm install -g @turbodl/cli      # then: turbodl <url> -o out -c 64
npx @turbodl/cli <url> --json    # zero-install, agent-friendly NDJSON output

# Requires Java 17+; jar cached in ~/.turbodl/; TURBODL_JAR overrides the jar path.

Source: npm/cli. Published as @turbodl/cli on npm.

Build#

./gradlew build        # compile + run unit tests

Unit tests use an embedded HTTP(Range) server and cover: multi-threaded byte-level correctness, dynamic segmentation, fallback when Range is unsupported, fallback when Range is tampered, transient 503 retrying only the affected segment, and global speed limiting.

Modules#

• turbodl-core: the download engine SDK (the published library, usable standalone, no plugin-framework dependency).
• turbodl-cli: command-line example demonstrating SDK usage.
• turbo-plugin-runtime: optional plugin runtime kernel (lifecycle / disposer / event bus / service registry / extension points / version handshake / diagnostics). core does not depend on it; when not included, core works as usual.
• turbo-plugin-bootstrap: optional bootstrap module for one-click wiring of base plugins (Kotlin loader + HTTP backend); not a mandatory dependency.
• turbo-plugin-hls: optional HLS VOD protocol adapter plugin — resolves master/media M3U8 playlists, downloads segments concurrently (with per-segment retry), decrypts AES-128, honors EXT-X-BYTERANGE, and returns ordered parts for the engine to merge. Registers itself as a routed DownloadBackend; unsupported constructs (live streams, DRM/SAMPLE-AES, fMP4 EXT-X-MAP, discontinuities) fail explicitly instead of producing corrupt output. It also declares the hls protocol (manifest protocols + a load-time ProtocolClaim; the declaration grants no permission and routes nothing — .m3u8 stays the DownloadBackend predicate's job).
• turbo-plugin-js: optional JavaScript plugin loader — loads TurboDL plugins written in JavaScript. Embeds QuickJS (one runtime and one context per script) and exposes a stable host/plugin ABI with coarse-grained capabilities (http, crypto, log, storage, env, time, timers). Bridges JS link parsers and task hooks into the ordinary extension points, and unloads through an ACTIVE → STOPPING → DRAINING → DISPOSED sequence that never closes a runtime while its script is still executing. Only this module pulls in a JS engine; leave it out and nothing about QuickJS reaches your classpath.
• turbo-plugin-parser: optional link-parser plugin — turns a share link from a video/social platform into concrete download requests. Ships parsers for Weibo (MP4 + image sets), Xiaohongshu (watermark-free video + original images) and Pixiv (every page's urls.original). Registers one LinkParser at LINK_PARSER plus one ProtocolClaim per platform at PROTOCOL_HANDLER; core and runtime are untouched. Holds no credentials — where a platform needs a login it returns null and the host's own session handling takes over. Its public result type (ParsedPost) is deliberately path-free, because a DownloadRequest needs a destination: File while a parser runs before the host has chosen a directory; the temp-dir destination is filled in only at the boundary.
• demo: three runnable examples — Kotlin-native plugin, bootstrap usage, and a shim-adapter template. Run with ./gradlew :demo:run --args="1" (or 2, 3, all).

Plugin framework (optional)#

TurboDL is a standalone engine and an optional plugin platform. Three ideas drive the design:

1. Core works alone. turbodl-core is a complete multi-threaded engine with zero plugin dependency. Plugins are strictly additive.
2. Everything is a plugin; the kernel is only mechanism. The runtime kernel knows nothing about any protocol — it provides only lifecycle, cleanup (disposer), a service registry, a type-safe event bus, an extension-point registry, a version handshake, and diagnostics. The Kotlin loader, the HTTP backend and the HLS backend are all ordinary plugins.
3. Hybrid A+B. Core ships a built-in HTTP backend (A); a plugin backend can override it or add protocols through the registry (B) — without core ever depending on the runtime. Dependency direction is strict: runtime → core, never the reverse.

A versioned public API (ApiVersion, currently 1.0.0) plus a load-time handshake means a future breaking core release fails loudly (a plugin is marked INCOMPATIBLE and never loaded) instead of silently corrupting behavior.

Plugin documentation:

• Plugin authoring guide — how to build and integrate a plugin.
• Development Convention — the official compatibility rulebook (stable API, versioning, naming, safety).
• Plugin Market — publish/discover plugins via GitHub topics + a turbodl-plugin.json manifest.
• Manifest schema — JSON Schema for turbodl-plugin.json.

Plugins written in JavaScript#

Adding turbo-plugin-js lets a plugin be a script instead of a JVM class:

val boot = TurboBootstrap.create(
    config = TurboConfig(maxConnectionsPerTask = 16),
    extraPlugins = listOf(HlsPlugin(), JsPluginLoaderPlugin()),
)

boot.host.loadSource(PluginSource(kind = "js", uri = "/opt/turbodl-plugins/acme-parser.js"))

A script gets a host/plugin ABI and nothing more: it computes URLs, headers and signatures, reads and writes its own sandbox directory, and logs. It never touches the download data stream and cannot register a DownloadBackend or publish a host service — that boundary is what keeps a script from becoming a bandwidth bottleneck. Scripts are local files (or inline: / data: sources); require, npm and remote fetch are not part of the ABI.

The loader owns one QuickJS runtime and one context per script, and unloads in four stages so a runtime is never closed while its JavaScript is still running. See turbo-plugin-js/README.md for the ABI reference and turbo-plugin-js/ARCHITECTURE.md for why it is shaped that way.

Design notes & acknowledgements#

TurboDL's design draws on the ideas of the following open-source projects (ideas only, no source copied), with thanks: aria2, Xtreme Download Manager, axel, Persepolis, Motrix, ab-download-manager.

License#

MIT License, with "plugin-ecosystem supplemental terms" (clarifying that plugins are independent works, may be licensed freely, and are not considered derivative works merely by interacting through public APIs / extension points).