Skip to main content
Version: v2.1.x

Dfdaemon

dfget​

dfget is the client of Dragonfly used to download and upload files

Usage​

dfget is the client of Dragonfly which takes a role of peer in a P2P network. When user triggers a file downloading task, dfget will download the pieces of file from other peers. Meanwhile, it will act as an uploader to support other peers to download pieces from it if it owns them. In addition, dfget has the abilities to provide more advanced functionality, such as network bandwidth limit, transmission encryption and so on.

dfget url -O path [flags]
dfget [command]

Available Commands​

  completion  generate the autocompletion script for the specified shell
  daemon      start the client daemon of dragonfly
  doc         generate documents
  help        Help about any command
  plugin      show plugin
  version     show version

Options​

      --accept-regex string   Recursively download only. Specify a regular expression to accept the complete URL. In this case, you have to enclose the pattern into quotes to prevent your shell from expanding it
      --config string         the path of configuration file with yaml extension name, it can also be set by env var: DFGET_CONFIG
      --console               whether logger output records to the stdout
      --daemon-sock string    Download socket path of daemon. In linux, default value is /var/run/dfdaemon.sock, in macos(just for testing), default value is /tmp/dfdaemon.sock
      --digest string         Check the integrity of the downloaded file with digest, in format of md5:xxx or sha256:yyy
      --disable-back-source   Disable downloading directly from source when the daemon fails to download file
      --filter string         Filter the query parameters of the url, P2P overlay is the same one if the filtered url is same, in format of key&sign, which will filter 'key' and 'sign' query parameters
  -H, --header strings        url header, eg: --header='Accept: *' --header='Host: abc'
  -h, --help                  help for dfget
      --jaeger string         jaeger endpoint url, like: http://localhost:14250/api/traces
      --level uint            Recursively download only. Set the maximum number of subdirectories that dfget will recurse into. Set to 0 for no limit (default 5)
  -l, --list                  Recursively download only. List all urls instead of downloading them.
      --logdir string         Dfget log directory
      --original-offset       Range request only. Download ranged data into target file with original offset. Daemon will make a hardlink to target file. Client can download many ranged data into one file for same url. When enabled, back source in client will be disabled
  -O, --output string         Destination path which is used to store the downloaded file, it must be a full path
  -P, --priority string       Scheduler will schedule task according to priority
      --pprof-port int        listen port for pprof, 0 represents random port (default -1)
      --range string          Download range. Like: 0-9, stands download 10 bytes from 0 -9, [0:9] in real url
      --ratelimit string      The downloading network bandwidth limit per second in format of G(B)/g/M(B)/m/K(B)/k/B, pure number will be parsed as Byte, 0 is infinite (default "100.0MB")
  -r, --recursive             Recursively download all resources in target url, the target source client must support list action
      --reject-regex string   Recursively download only. Specify a regular expression to reject the complete URL. In this case, you have to enclose the pattern into quotes to prevent your shell from expanding it
      --service-name string   name of the service for tracer (default "dragonfly-dfget")
  -b, --show-progress         Show progress bar, it conflicts with --console
      --tag string            Different tags for the same url will be divided into different P2P overlay, it conflicts with --digest
      --timeout duration      Timeout for the downloading task, 0 is infinite
  -u, --url string            Download one file from the url, equivalent to the command's first position argument
      --verbose               whether logger use debug level
      --workhome string       Dfget working directory

Example​

Download with HTTP protocol​

dfget -u "http://example.com/object" -O /path/to/output

Download with OSS protocol​

Dragonfly supports download objects from Alibaba Cloud Object Storage Service (OSS). You can download via dfget command.

All arguments is necessary to download from private OSS bucket.

Header explain:

Endpoint: OSS Endpoint, refer: Alibaba Cloud

AccessKeyID: OSS AccessKey ID

AccessKeySecret: OSS AccessKey Secret

--filter "Expires&Signature" is used for generating unique task id for same object in different machines.

oss://bucket/path/to/object is the object bucket and path.

dfget --header "Endpoint: https://oss-cn-hangzhou.aliyuncs.com" \
    --header "AccessKeyID: id" \
    --header "AccessKeySecret: secret" \
    --url oss://bucket/path/to/object \
    --output /path/to/output \
    --filter "Expires&Signature"

Log configuration​

1. set option --console if you want to print logs to Terminal
2. log path: /var/log/dragonfly/dfget/

dfget daemon​

Daemon Options​

      --accept-regex string   Recursively download only. Specify a regular expression to accept the complete URL. In this case, you have to enclose the pattern into quotes to prevent your shell from expanding it
      --config string         the path of configuration file with yaml extension name, default is /etc/dragonfly/dfget.yaml, it can also be set by env var: DFGET_CONFIG
      --console               whether logger output records to the stdout
      --digest string         Check the integrity of the downloaded file with digest, in format of md5:xxx or sha256:yyy
      --disable-back-source   Disable downloading directly from source when the daemon fails to download file
      --filter string         Filter the query parameters of the url, P2P overlay is the same one if the filtered url is same, in format of key&sign, which will filter 'key' and 'sign' query parameters
  -H, --header strings        url header, eg: --header='Accept: *' --header='Host: abc'
  -h, --help                  help for dfget
      --jaeger string         jaeger endpoint url, like: http://localhost:14250/api/traces
      --level uint            Recursively download only. Set the maximum number of subdirectories that dfget will recurse into. Set to 0 for no limit (default 5)
      --limit string          The downloading network bandwidth limit per second in format of G(B)/g/M(B)/m/K(B)/k/B, pure number will be parsed as Byte, 0 is infinite (default "0.0B")
  -l, --list                  Recursively download only. List all urls instead of downloading them.
      --logdir string         Dfget log directory
  -O, --output string         Destination path which is used to store the downloaded file, it must be a full path
      --pprof-port int        listen port for pprof, 0 represents random port (default -1)
  -r, --recursive             Recursively download all resources in target url, the target source client must support list action
      --reject-regex string   Recursively download only. Specify a regular expression to reject the complete URL. In this case, you have to enclose the pattern into quotes to prevent your shell from expanding it
      --service-name string   name of the service for tracer (default "dragonfly-dfget")
  -b, --show-progress         Show progress bar, it conflicts with --console
      --tag string            Different tags for the same url will be divided into different P2P overlay, it conflicts with --digest
      --timeout duration      Timeout for the downloading task, 0 is infinite
  -u, --url string            Download one file from the url, equivalent to the command's first position argument
      --verbose               whether logger use debug level
      --workhome string       Dfget working directory

Daemon Log configuration​

1. set option --console if you want to print logs to Terminal
2. log path: /var/log/dragonfly/daemon/