Preheat
This document will help you experience how to use Dragonfly's Open API for preheating.
Create personal access token
Click the ADD PERSONAL ACCESS TOKENS
button to create personal access token.
Name: Set your token a descriptive name.
Description: Set a description.
Expiration: Set your token an expiration.
Scopes: Select the access permissions for the token.
Click SAVE
and copy the token and store it. For your security, it doesn't display again.
Preheat image
Use Open API for preheating image. First create a POST request for preheating.
args: Parameters for the preheat job.
-
url: URL used to specify the resource to be preheated.
-
concurrent_task_count: Used to specify the maximum number of tasks (e.g., image layers) to preheat concurrently. For example, if preheating 100 layers with ConcurrentTaskCount set to 10, up to 10 layers are processed simultaneously. If ConcurrentPeerCount is 10 for 1000 peers, each layer is preheated by 10 peers concurrently.Default is 8, maximum is 100.
-
concurrent_peer_count: Used to specify the maximum number of peers to preheat concurrently for a single task (e.g., an image layer). For example, if preheating a layer with ConcurrentPeerCount set to 10, up to 10 peers process that layer simultaneously. Default is 500, maximum is 1000.
-
platform: The image type preheating task can specify the image architecture type. eg: linux/amd64、linux/arm64.
-
scope: Select the scope of preheat as needed.
-
single_seed_peer: Preheat to a seed peer.
-
all_seed_peers: Preheat to each seed peer in the P2P cluster.
- count: The count of preheat seed peers desired.
This field is used only when
IPs
is not specified. It has priority overPercentage
. It must be a value between 1 and 200 (inclusive) if provided. - percentage: The percentage of preheat seed peers desired.
This field has the lowest priority and is only used if both
IPs
andCount
are not provided. It must be a value between 1 and 100 (inclusive) if provided. - ips: By setting the IPs, can specify a seed peer IP for preheating. This field has the highest priority: if provided, both
Count
andPercentage
will be ignored.
- count: The count of preheat seed peers desired.
This field is used only when
-
all_peers: Preheat to each peer in the P2P cluster.
- count: The count of preheat peers desired.
This field is used only when
IPs
is not specified. It has priority overPercentage
. It must be a value between 1 and 200 (inclusive) if provided. - percentage: The percentage of preheat peers desired.
This field has the lowest priority and is only used if both
IPs
andCount
are not provided. It must be a value between 1 and 100 (inclusive) if provided. - ips: By setting the IPs, can specify a peer IP for preheating. This field has the highest priority: if provided, both
Count
andPercentage
will be ignored.
- count: The count of preheat peers desired.
This field is used only when
-
-
username: The username used to authenticate the image manifest.
-
password: The password used to authenticate the image manifest.
-
tag: When the URL of the preheat task are the same but the Tag are different, they will be distinguished based on the tag and the generated preheat task will be different.
-
application: When the URL of the preheat tasks are the same but the application are different, they will be distinguished based on the application and the generated preheat tasks will be different.
-
filtered_query_params: By setting the filter parameter, you can specify the file type of the resource that needs to be preheated. The filter is used to generate a unique preheat task and filter unnecessary query parameters in the URL, separated by & characters.
-
headers: Add headers for preheat requests.
scheduler_cluster_ids: Specify the preheated scheduler cluster id,
if scheduler_cluster_ids
is empty, it means preheating all scheduler clusters.
curl --location --request POST 'http://dragonfly-manager:8080/oapi/v1/jobs' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer your_dragonfly_personal_access_token' \
--data-raw '{
"type": "preheat",
"args": {
"type": "image",
"url": "https://index.docker.io/v2/library/alpine/manifests/3.19",
"username": "your_registry_username",
"password": "your_registry_password",
"scope": "single_seed_peer"
},
"scheduler_cluster_ids":[1]
}'
The command-line log returns the preheat job id.
{
"id": 1,
"created_at": "0001-01-01T00:00:00Z",
"updated_at": "0001-01-01T00:00:00Z",
"task_id": "group_9523f30a-877d-41f7-a25f-0854228341f6",
"type": "preheat",
"state": "PENDING",
"args": {
"scope": "single_seed_peer",
"type": "image",
"url": "https://dockerpull.org/v2/library/alpine/manifests/3.19"
},
"result": null,
"scheduler_clusters": [
{
"id": 1,
"created_at": "2024-12-11T07:57:44Z",
"updated_at": "2024-12-11T07:57:44Z",
"name": "cluster-1"
}
]
}
Polling the preheating status with job id.
curl --request GET 'http://dragonfly-manager:8080/oapi/v1/jobs/1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer your_dragonfly_personal_access_token'
If the status is SUCCESS
, the preheating is successful.
{
"id": 1,
"created_at": "0001-01-01T00:00:00Z",
"updated_at": "0001-01-01T00:00:00Z",
"task_id": "group_9523f30a-877d-41f7-a25f-0854228341f6",
"type": "preheat",
"state": "SUCCESS",
"args": {
"scope": "single_seed_peer",
"type": "image",
"url": "https://dockerpull.org/v2/library/alpine/manifests/3.19"
},
"result": null,
"scheduler_clusters": [
{
"id": 1,
"created_at": "2024-12-11T07:57:44Z",
"updated_at": "2024-12-11T07:57:44Z",
"name": "cluster-1"
}
]
}
Preheat file
Use Open API for preheating file. First create a POST request for preheating.
args: Parameters for the preheat job.
-
urls: Used to specify the URL addresses of resources requiring preheating, supporting multiple URLs in a single preheat request.
-
concurrent_task_count: Used to specify the maximum number of tasks (e.g., image layers) to preheat concurrently. For example, if preheating 100 layers with ConcurrentTaskCount set to 10, up to 10 layers are processed simultaneously. If ConcurrentPeerCount is 10 for 1000 peers, each layer is preheated by 10 peers concurrently.Default is 8, maximum is 100.
-
concurrent_peer_count: Used to specify the maximum number of peers to preheat concurrently for a single task (e.g., an image layer). For example, if preheating a layer with ConcurrentPeerCount set to 10, up to 10 peers process that layer simultaneously. Default is 500, maximum is 1000.
-
scope: Select the scope of preheat as needed.
-
single_seed_peer: Preheat to a seed peer.
-
all_seed_peers: Preheat to each seed peer in the P2P cluster.
- count: The count of preheat seed peers desired.
This field is used only when
IPs
is not specified. It has priority overPercentage
. It must be a value between 1 and 200 (inclusive) if provided. - percentage: The percentage of preheat seed peers desired.
This field has the lowest priority and is only used if both
IPs
andCount
are not provided. It must be a value between 1 and 100 (inclusive) if provided. - ips: By setting the IPs, can specify a peer IP for preheating. This field has the highest priority: if provided, both
Count
andPercentage
will be ignored.
- count: The count of preheat seed peers desired.
This field is used only when
-
all_peers: Preheat to each peer in the P2P cluster.
- count: The count of preheat peers desired.
This field is used only when
IPs
is not specified. It has priority overPercentage
. It must be a value between 1 and 200 (inclusive) if provided. - percentage: The percentage of preheat peers desired.
This field has the lowest priority and is only used if both
IPs
andCount
are not provided. It must be a value between 1 and 100 (inclusive) if provided. - ips: By setting the IPs, can specify a seed peer IP for preheating. This field has the highest priority: if provided, both
Count
andPercentage
will be ignored.
- count: The count of preheat peers desired.
This field is used only when
-
-
tag: When the URL of the preheat task are the same but the Tag are different, they will be distinguished based on the tag and the generated preheat task will be different.
-
application: When the URL of the preheat tasks are the same but the Application are different, they will be distinguished based on the Application and the generated preheat tasks will be different.
-
filtered_query_params: By setting the filter parameter, you can specify the file type of the resource that needs to be preheated. The filter is used to generate a unique preheat task and filter unnecessary query parameters in the URL, separated by & characters.
-
headers: Add headers for preheat requests.
scheduler_cluster_ids: Specify the preheated scheduler cluster id,
if scheduler_cluster_ids
is empty, it means preheating all scheduler clusters.
curl --location --request POST 'http://dragonfly-manager:8080/oapi/v1/jobs' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer your_dragonfly_personal_access_token' \
--data-raw '{
"type": "preheat",
"args": {
"type": "file",
"urls": ["https://example.com"],
"scope": "single_seed_peer"
},
"scheduler_cluster_ids":[1]
}'
The command-line log returns the preheat job id.
{
"id": 1,
"created_at": "2024-12-11T08:30:12Z",
"updated_at": "2024-12-11T08:30:51Z",
"task_id": "group_4dd1da54-96ca-48ff-8f20-4fae665f677f",
"type": "preheat",
"state": "PENDING",
"args": {
"scope": "single_seed_peer",
"type": "file",
"url": "https://example.com"
},
"scheduler_clusters": [
{
"id": 1,
"created_at": "2024-12-11T07:57:44Z",
"updated_at": "2024-12-11T07:57:44Z",
"name": "cluster-1"
}
]
}
Polling the preheating status with job id.
curl --request GET 'http://dragonfly-manager:8080/oapi/v1/jobs/1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer your_dragonfly_personal_access_token'
If the status is SUCCESS
, the preheating is successful.
{
"id": 1,
"created_at": "2024-12-11T08:30:12Z",
"updated_at": "2024-12-11T08:30:51Z",
"task_id": "group_4dd1da54-96ca-48ff-8f20-4fae665f677f",
"bio": "",
"type": "preheat",
"state": "SUCCESS",
"args": {
"platform": "",
"scope": "single_seed_peer",
"type": "file",
"url": "https://example.com"
},
"scheduler_clusters": [
{
"id": 1,
"created_at": "2024-12-11T07:57:44Z",
"updated_at": "2024-12-11T07:57:44Z",
"name": "cluster-1"
}
]
}
Harbor
Use harbor for preheating image, please refer to the harbor documentation for details.
Configure self-signed certificates for registry
Notice: If harbor is not configured self-signed certificates, please ignore the following.
To support preheating for harbor with self-signed certificates, the Manager configuration needs to be modified.
Configure manager.yaml
, the default path is /etc/dragonfly/manager.yaml
,
refer to manager config.
Notice:
yourdomain.crt
is Harbor's ca.crt.
job:
# Preheat configuration.
preheat:
tls:
# insecureSkipVerify controls whether a client verifies the server's certificate chain and hostname.
insecureSkipVerify: false
# # caCert is the CA certificate for preheat tls handshake, it can be path or PEM format string.
caCert: /etc/certs/yourdomain.crt
Skip TLS verification, set job.preheat.tls.insecureSkipVerify
to true.
job:
# Preheat configuration.
preheat:
tls:
# insecureSkipVerify controls whether a client verifies the server's certificate chain and hostname.
insecureSkipVerify: true
# # caCert is the CA certificate for preheat tls handshake, it can be path or PEM format string.
# caCert: ''
Create personal access token
Click the ADD PERSONAL ACCESS TOKENS
button to create personal access token.
Name: Set your token a descriptive name.
Description: Set your token a descriptive information.
Expiration: Set your token an expiration.
Scopes: Select the access permissions for the token.
Click SAVE
and copy the token and store it. For your security, it doesn't display again.
Create instance
Open the harbor UI, go to Distributions
item under Administration
and click the NEW INSTANCE
button to create create instance.
Step 1: Enter REST address of the Dragonfly Manager.
Step 2: Auth Mode selects OAuth for authentication.
Step 3: Enter personsal assess token.
Click the TEST CONNECTION
button to test the connectivity of the creating instance.
If the connectivity testing is successful, click the OK
button to save the creating instance.
Notice: Instance status must be
Healthy
.
Create P2P provider policy
Go to Projects
and open your project from the project list, and open the P2P Preheat
tab.
Click the NEW POLICY
button to create P2P provider policy.
Scope: Select the scope of preheat as needed.
-
Single Seed Peer: Preheat to a seed peer.
-
All Seed Peers: Preheat to each seed peer in the P2P cluster.
-
All Peers: Preheat to each peer in the P2P cluster.
Step 2: Enter the cluster id of Dragonfly Manager in Cluster IDs
to specify the preheated cluster.
If Cluster IDs
is empty, it means to preheat all clusters.
Executions Preheat policy
Click the EXECUTE
to execute the preheating task.
If the status is SUCCESS, the preheating is successful.
Click the executions ID
to view the detailed information of the preheating task, and click the Logs icon to view the log.
The expected output is as follows.