Handling partial downloads in self-hosted batch scripts
A developer shares a post-mortem on a download script that failed to verify existing folders, highlighting the need for robust restart policies in unattended jobs.
A recent technical analysis published on November 16, 2026, details a subtle but critical flaw in a self-hosted batch download script. The author, writing for the DEV Community, discovered that their automation tool treated pre-existing directories as complete without verifying their contents, leading to ambiguous failure states during restarts. This case study highlights the complexities of managing stateful operations in unattended environments where network interruptions or disk space issues are common.
What happened
The script in question was designed to archive a batch of machine learning models, including image, audio, and video repositories, each pinned to a specific revision. During a successful run, all seven repositories were marked as verified after passing checksum checks. However, the author noticed a gap in the logic when simulating a restart scenario. If a destination folder already existed, the script would skip both the download and the verification steps, recording the status simply as existing.
This behavior caused the entire batch job to exit with a failure code because the final status check expected every entry to be explicitly verified. While this prevented a false positive success report, it also meant the script could not certify that the existing files were actually complete or correct. The author reproduced this issue in an isolated test environment using mocked commands, confirming that the script offered no way to distinguish between a fully downloaded archive and a partially corrupted one from a previous interrupted run.
Beyond the restart logic, the script included several safeguards for disk management. Before downloading each repository, it checked the free space on the target volume against the estimated size of the file plus a 200 GB reserve. For the largest model, estimated at 127 GB, this required at least 327 GB of free space. The script also verified that the target mount point was active to prevent accidental writes to the system disk. However, these checks occurred only before each transfer, not during it, meaning concurrent processes could still consume the reserved margin while a download was in progress.
Key details
- The script records three primary statuses:
download_failed,verification_failed, andverified, with a fourth ambiguous state calledexistingfor skipped folders. - A 200 GB reserve buffer was added to each download estimate to account for temporary files and overhead, using decimal gigabytes.
- The verification step compares downloaded files against a pinned revision, treating missing files as a failure, but does not assess model quality or memory fit.
- Free space checks are performed immediately before each repository download but do not lock the space or monitor usage during the transfer.
- The final results file is written only at the end of the batch, meaning an abrupt process termination leaves no complete ledger of what was accomplished.
- In the tested restart scenario, the script exited with code 1 due to the
existingstatus, even though the other six mocked downloads reported success.
Background
Self-hosted automation scripts often run unattended for long periods, making them vulnerable to transient failures like network drops or power outages. A robust script must be idempotent, meaning it can be restarted multiple times without causing errors or duplicating work. This usually involves checking the current state of the system before acting. However, determining whether a partially completed task is valid requires more than just checking for the existence of a file or folder. Checksums and revision pins provide a way to verify data integrity, ensuring that the bits on the disk match the expected source exactly.
In batch processing, managing state is critical. If a job processes ten items and fails on the fifth, a naive restart might reprocess the first four, wasting resources, or skip them entirely, risking data corruption if they were incomplete. Advanced strategies involve checkpointing, where the progress of each individual item is recorded persistently as it completes. This allows the job to resume precisely where it left off, verifying only the necessary components rather than assuming safety based on folder presence.
Why it matters
For teams running their own infrastructure, silent data corruption is a significant risk. If a backup or archive script assumes that an existing folder is valid, it may propagate incomplete or corrupted data to downstream systems. In the context of machine learning, using a partially downloaded model could lead to subtle training errors or inference failures that are difficult to trace back to the source. The ambiguity of the existing status in this script meant that operators had to manually inspect directories to determine if a rerun was safe, defeating the purpose of automation.
Furthermore, the lack of per-item checkpointing makes incident response harder. When a long-running job fails, knowing exactly which items succeeded and which failed is essential for efficient recovery. Writing the results file only at the end means that if the process crashes mid-batch, there is no record of what was already done. This forces administrators to either rerun the entire batch, potentially wasting bandwidth and time, or manually reconstruct the state, which is error-prone. Clear, granular status reporting enables faster troubleshooting and more reliable operational workflows.
What you can do
- Implement verification checks for any existing destinations before skipping them, ensuring that partial downloads are detected and repaired.
- Use per-item checkpointing to record the status of each repository or file as it completes, rather than waiting until the entire batch finishes.
- Distinguish between fresh downloads and previously verified archives in your logs to maintain a clear history of data provenance.
- Consider monitoring disk usage during transfers, not just before, if your workload involves large files and concurrent processes.
- Ensure your scripts exit with clear, distinct error codes that indicate whether the failure was due to a missing resource, a verification error, or an incomplete state.
- Test restart scenarios explicitly by creating fake existing directories and mocking failures to confirm that your recovery logic works as intended.



