epsman.repo.remoteUpload module

epsman

Local python script for job upload to repo.

Can be called from Fabric for remote run case, only requires standard libs + local nbDetails file defining uploads.

Currently duplicates some functions in _repo.py, in stripped-down form.

20/01/20 Testing for Zenodo uploads. Issue with files >100Mb… drops out with errors, but not messages.

09/01/20 v1

epsman.repo.remoteUpload.convert_bytes(num)

This function will convert bytes to MB…. GB… etc

epsman.repo.remoteUpload.readNBdetailsJSON(jsonProcFile)

Read previously written nbDetails dictionary from JSON file.

See _repo.readNBdetailsJSON() for local version.

epsman.repo.remoteUpload.splitArchFiles(nbDetails, key, dryRun=True, chunk=90, verbose=True)

Basic routine to split existing archive files into chunks for upload.

Split existing archives into size = chunk (MB) files using system zip package. See, e.g., https://serverfault.com/questions/760337/how-to-zip-files-with-a-size-limit/760341

TODO: replace with better logic…? Package files for E sets to avoid large archives? Use Tar?

epsman.repo.remoteUpload.uploadRepoFiles(nbDetails, key, ACCESS_TOKEN, dryRun=True)

Upload files to repo (from local machine)

See _repo.uploadRepoFiles() for local version.

epsman.repo.remoteUpload.writeNBdetailsJSON(jsonProcFile, nbDetails)

Write nbDetails dictionary to JSON file.

See _repo.writeNBdetailsJSON() for local version.