Download List

The Synapse Download List (cart) lets you queue files for bulk download via the Synapse web UI or API. Files are downloaded individually rather than packaged into a zip because download lists can exceed 100 GB. Successfully downloaded files are removed from the cart automatically, so interrupted runs are safely resumable.

API Reference

synapseclient.operations.download_list_files_async async

download_list_files_async(download_location: Optional[str] = None, *, parallel: bool = False, max_concurrent: int = 10, synapse_client: Optional[Synapse] = None) -> str

Download all files in the Synapse download list (cart) to a local directory.

Files are downloaded individually. The cart is not packaged into a zip because download lists can exceed 100 GB. Only successfully downloaded files are removed from the cart after the full pass completes, so interrupted runs are safely resumable.

Files that cannot be accessed or fail to download are left in the cart and recorded with an error value in the result manifest.

PARAMETER	DESCRIPTION
download_location	Directory to download files to. Defaults to the current working directory. TYPE: Optional[str] DEFAULT: None
parallel	If True, files are downloaded concurrently up to max_concurrent at a time using asyncio.gather. If False (default), files are downloaded sequentially. TYPE: bool DEFAULT: False
max_concurrent	Maximum number of files to download concurrently when parallel=True. Defaults to 10. Has no effect when parallel=False. TYPE: int DEFAULT: 10
synapse_client	If not passed in and caching was not disabled by Synapse.allow_client_caching(False) this will use the last created instance from the Synapse class constructor. TYPE: Optional[Synapse] DEFAULT: None

RETURNS	DESCRIPTION
str	Path to the result manifest CSV, which contains all original manifest
str	columns plus path (local file path) and error (error message or
str	empty string) columns.

RAISES	DESCRIPTION
SynapseHTTPError	If the manifest async job fails or the cart is empty ("No files available for download").
SynapseError	If the manifest job completes but produces no local file, or if the downloaded CSV has no headers or contains reserved column names ("path" or "error").

Download all files in the cart

  Download all files in the user's download list to a local directory.

import asyncio
from synapseclient import Synapse
from synapseclient.operations import download_list_files_async

async def main():
    syn = Synapse()
    syn.login()

    manifest_path = await download_list_files_async(download_location="./data")

asyncio.run(main())

Source code in synapseclient/operations/download_list_operations.py

async def download_list_files_async(
    download_location: Optional[str] = None,
    *,
    parallel: bool = False,
    max_concurrent: int = 10,
    synapse_client: Optional["Synapse"] = None,
) -> str:
    """Download all files in the Synapse download list (cart) to a local directory.

    Files are downloaded individually. The cart is not packaged into a zip because
    download lists can exceed 100 GB. Only successfully downloaded files are removed
    from the cart after the full pass completes, so interrupted runs are safely
    resumable.

    Files that cannot be accessed or fail to download are left in the cart and
    recorded with an error value in the result manifest.

    Arguments:
        download_location: Directory to download files to. Defaults to the
            current working directory.
        parallel: If True, files are downloaded concurrently up to
            max_concurrent at a time using asyncio.gather. If False
            (default), files are downloaded sequentially.
        max_concurrent: Maximum number of files to download concurrently when
            parallel=True. Defaults to 10. Has no effect when
            parallel=False.
        synapse_client: If not passed in and caching was not disabled by
            Synapse.allow_client_caching(False) this will use the last created
            instance from the Synapse class constructor.

    Returns:
        Path to the result manifest CSV, which contains all original manifest
        columns plus path (local file path) and error (error message or
        empty string) columns.

    Raises:
        SynapseHTTPError: If the manifest async job fails or the cart is empty
            ("No files available for download").
        SynapseError: If the manifest job completes but produces no local file,
            or if the downloaded CSV has no headers or contains reserved column
            names ("path" or "error").

    Example: Download all files in the cart
         
        Download all files in the user's download list to a local directory.
        ```python
        import asyncio
        from synapseclient import Synapse
        from synapseclient.operations import download_list_files_async

        async def main():
            syn = Synapse()
            syn.login()

            manifest_path = await download_list_files_async(download_location="./data")

        asyncio.run(main())
        ```
    """
    from synapseclient import Synapse

    client = Synapse.get_client(synapse_client=synapse_client)

    if download_location is not None:
        download_location = os.path.expandvars(os.path.expanduser(download_location))

    # 1. Fetch the server-generated manifest and read it into memory
    manifest_path = await download_list_manifest_async(synapse_client=client)
    try:
        columns, rows = await asyncio.to_thread(_read_manifest_rows, manifest_path)
    finally:
        os.remove(manifest_path)

    # 2. Validate manifest columns and append result columns
    columns = _validate_and_extend_columns(columns)

    # 3. Download each file in the manifest
    downloaded_files = await _download_all_manifest_files(
        rows=rows,
        download_location=download_location,
        parallel=parallel,
        max_concurrent=max_concurrent,
        synapse_client=client,
    )

    # 4. Write the result manifest with path/error columns
    new_manifest_path = await _save_result_manifest(
        rows=rows,
        columns=columns,
        download_location=download_location,
    )

    # 5. Remove successfully downloaded files from the cart. The Synapse API
    #    requires the (fileEntityId, versionNumber) pair at removal to match
    #    exactly what was used at add time -- a no-version add is only matched
    #    by a no-version remove. The manifest always carries a resolved
    #    versionNumber, so a versioned remove silently fails (returns 0) for
    #    entries that were added without a version. When that happens, retry
    #    the same entity with version_number=None to match the no-version add.
    if downloaded_files:
        for item in downloaded_files:
            removed = await remove_from_download_list_async(
                files=[item],
                synapse_client=client,
            )
            if removed == 0:
                await remove_from_download_list_async(
                    files=[DownloadListItem(file_entity_id=item.file_entity_id)],
                    synapse_client=client,
                )
    else:
        client.logger.warning("A manifest was created, but no files were downloaded")

    return new_manifest_path

synapseclient.operations.download_list_manifest_async async

download_list_manifest_async(*, csv_table_descriptor: Optional[CsvTableDescriptor] = None, destination: str = '.', synapse_client: Optional[Synapse] = None) -> str

Generate and download the manifest CSV for the current cart contents.

Submits an async job to Synapse to generate the manifest, then downloads the resulting CSV. The manifest contains the same columns as the zip manifest downloaded from the Synapse web UI.

PARAMETER	DESCRIPTION
csv_table_descriptor	Optional CsvTableDescriptor controlling the format of the generated CSV (separator, quote character, escape character, line ending, and whether the first line is a header). When omitted the Synapse defaults are used. TYPE: Optional[CsvTableDescriptor] DEFAULT: None
destination	Directory to download the manifest CSV to. Defaults to the current working directory. TYPE: str DEFAULT: '.'
synapse_client	If not passed in and caching was not disabled by Synapse.allow_client_caching(False) this will use the last created instance from the Synapse class constructor. TYPE: Optional[Synapse] DEFAULT: None

RAISES	DESCRIPTION
SynapseError	If the async job completes without producing a manifest.

RETURNS	DESCRIPTION
str	Path to the downloaded manifest CSV.

Get the download list manifest

  Inspect the cart contents before downloading.

import asyncio
from synapseclient import Synapse
from synapseclient.operations import download_list_manifest_async

async def main():
    syn = Synapse()
    syn.login()

    manifest_path = await download_list_manifest_async()

asyncio.run(main())

Source code in synapseclient/operations/download_list_operations.py

async def download_list_manifest_async(
    *,
    csv_table_descriptor: Optional[CsvTableDescriptor] = None,
    destination: str = ".",
    synapse_client: Optional["Synapse"] = None,
) -> str:
    """Generate and download the manifest CSV for the current cart contents.

    Submits an async job to Synapse to generate the manifest, then downloads
    the resulting CSV. The manifest contains the same columns as the zip
    manifest downloaded from the Synapse web UI.

    Arguments:
        csv_table_descriptor: Optional CsvTableDescriptor controlling the
            format of the generated CSV (separator, quote character, escape
            character, line ending, and whether the first line is a header).
            When omitted the Synapse defaults are used.
        destination: Directory to download the manifest CSV to. Defaults to
            the current working directory.
        synapse_client: If not passed in and caching was not disabled by
            Synapse.allow_client_caching(False) this will use the last created
            instance from the Synapse class constructor.

    Raises:
        SynapseError: If the async job completes without producing a manifest.

    Returns:
        Path to the downloaded manifest CSV.

    Example: Get the download list manifest
         
        Inspect the cart contents before downloading.
        ```python
        import asyncio
        from synapseclient import Synapse
        from synapseclient.operations import download_list_manifest_async

        async def main():
            syn = Synapse()
            syn.login()

            manifest_path = await download_list_manifest_async()

        asyncio.run(main())
        ```
    """
    manifest_request = _DownloadListManifestRequest(
        csv_table_descriptor=csv_table_descriptor or CsvTableDescriptor(),
    )
    await manifest_request.send_job_and_wait_async(
        post_exchange_args={"destination": destination},
        synapse_client=synapse_client,
    )
    if manifest_request.manifest_path is None:
        raise SynapseError(
            "Manifest job completed but no local file was produced. "
            "The download from Synapse may have failed silently."
        )
    return manifest_request.manifest_path

synapseclient.operations.download_list_add_async async

download_list_add_async(files: list[DownloadListItem], *, synapse_client: Optional[Synapse] = None) -> int

Add files to the Synapse download list.

If a file is added with no version specified, the latest version will be downloaded.

PARAMETER	DESCRIPTION
files	List of DownloadListItem objects identifying the file versions to add. TYPE: list[DownloadListItem]
synapse_client	If not passed in and caching was not disabled by Synapse.allow_client_caching(False) this will use the last created instance from the Synapse class constructor. TYPE: Optional[Synapse] DEFAULT: None

RETURNS	DESCRIPTION
int	The number of files added.

Add files to the download list

  Add specific file versions to the cart.

import asyncio
from synapseclient import Synapse
from synapseclient.operations import download_list_add_async, DownloadListItem

async def main():
    syn = Synapse()
    syn.login()

    count = await download_list_add_async([
        DownloadListItem(file_entity_id="syn123", version_number=1),
        DownloadListItem(file_entity_id="syn456", version_number=2),
    ])

asyncio.run(main())

Source code in synapseclient/operations/download_list_operations.py

async def download_list_add_async(
    files: list[DownloadListItem],
    *,
    synapse_client: Optional["Synapse"] = None,
) -> int:
    """Add files to the Synapse download list.

    If a file is added with no version specified, the latest version will be downloaded.

    Arguments:
        files: List of DownloadListItem objects identifying the file
            versions to add.
        synapse_client: If not passed in and caching was not disabled by
            Synapse.allow_client_caching(False) this will use the last created
            instance from the Synapse class constructor.

    Returns:
        The number of files added.

    Example: Add files to the download list
         
        Add specific file versions to the cart.
        ```python
        import asyncio
        from synapseclient import Synapse
        from synapseclient.operations import download_list_add_async, DownloadListItem

        async def main():
            syn = Synapse()
            syn.login()

            count = await download_list_add_async([
                DownloadListItem(file_entity_id="syn123", version_number=1),
                DownloadListItem(file_entity_id="syn456", version_number=2),
            ])

        asyncio.run(main())
        ```
    """
    return await add_to_download_list_async(
        files=files,
        synapse_client=synapse_client,
    )

synapseclient.operations.download_list_remove_async async

download_list_remove_async(files: list[DownloadListItem], *, synapse_client: Optional[Synapse] = None) -> int

Remove files from the Synapse download list.

If a file was added with a version specified, then that version must be specified to remove it. If a file was added with no version specified, then no version must be specified to remove it.

PARAMETER	DESCRIPTION
files	List of DownloadListItem objects identifying the file versions to remove. TYPE: list[DownloadListItem]
synapse_client	If not passed in and caching was not disabled by Synapse.allow_client_caching(False) this will use the last created instance from the Synapse class constructor. TYPE: Optional[Synapse] DEFAULT: None

RETURNS	DESCRIPTION
int	The number of files removed.

Remove files from the download list

  Remove specific file versions from the cart.

import asyncio
from synapseclient import Synapse
from synapseclient.operations import download_list_remove_async, DownloadListItem

async def main():
    syn = Synapse()
    syn.login()

    count = await download_list_remove_async([
        DownloadListItem(file_entity_id="syn123", version_number=1),
    ])

asyncio.run(main())

Source code in synapseclient/operations/download_list_operations.py

async def download_list_remove_async(
    files: list[DownloadListItem],
    *,
    synapse_client: Optional["Synapse"] = None,
) -> int:
    """Remove files from the Synapse download list.

    If a file was added with a version specified, then that version must be specified to remove it.
    If a file was added with no version specified, then no version must be specified to remove it.

    Arguments:
        files: List of DownloadListItem objects identifying the file versions to remove.
        synapse_client: If not passed in and caching was not disabled by
            Synapse.allow_client_caching(False) this will use the last created
            instance from the Synapse class constructor.

    Returns:
        The number of files removed.

    Example: Remove files from the download list
         
        Remove specific file versions from the cart.
        ```python
        import asyncio
        from synapseclient import Synapse
        from synapseclient.operations import download_list_remove_async, DownloadListItem

        async def main():
            syn = Synapse()
            syn.login()

            count = await download_list_remove_async([
                DownloadListItem(file_entity_id="syn123", version_number=1),
            ])

        asyncio.run(main())
        ```
    """
    return await remove_from_download_list_async(
        files=files,
        synapse_client=synapse_client,
    )

synapseclient.operations.download_list_clear_async async

download_list_clear_async(*, synapse_client: Optional[Synapse] = None) -> None

Clear all files from the Synapse download list (cart).

PARAMETER	DESCRIPTION
synapse_client	If not passed in and caching was not disabled by Synapse.allow_client_caching(False) this will use the last created instance from the Synapse class constructor. TYPE: Optional[Synapse] DEFAULT: None

Clear the download list

  Remove all files from the cart.

import asyncio
from synapseclient import Synapse
from synapseclient.operations import download_list_clear_async

async def main():
    syn = Synapse()
    syn.login()

    await download_list_clear_async()

asyncio.run(main())

Source code in synapseclient/operations/download_list_operations.py

async def download_list_clear_async(
    *,
    synapse_client: Optional["Synapse"] = None,
) -> None:
    """Clear all files from the Synapse download list (cart).

    Arguments:
        synapse_client: If not passed in and caching was not disabled by
            Synapse.allow_client_caching(False) this will use the last created
            instance from the Synapse class constructor.

    Example: Clear the download list
         
        Remove all files from the cart.
        ```python
        import asyncio
        from synapseclient import Synapse
        from synapseclient.operations import download_list_clear_async

        async def main():
            syn = Synapse()
            syn.login()

            await download_list_clear_async()

        asyncio.run(main())
        ```
    """
    await clear_download_list_async(synapse_client=synapse_client)

---

DownloadListItem

Identifies a specific file version in the download list. Used as input to download_list_add_async and download_list_remove_async.

synapseclient.operations.DownloadListItem dataclass

A single item for a user's download list.

https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/download/DownloadListItem.html

ATTRIBUTE	DESCRIPTION
file_entity_id	Synapse ID of the file entity (e.g. "syn123"). TYPE: str
version_number	Version of the file to target. TYPE: Optional[int]

Source code in synapseclient/operations/download_list_operations.py

@dataclass
class DownloadListItem:
    """A single item for a user's download list.

    <https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/download/DownloadListItem.html>

    Attributes:
        file_entity_id: Synapse ID of the file entity (e.g. "syn123").
        version_number: Version of the file to target.
    """

    file_entity_id: str
    """Synapse ID of the file entity (e.g. "syn123")."""

    version_number: Optional[int] = None
    """Version of the file to target."""

Attributes

file_entity_id instance-attribute

file_entity_id: str

Synapse ID of the file entity (e.g. "syn123").

version_number class-attribute instance-attribute

version_number: Optional[int] = None

Version of the file to target.