Downloading files
Download a single file
hf_hub_download
huggingface_hub.hf_hub_download
< source >( repo_id: strfilename: strsubfolder: typing.Optional[str] = Nonerepo_type: typing.Optional[str] = Nonerevision: typing.Optional[str] = Nonelibrary_name: typing.Optional[str] = Nonelibrary_version: typing.Optional[str] = Nonecache_dir: typing.Union[str, pathlib.Path, NoneType] = Nonelocal_dir: typing.Union[str, pathlib.Path, NoneType] = Noneuser_agent: typing.Union[typing.Dict, str, NoneType] = Noneforce_download: bool = Falseproxies: typing.Optional[typing.Dict] = Noneetag_timeout: float = 10token: typing.Union[bool, str, NoneType] = Nonelocal_files_only: bool = Falseheaders: typing.Optional[typing.Dict[str, str]] = Noneendpoint: typing.Optional[str] = Noneresume_download: typing.Optional[bool] = Noneforce_filename: typing.Optional[str] = Nonelocal_dir_use_symlinks: typing.Union[bool, typing.Literal['auto']] = 'auto' ) β str
Parameters
- repo_id (
str
) β A user or an organization name and a repo name separated by a/
. - filename (
str
) β The name of the file in the repo. - subfolder (
str
, optional) β An optional value corresponding to a folder inside the model repo. - repo_type (
str
, optional) β Set to"dataset"
or"space"
if downloading from a dataset or space,None
or"model"
if downloading from a model. Default isNone
. - revision (
str
, optional) β An optional Git revision id which can be a branch name, a tag, or a commit hash. - library_name (
str
, optional) β The name of the library to which the object corresponds. - library_version (
str
, optional) β The version of the library. - cache_dir (
str
,Path
, optional) β Path to the folder where cached files are stored. - local_dir (
str
orPath
, optional) β If provided, the downloaded file will be placed under this directory. - user_agent (
dict
,str
, optional) β The user-agent info in the form of a dictionary or a string. - force_download (
bool
, optional, defaults toFalse
) β Whether the file should be downloaded even if it already exists in the local cache. - proxies (
dict
, optional) β Dictionary mapping protocol to the URL of the proxy passed torequests.request
. - etag_timeout (
float
, optional, defaults to10
) β When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed torequests.request
. - token (
str
,bool
, optional) β A token to be used for the download.- If
True
, the token is read from the HuggingFace config folder. - If a string, itβs used as the authentication token.
- If
- local_files_only (
bool
, optional, defaults toFalse
) β IfTrue
, avoid downloading the file and return the path to the local cached file if it exists. - headers (
dict
, optional) β Additional headers to be sent with the request.
Returns
str
Local path of file or if networking is off, last version of file cached on disk.
Raises
RepositoryNotFoundError or RevisionNotFoundError or EntryNotFoundError or LocalEntryNotFoundError or EnvironmentError
or OSError
or ValueError
- RepositoryNotFoundError β
If the repository to download from cannot be found. This may be because it doesnβt exist,
or because it is set to
private
and you do not have access. - RevisionNotFoundError β If the revision to download from cannot be found.
- EntryNotFoundError β If the file to download cannot be found.
- LocalEntryNotFoundError β If network is disabled or unavailable and file is not found in cache.
EnvironmentError
β Iftoken=True
but the token cannot be found.OSError
β If ETag cannot be determined.ValueError
β If some parameter value is invalid.
Download a given file if itβs not already present in the local cache.
The new cache file layout looks like this:
- The cache directory contains one subfolder per repo_id (namespaced by repo type)
- inside each repo folder:
- refs is a list of the latest known revision => commit_hash pairs
- blobs contains the actual file blobs (identified by their git-sha or sha256, depending on whether theyβre LFS files or not)
- snapshots contains one subfolder per commit, each βcommitβ contains the subset of the files that have been resolved at that particular commit. Each filename is a symlink to the blob at that particular commit.
[ 96] .
βββ [ 160] models--julien-c--EsperBERTo-small
βββ [ 160] blobs
β βββ [321M] 403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
β βββ [ 398] 7cb18dc9bafbfcf74629a4b760af1b160957a83e
β βββ [1.4K] d7edf6bd2a681fb0175f7735299831ee1b22b812
βββ [ 96] refs
β βββ [ 40] main
βββ [ 128] snapshots
βββ [ 128] 2439f60ef33a0d46d85da5001d52aeda5b00ce9f
β βββ [ 52] README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812
β βββ [ 76] pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
βββ [ 128] bbc77c8132af1cc5cf678da3f1ddf2de43606d48
βββ [ 52] README.md -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e
βββ [ 76] pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
If local_dir
is provided, the file structure from the repo will be replicated in this location. When using this
option, the cache_dir
will not be used and a .cache/huggingface/
folder will be created at the root of local_dir
to store some metadata related to the downloaded files. While this mechanism is not as robust as the main
cache-system, itβs optimized for regularly pulling the latest version of a repository.
hf_hub_url
huggingface_hub.hf_hub_url
< source >( repo_id: strfilename: strsubfolder: typing.Optional[str] = Nonerepo_type: typing.Optional[str] = Nonerevision: typing.Optional[str] = Noneendpoint: typing.Optional[str] = None )
Parameters
- repo_id (
str
) β A namespace (user or an organization) name and a repo name separated by a/
. - filename (
str
) β The name of the file in the repo. - subfolder (
str
, optional) β An optional value corresponding to a folder inside the repo. - repo_type (
str
, optional) β Set to"dataset"
or"space"
if downloading from a dataset or space,None
or"model"
if downloading from a model. Default isNone
. - revision (
str
, optional) β An optional Git revision id which can be a branch name, a tag, or a commit hash.
Construct the URL of a file from the given information.
The resolved address can either be a huggingface.co-hosted url, or a link to Cloudfront (a Content Delivery Network, or CDN) for large files which are more than a few MBs.
Example:
>>> from huggingface_hub import hf_hub_url
>>> hf_hub_url(
... repo_id="julien-c/EsperBERTo-small", filename="pytorch_model.bin"
... )
'https://huggingface.co/julien-c/EsperBERTo-small/resolve/main/pytorch_model.bin'
Notes:
Cloudfront is replicated over the globe so downloads are way faster for the end user (and it also lowers our bandwidth costs).
Cloudfront aggressively caches files by default (default TTL is 24 hours), however this is not an issue here because we implement a git-based versioning system on huggingface.co, which means that we store the files on S3/Cloudfront in a content-addressable way (i.e., the file name is its hash). Using content-addressable filenames means cache canβt ever be stale.
In terms of client-side caching from this library, we base our caching
on the objectsβ entity tag (ETag
), which is an identifier of a
specific version of a resource [1]_. An objectβs ETag is: its git-sha1
if stored in git, or its sha256 if stored in git-lfs.
References:
Download a snapshot of the repo
huggingface_hub.snapshot_download
< source >( repo_id: strrepo_type: typing.Optional[str] = Nonerevision: typing.Optional[str] = Nonecache_dir: typing.Union[str, pathlib.Path, NoneType] = Nonelocal_dir: typing.Union[str, pathlib.Path, NoneType] = Nonelibrary_name: typing.Optional[str] = Nonelibrary_version: typing.Optional[str] = Noneuser_agent: typing.Union[typing.Dict, str, NoneType] = Noneproxies: typing.Optional[typing.Dict] = Noneetag_timeout: float = 10force_download: bool = Falsetoken: typing.Union[bool, str, NoneType] = Nonelocal_files_only: bool = Falseallow_patterns: typing.Union[typing.List[str], str, NoneType] = Noneignore_patterns: typing.Union[typing.List[str], str, NoneType] = Nonemax_workers: int = 8tqdm_class: typing.Optional[tqdm.asyncio.tqdm_asyncio] = Noneheaders: typing.Optional[typing.Dict[str, str]] = Noneendpoint: typing.Optional[str] = Nonelocal_dir_use_symlinks: typing.Union[bool, typing.Literal['auto']] = 'auto'resume_download: typing.Optional[bool] = None ) β str
Parameters
- repo_id (
str
) β A user or an organization name and a repo name separated by a/
. - repo_type (
str
, optional) β Set to"dataset"
or"space"
if downloading from a dataset or space,None
or"model"
if downloading from a model. Default isNone
. - revision (
str
, optional) β An optional Git revision id which can be a branch name, a tag, or a commit hash. - cache_dir (
str
,Path
, optional) β Path to the folder where cached files are stored. - local_dir (
str
orPath
, optional) β If provided, the downloaded files will be placed under this directory. - library_name (
str
, optional) β The name of the library to which the object corresponds. - library_version (
str
, optional) β The version of the library. - user_agent (
str
,dict
, optional) β The user-agent info in the form of a dictionary or a string. - proxies (
dict
, optional) β Dictionary mapping protocol to the URL of the proxy passed torequests.request
. - etag_timeout (
float
, optional, defaults to10
) β When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed torequests.request
. - force_download (
bool
, optional, defaults toFalse
) β Whether the file should be downloaded even if it already exists in the local cache. - token (
str
,bool
, optional) β A token to be used for the download.- If
True
, the token is read from the HuggingFace config folder. - If a string, itβs used as the authentication token.
- If
- headers (
dict
, optional) β Additional headers to include in the request. Those headers take precedence over the others. - local_files_only (
bool
, optional, defaults toFalse
) β IfTrue
, avoid downloading the file and return the path to the local cached file if it exists. - allow_patterns (
List[str]
orstr
, optional) β If provided, only files matching at least one pattern are downloaded. - ignore_patterns (
List[str]
orstr
, optional) β If provided, files matching any of the patterns are not downloaded. - max_workers (
int
, optional) β Number of concurrent threads to download files (1 thread = 1 file download). Defaults to 8. - tqdm_class (
tqdm
, optional) β If provided, overwrites the default behavior for the progress bar. Passed argument must inherit fromtqdm.auto.tqdm
or at least mimic its behavior. Note that thetqdm_class
is not passed to each individual download. Defaults to the custom HF progress bar that can be disabled by settingHF_HUB_DISABLE_PROGRESS_BARS
environment variable.
Returns
str
folder path of the repo snapshot.
Raises
RepositoryNotFoundError or RevisionNotFoundError or EnvironmentError
or OSError
or ValueError
- RepositoryNotFoundError β
If the repository to download from cannot be found. This may be because it doesnβt exist,
or because it is set to
private
and you do not have access. - RevisionNotFoundError β If the revision to download from cannot be found.
EnvironmentError
β Iftoken=True
and the token cannot be found.OSError
β if ETag cannot be determined.ValueError
β if some parameter value is invalid.
Download repo files.
Download a whole snapshot of a repoβs files at the specified revision. This is useful when you want all files from
a repo, because you donβt know which ones you will need a priori. All files are nested inside a folder in order
to keep their actual filename relative to that folder. You can also filter which files to download using
allow_patterns
and ignore_patterns
.
If local_dir
is provided, the file structure from the repo will be replicated in this location. When using this
option, the cache_dir
will not be used and a .cache/huggingface/
folder will be created at the root of local_dir
to store some metadata related to the downloaded files. While this mechanism is not as robust as the main
cache-system, itβs optimized for regularly pulling the latest version of a repository.
An alternative would be to clone the repo but this requires git and git-lfs to be installed and properly configured. It is also not possible to filter which files to download when cloning a repository using git.
Get metadata about a file
get_hf_file_metadata
huggingface_hub.get_hf_file_metadata
< source >( url: strtoken: typing.Union[bool, str, NoneType] = Noneproxies: typing.Optional[typing.Dict] = Nonetimeout: typing.Optional[float] = 10library_name: typing.Optional[str] = Nonelibrary_version: typing.Optional[str] = Noneuser_agent: typing.Union[typing.Dict, str, NoneType] = Noneheaders: typing.Optional[typing.Dict[str, str]] = None )
Parameters
- url (
str
) β File url, for example returned by hf_hub_url(). - token (
str
orbool
, optional) β A token to be used for the download.- If
True
, the token is read from the HuggingFace config folder. - If
False
orNone
, no token is provided. - If a string, itβs used as the authentication token.
- If
- proxies (
dict
, optional) β Dictionary mapping protocol to the URL of the proxy passed torequests.request
. - timeout (
float
, optional, defaults to 10) β How many seconds to wait for the server to send metadata before giving up. - library_name (
str
, optional) β The name of the library to which the object corresponds. - library_version (
str
, optional) β The version of the library. - user_agent (
dict
,str
, optional) β The user-agent info in the form of a dictionary or a string. - headers (
dict
, optional) β Additional headers to be sent with the request.
Fetch metadata of a file versioned on the Hub for a given url.
HfFileMetadata
class huggingface_hub.HfFileMetadata
< source >( commit_hash: typing.Optional[str]etag: typing.Optional[str]location: strsize: typing.Optional[int] )
Parameters
- commit_hash (
str
, optional) β The commit_hash related to the file. - etag (
str
, optional) β Etag of the file on the server. - location (
str
) β Location where to download the file. Can be a Hub url or not (CDN). - size (
size
) β Size of the file. In case of an LFS file, contains the size of the actual LFS file, not the pointer.
Data structure containing information about a file versioned on the Hub.
Returned by get_hf_file_metadata() based on a URL.
Caching
The methods displayed above are designed to work with a caching system that prevents re-downloading files. The caching system was updated in v0.8.0 to become the central cache-system shared across libraries that depend on the Hub.
Read the cache-system guide for a detailed presentation of caching at at HF.
< > Update on GitHub