logitorch.datasets.utils

Attributes

CURRENT_PATH
DATASETS_FOLDER
DATASETS_ZIP_FOLDER
SPLIT_SETS
SPLIT_SETS_TRAIN_VAL

Functions

download_dataset(→ None)	Function to download datasets
read_json(→ List[Dict[str, Any]])	Function to read a JSON file
read_jsonl(→ List[Dict[str, Any]])	Function to read a JSONL file

Module Contents

logitorch.datasets.utils.download_dataset(url: str, dataset_name: str) → None

Function to download datasets

Parameters:

• url (str) – url of the dataset

• dataset_name (str) – dataset name

Raises:

FileSizeError – an error is raised if the dataset is not downloaded properly

logitorch.datasets.utils.read_json(dataset_path: str) → List[Dict[str, Any]]

Function to read a JSON file

Parameters:

dataset_path (str) – path of the dataset

Returns:

list of JSON objects

Return type:

List[Dict[str, Any]]

logitorch.datasets.utils.read_jsonl(dataset_path: str) → List[Dict[str, Any]]

Function to read a JSONL file

Parameters:

dataset_path (str) – path of the dataset

Returns:

list of JSON objects

Return type:

List[Dict[str, Any]]

logitorch.datasets.utils.CURRENT_PATH = ''

logitorch.datasets.utils.DATASETS_FOLDER = '/logitorch_datasets'

logitorch.datasets.utils.DATASETS_ZIP_FOLDER = '/logitorch_datasets/tmp'

logitorch.datasets.utils.SPLIT_SETS = ['train', 'val', 'test']

logitorch.datasets.utils.SPLIT_SETS_TRAIN_VAL = ['train', 'val']