logitorch.datasets.utils
========================

.. py:module:: logitorch.datasets.utils


Attributes
----------

.. autoapisummary::

   logitorch.datasets.utils.CURRENT_PATH
   logitorch.datasets.utils.DATASETS_FOLDER
   logitorch.datasets.utils.DATASETS_ZIP_FOLDER
   logitorch.datasets.utils.SPLIT_SETS
   logitorch.datasets.utils.SPLIT_SETS_TRAIN_VAL


Functions
---------

.. autoapisummary::

   logitorch.datasets.utils.download_dataset
   logitorch.datasets.utils.read_json
   logitorch.datasets.utils.read_jsonl


Module Contents
---------------

.. py:function:: download_dataset(url: str, dataset_name: str) -> None

   Function to download datasets

   :param url: url of the dataset
   :type url: str
   :param dataset_name: dataset name
   :type dataset_name: str
   :raises FileSizeError: an error is raised if the dataset is not downloaded properly


.. py:function:: read_json(dataset_path: str) -> List[Dict[str, Any]]

   Function to read a JSON file

   :param dataset_path: path of the dataset
   :type dataset_path: str
   :return: list of JSON objects
   :rtype: List[Dict[str, Any]]


.. py:function:: read_jsonl(dataset_path: str) -> List[Dict[str, Any]]

   Function to read a JSONL file

   :param dataset_path: path of the dataset
   :type dataset_path: str
   :return: list of JSON objects
   :rtype: List[Dict[str, Any]]


.. py:data:: CURRENT_PATH
   :value: ''


.. py:data:: DATASETS_FOLDER
   :value: '/logitorch_datasets'


.. py:data:: DATASETS_ZIP_FOLDER
   :value: '/logitorch_datasets/tmp'


.. py:data:: SPLIT_SETS
   :value: ['train', 'val', 'test']


.. py:data:: SPLIT_SETS_TRAIN_VAL
   :value: ['train', 'val']