llmcompressor.transformers.finetune.data.custom

`CustomDataset`

Bases: TextGenerationDataset

Child text generation class for custom local dataset supporting load for csv and json

Parameters:

Name	Type	Description	Default
`dataset_args`	`DatasetArguments`	configuration settings for dataset loading	required
`split`	`str`	split from dataset to load, for instance `test` or `train[:5%]` Can also be set to None to load all the splits	required
`processor`	`Processor`	processor or tokenizer to use on dataset	required

Source code in llmcompressor/transformers/finetune/data/custom.py

@TextGenerationDataset.register(name="custom", alias=["json", "csv"])
class CustomDataset(TextGenerationDataset):
    """
    Child text generation class for custom local dataset supporting load
    for csv and json

    :param dataset_args: configuration settings for dataset loading
    :param split: split from dataset to load, for instance `test` or `train[:5%]`
        Can also be set to None to load all the splits
    :param processor: processor or tokenizer to use on dataset

    """

    pass