barrage.dataset package

Submodules

barrage.dataset.core module

class barrage.dataset.core.RecordAugmentor(funcs)[source]

Bases: object

Class for applying a list of data augmentation functions to a data record.

Parameters

funcs (List[dict]) – list[dict], list of augmentation functions {“import”: “python_path”, “params”: {…}}.

augment(data_record)[source]

Apply augmentation to a train data record.

Parameters

data_record (Tuple[Dict[str, Union[ndarray, float]], …]) – DataRecord, data record.

Return type

Tuple[Dict[str, Union[ndarray, float]], …]

Returns

DataRecord, augmented data record.

static reduce_compose(*funcs)[source]

Compose a list of functions into a single function.

barrage.dataset.core.batchify_data_records(data_records)[source]

Stack a list of DataRecord into BatchRecord. This process converts a list of tuples comprising of dicts {str: float/array} into tuples of dict {str: array}. Float/array is concatenated along the first dimension. See Example.

Parameters

data_records (List[Tuple[Dict[str, Union[ndarray, float]], …]]) – list[DataRecord], list of individual data records.

Return type

Tuple[Dict[str, ndarray], …]

Returns

BatchDataRecords, batch data records.

Example:

data_record_1 = ({"input_1": 1, "input_2": 2}, {"output_1": 3})
data_record_2 = ({"input_1": 2, "input_2": 4}, {"output_1": 6})
batch_data_records = (
    {"input_1": arr([1, 2], "input_2": arr([2, 4])},
    {"output_1": arr([3, 6])}
)
barrage.dataset.core.batchify_network_output(network_output, output_names)[source]

Convert network output scores to BatchRecordScores. This process converts a single numpy array or list of numpy arrays into a list of dictionaries. See example.

Parameters

network_output (Union[ndarray, List[ndarray]]) – union[np.ndarray, list[np.ndarray], network output.

Return type

List[Dict[str, ndarray]]

Returns

BatchRecordScores, batch scores.

Example:

network_output == np.array([[1], [2]])
output_names = ["y"]
batch_scores = [{"y": np.array([1])}, {"y": np.array([2])}]

barrage.dataset.loader module

class barrage.dataset.loader.KeySelector(mode, inputs, outputs, sample_weights=None)[source]

Bases: barrage.api.RecordLoader

Record loader for directly transforming keys from a record into a data record.

Parameters

  • mode (RecordMode) – RecordMode, load mode.

  • inputs (dict) – dict, {input: key or [keys], …}.

  • outputs (dict) – dict, {output: key or [keys], …}.

  • sample_weights (Optional[dict]) – dict or None (OPTIONAL), {output: key, …}.

Raises

TypeError, invalid params. –

load(record)[source]

Load a record by selecting keys corresponding to inputs, outputs, and maybe sample weights.

Parameters

record (Dict[str, Any]) – Record, record.

Return type

Tuple[Dict[str, Union[ndarray, float]], …]

Returns

DataRecord, data record.

barrage.dataset.transformer module

class barrage.dataset.transformer.IdentityTransformer(mode, loader, **params)[source]

Bases: barrage.api.RecordTransformer

Default transformer that does nothing (identity transform) that ensures every dataset has a transformer.

fit(records)[source]

Pass - no transform to fit.

Parameters

records (List[Dict[str, Any]]) – Records, records.

load(path)[source]

Pass - no objects to load.

Parameters

path (str) – str.

postprocess(score)[source]

Identity - return the record score unchanged.

Parameters

score (Dict[str, ndarray]) – RecordScore, record output from net.

Return type

Dict[str, ndarray]

Returns

RecordScore, record output from net.

save(path)[source]

Pass - no objects to save.

Parameters

path (str) – str.

transform(data_record)[source]

Identity - return the original data record unchanged.

Parameters

data_record (Tuple[Dict[str, Union[ndarray, float]], …]) – DataRecord, data record.

Return type

Tuple[Dict[str, Union[ndarray, float]], …]

Returns

DataRecord, data record.

Module contents