load-mnist ( -- data-set )

Vocabulary

machine-learning.data-sets

Inputs

None

Outputs

data-set

an object

Definition

USING: arrays combinators concurrency.combinators http.download
io.directories io.encodings.binary io.files io.pathnames kernel
sequences ;

IN: machine-learning.data-sets

: load-mnist ( -- data-set )
    datasets-path dup make-directories [
        {
            "https://github.com/golbin/TensorFlow-MNIST/raw/master/mnist/d..."
            "https://github.com/golbin/TensorFlow-MNIST/raw/master/mnist/d..."
            "https://github.com/golbin/TensorFlow-MNIST/raw/master/mnist/d..."
            "https://github.com/golbin/TensorFlow-MNIST/raw/master/mnist/d..."
        } [ [ download-once-into ] parallel-each ] [
            [
                dup file-stem file-exists?
                [ drop ] [ file-name gzip-decompress-file ] if
            ] each
        ] [ [ file-stem binary file-contents ] map ] tri first4 
        {
            [ mnist-data>array ]
            [ mnist-labels>array ]
            [ mnist-data>array ]
            [ mnist-labels>array ]
        } spread 4array
    ] with-directory ;