Tutorial 05: public data training (classification)

Posted on Wed 06 December 2017 in misc by Kazuhiro Terao

In this notebook we train a simple, 10-layers deep convolutional neural network for classifying 5 particle images in a simulated LArTPC detecotr available from the public dataset. We use tensorflow to train the network and larcv_threadio to fetch data from larcv files. If you are completely unfamiliar with larcv_threadio, go look at this quick start. First let's prepare data samples. For the setup of this example, I need to prepare practice_train_10k.root and practice_test_10k.root in the current directory. Let us make symbolic links.

In [1]:
%%bash
# Preparation: make symbolic links for practice_train_10k.root and practice_test_10k.root
PRACTICE_FILE_DIR=../..
ln -sf $PRACTICE_FILE_DIR/practice_train_5k.root ./train.root
ln -sf $PRACTICE_FILE_DIR/practice_test_5k.root ./test.root

Imports

In [3]:
from larcv import larcv
from larcv.dataloader2 import larcv_threadio
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
import os,sys,time

# tensorflow/gpu start-up configuration
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
%env CUDA_DEVICE_ORDER=PCI_BUS_ID
%env CUDA_VISIBLE_DEVICES=2
import tensorflow as tf
env: CUDA_DEVICE_ORDER=PCI_BUS_ID
env: CUDA_VISIBLE_DEVICES=2

We set os.environ['TF_CPP_MIN_LOG_LEVEL'] to suppress lots of non-error (standard) output from tensorflow because it can overwhelm ipython's capability to fetch stdout stream.

Configurations

Next, let's define configuration variables.

In [5]:
TUTORIAL_DIR     = '..'
TRAIN_IO_CONFIG  = os.path.join(TUTORIAL_DIR, 'tf/io_train.cfg')
TEST_IO_CONFIG   = os.path.join(TUTORIAL_DIR, 'tf/io_test.cfg' )
TRAIN_BATCH_SIZE = 50
TEST_BATCH_SIZE  = 100
LOGDIR           = 'log'
ITERATIONS       = 5000
SAVE_SUMMARY     = 20
SAVE_WEIGHTS     = 100

# Check log directory is empty
train_logdir = os.path.join(LOGDIR,'train')
test_logdir  = os.path.join(LOGDIR,'test')
if not os.path.isdir(train_logdir): os.makedirs(train_logdir)
if not os.path.isdir(test_logdir):  os.makedirs(test_logdir)
if len(os.listdir(train_logdir)) or len(os.listdir(test_logdir)):
  sys.stderr.write('Error: train or test log dir not empty...\n')
  raise OSError

The top block defines a set of constants in capitalized letters. The bottom part is simply checking if the directories where we will store the network training logs are empty or not (so that we won't mix with the previous attempt). So what do the constants do?

  • TUTORIAL_DIR ... points to the top-level directory of the larcv-tutorial repostitory.
  • TRAIN_IO_CONFIG ... a configuration file for larcv_threadio to read data for training.
  • TEST_IO_CONFIG ... a configuration file for larcv_threadio to read data for testing.
  • TRAIN_BATCH_SIZE ... a number of images (batch) to be used to calculate the average gradients for updating the network's weights.
  • TEST_BATCH_SIZE ... a number of images to be used to calculate the average accuracy using test data set.
  • LOGDIR ... the top-level directory to save the tensorboard logs.
  • ITERATIONS ... the total number of steps (batches) to train the network.
  • SAVE_SUMMARY ... a period in a training step count to save the log (tensorboard summaries).
  • SAVE_WEIGHTS ... a period in a training step count to save the network's weights.

Configure data reader

We prepare two data reader instances: one for training and another for testing the network. Testing is not absolutely needed but we try here to just cover in this example. We don't go in details of how larcv_threadio works here since there is a dedicated tutorial for that.

In [6]:
#
# Step 0: IO
#
# for "train" data set
train_io = larcv_threadio()  # create io interface
train_io_cfg = {'filler_name' : 'TrainIO',
                'verbosity'   : 0,
                'filler_cfg'  : TRAIN_IO_CONFIG}
train_io.configure(train_io_cfg)   # configure
train_io.start_manager(TRAIN_BATCH_SIZE) # start read thread
time.sleep(2)
train_io.next()

# for "test" data set
test_io = larcv_threadio()   # create io interface
test_io_cfg = {'filler_name' : 'TestIO',
               'verbosity'   : 0,
               'filler_cfg'  : TEST_IO_CONFIG}
test_io.configure(test_io_cfg)   # configure
test_io.start_manager(TEST_BATCH_SIZE) # start read thread
time.sleep(2)
test_io.next()
 setting verbosity 3
 setting verbosity 3

Defining a network

Let's construct a simple network for this exercise. We use 5x2 convolution layers with max-pooling operation followed after every 2 convolution layers except the last layer is average-pooling.

In [7]:
#
# Step 1: Define network
#
import tensorflow.contrib.slim as slim
import tensorflow.python.platform

def build(input_tensor, num_class=4, trainable=True, debug=True):

    net = input_tensor
    if debug: print('input tensor:', input_tensor.shape)

    filters = 32
    num_modules = 5
    with tf.variable_scope('conv'):
        for step in xrange(5):
            stride = 2
            if step: stride = 1
            net = slim.conv2d(inputs        = net,        # input tensor
                              num_outputs   = filters,    # number of filters (neurons) = # of output feature maps
                              kernel_size   = [3,3],      # kernel size
                              stride        = stride,     # stride size
                              trainable     = trainable,  # train or inference
                              activation_fn = tf.nn.relu, # relu
                              scope         = 'conv%da_conv' % step)

            net = slim.conv2d(inputs        = net,        # input tensor
                              num_outputs   = filters,    # number of filters (neurons) = # of output feature maps
                              kernel_size   = [3,3],      # kernel size
                              stride        = 1,          # stride size
                              trainable     = trainable,  # train or inference
                              activation_fn = tf.nn.relu, # relu
                              scope         = 'conv%db_conv' % step)
            if (step+1) < num_modules:
                net = slim.max_pool2d(inputs      = net,    # input tensor
                                      kernel_size = [2,2],  # kernel size
                                      stride      = 2,      # stride size
                                      scope       = 'conv%d_pool' % step)

            else:
                net = tf.layers.average_pooling2d(inputs = net,
                                                  pool_size = [net.get_shape()[-2].value,net.get_shape()[-3].value],
                                                  strides = 1,
                                                  padding = 'valid',
                                                  name = 'conv%d_pool' % step)
            filters *= 2

            if debug: print('After step',step,'shape',net.shape)

    with tf.variable_scope('final'):
        net = slim.flatten(net, scope='flatten')

        if debug: print('After flattening', net.shape)

        net = slim.fully_connected(net, int(num_class), scope='final_fc')

        if debug: print('After final_fc', net.shape)

    return net

Build the network

Build the network and define loss, accuracy metrics and our solver. Any optimizer should work but you may have to tune the parameters by yourself. Here, we use RMSPropOptimizer with base learning rate 0.0005 with no justification. Note we add minimal set of tensorflow variables into tf.summary to demonstrate later the tensorboard, a dedicated monitoring/visualization tool for network training with tensorflow.

In [8]:
#
# Step 2: Build network + define loss & solver
#
# retrieve dimensions of data for network construction
dim_data  = train_io.fetch_data('train_image').dim()
dim_label = train_io.fetch_data('train_label').dim()
# define place holders
data_tensor    = tf.placeholder(tf.float32, [None, dim_data[1] * dim_data[2] * dim_data[3]], name='image')
label_tensor   = tf.placeholder(tf.float32, [None, dim_label[1]], name='label')
data_tensor_2d = tf.reshape(data_tensor, [-1,dim_data[1],dim_data[2],dim_data[3]],name='image_reshape')

# Let's keep 10 random set of images in the log
tf.summary.image('input',data_tensor_2d,10)
# build net
net = build(input_tensor=data_tensor_2d, num_class=dim_label[1], trainable=True, debug=False)
# Define accuracy
with tf.name_scope('accuracy'):
    correct_prediction = tf.equal(tf.argmax(net,1), tf.argmax(label_tensor,1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    tf.summary.scalar('accuracy', accuracy)
# Define loss + backprop as training step
with tf.name_scope('train'):
    cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=label_tensor, logits=net))
    tf.summary.scalar('cross_entropy',cross_entropy)
    train_step = tf.train.RMSPropOptimizer(0.00005).minimize(cross_entropy)

Defining tensorflow IO

In the next cell we define tensorflow's IO

  • merged_summary ... is tensorflow operation to create summaries to be written into a log file for tensorboard.
  • writer_train ... writes monitoring data for training data sample into a log file.
  • writer_test ... is same as writer_train except it is for testing data sample.
  • saver ... is a handle to store the state of the network = trained network variable values (weights, biases, etc.).
In [9]:
#                                                                                                                                      
# Step 3: weight saver & summary writer                                                                                                
#                                                                                                                                      
# Create a bandle of summary                                                                                                           
merged_summary=tf.summary.merge_all()
# Create a session                                                                                                                     
sess = tf.InteractiveSession()
# Initialize variables                                                                                                                 
sess.run(tf.global_variables_initializer())
# Create a summary writer handle                                                                                                       
writer_train=tf.summary.FileWriter(train_logdir)
writer_train.add_graph(sess.graph)
writer_test=tf.summary.FileWriter(test_logdir)
writer_test.add_graph(sess.graph)
# Create weights saver                                                                                                                 
saver = tf.train.Saver()

Train!

In [10]:
#
# Step 4: Run training loop
#
for i in range(ITERATIONS):

    train_data  = train_io.fetch_data('train_image').data()
    train_label = train_io.fetch_data('train_label').data()

    feed_dict = { data_tensor  : train_data,
                  label_tensor : train_label }

    loss, acc, _ = sess.run([cross_entropy, accuracy, train_step], feed_dict=feed_dict)

    if (i+1)%SAVE_SUMMARY == 0:
        # Save train log
        sys.stdout.write('Training in progress @ step %d loss %g accuracy %g          \n' % (i,loss,acc))
        sys.stdout.flush()
        s = sess.run(merged_summary, feed_dict=feed_dict)
        writer_train.add_summary(s,i)
    
        # Calculate & save test log
        test_data  = test_io.fetch_data('test_image').data()
        test_label = test_io.fetch_data('test_label').data()
        feed_dict  = { data_tensor  : test_data,
                       label_tensor : test_label }
        loss, acc = sess.run([cross_entropy, accuracy], feed_dict=feed_dict)
        sys.stdout.write('Testing in progress @ step %d loss %g accuracy %g          \n' % (i,loss,acc))
        sys.stdout.flush()
        s = sess.run(merged_summary, feed_dict=feed_dict)
        writer_test.add_summary(s,i)
        
        test_io.next()

    train_io.next()

    if (i+1)%SAVE_WEIGHTS == 0:
        ssf_path = saver.save(sess,'weights/toynet',global_step=i)
        print('saved @',ssf_path)

# inform log directory
print()
print('Run `tensorboard --logdir=%s` in terminal to see the results.' % LOGDIR)
train_io.reset()
test_io.reset()
Training in progress @ step 19 loss 1.60804 accuracy 0.22          
Testing in progress @ step 19 loss 1.61259 accuracy 0.19          
Training in progress @ step 39 loss 1.59938 accuracy 0.28          
Testing in progress @ step 39 loss 1.61911 accuracy 0.14          
Training in progress @ step 59 loss 1.60463 accuracy 0.1          
Testing in progress @ step 59 loss 1.60371 accuracy 0.27          
Training in progress @ step 79 loss 1.60484 accuracy 0.18          
Testing in progress @ step 79 loss 1.61089 accuracy 0.13          
Training in progress @ step 99 loss 1.56293 accuracy 0.28          
Testing in progress @ step 99 loss 1.57563 accuracy 0.24          
saved @ weights/toynet-99
Training in progress @ step 119 loss 1.58767 accuracy 0.24          
Testing in progress @ step 119 loss 1.61108 accuracy 0.19          
Training in progress @ step 139 loss 1.55079 accuracy 0.3          
Testing in progress @ step 139 loss 1.56534 accuracy 0.34          
Training in progress @ step 159 loss 1.41064 accuracy 0.4          
Testing in progress @ step 159 loss 1.51217 accuracy 0.37          
Training in progress @ step 179 loss 1.49059 accuracy 0.42          
Testing in progress @ step 179 loss 1.4979 accuracy 0.3          
Training in progress @ step 199 loss 1.56263 accuracy 0.24          
Testing in progress @ step 199 loss 1.49698 accuracy 0.34          
saved @ weights/toynet-199
Training in progress @ step 219 loss 1.42051 accuracy 0.38          
Testing in progress @ step 219 loss 1.40418 accuracy 0.36          
Training in progress @ step 239 loss 1.42 accuracy 0.34          
Testing in progress @ step 239 loss 1.35263 accuracy 0.42          
Training in progress @ step 259 loss 1.23509 accuracy 0.52          
Testing in progress @ step 259 loss 1.34543 accuracy 0.35          
Training in progress @ step 279 loss 1.33113 accuracy 0.46          
Testing in progress @ step 279 loss 1.30162 accuracy 0.49          
Training in progress @ step 299 loss 1.63349 accuracy 0.3          
Testing in progress @ step 299 loss 1.46306 accuracy 0.38          
saved @ weights/toynet-299
Training in progress @ step 319 loss 1.31014 accuracy 0.46          
Testing in progress @ step 319 loss 1.34693 accuracy 0.44          
Training in progress @ step 339 loss 1.03612 accuracy 0.66          
Testing in progress @ step 339 loss 1.45878 accuracy 0.41          
Training in progress @ step 359 loss 1.23237 accuracy 0.5          
Testing in progress @ step 359 loss 1.22048 accuracy 0.52          
Training in progress @ step 379 loss 1.31127 accuracy 0.46          
Testing in progress @ step 379 loss 1.29868 accuracy 0.45          
Training in progress @ step 399 loss 1.49949 accuracy 0.32          
Testing in progress @ step 399 loss 1.27982 accuracy 0.48          
saved @ weights/toynet-399
Training in progress @ step 419 loss 1.15013 accuracy 0.56          
Testing in progress @ step 419 loss 1.27892 accuracy 0.45          
Training in progress @ step 439 loss 0.865968 accuracy 0.66          
Testing in progress @ step 439 loss 1.15786 accuracy 0.5          
Training in progress @ step 459 loss 1.54451 accuracy 0.38          
Testing in progress @ step 459 loss 1.38144 accuracy 0.41          
Training in progress @ step 479 loss 1.20095 accuracy 0.48          
Testing in progress @ step 479 loss 1.19787 accuracy 0.46          
Training in progress @ step 499 loss 1.06143 accuracy 0.48          
Testing in progress @ step 499 loss 1.31291 accuracy 0.45          
saved @ weights/toynet-499
Training in progress @ step 519 loss 1.20675 accuracy 0.48          
Testing in progress @ step 519 loss 1.13593 accuracy 0.54          
Training in progress @ step 539 loss 0.780554 accuracy 0.66          
Testing in progress @ step 539 loss 1.21183 accuracy 0.48          
Training in progress @ step 559 loss 1.10059 accuracy 0.58          
Testing in progress @ step 559 loss 1.07109 accuracy 0.5          
Training in progress @ step 579 loss 1.13958 accuracy 0.5          
Testing in progress @ step 579 loss 1.25944 accuracy 0.45          
Training in progress @ step 599 loss 0.982788 accuracy 0.56          
Testing in progress @ step 599 loss 1.16895 accuracy 0.47          
saved @ weights/toynet-599
Training in progress @ step 619 loss 1.16153 accuracy 0.54          
Testing in progress @ step 619 loss 1.23791 accuracy 0.54          
Training in progress @ step 639 loss 1.1081 accuracy 0.46          
Testing in progress @ step 639 loss 1.15263 accuracy 0.55          
Training in progress @ step 659 loss 1.02257 accuracy 0.62          
Testing in progress @ step 659 loss 1.12234 accuracy 0.52          
Training in progress @ step 679 loss 1.1435 accuracy 0.5          
Testing in progress @ step 679 loss 1.13025 accuracy 0.43          
Training in progress @ step 699 loss 1.67424 accuracy 0.3          
Testing in progress @ step 699 loss 1.36547 accuracy 0.43          
saved @ weights/toynet-699
Training in progress @ step 719 loss 1.21574 accuracy 0.56          
Testing in progress @ step 719 loss 1.27319 accuracy 0.49          
Training in progress @ step 739 loss 1.10817 accuracy 0.48          
Testing in progress @ step 739 loss 1.06292 accuracy 0.6          
Training in progress @ step 759 loss 0.988775 accuracy 0.62          
Testing in progress @ step 759 loss 1.02028 accuracy 0.5          
Training in progress @ step 779 loss 1.07787 accuracy 0.58          
Testing in progress @ step 779 loss 1.07664 accuracy 0.48          
Training in progress @ step 799 loss 0.924829 accuracy 0.58          
Testing in progress @ step 799 loss 1.16148 accuracy 0.51          
saved @ weights/toynet-799
Training in progress @ step 819 loss 1.02027 accuracy 0.58          
Testing in progress @ step 819 loss 1.06231 accuracy 0.44          
Training in progress @ step 839 loss 1.08011 accuracy 0.48          
Testing in progress @ step 839 loss 0.939656 accuracy 0.51          
Training in progress @ step 859 loss 1.09139 accuracy 0.56          
Testing in progress @ step 859 loss 1.40887 accuracy 0.48          
Training in progress @ step 879 loss 1.06985 accuracy 0.54          
Testing in progress @ step 879 loss 0.946764 accuracy 0.55          
Training in progress @ step 899 loss 1.26168 accuracy 0.4          
Testing in progress @ step 899 loss 1.02387 accuracy 0.55          
saved @ weights/toynet-899
Training in progress @ step 919 loss 0.985469 accuracy 0.58          
Testing in progress @ step 919 loss 0.913799 accuracy 0.62          
Training in progress @ step 939 loss 0.851035 accuracy 0.64          
Testing in progress @ step 939 loss 0.912754 accuracy 0.56          
Training in progress @ step 959 loss 0.919744 accuracy 0.66          
Testing in progress @ step 959 loss 0.927394 accuracy 0.55          
Training in progress @ step 979 loss 0.888268 accuracy 0.62          
Testing in progress @ step 979 loss 0.854074 accuracy 0.6          
Training in progress @ step 999 loss 0.792844 accuracy 0.62          
Testing in progress @ step 999 loss 0.876297 accuracy 0.62          
saved @ weights/toynet-999
Training in progress @ step 1019 loss 0.923724 accuracy 0.58          
Testing in progress @ step 1019 loss 0.982102 accuracy 0.52          
Training in progress @ step 1039 loss 0.85751 accuracy 0.64          
Testing in progress @ step 1039 loss 1.02859 accuracy 0.56          
Training in progress @ step 1059 loss 0.91098 accuracy 0.62          
Testing in progress @ step 1059 loss 0.747974 accuracy 0.64          
Training in progress @ step 1079 loss 0.854377 accuracy 0.6          
Testing in progress @ step 1079 loss 0.921234 accuracy 0.6          
Training in progress @ step 1099 loss 1.16292 accuracy 0.38          
Testing in progress @ step 1099 loss 0.960073 accuracy 0.52          
saved @ weights/toynet-1099
Training in progress @ step 1119 loss 0.845744 accuracy 0.6          
Testing in progress @ step 1119 loss 0.912497 accuracy 0.59          
Training in progress @ step 1139 loss 0.569651 accuracy 0.7          
Testing in progress @ step 1139 loss 0.772228 accuracy 0.63          
Training in progress @ step 1159 loss 0.897686 accuracy 0.62          
Testing in progress @ step 1159 loss 0.799776 accuracy 0.64          
Training in progress @ step 1179 loss 0.842747 accuracy 0.62          
Testing in progress @ step 1179 loss 1.09458 accuracy 0.52          
Training in progress @ step 1199 loss 1.40838 accuracy 0.32          
Testing in progress @ step 1199 loss 0.960923 accuracy 0.61          
saved @ weights/toynet-1199
Training in progress @ step 1219 loss 0.837344 accuracy 0.74          
Testing in progress @ step 1219 loss 0.89485 accuracy 0.63          
Training in progress @ step 1239 loss 0.627333 accuracy 0.74          
Testing in progress @ step 1239 loss 0.902356 accuracy 0.57          
Training in progress @ step 1259 loss 0.889504 accuracy 0.68          
Testing in progress @ step 1259 loss 0.889847 accuracy 0.59          
Training in progress @ step 1279 loss 0.780373 accuracy 0.64          
Testing in progress @ step 1279 loss 0.807954 accuracy 0.66          
Training in progress @ step 1299 loss 0.746441 accuracy 0.62          
Testing in progress @ step 1299 loss 0.794658 accuracy 0.65          
saved @ weights/toynet-1299
Training in progress @ step 1319 loss 0.871986 accuracy 0.58          
Testing in progress @ step 1319 loss 0.988803 accuracy 0.55          
Training in progress @ step 1339 loss 0.60549 accuracy 0.74          
Testing in progress @ step 1339 loss 0.917192 accuracy 0.54          
Training in progress @ step 1359 loss 0.913408 accuracy 0.54          
Testing in progress @ step 1359 loss 0.850111 accuracy 0.56          
Training in progress @ step 1379 loss 0.775041 accuracy 0.62          
Testing in progress @ step 1379 loss 0.848526 accuracy 0.61          
Training in progress @ step 1399 loss 0.731224 accuracy 0.62          
Testing in progress @ step 1399 loss 1.04472 accuracy 0.58          
saved @ weights/toynet-1399
Training in progress @ step 1419 loss 0.809716 accuracy 0.66          
Testing in progress @ step 1419 loss 0.828566 accuracy 0.61          
Training in progress @ step 1439 loss 0.620828 accuracy 0.7          
Testing in progress @ step 1439 loss 0.839731 accuracy 0.57          
Training in progress @ step 1459 loss 0.883545 accuracy 0.56          
Testing in progress @ step 1459 loss 0.774318 accuracy 0.64          
Training in progress @ step 1479 loss 0.782009 accuracy 0.62          
Testing in progress @ step 1479 loss 0.80091 accuracy 0.63          
Training in progress @ step 1499 loss 0.715672 accuracy 0.64          
Testing in progress @ step 1499 loss 0.800926 accuracy 0.67          
saved @ weights/toynet-1499
Training in progress @ step 1519 loss 0.794199 accuracy 0.7          
Testing in progress @ step 1519 loss 0.899569 accuracy 0.61          
Training in progress @ step 1539 loss 0.56989 accuracy 0.76          
Testing in progress @ step 1539 loss 0.813612 accuracy 0.67          
Training in progress @ step 1559 loss 0.802496 accuracy 0.68          
Testing in progress @ step 1559 loss 0.800578 accuracy 0.62          
Training in progress @ step 1579 loss 0.76987 accuracy 0.68          
Testing in progress @ step 1579 loss 0.985612 accuracy 0.6          
Training in progress @ step 1599 loss 1.0418 accuracy 0.46          
Testing in progress @ step 1599 loss 0.884355 accuracy 0.67          
saved @ weights/toynet-1599
Training in progress @ step 1619 loss 0.788766 accuracy 0.66          
Testing in progress @ step 1619 loss 0.815026 accuracy 0.65          
Training in progress @ step 1639 loss 0.869168 accuracy 0.64          
Testing in progress @ step 1639 loss 0.952737 accuracy 0.59          
Training in progress @ step 1659 loss 0.806298 accuracy 0.7          
Testing in progress @ step 1659 loss 0.7895 accuracy 0.64          
Training in progress @ step 1679 loss 0.799824 accuracy 0.66          
Testing in progress @ step 1679 loss 0.870121 accuracy 0.56          
Training in progress @ step 1699 loss 1.05948 accuracy 0.46          
Testing in progress @ step 1699 loss 0.786247 accuracy 0.72          
saved @ weights/toynet-1699
Training in progress @ step 1719 loss 0.73907 accuracy 0.68          
Testing in progress @ step 1719 loss 0.845045 accuracy 0.57          
Training in progress @ step 1739 loss 0.609573 accuracy 0.72          
Testing in progress @ step 1739 loss 0.874041 accuracy 0.64          
Training in progress @ step 1759 loss 0.799591 accuracy 0.66          
Testing in progress @ step 1759 loss 0.868031 accuracy 0.53          
Training in progress @ step 1779 loss 0.74723 accuracy 0.68          
Testing in progress @ step 1779 loss 0.840827 accuracy 0.63          
Training in progress @ step 1799 loss 1.1167 accuracy 0.46          
Testing in progress @ step 1799 loss 0.819905 accuracy 0.64          
saved @ weights/toynet-1799
Training in progress @ step 1819 loss 0.736943 accuracy 0.68          
Testing in progress @ step 1819 loss 0.841028 accuracy 0.56          
Training in progress @ step 1839 loss 0.587976 accuracy 0.72          
Testing in progress @ step 1839 loss 0.836034 accuracy 0.6          
Training in progress @ step 1859 loss 1.04782 accuracy 0.58          
Testing in progress @ step 1859 loss 0.940606 accuracy 0.57          
Training in progress @ step 1879 loss 0.771269 accuracy 0.68          
Testing in progress @ step 1879 loss 0.821382 accuracy 0.58          
Training in progress @ step 1899 loss 1.0323 accuracy 0.4          
Testing in progress @ step 1899 loss 0.961747 accuracy 0.6          
saved @ weights/toynet-1899
Training in progress @ step 1919 loss 0.759074 accuracy 0.7          
Testing in progress @ step 1919 loss 0.664692 accuracy 0.72          
Training in progress @ step 1939 loss 0.777342 accuracy 0.64          
Testing in progress @ step 1939 loss 0.764626 accuracy 0.64          
Training in progress @ step 1959 loss 0.892178 accuracy 0.54          
Testing in progress @ step 1959 loss 0.868235 accuracy 0.54          
Training in progress @ step 1979 loss 0.778631 accuracy 0.64          
Testing in progress @ step 1979 loss 0.884256 accuracy 0.61          
Training in progress @ step 1999 loss 0.997717 accuracy 0.48          
Testing in progress @ step 1999 loss 0.72596 accuracy 0.68          
saved @ weights/toynet-1999
Training in progress @ step 2019 loss 0.722517 accuracy 0.68          
Testing in progress @ step 2019 loss 0.845509 accuracy 0.59          
Training in progress @ step 2039 loss 0.611634 accuracy 0.68          
Testing in progress @ step 2039 loss 0.964 accuracy 0.65          
Training in progress @ step 2059 loss 0.904469 accuracy 0.6          
Testing in progress @ step 2059 loss 0.681892 accuracy 0.66          
Training in progress @ step 2079 loss 0.761023 accuracy 0.68          
Testing in progress @ step 2079 loss 0.74166 accuracy 0.67          
Training in progress @ step 2099 loss 1.04678 accuracy 0.4          
Testing in progress @ step 2099 loss 0.786077 accuracy 0.59          
saved @ weights/toynet-2099
Training in progress @ step 2119 loss 0.706591 accuracy 0.68          
Testing in progress @ step 2119 loss 0.822638 accuracy 0.59          
Training in progress @ step 2139 loss 0.731965 accuracy 0.66          
Testing in progress @ step 2139 loss 0.650216 accuracy 0.71          
Training in progress @ step 2159 loss 0.82419 accuracy 0.6          
Testing in progress @ step 2159 loss 0.63992 accuracy 0.73          
Training in progress @ step 2179 loss 0.75396 accuracy 0.68          
Testing in progress @ step 2179 loss 1.04561 accuracy 0.52          
Training in progress @ step 2199 loss 0.655408 accuracy 0.7          
Testing in progress @ step 2199 loss 0.844042 accuracy 0.67          
saved @ weights/toynet-2199
Training in progress @ step 2219 loss 0.668467 accuracy 0.8          
Testing in progress @ step 2219 loss 0.750227 accuracy 0.65          
Training in progress @ step 2239 loss 0.873769 accuracy 0.66          
Testing in progress @ step 2239 loss 0.777166 accuracy 0.63          
Training in progress @ step 2259 loss 0.84647 accuracy 0.56          
Testing in progress @ step 2259 loss 0.789617 accuracy 0.67          
Training in progress @ step 2279 loss 0.681089 accuracy 0.7          
Testing in progress @ step 2279 loss 0.764521 accuracy 0.68          
Training in progress @ step 2299 loss 1.00233 accuracy 0.52          
Testing in progress @ step 2299 loss 0.768911 accuracy 0.63          
saved @ weights/toynet-2299
Training in progress @ step 2319 loss 0.710802 accuracy 0.7          
Testing in progress @ step 2319 loss 0.895237 accuracy 0.61          
Training in progress @ step 2339 loss 0.771297 accuracy 0.64          
Testing in progress @ step 2339 loss 0.826053 accuracy 0.67          
Training in progress @ step 2359 loss 0.816656 accuracy 0.62          
Testing in progress @ step 2359 loss 0.868144 accuracy 0.63          
Training in progress @ step 2379 loss 0.665603 accuracy 0.72          
Testing in progress @ step 2379 loss 0.761712 accuracy 0.7          
Training in progress @ step 2399 loss 0.943084 accuracy 0.52          
Testing in progress @ step 2399 loss 0.89798 accuracy 0.59          
saved @ weights/toynet-2399
Training in progress @ step 2419 loss 0.6713 accuracy 0.68          
Testing in progress @ step 2419 loss 0.768299 accuracy 0.63          
Training in progress @ step 2439 loss 0.604608 accuracy 0.74          
Testing in progress @ step 2439 loss 0.759793 accuracy 0.64          
Training in progress @ step 2459 loss 0.878221 accuracy 0.62          
Testing in progress @ step 2459 loss 0.943683 accuracy 0.64          
Training in progress @ step 2479 loss 0.655328 accuracy 0.68          
Testing in progress @ step 2479 loss 0.674722 accuracy 0.71          
Training in progress @ step 2499 loss 0.631793 accuracy 0.76          
Testing in progress @ step 2499 loss 0.726522 accuracy 0.68          
saved @ weights/toynet-2499
Training in progress @ step 2519 loss 0.645802 accuracy 0.74          
Testing in progress @ step 2519 loss 0.813128 accuracy 0.66          
Training in progress @ step 2539 loss 0.582686 accuracy 0.72          
Testing in progress @ step 2539 loss 0.753299 accuracy 0.69          
Training in progress @ step 2559 loss 0.862092 accuracy 0.58          
Testing in progress @ step 2559 loss 0.649019 accuracy 0.74          
Training in progress @ step 2579 loss 0.703706 accuracy 0.74          
Testing in progress @ step 2579 loss 1.05466 accuracy 0.64          
Training in progress @ step 2599 loss 0.991806 accuracy 0.52          
Testing in progress @ step 2599 loss 0.772276 accuracy 0.7          
saved @ weights/toynet-2599
Training in progress @ step 2619 loss 0.61569 accuracy 0.72          
Testing in progress @ step 2619 loss 0.762828 accuracy 0.62          
Training in progress @ step 2639 loss 0.681346 accuracy 0.68          
Testing in progress @ step 2639 loss 1.0182 accuracy 0.66          
Training in progress @ step 2659 loss 0.734712 accuracy 0.68          
Testing in progress @ step 2659 loss 0.751037 accuracy 0.65          
Training in progress @ step 2679 loss 0.720828 accuracy 0.68          
Testing in progress @ step 2679 loss 0.822734 accuracy 0.63          
Training in progress @ step 2699 loss 0.606319 accuracy 0.74          
Testing in progress @ step 2699 loss 0.873934 accuracy 0.71          
saved @ weights/toynet-2699
Training in progress @ step 2719 loss 0.638485 accuracy 0.72          
Testing in progress @ step 2719 loss 0.838632 accuracy 0.61          
Training in progress @ step 2739 loss 0.582639 accuracy 0.72          
Testing in progress @ step 2739 loss 0.839116 accuracy 0.69          
Training in progress @ step 2759 loss 0.723348 accuracy 0.66          
Testing in progress @ step 2759 loss 0.781844 accuracy 0.58          
Training in progress @ step 2779 loss 0.804298 accuracy 0.62          
Testing in progress @ step 2779 loss 0.781691 accuracy 0.7          
Training in progress @ step 2799 loss 0.566278 accuracy 0.78          
Testing in progress @ step 2799 loss 0.629277 accuracy 0.68          
saved @ weights/toynet-2799
Training in progress @ step 2819 loss 0.598975 accuracy 0.7          
Testing in progress @ step 2819 loss 0.72805 accuracy 0.63          
Training in progress @ step 2839 loss 0.640852 accuracy 0.7          
Testing in progress @ step 2839 loss 0.827767 accuracy 0.64          
Training in progress @ step 2859 loss 0.748543 accuracy 0.64          
Testing in progress @ step 2859 loss 1.01029 accuracy 0.6          
Training in progress @ step 2879 loss 0.646093 accuracy 0.76          
Testing in progress @ step 2879 loss 0.717815 accuracy 0.67          
Training in progress @ step 2899 loss 0.594118 accuracy 0.78          
Testing in progress @ step 2899 loss 1.0105 accuracy 0.61          
saved @ weights/toynet-2899
Training in progress @ step 2919 loss 0.647871 accuracy 0.72          
Testing in progress @ step 2919 loss 0.680736 accuracy 0.72          
Training in progress @ step 2939 loss 0.579024 accuracy 0.76          
Testing in progress @ step 2939 loss 0.810344 accuracy 0.62          
Training in progress @ step 2959 loss 0.687057 accuracy 0.72          
Testing in progress @ step 2959 loss 0.968042 accuracy 0.6          
Training in progress @ step 2979 loss 0.605305 accuracy 0.74          
Testing in progress @ step 2979 loss 0.766425 accuracy 0.67          
Training in progress @ step 2999 loss 0.865925 accuracy 0.52          
Testing in progress @ step 2999 loss 0.760901 accuracy 0.64          
saved @ weights/toynet-2999
Training in progress @ step 3019 loss 0.597984 accuracy 0.7          
Testing in progress @ step 3019 loss 0.896497 accuracy 0.58          
Training in progress @ step 3039 loss 0.627448 accuracy 0.74          
Testing in progress @ step 3039 loss 0.933452 accuracy 0.67          
Training in progress @ step 3059 loss 0.728232 accuracy 0.66          
Testing in progress @ step 3059 loss 0.609125 accuracy 0.72          
Training in progress @ step 3079 loss 0.583779 accuracy 0.76          
Testing in progress @ step 3079 loss 0.693111 accuracy 0.7          
Training in progress @ step 3099 loss 0.969401 accuracy 0.46          
Testing in progress @ step 3099 loss 0.833739 accuracy 0.57          
saved @ weights/toynet-3099
Training in progress @ step 3119 loss 0.588207 accuracy 0.82          
Testing in progress @ step 3119 loss 0.813724 accuracy 0.64          
Training in progress @ step 3139 loss 0.535594 accuracy 0.76          
Testing in progress @ step 3139 loss 0.588565 accuracy 0.76          
Training in progress @ step 3159 loss 0.678258 accuracy 0.7          
Testing in progress @ step 3159 loss 0.591031 accuracy 0.71          
Training in progress @ step 3179 loss 0.690407 accuracy 0.74          
Testing in progress @ step 3179 loss 0.996247 accuracy 0.61          
Training in progress @ step 3199 loss 0.545951 accuracy 0.76          
Testing in progress @ step 3199 loss 0.869521 accuracy 0.69          
saved @ weights/toynet-3199
Training in progress @ step 3219 loss 0.611411 accuracy 0.7          
Testing in progress @ step 3219 loss 0.747814 accuracy 0.73          
Training in progress @ step 3239 loss 0.500571 accuracy 0.76          
Testing in progress @ step 3239 loss 0.841545 accuracy 0.58          
Training in progress @ step 3259 loss 0.661496 accuracy 0.74          
Testing in progress @ step 3259 loss 0.876797 accuracy 0.62          
Training in progress @ step 3279 loss 0.773342 accuracy 0.64          
Testing in progress @ step 3279 loss 0.769359 accuracy 0.68          
Training in progress @ step 3299 loss 0.904071 accuracy 0.5          
Testing in progress @ step 3299 loss 0.708336 accuracy 0.69          
saved @ weights/toynet-3299
Training in progress @ step 3319 loss 0.625595 accuracy 0.7          
Testing in progress @ step 3319 loss 0.843951 accuracy 0.6          
Training in progress @ step 3339 loss 0.575792 accuracy 0.74          
Testing in progress @ step 3339 loss 0.8904 accuracy 0.63          
Training in progress @ step 3359 loss 0.754028 accuracy 0.66          
Testing in progress @ step 3359 loss 0.758322 accuracy 0.65          
Training in progress @ step 3379 loss 0.661254 accuracy 0.76          
Testing in progress @ step 3379 loss 0.747814 accuracy 0.64          
Training in progress @ step 3399 loss 0.915007 accuracy 0.52          
Testing in progress @ step 3399 loss 0.924655 accuracy 0.57          
saved @ weights/toynet-3399
Training in progress @ step 3419 loss 0.592136 accuracy 0.7          
Testing in progress @ step 3419 loss 0.688503 accuracy 0.66          
Training in progress @ step 3439 loss 0.53446 accuracy 0.72          
Testing in progress @ step 3439 loss 0.713104 accuracy 0.62          
Training in progress @ step 3459 loss 0.641942 accuracy 0.72          
Testing in progress @ step 3459 loss 0.793349 accuracy 0.66          
Training in progress @ step 3479 loss 0.651127 accuracy 0.76          
Testing in progress @ step 3479 loss 0.697957 accuracy 0.7          
Training in progress @ step 3499 loss 0.811335 accuracy 0.56          
Testing in progress @ step 3499 loss 0.717807 accuracy 0.71          
saved @ weights/toynet-3499
Training in progress @ step 3519 loss 0.619678 accuracy 0.7          
Testing in progress @ step 3519 loss 0.823851 accuracy 0.66          
Training in progress @ step 3539 loss 0.858907 accuracy 0.66          
Testing in progress @ step 3539 loss 0.833918 accuracy 0.65          
Training in progress @ step 3559 loss 0.738354 accuracy 0.6          
Testing in progress @ step 3559 loss 0.676696 accuracy 0.7          
Training in progress @ step 3579 loss 0.703173 accuracy 0.74          
Testing in progress @ step 3579 loss 1.09339 accuracy 0.67          
Training in progress @ step 3599 loss 0.604465 accuracy 0.76          
Testing in progress @ step 3599 loss 0.649062 accuracy 0.68          
saved @ weights/toynet-3599
Training in progress @ step 3619 loss 0.566989 accuracy 0.76          
Testing in progress @ step 3619 loss 0.73983 accuracy 0.67          
Training in progress @ step 3639 loss 0.565764 accuracy 0.76          
Testing in progress @ step 3639 loss 0.97306 accuracy 0.66          
Training in progress @ step 3659 loss 0.609621 accuracy 0.74          
Testing in progress @ step 3659 loss 0.661985 accuracy 0.72          
Training in progress @ step 3679 loss 0.634706 accuracy 0.7          
Testing in progress @ step 3679 loss 0.828382 accuracy 0.64          
Training in progress @ step 3699 loss 0.530048 accuracy 0.78          
Testing in progress @ step 3699 loss 1.06915 accuracy 0.65          
saved @ weights/toynet-3699
Training in progress @ step 3719 loss 0.608648 accuracy 0.82          
Testing in progress @ step 3719 loss 0.802318 accuracy 0.66          
Training in progress @ step 3739 loss 0.707695 accuracy 0.64          
Testing in progress @ step 3739 loss 0.796944 accuracy 0.75          
Training in progress @ step 3759 loss 0.629873 accuracy 0.74          
Testing in progress @ step 3759 loss 0.786761 accuracy 0.59          
Training in progress @ step 3779 loss 0.699369 accuracy 0.68          
Testing in progress @ step 3779 loss 0.820317 accuracy 0.65          
Training in progress @ step 3799 loss 0.496323 accuracy 0.86          
Testing in progress @ step 3799 loss 0.630245 accuracy 0.67          
saved @ weights/toynet-3799
Training in progress @ step 3819 loss 0.558216 accuracy 0.74          
Testing in progress @ step 3819 loss 0.658045 accuracy 0.69          
Training in progress @ step 3839 loss 0.521911 accuracy 0.76          
Testing in progress @ step 3839 loss 0.706592 accuracy 0.75          
Training in progress @ step 3859 loss 0.705313 accuracy 0.58          
Testing in progress @ step 3859 loss 1.0322 accuracy 0.61          
Training in progress @ step 3879 loss 0.669163 accuracy 0.74          
Testing in progress @ step 3879 loss 0.74675 accuracy 0.67          
Training in progress @ step 3899 loss 0.778846 accuracy 0.64          
Testing in progress @ step 3899 loss 1.09875 accuracy 0.65          
saved @ weights/toynet-3899
Training in progress @ step 3919 loss 0.562154 accuracy 0.78          
Testing in progress @ step 3919 loss 0.528436 accuracy 0.76          
Training in progress @ step 3939 loss 0.527017 accuracy 0.76          
Testing in progress @ step 3939 loss 0.777258 accuracy 0.67          
Training in progress @ step 3959 loss 0.644141 accuracy 0.74          
Testing in progress @ step 3959 loss 0.955519 accuracy 0.63          
Training in progress @ step 3979 loss 0.483887 accuracy 0.82          
Testing in progress @ step 3979 loss 0.741936 accuracy 0.69          
Training in progress @ step 3999 loss 0.801905 accuracy 0.56          
Testing in progress @ step 3999 loss 0.740641 accuracy 0.68          
saved @ weights/toynet-3999
Training in progress @ step 4019 loss 0.534196 accuracy 0.76          
Testing in progress @ step 4019 loss 0.836735 accuracy 0.61          
Training in progress @ step 4039 loss 0.708263 accuracy 0.64          
Testing in progress @ step 4039 loss 0.865541 accuracy 0.76          
Training in progress @ step 4059 loss 0.577173 accuracy 0.72          
Testing in progress @ step 4059 loss 0.557551 accuracy 0.73          
Training in progress @ step 4079 loss 0.658289 accuracy 0.7          
Testing in progress @ step 4079 loss 0.737281 accuracy 0.71          
Training in progress @ step 4099 loss 0.764987 accuracy 0.58          
Testing in progress @ step 4099 loss 0.755877 accuracy 0.6          
saved @ weights/toynet-4099
Training in progress @ step 4119 loss 0.559188 accuracy 0.82          
Testing in progress @ step 4119 loss 0.798659 accuracy 0.66          
Training in progress @ step 4139 loss 0.498029 accuracy 0.78          
Testing in progress @ step 4139 loss 0.564819 accuracy 0.75          
Training in progress @ step 4159 loss 0.612597 accuracy 0.7          
Testing in progress @ step 4159 loss 0.571949 accuracy 0.77          
Training in progress @ step 4179 loss 0.538559 accuracy 0.8          
Testing in progress @ step 4179 loss 0.945541 accuracy 0.66          
Training in progress @ step 4199 loss 0.809389 accuracy 0.6          
Testing in progress @ step 4199 loss 0.916501 accuracy 0.66          
saved @ weights/toynet-4199
Training in progress @ step 4219 loss 0.552245 accuracy 0.8          
Testing in progress @ step 4219 loss 0.71138 accuracy 0.68          
Training in progress @ step 4239 loss 0.598316 accuracy 0.78          
Testing in progress @ step 4239 loss 0.804888 accuracy 0.59          
Training in progress @ step 4259 loss 0.574028 accuracy 0.72          
Testing in progress @ step 4259 loss 1.17863 accuracy 0.66          
Training in progress @ step 4279 loss 0.507018 accuracy 0.82          
Testing in progress @ step 4279 loss 0.717412 accuracy 0.72          
Training in progress @ step 4299 loss 0.725069 accuracy 0.54          
Testing in progress @ step 4299 loss 0.763786 accuracy 0.68          
saved @ weights/toynet-4299
Training in progress @ step 4319 loss 0.547566 accuracy 0.7          
Testing in progress @ step 4319 loss 0.897501 accuracy 0.62          
Training in progress @ step 4339 loss 0.437541 accuracy 0.82          
Testing in progress @ step 4339 loss 0.861329 accuracy 0.69          
Training in progress @ step 4359 loss 0.587879 accuracy 0.76          
Testing in progress @ step 4359 loss 0.747539 accuracy 0.68          
Training in progress @ step 4379 loss 0.502393 accuracy 0.76          
Testing in progress @ step 4379 loss 0.749756 accuracy 0.72          
Training in progress @ step 4399 loss 0.777657 accuracy 0.62          
Testing in progress @ step 4399 loss 1.01437 accuracy 0.63          
saved @ weights/toynet-4399
Training in progress @ step 4419 loss 0.514751 accuracy 0.84          
Testing in progress @ step 4419 loss 0.777094 accuracy 0.66          
Training in progress @ step 4439 loss 0.512706 accuracy 0.76          
Testing in progress @ step 4439 loss 0.65704 accuracy 0.67          
Training in progress @ step 4459 loss 0.702033 accuracy 0.64          
Testing in progress @ step 4459 loss 0.709192 accuracy 0.67          
Training in progress @ step 4479 loss 0.643489 accuracy 0.74          
Testing in progress @ step 4479 loss 0.710084 accuracy 0.71          
Training in progress @ step 4499 loss 0.472949 accuracy 0.84          
Testing in progress @ step 4499 loss 0.637643 accuracy 0.73          
saved @ weights/toynet-4499
Training in progress @ step 4519 loss 0.526931 accuracy 0.78          
Testing in progress @ step 4519 loss 0.802738 accuracy 0.66          
Training in progress @ step 4539 loss 0.71399 accuracy 0.66          
Testing in progress @ step 4539 loss 0.725557 accuracy 0.68          
Training in progress @ step 4559 loss 0.59562 accuracy 0.72          
Testing in progress @ step 4559 loss 0.768135 accuracy 0.7          
Training in progress @ step 4579 loss 0.513815 accuracy 0.78          
Testing in progress @ step 4579 loss 1.30597 accuracy 0.69          
Training in progress @ step 4599 loss 0.846219 accuracy 0.58          
Testing in progress @ step 4599 loss 0.708973 accuracy 0.69          
saved @ weights/toynet-4599
Training in progress @ step 4619 loss 0.545314 accuracy 0.72          
Testing in progress @ step 4619 loss 0.704354 accuracy 0.71          
Training in progress @ step 4639 loss 0.508829 accuracy 0.74          
Testing in progress @ step 4639 loss 0.98429 accuracy 0.69          
Training in progress @ step 4659 loss 0.644264 accuracy 0.68          
Testing in progress @ step 4659 loss 0.600745 accuracy 0.74          
Training in progress @ step 4679 loss 0.46995 accuracy 0.82          
Testing in progress @ step 4679 loss 0.820408 accuracy 0.61          
Training in progress @ step 4699 loss 0.732839 accuracy 0.62          
Testing in progress @ step 4699 loss 0.740484 accuracy 0.67          
saved @ weights/toynet-4699
Training in progress @ step 4719 loss 0.495116 accuracy 0.8          
Testing in progress @ step 4719 loss 0.878607 accuracy 0.64          
Training in progress @ step 4739 loss 0.458679 accuracy 0.8          
Testing in progress @ step 4739 loss 0.842409 accuracy 0.71          
Training in progress @ step 4759 loss 0.54042 accuracy 0.76          
Testing in progress @ step 4759 loss 1.25367 accuracy 0.64          
Training in progress @ step 4779 loss 0.484868 accuracy 0.84          
Testing in progress @ step 4779 loss 0.915994 accuracy 0.63          
Training in progress @ step 4799 loss 0.696491 accuracy 0.68          
Testing in progress @ step 4799 loss 0.734606 accuracy 0.72          
saved @ weights/toynet-4799
Training in progress @ step 4819 loss 0.495629 accuracy 0.82          
Testing in progress @ step 4819 loss 0.731098 accuracy 0.64          
Training in progress @ step 4839 loss 0.513645 accuracy 0.82          
Testing in progress @ step 4839 loss 0.820735 accuracy 0.73          
Training in progress @ step 4859 loss 0.556325 accuracy 0.74          
Testing in progress @ step 4859 loss 0.983378 accuracy 0.65          
Training in progress @ step 4879 loss 0.538882 accuracy 0.8          
Testing in progress @ step 4879 loss 0.885603 accuracy 0.68          
Training in progress @ step 4899 loss 0.801503 accuracy 0.56          
Testing in progress @ step 4899 loss 1.21059 accuracy 0.65          
saved @ weights/toynet-4899
Training in progress @ step 4919 loss 0.482372 accuracy 0.78          
Testing in progress @ step 4919 loss 0.521534 accuracy 0.76          
Training in progress @ step 4939 loss 0.709935 accuracy 0.7          
Testing in progress @ step 4939 loss 0.673117 accuracy 0.7          
Training in progress @ step 4959 loss 0.614147 accuracy 0.68          
Testing in progress @ step 4959 loss 0.833616 accuracy 0.65          
Training in progress @ step 4979 loss 0.623204 accuracy 0.82          
Testing in progress @ step 4979 loss 0.713888 accuracy 0.66          
Training in progress @ step 4999 loss 0.62346 accuracy 0.7          
Testing in progress @ step 4999 loss 0.692974 accuracy 0.74          
saved @ weights/toynet-4999

Run `tensorboard --logdir=log` in terminal to see the results.

Checking log on the tensorboard

As the last line above says you can visualize your log using tensorboard. This command

tensorboard --logdir=log

on the terminal instantiates the tensorboard server and tells the localhost address to access through your web-browser. You can certainly ssh-tunnel to access the localhost of your remote machine to check it on your local machine's web-browser as well. For the above training, here's the screenshot of the loss and accuracy curve for train and test samples where the blue line represents metric measured on the training set and orange line is for the same on the test sample.

loss

accuracy

This notebook covered training convolutional neural networks to perform image classification of 5 LArTPC particles using a practice files. We encourage you to design your own network and train on our public dataset! We provide 50,000 entries of 5 particle images (10,000 per particle) for training and separate 40,000 for testing your network. When you are confident, try our data challenge, yet another set of 40,000 events without answers (i.e. no particle information). Share your awesome result in the CSV format to us with your network architecture made available on a github repository.