BigDL release 0.10.0

Highlights

• Continue RNN optimization. We support both LSTM and GRU integration with MKL-DNN which acheives ~3x performance

• ONNX support. We support loading third party framework models via ONNX

• Richer data preprocssing support and segmentation inference pipeline support

Details

• [New Feature] Full MaskRCNN model support with data processing
• [New Feature] Support variable-size Resize
• [New Feature] Support batch input for region proposal
• [New Feature] Support samples of different size in one minibatch
• [New Feature] MAP validation method implementation
• [New Feature] ROILabel enhancement to support both object detection and segmentation
• [New Feature] Grey image support for segmentation
• [New Feature] Add TopBlocks support for Feature Pyramid Networks (FPN)
• [New Feature] GRU integration with MKL-DNN support
• [New Feature] MaskHead support for MaskRCNN
• [New Feature] BoxHead support for MaskRCNN
• [New Feature] RegionalProposal support for MaskRCNN
• [New Feature] Shape operation support for ONNX
• [New Feature] Gemm operation support for ONNX
• [New Feature] Gather operation support for ONNX
• [New Feature] AveragePool operation support for ONNX
• [New Feature] BatchNormalization operation support for ONNX
• [New Feature] Concat operation support for ONNX
• [New Feature] Conv operation support for ONNX
• [New Feature] MaxPool operation support for ONNX
• [New Feature] Reshape operation support for ONNX
• [New Feature] Relu operation support for ONNX
• [New Feature] SoftMax operation support for ONNX
• [New Feature] Sum operation support for ONNX
• [New Feature] Squeeze operation support for ONNX
• [New Feature] Const operation support for ONNX
• [New Feature] ONNX model loader implementation
• [New Feature] RioAlign layer support
• [Enhancement] Align batch normalization layer between mklblas and mkl-dnn
• [Enhancement] Python API enhancement to support nested list input
• [Enhancement] Multi-model training/inference support with MKL-DNN
• [Enhancement] BatchNormalization fusion with Scale
• [Enhancement] SoftMax companion object support no argument initialization
• [Enhancement] Python support for training with MKL-DNN
• [Enhancement] Docs enhancement
• [Bug Fix] Fix model version comparison
• [Bug Fix] Fix graph backward bug for ParallelTable
• [Bug Fix] Fix memory leak for training with MKL-DNN
• [Bug Fix] Fix performance caused by denormal values during training
• [Bug Fix] Fix SoftMax segment fault issue under MKL-DNN
• [Bug Fix] Fix TimeDistributedCriterion python API inconsistent with Scala

BigDL release 0.9.0

Highlights

• Continue VNNI acceleration support, we add optimization for more CNN models including object detection models, enhance model scales generation support for VNNI.

• Add attention based model support, we add Transformer implementation for both lanuage model and translation model.

• RNN optimization, We support LSTM integration with MKL-DNN which acheives ~3x performance speedup.

Details

• [New Feature] Add attention layer support
• [New Feature] Add FeedForwardNetwork layer support
• [New Feature] Add ExpandSize layer support
• [New Feature] Add TableOperation layer to support table calculation with different input sizes
• [New Feature] Add LayerNormalizaiton layer support
• [New Feature] Add Transformer support for both language and translation models
• [New Feature] Add beam search support in Transformer model
• [New Feature] Add Layer-wise adaptve rate scaling optim method
• [New Feature] Add LSTM integration with MKL-DNN support
• [New Feature] Add dilated convolution integration with MKL-DNN support
• [New Feature] Add parameter process for LarsSGD optim method
• [New Feature] Support Affinity binding option with mkl-dnn
• [Enhancement] Document enhancement for configuration and build
• [Enhancement] Reflection enhancement to get default values for constructor parameters
• [Enhhancement] User one AllReducemParameter for multi-optim method training
• [Enhancement] CAddTable layer enhancement to support input expansion along specific dimension
• [Enhancement] Resnet-50 preprocessing pipeline enhancement to replace RandomCropper with CenterCropper
• [Enhancement] Calculate model scales for arbitrary mask
• [Enhancment] Enable global average pooling
• [Enhancement] Check input shape and underlying MKL-DNN layout consistency
• [Enhancement] Threadpool enhancement to throw proper exception at executor runtime
• [Enhancement] Support mkl-dnn format conversion from ntc to tnc
• [Bug Fix] Fix backward graph generation topology ordering issue
• [Bug Fix] Fix MemoryData hash code calculation
• [Bug Fix] Fix log output for BCECriterion
• [Bug Fix] Fix setting mask for container quantization
• [Bug Fix] Fix validation accuracy issue when multi-executor running with the same worker
• [Bug Fix] Fix INT8 layer fusion between conlution with multi-group masks and BatchNormalization
• [Bug Fix] Fix JoinTable scales generation issue
• [Bug Fix] Fix CMul forward issue with special input format
• [Bug Fix] Fix weights change issue after model fusion issue
• [Bug Fix] Fix SpatinalConvolution primitives initializaiton issue

BigDL release 0.8.0

Highlights

• Add MKL-DNN Int8 support, especially for VNNI acceleration support. Low precision inference accelerates both latency and throughput significantly
• Add support for runnning MKL-BLAS models under MKL-DNN. We leverage MKL-DNN to speed up both training and inference for MKL-BLAS models
• Add Spark 2.4 support. Our examples and APIs are fully compatible with Spark 2.4, we released the binary for Spark 2.4 together with other Spark versions

Details

• [New Feature] Add MKL-DNN Int8 support, especially for VNNI support
• [New Feature] Add support for runnning MKL-BLAS models under MKL-DNN
• [New Feature] Add Spark 2.4 support
• [New Feature] Add auto fusion to speed up model inference
• [New Feature] Memoery reorder support for low precision inference
• [New Feature] Add bytes support for DNN Tensor
• [New Feature] Add SAME padding in MKL-DNN layers
• [New Feature] Add combined (add/or) triggers for training completion
• [Enhancement] Inception-V1 python training support enhancement
• [Enhancement] Distributed Optimizer enhancement to support customized optimizer
• [Enhancement] Add compute output shape for DNN supported layers
• [Enhancement] New MKL-DNN computing thread pool
• [Enhancement] Add MKL-DNN support for Predictor
• [Enhancement] Documentation enhancement for Sparse Tensor, MKL-DNN support, etc
• [Enhancement] Add ceilm mode for AvgPooling and MaxPooling layers
• [Enhacement] Add binary classification support for DLClassifierModel
• [Enhacement] Improvement to support conversion between NHWC and NCHW for memory reoder
• [Bug Fix] Fix SoftMax layer with narrowed input
• [Bug Fix] TensorFlow loader to support checking all data types
• [Bug Fix] Fix Add operation bug to support double type when loading TensorFlow graph
• [Bug Fix] Fix one-step weight update missing issue in validation during training
• [Bug Fix] Fix scala compiler security issue in 2.10 & 2.11
• [Bug Fix] Fix model broadcast cache UUID issue
• [Bug Fix] Fix predictor issue for batch size == 1

BigDL release 0.7.0

Highlights

• MKL-DNN support enhancement, which includes training optimization, more models training support and model serialization support
• A new distributed optimizer for models powered by MKL-DNN. This optimizer can overlap training and communication during the distributed training, which lead to a better scalability on multi-nodes

Details

• [New Feature] A new optim method ParallelAdam which leverages the multi-thread capacity
• [New Feature] Add new validation methods HitRate, which is widely used in recommendation
• [New Feature] Add new validation methods NDCG, which is widely used in recommendation
• [New Feature] Support communication priority when synchronize parameter in the distributed training
• [New Feature] Support ModelBroadcast customization
• [New Feature] Add a new distributed optimizer for models powered by MKL-DNN. This optimizer can overlap training and communication during the distributed training, which lead to a better scalability on multi-nodes
• [API Change] Add batch size into the Python model.predict API
• [Enhancement] Add MKL-DNN training example for LeNet
• [Enhancement] Improve the training performance by getting rid of narrowing gradients and zero gradients for model powered by MKL-DNN
• [Enhancement] Add training example for VGG-16 based on MKL-DNN
• [Enhancement] Support nested table in Graph output
• [Enhancement] Enhancement on thread pool to make it compatible with MKL-DNN engine
• [Enhancement] MKL-DNN model serialization support
• [Enhancement] Add VGG-16 validation example
• [Bug Fix] Fix JoinTable throwing exception during backward if batch size is changed
• [Bug Fix] Change Reshape to InferReShape in ReshapeLoadTF
• [Bug Fix] Fix splitBatch issue in Predictor, where the model has multiple Graph and each Graph outputs a table
• [Bug Fix] Fix MDL-DNN inference performance issue not to copy weights at inference
• [Bug Fix] Fix the issue that the training will crash if there are unlabeled data
• [Bug Fix] Fix the issue that the input is grey image while the model needs 3 channels input
• [Bug Fix] Correct the style check job to make both input and output file format to UTF-8 format
• [Bug Fix] Load the relevant library only if MKL-DNN engine specified
• [Bug Fix] Shade org.tensorflow.framework to avoid conflict
• [Bug Fix] Fix dlframes not packaged in pip issue
• [Bug Fix] Fix LocalPredictor cannot be serialized because of nested logger variable
• [Bug Fix] Need to clear Recurrent preTopology's output while cloneCells
• [Bug Fix] MM layer output different output for same input if ran multiple times
• [Bug Fix] Distribute predictor will send model twice when do mapPartition
• [Document] Kubernetes programming guide to spark2.3
• [Document] Add document for wrap preprocessor and model in one graph and add its python API

BigDL release 0.6.0

Highlights

• We integrate MKL-DNN as an alternative execution engine for CNN models. MKL-DNN provides better training/inference performance and less memory consuming. On some CNN models, we find there’s 2x throughput improvement in our experiment.
• Support using different optimization methods to optimize different parts of the model. This is necessary when train some models.
• Spark 2.3 support. We have tested our code and examples on Spark 2.3. We release the binary for Spark 2.3, and Spark 1.5 will not be supported.

Details

• [New Feature] MKL-DNN integration. We integrate MKL-DNN as an alternative execution engine for CNN models. It supports speedup layers like: AvgPooling, MaxPooling, CAddTable, LRN, JoinTable, Linear, ReLU, SpatialConvolution, SpatialBatchnormalization, Softmax. MKL-DNN provides better training/inference performance and less memory consuming.
• [New Feature] Layer fusion. Support layer fusion on conv + relu, batchnorm + relu, conv + batchnorm and conv + sum(some of the fusion can only be applied in the inference). Layer fusion provides better performance especially on inference. Currently layer fusion are only available for MKL-DNN related layers.
• [New Feature] Multiple optimization method support in optimizer. Support using different optimization methods to optimize different parts of the model.
• [New Feature] Add a new optimization method Ftrl, which is often used in recommendation model training.
• [New Feature] Add a new example: Training Resnet50 on ImageNet dataset.
• [New Feature] Add new OpenCV based image preprocessing transformer ChannelScaledNormalizer.
• [New Feature] Add new OpenCV based image preprocessing transformer RandomAlterAspect.
• [New Feature] Add new OpenCV based image preprocessing transformer RandomCropper.
• [New Feature] Add new OpenCV based image preprocessing transformer RandomResize.
• [New Feature] Support loading Tensorflow Max operation.
• [New Feature] Allow user to specify input port when loading Tensorflow model. If the input operation accepts multiple tensors as input, user can specify which to feed data to instead of feed all tensors.
• [New Feature] Support loading Tensorflow Gather operation.
• [New Feature] Add random split for ImageFrame
• [New Feature] Add setLabel and getURI API into ImageFrame
• [API Change] Add batch size into the Python model.predict API.
• [API Change] Add generateBackward into load Tensorflow model API, which allows user choose whether to generate backward path when load Tensorflow model.
• [API Change] Add feature() and label() to the Sample.
• [API Change] Deprecate the DLClassifier/DLEstimator in org.apache.spark.ml. Prefer using DLClassifier/DLEstimator under com.intel.analytics.bigdl.dlframes.
• [Enhancement] Refine StridedSlice. Support begin/end/shrinkAxis mask just like Tensorflow.
• [Enhancement] Add layer sync to SpatialBatchNormalization. SpatialBatchNormalization can calculate mean/std on a larger batch size. The model with SpatialBatchNormalization layer can converge to a better accuracy even the local batch size is small.
• [Enhancement] Code refactor in DistriOptimizer for advanced parameter operations, e.g. global gradient clipping.
• [Enhancement] Add more models into the LoadModel example.
• [Enhancement] Share Const values when broadcast the model. The Const value will not be changed and we can share it when use multiple model for inference on a same node, which will reduce memory usage.
• [Enhancement] Refine the getTime and time counting implementation.
• [Enhancement] Support group serializer so that layers of the same hierarchy could share the same serializer.
• [Enhancement] Dockerfile use Python 2.7.
• [Bug Fix] Fix memory leak problem when using quantized model in predictor.
• [Bug Fix] Fix PY4J Java gateway not compatible in Spark local mode for Spark 2.3.
• [Bug Fix] Fix a bug in python inception example.
• [Bug Fix] Fix a bug when run Tensorflow model using loop.
• [Bug Fix] Fix a bug in the Squeeze layer.
• [Bug Fix] Fix python API for random split.
• [Bug Fix] Using parameters() instead of getParameterTable() to get weight and bias in serialization.
• [Document] Fix incorrectness in Quantized model document.
• [Document] Fix incorrect instructions when generate Sequence files for ImageNet 2012 dataset in the document.
• [Document] Move bigdl-core build document into a separated page and refine the format.
• [Document] Fix incorrect command in Tensorflow load and transfer learning examples.

BigDL release 0.5.0

Highlights

• Bring in a Keras-like API(Scala and Python). User can easily run their Keras code (training and inference) on Apache Spark through BigDL. For more details, see this link.
• Support load Tensorflow dynamic models(e.g. LSTM, RNN) in BigDL and support more Tensorflow operations, see this page.
• Support combining data preprocessing and neural network layers in the same model (to make model deployment easy )
• Speedup various modules in BigDL (BCECriterion, rmsprop, LeakyRelu, etc.)
• Add DataFrame-based image reader and transformer

New Features

• Tensor can be converted to OpenCVMat
• Bring in a new Keras-like API for scala and python
• Support load Tensorflow dynamic models(e.g. LSTM, RNN)
• Support load more Tensorflow operations(InvertPermutation, ConcatOffset, Exit, NextIteration, Enter, RefEnter, LoopCond, ControlTrigger, TensorArrayV3,TensorArrayGradV3, TensorArrayGatherV3, TensorArrayScatterV3, TensorArrayConcatV3, TensorArraySplitV3, TensorArrayReadV3, TensorArrayWriteV3, TensorArraySizeV3, StackPopV2, StackPop, StackPushV2, StackPush, StackV2, Stack)
• ResizeBilinear support NCHW
• ImageFrame support load Hadoop sequence file
• ImageFrame support gray image
• Add Kv2Tensor Operation(Scala)
• Add PGCriterion to compute the negative policy gradient given action distribution, sampled action and reward
• Support gradual increase learning rate in LearningrateScheduler
• Add FixExpand and add more options to AspectScale for image preprocessing
• Add RowTransformer(Scala)
• Support to add preprocessors to Graph, which allows user combine preprocessing and trainable model into one model
• Resnet on cifar-10 example support load images from HDFS
• Add CategoricalColHashBucket operation(Scala)
• Predictor support Table as output
• Add BucketizedCol operation(Scala)
• Support using DenseTensor and SparseTensor together to create Sample
• Add CrossProduct Layer (Scala)
• Provide an option to allow user bypass the exception in transformer
• DenseToSparse layer support disable backward propagation
• Add CategoricalColVocaList Operation(Scala)
• Support imageframe in python optimizer
• Support get executor number and executor cores in python
• Add IndicatorCol Operation(Scala)
• Add TensorOp, which is an operation with Tensor[T]-formatted input and output, and provides shortcuts to build Operations for tensor transformation by closures. (Scala)
• Provide a docker file to make it easily to setup testing environment of BigDL
• Add CrossCol Operation(Scala)
• Add MkString Operation(Scala)
• Add a prediction service interface for concurrent calls and accept bytes input
• Add SparseTensor.cast & SparseTensor.applyFun
• Add DataFrame-based image reader and transformer
• Support load tensoflow model files saved by tf.saved_model API
• SparseMiniBatch supporting multiple TensorDataTypes

Enhancement

• ImageFrame support serialization
• A default implementation of zeroGradParameter is added to AbstractModule
• Improve the style of the document website
• Models in different threads share weights in model training
• Speed up leaky relu
• Speed up Rmsprop
• Speed up BCECriterion
• Support Calling Java Function in Python Executor and ModelBroadcast in Python
• Add detail instructions to run-on-ec2
• Optimize padding mechanism
• Fix maven compiling warnings
• Check duplicate layers in the container
• Refine the document which introduce how to automatically Deploy BigDL on Dataproc cluster
• Refactor adding extra jars/python packages for python user. Now only need to set env variable BIGDL_JARS & BIGDL_PACKAGES
• Implement appendColumn and avoid the error caused by API mismatch between different Spark version
• Add python inception training on ImageNet example
• Update "can't find locality partition for partition ..." to warning message

API change

• Move DataFrame-based API to dlframe package
• Refine the Container hierarchy. The add method(used in Sequential, Concat…) is moved to a subclass DynamicContainer
• Refine the serialization code hierarchy
• Dynamic Graph has been an internal class which is only used to run tensorflow models
• Operation is not allowed to use outside Graph
• The getParamter method as final and private[bigdl], which should be only used in model training
• remove the updateParameter method, which is only used in internal test
• Some Tensorflow related operations are marked as internal, which should be only used when running Tensorflow models

Bug Fix

• Fix Sparse sample batch bug. It should add another dimension instead of concat the original tensor
• Fix some activation or layers don’t work in TimeDistributed and RnnCell
• Fix a bug in SparseTensor resize method
• Fix a bug when convert SparseTensor to DenseTensor
• Fix a bug in SpatialFullConvolution
• Fix a bug in Cosine equal method
• Fix optimization state mess up when call optimizer.optimize() multiple times
• Fix a bug in Recurrent forward after invoking reset
• Fix a bug in inplace leakyrelu
• Fix a bug when save/load bi-rnn layers
• Fix getParameters() in submodule will create new storage when parameters has been shared by parent module
• Fix some incompatible syntax between python 2.7 and 3.6
• Fix save/load graph will loss stop gradient information
• Fix a bug in SReLU
• Fix a bug in DLModel
• Fix sparse tensor dot product bug
• Fix Maxout ser issue
• Fix some serialization issue in some customized faster rcnn model
• Fix and refine some example document instructions
• Fix a bug in export_tf_checkpoint.py script
• Fix a bug in set up python package.
• Fix picklers initialization issues
• Fix some race condition issue in Spark 1.6 when broadcasting model
• Fix Model.load in python return type is wrong
• Fix a bug when use pyspark-with-bigdl.sh to run jobs on Yarn
• Fix empty tensor call size and stride not throw null exception

BigDL release 0.4.0

Highlights

• Supported all Keras layers, and support Keras 1.2.2 model loading. See keras-support for detail
• Python 3.6 support
• OpenCV support, and add a dozen of image transformer based on OpenCV
• More layers/operations

New Features

• Models & Layers & Operations & Loss function
  • Add layers for Keras: Cropping2D, Cropping3D, UpSampling1D, UpSampling2D, UpSampling3D, masking,Maxout,HighWay,GaussianDropout, GaussianNoise, CAveTable, VolumetricAveragePooling, HardSigmoidSReLU, LocallyConnected1D, LocallyConnected2D, SpatialSeparableConvolution, ActivityRegularization, SpatialDropout1D, SpatialDropout2D, SpatialDropout3D
  • Add Criterion for keras: PoissonCriterion, KullbackLeiblerDivergenceCriterion, MeanAbsolutePercentageCriterion, MeanSquaredLogarithmicCriterion, CosineProximityCriterion
  • Support NHWC for LRN and BatchNormalization
  • Add LookupTableSparse (lookup table for multivalue)
  • Add activation argument for recurrent layers
  • Add MultiRNNCell
  • Add SpatialSeparableConvolution
  • Add MSRA filler
  • Support SAME padding in 3d conv and allows user config padding size in convlstm and convlstm3d
  • TF opteration: SegmentSum, conv3d related operations, Dilation2D, Dilation2DBackpropFilter, Dilation2DBackpropInput, Digamma, Erf, Erfc, Lgamma, TanhGrad, depthwise, Rint, All, Any, Range, Exp, Expm1, Round, FloorDiv, TruncateDiv, Mod, FloorMod, TruncateMod, IntopK, Round, Maximum, Minimum, BatchMatMu, Sqrt, SqrtGrad, Square, RsqrtGrad, AvgPool, AvgPoolGrad, BiasAddV1, SigmoidGrad, Relu6, Relu6Grad, Elu, EluGrad, Softplus, SoftplusGrad, LogSoftmax, Softsign, SoftsignGrad, Abs, LessEqual, GreaterEqual, ApproximateEqual, Log, LogGrad, Log1p, Log1pGrad, SquaredDifference, Div, Ceil, Inv, InvGrad, IsFinite, IsInf, IsNan, Sign, TopK. See details at tensorflow_ops_list)
  • Add object detection related layers: PriorBox, NormalizeScale, Proposal, DetectionOutputSSD, DetectionOutputFrcnn, Anchor
• Transformer
  • Add image Transformer based on OpenCV: Resize, Brightness, ChannelOrder, Contrast, Saturation, Hue, ChannelNormalize, PixelNormalize, RandomCrop, CenterCrop, FixedCrop, DetectionCrop, Expand, Filler, ColorJitter, RandomSampler, MatToFloats, AspectScale, RandomAspectScale, BytesToMat
  • Add Transformer: RandomTransformer, RoiProject, RoiHFlip, RoiResize, RoiNormalize
• API change
  • Add predictImage function in LocalPredictor
  • Add partition number option for ImageFrame read
  • Add an API to get node from graph model with given name
  • Support List of JTensors for label in Python API
  • Expose local optimizer and predictor in Python API
• Install & Deploy
  • Support BigDL on Spark on k8s
• Model Save/Load
  • Support big-sized model (parameter exceed > 2.1G) for both java and protobuffer
  • Support keras model loading
• Training
  • Allow user to set new train data or new criterion for optimizer reusing
  • Support gradient clipping (constant clip and clip by L2-norm)

Enhancement

• Speed up BatchNormalization.
• Speed up MSECriterion
• Speed up Adam
• Speed up static graph execution
• Support reading TFRecord files from HDFS
• Support reading raw binary files from HDFS
• Check input size in concat layer
• Add proper exception handling for CaffeLoader&Persister
• Add serialization support for multiple tensor numeric
• Add an Activity wrapper for Python to simplify the returning value
• Override joda-time in hadoop-aws to reduce compile time
• LocalOptimizer-use modelbroadcast-like method to clone module
• Time counting for paralleltable's forward/backward
• Use shade to package jar-with-dependencies to manage some package conflict
• Support loading bigdl_conf_file in multiple python zip files

Bug Fix

• Fix getModel failed in DistriOptimizer when model parameters exceed 2.1G
• Fix core number is 0 where there's only one core in system
• Fix SparseJoinTable throw exception if input’s nElement changed.
• Fix some issues found when save bigdl model to tensorflow format file
• Fix return object type error of DLClassifier.transform in Python
• Fix graph generatebackward is lost in serialization
• Fix resizing tensor to empty tensor doesn’t work properly
• Fix Adapter layer does not support different batch size at runtime
• Fix Adaper layer cannot be serialized directly
• Fix calling wrong function when set user-defined mkl threads
• Fix SmoothL1Criterion and SoftmaxWithCriterion doesn’t deal with input’s offset.
• Fix L1Regularization throw NullPointerException while broadcasting model.
• Fix CMul layer will crash for certain configure

BigDL release 0.3.0

Highlights

• New protobuf-based model storage format
• Support model quantization
• Support sparse tensor and model
• Easier and broader Tensorflow model load support
• More layers/operations
• Apache Spark 2.2 support

New Features

• Models & Layers & Operations & Loss function
  • Support convlstm3D model
  • Support Variational Auto Encoder
  • Support Unet
  • Support PTB model
  • Add SpatialWithinChannelLRN layer
  • Add 3D-deconv layer
  • Add BifurcateSplitTable layer
  • Add KLD criterion
  • Add Gaussian layer
  • Add Sampler layer
  • Add RNN decoder layer
  • Support NHWC data format in 2D-conv, 2D-pooling layers
  • Support same/valid padding type in 2D-conv and 2D-pooling layers
  • Support dynamic execution flow in Graph
  • Graph node can pass nested tensors
  • Layer/Operation can support different input and output numeric tensor
  • Start to support operations in BigDL, add following operations: LogicalNot, LogicalOr, LogicalAnd, 1D Max Pooling, Squeeze, Prod, Sum, Reshape, Identity, ReLU, Equals, Greater, Less, Switch, Merge, Floor, L2Loss, RandomUniform, Rank, MatMul, SoftMax, Conv2d, Add, Assert, Onehot, Assign, Cast, ExpandDims, MaxPool, Realdiv, BiasAdd, Pad, Tile, StridedSlice, Transpose, Negative, AssignGrad, BiasAddGrad, Deconv2D, Conv2DBackFilter CrossEntropy, MaxPoolGrad, NoOp, RandomUniform, ReluGrad, Select, Sum, Pow, BroadcastGradientArgs, Control Dependency
  • Start to support sparse layers in BigDL, add following sparse layers: SparseLinear, SparseJoinTable, DenseToSparse
• Tensor
  • Support sparse tensor
  • Support scalar (0-D tensor)
  • Tensor support more numeric type: boolean, short, int, long, string, char, bytestring
  • Tensor don’t display full content in toString when there’re too many elements
• API change
  • Expose evaluate API to python
  • Add a predictClass API to model to simplify the code when user want to use model in classification
  • Change model.test to model.evaluate in Python
  • Refine Recurrent, BiRecurrent and RnnCell API
  • Sample.features from ndarray to JTensor/List[JTensor]
  • Sample.label from ndarray to JTensor
• Install & Deploy
  • Support Apache Spark 2.2
  • Add script to run BigDL on Google DataProc platform
  • Refine run-example.sh scripts to run bigdl examples on AWS with build-in Spark
  • Pip install will now auto install spark-2.2
  • Add a docker file
• Model Save/Load
  • New model persistent format(protobuf based) to provide a better user experience when save/load bigdl models
  • Support load more operations from Tensorflow
  • Support read tensor content from Tensorflow checkpoint
  • Support load a subset of Tensorflow graph
  • Support load Tensorflow preprocessing graph(read/parse tfrecord data, image decoders and queues)
  • Automatically convert data in Tensorflow queue to RDD and feeding model training in BigDL
  • Support load deconv layer from caffe and Tensorflow
  • Support save/load SpatialCrossLRN torch module
• Training
  • Allow user to modify the optimization algorithm status when resuming the training in Python
  • Allow user to specify optimization algorithms, learning rate and learning rate decay when use BigDL in Spark * ML pipeline
  • Allow user to stop gradient on some layers in backpropagation
  • Allow user to freeze layer parameters in training
  • Add ML pipeline python API, user can use BigDL with ML pipeline in python code

Enhancement

1. Support model quantization. User can speed up model inference by quantize the model
2. Display bigdl model in Tensorboard
3. User can easily convert a sequential model to graph model by invoking new added toGraph method
4. Remove unnecessary contiguous check in 3D conv
5. Support global average pooling
6. Support regularizer in 3D convolution layer
7. Add regularizer for convlstmpeephole3d
8. Throw more meaningful messages in layers and criterions
9. Migrate GRU/LSTM/RNN/LSTM-Peehole definition from sequence to graph
10. Switch to pytest for python unit tests
11. Speed up tanh layer
12. Speed up sigmoid layer
13. Speed up recurrent layer
14. Support batch normalization in recurrent
15. Speedup Python ndarray to scala tensor convertion
16. Improve gradient sync performance in distributed training
17. Speedup tensor dot operation with mkl dot
18. Speedup copy operation in recurrent container
19. Speedup logsoftmax
20. Move classes.lst and img_class.lst to the model example folder, so user can easier to find them.
21. Ensure spark.speculation is set to false to get a better performance in training
22. Easier to turn on performance data in distributed training log
23. Optimize memory usage when broadcasting the model
24. Support mllib vector as feature for BigDL
25. Support create multiple tensors Sample in python
26. Support resizing in BytesToBGRImg

Bug Fix

1. Fix TemporalConv layer cannot return parameter table
2. Fix some bugs when loading dilated group convolution from caffe
3. Fix some bugs when loading caffe v1 layers
4. Fix a bug in TimeDistributed layer
5. Fix get incorrect execution time in recurrent layers
6. Fix inplace layer clear state bug
7. Fix incorrect training data sample count under some input
8. Remove label check in BytesToGreyImg
9. Fix a bug in concat table when it contains no layer
10. Fix a bug in maptable
11. Fix some typos in document
12. Use newInstance method to obtain FileSystem

BigDL release 0.2.0

New feature

• A new BigDL document website online https://bigdl-project.github.io/, which replace the original BigDL wiki
• Added New Models & Layers
  • TreeLSTM and examples for sentiment analytics
  • convLSTM layer
  • 1D convolution layer
  • Mean Absolute Error (MAE) metrics
  • TimeDistributed Layer
  • VolumetricConvolution(3D convolution)
  • VolumetricMaxPooling
  • RoiPooling layer
  • DiceCoefficient loss
  • bi-recurrent layers
• API change
  • Allow user to set regularization per layer
  • Allow user to set learning rate per layer
  • Add predictClass API for python
  • Add DLEstimator for Spark ML pipeline
  • Add Functional API for model definition
  • Add movie length dataset API
  • Add 4d normalize support
  • Add evaluator API to simplify model test
• Install & Deploy
  • Allow user to install BigDL from pip
  • Support win64 platform
  • A new script to auto pack/distribute python dependency on yarn cluster mode
• Model Save/Load
  • Allow user to save BigDL model as Caffe model file
  • Allow user to load/save some Tensorflow model(cover tensorflow slim APIs)
  • Support save/load model file from/to s3/hdfs
• Optimization
  • Add plateau learning rate schedule
  • Allow user to adjust optimization process based on loss and score
  • Add Exponential learning rate decay
  • Add natural exp decay learning rate schedule
  • Add multistep learning rate policy

Enhancement

1. Optimization method API refactor
2. Allow user to load a Caffe model without pre-defining a BigDL model
3. Optimize Recurrent Layers performance
4. Refine the ML pipeline related API, and add more examples
5. Optimize JoinTable layer performance
6. Allow user to use nio blockmanager on Spark 1.5
7. Refine layer parameter initialization algorithm API
8. Refine Sample class to save memory usage when cache train/test dataset as tensor format
9. Refine MiniBatch API to support padding and multiple tensors
10. Remove bigdl.sh. BigDL will set MKL behavior through MKL Java API, and user can control this via Java properties
11. Allow user to remove Spark log in redirecting log file
12. Allow user create a SpatialConvultion layer without bias
13. Refine validation metrics API
14. Refine smoothL1Criterion and reduce tensor storage usage
15. Use reflection to handle difference of Spark2 platforms, and user need not to recompile BigDL for different Spark2 platform
16. Optimize FlattenTable performance
17. Use maven package instead of script to copy dist artifacts together

Bug Fix

1. Fix some error in Text-classifier document
2. Fix a bug when call JoinTable after clearState()
3. Fix a bug in Concat layer when the dimension concatenated along is larger than 2
4. Fix a bug in MapTable layer
5. Fix some multi-thread error not catch issue
6. Fix maven artifact dependency issue
7. Fix model save method won’t close the stream issue
8. Fix a bug in BCECriterion
9. Fix some ConcatTable don’t clear gradInput buffer
10. Fix SpatialDilatedConvolution not clear gradInput content

BigDL maintenance release 0.1.1

Release Notes

• API Change

1. Use bigdl as the top level package name for all bigdl python module
2. Allow user to change the model in the optimizer
3. Allow user to define a model in python API
4. Allow user to invoke BigDL scala code from python in 3rd prject
5. Allow user to use BigDL random generator in python
6. Allow user to use forward/backward method in python
7. Add BiRnn layer to python
8. Remove useless CriterionTable layer

• Enhancement

1. Load libjmkl.so in the class load phase
2. Support python 3.5
3. Initialize gradient buffer at the start of backward to reduce the memory usage
4. Auto pack python dependency in yarn cluster mode

• Bug Fix

1. Fix optimizer continue without failure after retry maximum number
2. Fix LookupTable python API throw noSuchMethod error
3. Fix an addmv bug for 1x1 matrix
4. Fix lenet python example error
5. Fix python load text file encoding issue
6. Fix HardTanh performance issue
7. Fix data may distribute unevenly in vgg example when input partition is too large
8. Fix a bug in SpatialDilatedConvolution
9. Fix a bug in BCECriterion loss function
10. Fix a bug in Add layer
11. Fix runtime error when run BigDL on Pyspark 1.5