Apache SystemDS 1.1.0 Release Notes

The Apache SystemML 1.1.0 release was approved on March 28, 2018. The release includes enhancements, features, and additions as listed below.

New Builtin Functions/Operations

  • ifelse
  • assert
  • Deep learning builtin functions: avg_pool, avg_pool_backward
  • Second-order eval
  • Accumulator operator +=
  • Bitwise operators not, and, or, xor, & LShift, Rshift
  • Logical operator support over matrices AND/OR/NOT/XOR

Additional Layers in the NN library

  • Average pooling
  • Upsampling
  • Low-rank fully connected

New Capabilities/Features

  • Dense matrix blocks >16GB, and operations
  • Enable additional ParFor result aggregation operations
  • UDFs callable in expressions
  • Zero rows/columns matrices and updated operations such as removeEmpty
  • Matrix-matrix multiplication over compressed matrices
  • Extended Caffe2DML and Keras2DML APIs

Compiler & Runtime

  • Use common thread pool
  • Single-precision support for native conv2d and mm operations
  • Improved nnz maintenance, runtime propogation and memory management
  • Robustness for matrices with larger than int dimensions
  • [Experimental] Codegen extensions: operation support, extended optimizer, see SYSTEMML-2065

Performance Improvements

  • sparse left indexing, sparse reshape, ultra-sparse transpose, ultra-sparse rand, binary in-place operations, sparse relu backward, maxpooling, sparse im2col, ultra-sparse conv2d, read of short-wide sparse matrices, avoid unnecessary evictions, lock-free statistics maintenance, spark cpmm, spark aggregates, spark reshape, spark binary ops, etc.

Bug Fixes

  • in APIs, performance, optimizer, runtime, GPU backend, Spark backend

Deprecate

  • Support for Spark 2.1 / 2.2 (make switch to newer ANTLR version)

JIRA release notes