Summary

Important highlights from this release

1. Highlight 1
2. Highlight 2
3. Highlight 3

All updates for this release

Epic

• SYSTEMDS-3572 This epic contains all the task associated with morphing of columngroups.
• SYSTEMDS-3019 JUnit test suite for GPU
• SYSTEMDS-2840 Add Builtin Functions
• SYSTEMDS-493 Modularize Existing DML Algorithms

New Feature

• SYSTEMDS-3640 Hash Column in Frames
• SYSTEMDS-3635 Cast as frame with column names
• SYSTEMDS-3602 Settings warnings (Memory)
• SYSTEMDS-3601 Performance Jar
• SYSTEMDS-3592 Compress Frame
• SYSTEMDS-3590 Frame DDC Encoding
• SYSTEMDS-3578 ColGroup Info for Morphing individual groups.
• SYSTEMDS-3576 Morph ColumnGroup instruction
• SYSTEMDS-3575 Column Group get compression Info
• SYSTEMDS-3574 Recompress Compressed Matrix
• SYSTEMDS-3573 CLALibCombineGroups
• SYSTEMDS-3570 Error statistics builtins
• SYSTEMDS-3495 Parallel Compressed Encode
• SYSTEMDS-3405 Federated Write at site
• SYSTEMDS-3317 Lineage-based pruning of ML tasks
• SYSTEMDS-2879 Cspline Predict function
• SYSTEMDS-1027 Compile to Julia

Improvement

• SYSTEMDS-3663 Low overhead join indexes
• SYSTEMDS-3662 Parfor Merge Sparse
• SYSTEMDS-3652 Add row direction for unique() builtin
• SYSTEMDS-3637 Manifest for SystemDS
• SYSTEMDS-3619 IO save Encoded
• SYSTEMDS-3618 Compresseed Schemas
• SYSTEMDS-3610 BuiltinSwitch Aggregate Skip
• SYSTEMDS-3609 Common ThreadPool Reuse ThreadLocal Pools
• SYSTEMDS-3608 Cocode shortcut
• SYSTEMDS-3607 Split Balanced Internal
• SYSTEMDS-3532 CLA Compressed Parallel Writing
• SYSTEMDS-3531 Unify Application Function and Component Test GitHub Action
• SYSTEMDS-3528 Refine allocation of GitHub Artifacts
• SYSTEMDS-3527 CLA DeltaOffset Skip list
• SYSTEMDS-3526 Transpose Sparse to Dense
• SYSTEMDS-3525 Binary Inplace Operations
• SYSTEMDS-3508 Remove Spark Checkpoint if CP
• SYSTEMDS-3491 CLA Specialized Column Indexes
• SYSTEMDS-3490 Compressed Transform Encode
• SYSTEMDS-2985 List nested access
• SYSTEMDS-2753 Approximation and information loss information for LCLA
• SYSTEMDS-2102 Vectorize gradients for Factorization Machines function
• SYSTEMDS-496 Running dml on hadoop writes files to dir that requires root permission
• SYSTEMDS-339 DML-Improvement. Unneccessary warn in If else

Bug

• SYSTEMDS-3673 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder"
• SYSTEMDS-3672 GIO Test Cases false positive
• SYSTEMDS-3667 Erroneous cleanup of List objects after function return
• SYSTEMDS-3659 Federated GitHub Actions Fail
• SYSTEMDS-3657 smaller adjustments around the word embedding encoder
• SYSTEMDS-3654 FederatedCentralMomentTest infinite cloud runtime
• SYSTEMDS-3642 CLA NaN in Dictionaries replace
• SYSTEMDS-3639 logLoss Nan
• SYSTEMDS-3622 Single-threaded libsvm writer hangs forever
• SYSTEMDS-3617 Offsets Separated
• SYSTEMDS-3612 Persistence is not respected for RDD
• SYSTEMDS-3606 pnmf with ultra-sparse data never finishes
• SYSTEMDS-3600 POM Shader bug
• SYSTEMDS-3598 Federated Compression Redundant Release
• SYSTEMDS-3596 Incorrect heavy hitter statistics
• SYSTEMDS-3593 Fix test coverage badges
• SYSTEMDS-3591 Recompilation Issue XGBoost
• SYSTEMDS-3588 Federated Remote Monitoring
• SYSTEMDS-3587 Federated missing variable on toString
• SYSTEMDS-3586 Federated missing release variable
• SYSTEMDS-3584 OOM while reading matrix from HDFS
• SYSTEMDS-3577 Fix default storage memory fractions in Spark
• SYSTEMDS-3520 Error if multiple encoders applied on one column
• SYSTEMDS-3514 Parfor optimizer adds redundant checkpointing statement blocks
• SYSTEMDS-3512 Bug in Decision Trees
• SYSTEMDS-3509 Function call with named arguments misassigns variables
• SYSTEMDS-3504 GPU Junit tests
• SYSTEMDS-3502 Perftest: lmDS on 1M_1k_dense with unnecessary read from checkpoint
• SYSTEMDS-3441 Monitoring figues lies enter bottom.
• SYSTEMDS-3423 Debug runtime of Animal dataset
• SYSTEMDS-3391 Correct the release artifact generation date
• SYSTEMDS-3372 Regression in Federated SSL Tests
• SYSTEMDS-3315 Fix lineage exploitation in the buffer pool
• SYSTEMDS-3292 Decision tree training built-in does not stop when ran with data generated for logistic regression
• SYSTEMDS-3251 Unstable Tests: Async PrefetchRDD
• SYSTEMDS-3200 Federated Lineage full reuse bug
• SYSTEMDS-3110 Codegen CNodeBinary - No enum constant RBIND
• SYSTEMDS-3077 Codegen failing w/ unsupported operation
• SYSTEMDS-3071 Builtin MultiLogReg Fails if forced spark mode.
• SYSTEMDS-3070 Spark aggregation on 0 row or col matrix
• SYSTEMDS-3038 CLA Spark Left Multiplication No Unification of matrix
• SYSTEMDS-3029 Missing estimate of intermediate size in codegen hop SpoofFusedOp
• SYSTEMDS-2672 Range Indexing (all rows, columns) recompile
• SYSTEMDS-2541 Runtime exception in frame cbind operation in Spark context
• SYSTEMDS-2363 Inflexible write format parameter requires constant propagation
• SYSTEMDS-2341 Improve right indexing recompilation
• SYSTEMDS-2316 Data type propagation issue with mixed type accumulate
• SYSTEMDS-2080 Lenet over mnist shows accuracy issues on CPU backend
• SYSTEMDS-1965 Refactor nn layers to move the computation in forward/backward function known at compile time to init function
• SYSTEMDS-1597 Random Forest With Categorical Variables
• SYSTEMDS-1371 The "sparsity" parameter for the "rand" function is confusingly named
• SYSTEMDS-624 writeMM and write.csv code currently not called
• SYSTEMDS-614 Single variable DML script reports wrong line number

Task

• SYSTEMDS-3661 Native builtin for contains row vector operations
• SYSTEMDS-3636 Sparse Transpose-Self MatMult w/ Sparse Output
• SYSTEMDS-3615 Array-based lookups in recode transformdecode
• SYSTEMDS-3613 Missing Size Propagation transformapply/transformdecode
• SYSTEMDS-3595 Extended lineage exploitation in the buffer pool
• SYSTEMDS-3542 Compressed transfer to/from federated workers
• SYSTEMDS-3329 Builtin for pageRank (multiple methods)
• SYSTEMDS-3153 Missing value imputation using KNN
• SYSTEMDS-3149 Decision Tree cleanups and Builtin
• SYSTEMDS-3135 Reader Generation for Custom Text Formats
• SYSTEMDS-346 Document SystemML best practices
• SYSTEMDS-229 New result merge function 'append' (row and column append)

Sub-task

• SYSTEMDS-3646 Extend max_parallelize operator ordering to support GPU OPs
• SYSTEMDS-3582 Optional Data Compaction in decisionTree/randomForest
• SYSTEMDS-3567 Cost-based eviction from GPU lineage cache
• SYSTEMDS-3564 Cost-based operator ordering
• SYSTEMDS-3562 Compile-time chkpoint placement for Spark operators in loop
• SYSTEMDS-3555 Add unit tests for word embedding
• SYSTEMDS-3522 Add missing binning decoder to transformdecode
• SYSTEMDS-3513 Multi-level reuse for GPU intermediates
• SYSTEMDS-3510 Recycling intermediates from GPU lineage cache
• SYSTEMDS-3506 Additional decisionTree/randomForest inference methods
• SYSTEMDS-3505 Cleanup randomForest builtin
• SYSTEMDS-3483 Compile-time chkpoint placement for shared Spark operators
• SYSTEMDS-3277 Cleanup decisionTree builtin
• SYSTEMDS-3140 Structured data formats (JSON, XML)
• SYSTEMDS-3139 Performance improvements readers
• SYSTEMDS-3138 Performance improvements analysis
• SYSTEMDS-3137 Matrix and Frame Reader Generation
• SYSTEMDS-3136 Analysis: structure Identification
• SYSTEMDS-2947 Lineage cache eviction in GPU
• SYSTEMDS-1320 Map writeMM and write.csv to write

Dependency

• SYSTEMDS-3656 Update Hadoop 3.3.6
• SYSTEMDS-3655 Update Spark Version 3.5.0
• SYSTEMDS-3375 CUDA11 / CUDNN8 support

Dependency upgrade

• SYSTEMDS-3515 node-sass and browser-sync upgrade

Story

• SYSTEMDS-3087 Memory management and lazy evaluation in dynamic environments

Wish

• SYSTEMDS-3428 Code coverage in GitHub Actions (Java)
• SYSTEMDS-3427 Code coverage in Github Actions (Python)