You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

IDIEP-20
AuthorVladimir Ozerov Ozerov
SponsorVladimir Ozerov Ozerov
Created25 Apr 2018
StatusDRAFT


Motivation

Compression is used extensively by all database vendors to reduce TCO and improve performance. Compression could be applied to different parts of the system to achieve the following goals:

  • Less writes to disk - less IO call, less disk space needed
  • Less reads from disk - less IO call, less RAM is needed to accommodate the same number of records.

Performance numbers of other vendors demonstrate that we could expect 2x-4x decrease in required disk space and >1.5x increase in throughput (ops/sec) on typical workloads.

Competitive analysis

This section describes general compression approaches and their pros and cons. The following compression mechanisms are implemented in practice:

  1. Data format improvements
  2. Index prefix compression
  3. Page-level compression
  4. WAL compression
  5. Per-column compression
  6. Compression on file system level
  7. Column store

Data Format Improvements

Efficient disk usage starts with proper data layout. Vendors strive to place data in pages in such a way that total overhead is kept as low as possible while still maintaining high read speed. Typically this is achieved as follows:

  1. Common metadata such is stored outside of data page
  2. Numeric types are written using varlen encoding (e.g. int data type may take 1-5 bytes instead of 4)
  3. Fixed-length string data types (CHAR, NCHAR) are trimmed
  4. NULL and zero values are optimized to consume no space.

Examples:

  1. SQL Server \[1\]

\[1\] https://docs.microsoft.com/en-us/sql/relational-databases/data-compression/row-compression-implementation?view=sql-server-2017

 

Draft materials

Compression options:
1) FS
2) Sparse files
3) Prefix compression (indexes)
4) Better format (varlen, common header, null fields)
5) Column store
6) Block compression
7) Per-column compression
8) Row compression
9) WAL compression
10) New algorithms (LSM, BWTree, ...)
------------------------------
Postgres:
  • Compress full pages
  • Remove holes in the page
------------------------------
MySQL: 
3) COMPRESS/UNCOMPRESS functions
"The code changes required to get the old InnoDB compression to work properly were extensive and complex. Its tentacles are everywhere—I think that just about every module inside InnoDB has had modifications done to it in order to make it work properly when compression is in use. This complexity has its challenges, both in terms of maintainability and when it comes to improving the feature. We have been debating internally about what we should do about this over the long run. As much as we would like to redesign and rewrite the entire old compression code, it is impractical. Therefore we are only fixing issues in the old compression code reported by customers. We think there are better ways to solve the underlying performance and compression problems around B-Trees. For example by adding support for other types of indexes e.g. LSM tree and/or BW-Tree or some variation of the two. "
------------------------------
MariaDB
1) WAL compression - compress event data once certain threshold is reached https://mariadb.com/kb/en/library/compressing-events-to-reduce-size-of-the-binary-log/
2) Page compression - uncompressed in memory, compressed on disk https://mariadb.com/kb/en/library/compression/
3) Old good MySQL COMPRESSED format (stores both compressed and uncompressed data in memory, use special room in page to store current modifications without recompression) https://mariadb.com/kb/en/library/xtradbinnodb-storage-formats/
3) Independent Column Compression - automatically compress and uncompress, cannot create indexes on these columns https://mariadb.com/kb/en/library/storage-engine-independent-column-compression/
4) COMPRESS function (similar as in MySQL?) https://mariadb.com/kb/en/library/compress/
?? 5) ColumnStore
------------------------------
SQL Server
1.1) NULL and 0 take no bytes!
------------------------------
Oracle
1) Basic compression - compression during bulk loads only (direct load, CREATE TABLE AS SELECT)
2) Advanced Row Compression (ex. OLTP Table Compression) - used for DML, keep data compressed in-memory; more CPU but less IO - reads gets gain anyway
3) Advanced LOB compression and deduplication - appears to be something similar to PG TOAST (?)
?? 5) Advanced Index Compression 
?? 6) Hybrid columnar compression (query level, archive level) - tremendous compression rates (up to 50x), 5-15x typical, "query" - improves scan performance, "archive" - for data archives
7) Compression at tablespace and table levels
8) Tablespace encryption - after compression, column encryption - before compression, no effect
9) Indexes are compressed separately from data!
10) Clustered tables - only prefix compression is applicable
11) Heat Map - insight on how data is accessed
12) Advanced Row Compression - can read specific attributes without full decompression
"With Advanced Row Compression, when the block is full, it is compressed. More rows are then added (since more rows can now fit into the block), and the process of recompression is repeated several times until the rows in the block cannot be compressed further. Blocks are usually compressed and reformatted in their entirety, but, starting with Oracle Database 12c Release 2, in some cases the block can be partially compressed, hence resulting in CPU savings and extra compression."
------------------------------
MongoDB
4) Configurable per-collection and per-index
"The cache generally stores uncompressed changes (the exception is for very large documents). The default snappy compression is fairly straightforward: it gathers data up to a maximum of 32KB, compresses it, and if compression is successful, writes the block rounded up to the nearest 4KB.
The alternative zlib compression works a little differently: it will gather more data and compress enough to fill a 32KB block on disk. This is more CPU intensive but generally results in better compression ratios (independent of the inherent differences between snappy and zlib)."
—Michael Cahill

Motivation

TBD

Proposed changes

TBD

 

Tickets

key summary type created updated due assignee reporter priority status resolution

JQL and issue key arguments for this macro require at least one Jira application link to be configured

  • No labels

Apache Ignite