THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Page History
...
Data is stored in-memory as is, in uncompressed form. When it is time to flush data to disk compression is applied. If data size is reduced significantly, data is stored in compressed form. Otherwise it is stored in plain form (compression faiure). Big block sizes (e.g. 32Kb) is typically used in this case to achieve higher compression rates. Data is still being written to disk in blocks of smaller sizes. E.g. one may have 32Kb block in-memory, which is compressed to 7Kb, which is then written as two 4Kb blocks to disk. Vendors allow to select compression algorithm (Snappy, zlib, lz4, etc.).
Page compression is not applicable to index pages because it incurs serious slowdow no reads.
Hole punching with fallocate [1] might be added if underlying file system supports it. In this case compressed block is written as is, but then empty space is trimmed with separate system call. E.g. if 32Kb block is compressed to 6.5Kb, then 32Kb is written as is, and then 32 - 7 = 25 Kb are released.
...
- High RAM usage
- Need to re-compress data frequently (spe
- Hole-punching is supported by very few file systems (XFS, ext4, Btrfs), and may lead to heavy file maintenance [2], [7]
Examples:
- MySQL Table Compression - uses different in-memory and disk block sizes, block data is fully re-compressed on every access [3]
- MySQL Page Compression - uses hole-punching instead [4]
- MongoDB with Snappy codec - gathers up to 32Kb of data and then try to compress it [5]5
- Postgres Professional attempts to implement similar approach - data will be uncompressed when read from disk [6]
[1] http://man7.org/linux/man-pages/man2/fallocate.2.html
[2] https://mariadb.org/innodb-holepunch-compression-vs-the-filesystem-in-mariadb-10-1/
[3] https://dev.mysql.com/doc/refman/5.7/en/innodb-compression-background.html
[4] https://mysqlserverteam.com/innodb-transparent-page-compression/
[5] https://www.objectrocket.com/blog/company/mongodb-3-0-wiredtiger-compression-and-performance/
[6] https://habr.com/company/postgrespro/blog/337180/
[7] https://www.percona.com/blog/2017/11/20/innodb-page-compression/
With in-memory compression
Data is stored in-memory in compressed form. Data is accomodated in data block in raw form
Draft materials
Compression options:
1) FS
2) Sparse files
3) Prefix compression (indexes)
4) Better format (varlen, common header, null fields)
5) Column store6) Block compression
7) Per-column compression
...
- Compress full pages
- Remove holes in the page
------------------------------
MySQL: 1) Table Compression https://dev.mysql.com/doc/refman/5.7/en/innodb-compression-background.html
2) Page Compression https://mysqlserverteam.com/innodb-transparent-page-compression/
3) COMPRESS/UNCOMPRESS functions
...
7) Column compression in Percona - https://www.percona.com/live/e17/sites/default/files/slides/Percona%20XtraDB_%20Compressed%20Columns%20with%20Dictionaries%20-%20An%20Alternative%20to%20InnoDB%20Table%20Compression%20-%20FileId%20-%20115392.pdf
"The code changes required to get the old InnoDB compression to work properly were extensive and complex. Its tentacles are everywhere—I think that just about every module inside InnoDB has had modifications done to it in order to make it work properly when compression is in use. This complexity has its challenges, both in terms of maintainability and when it comes to improving the feature. We have been debating internally about what we should do about this over the long run. As much as we would like to redesign and rewrite the entire old compression code, it is impractical. Therefore we are only fixing issues in the old compression code reported by customers. We think there are better ways to solve the underlying performance and compression problems around B-Trees. For example by adding support for other types of indexes e.g. LSM tree and/or BW-Tree or some variation of the two. "
------------------------------
...
1) WAL compression - compress event data once certain threshold is reached https://mariadb.com/kb/en/library/compressing-events-to-reduce-size-of-the-binary-log/
2) Page compression - uncompressed in memory, compressed on disk https://mariadb.com/kb/en/library/compression/
3) Old good MySQL COMPRESSED format (stores both compressed and uncompressed data in memory, use special room in page to store current modifications without recompression) https://mariadb.com/kb/en/library/xtradbinnodb-storage-formats/3) Independent Column Compression - automatically compress and uncompress, cannot create indexes on these columns https://mariadb.com/kb/en/library/storage-engine-independent-column-compression/
4) COMPRESS function (similar as in MySQL?) https://mariadb.com/kb/en/library/compress/
?? 5) ColumnStore
General link: https://mariadb.com/kb/en/library/optimization-and-tuning-compression/Bad experience with punch
-holes in MySQL: https://mariadb.org/innodb-holepunch-compression-vs-the-filesystem-in-mariadb-10-1/------------------------------
SQL Server
1) Row compression - metadata, varlen for numeric types, trimming for CHAR types https://docs.microsoft.com/en-us/sql/relational-databases/data-compression/row-compression-implementation?view=sql-server-2017
...
------------------------------
MongoDB
1) Block compression (Snappy, zlib) https://docs.mongodb.com/manual/core/wiredtiger/#compression
2) Prefix compression (indexes, once per page) https://docs.mongodb.com/manual/reference/glossary/#term-prefix-compression
3) WAL compression https://docs.mongodb.com/manual/core/wiredtiger/#storage-wiredtiger-journal
4) Configurable per-collection and per-indexindex https://www.mongodb.com/blog/post/new-compression-options-mongodb-30
"The cache generally stores uncompressed changes (the exception is for very large documents). The default snappy compression is fairly straightforward: it gathers data up to a maximum of 32KB, compresses it, and if compression is successful, writes the block rounded up to the nearest 4KB.
The alternative zlib compression works a little differently: it will gather more data and compress enough to fill a 32KB block on disk. This is more CPU intensive but generally results in better compression ratios (independent of the inherent differences between snappy and zlib)."
—Michael Cahill
Motivation
TBD
Proposed changes
...
Overview
Content Tools
Apps