Abstract

A system and method for filesystem data compression using codebooks, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams. Further, the system comprises a system for integrating the compression into a filesystem for both system- wide compression on a per-file or filegroup basis, and intrusion or alteration detection of files.

IPC Classes  ?

• H03M 7/30 - CompressionExpansionSuppression of unnecessary data, e.g. redundancy reduction
• G06N 20/00 - Machine learning

Abstract

Codebook data compaction using a universal codebook and mismatch probability estimations to improve entropy encoding methods. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Entropy encoding is used to generate codebooks comprising codewords for data sourceblocks based on the frequency of occurrence of each sourceblock. A "mismatch codeword" is inserted into the codebook based on the mismatch probability estimate to represent those cases when a block of data to be encoded does not have a codeword in the codebook.

IPC Classes  ?

• H04N 19/13 - Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• G06F 3/06 - Digital input from, or digital output to, record carriers

Abstract

A system and method for data compression with protocol adaptation, that utilizes a codebook generator which leverages one or more machine/ deep learning algorithms trained on at least a plurality of protocol policies in order to generate a protocol appendix and codebook, wherein original data is encoded by an encoder according to the codebook and sent to a decoder, but instead of just decoding the data according to the codebook to reconstruct the original data, data manipulation rules such as mapping and transformation are applied at the decoding stage to transform the decoded data into protocol formatted data.

IPC Classes  ?

• H03M 7/30 - CompressionExpansionSuppression of unnecessary data, e.g. redundancy reduction
• G06N 20/00 - Machine learning

Abstract

A system for operating system virtualization using codebook encoding to improve entropy encoding methods. Training data sets are analyzed to determine occurrence frequency of each sourceblock in the data sets. A mismatch probability estimate is calculated at which any given data sourceblock received will not have a codeword in the codebook. Entropy encoding is used to generate codebooks comprising codewords for sourceblocks based on the frequency of occurrence of each. A "mismatch codeword" is inserted into the codebook to represent those cases when a block of data does not have a codeword.

IPC Classes  ?

• G06F 9/44 - Arrangements for executing specific programs
• G06F 9/455 - EmulationInterpretationSoftware simulation, e.g. virtualisation or emulation of application or operating system execution engines
• H03M 7/30 - CompressionExpansionSuppression of unnecessary data, e.g. redundancy reduction

Abstract

A system and method for data compaction utilizing distributed codebook encoding to improve entropy encoding methods to account for, and efficiently handle, previously-unseen data in data to be compacted, allow for distributed encoding and decoding capabilities, and allow for parametrized codebook encoding methods. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability' estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Further, a codebook and a behavior codebook may both be maintained or altered in a distributed fashion across multiple devices or services, for widespread, or permission-based, or parametrized codebook encoding.

IPC Classes  ?

• H04N 19/13 - Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• G06F 3/06 - Digital input from, or digital output to, record carriers

Abstract

A system and methods for data compression with encryption using asymmetric codebooks, that utilizes machine learning to generate asymmetric encoder and decoder codebooks for data encoding and compression. The use of asymmetric codebooks enables a variety of data obfuscation techniques such that output data remains useful to the decoder and any downstream entity, while concealing the original sourceblock data. The asymmetric codebooks may be utilized in reverse order to restore the original data as needed, facilitating a two-way reversible operation.

IPC Classes  ?

• G06N 20/00 - Machine learning
• H03M 7/30 - CompressionExpansionSuppression of unnecessary data, e.g. redundancy reduction

Abstract

A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams.

IPC Classes  ?

• H03M 7/30 - CompressionExpansionSuppression of unnecessary data, e.g. redundancy reduction

Abstract

A system and methods for bandwidth-efficient cryptographic data transfer, utilizing an encoding endpoint device, a decoding endpoint device, a reference codebook, and a plurality of data to encode and decode, which may use specific algorithms on top of block cipher encryption to achieve higher data security and ease the burden on users with regards to computational power, complexity, and bandwidth for communication.

IPC Classes  ?

• G06F 16/174 - Redundancy elimination performed by the file system
• H04L 9/00 - Arrangements for secret or secure communicationsNetwork security protocols
• G06N 20/00 - Machine learning

Abstract

A system and method for data storage, transfer, synchronization, and security using recursive encoding, wherein data is deconstructed into chunklets, and is processed through a series of reference code libraries that reduce the data to a sequence of reference codes, and where the output of each reference library is used as the input to the next.

IPC Classes  ?

• G06F 16/174 - Redundancy elimination performed by the file system
• H04L 9/00 - Arrangements for secret or secure communicationsNetwork security protocols

Abstract

A system and method for high-speed transfer of small data sets, that provides near-instantaneous bit- level lossless compression, that is ideal for communications environments that cannot tolerate even small amounts of data corruption, have very low latency tolerance, where data has a low entropy rate, and where every bit costs the user bandwidth, power, or time so that deflation is worthwhile. Where some loss of data can be tolerated, the system and method can be configured for use as lossy compression.

IPC Classes  ?

• H03M 7/40 - Conversion to or from variable length codes, e.g. Shannon-Fano code, Huffman code, Morse code
• H03M 7/30 - CompressionExpansionSuppression of unnecessary data, e.g. redundancy reduction