Question

I have to write a tool that manages very large data sets (well, large for an ordinary workstations). I need basically something that works the opposite that the jpeg format. I need the dataset to be intact on disk where it can be arbitrarily large, but then it needs to be lossy compressed when it gets read in memory and only the sub-part used at any given time need to be uncompressed on the flight. I have started looking at ipp (Intel Integrated Performance Primitives) but it s not really clear for now if I can use them for what I need to do. Can anyone point me in the right direction?

Thank you.

Answer 1

Given the nature of your data, it seems you are handling some kind of raw sample. So the easiest and most generic "lossy" technique will be to drop the lower bits, reducing precision, up to the level you want.

请注意,你需要“选择低轨道”,这与“四舍五入到十强”截然不同。第2基地的计算机工作,你希望你所有的下限都是“000”,以便尽可能压缩。这种方法假定,选定的压缩算法将利用可预测的0轨道模式。

另一种更为复杂、更具体的方法是把你的价值观作为指数转化为一个表。好处是,在你想要时,你可以“瞄准”精确度。明显的缺点是,表将专门针对分配模式。

其中最重要的是,如果两者之间有某种关系,你可能不会储存价值本身,而是储存价值上值的三角洲。这将有助于压缩。

为了压缩数据,你将需要通过适当规模的包装,如64KB,“组别”。在单一领域,任何压缩算法都不会给你带来适当的结果。这反过来又意味着,每当你想要进入某个领域时,你都需要压缩整个包装,这样,它才能根据你想要做些什么加以调整。在这种情形下,定期接触更容易处理。

关于压缩算法,由于这些数据将属于“寿命”,你需要非常快,因此获得数据的影响很小。

There are several open-source alternatives out there for that use. For easier license management, i would recommend a BSD alternative. Since you use C++, the following ones look suitable : http://code.google.com/p/snappy/ and http://code.google.com/p/lz4/

友情链接