Question

I have some data which takes up more than 50MB in an uncompressed file, but compresses down to less than half a MB using gzip.

Most of this is numerical data. I m trying to figure out how to process this data without having to uncompress it completely. For example, if this data contains a couple of strings and 5 or so numerical values per record, is there a way I can uncompress a single row (or a small set of rows), process them, then discard them?

Unix provides utilities such as zcat, grep, etc. which operate directly on compressed data, I d like to do the same in Java.

Thanks

Answer 1

Just wrap your FileInputStream in a GZipInputStream:

public static BufferedReader createReader (File f, String encoding) throws IOException
{
    try
    {
        InputStream in = new FileInputStream (f);
        if (f.getName ().endsWith (".gz"))
            in = new GZIPInputStream (in, 10240);

        return new BufferedReader (new InputStreamReader (in, encoding));
    }
    catch (UnsupportedEncodingException e)
    {
        throw new RuntimeException("Missing encoding "+encoding, e);
    }
}

友情链接