Proposal: uncompressed input size

[natefoo]

Currently input_size is the size of the raw input, which can be either compressed or uncompressed. When scaling memory based on input size you probably only care about the uncompressed size. But gzip does store the uncompressed size, which we could read into a separate uncompressed_jnput_size variable. The uncompressed size is stored in the last 4 bytes, this seems to work for me:

#!/usr/bin/env python3
import os
import sys

path = sys.argv[1]

with open(path, 'rb') as f:
    f.seek(-4, os.SEEK_END)
    size = int.from_bytes(f.read(4), 'little')
    print(size)

The uncompressed size also isn't always set properly:

nate@pdp-11% gzip -l /home/nate/work/galaxy/test-data/1.bam
         compressed        uncompressed  ratio uncompressed_name
               3592                   0   0.0% /home/nate/work/galaxy/test-data/1.bam

So we should have a default... actual size, or actual size * some constant factor.

[mvdbeek]

Currently input_size is the size of the raw input,

That seems like a bug and something to be fixed in that helper ? Would that close galaxyproject/galaxy#19280 ?

[natefoo]

Sure - you think this should be implemented in TPV though, not in Galaxy? I am not thrilled that TPV would need to pull the data into the cache if it's on object storage.

[mvdbeek]

Ah, i would be the last person to do that ;). I'm saying either we know which converted dataset we use, or we fix that.

[natefoo]

We discussed this out of band but for everyone else, I am not talking here about the converted dataset, I mean having access to the uncompressed size of native compressed (e.g. fastqsanger.gz) inputs so that you can scale memory based on the size of the uncompressed reads regardless of whether the input is compressed.

As a stopgap or maybe alternative if we don't want to store this in Galaxy, we did discuss that you could instead query and calculate stats on memory usage by input separated by compressed and uncompressed (i need to write a query for this) and then use a rule construct such as the following:

- if: |
    datasets = [jtid.dataset.dataset for jtid in job.input_datasets if jtid.dataset]
    input_is_compressed = any([d.ext.endswith(".gz") or d.ext.endswith(".bz2") for d in datasets])
    (input_is_compressed and 0.1 < input_size < 2.0) or (not input_is_compressed and 0.5 < input_size < 4.0)
  cores: 4
  mem: 28

Or without the helper:

[mvdbeek]

That rule is not good for performance (ORM overhead is likely significant here), if you do that I would suggest you write a helper that builds a core statement. That helper might also estimate uncompressed size so you don't need the 2 or statements.

[natefoo]

What's the main penalty? Do we lazy load each jtid -> (h)da or something?

[mvdbeek]

yes, plus the ORM overhead. we've optimized this code so much on the galaxy side it's painful to tear it back down in tpv :(

[mvdbeek]

you can scale memory based on the size of the uncompressed reads regardless of whether the input is compressed.

having looked at this, I think what generalizes much better and is more amenable to automatic regression is having rules per input extensions, that'll also cover tools that have different characteristic per datatype (e.g. bandage which accepts gfa and fasta, and has a different intercept and slope but where both can produce a good fit). xref galaxyproject/gxadmin#173 which gives you stats per input extension