Support for hparams

[jlperla]

It is very useful to have hyperparameters when looking at logs. i.e. https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_hparams

Is this possible to add to the TensorBoardLogger.jl? Either giving a dictionary or just calling yourself on each dictionary item.

[PhilipVinc]

I agree it'd be very useful.

TensorBoardLogger does not know about the hparams plugin at the moment. However adding support for that wouldn't be too hard. I already outlined the steps necessary to do so in this comment.

I'm (unfortunately) on a python's hiatus which will last a while, and don't have time to dedicate to this.
However, if you were interested in putting in the work for this (I'd say it would take no more than a few days of work, including the time to get familiar with the codebase), I'd be glad to help you and maybe could take care of the more technical part (generating the right .proto definitions)

[PhilipVinc]

By the way, I'd be \infty-ely grateful if you found the time to contribute the hparams plugin.

[jlperla]

Thanks. Alas, I fear it is outside of my competence, knowledge, or bandwidth at this point. But if I come across an interested grad student/etc. I will point them this way.

[dangirsh]

I've started working on this here

[PhilipVinc]

@dangirsh Great!
If you open a PR I'll keep an eye on it from time to time.

[PhilipVinc]

@JamieMair as you expressed interest in tackling this...

@dangirsh had already started the work, and I pushed it a bit further, so you should start from PR #99 . You should also give a look at my remarks in the review of #87 , which I'm not sure I had addressed fully in #99.

IIRC (but you should check) we had already wrapped the functionality , the remaining work is mainly about identifying what quantities/structures can be logged as hparams and design the API that TBLogger should expose, and implement it.

If you take #99 you should already be capable of logging hparams, but the api is terrible.

[PhilipVinc]

The 'back-end' is implemented in src/Loggers/HParams.jl. The back-end entry point is log_hparams which creates a SummaryColelction containing a hparams_summary and serialises it to disk.

Most of the work will therefore be in hparams_summary and the methods it calls.

From-end wise, the nice @info myvalue API system that dispatches objects of certain types to certain backends (like Audio, Images, Scalars, and Hparams) are contained in logger_dispatch_overrides, but that should come after.

[JamieMair]

Thanks for this! I'll take a look and see what I can do

[JamieMair]

@PhilipVinc

I had some time to look at this, right now I've made very few changes, but I've made a list of tasks I need to do. Right now, I have focused on getting the unit tests to pass (they still need a lot of work!). See changes - JamieMair@b54d430
The first test HParamConfig Logger appears to mostly work now, except for the metrics, as it does not track metrics written by the "log_value" function which mimics the "tf.summary.scalar" function in the python API. The second test does not show any values, but shows the headers, I believe you need to write a config first. But this can be an easy fix in the API.

Are there any resources that you recommend for understanding the ProtoBuf code? I am trying to follow the python source https://github.com/tensorflow/tensorboard/tree/master/tensorboard/plugins/hparams, but most of the functionality is hidden behind the api_pb2 module, which I cannot find the source code for.

[JamieMair]

This issue should be closed as the new API is merged.