mts: General-purpose meta data tags in PSI

[Alan Noble]

This is an enhancement request to generalise how we read and write meta data using MTS PSI.

I propose adding the following methods to read/write arbitrary key/value pairs, where keys are strings and values are strings or numbers (64-bit floats would suffice).

MetaNumber(key string) float64
MetaString(key string) string
Meta() map[string]interface{} // return all metadata

AddMetaNumber(key string, val float64)
AddMetaString(key, val string)

psi.TimeFrom() then becomes psi.MetaNumber("time").

psi.LocationFrom() then becomes psi.MetaString("location"), or or if we prefer to store the numbers, psi.MetaNumber("lat") and psi.MetaNumber("lng").

proposal doc: https://docs.google.com/document/d/1ObOzYjQNy-eOzbr_AICERvIL6gVfocKuw54a2TyJEyA/edit

[Saxon Milton]

Because we only deal with a struct on creation of a psi, and then just use a slice representation from then on, rather than having psi.MetaNumber("lng"), should we just have Meta(psi []byte, descTag int) []byte, where psi is the byte slice of the psi, and descTag is the metadata descriptor used to identify that particular type of meta and the return is the descriptor data, i.e. the metadata. We then leave it up to the caller to convert the returned slice to the desired type.

Similarly, if we would like to add metadata to a slice representation of a psi, we would have AddMeta(psi []byte, descTag int, data []byte) error.

Alternatively, we could still have the psi struct, but we actually store the byte representation of this struct as field. So then you can call receiver functions on the struct to modify the slice representation and also update the struct fields. i.e. what you have suggested above. It just means you're doing more work by potentially allocating data in the psi struct.

[Alan Noble]

I would prefer if keep it more abstract, and keep the internal []byte representation hidden from the caller.

As a simplification, we could probably just use strings for everything, so 3 methods:

Meta.Get(key string) string
Meta.All() map[string]interface{} // returns all metadata
Meta.Add(key, val string)

The mapping to []byte is easy if everything is a string. We could simply write to the []byte in CSV format, i.e., "key1=value1,key2=value2,..."

Of we could use amf encoding/decoding which go to/from []byte.

[kortschak]

Agree with Alan. Also remember that there is a good prospect of moving to gots, which does not use a struct marshaling approach.

[Alan Noble]

And the All() method would of course be a map of string values, i.e.,

Meta.All() map[string]string{}

In fact, the map could be readable and writable and no other methods would be required.

[Saxon Milton]

So you would like to have 'Meta' global inside encoder.go ? How do you think we should modify the actual psi ? Should we have receiver functions for the psi struct that can update, add or remove metadata by altering the psi's byte slice representation field?

[Saxon Milton]

Also, does comcast gots actually provide the capability of creating and altering psi ? From what I can see, it can only parse pmt bytes, and populate a PMT struct ?

[Saxon Milton]

Also, why is the struct marshaling a bad idea ? Just a reminder that when we do update the meta data in a pmt, I am just dealing with the bytes slice and copying over existing fields, I don't re-marshal the struct.

[Alan Noble]

In general, I think our encoder needs to somehow support the notion of "out of band" data, i.e., there is the multimedia data (which it handles now) and there is separate metadata that gets passed in (that we store in the PSI). The OOB data might lend itself to a callback approach.

That said, I prefer not to design it on the fly in these tiny comment boxes! Please write a short design note proposal in Google Docs and share.

[kortschak]

Writing a gots packet.Packet involves zero allocations, being Write(p[:]) where p is a *packet.Packet. In the life span of a packet, there is no allocation except for the initial one, and this can be done on the stack, and reused.

What do you want to see in PSI that gots doesn't have? These can be added.

[Saxon Milton]

Alright I will write a google doc

[Saxon Milton]

@kortschak While I would like to improve how we do the mts stuff, i.e. move to comcast gots approach, I think it can hold for now ? I also don't entirely understand how you would like to do it - could you write a detailed proposal when you get a chance?

[kortschak]

I don't think it needs a detailed google docs proposal (by you or me). It just needs a description here about what you are writing into the packets the way that you would if you were doing it with the approach you are using since they will be isomorphic.

[Saxon Milton]

here or on another issue ? I don't really want to deal with the comcast gots stuff with this issue.

[kortschak]

Feel free to open another issue. It was included because of https://bitbucket.org/ausocean/av/issues/36/mts-general-purpose-meta-data-tags-in-psi#comment-50198150 and the response https://bitbucket.org/ausocean/av/issues/36/mts-general-purpose-meta-data-tags-in-psi#comment-50198892. There is an implication in the first that there is an approach that is a consequence of the struct-based approach.

[Saxon Milton]

This proposal was implemented in PR #131