datetime value units

[robambalu]

Hey, for the datetime type it looks like perspective only supports sending datetime as epoch time in milliseconds.
Our systems can generate time in various units ( seconds, millis, micros, etc ) so we find ourselves converting every single datetime data event on the front end into millis before pushing into perspective.
Is there a way to control the expected units of a datetime column, and if not is there an appetite to add support for it?

Thanks

• Rob

[sc1f]

From 0.10.0 onwards you should be able to create an expression column to do the conversion within perspective: ``` // datetime column datetime(“seconds_timestamp” * 1000) ``` In this case seconds_timestamp is a float column, and the output of the expression is a datetime. On Jul 30, 2021, at 11:08 PM, robambalu ***@***.***> wrote:  Hey, for the datetime type it looks like perspective only supports sending datetime as epoch time in milliseconds. Our systems can generate time in various units ( seconds, millis, micros, etc ) so we find ourselves converting every single datetime data event on the front end into millis before pushing into perspective. Is there a way to control the expected units of a datetime column, and if not is there an appetite to add support for it? Thanks * Rob — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#1500>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADE3TKYCOGN2MIOQQNKV3HTT2K553ANCNFSM5BIWQYKA>.

[sc1f]

A handy GIF of the fix in action:

If you have many columns that need the conversion, you can auto-generate the expressions and apply them to the viewer/PerspectiveWidget using the expressions attribute, which takes a list of string expressions.

[robambalu]

Understood, thank you for that. Do you think it make sense to avoid the need for this type of conversation and support different datetime units when defining the schema? Similar to how arrow has it From: Jun Tan ***@***.***> Sent: Wednesday, August 4, 2021 12:16 PM To: finos/perspective ***@***.***> Cc: Ambalu, Robert ***@***.***>; Author ***@***.***> Subject: Re: [finos/perspective] datetime value units (#1500) A handy GIF of the fix in action: [dtconvert]<https://urldefense.com/v3/__https:/user-images.githubusercontent.com/13220267/128216675-37e43fa4-ce32-4545-9e8f-d05ce77fce3d.gif__;!!K_TC0FI_KA!9xeeYYc_ltcz5HexCGFAFGN6nEFvcY2qNe27o9pO18fMfVfQss2vu501yUrSym_7gggTQQ$> If you have many columns that need the conversion, you can auto-generate the expressions and apply them to the viewer/PerspectiveWidget using the expressions attribute, which takes a list of string expressions. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/finos/perspective/issues/1500*issuecomment-892790793__;Iw!!K_TC0FI_KA!9xeeYYc_ltcz5HexCGFAFGN6nEFvcY2qNe27o9pO18fMfVfQss2vu501yUrSym-6u2wUbA$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AG22D7D6JWLVFBX65WF26G3T3FRT3ANCNFSM5BIWQYKA__;!!K_TC0FI_KA!9xeeYYc_ltcz5HexCGFAFGN6nEFvcY2qNe27o9pO18fMfVfQss2vu501yUrSym8ApD2rbw$>. Triage notifications on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https:/apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!K_TC0FI_KA!9xeeYYc_ltcz5HexCGFAFGN6nEFvcY2qNe27o9pO18fMfVfQss2vu501yUrSym-e9zERdQ$> or Android<https://urldefense.com/v3/__https:/play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email__;!!K_TC0FI_KA!9xeeYYc_ltcz5HexCGFAFGN6nEFvcY2qNe27o9pO18fMfVfQss2vu501yUrSym9UPvX7nA$>. DISCLAIMER: This e-mail message and any attachments are intended solely for the use of the individual or entity to which it is addressed and may contain information that is confidential or legally privileged. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and permanently delete this message and any attachments.

[texodus]

@robambalu Not avoid per se (the conversions are useful in themselves), but the we're working on a plan to replace parts of Perspective's internal data structures with Arrow directly, like the columnar store itself, in which case Perspective would support all (or all useful) Arrow data types.

[nickroci]

FYI you can get around this by creating arrow table bytes and passing them into perspective (can choose precision in arrow conversion to dt), I find it much faster and more reliable, especially compare to df

[texodus]

Perspective should now support ingest from Arrow with any timestamp unit type (#3210). Internally, there is still only a millisecond-epoch datetime - so you will get "correct" values for ns+ typed timestamps, floored to the nearest millisecond.

I'm down to elaborate on this feature with richer unit types, but we'd still need to downcast to convert them to JavaScript types when we want to use Intl APIs to format them for presentation (or locale formatting, e.g.); ergo, I'll leave this for another Issue if there's any future interest in this.