Got basic timeseries with linestack working. I've also got some code snippets for interpolating to display heatmap with non-uniformly sampled timeseries data. I should be able to have this fully working with time series very soon :D

Got heatmap to display timeseries working. It should also work with non-uniformly sampled data by interpolating, need to test.

Also need to implementing switching between heatmap and line representations, need to delete the graphic when switching.

So timeseries can be represented with arrays of one of the following shapes (let's ignore x-axis values for now).

If we have:

l: number of timeseries
p: number of datapoints in a timeseries

We can have the following shapes:

p: only y-values
p, 2: yz vals
l, p: l-timeseries with y values
l, p 2: l-timeseries with yz values

Extended to n-dimensional arrays (for example, trajectories projected onto principal components?). If each non-timeseries dim is $d_1, d_2, ... d_n$, then the above becomes:

I don't think we can auto-detect if l is present or not and the user should specify, something like:

multi_timeseries: bool = True

Scatters can be similar to some cases of nd-lines 🤔 , but x values would be directly specified and the current index is parametric (example with time indicating position in a low dim space). This would actually be true for lines as well if representing trajectories.

So for nd-line maybe we have two versions, parametric (y and z are not functions of x, but x, y, z are a function of some other dim) and non-parametric (simple timeseries lines where y and z are functions of x).

I made a more generalist NDPositions which can map data that is:

[s1, s2, ... sn, l, p, 2 | 3]

where:
s1, s2, .... sn are slider dims
l is number of lines or number of scatters, this dimension is optional and the user must specify whether or not it exists
2 | 3 is the last dim, indicating xy or xyz positions.

It can map arrays of these dims to a line, line collection, line stack, scatter, or list of scatters (similar to multi-line).

I think this is a much more elegant way to deal with things, and NDTimeSeries is not necessary. The user can provide a slider mapping (to map from reference units to array index) for the p dimension which is the same as the "x-axis" for time series data!

Example if we have data that is [n_timepoints, 2], and the x-positions here (in the last dim) are in seconds. The NDWidget reference units for the slider can also be in seconds, and we can provide a mapping function that goes from the slider reference units to the n_timepoints index.

I think we can also use this for heatmaps and interpolation. Use the reference units to determine a uniform x-range for the current display window, and we can interpolate using [n_timepoints, 2] data.

EDIT: I think that the NDPositions will also work for PolygonGraphic ! Can think about meshes in general later.

For positions graphics, I should actually do [n_datapoints, n_lines, 2 | 3] so everything before the last 1 or 2 dims is always sliders.

Some more ideas:

Allow any 2-3 dims to be used as the graphic dimensions and specify the slider dims. This would also allow using named dims (such as those used in xarray).
API like:

add_nd_<scatter|lines|heatmap>(
  data=<array>, # array like, defaults to last 2-3 dims are graphical, first (n - 2 | 3) dims are sliders and in order
  x=<int | str | None>  # optionally specify x graphical dim using the integer dim index or dim name
  y = ... # same for other graphical dims
  slider_dims=<None | tuple[int | str]> # specify slider dims, None is auto first (n - 2 | 3), or specify a tuple of slider dims as int or str, examples: (0, 1, 2, 3), ("time", "plane")
  ... # other args
)

We interpret the given order of the slider_dims passed as $$s_1, s_2, ... s_n$$, regardless of their order within the actual array. This will make it clear to users which dims they are syncing when they use the sliders.

EDIT: A limitation of the above is that a user can't collapse multiple "graphic/display dimensions" into "final graphic/display dimensions" if they're hard-coded this like. So something like:

add_nd_<...>(
  data=<array>,
  display_dims=<tuple[int | str]>, # specify display dims in [x, y, z] order, OR display dims that collapse to xyz after the finalizer function
...
)

An example for images would be collapsing [z, m, n] to display a projection over z as slider dims are moved. But we could also have > 3 dims that are used, and then collapsed to 3 or fewer dims for the "final graphic/display dims".

We can use LineCollections to display multiple lines, like behavior tracks of keypoints, with shape [n_keypoints, n_timepoints, 2]. This works well with NDPositions and display windows.

I was thinking of what's the best way to show a scatter for each keypoint, and I think I should make a ScatterCollection that behaves like a LineCollection so the same array with the same shape can be given, the only difference is the graphical representation would be a scatter instead of lines. For typical behavior keypoint viz, the display_window would usually be just 1, but it can be greater than 1 for any viz that needs to show a window of scatter points.

ok I think stuff is working

I think I need to make a PolygonCollection too 🤔 . Would be very similar to the ScatterCollection.

so the basics are all working 🥳

LineStack and heatmap representations are swappable:

Scatter and line collection to show behavior trajectories:

A set of imgui UIs that allow controlling some aspects of the "nd graphics" could be useful, such as:

• graphical representation, dropdown menu to choose scatter, line, heatmap
• display_window
• multi bool

Stuff I should finish before implementing the orchestrator:

• merge auto-replace buffers #974 , so I can implement changing the display_window
• implement logic to allow the display_window to centered, on the left or on the right
• adding a linear selector, it's "default" positions and its position when the main slider for the n_datapoints dimension changes should correspond whether the display_window is in "center", "start" or "end" mode. Moving the linear selector should probably change the index in all other NDobjects but not its "parent" object. So that the visual representation of the parent object doesn't change, ex: if the linear selector is moved we don't want a linestack to also "move"
• spatial function
• implement mapping from a slider reference index with units (such as time) to array index.
• figure out how to implement stuff like colors, cmaps given that we're essentially performing out of core rendering with these graphics
• auto display_window? Zoom in/out and set graphic data at different subsample levels

Things that make me "uncomfortable" that I need to settle:

dim shapes

Dim shape for nd-positions is [s1, s2, ... sn, n, p, 2 | 3] where:

n: number of lines, scatters, or heatmap rows (optional, can be 1)
p: number of datapoints, often corresponds to a time index.

p would be a slider dim, but there's this n dim that's in between p and all the other slider dims. We could instead use a shape of [s1, ... sn, p, n, 2 | 3] but that feels weird and we'd have to do a transpose to get the array for the graphical representation, i.e. [p, n, 2 | 3] -> [n, p, 2 | 3] is required for the graphical representation.

Do we just document this well, that the n dimension is not a slider dim but p is?

When using nd-positions data in conjunction with nd-image data, we'd have something like this:

# nd-positions
[s1, ... sn, n, p, 2 | 3]

# nd-image
[s1, ... sn, p, r, c, 1 | 3 | 4]

Where r: rows, cols: columns. 1 | 3 | 4 denotes grayscale or RGB(A).

The p dimension in the nd-image array above would correspond to the p in the nd-positions. nd-images don't have any n dimension, it doesn't make sense there.

Working on "implement mapping from a slider reference index with units (such as time) to array index.", which requires proper implementation of slider_dims and n_slider_dims properties. Need to figure out how to properly separate the p n_datapoints dimension from other slider dims, and also apply the window funcs on the p dim. Will do tomorrow.

Window funcs working for p (n_datapoints) dim and all graphical representations

ok got the behavior viz working again, now with named dims.

@apasarkar, @FlynnOConnell you will have to adjust your lazy array base classes slightly to work with xarray:

You need to add these methods, they don't actually have to be implemented

    def __array__(self, dtype=None, copy=None):
        if copy:
            return copy(self)
            
        return self

    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
        raise NotImplementedError

    def __array_function__(self, func, types, *args, **kwargs):
        raise NotImplementedError

You will need to account for an Ellipsis object as the last object in the indexing tuple. You may need to adjust any validation logic accordingly, but I think that if you just ignore/get rid of the last object in the tuple (the Ellipsis obj) you're fine.

    def __getitem__(self, indices: tuple[slice, ...]) -> np.ndarray:
        # indices can be a tuple of slice | Ellipsis
        # need to accoutn for Ellipsis as the last object in the tuple
        pass

example implementation with the decord lazy video reader wrapper:

class LazyVideo(LazyArray):
    def __init__(
        self,
        path: Union[Path, str],
        min_max: Tuple[int, int] = None,
        **kwargs,
    ):
        self._video_reader = VideoReader(str(path), **kwargs)

        try:
            frame0 = self._video_reader[10].asnumpy()
            self._video_reader.seek(0)
        except IndexError:
            frame0 = self._video_reader[0].asnumpy()
            self._video_reader.seek(0)

        self._shape = (self._video_reader._num_frame, *frame0.shape)

        self._dtype = frame0.dtype

        if min_max is not None:
            self._min, self._max = min_max
        else:
            self._min = frame0.min()
            self._max = frame0.max()

    @property
    def dtype(self) -> str:
        return self._dtype

    @property
    def shape(self) -> Tuple[int, int, int]:
        """[n_frames, m, n, 3 | 4]"""
        return self._shape

    @property
    def min(self) -> float:
        warn("min not implemented for LazyTiff, returning min of 0th index")
        return self._min

    @property
    def max(self) -> float:
        warn("max not implemented for LazyTiff, returning min of 0th index")
        return self._max

    @lru_cache(maxsize=32)
    def __getitem__(self, indices: tuple[slice, ...]) -> np.ndarray:
        # indices can be a tuple of slice | Ellipsis

        # apply to frame index, then remaining dims
        a = self._video_reader[indices[0]].asnumpy()[indices[1:]]
        self._video_reader.seek(0)
        return a

ideas:

If a ref range is not specified for a slider dim, it auto makes one just using the identity transform using the max of that array for that slider dim.
default_image_dims and default_positional_dims kwargs to NDWidget could be used, better than a config and hard-coded guessing

ok update imgui sliders UI, need to do right click menus for NDGraphic properties, as well as some image properties like brightness, contrast, gamma for any ImageGraphic in general.

nd graphic options are now accessible through a right click menu popup:

Which then opens a window to change stuff until it's closed explicitly:

I think these kinds of floating windows can be pretty useful for many things 🤔 @FlynnOConnell

Checkboxes in the right click window to select the spatial dims.

Another thing I should look into, when the display_window and indices auto-adjusts for timeseries by polling the Subplot.x_range, it also should adjust for other subplots that share the same controller. Useful for @GilyGinosar spectrograms