Convert Z-normalized Matrix Profile Distances to Correlation Values

[ssahaxd]

Is it possible to use some other measures of distance instead of the euclidian distance for calculating the distance profile?
Before I knew stumpy, I used the correlation coefficient as the metric to find similarities between the pattern and the time series.

[seanlaw]

@ssahaxd Welcome to the STUMPY community and thank you for your question. Generally speaking, since there is a computational speed-up trick (traversing diagonals) being utilized to calculate z-normalized Euclidean distances super quickly, it is not possible to measure other custom distances (at least, not easily). However, correlation is a special case because there is a direct correspondence between z-normalized Euclidean distance and correlation (see Section 3):

Where D is the z-normalized Euclidean distance (computed by STUMPY), m is the subsequence window size, and P is the Pearson Correlation. So, rearranging, we get:

Thus, you can take the output of STUMPY and convert it to correlation:

import stumpy 
import numpy as np

T = np.random.rand(20)
m = 5
mp = stumpy.stump(T, m)  # Don NOT use `normalize=False` here if you are trying to compute correlation!!! 

pearson = 1.0 - (np.square(mp[:, 0]) / (2.0 * m))
print(pearson)

array([0.45635993707390354, 0.5227874872060077, 0.675953192999289,
       0.6769874021058575, 0.7972140720190464, 0.7774197297618404,
       0.9215246389104172, 0.8127424056523954, 0.7794595806608301,
       0.7793923781218768, 0.6928858682807422, 0.7972140720190464,
       0.7774197297618404, 0.9215246389104172, 0.8127424056523954,
       0.7794595806608301], dtype=object)

And this should be equivalent to computing the correlation coefficients directly with NumPy:

for i in range(len(T)-m+1):
    nni = int(mp[i, 1])  # This is the "nearest-neighbor" index
    print(np.corrcoef(T[i:i+m], T[nni:nni+m])[0, 1])

0.4563599370739034
0.5227874872060075
0.6759531929992889
0.6769874021058572
0.7972140720190461
0.7774197297618402
0.9215246389104169
0.8127424056523949
0.7794595806608295
0.779392378121876
0.6928858682807422
0.7972140720190461
0.7774197297618402
0.9215246389104169
0.8127424056523949
0.7794595806608295

To summarize, the z-normalized Euclidean distance is a special case of correlation (note that this relationship does NOT hold if you use normalize=False). I hope that you find this useful.