Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Codecov Report

Patch coverage has no change and project coverage change: -0.13 ⚠️

Comparison is base (a4bb1e1) 99.25% compared to head (5407968) 99.12%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main     #800      +/-   ##
==========================================
- Coverage   99.25%   99.12%   -0.13%     
==========================================
  Files          82       83       +1     
  Lines       13101    13898     +797     
==========================================
+ Hits        13003    13776     +773     
- Misses         98      122      +24     

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@ken-maeda Thank you for this contribution. Please allow me some time to review

View / edit / reply to this conversation on ReviewNB

seanlaw commented on 2023-03-07T11:21:35Z
----------------------------------------------------------------

T(+) requires at least two behaviors.

Can you explain why it requires at least two behaviors?

Maybe "behaviors" isn't the right word and you mean "at least two instances of the positive case"?

View / edit / reply to this conversation on ReviewNB

seanlaw commented on 2023-03-07T11:21:36Z
----------------------------------------------------------------

Line #1.    ecg_df = pd.read_csv("14172m.csv", index_col=0)

Instead of repeating astype(float) so many times later, you should just do:

ecg_df = pd.read_csv("14172m.csv", index_col=0, usecols=[1]).astype(float) 

Also, it would be nice just to show or print out what is in ecg_df.head() . What does the dataframe look like?

View / edit / reply to this conversation on ReviewNB

seanlaw commented on 2023-03-07T11:21:37Z
----------------------------------------------------------------

Can we add some comments on what we are looking at? Why does the bottom look so much more regular and with a repeated pattern?

View / edit / reply to this conversation on ReviewNB

seanlaw commented on 2023-03-07T11:21:39Z
----------------------------------------------------------------

Line #1.    v_query = ecg_df.iloc[5930:5930+127, 1].values.astype(float)

Why is the window size 127 and not 128?

View / edit / reply to this conversation on ReviewNB

seanlaw commented on 2023-03-07T11:21:40Z
----------------------------------------------------------------

I don't understand the significance of this. Your point isn't clear. Where did v_query come from? Why should the reader care about that?

It would be useful to discuss the bottom plot (distance profile) and how to interpret it.

Why is it useful/important to show distance profile?

ken-maeda commented on 2023-03-07T12:08:54Z
----------------------------------------------------------------

v_query is typical norml ecg query we can see everywhere in dataset. so it is picked up one randomly.

The purpose of this distance profile is finding desired behavior by just comparing v_query(typical ecg signal) with desired behavior(rare signal). As assumption, it could be highest in distance profile. But it didn't happen.

View / edit / reply to this conversation on ReviewNB

seanlaw commented on 2023-03-07T11:21:41Z
----------------------------------------------------------------

What is "plato"?

ken-maeda commented on 2023-03-07T12:10:04Z
----------------------------------------------------------------

I add more descriptino to

Contrast Profile

The subsequence in 𝐓(+) corresponding to the highest point in the Contrast Profile is called the Plato.

v_query is typical norml ecg query we can see everywhere in dataset. so it is picked up one randomly.

The purpose of this distance profile is finding desired behavior by just comparing v_query(typical ecg signal) with desired behavior(rare signal). As assumption, it could be highest in distance profile. But it didn't happen.

View entire conversation on ReviewNB

I add more descriptino to

Contrast Profile

The subsequence in 𝐓(+) corresponding to the highest point in the Contrast Profile is called the Plato.

View entire conversation on ReviewNB

I appriciate your feedback, I fixed those.

@ken-maeda
I have a suggestion for you.

I think it is better to develop the notebook section by section. So, for each section that you add, you can wait to get some feedback and then apply those, and then after getting the green light, you can move forward. Right now, you may know what is going on in your notebook, however, the main goal is to make sure the reader can understand what is going on! You can keep the current notebook somewhere in your local pc. Then, you can start again by just providing the first section or a couple of sections. So, your notebook should only contain a couple of parts in the beginning. Then, you can add sections to it step by step.

Currently, if I do not understand a part of your notebook, I try to read other parts to better understand the concept. However, this is not desirable. I think this is a red flag. There should be a flow in your tutorial and I believe each segment should be understandable on its own.

Also, the text is as important as the code. In fact, I think it is more important particularly in tutorials. So, try to be extra careful when you explain a concept. You want to be crystal clear in every single step as much as possible.

Regarding contrast profile, this is how I see it:

we can see each subsequence of length m as a data point in $R^{m}$ space. For the sake of visualization, let's illustrate the problem in 2D space.

First, let's review the definitions of T(+) and T(-).

T(+) : contains at least two instances that are unique to the phenomena of interest.
T(-) : contains no instances of interest (and instead, I think we should say it contains the regular, obvious patterns in `T(+)`)

Before we talk about T(-) and T(+), it is better to just talk about T. Let's assume the figure below shows the subsequences of T.

If I look at this data, I can see that the regular, obvious pattern is where the crowded part is. However, the motif we might be interested in can be the motif pair (A, B). Note that this motif pair may not be easily captured as their distance is greater than the distance of any other point and its nearest neighbour.

So, what can we do? We can create T(-) which just contains the regular behaviour of our Data. We then use T(+) to denote the remaining ones.

Now, we can see that the d = dist(A, A_nn_in_Tneg) - dist(A, A_nn_in_Tpos) has a high value. Let's call it contrast distance. The contrast profile, cp, is an array where cp[i] is the contrast distance that corresponds to the i-th subsequence in T(+). The peak of this contrast profile can reveal the motif pair (A, B).

Question: Can we see it as twin-freak problem? In other words, this might be an anomaly that appears more than once. So, we can easily find the motif pair (A, B) by finding the subsequence that has the greatest distance to its 2nd nearest neighbour.

Answer: I do not know! I think a good way to investigate this is to get some data and find a pair of subsets using each of these two approaches: (1) twin-freak (2) contrast profile, and see if they result in different outcomes.

@NimaSarajpoor
I'm sorry for causing trouble, and greatful your kind guidance. I uploaded new notebook first section of Tutorial_Contrast_Profile2.ipynb. I should have considered the contrast profile concept more.

I added its scatter plot should best to understand the "constrat" concept to the notebook.

T(+) : contains at least two instances that are unique to the phenomena of interest.
T(-) : contains no instances of interest (and instead, I think we should say it contains the regular, obvious patterns in T(+))

This might be tricky precondition, the robustness for this precondition also is argued. I thought the notebook should be enoguh only for expalining contrast profile conceptl.

@ken-maeda

I'm sorry for causing trouble, and greatful your kind guidance.

No need to be sorry. I provided a few comments. Let's start with those. Please do not add any new section. Let's take care of the current sections first. Please feel free to discuss something if you feel there is a need for that.

@ken-maeda Instead of uploading a .png file, I would prefer if you could add the code that could create/recreate the image and have it inside the notebook

@ken-maeda
I reviewed up to the section Loading the ECG data for Contrast Profile. Whenever you address a comment, you can go to ReviewNB (see the top of this PR) and then click on "Resolve Conversation" whenever you are done with that comment.

I think you have done a great job so far, and the notebook becomes more and more clear.

@NimaSarajpoor I appriciate your kind feedback, I updated the notebook markdown.

@ken-maeda
From my point of view, things are good so far. My only comment is the last figure as I think it is a little bit crowded (and I am not sure if there is a better way to break it down or not). Also, you may want to revise the last sentence. I think the last sentence talks about the desirable patterns but what you should have discussed is that the discovered discords via matrix profile (shown in "orange") is not desirable and someting like that.

@NimaSarajpoor I changed to separate last plot(maybe redundant?), as you mentioned, it was crowded it is hard to recognized where is indicated. I fixed the last sentence also.

@ken-maeda
I provided my final touch on the notebook. After addressing the comments, we should see what @seanlaw thinks about this notebook.

If everything is okay, we can then move forward and add the part where you compute the contrast profile and show that it can discover the subsequences of our interest.

@ken-maeda
How do you feel about the progress so far?

@NimaSarajpoor
I'm sorry for the delay, I have fixed the notebook in the point you mentioned. I hope the notebook I created is fine now.

@ken-maeda
Thank you for addressing the comments. While there is still some room for improvement, we can do it later. I think @seanlaw can take a look at the notebook now and see if he has any opinion / suggestion.

@seanlaw
Do you have any comment on the second notebook, docs/Tutorial_Contrast_Profile2.ipynb ?

Let me find some time to provide some comments

Apply stump to the entire time series
what is the simplest thing that the user would have tried?

Regarding the point whether it is natural to calculate the Matrix Profile for the entire series first, the characteristic we are trying to find this time is neither motif nor discord, so I feel there is no motivation to calculate stump directly. Therefore, even if you simply calculate it for the entire series, I don't think there is much to say from the event itself, so I compared when there is a single discord and when there are multiple discords, and built on what can be said from there.
It might be better to make it easy to understand what I'm trying to do from the very beginning.

Regarding the point whether it is natural to calculate the Matrix Profile for the entire series first, the characteristic we are trying to find this time is neither motif nor discord, so I feel there is no motivation to calculate stump directly. Therefore, even if you simply calculate it for the entire series, I don't think there is much to say from the event itself, so I compared when there is a single discord and when there are multiple discords, and built on what can be said from there.
It might be better to make it easy to understand what I'm trying to do from the very beginning.

I think the point is that stump will not be able to help you here precisely because the subsequence is neither a top motif or a top discord. Certainly, if you traverse down to the top-N motifs then you might eventually find it. I think it is important to motivate "why" computing the full matrix profile is not enough and also demonstrate its ineffectiveness for this particular problem (i.e., when the subsequence of interest does not have a nearest neighbor that is as close as other motifs)

I think it is important to motivate "why" computing the full matrix profile is not enough and also demonstrate its ineffectiveness for this particular problem

1. It is challenging to set a goal prior to calculating the Matrix Profile across the whole data.
   When we have a signal pattern we want to find, searching for motifs or discords with the Matrix Profile may not seem natural. Users might question what to do in such cases and may find it unnatural to take action using the naive Matrix Profile.

2. Analyzing the result of applying the Matrix Profile to the whole data is difficult.
   As you mentioned, whether we can find what we're looking for largely depends on parts of the signal other than the current characteristic. Therefore, it's hard to say from the results what would be better from the perspective of the current characteristic .

3. Elements that should be explained in the introduction and elements that can be explained.
   I want to determine that.
   Currently, the overall flow is:
   3-1. If a discord is included once, it can be found.
   3-2. If a discord is included twice, it cannot be found.
   3-3. So, what should we do?
   Regarding this flow, I thought that it would be better to write more concisely at the beginning about what the Contrast Profile brings. However, what do you think should be written in the introduction?

3-1. If a discord is included once, it can be found.
3-2. If a discord is included twice, it cannot be found.
3-3. So, what should we do?
Regarding this flow, I thought that it would be better to write more concisely at the beginning about what the Contrast Profile brings. However, what do you think should be written in the introduction?

If you look at the comments, I don't think "discord" or "anomaly" is the right word here as a discord is referring to "a subsequence that has a one-nearest neighbor that is very far away". In our example, the one nearest neighbor may not necessarily be very far away and, instead, it is that the subsequence of interest isn't discovered in the first few motifs. The paper refers to them as "phenomena of interest" and not "discord"

When I look at the contrast profile paper, it presents the problem as:

1. Imagine that you have a time series that usually has a reasonably well defined set of one or more repeating patterns and other subsequences are not expected
2. Then, something occurs that induces a new subsequence (a phenomena) that has never been observed before AND it isn't quite a discord because the phenomena has repeated itself at least once more and while this nearest neighbor might be "close" to the first manifestation, it isn't as "close" to the nearest neighbor of previously known motifs (i.e., the repeating patterns in 1.)

So, you have some historical data where everything is well known and then you encounter an event that causes a new, never seen before subsequence AND it happens a second time. And so it becomes obvious to ask the question, "given how rare the phenomena is, how might we go about detecting it in a systematic way?"

The first (naive) thing that you might try is to compute the matrix profile for the entire time series and then iterate through the top-N motifs via stumpy.motif and then consider how similar the ith motif is to the first i-1 motifs. However, there is a "better" way... Then introduce the contrast profile (including its assumptions, what it aims to do, and what it isn't/can't do)

What I'm trying to say is that it doesn't make sense to me why we're calculating the matrix profile for the entire series when there's not enough reason to expect that the phenomena we're trying to discover will be detected as motifs. It feels like we're just saying, 'We tried calculating it and this is what happened.'

So, you have some historical data where everything is well known and then you encounter an event that causes a new, never seen before subsequence AND it happens a second time. And so it becomes obvious to ask the question, "given how rare the phenomena is, how might we go about detecting it in a systematic way?"

In this way, the phenomenon we wish to find this time is mentioned, but I believe no mention is made of the relationship with other intervals or motifs that occur in other intervals. Under these circumstances, would a user try to find it as a motif?

Particularly in this case, where other parts clearly have repetitive subsequence, it feels unnatural to calculate motifs to find the phenomena.

.

Particularly in this case, where other parts clearly have repetitive subsequence, it feels unnatural to calculate motifs to find the phenomena.

@ken-maeda If this is the case, you really need to explain this fact about the data at the beginning when you present the data. I don't think it is enough to assume that the reader will notice the repetition. Instead, you must point out the facts even if it is obvious. Then, you should explicitly explain that computing the full matrix profile won't really help you find the phenomena and why that is. Better yet, I was advocating that you just compute the matrix profile and show its limitations in that the phenomena won't be captured easily by the matrix profile (or using the motifs function).

As you suggested, I tried it. Does this seem to be okay?

top-motif location by motifs

We were able to find something close to the motif we want to find as the top 10th motif.

As you suggested, I tried it. Does this seem to be okay? We were able to find something close to the motif we want to find as the top 10th motif.

Yes! What code did you use to get that information? Was it stumpy.stump followed by stumpy.motifs?

So, I think it's important to explicitly point out/motivate that, in this example, we only know to search to the 10th motif because we are lucky enough to see/visualize where the phenomena is but we have no idea if it will be the 10th motif or the 100th motif. In the real world, the phenomena might be much harder to spot with the naked eye because the repetition within data might be much longer/noisier. And so we need a different way to solve this problem.

Maybe a poor-man's version would be to compare ith motif with the all of the motifs that were discovered before it and ask how different/similar it is? But this is painful and more art than science.

mp = stumpy.stump(T, m, normalize=False)
motifs = stumpy.motifs(T, mp[:, 0])

I simply calculated the above example from the motifs.
Considering the following flow for the introduction, how far do you think it's appropriate to explain?

1."By naively using stump, you can discover motifs and discords.
2.In this case, the signal we want to find are similar, so it should be possible to findthem as some top-motif when we compute stump.
3.We are calculating stump, but the signal we want to find have clear characteristics in their size(amplitude), so we calculate with normalization set to False, which is True as the default in stump.
4.It could be calculated as the 10th top-motif.
5.However, in reality, we don't know as which top-motif it will be calculated. There is a more reliable way to find signal with these characteristics. This can be achieved with a concept called Contrast Profile, which I would like to explain in detail."

@ken-maeda I think this level of detail is more than fine. However, I think it is important to point out that there is repetition in the data and so finding the "phenomena" (only repeated once) is challenging since the majority of the top motifs will look like the repeated pattern.

@seanlaw OK, I'm going to create another notebook from the introduction, following your new advice.