i think we avoided using this name in the label import project because it didnt exist anywhere else in the application. Can you double check that we want to bake this into a public-facing method? I think maybe we can just call them label?

Great point, we need to come up with some consistent language. The problem here is that Label means "data and annotations" not "human made annotations". There needs to be some distinction between human generated and machine generated labels. I am open to changing the name but I already exposed this concept with the iou metric functions for our first mea release :). Although these are positional args so if we want to change this we probably can. We should probably do it soon though.

i honestly have no stake in it, i guess i just wanted to point it out. I think Gareth asked if we could change the name in label imports, so he might be a person to run it by.
I definitely dont have a problem with it, and honestly, i think ground truth label is a good name, as thats the name in the industry, so i dont know why labelbox wouldnt go that direction.

i dont follow why were returning the confusion matrices for just the first prediction/ground truth values? dont we need to find an average for all of the values?

Because at this point we have grouped annotations by class. So there can only be one or zero of a classification in an image. I updated the docstring to explain this.

what is a matching annotation? wouldnt a matching prediction and label have the same schemaId? Would it make sense to check for that in this function, or is it not that simple?
Would it make sense to make sure the lengths are the same?

All annotations passed to this function have the same schema ids. So this function checks if we even need to run the metric calculation code - if there is no ground truth or no prediction then there is nothing to compute.

isnt this whole block just re-generating a list, expected, which is the same list as example.expected.values()?
Do we need this block at all? can we just do

            assert score[0].value == tuple(
                example.expected.values()), f"{example.predictions},{example.ground_truths}"

?

We are actually accumulating all expected values.

expected = [0, 0, 0, 0]
for expected_values in example.expected.values():
    for idx in range(4):
        expected[idx] += expected_values[idx]
assert ... # Each of the values are added to expected before this assert statement is reached.

if the assert was within the outer for loop, then what you are suggesting would be true.

how does this test test_custom_confusion_matrix_metric ? it seems like it just deserialize and re-serializes a json file, then asserts that its equal to the file data?

This tests the serialization and deserialization of the custom metric. Not the actual calculation.

got it. I might recommend naming it something about the serialization, then but that just a nit!

That seems like a lot of time. Is there any way to avoid this? I assume that things were not being created in time? is there a way to "await" (for lack of a pythonic verison of the word) these function calls?

everything is technically awaited in python (unless using asyncio or some other async abstraction). The problem is the async stuff on the backend that can't be awaited like items that are added to the queue to be processed.

Also once label imports are complete, I can throw out this garbage hack to upload labels for tests.

ah, yeah. dang. 10s is a while to add to a test, but if the processing takes that long, makes sense!

what is common?

It is the standard format for annotations in the repo. This terminology is used elsewhere. I am not a huge fan of this naming convention but I didn't have a great alternative. The type hints should make this a little less ambiguous.

I dont feel like i was able to review these functions at all, fyi