Unexpected KeyError while running test suite inside GitLab Worker #1237

konstista · 2024-03-12T23:32:44Z

konstista
Mar 12, 2024

Hello there, I'm looking for help.
Basically all of a sudden started getting unexpected KeyError while running test suite inside GitLab Worker.

It seems like no dependencies were updated, no extra code was recently added. Pipelines seem to fail even for the code that previously passed all tests. Interesting that everything works fine locally.

Error Trace

ImportError while loading conftest '.../tests/conftest.py'.
/usr/local/lib/python3.8/site-packages/ddtrace/internal/module.py:
[208](https://gitlab.com/.../jobs/...#L208): in _exec_module
    self.loader.exec_module(module)
tests/conftest.py:5: in <module>
    from service.main import app
...
...
...
service/utils.py:10: in <module>
    from jsonschema import validate
/usr/local/lib/python3.8/site-packages/ddtrace/internal/module.py:208: in _exec_module
    self.loader.exec_module(module)
/usr/local/lib/python3.8/site-packages/jsonschema/__init__.py:16: in <module>
    from jsonschema.validators import (
/usr/local/lib/python3.8/site-packages/ddtrace/internal/module.py:208: in _exec_module
    self.loader.exec_module(module)
/usr/local/lib/python3.8/site-packages/jsonschema/validators.py:589: in <module>
    meta_schema=SPECIFICATIONS.contents(
/usr/local/lib/python3.8/site-packages/referencing/_core.py:479: in contents
    return self._resources[uri.rstrip("#")].contents
E   KeyError: 'http://json-schema.org/draft-03/schema'

Error happens when importing validate() from jsonschema.
In all test cases the function is not being called, it fails on import.

Implementation

Implementation is fairly straightforward: try/except block wrapped in a function.

 try:
     validate(entity, schema)
     return True
 except Exception as e:
     return False

Dependencies

jsonschema==4.21.1
├── attrs [required: >=22.2.0, installed: 23.2.0]
├── jsonschema-specifications [required: >=2023.03.6, installed: 2023.12.1]
│   └── referencing [required: >=0.31.0, installed: 0.33.0]
│       ├── attrs [required: >=22.2.0, installed: 23.2.0]
│       └── rpds-py [required: >=0.7.0, installed: 0.18.0]
├── referencing [required: >=0.28.4, installed: 0.33.0]
│   ├── attrs [required: >=22.2.0, installed: 23.2.0]
│   └── rpds-py [required: >=0.7.0, installed: 0.18.0]
└── rpds-py [required: >=0.7.1, installed: 0.18.0]

konstista · 2024-03-12T23:37:13Z

konstista
Mar 12, 2024
Author

The issue seems to be somewhere on my side, but I don't think something changed in the docker executor. The code itself is perfectly functional. Would be very appreciated for any help.

I can overcome the issue with moving import to function local scope. As the function is never called in the test suite - the import never happens. But I'm just curious about the behaviour.

0 replies

silvanocerza · 2024-03-13T11:57:38Z

silvanocerza
Mar 13, 2024

Same issue when running with Github runners. Locally it works without any issue.

0 replies

Julian · 2024-03-13T12:49:29Z

Julian
Mar 13, 2024
Maintainer

There isn't enough info to be helpful I'm afraid. All passes here in CI in GitHub Actions without doing anything special. The error message points to you possibly doing something wrong to break normal installation of static (non-Python) files but hard to say what.

(Going to convert this to a discussion. If there's more info there's a small chance I can help you figure it out but no promises.)

0 replies

konstista · 2024-03-13T13:27:47Z

konstista
Mar 13, 2024
Author

Yeah, no worries and thanks for the fast response. We'll check some things on our end.

0 replies

konstista · 2024-03-13T14:48:06Z

konstista
Mar 13, 2024
Author

Okay, workaround with moving dependency to a local scope does not work. When code is deployed to AWS Lambda as docker image it fails on import with the same issue. Docker image used for deployment was updated 8 days ago, so its not clear why it started failing around 20 hours ago.

0 replies

konstista · 2024-03-13T17:17:05Z

konstista
Mar 13, 2024
Author

Ok, managed to solve the issue. The problem was in two latest images: AWS Lambda Python Image (released 9 days ago from this day) we used as deployment container and Debian Bullseye (released 1 day ago from this day) we used for running pytest tests. For both cases we could use old (outdated) versions of images and things would work.

But the better solution was to migrate from Python 3.8 to Python 3.9. Latest Python 3.9 images are fully compatible and seem to work without issues.

So @Julian it might make sense to remove Python 3.8 from a list of supported versions, since the package might not work without issues. Especially in CI environment, as normally images are not locked there.

4 replies

Julian Mar 13, 2024
Maintainer

It's Lambda that isn't really supported and generally has bad packaging behavior. 3.8 is definitely supported and definitely works fine in normal environments!

konstista Mar 13, 2024
Author

AWS Lambda container (default runtime) is launched on top of Amazon Linux 2 what it a somewhat similar of CenOS and Fedora. Its very highly likely that things would brake on latest RHEL-based distros sooner or later.

Many people pack app/services in lambda containers, some of them even don't care about runtime that are using. This approach is becoming more and more popular due to simplicity and scalability. For sure not many devs would use Python 3.8 as its end-of-life already this year, but hopefully they will find this discussion helpful.

Julian Mar 13, 2024
Maintainer

I'm not telling you how to run your infrastructure or app, I'm just telling you I would estimate 90% of the packaging issues filed on this package (or in my experience often in general across the ecosystem) are caused by users using Lambda! It's Lambda that I don't support (which just means "I won't help you figure out how to get Lambda to do the right thing").

3.8 is supported until it's EOL'ed essentially, and again as I say as far as I know there are no 3.8-specific bugs or issues related to this library. If there were, I'd fix it, at least until 3.8 support is dropped.

konstista Mar 13, 2024
Author

Totally understandable and I get your pain. However just a friendly reminder that initial discussion was not about Lambdas but about some issues inside GitLab runners.

Anyway, thanks for your help and contribution!

patrick91 · 2024-04-19T19:51:46Z

patrick91
Apr 19, 2024

Getting this issue on GitHub actions as well: https://github.com/strawberry-graphql/strawberry/actions/runs/8758073573/job/24040028878?pr=3116

I might skip adding this dep on 3.8 (since it's optional), but if there's another workaround I'll be happy to use that :D

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python + JSON Schema

Unexpected KeyError while running test suite inside GitLab Worker #1237

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Python + JSON Schema

Unexpected KeyError while running test suite inside GitLab Worker #1237

konstista Mar 12, 2024

Error Trace

Implementation

Dependencies

Replies: 7 comments · 4 replies

konstista Mar 12, 2024 Author

silvanocerza Mar 13, 2024

Julian Mar 13, 2024 Maintainer

konstista Mar 13, 2024 Author

konstista Mar 13, 2024 Author

konstista Mar 13, 2024 Author

Julian Mar 13, 2024 Maintainer

konstista Mar 13, 2024 Author

Julian Mar 13, 2024 Maintainer

konstista Mar 13, 2024 Author

patrick91 Apr 19, 2024

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

konstista
Mar 12, 2024

Replies: 7 comments 4 replies

konstista
Mar 12, 2024
Author

silvanocerza
Mar 13, 2024

Julian
Mar 13, 2024
Maintainer

konstista
Mar 13, 2024
Author

konstista
Mar 13, 2024
Author

konstista
Mar 13, 2024
Author

Julian Mar 13, 2024
Maintainer

konstista Mar 13, 2024
Author

Julian Mar 13, 2024
Maintainer

konstista Mar 13, 2024
Author

patrick91
Apr 19, 2024