feat: Update kafka python instrumentation to support kafka-python>=3

[rjduffner]

Description

While kafka-python version 3.0.0 is actually a pretty large change, https://github.com/dpkp/kafka-python/releases/tag/3.0.0, the breaking changes don't seem to be exercised by the instrumentation.

I updated the instrumentation to include support for versions > 3 but also kept testing older versions as well.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Make sure the current tests pass with the new library versions.

Does This PR Require a Core Repo Change?

• No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[herin049]

Also here, I'd prefer we keep this pinned. You can also look at

opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-fastapi/test-requirements.in

Line 4 in ed72afb

urllib3~=2.7.0

to see how we can make this easier to maintain by using uv pip compile.

[xrmx]

The rage committing 🤣

[rjduffner]

@xrmx

haha, ya. A mentor when i was very junior always made me commit often. However, since we squashed commits, the messages never had to be good. Well the habit stuck. whoops.

[rjduffner]

@herin049, thanks for the comment. I actually like that earliest latest pattern instead of the 0,1,2 I stole from another instrumentation. I have some questions though and forgive me, I haven't contributed in a while and it was before the switch to uv.

I am trying to figure out how to use uv pip compile to do what you say. I see the flag --resolution

So is your workflow something like this to create the "low" requirements?

uv pip compile --resolution lowest test-requirements.in --output-file test-requirements.txt.oldest

Or am I missing something?

Apologies if i missed something simple.

[herin049]

@herin049, thanks for the comment. I actually like that earliest latest pattern instead of the 0,1,2 I stole from another instrumentation. I have some questions though and forgive me, I haven't contributed in a while and it was before the switch to uv.

I am trying to figure out how to use uv pip compile to do what you say. I see the flag --resolution

So is your workflow something like this to create the "low" requirements?

uv pip compile --resolution lowest test-requirements.in --output-file test-requirements.txt.oldest

Or am I missing something?

Apologies if i missed something simple.

We typically use lowest-direct for the earliest dependencies and highest for latest.

Quoting from the uv docs:

lowest-direct: Forces the lowest compatible version for your direct dependencies (the ones explicitly listed in your pyproject.toml or requirements.in). It continues to pull the highest compatible version for any transitive (indirect) dependencies.

You can see an example of the exact command used here:

opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-fastapi/test-requirements.oldest.txt

Line 2 in ed72afb

# uv pip compile --python 3.10 --universal --resolution lowest-direct --no-emit-package opentelemetry-api --no-emit-package opentelemetry-semantic-conventions -c dev-requirements.txt instrumentation/opentelemetry-instrumentation-fastapi/test-requirements.in -o instrumentation/opentelemetry-instrumentation-fastapi/test-requirements.oldest.txt

[rjduffner]

@herin049

Yup it was something simple. Totally missed the comment in the generated file. Thank you and sorry about that. Changes made.

[emdneto]

Not related to this PR, but I think we can drop support for kafka-python-ng now kafka-python development is active again. @rjduffner

[rjduffner]

@emdneto, I agree. My original PR had the drop of kafka-python-ng in it but I pulled it out as it would technically be a breaking change and I wasn't sure if we wanted to do that. I can follow up with ripping that out in a second pr if desired.

[emdneto]

@emdneto, I agree. My original PR had the drop of kafka-python-ng in it but I pulled it out as it would technically be a breaking change and I wasn't sure if we wanted to do that. I can follow up with ripping that out in a second pr if desired.

Sure. Created an issue about that.