Skip to content

fix: do not crash on unparseable cursor state value during concurrent cursor construction#1070

Draft
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/1783684033-fix-datetime-state-converter-bad-cursor-value
Draft

fix: do not crash on unparseable cursor state value during concurrent cursor construction#1070
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/1783684033-fix-datetime-state-converter-bad-cursor-value

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a hard source startup crash where a boolean value under the cursor field key reaches the datetime state converter during ConcurrentCursor construction and raises ValueError: No format in [...] matching True, failing the entire source before any stream is read.

Investigated for oncall issue airbytehq/oncall#13084 (HubSpot). The customer's saved connection state contained boolean bookkeeping markers (use_global_cursor: true, __ab_no_cursor_state_message: true); one reached the parent cursor's datetime parser via the associations_* substream partition router. Because the source dies at startup, no new state is ever committed — which is why manual state edits appeared to "revert" after each failed sync.

The read/observe path already tolerates a bad cursor value (ConcurrentPerPartitionCursor.observe, #758). The construction / state-init path (ConcurrentCursor.__init___get_concurrent_stateconvert_from_sequential_state_get_sync_start) was still unguarded.

Change

In DateTimeStreamStateConverter._get_sync_start, ignore a boolean saved cursor value (never a valid datetime) and fall back to the configured start date instead of crashing:

saved_cursor_value = stream_state[cursor_field.cursor_field_key]
if isinstance(saved_cursor_value, bool):
    logger.warning("Ignoring saved state ... not a valid datetime. Falling back to the start date", ...)
else:
    prev_sync_low_water_mark = self.parse_timestamp(saved_cursor_value)

The guard is intentionally scoped to bool rather than swallowing every ValueError. A genuinely malformed datetime string must still raise: StateDelegatingStream's api_retention_period cursor-age validation (#890) deliberately fails fast on unparseable cursor strings, and test_cursor_age_validation_raises_error_for_unparseable_cursor asserts that. Booleans are unambiguously invalid cursor values (they originate from internal state markers, not user data), so skipping only them fixes the crash without regressing that behavior.

This is a robustness fix (fall back rather than abort); no public interface changes, non-breaking.

Test Coverage

Added regression tests in unit_tests/sources/streams/concurrent/test_datetime_state_converter.py:

  • Parametrized _get_sync_start cases with a boolean state value (CustomFormat converter) — falls back to start and to zero_value.
  • test_get_sync_start_with_boolean_state_value_does_not_raise and test_convert_from_sequential_state_with_boolean_state_value_does_not_raise.

All fail without the fix (reproducing the ValueError) and pass with it. Full converter file passes (36 tests) and test_cursor_age_validation_raises_error_for_unparseable_cursor still passes. ruff check, ruff format, and mypy clean on the changed files.

Related to https://github.com/airbytehq/oncall/issues/13084:

Note: this addresses the hard startup crash. The customer's original, softer complaint (slow syncs / repeated *_property_history re-emissions) is a separate performance concern tracked in the same issue.

Link to Devin session: https://app.devin.ai/sessions/ee9a064393c44676a341378e32c2a746

… cursor construction

Co-Authored-By: bot_apk <apk@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1783684033-fix-datetime-state-converter-bad-cursor-value#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1783684033-fix-datetime-state-converter-bad-cursor-value

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

PyTest Results (Fast)

4 131 tests  +4   4 119 ✅ +4   8m 0s ⏱️ +19s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit a049329. ± Comparison against base commit dd9558c.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

PyTest Results (Full)

4 134 tests  +4   4 122 ✅ +4   11m 38s ⏱️ + 4m 23s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit a049329. ± Comparison against base commit dd9558c.

♻️ This comment has been updated with latest results.

…il-fast

Co-Authored-By: bot_apk <apk@cognition.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants