[pull] master from scrapinghub:master - #99
Merged
Conversation
* Parse dates wrapped in harmless extra text dateparser.parse() returned None for strings such as "Actualisé le 17 avril 2019" (fr) or "Published on 16 April 2019" (en): an unrecognised leading/trailing word makes the whole string fail the locale applicability check (Dictionary.are_tokens_valid), so no locale is ever tried and parsing gives up before it starts. When strict parsing of the whole string finds nothing and the language is known (given or detected), retry after stripping unrecognised words from the leading/trailing edges. Stripping is applied only when the surviving core is still a valid date for that locale AND it contains a month, weekday or relative expression, so genuine non-dates and bare numbers are not rescued and interior noise still fails. The fallback lives in dateparser.parse() only; get_date_data(), translation, applicability and search_dates() are left untouched, so language detection and date search keep their existing strictness. Related to #521: its search_dates facet is a separate translate_search chunking behaviour and is not addressed here; its parse facet ("Il giorno ...") additionally needs handling of a dangling unit word ("giorno" -> "day") and is left as a known limitation. Close #518 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Rework #518 fix as an opt-in IGNORE_SURROUNDING_TEXT setting Address review feedback on #1356: - Implement the fallback inside DateDataParser.get_date_data() behind a new IGNORE_SURROUNDING_TEXT setting (default off), so parse() and get_date_data() agree and every language hint source (languages, locales, region, DEFAULT_LANGUAGES, detect_languages_function, or none at all) behaves consistently. search_dates and language detection are unaffected by default. - Strip unknown edge tokens inside Locale.translate()/is_applicable() instead of re-joining raw tokens and re-entering the pipeline, so numerals translation, normalization and simplifications run exactly once and no fragile "".join() is needed. - Drop the month/weekday anchor heuristic and the magic KNOWN_WORD_TOKENS[:19] slice: the setting is explicit opt-in, so purely numeric dates, date_formats and relative expressions work too. - Parameterise _get_applicable_locales() instead of duplicating the locale enumeration; delete the dead code paths flagged in review (unreachable TypeError, no-op previous_locales write, dead default parameter). - Move parse()-level tests to test_clean_api.py, add locale-level tests to test_languages.py and parser-level tests to test_date.py; lock in that get_date_data and search_dates stay strict by default. - Document the new setting in docs/settings.rst. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Keep recognized trailing timezones when IGNORE_SURROUNDING_TEXT strips edges Follow-up to the review of #1356 (issue #518). When IGNORE_SURROUNDING_TEXT retries parsing after dropping unrecognized tokens from the edges of the string, Dictionary._strip_unknown_edge_tokens() judged a token "known" via token.isdigit(), a relative-expression match, or membership in the locale dictionary. The only timezone-shaped strings the dictionary lists are ["am", "pm", "UTC", "GMT", "Z"], so every other recognized zone abbreviation (EST, PST, CET, ...) was treated as noise and stripped off the trailing edge before translation. As a result parse('Updated 23 March 2000 1:21 PM EST', settings={'IGNORE_SURROUNDING_TEXT': True}) silently dropped the timezone, while the same string without the leading "Updated " kept it -- a silent correctness bug, not a missing feature. - Add dateparser.timezone_parser.is_timezone_token(): a case-insensitive *full* match on the stripped token, unlike the case-sensitive, unanchored prefix match of word_is_tz(). Edge tokens reaching _strip_unknown_edge_tokens are already lowercased and may carry embedded whitespace (e.g. " est"), and a fullmatch keeps a noise word such as "actualise" from spuriously matching the ACT zone. - Keep a recognized timezone at the trailing edge only: a timezone follows the date it qualifies and does not precede it, so the leading edge keeps stripping unconditionally. The choice, and the limitation that the timezone is kept only when it is the final token (trailing prose after it merges the tokens and drops it), are stated in the method docstring, inline comments and the docs. - are_tokens_valid() now calls the shared _is_known_token() helper instead of restating the same per-token condition, so the two cannot drift. - Document the actual stripping rule in docs/settings.rst: stripping stops at the first token the locale recognizes as a number, relative expression or dictionary word, so an incidental edge number blocks the feature (e.g. 'invoice 12345 paid on 3 March 2019' -> None); point such input at search_dates(). Clarify in the warning that a trailing timezone is applied only as the last token, while a leading, trailing-followed-by-text, or unrecognized one is dropped. - Tests: assert a wrapped trailing timezone (EST, PST, JST, and a parenthesized GMT) yields the same tz-aware instant as the bare date, checking the timezone *name* so a case cannot pass by the dropped offset coinciding with the local timezone; pin that only the trailing edge keeps a timezone; lock both the "trailing timezone followed by more text is dropped" and "edge number blocks stripping" limitations; unit-test is_timezone_token against lowercased, space-padded and look-alike tokens. Refs #518, #1356 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )