feat: add InBerlinWohnen provider#367
Conversation
| async function fetchPage(page) { | ||
| const pageUrl = new URL(url); | ||
| if (page > 1) pageUrl.searchParams.set('page', String(page)); | ||
| const response = await fetch(pageUrl, { headers: { Accept: 'text/html' } }); |
There was a problem hiding this comment.
why do you use fetch instead of puppeteer? Fetch is known to be caught by bot detection pretty easily
There was a problem hiding this comment.
Good point. I replaced the direct listing-page fetches with the shared CloakBrowser. The provider now declares requiresBrowser, receives the job browser in getListings(url, browser), and reuses it across all result pages and detail requests, including the configured proxy and bot-detection handling. Structurally incomplete pages now fail the crawl instead of looking like an empty result.
| }, | ||
| normalize, | ||
| filter: applyBlacklist, | ||
| activeTester: isListingActive, |
There was a problem hiding this comment.
would you mind also implementing fetchDetails?
There was a problem hiding this comment.
Implemented. fetchDetails reuses the shared browser and has explicit parsers for Berlinovo, degewo, GESOBAU, Gewobag, HOWOGE, and WBM. STADT UND LAND currently exposes no useful narrative detail block, so it keeps the structured snapshot description. Unknown hosts, bot detection, and parser failures always return the original listing. The live smoke test verifies one current page for every enrichable partner domain.
| const newCache = new Set(); | ||
| for (const entry of allEntries) { | ||
| newCache.add(toHash(entry?.title, entry?.price, entry?.address)); | ||
| newCache.add(toHash(entry?.job_id, entry?.title, entry?.price, entry?.address)); |
There was a problem hiding this comment.
What do you want to accomplish here?
There was a problem hiding this comment.
The goal is to keep cross-provider similarity deduplication within one job while preventing one job from hiding the same listing from another job. Persistent listing uniqueness is already scoped by (job_id, hash), so the in-memory cache now follows the same boundary. I expanded the tests and fixed the related hard-delete path: hydration and eviction include job_id, addresses use the same normalization as storage, and deleting an already hidden duplicate no longer evicts the active listing cache entry.
|
Addressed all three review threads in |
| } | ||
|
|
||
| if (!browser && matchedProvider.config.getListings == null) { | ||
| if (!browser && (matchedProvider.config.getListings == null || matchedProvider.config.requiresBrowser)) { |
There was a problem hiding this comment.
Why is this needed? I know you're using claude.. While this is fine, you should tell it not to push in vars/methods for the sake of doing so.
please remove the requiredBrowser again from where you/claude added it. The shared browser instance should be available in your provider whether or not you make use out of it ;)
The implementation of your provider is already much more complicated than the rest.
Thanks.
2 participants
Summary
inberlinwohnen.deas a native Fredy providerImplementation
The provider reuses one shared browser for pagination and optional detail enrichment, preserving Fredy's proxy, timeout, and bot-detection behavior. It normalizes stable partner IDs, canonical allowlisted deeplinks, total rent, size, rooms, address, image, provider details, WBS, and availability.
fetchDetailsuses explicit selectors for Berlinovo, degewo, GESOBAU, Gewobag, HOWOGE, and WBM. STADT UND LAND and any failed or unsupported detail extraction retain the structured InBerlinWohnen snapshot description.Similarity hashes now include
jobId, preserving cross-provider deduplication within a job without suppressing the same listing in another job. Cache hydration, address normalization, hard-delete eviction, and hidden-duplicate handling use the same identity.Verification
npm run test:offline: 41 files, 252 tests passed, 1 live-only test skippednpm test -- test/provider/inberlinwohnen.test.js --testTimeout=180000: 5 live tests passed, 13 offline-only tests skippednpm run lint: passednpm run format:check: passednpm run build:frontend: passed