[tests] Analysis of potential flaky e2e tests (AI generated)

[susnux]

Playwright flakiness audit — findings & todo

Audit of tests/playwright (specs, page objects, utils, fixtures) correlated with
flaky-test notices from CI runs on 2026-07-17. Ordered by severity.

CI context: workers: 1, retries: 1, one shared Nextcloud server per shard.
Local context: fullyParallel with many workers, reuseExistingServer — several
findings only bite locally today but will bite CI once shard workers > 1.

Confirmed flaky on CI

• Copy-to-same-folder tests time out at 30 s — e2e/files/files-copy-move.spec.ts:84,95,111 (flaked in 2 runs, 3 tests) and e2e/files/live-photos.spec.ts:39,50 (2 runs).
  Failure surfaces at support/sections/CopyMoveDialogPage.ts:68 (waitForResponse for COPY), but the helper is correct — the client's same-folder copy path (apps/files/src/actions/moveOrCopyAction.ts) does PROPFIND (conflict check) → COPY → PROPFIND (stat) serially. On a loaded shard this plus fixture setup blows the default 30 s test timeout.
  Fix: raise the global timeout (e.g. 60 s) in playwright.config.ts, or test.slow() on the copy-heavy tests.

• handlePasswordConfirmation helper — support/utils/password-confirmation.ts (~12 call sites; flaked in e2e/appstore/admin-settings-apps.spec.ts:159 via line 205).
  Two defects: (1) line 19 probes for the dialog with a hardcoded 500 ms window and silently returns if it isn't visible yet — on a slow shard the confirmation is skipped and the guarded action never fires; (2) line 32 waits for hidden with no bound or retry — today's CI failure.
  Fix: race the dialog appearance against the expected success response instead of a 500 ms probe; wrap fill + confirm + hidden in an expect(...).toPass() retry loop.

High

• clearTags() wipes ALL system tags globally — support/utils/systemtags.ts:47, wired into afterAll of 5 systemtags specs (files-view, files-sidebar, files-inline-action, files-bulk-action, admin-settings-restrictions). System tags are server-global; under local fullyParallel one spec's teardown deletes tags another spec is mid-assertion on.
  Fix: track and delete only the tags each test created.

• systemtags/admin-settings.spec.ts mass-deletes all tags in beforeEach + fixed names foo/bar — e2e/systemtags/admin-settings.spec.ts:10-19. Races against concurrently created tags of the parallel default-project systemtags specs; deleteTag (non-forced) throws if a tag vanishes underneath it.
  Fix: random tag names + targeted cleanup.

• admin-settings project is not serialized against the default project — playwright.config.ts. Per-project workers: 1 only serializes admin-settings tests among themselves; locally they interleave with default-project tests while mutating global server state. Worst offenders:

  • e2e/files_external/admin-settings-home-folder-root-mount.spec.ts:22-43 — mounts a read-only storage over every user's home and flips overwrites_home_folders
  • e2e/files_external/admin-settings-external-storage.spec.ts:20 — deleteAllGlobalStorages() in beforeEach
  • e2e/appstore/admin-settings-apps.spec.ts:12-23 — enables/disables apps globally with no restore

  Fix: add project dependencies (or a separate serialized invocation) so global-state specs never overlap parallel file tests.

• Shared admin account preferences raced by parallel specs — e2e/users/users-columns.spec.ts:11-30 persists column visibility against the single admin account while users.spec.ts, users-disable, users-search, users-manager, users-modify, users-groups (all default project, parallel locally) assert on the same list's columns. Also users-groups.spec.ts:176-205 persists the group sort-order preference.
  Fix: serialize the admin user-list specs or isolate per-test accounts.

• Fixture login/user-creation band-aid: single retry after fixed 800 ms — support/fixtures/random-user.ts:23, random-user-session.ts:20, admin-session.ts:18. The transient is real (login returned HTTP 500 twice in a row in a local run on 2026-07-17, failing a test). One fixed-delay retry is not robust.
  Fix: retry with backoff (e.g. 3 attempts) or poll the login endpoint until healthy.

• drag-n-drop.spec.ts one-shot putFired boolean — e2e/files/drag-n-drop.spec.ts:111-123. Read the instant the conflict dialog appears; a slightly-late PUT is missed, so the assertion can pass when the guarantee is violated. The multi-file drop test (:29-42) also has no waitForResponse for the PUTs, unlike the single-file test. Whole file uses synthetic dispatchEvent DnD — inherently fragile.
  Fix: wait for a settle window / expected request count before asserting; add PUT waits to the multi-file test.

Medium

• ContactsMenuPage.search never awaits the debounced request — support/sections/ContactsMenuPage.ts:67-69. Docstring promises to wait for the /contactsmenu/contacts response but only fill()s; callers assert on results immediately.
  Fix: register waitForResponse before fill() (same pattern as open()).

• SettingsUsersPage.saveEditDialog — support/sections/SettingsUsersPage.ts:85-92. force: true click (can fire before the handler is wired) combined with the 500 ms password-confirmation probe.
  Fix: drop force, and fix handlePasswordConfirmation (above).

• FilesListPage.selectAll/deselectAll force-click with no state assertion — support/sections/FilesListPage.ts:297-308. A force-click landing mid-render can no-op; the caller only fails later (feeds the flaky files-delete test).
  Fix: assert the resulting checkbox/selection state after toggling.

• theming.pickColor — support/utils/theming.ts:67-79. Force-click on the trigger races the picker opening; swatch chosen by positional .nth(index) with no stable identity; "Choose" clicked without confirming the swatch registered.
  Fix: wait for the picker to open, select by color value/label, verify selection before confirming.

• AppstorePage.appRow substring match + .first() — support/sections/AppstorePage.ts:61-63. hasText is a substring match, so "Files" also matches "Files sharing"; .first() silently picks by DOM order. Feeds enableButton/disableButton/appLink.
  Fix: match the app-name cell exactly.

• files_trashbin/files.spec.ts sharee deletes before share propagation is confirmed — e2e/files_trashbin/files.spec.ts:67-79. Bob rms via his DAV path with no waitForShare poll (the sharing copy-move spec added exactly that); rm throws on 404.
  Fix: reuse the waitForShare poll before acting as the sharee.

• scrolling.spec.ts / viewport.ts virtualization assumptions — support/utils/viewport.ts:34-51 hardcodes row-height constants mirrored from the component; isFullyInViewport does toHaveCount(1) first, so a row virtualized fully out of the DOM makes the poll spin to timeout instead of resolving false (e2e/files/scrolling.spec.ts:45,84).
  Fix: treat count 0 as "not in viewport"; derive row height from a rendered row instead of constants.

• hotkeys.spec.ts races keyboard focus — e2e/files/hotkeys.spec.ts:36-73. Key presses right after URL-driven active-row changes; the Delete test asserts row count with no DELETE response wait.
  Fix: wait for focus/selection state before pressing; pair Delete with a waitForResponse.

• files-renaming.spec.ts layout-measured virtualization test + one-shot selection read — e2e/files/files-renaming.spec.ts:43-45 reads selectionStart/End once with no retry; :132-174 computes exact viewport heights and scrolls to trigger re-render.
  Fix: poll the selection read; loosen the layout assumptions.

• files_external/admin-settings-home-folder-root-mount.spec.ts one-shot occ config reads + admitted HACK — lines 26/36/41 expect(await getOverwritesHomeFolders()).toBe(...) without expect.poll; lines 34-35 double open() as a propagation workaround.
  Fix: wrap the config reads in expect.poll; replace the double-open with an explicit wait on the observable effect.

• Hardcoded global-state counts — e2e/core/header-access-levels.spec.ts:19,48 (toHaveCount(6)/(9) account-menu entries), e2e/theming/user-settings-app-order.spec.ts:17 and e2e/theming/admin-settings-default-app.spec.ts:50 (toHaveCount(2) nav apps). Brittle to any globally enabled app (see admin-settings-apps leak above).
  Fix: assert on the specific expected entries, not totals.

• login.spec.ts global brute-force toggle + shared IP — e2e/login/login.spec.ts:32-38. Failed-login tests are only safe while protection is off; once afterAll re-enables it, accumulated failures from the shared runner IP can 429-throttle other suites' logins.
  Fix: reset the brute-force attempt table in afterAll, or isolate these tests.

• personal-info.spec.ts global config mutation + the repo's only networkidle waits — e2e/settings/personal-info.spec.ts:59-72 (force_language, force_locale, has_internet_connection from a parallel-project spec) and :88, :127 (networkidle is timing-fragile with background polling).
  Fix: move to the serialized project or scope the config changes; replace networkidle with assertions on concrete post-reload UI state.

• header-contacts-menu.spec.ts enumeration on a shared server — e2e/core/header-contacts-menu.spec.ts:33-57,97-117. Asserts random users appear in a limited/paginated contacts list — accumulated users on a reused server can push them out; also toggles global shareapi_restrict_user_enumeration_to_group while parallel enumeration tests run.
  Fix: search for the exact user instead of relying on the default listing; serialize the enumeration-config window.

• a11y-color-contrast.spec.ts reads global theming state from the parallel project — e2e/theming/a11y-color-contrast.spec.ts:49-116. Primary color is global admin state mutated by admin-settings-colors.spec.ts; axe runs once after goto with no wait for theming CSS application.
  Fix: wait for the theming stylesheet/CSS variables before running axe; covered further by the project-serialization fix above.

• files-external-failed.spec.ts one-shot isVisible() + reload-once — e2e/files_external/files-external-failed.spec.ts:33-36,65-67; also synchronous page.url() comparison right after a click (:42-44, :73-75).
  Fix: replace with retrying toBeVisible; assert URL with expect(page).toHaveURL.

Low

• dav/availability.spec.ts fixed replacement-user — e2e/dav/availability.spec.ts:48-95. Pre-delete + finally mostly handle it, but a fixed global username remains a latent collision.
  Fix: random user name.

• Generic unnamed getByRole('dialog') locators — support/sections/CopyMoveDialogPage.ts:23-24, BackgroundFilePickerDialogPage.ts:16-17, AppstorePage.ts:146-148. Match any open dialog; fragile if a second dialog/toast overlaps.
  Fix: name-scope the dialogs.

• FilesNavigationPage.setShowHiddenFiles force-toggle without state assertion — support/sections/FilesNavigationPage.ts:78-88. A dropped force-toggle goes unnoticed until a later assertion.
  Fix: assert the switch state after toggling (as SetupPage.selectDatabase does).

• SetupPage.installRecommendedApps swallows a genuine timeout — support/sections/SetupPage.ts:122-141. .catch(() => {}) hides "prompt never consumed"; loop trusts caller-supplied appCount.
  Fix: fail loudly on the timeout; derive the prompt count from the page.

• users-modify.spec.ts loose group-name regex — e2e/users/users-modify.spec.ts:124-143. 4–6 char hex fragment matched via unanchored RegExp could hit another group.
  Fix: exact-match the option.

• login-redirect.spec.ts URL fragment in redirect regex — e2e/login/login-redirect.spec.ts:42-44. Fragments aren't sent to the server; brittle if the app normalizes the value.

Done

• files-settings.spec.ts "Can set it to personal files" re-navigates before the default_view config PUT persists — fixed by registering waitForResponse for /apps/files/api/v1/config/default_view before clicking the radio (same idiom as FilesListPage.enableGridView). Verified locally. test(files): wait for PUT before continue with tests #62284
• Setup wizard DB installs run under the default 30 s timeout — e2e/core/setup.spec.ts:136 (MySQL) and :145 (MariaDB) each flaked once. Only Oracle gets test.setTimeout(200_000) (line 165), yet MySQL/MariaDB/Postgres also perform a full Nextcloud installation. test: mark installation tests as slow #62282
• users.spec.ts fixed username with cleanup outside finally — e2e/users/users.spec.ts:14-31. newuser-basic leaks on assertion failure and collides on the reused local server. Fix: random id + finally, like the file's other tests.
  test(users): prevent flaky tests with hardcoded values #62283
• files-delete "can delete multiple files" — e2e/files/files-delete.spec.ts:39-54 (flaked once, non-204 at line 53).
  Two defects: (1) the 5 waitForResponse listeners have identical predicates, so they can all resolve to the same first DELETE response — 5 distinct deletes are never actually verified; (2) expect(response.status()).toBe(204) is one-shot on raw status; parallel DAV DELETEs can transiently return 423 (file locked, e.g. preview generation).
  Fix: match each file's distinct URL per listener; assert the UI end state (rows gone) instead of, or in addition to, raw status.
  test(files): wait for actual requests when deleting files #62286