Skip to content

feat: Adding a connection pool to reduce the time spent opening connection to Postgres#4851

Open
Ma77Ball wants to merge 6 commits intoapache:mainfrom
Ma77Ball:fix/connection-pool
Open

feat: Adding a connection pool to reduce the time spent opening connection to Postgres#4851
Ma77Ball wants to merge 6 commits intoapache:mainfrom
Ma77Ball:fix/connection-pool

Conversation

@Ma77Ball
Copy link
Copy Markdown
Contributor

@Ma77Ball Ma77Ball commented May 3, 2026

What changes were proposed in this PR?

This PR adds a HikariCP connection pool to SqlServer.scala so that jOOQ queries borrow pre-authenticated connections from a pool instead of opening a new TCP connection and performing SCRAM-SHA-256 authentication on every database call.

What changed in SqlServer.scala:

  • Added a HikariConfig with maximumPoolSize=10, minimumIdle=2, connectionTimeout=30s, idleTimeout=10min, maxLifetime=30min (chosen to stay below typical PostgreSQL idle and load-balancer connection-reaping windows)
  • Initialized a HikariDataSource and built the jOOQ DSLContext via DSL.using(dataSource, SQL_DIALECT) so queries draw from the pool
  • Added a close() method to shut down the pool on application or test teardown, and updated clearInstance() to call it, so test classes that replace the singleton don't leak pool threads
  • Added a documented warning against caching the DSLContext in a val or lazy val. MockTexeraDB swaps the singleton between test classes, and a cached context would hold a stale reference to a dead pool

Any related issues, documentation, discussions?

closes: #4852

How was this PR tested?

  • Existing unit and integration tests pass — the pool-backed DSLContext is a drop-in replacement for the previous direct-driver context
  • Verified the MockTexeraDB flow still works across test classes by running the full test suite and confirming no "Connection refused" failures from stale contexts

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude opus 4.6 (Anthropic) in compliance with ASF

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file fix common labels May 3, 2026
@Ma77Ball Ma77Ball changed the title Adding a connection pool to reduce the time spent opening connection to Postgres refactor: Adding a connection pool to reduce the time spent opening connection to Postgres May 3, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.17%. Comparing base (056e39d) to head (c37711b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #4851   +/-   ##
=========================================
  Coverage     42.16%   42.17%           
  Complexity     2180     2180           
=========================================
  Files           980      980           
  Lines         36305    36297    -8     
  Branches       3783     3783           
=========================================
  Hits          15309    15309           
+ Misses        20068    20060    -8     
  Partials        928      928           
Flag Coverage Δ
access-control-service 39.53% <ø> (ø)
computing-unit-managing-service 0.00% <ø> (ø)
config-service 0.00% <ø> (ø)
file-service 33.24% <ø> (ø)
workflow-compiling-service 47.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

@Yicong-Huang Yicong-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Left inline comments

Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala
Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala Outdated
Comment thread common/dao/build.sbt
Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala Outdated
@Yicong-Huang
Copy link
Copy Markdown
Contributor

please also change the title. this is not a refactor. we are adding a new feature.

@Ma77Ball Ma77Ball changed the title refactor: Adding a connection pool to reduce the time spent opening connection to Postgres feat: Adding a connection pool to reduce the time spent opening connection to Postgres May 4, 2026
@github-actions github-actions Bot added the platform Non-amber Scala service paths label May 4, 2026
@Ma77Ball Ma77Ball requested a review from Yicong-Huang May 5, 2026 00:03
Comment thread common/dao/build.sbt

libraryDependencies ++= Seq(
"org.postgresql" % "postgresql" % "42.7.4",
"com.zaxxer" % "HikariCP" % "5.1.0"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the latest version is HikariCP-7.0.2, any particular reason to stay on v5.1.0?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file fix platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HikariCP connection pool to SqlServer

3 participants