Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
github: ddc
ko_fi: ddcsta
ko_fi: ddc
custom: "https://www.paypal.com/ncp/payment/6G9Z78QHUD4RJ"
3 changes: 0 additions & 3 deletions .github/PULL_REQUEST_TEMPLATE
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
## Summary
<!-- Brief description of what this PR does and why -->

## Changes Made
<!-- List the key changes -->

Expand Down
42 changes: 20 additions & 22 deletions .github/workflows/workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ jobs:
name: Lint (ruff)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: astral-sh/ruff-action@v3
- uses: actions/checkout@v7
- uses: astral-sh/ruff-action@v4.1.0
with:
args: "check"

Expand All @@ -28,10 +28,12 @@ jobs:
os: ['ubuntu-latest', 'macos-latest', 'windows-latest']
python-version: ['3.12', '3.14']
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
with:
fetch-depth: 0

- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v9.0.0
with:
enable-cache: true

Expand All @@ -54,39 +56,35 @@ jobs:
uv sync --locked --all-extras --dev --no-install-package mysqlclient --no-install-package aiomysql
elif [[ '${{ matrix.os }}' == 'macos-latest' ]]; then
export PKG_CONFIG_PATH="$(brew --prefix mysql-client)/lib/pkgconfig"
uv sync --locked --all-extras --group dev
uv sync --locked --all-extras --all-groups
else
uv sync --locked --all-extras --group dev
uv sync --locked --all-extras --all-groups
fi
shell: bash

- name: Run tests with coverage
uses: nick-fields/retry@v3
uses: nick-fields/retry@v4
with:
timeout_minutes: 2
max_attempts: 3
command: uv run --no-sync coverage run -m pytest tests/unit && uv run --no-sync coverage report && uv run --no-sync coverage xml
shell: bash

- name: Upload coverage to Codecov
if: matrix.python-version == env.LATEST_PYTHON_VERSION && matrix.os == 'ubuntu-latest'
uses: codecov/codecov-action@v5

- name: Upload test results to Codecov
- name: SonarQube Scan
if: matrix.python-version == env.LATEST_PYTHON_VERSION && matrix.os == 'ubuntu-latest'
uses: codecov/codecov-action@v5
with:
report_type: test_results
uses: SonarSource/sonarqube-scan-action@v8
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

integration-test:
name: Integration Tests
runs-on: ubuntu-latest
needs: lint
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7

- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v9.0.0
with:
enable-cache: true

Expand All @@ -97,7 +95,7 @@ jobs:
run: sudo apt-get update && sudo apt-get install -y default-libmysqlclient-dev pkg-config

- name: Install dependencies
run: uv sync --locked --all-extras --group dev
run: uv sync --locked --all-extras --all-groups

- name: Install ODBC driver for MSSQL
run: |
Expand All @@ -110,7 +108,7 @@ jobs:
shell: bash

- name: Run integration tests
uses: nick-fields/retry@v3
uses: nick-fields/retry@v4
with:
timeout_minutes: 3
max_attempts: 3
Expand All @@ -123,10 +121,10 @@ jobs:
needs: [test, integration-test]
if: startsWith(github.ref, 'refs/tags/v')
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7

- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v9.0.0
with:
enable-cache: true

Expand Down Expand Up @@ -164,7 +162,7 @@ jobs:
path: release-assets/

- name: Create Release
uses: softprops/action-gh-release@v2
uses: softprops/action-gh-release@v3
with:
name: Release ${{ github.ref_name }}
files: release-assets/*
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
repos:
- repo: https://github.com/astral-sh/uv-pre-commit
rev: 0.10.0
rev: 0.12.2
hooks:
- id: uv-lock
96 changes: 52 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,16 @@
<a href="https://ko-fi.com/ddc"><img src="https://img.shields.io/badge/Ko--fi-ddc-FF5E5B?style=plastic&logo=kofi&logoColor=white&color=brightgreen" alt="Ko-fi"/></a>
<a href="https://www.paypal.com/ncp/payment/6G9Z78QHUD4RJ"><img src="https://img.shields.io/badge/Donate-PayPal-brightgreen.svg?style=plastic&logo=paypal&logoColor=white" alt="Donate"/></a>
<br>
<a href="https://pypi.python.org/pypi/ddcDatabases"><img src="https://img.shields.io/pypi/v/ddcDatabases.svg?style=plastic&logo=python&cacheSeconds=3600" alt="PyPi"/></a>
<a href="https://pepy.tech/projects/ddcDatabases"><img src="https://img.shields.io/pepy/dt/ddcDatabases?style=plastic&logo=pypi&logoColor=3776AB" alt="PyPI Downloads"/></a>
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=plastic&logo=creativecommons&logoColor=white" alt="License: MIT"/></a>
<br>
<a href="https://www.python.org/downloads"><img src="https://img.shields.io/pypi/pyversions/ddcDatabases.svg?style=plastic&logo=python&cacheSeconds=3600" alt="Python"/></a>
<a href="https://github.com/astral-sh/uv"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json&style=plastic" alt="uv"/></a>
<a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json&style=plastic" alt="Ruff"/></a>
<br>
<a href="https://pypi.python.org/pypi/ddcDatabases"><img src="https://img.shields.io/pypi/v/ddcDatabases.svg?style=plastic&logo=python&cacheSeconds=3600" alt="PyPi"/></a>
<a href="https://pepy.tech/projects/ddcDatabases"><img src="https://img.shields.io/pepy/dt/ddcDatabases?style=plastic&logo=pypi&logoColor=3776AB" alt="PyPI Downloads"/></a>
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=plastic&logo=creativecommons&logoColor=white" alt="License: MIT"/></a>
<br>
<a href="https://github.com/ddc/ddcDatabases/issues"><img src="https://img.shields.io/github/issues/ddc/ddcDatabases?style=plastic&logo=github&logoColor=white" alt="issues"/></a>
<a href="https://codecov.io/gh/ddc/ddcDatabases"><img src="https://img.shields.io/codecov/c/github/ddc/ddcDatabases?token=XWB53034GI&style=plastic&logo=codecov" alt="codecov"/></a>
<a href="https://sonarcloud.io/component_measures?id=ddc_ddcDatabases&metric=coverage"><img src="https://img.shields.io/sonar/coverage/ddc_ddcDatabases?server=https%3A%2F%2Fsonarcloud.io&style=plastic&logo=sonarqubecloud&logoColor=white" alt="SonarCloud Coverage"/></a>
<a href="https://sonarcloud.io/dashboard?id=ddc_ddcDatabases"><img src="https://img.shields.io/sonar/quality_gate/ddc_ddcDatabases?server=https%3A%2F%2Fsonarcloud.io&style=plastic&logo=sonarqubecloud&logoColor=white" alt="Quality Gate Status"/></a>
<a href="https://github.com/ddc/ddcDatabases/actions/workflows/workflow.yml"><img src="https://img.shields.io/github/actions/workflow/status/ddc/ddcDatabases/workflow.yml?style=plastic&logo=github&logoColor=white&label=CI%2FCD%20Pipeline" alt="CI/CD Pipeline"/></a>
<a href="https://actions-badge.atrox.dev/ddc/ddcDatabases/goto?ref=main"><img src="https://img.shields.io/endpoint.svg?url=https%3A//actions-badge.atrox.dev/ddc/ddcDatabases/badge?ref=main&label=build&logo=github&style=plastic" alt="Build Status"/></a>
Expand Down Expand Up @@ -127,10 +127,10 @@ from ddcdatabases import PostgreSQL, PostgreSQLConnectionRetryConfig

with PostgreSQL(
connection_retry_config=PostgreSQLConnectionRetryConfig(
enable_retry=True, # Enable/disable retry (default: True)
max_retries=3, # Maximum retry attempts (default: 3)
initial_retry_delay=1.0, # Initial delay in seconds (default: 1.0)
max_retry_delay=30.0, # Maximum delay in seconds (default: 30.0)
enable_retry=True, # Enable/disable retry (default: True)
max_retries=3, # Maximum retry attempts (default: 3)
initial_retry_delay=1.0, # Initial delay in seconds (default: 1.0)
max_retry_delay=30.0, # Maximum delay in seconds (default: 30.0)
),
) as session:
# Connection errors will automatically retry with exponential backoff
Expand All @@ -143,11 +143,11 @@ from ddcdatabases import DBUtils, PostgreSQL, PostgreSQLOperationRetryConfig

with PostgreSQL(
operation_retry_config=PostgreSQLOperationRetryConfig(
enable_retry=True, # Enable/disable (default: True)
max_retries=3, # Max attempts (default: 3)
initial_retry_delay=1.0, # Initial delay in seconds (default: 1.0)
max_retry_delay=30.0, # Max delay in seconds (default: 30.0)
jitter=0.1, # Randomization factor (default: 0.1)
enable_retry=True, # Enable/disable (default: True)
max_retries=3, # Max attempts (default: 3)
initial_retry_delay=1.0, # Initial delay in seconds (default: 1.0)
max_retry_delay=30.0, # Max delay in seconds (default: 30.0)
jitter=0.1, # Randomization factor (default: 0.1)
),
) as session:
db_utils = DBUtils(session)
Expand Down Expand Up @@ -189,22 +189,22 @@ conn = PostgreSQLPersistent(
password="postgres",
database="mydb",
config=PersistentConnectionConfig(
idle_timeout=300, # seconds before idle disconnect (default: 300)
health_check_interval=30, # seconds between health checks (default: 30)
auto_reconnect=True, # auto-reconnect on failure (default: True)
idle_timeout=300, # seconds before idle disconnect (default: 300)
health_check_interval=30, # seconds between health checks (default: 30)
auto_reconnect=True, # auto-reconnect on failure (default: True)
),
connection_retry_config=PostgreSQLConnectionRetryConfig(
enable_retry=True, # enable connection retry (default: True)
max_retries=5, # max connection attempts (default: 5)
initial_retry_delay=1.0, # initial delay in seconds (default: 1.0)
max_retry_delay=30.0, # max delay in seconds (default: 30.0)
enable_retry=True, # enable connection retry (default: True)
max_retries=5, # max connection attempts (default: 5)
initial_retry_delay=1.0, # initial delay in seconds (default: 1.0)
max_retry_delay=30.0, # max delay in seconds (default: 30.0)
),
operation_retry_config=PostgreSQLOperationRetryConfig(
enable_retry=True, # enable operation retry (default: True)
max_retries=3, # max operation attempts (default: 3)
initial_retry_delay=0.5, # initial delay in seconds (default: 0.5)
max_retry_delay=10.0, # max delay in seconds (default: 10.0)
jitter=0.1, # randomization factor (default: 0.1)
enable_retry=True, # enable operation retry (default: True)
max_retries=3, # max operation attempts (default: 3)
initial_retry_delay=0.5, # initial delay in seconds (default: 0.5)
max_retry_delay=10.0, # max delay in seconds (default: 10.0)
jitter=0.1, # randomization factor (default: 0.1)
),
)

Expand Down Expand Up @@ -235,19 +235,15 @@ The `execute_with_retry` method provides automatic session management with retry
from ddcdatabases import PostgreSQLPersistent

db = PostgreSQLPersistent(logger=logger)
result = db.execute_with_retry(
lambda session: MyDal(session).do_something()
)
result = db.execute_with_retry(lambda session: MyDal(session).do_something())
```

**Asynchronous:**
```python
from ddcdatabases import PostgreSQLPersistent

db = PostgreSQLPersistent(async_mode=True, logger=logger)
result = await db.execute_with_retry(
lambda session: MyDal(session).do_something()
)
result = await db.execute_with_retry(lambda session: MyDal(session).do_something())
```

The method automatically:
Expand Down Expand Up @@ -390,13 +386,16 @@ import sqlalchemy as sa
from ddcdatabases import DBUtilsAsync, MSSQL
from your_models import Model


async def main():
async with MSSQL(host="127.0.0.1", database="master") as session:
stmt = sa.select(Model).where(Model.id == 1)
db_utils = DBUtilsAsync(session)
results = await db_utils.fetchall(stmt)
for row in results:
print(row)


asyncio.run(main())
```

Expand Down Expand Up @@ -429,10 +428,10 @@ with PostgreSQL(
autocommit=True,
),
ssl_config=PostgreSQLSSLConfig(
ssl_mode="disable", # disable, allow, prefer, require, verify-ca, verify-full
ssl_ca_cert_path=None, # Path to CA certificate
ssl_client_cert_path=None, # Path to client certificate
ssl_client_key_path=None, # Path to client key
ssl_mode="disable", # disable, allow, prefer, require, verify-ca, verify-full
ssl_ca_cert_path=None, # Path to CA certificate
ssl_client_cert_path=None, # Path to client certificate
ssl_client_key_path=None, # Path to client key
),
) as session:
stmt = sa.select(Model).where(Model.id == 1)
Expand All @@ -450,13 +449,16 @@ import sqlalchemy as sa
from ddcdatabases import DBUtilsAsync, PostgreSQL
from your_models import Model


async def main():
async with PostgreSQL(host="127.0.0.1", database="postgres") as session:
stmt = sa.select(Model).where(Model.id == 1)
db_utils = DBUtilsAsync(session)
results = await db_utils.fetchall(stmt)
for row in results:
print(row)


asyncio.run(main())
```

Expand Down Expand Up @@ -496,7 +498,7 @@ with MySQL(
autocommit=True,
),
ssl_config=MySQLSSLConfig(
ssl_mode="DISABLED", # DISABLED, PREFERRED, REQUIRED, VERIFY_CA, VERIFY_IDENTITY
ssl_mode="DISABLED", # DISABLED, PREFERRED, REQUIRED, VERIFY_CA, VERIFY_IDENTITY
ssl_ca_cert_path=None,
ssl_client_cert_path=None,
ssl_client_key_path=None,
Expand All @@ -516,13 +518,16 @@ import asyncio
import sqlalchemy as sa
from ddcdatabases import DBUtilsAsync, MySQL


async def main() -> None:
async with MySQL(host="127.0.0.1", database="dev") as session:
stmt = sa.text("SELECT * FROM users")
db_utils = DBUtilsAsync(session)
results = await db_utils.fetchall(stmt)
for row in results:
print(row)


asyncio.run(main())
```

Expand Down Expand Up @@ -587,7 +592,7 @@ with MongoDB(
query_config=MongoDBQueryConfig(
query={"_id": ObjectId("689c9f71dd642a68cfc60477")},
sort_column="_id",
sort_order="asc", # asc or desc
sort_order="asc", # asc or desc
batch_size=2865,
limit=0,
),
Expand Down Expand Up @@ -624,11 +629,13 @@ with PostgreSQL() as session:
import asyncio
from ddcdatabases import PostgreSQL


async def main():
async with PostgreSQL() as session:
engine = session.bind
# Use engine for advanced operations


asyncio.run(main())
```

Expand All @@ -645,13 +652,13 @@ from ddcdatabases import DBUtils, DBUtilsAsync, PostgreSQL
# Synchronous utilities
with PostgreSQL() as session:
db_utils = DBUtils(session)
results = db_utils.fetchall(stmt) # Returns list of RowMapping objects
results = db_utils.fetchall(stmt) # Returns list of RowMapping objects
results = db_utils.fetchall(stmt, as_dict=True) # Returns list of dictionaries
value = db_utils.fetchvalue(stmt) # Returns single value as string
db_utils.insert(model_instance) # Insert into model table
db_utils.deleteall(Model) # Delete all records from model
db_utils.insertbulk(Model, data_list) # Bulk insert from list of dictionaries
db_utils.execute(stmt) # Execute any SQLAlchemy statement
value = db_utils.fetchvalue(stmt) # Returns single value as string
db_utils.insert(model_instance) # Insert into model table
db_utils.deleteall(Model) # Delete all records from model
db_utils.insertbulk(Model, data_list) # Bulk insert from list of dictionaries
db_utils.execute(stmt) # Execute any SQLAlchemy statement

# Asynchronous utilities (similar interface with await)
async with PostgreSQL() as session:
Expand Down Expand Up @@ -685,6 +692,7 @@ with PostgreSQL(host="localhost", database="mydb", logger=log) as session:

```python
import logging

logging.getLogger("ddcdatabases").setLevel(logging.DEBUG)
logging.getLogger("ddcdatabases").addHandler(logging.StreamHandler())
```
Expand Down
Loading
Loading