added documents marked as to be removed clean script (file_garbage_remover) - #145
added documents marked as to be removed clean script (file_garbage_remover)#145idelcano wants to merge 25 commits into
Conversation
cgbautista
left a comment
cgbautista
left a comment
There was a problem hiding this comment.
Looks good from my side. Suggested some changes, but would work as is.
|
Hi @cgbautista |
|
|
||
| Test Mode (dry-run): | ||
|
|
||
| ```./file_garbage_remover_tomcat.py --test --config /path/to/config.json``` |
There was a problem hiding this comment.
Test mode also requires the db password to connect and retrieve the list of fileResources, so the code example should contain the export line as the --force example
| def ensure_audit_table_exists(cursor, dry_run=False): | ||
| if dry_run: | ||
| log("[DRY RUN] Would execute:") | ||
| log(" CREATE TABLE IF NOT EXISTS fileresourcesaudit AS TABLE fileresource WITH NO DATA;") |
There was a problem hiding this comment.
I would include this SQL query (and the same query in line 109) the same way as the other 3 queries in lines 12-30
|
|
||
| db_password = os.environ.get("DB_PASSWORD_FILE_G") | ||
| if not db_password: | ||
| log("❌ Missing required environment variable: DB_PASSWORD") |
There was a problem hiding this comment.
Error message informs that the user did not inform an environmental variable that is not the same that the code checks.
If the password is expected to be as plain text in the env., it should not be called FILE
There was a problem hiding this comment.
Yes, the name was really bad
I used it to avoid introducing another DB_PASSWORD environment variable (definitely improvable) and was thinking in FILE_GARBAGE.py, but it doesn't make sense. I've renamed it to DB_PASSWORD_FG.
|
Thanks @cgbautista I did a new commits to fix this minus issues |
…environmental variable
cgbautista
left a comment
cgbautista
left a comment
There was a problem hiding this comment.
Tested in local enviroment both on docker and tomcat instances.
… event, datavalue, trackedentityattribute values.
…), and add save-all-as-notified
idelcano
changed the title
|
Hi @cgbautista , I added several commits to the file garbage remover with big changes from your review. Changes summary (grouped): Orphan detection: added 24h safety window; improved SQL for data value file resources; refactor of SQL queries. |
2 participants
This script removes documents that should have been deleted by DHIS2 but weren't.
to use
python3 file_garbage_remover.py -i instancename
Introduced file_garbage_remover_tomcat.py:
A robust script designed specifically for production (Tomcat) environments. This script safely handles orphaned document resources by moving them to a temporary directory, creating an audit trail in the database, and then cleaning up original entries. It provides test (--test) and force (--force) modes for secure operations.
Updated Documentation:
Clearly documented both scripts (file_garbage_remover_tomcat.py and the previously existing Docker test script) to distinguish their usage, operation, and precautions.
This improvement allows safer file handling and audit tracking in production, complementing the simpler Docker-based script intended for testing purposes.
I have create a wrapper in dev to test it in dry run mode:
/home/tomcatuser/bin/file_garbage_remover/run_file_garbage_remover.sh test