fix: resolve bug on add properties from records related methods by mcllerena · Pull Request #129 · NatLabRockies/plexosdb

mcllerena · 2026-04-29T20:47:44Z

No description provided.

codecov-commenter · 2026-04-29T21:20:37Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.66%. Comparing base (c5d9a38) to head (41a380b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #129      +/-   ##
==========================================
+ Coverage   95.64%   95.66%   +0.02%     
==========================================
  Files           8        8              
  Lines        1721     1729       +8     
==========================================
+ Hits         1646     1654       +8     
  Misses         75       75

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pesap · 2026-04-30T16:15:32Z

+        param_tuple = (membership_id, property_id, record["value"])
+        row_key = (membership_id, property_id, record["value"], row_index)
+        params.append(param_tuple)
+        metadata_map[row_key] = {


Index source can drift here. metadata_map keys use row_index from normalized_records, but downstream maps and lookups key off enumerate(params). If a row is skipped before append, metadata can miss on later valid rows.

Suggested change:

Suggested change

param_tuple = (membership_id, property_id, record["value"])

row_key = (membership_id, property_id, record["value"], row_index)

params.append(param_tuple)

metadata_map[row_key] = {

insert_index = len(params)

param_tuple = (membership_id, property_id, record["value"])

row_key = (membership_id, property_id, record["value"], insert_index)

params.append(param_tuple)

metadata_map[row_key] = {

Then use the same insert_index convention in later key lookups too (insert_property_values, apply_scenario_tags, _collect_text_rows).

pesap · 2026-04-30T16:15:51Z

+    assert len(rows) == 2
+    assert {row[1] for row in rows} == {1, 2}
+    assert {row[2] for row in rows} == {1}
+    assert {row[3] for row in rows} == {"row1.csv", "row2.csv"}


Great coverage for duplicate value rows. Please add one regression with a skipped row before valid rows to lock the index alignment fix.

Suggested change:

Suggested change

assert {row[3] for row in rows} == {"row1.csv", "row2.csv"}

assert {row[3] for row in rows} == {"row1.csv", "row2.csv"}

def test_add_properties_from_records_duplicate_rows_after_skipped_row_preserve_links(

db_instance_with_schema: PlexosDB,

) -> None:

from plexosdb import ClassEnum, CollectionEnum

db = db_instance_with_schema

db.add_object(ClassEnum.Generator, "GEN_MIX")

records = [

{

"name": "MISSING_OBJECT",

"property": "Max Capacity",

"value": 0.0,

"band": 9,

"datafile_text": "skip.csv",

},

{

"name": "GEN_MIX",

"property": "Max Capacity",

"value": 0.0,

"band": 1,

"datafile_text": "row1.csv",

},

{

"name": "GEN_MIX",

"property": "Max Capacity",

"value": 0.0,

"band": 2,

"datafile_text": "row2.csv",

},

]

db.add_properties_from_records(

records,

object_class=ClassEnum.Generator,

parent_class=ClassEnum.System,

collection=CollectionEnum.Generators,

scenario="ScenarioMix",

)

rows = db.query(

"""

SELECT d.data_id, b.band_id

FROM t_data d

LEFT JOIN t_band b ON b.data_id = d.data_id

JOIN t_membership m ON d.membership_id = m.membership_id

JOIN t_object o ON m.child_object_id = o.object_id

JOIN t_property p ON d.property_id = p.property_id

WHERE o.name = ? AND p.name = ?

ORDER BY d.data_id

""",

("GEN_MIX", "Max Capacity"),

)

assert len(rows) == 2

assert {row[1] for row in rows} == {1, 2}

pesap

Left two inline comments with concrete suggestions. Main blocker is row index key drift between normalized_records and params, which can drop metadata/text/tag links after skipped rows. Please fix index source and add a mixed valid plus skipped regression test.

…oach

Bobby-Heyer · 2026-05-11T05:49:23Z

Hi @mcllerena

Thanks for actioning this so quickly, I’ve tested the fix on branch ml/fix-bug. It works exactly as described. All band and date metadata survives.

minimum reproducible test:

def bulk_add(db, flat_records: list, object_class: str) -> None:
    logging.info("=" * 60)
    logging.info(f"STEP: add bids for {object_class}")
    logging.info("=" * 60)

    # Map object_class → enums (avoids duplicating the try/except per branch)
    class_map = {
        'Generators': (ClassEnum.Generator, CollectionEnum.Generators),
        'Batteries':  (ClassEnum.Battery,   CollectionEnum.Batteries),
    }
    if object_class not in class_map:
        logging.error(f"Unknown object_class: {object_class}")
        return
    obj_enum, coll_enum = class_map[object_class]

    try:
        db.add_properties_from_records(
            flat_records,
            object_class=obj_enum,
            parent_class=ClassEnum.System,
            collection=coll_enum,
            scenario=None,
        )
        logging.error(f"  ✓ SUCCESS in adding Records")
    except Exception as e:
        logging.error(f"  ❌ FAILED to Add Records: {e}")

flat_records = [
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 264.0, 'band': 1, 'date_from': datetime.datetime(2025, 7, 1, 0, 0), 'date_to': datetime.datetime(2026, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 286.71, 'band': 1, 'date_from': datetime.datetime(2026, 7, 1, 0, 0), 'date_to': datetime.datetime(2027, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 264.0, 'band': 1, 'date_from': datetime.datetime(2027, 7, 1, 0, 0), 'date_to': datetime.datetime(2028, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 264.0, 'band': 1, 'date_from': datetime.datetime(2028, 7, 1, 0, 0), 'date_to': datetime.datetime(2029, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 264.0, 'band': 1, 'date_from': datetime.datetime(2029, 7, 1, 0, 0), 'date_to': datetime.datetime(2030, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 264.0, 'band': 1, 'date_from': datetime.datetime(2030, 7, 1, 0, 0), 'date_to': datetime.datetime(2031, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2031, 7, 1, 0, 0), 'date_to': datetime.datetime(2032, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2032, 7, 1, 0, 0), 'date_to': datetime.datetime(2033, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2033, 7, 1, 0, 0), 'date_to': datetime.datetime(2034, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2034, 7, 1, 0, 0), 'date_to': datetime.datetime(2035, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2035, 7, 1, 0, 0), 'date_to': datetime.datetime(2036, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2036, 7, 1, 0, 0), 'date_to': datetime.datetime(2037, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2037, 7, 1, 0, 0), 'date_to': datetime.datetime(2038, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2038, 7, 1, 0, 0), 'date_to': datetime.datetime(2039, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2039, 7, 1, 0, 0), 'date_to': datetime.datetime(2040, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2040, 7, 1, 0, 0), 'date_to': datetime.datetime(2041, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2041, 7, 1, 0, 0), 'date_to': datetime.datetime(2042, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2042, 7, 1, 0, 0), 'date_to': datetime.datetime(2043, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2043, 7, 1, 0, 0), 'date_to': datetime.datetime(2044, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2044, 7, 1, 0, 0), 'date_to': datetime.datetime(2045, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2045, 7, 1, 0, 0), 'date_to': datetime.datetime(2046, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2046, 7, 1, 0, 0), 'date_to': datetime.datetime(2047, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2047, 7, 1, 0, 0), 'date_to': datetime.datetime(2048, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2048, 7, 1, 0, 0), 'date_to': datetime.datetime(2049, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 1, 'date_from': datetime.datetime(2049, 7, 1, 0, 0), 'date_to': datetime.datetime(2050, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 112.41, 'band': 2, 'date_from': datetime.datetime(2025, 7, 1, 0, 0), 'date_to': datetime.datetime(2026, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 124.58, 'band': 2, 'date_from': datetime.datetime(2026, 7, 1, 0, 0), 'date_to': datetime.datetime(2027, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 117.45, 'band': 2, 'date_from': datetime.datetime(2027, 7, 1, 0, 0), 'date_to': datetime.datetime(2028, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 123.36, 'band': 2, 'date_from': datetime.datetime(2028, 7, 1, 0, 0), 'date_to': datetime.datetime(2029, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 135.92, 'band': 2, 'date_from': datetime.datetime(2029, 7, 1, 0, 0), 'date_to': datetime.datetime(2030, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 129.72, 'band': 2, 'date_from': datetime.datetime(2030, 7, 1, 0, 0), 'date_to': datetime.datetime(2031, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2031, 7, 1, 0, 0), 'date_to': datetime.datetime(2032, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2032, 7, 1, 0, 0), 'date_to': datetime.datetime(2033, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2033, 7, 1, 0, 0), 'date_to': datetime.datetime(2034, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2034, 7, 1, 0, 0), 'date_to': datetime.datetime(2035, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2035, 7, 1, 0, 0), 'date_to': datetime.datetime(2036, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2036, 7, 1, 0, 0), 'date_to': datetime.datetime(2037, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2037, 7, 1, 0, 0), 'date_to': datetime.datetime(2038, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2038, 7, 1, 0, 0), 'date_to': datetime.datetime(2039, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2039, 7, 1, 0, 0), 'date_to': datetime.datetime(2040, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2040, 7, 1, 0, 0), 'date_to': datetime.datetime(2041, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2041, 7, 1, 0, 0), 'date_to': datetime.datetime(2042, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2042, 7, 1, 0, 0), 'date_to': datetime.datetime(2043, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2043, 7, 1, 0, 0), 'date_to': datetime.datetime(2044, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2044, 7, 1, 0, 0), 'date_to': datetime.datetime(2045, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2045, 7, 1, 0, 0), 'date_to': datetime.datetime(2046, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2046, 7, 1, 0, 0), 'date_to': datetime.datetime(2047, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2047, 7, 1, 0, 0), 'date_to': datetime.datetime(2048, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2048, 7, 1, 0, 0), 'date_to': datetime.datetime(2049, 6, 30, 0, 0)}, 
{'name': 'BW01', 'property': 'Offer Quantity', 'value': 0.0, 'band': 2, 'date_from': datetime.datetime(2049, 7, 1, 0, 0), 'date_to': datetime.datetime(2050, 6, 30, 0, 0)}, 
]

db = PlexosDB.from_xml(INPUT_XML)
bulk_add(db, flat_records=flat_records, object_class='Generators')

fix: resolve bug on add properties from records related methods

67cabd0

mcllerena mentioned this pull request Apr 29, 2026

bug: add_properties_from_records silently drops band/date metadata for records sharing (name, property, value) #126

Closed

mcllerena requested a review from pesap April 29, 2026 20:51

mcllerena linked an issue Apr 29, 2026 that may be closed by this pull request

bug: add_properties_from_records silently drops band/date metadata for records sharing (name, property, value) #126

Closed

fix: resolve same issue on text field data filesclear

146d17f

github-actions Bot added the tests label Apr 29, 2026

pesap reviewed Apr 30, 2026

View reviewed changes

pesap requested changes Apr 30, 2026

View reviewed changes

mcllerena and others added 2 commits April 30, 2026 12:18

Merge branch 'main' into ml/fix-bug

db9fc5e

fix: convert to normalized records back and update test with new appr…

41a380b

…oach

mcllerena requested a review from pesap April 30, 2026 18:46

pesap approved these changes May 5, 2026

View reviewed changes

pesap merged commit b3f4dbb into main May 11, 2026
28 checks passed

pesap deleted the ml/fix-bug branch May 11, 2026 16:24

github-actions Bot mentioned this pull request May 11, 2026

chore(main): release 1.4.0 #131

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: resolve bug on add properties from records related methods#129

fix: resolve bug on add properties from records related methods#129
pesap merged 4 commits into
mainfrom
ml/fix-bug

mcllerena commented Apr 29, 2026

Uh oh!

codecov-commenter commented Apr 29, 2026 •

edited

Loading

Uh oh!

pesap Apr 30, 2026

Uh oh!

pesap Apr 30, 2026

Uh oh!

pesap left a comment

Uh oh!

Bobby-Heyer commented May 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

-    assert {row[3] for row in rows} == {"row1.csv", "row2.csv"}
+    assert {row[3] for row in rows} == {"row1.csv", "row2.csv"}
+def test_add_properties_from_records_duplicate_rows_after_skipped_row_preserve_links(
+    db_instance_with_schema: PlexosDB,
+) -> None:
+    from plexosdb import ClassEnum, CollectionEnum
+    db = db_instance_with_schema
+    db.add_object(ClassEnum.Generator, "GEN_MIX")
+    records = [
+        {
+            "name": "MISSING_OBJECT",
+            "property": "Max Capacity",
+            "value": 0.0,
+            "band": 9,
+            "datafile_text": "skip.csv",
+        },
+        {
+            "name": "GEN_MIX",
+            "property": "Max Capacity",
+            "value": 0.0,
+            "band": 1,
+            "datafile_text": "row1.csv",
+        },
+        {
+            "name": "GEN_MIX",
+            "property": "Max Capacity",
+            "value": 0.0,
+            "band": 2,
+            "datafile_text": "row2.csv",
+        },
+    ]
+    db.add_properties_from_records(
+        records,
+        object_class=ClassEnum.Generator,
+        parent_class=ClassEnum.System,
+        collection=CollectionEnum.Generators,
+        scenario="ScenarioMix",
+    )
+    rows = db.query(
+        """
+        SELECT d.data_id, b.band_id
+        FROM t_data d
+        LEFT JOIN t_band b ON b.data_id = d.data_id
+        JOIN t_membership m ON d.membership_id = m.membership_id
+        JOIN t_object o ON m.child_object_id = o.object_id
+        JOIN t_property p ON d.property_id = p.property_id
+        WHERE o.name = ? AND p.name = ?
+        ORDER BY d.data_id
+        """,
+        ("GEN_MIX", "Max Capacity"),
+    )
+    assert len(rows) == 2
+    assert {row[1] for row in rows} == {1, 2}

Conversation

mcllerena commented Apr 29, 2026

Uh oh!

codecov-commenter commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pesap Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

pesap Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

pesap left a comment

Choose a reason for hiding this comment

Uh oh!

Bobby-Heyer commented May 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Apr 29, 2026 •

edited

Loading