Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
152 commits
Select commit Hold shift + click to select a range
7577f6f
Init local resolver
jecisc Jun 25, 2026
c9e9201
Manage multiple assignments to the same variable
jecisc Jun 25, 2026
44ab647
Rename test
jecisc Jun 29, 2026
ad72c3f
Set tree sitter version to the last release
jecisc Jun 29, 2026
41ddf78
Merge branch 'main' into local-resolver
jecisc Jun 29, 2026
92f930b
Declaration should count in local uses + add skipped tests
jecisc Jun 29, 2026
9fdf72f
Fix wrong parent name for entities in left and right side of a for st…
jecisc Jun 29, 2026
79ff1e2
Merge branch 'main' into local-resolver
jecisc Jun 29, 2026
4233076
Manage for statement left and right
jecisc Jun 29, 2026
2a25f4b
Start to manage local variables
jecisc Jun 30, 2026
4836c42
Implement a visitor specific to python because it is too different fr…
jecisc Jun 30, 2026
73f5979
Add tests on edge cases of loops
jecisc Jun 30, 2026
169ce55
Manage variables declared in while or for
jecisc Jul 3, 2026
6792292
Mannage globals accessed in lambdas
jecisc Jul 15, 2026
4b8ab0a
Manage lambda parameters
jecisc Jul 15, 2026
6790e5e
Begin to manage function parameters
jecisc Jul 15, 2026
34761a3
Manage collection initializers
jecisc Jul 15, 2026
a1200fd
Get further in the analysis of the notebooks by ignoring variables no…
jecisc Jul 16, 2026
6f57fed
Add doc
jecisc Jul 16, 2026
7e3e75f
Manage unbinding of variables if an import alias has the same name
jecisc Jul 16, 2026
3ee2a2a
Refactoring to support more resolution than just variables
jecisc Jul 17, 2026
3f0981f
Commit some tests I'll need to rework
jecisc Jul 17, 2026
ed832f6
Begin to manage resolution of imports
jecisc Jul 17, 2026
16d8d1c
Manage aliases of imports
jecisc Jul 17, 2026
9f65882
Start shadowing management
jecisc Jul 17, 2026
b7a5153
Ensure from imports are working as imports
jecisc Jul 20, 2026
ae685e1
Update documentation
jecisc Jul 20, 2026
1dced69
Support shadowing of parameters
jecisc Jul 20, 2026
f72a1d6
Fix import shadowed by other imports
jecisc Jul 20, 2026
63e2aa7
Start to manage non local declarations
jecisc Jul 20, 2026
7403461
Add doc
jecisc Jul 21, 2026
0eead1e
Begin to manage attribute accesses by creating non local declarations…
jecisc Jul 21, 2026
4536897
Add doc on limitations of the local resolver
jecisc Jul 21, 2026
8f6bbe1
Add testing method on Tuples
jecisc Jul 21, 2026
f2a044c
Ensure we can resolve multiple times a model without clashing
jecisc Jul 21, 2026
5f0ccee
Manage basic assignment in tuples
jecisc Jul 21, 2026
fa6996d
Simplify scoping because it does not seems I need it for now and I pr…
jecisc Jul 21, 2026
6103fbf
Begin to manage comprehensions
jecisc Jul 21, 2026
2d1db57
Manage scoping of list comprehensions
jecisc Jul 21, 2026
df60c21
Add a method to know if a local declaration is a variable assignment.…
jecisc Jul 29, 2026
082b78b
Group some classes in the same package tag
jecisc Jul 29, 2026
7492ac6
Add API for SSA building (This will probably move to FAST)
jecisc Jul 29, 2026
6e5f51e
Init SSA Visitor
jecisc Jul 29, 2026
0c477d2
Init tests for SSA
jecisc Jul 29, 2026
41c1ed8
Move some methods to FAST + rearrange tags
jecisc Jul 29, 2026
12431b7
Start to manage usage of assigned variables when there is no condition
jecisc Jul 29, 2026
36ec065
Manage undeclared var usage
jecisc Jul 29, 2026
8f0ee70
Avoid names that can be confusing
jecisc Jul 29, 2026
099637d
Fix bug causing too many creation of SSAVariable
jecisc Jul 29, 2026
c230c7c
Begin to manage parameters
jecisc Jul 29, 2026
8ddacd7
Add tests to ensure parameters knows they are declarations
jecisc Jul 30, 2026
39ba4e0
Fix bug in local resolutions of method parameters
jecisc Jul 30, 2026
9af81bf
Add rule to avoid inconsistant classification violations since I cate…
jecisc Jul 30, 2026
bf88294
Fix bug in method redefining another method
jecisc Jul 30, 2026
27f7ed5
Add test ensuring the variable of a for can be redeclared in the loca…
jecisc Jul 30, 2026
d4c1efd
Cleanup dead code
jecisc Jul 30, 2026
170add2
Variables declared in for should know they are a variable assignment
jecisc Jul 30, 2026
56fe964
SSA: Fixing declaration of variables in for
jecisc Jul 30, 2026
a50bce2
Rename method to make it more generic between LR and SSA
jecisc Jul 30, 2026
491bcf7
Improve implementation of #isVariableWriteAccess
jecisc Jul 30, 2026
79f1293
LR: Manage tuple declaration in the left field of foreach
jecisc Jul 30, 2026
0f30e13
Make identifier a TNamedEntity
jecisc Jul 30, 2026
03a534a
Simplify code now that identifiers are named entities
jecisc Jul 30, 2026
44d3f26
Update LocalResolver to make it easier to implement the SSA with Python
jecisc Jul 30, 2026
228eb50
Clean dead code
jecisc Jul 30, 2026
aa675c3
SSA: Begin to handle Tuples in for declaration
jecisc Jul 30, 2026
333f666
Test local variables with tuples
jecisc Jul 31, 2026
f8c15a1
LR: Manage assignment of tuples in tuples
jecisc Jul 31, 2026
f679720
Add tests on SSA of tuples assignments and tuples in tuples assignments
jecisc Jul 31, 2026
396a200
Remove dead code
jecisc Jul 31, 2026
f33275c
Start to manage read accesses in SSA
jecisc Jul 31, 2026
73b1b63
Test reassignment of variables
jecisc Aug 3, 2026
61eb236
Begin to manage phi versions (WIP)
jecisc Aug 3, 2026
f8a2479
Add broken code to save it while I work on another branch
jecisc Aug 4, 2026
2dd40e5
Support merge block in CFG
jecisc Aug 4, 2026
42c307a
Merge branch 'cfg' into local-resolver
jecisc Aug 4, 2026
edf51ed
Make new test pass by using the new CFG visit logic
jecisc Aug 5, 2026
62934f1
Improve the logic of phi versions to work when we do not reassign a v…
jecisc Aug 5, 2026
7de8122
Add some more tests on if and phi versions
jecisc Aug 5, 2026
3ca992d
Do not create a phi version if we have only one variable as a choice
jecisc Aug 5, 2026
1eedf2e
Ensure multiple phi are working + add more tests
jecisc Aug 5, 2026
5cb628a
Add tests on match statements
jecisc Aug 5, 2026
f814ae2
SSA: Add tests on for each
jecisc Aug 5, 2026
53311e6
Fix bug in Python CFG
jecisc Aug 5, 2026
e57df04
Simplify tests + start to fix some to match the new for handling (CFG)
jecisc Aug 5, 2026
2aa103c
Fix CFG tests after fixing a bug un for management
jecisc Aug 6, 2026
6e1638b
Unskip test that now passes
jecisc Aug 6, 2026
0137be6
LR: Begin to manage attribute access
jecisc Aug 6, 2026
6669a26
Simplify code by removing lines that cannot happen
jecisc Aug 6, 2026
443a990
Manage SSA write access to attributes (WIP)
jecisc Aug 6, 2026
cdadf14
Finish the local resolution of attribute accesses (I think?)
jecisc Aug 6, 2026
5bb3d5b
SSA: Manage read access of attribute accesses
jecisc Aug 6, 2026
f77c8d3
CFG: fix bug in gandling of lambdas
jecisc Aug 6, 2026
b6d5986
SSA: Test lambda parameters
jecisc Aug 6, 2026
879b86a
Fix SSA for method parameters
jecisc Aug 6, 2026
b00ab8f
LR: fix reassignment of parameters in functions
jecisc Aug 6, 2026
e4dd94e
SSA: Add visit of the CFG in definitions + fix a test
jecisc Aug 6, 2026
39554ec
Manage subscripts in LR (only if the indexes are exactly the same sou…
jecisc Aug 7, 2026
7fc901b
Simplify visitors by adding TCanBeVariable
jecisc Aug 7, 2026
eefe218
SSA: test subscript access and reference in default value of parameters
jecisc Aug 7, 2026
dcb9281
SSA: Ensure separators are in the parameters
jecisc Aug 7, 2026
c6b2963
SSA: Ensure typed parameters are working
jecisc Aug 7, 2026
e43172d
SSA: Manage splat parameters (*args or **kwargs as function parameters)
jecisc Aug 7, 2026
9c73a2e
SSA: Ensure phi version only gets last version of the branch
jecisc Aug 7, 2026
2cc4ef6
LR: Simplify the code by inlining the scopes
jecisc Aug 7, 2026
1f240e5
SSA: Rename var to be more explicit
jecisc Aug 7, 2026
c67ebc2
Ensure identifier and attributes do not clash
jecisc Aug 10, 2026
7000912
Assignment in assignment
jecisc Aug 10, 2026
2fe44b8
LR: Support assignment to lists
jecisc Aug 10, 2026
c9b0fb6
SSA: Support assignment to lists
jecisc Aug 10, 2026
b949b23
Ignore DS_STORE file on mac
jecisc Aug 10, 2026
b97f421
Start to add info on instance variables
jecisc Aug 11, 2026
eee18a2
Allow to know if an ivar is a write or read access
jecisc Aug 11, 2026
2cafff3
Add #isAbstract to methods
jecisc Aug 11, 2026
e7a70e7
Factorize code
jecisc Aug 11, 2026
aa080f8
Move some methods in the right location + add metadescriptions
jecisc Aug 12, 2026
2c76e47
Add more doc (WIP)
jecisc Aug 12, 2026
e8a8c9d
Add utility methods forn parsing and resolving in one step
jecisc Aug 12, 2026
95f4c13
Update documentation
jecisc Aug 12, 2026
2c7c1d8
Add test
jecisc Aug 12, 2026
3f92180
Fix shadowing of attribute accesses
jecisc Aug 13, 2026
d6ec1fd
Add test on new query API of FAST
jecisc Aug 13, 2026
be5642a
Update documentation
jecisc Aug 13, 2026
fda1137
Begin to implement the assigned expressions
jecisc Aug 13, 2026
7dcf0f7
Test assigned expressions map (basic test)
jecisc Aug 14, 2026
ece306f
Ensure the assigned variable map works on loops
jecisc Aug 14, 2026
99fe668
Update documentation
jecisc Aug 14, 2026
5b3929a
FIxes #142
jecisc Aug 14, 2026
783f4f7
Fix bug of impornt shadowing an attribute access global declaration
jecisc Aug 14, 2026
32b0274
Fix skipped tests
jecisc Aug 14, 2026
c3a7478
Finish to test shadowing of attributes
jecisc Aug 17, 2026
b482919
Manages variables used as call receivers
jecisc Aug 17, 2026
629a4a9
Test that imports in function do not shadow variables outside of the …
jecisc Aug 17, 2026
077a8f4
Fix bug in shadowing of variables from functions
jecisc Aug 17, 2026
c91c112
Add support for shadowing information in the LocalResolver
jecisc Aug 17, 2026
daef98b
Manage linking of SSA version even if the variable got shadowed in th…
jecisc Aug 17, 2026
75cdd4b
Add agents file
jecisc Aug 20, 2026
5a1d46d
Fix tests
jecisc Aug 20, 2026
9625e31
LR/SSA: Manage augmented assignments
jecisc Aug 20, 2026
609152b
LR/SSA: Test else clause in for loop
jecisc Aug 20, 2026
73a5223
Ensure nested for loops are working
jecisc Aug 20, 2026
91076bb
LR/SSA: Test reassignemnts in while loop
jecisc Aug 20, 2026
c69b5ba
LR/SSA: Fix bug when a while has an else block
jecisc Aug 20, 2026
a663972
SSA: Test redefinition of var in a for loop (should produce a phi ver…
jecisc Aug 20, 2026
e1a516b
LR/SSA: Fix bug in nested ifs + add tests
jecisc Aug 20, 2026
34422bc
Importer: Fix list splat in left side of assignment + test
jecisc Aug 21, 2026
4500e8b
LR/SSA: Add tests on elif (I mostly did if/then/else until now)
jecisc Aug 21, 2026
9c18280
LR/SSA: Begin to manage ternary epressions but it's a WIP
jecisc Aug 21, 2026
fae596c
LR/SSA: Tests assignemnts in try/catch
jecisc Aug 21, 2026
c999cbd
LR/SSA: Manage except in try
jecisc Aug 21, 2026
b3ab0be
LR/SSA: Fix problem with assignemnt to an unpacking
jecisc Aug 21, 2026
8a02917
Merge branch 'main' into local-resolver
jecisc Aug 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,5 @@ SqueakDebug.log
# Metacello-github cache
/github-cache
github-*.zip

**/.DS_STORE
90 changes: 90 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# AGENTS.md

Always read and follow the coding conventions in `~/.opencode/AGENTS.md`.

## What This Is

A **Pharo Smalltalk** project (not Python) that provides a Famix AST representation for Python code, built on TreeSitter. Part of the [moosetechnology/FAST](https://github.com/moosetechnology/FAST) ecosystem.

Source code format: **Tonel** (all source in `src/`).

## Build & Test

This is a Pharo Smalltalk project. There are no npm/pip/make commands.

**Install dependencies in a Pharo image:**
```smalltalk
Metacello new
githubUser: 'moosetechnology' project: 'FAST-Python' commitish: 'main' path: 'src';
baseline: 'FASTPython';
load
```

**CI command (what actually runs):**
```bash
smalltalkci -s Pharo64-13 # or Pharo64-14
```
CI is defined in `.github/workflows/tests.yml`. Tests run via `smalltalkci` using the `smalltalk.ston` spec. Coverage is collected in lcov format for `FAST-Python.*` packages.

There is no local test runner script outside of `smalltalkci`. You need a Pharo image with the project loaded to run tests interactively.

## Package Structure

```
src/
BaselineOfFASTPython/ -- Metacello baseline (dependency & group definitions)
FAST-Python-Model-Generator/ -- Metamodel generator (Famix code generation)
FAST-Python-Model/ -- Generated model classes (FASTPy*), do NOT edit by hand
FAST-Python-Model-Tests/ -- Model-level tests
FAST-Python-Tools/ -- Importer, CFG, local resolver, SSA, TreeSitter visitor
FAST-Python-Tools-Tests/ -- Tests for tools (importer, CFG, resolver, SSA)
```

**Baseline groups:**
- `Core` = Model + Tools
- `Generator` = Model-Generator only
- `Tests` = Model-Tests + Tools-Tests

**Dependencies** (from baseline):
- `FAST` v3: `github://moosetechnology/FAST:v3/src` (loads `'All'` group)
- `TreeSitter` v2.0.0: `github://Evref-BL/Pharo-Tree-Sitter:v.2.0.0/src`

## Critical: Model Files Are Generated

`FAST-Python-Model/` contains ~130 generated class files (`FASTPy*`). These are produced by `FASTPythonMetamodelGenerator` in `FAST-Python-Model-Generator/`. **Do not edit model files directly** -- edit the generator and re-run it. The generator also produces a visitor trait (`FASTPyTVisitor`).

## Key Entry Points

| Class | Role |
|-------|------|
| `FASTPythonImporter` | Parse Python source string or file → FAST model. Extends `TSFASTAbstractImporter`. |
| `FASTPythonTreeSitterVisitor` | Walks TreeSitter CST → builds FAST model. 1100+ lines, the core import logic. |
| `FASTPythonCFGVisitor` | Builds control flow graph. Uses `FASTTCFGUtility` trait from FAST. |
| `FASTPythonLocalResolverVisitor` | Links entity usages to their local declarations (scope-aware). |
| `FASTPythonSSAVisitor` | SSA transform. Uses `FASTCFGTVisitor` trait from FAST. Requires local resolution first. |
| `FASTPythonMetamodelGenerator` | Generates the Famix metamodel. Run via `FASTPythonMetamodelGenerator new generate`. |

**Typical analysis pipeline:**
```smalltalk
model := FASTPythonImporter parseFile: aFile.
FASTPythonLocalResolverVisitor resolve: model module.
model allFunctionDefinitions first cfg. "CFG"
FASTPythonSSAVisitor resolve: model allFunctionDefinitions first. "SSA (after resolution)"
```

## Testing

- **Base test class**: `FASTPythonAbstractTestCase` (in Tools-Tests) -- provides a `parse:` helper that wraps `FASTPythonImporter` with error reporting.
- **Importer tests**: `FASTPythonImporterTest` extends `TSFASTAbstractImporterTest` (from TreeSitter). The test file is ~53k lines -- generated tests for every AST node type. Test generation snippet is in the class comment.
- **CFG/Resolver/SSA tests**: `FASTPythonCFGTest`, `FASTPythonLocalResolverTest`, `FASTPythonSSATest`.

## Gotchas

- **TreeSitter python grammar version matters**: After v0.25, expression statements were removed from the tree. If tests fail unexpectedly, check your `tree-sitter-python` version first (see README).
- **Pharo 13 is the primary target**; Pharo 14 is also tested. The baseline references `FAST:v3` which requires Moose 13.
- **The importer uses the master branch** of `TSLibrariesPython` (hardcoded in `FASTPythonTreeSitterVisitor class >> initialize`).
- **`expression --|> statement`**: In Python, expressions can be expression statements. This is reflected in the model hierarchy and affects how the visitor processes nodes.
- **CFG uses `FASTTCFGUtility`** from FAST (the parent project). SSA uses `FASTCFGTVisitor`. These traits define the core graph traversal protocol.
- **Local resolver scope management**: The resolver maintains a `scopes` stack. Shadowing is handled by checking entity kind consistency (same kind = shared declaration, different kind = new declaration).
- **Python 3 only for resolver**: The local resolver is Python 3 scoped. Python 2 comprehension scoping (no scope) is not supported (issue #26).
- **Non-local declarations are not deduplicated**: Each access to an unknown field creates a separate `FASTNonLocalDeclaration` (issue #26 in the repo).
47 changes: 47 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,22 @@

Famix AST representation for Python based on TreeSitter

<!-- TOC -->

- [FAST-Python](#fast-python)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Documentation](#documentation)
- [Control flow graph](#control-flow-graph)
- [Local resolutions](#local-resolutions)
- [Single Static Assignment (SSA)](#single-static-assignment-ssa)
- [Moose versions compatibility](#moose-versions-compatibility)
- [TreeSitter python version compatibility](#treesitter-python-version-compatibility)
- [Python version compatibility](#python-version-compatibility)
- [Contact](#contact)

<!-- /TOC -->

## Installation

To install this project on your Pharo image, execute the following script:
Expand Down Expand Up @@ -47,6 +63,8 @@ Or

The best documentation to read about this project is located in Pharo Tree Sitter's repository here: [https://github.com/Evref-BL/Pharo-Tree-Sitter/blob/main/resources/doc/fast_importer.md](https://github.com/Evref-BL/Pharo-Tree-Sitter/blob/main/resources/doc/fast_importer.md) and here: [https://github.com/Evref-BL/Pharo-Tree-Sitter/blob/main/resources/doc/ts_utilities.md](https://github.com/Evref-BL/Pharo-Tree-Sitter/blob/main/resources/doc/ts_utilities.md)

You can find the documentation on some analysis possible here: [Analysis documentation](resources/doc/analysis.md).


## Control flow graph

Expand All @@ -64,6 +82,35 @@ A CFG can be done on a function, method, class, module or lambda.

You can visualize it in the inspector as a visualization and you can also export your CFG as a mermaid visualization using `#asMermaidScript`

For more information check the analysis documentation linked above.

## Local resolutions

It is possible to apply a local name resolution on a FAST-Python model by executing this piece of code:

```smalltalk
FASTPythonLocalResolverVisitor resolve: model module "Can be any behavioral entity"
```

Once this is executed, all nodes resolved will be able to provide a `#localDeclaration` pointing to the declaration/first use of the entity. This local declaration also knows all the `#localUses` of the entity.

For more information check the analysis documentation linked above.

## Single Static Assignment (SSA)

It is possible to build the SSA of a FAST-Python model like this:


It can be run like this:

```smalltalk
FASTPythonSSAVisitor resolve: model module
```

Then we can ask to any node representing a variable access `#ssaVersion`. This version is able to tell us where are the read and write accesses of this particular version of the variable.

For more information check the analysis documentation linked above.

## Moose versions compatibility

| Version | Compatible Moose versions |
Expand Down
Loading