Add parsed files and read them...

[rocky]

Add DumpParse to read Mathics3 source text and write it Python pickled to a file. Extend Get so that it will unpickle a file.

[rocky]

This might be one way to run source text programs that have copyrighted source text.

However, I see that many copyrighted source texts are distributed under an MIT license.

Still, this may lead to a convenient way to bundle several files making up a package into one file.

[rocky]

• Give an error message if the Pickle'd filename does not end in .mx3
• Allow optional .mx3 file? Use the basename of input Mathics3 file for automatic output file creation?
• DRY Get and DumpParse code?
• Add magic number in mx3 file, and version information.

[mmatera]

@rocky, this looks very nice. The only observation is that Pickle may not be the best choice for distributing binary packages: because they can contain active, malicious code, they are not safe; the latest version of the protocol can change among versions; and also, if Mathics3 changes the API, M-expression objects stored in these files could become invalid. I wonder if something like a zipped JSON file with expressions in plain string format would not be a better option.

[rocky]

@rocky, this looks very nice. The only observation is that Pickle may not be the best choice for distributing binary packages: because they can contain active, malicious code, they are not safe; the latest version of the protocol can change among versions; and also, if Mathics3 changes the API, M-expression objects stored in these files could become invalid. I wonder if something like a zipped JSON file with expressions in plain string format would not be a better option.

Ok. We should keep this in mind and benchmark performance.

[rocky]

@rocky, this looks very nice. The only observation is that Pickle may not be the best choice for distributing binary packages: because they can contain active, malicious code, they are not safe; the latest version of the protocol can change among versions; and also, if Mathics3 changes the API, M-expression objects stored in these files could become invalid. I wonder if something like a zipped JSON file with expressions in plain string format would not be a better option.

Ok. We should keep this in mind and benchmark performance.

The more I think about this, the more I think this isn't so much about Pickle, but about the distribution mechanism, binary or not.

Any time "Get" is used in Wolfram Language or Mathics3, that's just as unsafe. When was the last time you actually looked in depth for Get or a Python package, or an OS package for vulnerabilities?

I am sure most people don't. Instead, they rely on a check from the installation mechanism, and the validity of the organization that collects packages.

Typically, packages have a hash associated with them, e.g., MD5SUM or SHA1. Package installers use that and warn if a hash changes. And an organization has procedures to validate and take responsibility for the packages it distributes.

Needless to say, all of this stuff hasn't been fleshed out yet.

Mechanisms for validating a package are an issue. It's not something that necessarily is going to be fixed by just using JSON instead of Pickle. One can embed malicious code in JSON.