Skip to content

Disk Usage Duplication #1015

Description

@HoracioImagen

Hello.

I hope you'll see this as an improvement. I've found a problem, more of a bug than a bug, and I don't think I'm the only one...

In my case, I have very little disk space, only 512 GB, of which I have around 100 GB available. As you can imagine, I'm constantly trying to delete unnecessary files and applications. Using large file search tools, I've noticed that the downloaded AI models are "duplicated." I found the AI ​​model, which in my case is a small 3 GB one, and there's another one of the same size (3 GB). When I delete the model, both are deleted.

The files are located primarily in this path (I'm a Windows user):
%USERPROFILE%.docker\models\

From this path, the folders that matter in this case are "blobs" and "bundles." Both contain the "sha256" folder, and the large files are located within it:

  • Path: %USERPROFILE%.docker\models\blobs\sha256\

  • File: b8906b8c5e05e57b657646bbc657bd35814a269b2c20f0a2579047fafa1a67dd

  • Path: %USERPROFILE%.docker\models\bundles\sha256\2181595ff93ee2e979ccba03ab07c02ecbfaa31db978df52b429cc1e78fb83cc\model\

  • File: gemma-4-E2B-it-UD-Q4_K_XL.gguf

I don't know if it's a bug, intentional, or if it provides some benefit I'm missing, but with the current prices of hard drives and RAM, it's a problem, at least for me, since I don't have the budget and storage space is limited.

I asked Gordon, its AI, if I could delete any of those files, and it said no, explaining that one is the AI ​​model and the other is the uncompressed hash, and it's used to verify that the same model isn't downloaded again...

That's surprising, because Actually, it's not 3 GB being downloaded, but 6 GB. For me, that's too much, especially if you want to download a larger model.

If what Gordon says is true, why not use another mechanism to identify that the AI ​​model has already been downloaded, like Docker images? Yes, I also asked him that, and he said it's because the models don't use layers like Docker images, but I wasn't referring to that. I meant the use of tags, a smaller hash, or another methodology.

NOTE: When downloading the model, the two files aren't generated; instead, when using the downloaded model, the .gguf file is generated.

Oh! Also, in the list of AI models, when another model is being downloaded, the model being downloaded disappears; it's just not listed. It continues downloading in the background (apparently), but the download tracking is lost.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions