diff --git a/.agents/skills/localize-rustfs-docs/SKILL.md b/.agents/skills/localize-rustfs-docs/SKILL.md index 53f4c3f9..76fea008 100644 --- a/.agents/skills/localize-rustfs-docs/SKILL.md +++ b/.agents/skills/localize-rustfs-docs/SKILL.md @@ -1,74 +1,182 @@ --- name: localize-rustfs-docs -description: Localize, add, review, or repair RustFS documentation languages and FumaPress internationalization. Use for locale directories, translated Markdown or MDX, localized navigation and UI, terminology decisions, language routing, hreflang metadata, search indexes, redirects, and translation-quality reviews in docs.rustfs.com. +description: Localize, review, audit, or repair RustFS documentation and FumaPress internationalization. Use for translated Markdown or MDX, frontmatter, meta.json navigation, locale directories, localized links and UI, terminology decisions, language routing, hreflang metadata, search indexes, redirects, untranslated-content detection, and localization pull requests in docs.rustfs.com, especially for German, French, Japanese, or Chinese. --- -# Localize RustFS documentation +# Localize and review RustFS documentation -Build each locale as a complete RustFS documentation experience. Preserve product meaning and operational accuracy; do not perform literal sentence-by-sentence substitution. +Produce target-language documentation that is technically equivalent to the current English source, idiomatic for infrastructure developers, structurally valid for FumaPress, and safe to publish. Preserve meaning, operational behavior, warnings, constraints, and claim strength instead of translating sentence by sentence. -## Required context +## Load the required context -1. Read `../rustfs-docs/SKILL.md` and the repository `STYLE.md` before editing content. -2. Read [the current FumaPress internationalization guide](https://press.fumadocs.dev/docs/internationalization.md) before changing `press.config.tsx` or locale layout. -3. Read [references/terminology.md](references/terminology.md) before translating or reviewing terminology. +1. Read `../rustfs-docs/SKILL.md`, the repository `AGENTS.md`, and `STYLE.md`. +2. Read [references/terminology.md](references/terminology.md), then read the target-locale reference when one exists: + - [references/de.md](references/de.md) for German + - [references/fr.md](references/fr.md) for French + - [references/ja.md](references/ja.md) for Japanese +3. Read the [current FumaPress internationalization guide](https://press.fumadocs.dev/docs/internationalization.md) before changing `press.config.tsx`, routing, search, or locale layout. 4. Verify commands, flags, configuration keys, defaults, ports, APIs, and feature claims against `rustfs/rustfs`; never infer them from a translation. -## Translation standard +Use the current English page as the content baseline. For a pull request, compare against the English page at the base commit unless the pull request also proposes a newer English version. -- Translate meaning in the context of the RustFS product, its Console, and S3-compatible behavior. Rewrite sentence structure when necessary for natural target-language documentation. -- Keep `RustFS`, protocol names, API names, CLI commands, code, environment variables, configuration keys, file paths, URLs, JSON/YAML keys, and identifiers unchanged. -- Use the glossary consistently. When a technical term has no established translation, keep the English term and add a short target-language explanation on first use. -- Preserve visible Console labels exactly when instructing the reader to click them; add a translation in parentheses only when it improves comprehension. -- Preserve the source page's technical scope. Do not add capabilities, defaults, warnings, or recommendations during translation unless they are verified and added to every maintained locale. -- Keep code fences byte-for-byte equivalent across locales unless a localized string is itself part of the demonstrated behavior. Translate prose outside code fences instead. -- Preserve links and cited sources. Localize internal documentation links to the current locale while keeping external destinations unchanged. -- Review the completed page as native technical writing. Reject awkward calques, ambiguous pronouns, untranslated prose fragments, terminology drift, and sentences that are grammatically correct but unnatural. +## Select the operation mode -## Configure FumaPress i18n +- **Audit**: report prioritized findings without modifying files. +- **Repair**: correct existing translations and validate the affected locales. +- **Translate**: create missing locale counterparts from the current English source. +- **Integrate**: update navigation, routing, search, language metadata, or repository-owned UI strings. -1. Define all locales with `defineI18n` and choose the parser explicitly. This repository uses locale directories, so keep `parser: "dir"`. -2. Pass the resulting translations API to `defineConfig`. -3. Use an official `@fumapress/language` preset when available. A preset localizes Fumadocs UI and FumaPress strings; setting only `displayName` is insufficient. -4. Localize repository-owned navigation, footer text, labels, and calls to action separately because language packs cannot translate hard-coded strings. -5. Keep `content//` trees structurally aligned. Every maintained page must have a target-locale counterpart or an explicitly documented fallback decision. -6. Emit the correct ``, canonical URL, and `hreflang` alternates for every page. -7. Verify that search output is partitioned by locale and that the language switcher retains the corresponding page path. +Infer the mode from the request. Default to audit when the user asks only to check or review content. Do not hardcode a pull request, branch, locale, or file list in this skill. -## Content and navigation workflow +## Establish scope and coverage -1. Start from the current default-language page, not from an older translation. -2. Identify RustFS-specific terms and runtime claims before translating. -3. Translate headings and prose for reader intent, then reconcile terminology against the glossary and nearby translated pages. -4. Copy or share required images without changing the English, light-theme screenshot policy from `rustfs-docs`. -5. Update the target locale's `meta.json`. Use bare entries for pages inside the same folder; use locale-prefixed URLs for cross-folder navigation entries. -6. Check that frontmatter, heading levels, admonitions, links, images, code-fence languages, and MDX component structure match the source page. -7. Review the rendered page, including sidebar, breadcrumbs, table of contents, previous/next links, search, and custom navbar/footer text. +Map each target file to its English counterpart: -## Routing and deployment +```text +content// +content/en/ +``` + +Include `.md`, `.mdx`, `meta.json`, locale-aware configuration, repository-owned UI strings, internal links, redirects, search, and language metadata. Ignore copied binary images during linguistic review, but verify paths and avoid unnecessary binary duplication. + +For large changes, run exhaustive structural checks before semantic review. Review content in traceable batches and record exactly which files and locales were covered. Never claim full review after sampling only part of a change. + +Run the bundled audit before and after broad locale work: + +```bash +node .agents/skills/localize-rustfs-docs/scripts/audit-locales.mjs --locales de,fr,ja +``` + +Use `--strict` when warnings must fail the command and `--json` for machine-readable output. + +## Preserve technical truth + +Keep the translation semantically equivalent to the source. Preserve: + +- commands, flags, API operations, and configuration behavior; +- environment variables, identifiers, filenames, paths, ports, versions, regions, units, and numeric limits; +- prerequisites, conditions, exceptions, warnings, negation, and modal strength; +- feature status, compatibility scope, security guidance, and operational consequences. + +Distinguish `must`, `should`, `may`, and `can`; `supported`, `compatible`, `experimental`, and `unavailable`; local files and S3 Objects; Bucket replication and site replication; and replication, erasure coding, healing, rebalancing, and decommissioning. + +Never strengthen “S3-compatible” into “fully compatible” or “100% compatible”. Never turn “alternative to MinIO” into “drop-in replacement for MinIO” unless the English source and verified behavior support that claim. + +Report an English-source defect instead of silently correcting the fact in only one locale. + +## Protect non-translatable content + +Preserve these exactly unless a repository rule explicitly requires a locale-specific change: + +- fenced code contents, languages, and metadata; +- inline-code tokens; +- CLI commands, API actions, HTTP methods, headers, status codes, and protocol tokens; +- environment variables, configuration keys, JSON/YAML keys, metrics, and identifiers; +- class, function, type, field, package, and Kubernetes resource names; +- filenames, paths, domains, image tags, external URLs, and explicit anchors; +- MDX component names, imports, JavaScript expressions, and structural attributes. + +Do not reformat protected content for style. Preserve visible Console labels exactly when instructing the reader to select them; add a target-language explanation on first use only when it helps the reader. + +## Preserve document structure + +### Frontmatter + +Preserve delimiters, key names, key order where practical, and non-user-facing values. Translate user-facing values such as `title` and `description`. Keep `description` a complete sentence and do not introduce new claims. + +### Markdown + +Preserve heading hierarchy, list nesting, numbering, table dimensions, admonition types, footnote identifiers, code fences, explicit anchors, emphasis, and inline-code boundaries. Translate prose, headings, link labels, image alt text, table prose, and admonition titles. Do not add a body H1 when frontmatter renders the title. + +### Links + +Translate visible labels, preserve external destinations, and point internal documentation links to the corresponding target locale. Preserve relative links where practical. Verify fragments after translating headings because generated slugs can change. Preserve explicit anchors unless every reference is deliberately updated. -- Keep locale prefixes explicit (`/en/...`, `/zh/...`) while `hideLocale: "never"` is configured. -- Redirect `/` to the default locale and preserve every historical unprefixed URL by redirecting it to the equivalent default-locale URL. -- Treat hosting configuration as part of i18n. `_redirects` is used by Cloudflare-style static hosting; Vercel requires `vercel.json`, `vercel.ts`, or a configured bulk redirects file. -- Never delete a legacy redirect merely because the destination gained a locale prefix. Prefix its destination and retain the original source. -- Smoke-test redirects against the deployed preview. A successful build or deployment status does not prove that redirects work. +### MDX and HTML -## Validation +Preserve component and tag names, nesting, imports, expressions, and structural attributes. Translate visible children and user-facing properties such as `title`, `description`, `alt`, and `aria-label`. Keep `id`, `value`, `name`, `className`, and `aria-hidden` unchanged, except for required locale changes in internal `href` values. -Run all repository checks: +### `meta.json` + +Preserve valid JSON, keys, array order, page identifiers, and navigation structure. Translate visible section titles and Markdown link labels. Change only the locale segment of internal locale-prefixed URLs, and ensure every entry resolves within that locale. + +### Images + +Keep screenshots in English when required by `STYLE.md`. Translate captions and alt text. Do not edit image files merely to localize visible Console labels. + +## Apply terminology consistently + +Choose terms in this order: + +1. RustFS product names, source code, Console labels, APIs, and protocol identifiers. +2. The approved RustFS glossary for the target locale. +3. Consistent usage in nearby target-language RustFS pages. +4. Official localized terminology from Amazon S3, AWS IAM, Kubernetes, and the applicable ecosystem. +5. Established target-language infrastructure usage. +6. English with a concise target-language explanation when no stable translation exists. + +Do not preserve every S3-related noun in English indiscriminately. Preserve exact English for product names, API identifiers, SDK fields, commands, and visible UI labels. Use approved localized terms in explanatory prose. + +Protected identifiers include `CreateBucket`, `ListObjectsV2`, `MultipartUpload`, `AccessKeyId`, `SecretAccessKey`, `forcePathStyle`, `RUSTFS_ACCESS_KEY`, and `RUSTFS_SECRET_KEY`. + +Record a new canonical decision in the target-locale reference instead of allowing several translations for the same concept. + +## Review in priority order + +1. **Completeness**: detect untranslated pages, fragments, frontmatter, labels, omissions, and duplicated content. +2. **Technical accuracy**: compare commands, identifiers, values, conditions, warnings, behavior, negation, modality, compatibility scope, and feature status. +3. **Structural integrity**: compare frontmatter, headings, lists, tables, admonitions, links, images, code fences, MDX, and navigation. +4. **Terminology**: enforce the target glossary and product capitalization across related pages. +5. **Native-language quality**: reject awkward calques, ambiguous pronouns, unnatural word order, grammar errors, machine-translated phrasing, and casual slang. +6. **Site integration**: validate navigation, language switching, search partitioning, canonical URLs, `hreflang`, redirects, and rendered output. + +## Classify audit findings + +- **P0 — Blocker**: build failure, broken route, unsafe command change, security-critical mistranslation, wrong locale, or substantially untranslated page. +- **P1 — Major**: changed meaning, omitted requirement, incorrect negation or condition, unsupported compatibility claim, wrong operational terminology, or broken internal link. +- **P2 — Normal**: terminology drift, unnatural technical writing, inconsistent UI-label handling, accessibility text issue, or localized navigation defect. +- **P3 — Minor**: punctuation or stylistic polish with no effect on meaning. + +Report actionable findings as: + +```text +[P1] path/to/file:line — Short title +Problem: ... +Source meaning: ... +Recommended correction: ... +``` + +Group repeated mechanical problems by cause and include representative files plus the total affected count. Report locales reviewed, files reviewed, files skipped, automated checks, semantic-review coverage, and remaining uncertainty. + +## Repair and translate narrowly + +1. Make the smallest changes that correct the translation. +2. Keep locale trees structurally aligned and update affected navigation and links. +3. Update the locale terminology reference when establishing a new canonical term. +4. Do not mix unrelated English-source improvements into a localization repair. +5. Review the final diff to ensure protected content did not change. + +When the user supplies one document and explicitly requests “Markdown source only”, return only the corrected Markdown without a surrounding code fence or explanation. Do not apply that output rule to a pull-request audit unless explicitly requested. + +## Validate the result + +Run from the repository root: ```bash +node .agents/skills/localize-rustfs-docs/scripts/audit-locales.mjs --locales de,fr,ja npm run docs:check npm run types:check npm run build ``` -Then verify: +Replace the locale list with the comma-separated locales changed by the task. + +Also verify that locale file sets align, code blocks and identifiers remain protected, `meta.json` files parse, frontmatter contains localized `title` and `description`, target pages contain no unexpected source-language prose, links and fragments resolve, language switching preserves the corresponding page, language metadata is correct, and search results do not mix locales unexpectedly. + +Preview representative pages whenever routing, MDX, navigation, or visible UI text changes. Do not commit `node_modules`, `dist`, generated search indexes, preview output, or temporary translation artifacts. + +## Report completion -- `content/en` and every maintained locale contain the same relative file set. -- Commands and non-localized code blocks remain identical across translated counterparts. -- Every generated internal `href` resolves to a generated page or public asset. -- Target-language pages do not expose unexpected English framework UI strings. -- `/`, representative legacy URLs, and representative locale URLs return the intended status and destination on the deployment preview. +For an audit, return prioritized findings, coverage and limitations, validation performed, and merge readiness. For a repair, translation, or integration, return files changed, important terminology or structural decisions, validation commands and results, and remaining unverified issues. -Do not commit `node_modules/`, `dist/`, preview output, or temporary translation files. +Use the requester’s language for the report. Keep documentation content in its target locale and repository-owned technical identifiers in English. diff --git a/.agents/skills/localize-rustfs-docs/agents/openai.yaml b/.agents/skills/localize-rustfs-docs/agents/openai.yaml index 07b4e31e..24805413 100644 --- a/.agents/skills/localize-rustfs-docs/agents/openai.yaml +++ b/.agents/skills/localize-rustfs-docs/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Localize RustFS Docs" - short_description: "Build and verify localized RustFS documentation" - default_prompt: "Use $localize-rustfs-docs to add or review a documentation locale end to end." + short_description: "Audit and repair localized RustFS documentation" + default_prompt: "Use $localize-rustfs-docs to audit and repair a RustFS documentation locale end to end." diff --git a/.agents/skills/localize-rustfs-docs/references/de.md b/.agents/skills/localize-rustfs-docs/references/de.md new file mode 100644 index 00000000..ec54df1d --- /dev/null +++ b/.agents/skills/localize-rustfs-docs/references/de.md @@ -0,0 +1,70 @@ +# German localization reference + +Use this reference for `content/de`. Prefer natural, precise infrastructure documentation over literal translation. + +## Authoritative sources + +Use current official German documentation to confirm ecosystem terminology: + +- [Amazon S3 User Guide](https://docs.aws.amazon.com/de_de/AmazonS3/latest/userguide/Welcome.html) +- [Amazon S3 uploads](https://docs.aws.amazon.com/de_de/AmazonS3/latest/userguide/upload-objects.html) +- [Amazon S3 Bucket policies](https://docs.aws.amazon.com/de_de/AmazonS3/latest/userguide/bucket-policies.html) +- [Kubernetes documentation](https://kubernetes.io/de/docs/) + +Treat RustFS source code, APIs, and exact Console labels as authoritative for product-specific names. + +## Voice and grammar + +- Address the reader formally with `Sie` when direct address improves clarity. +- Use concise active instructions such as `Führen Sie ... aus` and `Öffnen Sie ...`. +- Apply standard German noun capitalization and form readable technical compounds. +- Use sentence-style capitalization for headings while respecting German noun rules. +- Prefer established German infrastructure usage; retain an English term only when German developers normally use it or no stable German term exists. +- Avoid casual slang, exaggerated claims, and English sentence structure copied into German. +- Keep the exact English Console label in bold when the reader must locate it; add a German explanation in parentheses on first use when useful. + +## Preferred terminology + +| English source | Preferred German prose | Guidance | +|---|---|---| +| RustFS | RustFS | Never alter the product name. | +| Amazon S3 / S3 | Amazon S3 / S3 | Keep the service and protocol name. | +| S3-compatible | S3-kompatibel | Never strengthen to complete or 100% compatibility. | +| object storage | Objektspeicher | Use for the storage model. | +| Bucket | Bucket / S3-Bucket | Follow established German AWS usage; preserve `Bucket` in identifiers and UI labels. | +| Object | Objekt / S3-Objekt | Distinguish it from a local `Datei`. | +| object key | Objektschlüssel | Use `Schlüsselname` when specifically describing the key name; preserve `Key` in APIs. | +| Bucket policy | Bucket-Richtlinie | Preserve policy JSON and API names. | +| access key ID | Zugriffsschlüssel-ID | Preserve `AccessKeyId` and environment variables. | +| secret access key | geheimer Zugriffsschlüssel | Preserve `SecretAccessKey` and placeholders. | +| endpoint | Endpunkt | Qualify as `S3-API-Endpunkt` when ambiguous. | +| Region | Region / AWS-Region | Preserve values such as `us-east-1`. | +| Console | Console | Treat as the RustFS product surface; use `Weboberfläche` only as an explanation. | +| storage pool | Speicherpool | Do not shorten to `Pool` when storage meaning is unclear. | +| erasure coding | Erasure Coding | Explain once as a redundancy scheme when the audience needs it; preserve `EC:n`. | +| healing | Datenwiederherstellung / Healing | Use one term consistently within an operational workflow; preserve exact UI and metric names. | +| rebalancing | Rebalancing / Neuverteilung | Preserve the RustFS operation name when it maps to a Console action. | +| decommission | Außerbetriebnahme | Explain that data is drained before the storage pool is deactivated. | +| replication | Replikation | Qualify as `Bucket-Replikation` or `Site-Replikation`. | +| versioning | Versionierung | Use for S3 Object versions, not source control. | +| Object Lock | Object Lock / Objektsperre | Preserve exact API and Console labels. | +| legal hold | Legal Hold | Preserve API/UI wording; explain the retention effect in German. | +| lifecycle management | Lebenszyklusverwaltung | Use for S3 lifecycle rules. | +| multipart upload | mehrteiliger Upload | Preserve API actions such as `CreateMultipartUpload`. | +| server-side encryption | serverseitige Verschlüsselung | Keep SSE-S3, SSE-KMS, and SSE-C unchanged. | +| presigned URL | vorsignierte URL | Keep URL uppercase and preserve API names. | +| path-style addressing | pfadbasierte Adressierung | Preserve SDK option names. | +| virtual-hosted-style addressing | Adressierung im Virtual-Hosted-Stil | Relate it to `RUSTFS_SERVER_DOMAINS` when relevant. | +| health check | Integritätsprüfung | Preserve endpoint paths and probe names. | +| readiness | Bereitschaft | Preserve `/health/ready` and Kubernetes field names. | +| observability | Observability / Beobachtbarkeit | Prefer the established term used by the surrounding German ecosystem documentation and keep it consistent. | +| rootless | ohne Root-Rechte | Prefer this over a literal calque. | +| root credentials | Root-Anmeldedaten | Distinguish RustFS deployment credentials from the operating-system root account. | + +## Common review failures + +- Reject unchanged English paragraphs, descriptions, headings, and navigation labels. +- Reject `100 % S3-kompatibel`, `vollständig S3-kompatibel`, or an unverified `Drop-in-Ersatz` claim. +- Do not translate API operations, environment variables, UI labels, or command output. +- Do not alternate arbitrarily between `Bucket`, `Speicher-Bucket`, and `Speicherbehälter`. +- Do not translate an S3 Object as `Datei` unless the source explicitly means a local file. diff --git a/.agents/skills/localize-rustfs-docs/references/fr.md b/.agents/skills/localize-rustfs-docs/references/fr.md new file mode 100644 index 00000000..79fff5b5 --- /dev/null +++ b/.agents/skills/localize-rustfs-docs/references/fr.md @@ -0,0 +1,71 @@ +# French localization reference + +Use this reference for `content/fr`. Write professional infrastructure documentation in natural French rather than reproducing English syntax. + +## Authoritative sources + +Use current official French documentation to confirm ecosystem terminology: + +- [Amazon S3 User Guide](https://docs.aws.amazon.com/fr_fr/AmazonS3/latest/userguide/Welcome.html) +- [Amazon S3 multipart uploads](https://docs.aws.amazon.com/fr_fr/AmazonS3/latest/userguide/mpuoverview.html) +- [Amazon S3 Object Lock](https://docs.aws.amazon.com/fr_fr/AmazonS3/latest/userguide/object-lock.html) +- [Kubernetes documentation](https://kubernetes.io/fr/docs/) + +Treat RustFS source code, APIs, and exact Console labels as authoritative for product-specific names. + +## Voice and grammar + +- Address the reader with `vous`; use concise active instructions. +- Prefer natural French clause order and avoid calques from English noun strings. +- Use sentence-style capitalization for headings. +- Apply French punctuation and non-breaking spacing where practical without damaging Markdown, URLs, code, or identifiers. +- Keep grammatical gender and number consistent across headings, prose, tables, and navigation. +- Retain established English ecosystem terms only when they are conventional in French technical writing. +- Avoid casual slang, promotional wording, and unjustified claims of seamless replacement. +- Keep the exact English Console label in bold when the reader must locate it; add a French explanation in parentheses on first use when useful. + +## Preferred terminology + +| English source | Preferred French prose | Guidance | +|---|---|---| +| RustFS | RustFS | Never alter the product name. | +| Amazon S3 / S3 | Amazon S3 / S3 | Keep the service and protocol name. | +| S3-compatible | compatible avec S3 | Never strengthen to complete or 100% compatibility. | +| object storage | stockage objet | Use for the storage model. | +| Bucket | compartiment / compartiment S3 | Preserve `Bucket` in APIs, identifiers, and exact UI labels. | +| Object | objet / objet S3 | Distinguish it from a local `fichier`. | +| object key | clé d’objet | Use `nom de clé` when specifically describing the key name; preserve `Key` in APIs. | +| Bucket policy | politique de compartiment | Preserve policy JSON and API names. | +| access key ID | ID de clé d’accès | Preserve `AccessKeyId` and environment variables. | +| secret access key | clé d’accès secrète | Preserve `SecretAccessKey` and placeholders. | +| endpoint | point de terminaison | Qualify as `point de terminaison de l’API S3` when ambiguous. | +| Region | Région / Région AWS | Preserve values such as `us-east-1`. | +| Console | Console | Treat as the RustFS product surface; use `interface web` only as an explanation. | +| storage pool | pool de stockage | Use consistently; avoid translating it as an application connection pool. | +| erasure coding | codage d’effacement | Explain once when required; preserve `EC:n`. | +| healing | réparation des données | Preserve exact UI, API, and metric names that contain `heal` or `healing`. | +| rebalancing | rééquilibrage | Preserve exact Console action names. | +| decommission | mise hors service | Explain that data is drained before the pool is deactivated. | +| replication | réplication | Qualify as `réplication de compartiment` or `réplication de site`. | +| versioning | gestion des versions | Use for S3 Object versions, not source control. | +| Object Lock | verrouillage d’objet | Preserve exact API and Console labels. | +| legal hold | conservation légale | Preserve API identifiers such as `ObjectLockLegalHold`. | +| lifecycle management | gestion du cycle de vie | Use for S3 lifecycle rules. | +| multipart upload | chargement partitionné | Preserve API actions such as `CreateMultipartUpload`. | +| server-side encryption | chiffrement côté serveur | Keep SSE-S3, SSE-KMS, and SSE-C unchanged. | +| presigned URL | URL présignée | Keep URL uppercase and preserve API names. | +| path-style addressing | adressage de type chemin | Preserve SDK option names. | +| virtual-hosted-style addressing | adressage de type hôte virtuel | Relate it to `RUSTFS_SERVER_DOMAINS` when relevant. | +| health check | contrôle d’intégrité | Preserve endpoint paths and probe names. | +| readiness | état de préparation | Preserve `/health/ready` and Kubernetes field names. | +| observability | observabilité | Keep OpenTelemetry and OTLP unchanged. | +| rootless | sans privilèges root | Prefer this over a literal calque. | +| root credentials | informations d’identification racine | Distinguish RustFS deployment credentials from the operating-system root account. | + +## Common review failures + +- Reject unchanged English paragraphs, descriptions, headings, and navigation labels. +- Reject `compatible à 100 % avec S3`, `entièrement compatible`, or an unverified `remplacement transparent` claim. +- Do not translate API operations, environment variables, UI labels, or command output. +- Do not alternate arbitrarily between `compartiment`, `bucket`, and `conteneur` in explanatory prose. +- Do not translate an S3 Object as `fichier` unless the source explicitly means a local file. diff --git a/.agents/skills/localize-rustfs-docs/references/ja.md b/.agents/skills/localize-rustfs-docs/references/ja.md new file mode 100644 index 00000000..ae5fa255 --- /dev/null +++ b/.agents/skills/localize-rustfs-docs/references/ja.md @@ -0,0 +1,71 @@ +# Japanese localization reference + +Use this reference for `content/ja`. Write natural Japanese technical documentation instead of preserving English sentence structure. + +## Authoritative sources + +Use current official Japanese documentation to confirm ecosystem terminology: + +- [Amazon S3 User Guide](https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/Welcome.html) +- [Amazon S3 bucket restrictions](https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/BucketRestrictions.html) +- [Amazon S3 Object Lock](https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/object-lock-managing.html) +- [Kubernetes documentation](https://kubernetes.io/ja/docs/) + +Treat RustFS source code, APIs, and exact Console labels as authoritative for product-specific names. + +## Voice and grammar + +- Use consistent professional `です・ます` style for explanatory prose and instructions. +- Omit unnecessary subjects and second-person pronouns when Japanese is clearer without them. +- Reorder clauses for natural Japanese; do not mirror English word order. +- Use Japanese punctuation in prose while preserving ASCII punctuation in code, URLs, identifiers, and syntax. +- Use established AWS and infrastructure katakana terminology, but avoid unnecessary katakana when a standard Japanese technical term is clearer. +- Avoid repeatedly ending adjacent steps with `してください`; prefer direct, concise action descriptions. +- Avoid casual slang, promotional wording, and unjustified claims of seamless replacement. +- Keep the exact English Console label in bold when the reader must locate it; add a Japanese explanation in parentheses on first use when useful. + +## Preferred terminology + +| English source | Preferred Japanese prose | Guidance | +|---|---|---| +| RustFS | RustFS | Never alter the product name. | +| Amazon S3 / S3 | Amazon S3 / S3 | Keep the service and protocol name. | +| S3-compatible | S3 互換 | Never strengthen to complete or 100% compatibility. | +| object storage | オブジェクトストレージ | Use for the storage model. | +| Bucket | バケット / S3 バケット | Preserve `Bucket` in APIs, identifiers, and exact UI labels. | +| Object | オブジェクト / S3 オブジェクト | Distinguish it from a local `ファイル`. | +| object key | オブジェクトキー | Use `キー名` when specifically describing the key name; preserve `Key` in APIs. | +| Bucket policy | バケットポリシー | Preserve policy JSON and API names. | +| access key ID | アクセスキー ID | Preserve `AccessKeyId` and environment variables. | +| secret access key | シークレットアクセスキー | Preserve `SecretAccessKey` and placeholders. | +| endpoint | エンドポイント | Qualify as `S3 API エンドポイント` when ambiguous. | +| Region | リージョン / AWS リージョン | Preserve values such as `us-east-1`. | +| Console | Console | Treat as the RustFS product surface; use `管理画面` only as an explanation. | +| storage pool | ストレージプール | Distinguish it from an application connection pool. | +| erasure coding | イレイジャーコーディング | Explain once when required; preserve `EC:n`. | +| healing | データ修復 | Preserve exact UI, API, and metric names that contain `heal` or `healing`. | +| rebalancing | リバランス | Preserve exact Console action names. | +| decommission | デコミッション / 使用停止 | Explain that data is drained before the pool is deactivated. | +| replication | レプリケーション | Qualify as `バケットレプリケーション` or `サイトレプリケーション`. | +| versioning | バージョニング | Use for S3 Object versions, not source control. | +| Object Lock | オブジェクトロック | Preserve exact API and Console labels. | +| legal hold | リーガルホールド | Preserve API identifiers such as `ObjectLockLegalHold`. | +| lifecycle management | ライフサイクル管理 | Use for S3 lifecycle rules. | +| multipart upload | マルチパートアップロード | Preserve API actions such as `CreateMultipartUpload`. | +| server-side encryption | サーバー側の暗号化 | Keep SSE-S3, SSE-KMS, and SSE-C unchanged. | +| presigned URL | 署名付き URL | Keep URL uppercase and preserve API names. | +| path-style addressing | パス形式のアドレス指定 | Preserve SDK option names. | +| virtual-hosted-style addressing | 仮想ホスト形式のアドレス指定 | Relate it to `RUSTFS_SERVER_DOMAINS` when relevant. | +| health check | ヘルスチェック | Preserve endpoint paths and probe names. | +| readiness | Readiness / 準備完了状態 | Preserve `/health/ready` and Kubernetes field names. | +| observability | オブザーバビリティ | Keep OpenTelemetry and OTLP unchanged. | +| rootless | root 権限なし | Prefer this over an unnatural literal translation. | +| root credentials | ルート認証情報 | Distinguish RustFS deployment credentials from the operating-system root account. | + +## Common review failures + +- Reject unchanged English paragraphs, descriptions, headings, and navigation labels. +- Reject `S3 と 100% 互換`, `完全互換`, or an unverified `ドロップイン代替` claim. +- Do not translate API operations, environment variables, UI labels, or command output. +- Do not alternate arbitrarily between `バケット` and improvised Japanese equivalents. +- Do not translate an S3 Object as `ファイル` unless the source explicitly means a local file. diff --git a/.agents/skills/localize-rustfs-docs/references/terminology.md b/.agents/skills/localize-rustfs-docs/references/terminology.md index 63fef5a8..2a7374c7 100644 --- a/.agents/skills/localize-rustfs-docs/references/terminology.md +++ b/.agents/skills/localize-rustfs-docs/references/terminology.md @@ -9,6 +9,14 @@ Use this hierarchy when choosing a translation: Never choose a dictionary translation that changes the RustFS concept. Search the repository and upstream source before adding a new translation, and update this glossary when a new term becomes canonical. +Read the locale-specific reference before translating or reviewing that locale: + +- [de.md](de.md) for German +- [fr.md](fr.md) for French +- [ja.md](ja.md) for Japanese + +Treat these references as project decisions, not substitutes for current upstream documentation. When an ecosystem changes its official localized term, verify the new usage, update the applicable reference, and use one term consistently across the locale. + ## Simplified Chinese baseline | Source term | Preferred Chinese | Guidance | diff --git a/.agents/skills/localize-rustfs-docs/scripts/audit-locales.mjs b/.agents/skills/localize-rustfs-docs/scripts/audit-locales.mjs new file mode 100644 index 00000000..c1edc161 --- /dev/null +++ b/.agents/skills/localize-rustfs-docs/scripts/audit-locales.mjs @@ -0,0 +1,470 @@ +#!/usr/bin/env node + +import { readdir, readFile } from "node:fs/promises"; +import path from "node:path"; +import process from "node:process"; + +const DEFAULT_SOURCE = "en"; +const DEFAULT_ROOT = "content"; +const DEFAULT_MAX_FINDINGS = 200; +const TITLE_ALLOWLIST = new Set([ + "amazon s3", + "boto3", + "cors", + "docker", + "ftps", + "helm", + "iam", + "java", + "javascript", + "kms", + "kubernetes", + "linux", + "macos", + "oidc", + "podman", + "python", + "rclone", + "rust", + "rustfs", + "s3", + "s3cmd", + "sdk", + "sftp", + "sts", + "terraform", + "traefik", + "typescript", + "webdav", + "windows", +]); + +function usage() { + return `Usage: node audit-locales.mjs [options] + +Options: + --root Content root (default: content) + --source Source locale (default: en) + --locales Target locales (default: discover all except source) + --strict Treat warnings as failures + --json Emit machine-readable JSON + --max-findings Limit displayed text findings (default: 200) + --help Show this help +`; +} + +function parseArgs(argv) { + const options = { + root: DEFAULT_ROOT, + source: DEFAULT_SOURCE, + locales: null, + strict: false, + json: false, + maxFindings: DEFAULT_MAX_FINDINGS, + }; + + for (let index = 0; index < argv.length; index += 1) { + const argument = argv[index]; + if (argument === "--help") { + options.help = true; + } else if (argument === "--strict") { + options.strict = true; + } else if (argument === "--json") { + options.json = true; + } else if (["--root", "--source", "--locales", "--max-findings"].includes(argument)) { + const value = argv[index + 1]; + if (!value) throw new Error(`${argument} requires a value`); + index += 1; + if (argument === "--root") options.root = value; + if (argument === "--source") options.source = value; + if (argument === "--locales") { + options.locales = [...new Set(value.split(",").map((item) => item.trim()).filter(Boolean))]; + } + if (argument === "--max-findings") { + options.maxFindings = Number.parseInt(value, 10); + if (!Number.isInteger(options.maxFindings) || options.maxFindings < 1) { + throw new Error("--max-findings must be a positive integer"); + } + } + } else { + throw new Error(`unknown option: ${argument}`); + } + } + + return options; +} + +function toPosix(relativePath) { + return relativePath.split(path.sep).join("/"); +} + +function isAuditedFile(filename) { + return filename === "meta.json" || filename.endsWith(".md") || filename.endsWith(".mdx"); +} + +async function listFiles(root) { + const files = []; + + async function walk(directory) { + const entries = await readdir(directory, { withFileTypes: true }); + for (const entry of entries) { + const absolute = path.join(directory, entry.name); + if (entry.isDirectory()) { + await walk(absolute); + } else if (entry.isFile() && isAuditedFile(entry.name)) { + files.push(toPosix(path.relative(root, absolute))); + } + } + } + + await walk(root); + return files.sort(); +} + +async function readFiles(root, files) { + const result = new Map(); + await Promise.all( + files.map(async (relative) => { + result.set(relative, await readFile(path.join(root, relative), "utf8")); + }), + ); + return result; +} + +function addIssue(issues, severity, code, locale, file, message) { + issues.push({ severity, code, locale, file, message }); +} + +function parseFrontmatter(content) { + const lines = content.replaceAll("\r\n", "\n").split("\n"); + if (lines[0]?.trim() !== "---") return null; + const end = lines.findIndex((line, index) => index > 0 && line.trim() === "---"); + if (end < 0) return null; + + const keys = []; + const values = new Map(); + for (const line of lines.slice(1, end)) { + const match = line.match(/^([A-Za-z0-9_-]+):\s*(.*)$/); + if (!match) continue; + keys.push(match[1]); + values.set(match[1], match[2].trim().replace(/^(["'])(.*)\1$/, "$2")); + } + return { keys, values, end }; +} + +function withoutFrontmatter(content) { + const frontmatter = parseFrontmatter(content); + if (!frontmatter) return content; + return content.replaceAll("\r\n", "\n").split("\n").slice(frontmatter.end + 1).join("\n"); +} + +function extractCodeFences(content) { + const lines = withoutFrontmatter(content).replaceAll("\r\n", "\n").split("\n"); + const fences = []; + + for (let index = 0; index < lines.length; index += 1) { + const opening = lines[index].match(/^\s*(`{3,}|~{3,})(.*)$/); + if (!opening) continue; + const marker = opening[1]; + const markerCharacter = marker[0]; + const body = []; + let closing = index + 1; + for (; closing < lines.length; closing += 1) { + const closeMatch = lines[closing].match(/^\s*(`{3,}|~{3,})\s*$/); + if (closeMatch && closeMatch[1][0] === markerCharacter && closeMatch[1].length >= marker.length) break; + body.push(lines[closing]); + } + fences.push({ info: opening[2].trim(), body: body.join("\n") }); + index = closing; + } + return fences; +} + +function removeCodeFences(content) { + const lines = withoutFrontmatter(content).replaceAll("\r\n", "\n").split("\n"); + const visible = []; + let marker = null; + + for (const line of lines) { + if (!marker) { + const opening = line.match(/^\s*(`{3,}|~{3,})/); + if (opening) { + marker = opening[1]; + } else { + visible.push(line); + } + continue; + } + + const closing = line.match(/^\s*(`{3,}|~{3,})\s*$/); + if (closing && closing[1][0] === marker[0] && closing[1].length >= marker.length) marker = null; + } + return visible.join("\n"); +} + +function extractInlineCode(content) { + const tokens = []; + const visible = removeCodeFences(content); + const pattern = /(? line.match(/^(#{1,6})\s+/)?.[1].length) + .filter(Boolean); +} + +function extractExternalUrls(content) { + const urls = []; + for (const match of removeCodeFences(content).matchAll(/https?:\/\/[^\s)<>{}"']+/g)) { + urls.push(match[0].replace(/[.,;:!?]+$/, "")); + } + return urls.sort(); +} + +function normalizeLocaleLinks(content, locale, source) { + return content.replaceAll(`/${locale}/`, `/${source}/`); +} + +function normalizeDocument(content, locale, source) { + return normalizeLocaleLinks(content, locale, source) + .replaceAll("\r\n", "\n") + .split("\n") + .map((line) => line.trimEnd()) + .join("\n") + .trim(); +} + +function visibleProseLines(content, locale, source) { + return removeCodeFences(content) + .split("\n") + .map((line) => normalizeLocaleLinks(line, locale, source).trim()) + .map((line) => line + .replace(/<[^>]*>/g, " ") + .replace(/`[^`\n]+`/g, " ") + .replace(/!\[([^\]]*)\]\([^)]*\)/g, "$1") + .replace(/\[([^\]]+)\]\([^)]*\)/g, "$1") + .replace(/https?:\/\/\S+/g, " ") + .replace(/^[#>*+\-\d.\s|]+/, "") + .replace(/[|*_~]/g, " ") + .replace(/\s+/g, " ") + .trim()) + .filter((line) => { + if (line.length < 60) return false; + if (/^(import|export)\s/.test(line)) return false; + if (/^<[^>]+>$/.test(line)) return false; + if (/^\|?\s*:?-{3,}/.test(line)) return false; + const words = line.match(/[A-Za-z]{2,}/g) ?? []; + return words.length >= 8; + }); +} + +function shapeOf(value) { + if (Array.isArray(value)) return value.map(shapeOf); + if (value && typeof value === "object") { + return Object.fromEntries(Object.keys(value).sort().map((key) => [key, shapeOf(value[key])])); + } + return typeof value; +} + +function visibleMetaStrings(value) { + const strings = []; + if (typeof value?.title === "string") strings.push(value.title); + if (Array.isArray(value?.pages)) { + for (const page of value.pages) { + if (typeof page !== "string") continue; + const label = page.match(/^\[([^\]]+)]\(/)?.[1]; + if (label) strings.push(label); + } + } + return strings; +} + +function requiresLocalization(value) { + const normalized = value.trim().toLowerCase(); + if (!normalized || TITLE_ALLOWLIST.has(normalized)) return false; + return /[A-Za-z]{3,}/.test(value); +} + +function same(left, right) { + return JSON.stringify(left) === JSON.stringify(right); +} + +function auditMeta(sourceContent, targetContent, sourceLocale, locale, file, issues) { + let source; + let target; + try { + source = JSON.parse(sourceContent); + } catch (error) { + addIssue(issues, "error", "INVALID_SOURCE_JSON", locale, file, error.message); + return; + } + try { + target = JSON.parse(targetContent); + } catch (error) { + addIssue(issues, "error", "INVALID_TARGET_JSON", locale, file, error.message); + return; + } + + if (!same(shapeOf(source), shapeOf(target))) { + addIssue(issues, "error", "META_STRUCTURE_MISMATCH", locale, file, "meta.json keys, types, or array structure differ from English"); + } + const sourceVisible = visibleMetaStrings(source); + const targetVisible = visibleMetaStrings(target); + const unchangedVisible = sourceVisible.filter((value, index) => value === targetVisible[index] && requiresLocalization(value)); + if (unchangedVisible.length > 0) { + addIssue(issues, "warning", "UNLOCALIZED_META", locale, file, `${unchangedVisible.length} visible navigation string(s) remain unchanged from English`); + } + if (targetContent.includes(`/${sourceLocale}/`)) { + addIssue(issues, "error", "SOURCE_LOCALE_LINK", locale, file, `contains an internal /${sourceLocale}/ link`); + } +} + +function auditDocument(sourceContent, targetContent, source, locale, file, issues) { + const untranslatedFile = normalizeDocument(sourceContent, source, source) === normalizeDocument(targetContent, locale, source); + if (untranslatedFile) { + addIssue(issues, "error", "UNTRANSLATED_FILE", locale, file, "target content is identical to English after normalizing locale links"); + } + + const sourceFrontmatter = parseFrontmatter(sourceContent); + const targetFrontmatter = parseFrontmatter(targetContent); + if (!sourceFrontmatter || !targetFrontmatter) { + if (Boolean(sourceFrontmatter) !== Boolean(targetFrontmatter)) { + addIssue(issues, "error", "FRONTMATTER_MISMATCH", locale, file, "frontmatter presence differs from English"); + } + } else { + if (!same(sourceFrontmatter.keys, targetFrontmatter.keys)) { + addIssue(issues, "error", "FRONTMATTER_KEYS", locale, file, "frontmatter keys or order differ from English"); + } + for (const key of ["title", "description"]) { + const sourceValue = sourceFrontmatter.values.get(key); + const targetValue = targetFrontmatter.values.get(key); + if (!sourceValue || sourceValue !== targetValue) continue; + const allowlistedTitle = key === "title" && TITLE_ALLOWLIST.has(sourceValue.toLowerCase()); + if (!untranslatedFile && !allowlistedTitle && (key === "description" || sourceValue.split(/\s+/).length > 1)) { + addIssue(issues, "warning", "UNLOCALIZED_FRONTMATTER", locale, file, `${key} is unchanged from English`); + } + } + } + + if (!same(extractCodeFences(sourceContent), extractCodeFences(targetContent))) { + addIssue(issues, "error", "CODE_FENCE_MISMATCH", locale, file, "code-fence metadata or content differs from English"); + } + if (!same(extractInlineCode(sourceContent), extractInlineCode(targetContent))) { + addIssue(issues, "warning", "INLINE_CODE_MISMATCH", locale, file, "inline-code tokens differ from English"); + } + if (!same(extractHeadings(sourceContent), extractHeadings(targetContent))) { + addIssue(issues, "error", "HEADING_STRUCTURE_MISMATCH", locale, file, "heading levels differ from English"); + } + if (!same(extractExternalUrls(sourceContent), extractExternalUrls(targetContent))) { + addIssue(issues, "error", "EXTERNAL_URL_MISMATCH", locale, file, "external URL set differs from English"); + } + + const visibleTarget = removeCodeFences(targetContent); + if (visibleTarget.includes(`/${source}/`)) { + addIssue(issues, "error", "SOURCE_LOCALE_LINK", locale, file, `contains an internal /${source}/ link`); + } + + const targetLines = new Set(visibleProseLines(targetContent, locale, source)); + const sharedLines = visibleProseLines(sourceContent, source, source).filter((line) => targetLines.has(line)); + const sharedCharacters = sharedLines.reduce((total, line) => total + line.length, 0); + if (!untranslatedFile && (sharedLines.length >= 2 || sharedCharacters >= 160)) { + addIssue(issues, "warning", "UNTRANSLATED_PROSE", locale, file, `${sharedLines.length} long prose line(s) remain identical to English`); + } +} + +async function discoverLocales(root, source) { + const entries = await readdir(root, { withFileTypes: true }); + return entries + .filter((entry) => entry.isDirectory() && entry.name !== source) + .map((entry) => entry.name) + .sort(); +} + +function printText(report, maxFindings) { + console.log(`Locale audit: ${report.source} -> ${report.locales.join(", ")}`); + for (const stats of report.coverage) { + console.log(` ${stats.locale}: ${stats.audited} audited, ${stats.missing} missing, ${stats.extra} extra`); + } + console.log(`Findings: ${report.summary.errors} error(s), ${report.summary.warnings} warning(s)`); + + for (const issue of report.issues.slice(0, maxFindings)) { + console.log(`${issue.severity.toUpperCase()} [${issue.locale}] ${issue.file} ${issue.code}: ${issue.message}`); + } + if (report.issues.length > maxFindings) { + console.log(`... ${report.issues.length - maxFindings} additional finding(s) omitted; use --json for complete output`); + } +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + process.stdout.write(usage()); + return; + } + + const root = path.resolve(options.root); + const sourceRoot = path.join(root, options.source); + const locales = options.locales ?? (await discoverLocales(root, options.source)); + if (locales.length === 0) throw new Error("no target locales found"); + if (locales.includes(options.source)) throw new Error("source locale cannot also be a target locale"); + + const sourceFiles = await listFiles(sourceRoot); + const sourceContents = await readFiles(sourceRoot, sourceFiles); + const sourceSet = new Set(sourceFiles); + const issues = []; + const coverage = []; + + for (const locale of locales) { + const targetRoot = path.join(root, locale); + const targetFiles = await listFiles(targetRoot); + const targetContents = await readFiles(targetRoot, targetFiles); + const targetSet = new Set(targetFiles); + const missing = sourceFiles.filter((file) => !targetSet.has(file)); + const extra = targetFiles.filter((file) => !sourceSet.has(file)); + const shared = sourceFiles.filter((file) => targetSet.has(file)); + + for (const file of missing) addIssue(issues, "error", "MISSING_TARGET_FILE", locale, file, "English counterpart has no target file"); + for (const file of extra) addIssue(issues, "warning", "EXTRA_TARGET_FILE", locale, file, "target file has no English counterpart"); + + for (const file of shared) { + const sourceContent = sourceContents.get(file); + const targetContent = targetContents.get(file); + if (file.endsWith("meta.json")) { + auditMeta(sourceContent, targetContent, options.source, locale, file, issues); + } else { + auditDocument(sourceContent, targetContent, options.source, locale, file, issues); + } + } + + coverage.push({ locale, audited: shared.length, missing: missing.length, extra: extra.length }); + } + + issues.sort((left, right) => { + const severity = { error: 0, warning: 1 }; + return severity[left.severity] - severity[right.severity] + || left.locale.localeCompare(right.locale) + || left.file.localeCompare(right.file) + || left.code.localeCompare(right.code); + }); + + const errors = issues.filter((issue) => issue.severity === "error").length; + const warnings = issues.length - errors; + const report = { source: options.source, locales, coverage, summary: { errors, warnings }, issues }; + + if (options.json) { + console.log(JSON.stringify(report, null, 2)); + } else { + printText(report, options.maxFindings); + } + + if (errors > 0 || (options.strict && warnings > 0)) process.exitCode = 1; +} + +main().catch((error) => { + console.error(`locale audit failed: ${error.message}`); + process.exitCode = 2; +}); diff --git a/README.md b/README.md index 6dfca43a..f95e9570 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,9 @@ RustFS is an S3-compatible distributed object storage engine written in Rust. Th ## Repository Snapshot -- `content/` – Markdown (`.md`) sources; each section has an `index.md` entry point. URLs mirror the folder path (e.g. `content/installation/linux/quick-start.md` → `/installation/linux/quick-start`). -- `content/meta.json` – root sidebar definition (section order, labels, and links). -- `content/**/meta.json` – per-folder sidebar titles and ordering for nested groups. +- `content//` – localized Markdown (`.md`) and MDX (`.mdx`) sources. URLs mirror the locale and folder path (for example, `content/en/installation/linux/quick-start.md` → `/en/installation/linux/quick-start`). +- `content//meta.json` – locale-specific root sidebar definition (section order, labels, and links). +- `content//**/meta.json` – locale-specific sidebar titles and ordering for nested groups. - `press.config.tsx` – site config: name, navbar links, social links, logo, analytics, SEO, plugins, and MDX components. - `source.config.ts` – content collection + global MDX options (KaTeX math, Mermaid diagrams). - `waku.config.ts` – Waku/Vite plugins (FumaPress, Fumadocs MDX, Tailwind). @@ -39,7 +39,9 @@ RustFS is an S3-compatible distributed object storage engine written in Rust. Th ### Translations -- Chinese docs: +The site exposes `en`, `zh`, `de`, `fr`, and `ja` locale routes. Simplified Chinese is fully localized. German, French, and Japanese currently localize the site UI, homepage, and navigation while using the current English page body as an explicit fallback. Translate fallback pages from `content/en`, keep each locale's relative file tree aligned, and preserve non-Mermaid technical code blocks exactly. + +Run `npm run docs:check`, `npm run types:check`, and `npm run build` before submitting localization changes. ## Local Development diff --git a/content/de/administration/console/images/rustfs-console-login.png b/content/de/administration/console/images/rustfs-console-login.png new file mode 100644 index 00000000..6282809d Binary files /dev/null and b/content/de/administration/console/images/rustfs-console-login.png differ diff --git a/content/de/administration/console/index.md b/content/de/administration/console/index.md new file mode 100644 index 00000000..489ef506 --- /dev/null +++ b/content/de/administration/console/index.md @@ -0,0 +1,69 @@ +--- +title: "Console" +description: "Enable the RustFS Console, connect it to a server, and sign in securely." +--- + +The **RustFS Console** is the web administration interface for RustFS. Use this page to enable the Console, open the login page, and choose the appropriate sign-in method. Detailed procedures for buckets, objects, and identity management are covered in their respective documentation sections. + +## Enable the Console + +The Console is enabled by default and listens on port `9001`, separately from the S3 API on port `9000`. You can set the behavior explicitly with the following environment variables: + +```ini title="/etc/default/rustfs" +RUSTFS_CONSOLE_ENABLE=true +RUSTFS_CONSOLE_ADDRESS=":9001" +``` + +Restart RustFS after changing these values. Set `RUSTFS_CONSOLE_ENABLE=false` when the Console must not run. + +The equivalent command-line options are `--console-enable` and `--console-address`. See the [CLI reference](/reference/cli) and [environment variable reference](/reference/environment-variables) for the complete server configuration. + +## Open the Console + +Open the following address, replacing `` with the RustFS server address: + +```text +http://:9001 +``` + +![RustFS Console login page with key, STS, and OIDC sign-in options](./images/rustfs-console-login.png) + +For a local deployment, use `http://localhost:9001`. Windows and macOS desktop launchers use port `7001` instead. + +If the login page cannot reach the target RustFS service, select **Server Configuration** or open `/config`. Enter the externally reachable RustFS service address and save it after the health check succeeds. **Reset** clears the saved address; **Skip** returns to login without changing it. + +## Log in + +The login methods shown depend on the deployment configuration: + +- **Key Login** uses the access key and secret key configured for the RustFS deployment. This is the standard login method for a local administrator. +- **STS Login** uses temporary Security Token Service (STS) credentials. Use it only when your identity workflow has issued a valid session token. +- **OIDC Login** appears when an OpenID Connect (OIDC) provider is configured. Select the provider and complete authentication with the identity provider. + +After login, the Console opens the first page your account can access. Menus and actions vary by account policy and enabled platform capabilities; a missing menu does not necessarily indicate a Console error. + +If login fails, verify the selected login method, credentials, target server address, and account status before retrying. + +:::warning[Do not expose default credentials] + +RustFS falls back to `rustfsadmin` / `rustfsadmin` when custom credentials are not configured. Use these defaults only for a throwaway local test. Configure a unique access key and a strong secret key before making the Console reachable by other users. + +::: + +## Operational notes + +- Use [TLS](/integration/tls-configured) before exposing the Console outside a trusted network. +- Restrict network access to the Console listener and configure [Console CORS](/administration/cors) only when cross-origin access is required. +- The Console session inherits the permissions of the signed-in identity. Use a least-privilege account for routine work. +- Signing out or an expired session returns you to the login page. Do not store administrator credentials in shared browsers. + +## Management workflows + +- [Create and manage buckets](/administration/data/bucket/creation) +- [Upload and manage objects](/administration/data/object/creation) +- [Manage access keys](/security-compliance/iam/access-token) +- [Configure identity and access management](/security-compliance/iam) + +## Next steps + +Review the [security checklist](/installation/requirement/checklists/security-checklists) before exposing the Console outside a trusted network. For OIDC-based login, continue with the [OIDC configuration guide](/security-compliance/oidc). diff --git a/content/de/administration/cors/index.md b/content/de/administration/cors/index.md new file mode 100644 index 00000000..d4e52db4 --- /dev/null +++ b/content/de/administration/cors/index.md @@ -0,0 +1,50 @@ +--- +title: "CORS Configuration" +description: "Configure allowed browser origins for the RustFS S3 API and Console." +--- + +Cross-Origin Resource Sharing (CORS) controls which browser origins can access the RustFS S3 API and Console. Configure each listener separately, then restart RustFS to apply the environment changes. + +## S3 API origins + +Set `RUSTFS_CORS_ALLOWED_ORIGINS` to a comma-separated list of trusted origins: + +```ini title="/etc/default/rustfs" +RUSTFS_CORS_ALLOWED_ORIGINS="https://app.example.com,https://admin.example.com" +``` + +When this variable is unset or empty, the S3 endpoint does not add generic CORS response headers. A list of explicit origins allows credentialed browser requests from matching origins. + +You can set the value to `*` to allow requests from any origin. Wildcard mode does not allow browser credentials. + +:::warning[Use explicit origins in production] + +Use a comma-separated allowlist for applications that send credentials. Reserve `*` for public resources that do not require credentialed browser requests. + +::: + +## Console origins + +The Console uses a separate variable: + +```ini title="/etc/default/rustfs" +RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="https://admin.example.com" +``` + +Use a comma-separated list when more than one browser origin must access the Console. Set `*` only when any origin must be allowed. + +## Verify the response + +Send a request with an `Origin` header and inspect the CORS response headers: + +```bash +curl -i \ + -H "Origin: https://app.example.com" \ + http://localhost:9000/ +``` + +Confirm that `Access-Control-Allow-Origin` contains the expected origin. Repeat the check with an unlisted origin and confirm that it is not allowed. + +## Next steps + +See the [environment variable reference](/reference/environment-variables#cors) for the verified defaults and configuration formats. \ No newline at end of file diff --git a/content/de/administration/data/bucket/creation.md b/content/de/administration/data/bucket/creation.md new file mode 100644 index 00000000..46804ffc --- /dev/null +++ b/content/de/administration/data/bucket/creation.md @@ -0,0 +1,52 @@ +--- +title: "RustFS Bucket Creation" +description: "Create buckets using the RustFS UI, rc, or the S3 API." +--- + +This guide explains how to create buckets using the RustFS UI, `rc`, or the S3 API. + +## Requirements + +- A running RustFS instance (see [Installation Guide](../../../installation/index.md)). +- [`rc`](/operations/rc) installed and configured with an alias for the command-line workflow. + +## Using the RustFS UI + +1. Log in to the RustFS Console. +2. On the Buckets page, in the top right corner, select **Create Bucket**. +3. Enter the bucket name and click **Create** to complete bucket creation. + +![bucket creation](images/bucket-creation-by-ui.png) + +## Using `rc` + +See the [`rc` guide](/operations/rc) for installation and alias configuration. + +Create a bucket: + +```bash +rc bucket create rustfs/my-bucket +rc bucket list rustfs/ +``` + +```text +✓ Bucket 'rustfs/my-bucket' created successfully. +``` + +## Using the API + +Create a bucket via API: + +```http +PUT /{bucketName} HTTP/1.1 +``` + +S3 requests must be signed with AWS Signature V4, so use an S3 client rather than hand-crafting headers. With the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured for your access keys: + +```bash +aws s3api create-bucket \ + --bucket bucket-creation-by-api \ + --endpoint-url http://localhost:9000 +``` + +Verify the bucket creation in the RustFS Console. diff --git a/content/de/administration/data/bucket/deletion.md b/content/de/administration/data/bucket/deletion.md new file mode 100644 index 00000000..b2234a81 --- /dev/null +++ b/content/de/administration/data/bucket/deletion.md @@ -0,0 +1,54 @@ +--- +title: "RustFS Bucket Deletion" +description: "Delete buckets using the RustFS UI, rc, or the S3 API." +--- + +This guide explains how to delete buckets using the RustFS UI, `rc`, or the S3 API. + +## Requirements + +- Install and configure [`rc`](/operations/rc) before using the command-line workflow. +- Empty the target bucket before deleting it, or use `--force` only after reviewing the objects that will be removed. + +**Warning**: Deleting a bucket is irreversible and may break applications relying on it. Ensure you have backed up any necessary data before proceeding. + +## Using the RustFS UI + +1. Log in to the RustFS Console. +2. On the homepage, select the bucket you want to delete. +3. On the far right, select the **Delete** button. +4. In the popup dialog, click **Confirm** to complete bucket deletion. + +![bucket deletion](images/bucket-deletion-on-ui.png) + +## Using `rc` + +See the [`rc` guide](/operations/rc) for installation and alias configuration. + +Delete a bucket: + +```bash +rc bucket remove rustfs/my-bucket +``` + +```text +✓ Bucket 'rustfs/my-bucket' removed successfully. +``` + +## Using the API + +Delete a bucket via API: + +```http +DELETE /{bucketName} HTTP/1.1 +``` + +S3 requests must be signed with AWS Signature V4, so use an S3 client rather than hand-crafting headers. With the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured for your access keys: + +```bash +aws s3api delete-bucket \ + --bucket bucket-creation-by-api \ + --endpoint-url http://localhost:9000 +``` + +Verify the bucket deletion in the RustFS Console. diff --git a/content/de/administration/data/bucket/images/bucket-creation-by-ui.png b/content/de/administration/data/bucket/images/bucket-creation-by-ui.png new file mode 100644 index 00000000..c3415a5a Binary files /dev/null and b/content/de/administration/data/bucket/images/bucket-creation-by-ui.png differ diff --git a/content/de/administration/data/bucket/images/bucket-deletion-on-ui.png b/content/de/administration/data/bucket/images/bucket-deletion-on-ui.png new file mode 100644 index 00000000..489b3237 Binary files /dev/null and b/content/de/administration/data/bucket/images/bucket-deletion-on-ui.png differ diff --git a/content/de/administration/data/bucket/images/bucket-quota-console.png b/content/de/administration/data/bucket/images/bucket-quota-console.png new file mode 100644 index 00000000..5e46df45 Binary files /dev/null and b/content/de/administration/data/bucket/images/bucket-quota-console.png differ diff --git a/content/de/administration/data/bucket/meta.json b/content/de/administration/data/bucket/meta.json new file mode 100644 index 00000000..48d5e82b --- /dev/null +++ b/content/de/administration/data/bucket/meta.json @@ -0,0 +1,10 @@ +{ + "title": "Buckets", + "pages": [ + "creation", + "deletion", + "quota", + "replication", + "policy" + ] +} \ No newline at end of file diff --git a/content/de/administration/data/bucket/policy.md b/content/de/administration/data/bucket/policy.md new file mode 100644 index 00000000..c13c11b5 --- /dev/null +++ b/content/de/administration/data/bucket/policy.md @@ -0,0 +1,205 @@ +--- +title: "Bucket Policy" +description: "Create, apply, inspect, and verify RustFS bucket policies with the S3 API." +--- + +RustFS bucket policies are S3-compatible resource policies attached directly to a bucket. Use them to grant or deny access to the bucket and its objects, including controlled anonymous access for public downloads. + +## Overview + +A bucket policy contains one or more statements that match a principal, action, resource, and optional conditions. RustFS evaluates the policy for requests to the bucket before allowing the storage operation. + +Bucket policies and IAM policies serve different purposes: + +| Policy type | Attached to | Typical use | +| --- | --- | --- | +| Bucket policy | A bucket | Grant public access, add resource-level restrictions, or authorize access to one bucket. | +| IAM policy | A user or group | Define what an authenticated identity can do across one or more resources. | + +An explicit `Deny` takes precedence over an `Allow`. Bucket owners remain able to get, replace, or delete the bucket policy so that a deny statement cannot permanently lock policy administration. + +RustFS implements the standard S3 operations `PutBucketPolicy`, `GetBucketPolicy`, `GetBucketPolicyStatus`, and `DeleteBucketPolicy`. + +:::warning[Public policies] + +A statement with `"Principal": "*"` can grant access without authentication. Keep the action and resource scope as narrow as possible, and verify the result anonymously before using the policy in production. + +::: + +## Configuration + +### Requirements + +- Create the target bucket before applying a policy. +- Configure the AWS CLI with a RustFS credential and region `us-east-1`. +- Use a credential allowed to perform the required policy-management action. + +Set reusable variables: + +```bash +export RUSTFS_ENDPOINT=http://localhost:9000 +export BUCKET_NAME=my-bucket +``` + +Policy management requires these actions: + +| Operation | Required action | +| --- | --- | +| Apply or replace a policy | `s3:PutBucketPolicy` | +| Read a policy | `s3:GetBucketPolicy` | +| Read public status | `s3:GetBucketPolicyStatus` | +| Remove a policy | `s3:DeleteBucketPolicy` | + +### Policy document structure + +A bucket policy uses version `2012-10-17` and a `Statement` array: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "PublicReadObjects", + "Effect": "Allow", + "Principal": "*", + "Action": ["s3:GetObject"], + "Resource": ["arn:aws:s3:::my-bucket/public/*"] + } + ] +} +``` + +The bucket ARN and object ARN are different: + +| Resource | ARN format | Example actions | +| --- | --- | --- | +| Bucket | `arn:aws:s3:::my-bucket` | `s3:ListBucket`, `s3:GetBucketLocation` | +| Objects | `arn:aws:s3:::my-bucket/*` | `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject` | + +Use a prefix in the object ARN, such as `arn:aws:s3:::my-bucket/public/*`, to limit access to part of the bucket. For the complete statement format and supported condition operators, see [Users, Groups, and Policies](/security-compliance/iam/policies#policy-document-format). + +### Public Access Block + +If the bucket's Public Access Block configuration has `BlockPublicPolicy` enabled, RustFS rejects a new policy containing an `Allow` statement with a wildcard principal. Keep this protection enabled unless anonymous access is intentional. + +Public Access Block does not replace careful policy review. An existing explicit `Deny` still overrides allows, and authenticated requests continue to be evaluated against IAM and bucket policies. + +## Usage + +### Create a public-read policy + +Create a policy that allows anonymous downloads only from the `public/` prefix: + +```bash +cat > /tmp/my-bucket-policy.json <<'EOF' +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "PublicReadObjects", + "Effect": "Allow", + "Principal": "*", + "Action": ["s3:GetObject"], + "Resource": ["arn:aws:s3:::my-bucket/public/*"] + } + ] +} +EOF +``` + +This policy does not allow anonymous bucket listing, uploads, overwrites, or deletes. + +### Apply the policy + +```bash +aws s3api put-bucket-policy \ + --bucket "$BUCKET_NAME" \ + --policy file:///tmp/my-bucket-policy.json \ + --endpoint-url "$RUSTFS_ENDPOINT" +``` + +Applying another policy replaces the complete existing policy. Retrieve and review the current document before updating it; statements are not merged automatically. + +### Read the policy + +```bash +aws s3api get-bucket-policy \ + --bucket "$BUCKET_NAME" \ + --endpoint-url "$RUSTFS_ENDPOINT" \ + --query Policy \ + --output text +``` + +RustFS preserves the submitted policy JSON. If no policy exists, the API returns `NoSuchBucketPolicy`. + +### Check public status + +```bash +aws s3api get-bucket-policy-status \ + --bucket "$BUCKET_NAME" \ + --endpoint-url "$RUSTFS_ENDPOINT" +``` + +The response contains a high-level public-access status: + +```json +{ + "PolicyStatus": { + "IsPublic": false + } +} +``` + +The current status check detects anonymous bucket listing (`s3:ListBucket`) and uploads (`s3:PutObject`). A policy that exposes only object downloads, such as the `public/` example above, can still report `false`. Always test the exact anonymous action, object prefix, and conditions instead of treating this status as a complete access analysis. + +### Remove the policy + +```bash +aws s3api delete-bucket-policy \ + --bucket "$BUCKET_NAME" \ + --endpoint-url "$RUSTFS_ENDPOINT" +``` + +Deleting a bucket policy removes only that resource policy. IAM policies attached to users and groups remain unchanged. + +## Verification + +Upload a test object with an authenticated credential: + +```bash +printf 'hello from RustFS\n' > /tmp/hello.txt + +aws s3api put-object \ + --bucket "$BUCKET_NAME" \ + --key public/hello.txt \ + --body /tmp/hello.txt \ + --endpoint-url "$RUSTFS_ENDPOINT" +``` + +Verify that the allowed object can be downloaded without credentials: + +```bash +curl --fail-with-body \ + "${RUSTFS_ENDPOINT}/${BUCKET_NAME}/public/hello.txt" +``` + +Then verify that an object outside the allowed prefix is not public: + +```bash +aws s3api put-object \ + --bucket "$BUCKET_NAME" \ + --key private/hello.txt \ + --body /tmp/hello.txt \ + --endpoint-url "$RUSTFS_ENDPOINT" + +curl --fail-with-body \ + "${RUSTFS_ENDPOINT}/${BUCKET_NAME}/private/hello.txt" +``` + +The second `curl` request should fail with `AccessDenied`. Also test each authenticated user role that relies on the policy, especially when the document contains conditions or explicit deny statements. + +## Next steps + +- [Manage IAM policies](/security-compliance/iam/policies) +- [Manage credentials](/operations/credentials) +- [Configure audit logs](/security-compliance/audit-logs) \ No newline at end of file diff --git a/content/de/administration/data/bucket/quota.md b/content/de/administration/data/bucket/quota.md new file mode 100644 index 00000000..ef975aac --- /dev/null +++ b/content/de/administration/data/bucket/quota.md @@ -0,0 +1,204 @@ +--- +title: "Bucket Quota" +description: "Configure, inspect, and verify hard storage quotas for individual RustFS buckets." +--- + +RustFS bucket quotas limit the total object data stored in an individual bucket. Use a quota to prevent one workload from consuming more than its assigned capacity while allowing other buckets to use the remaining storage. + +## Overview + +RustFS currently supports hard, byte-based quotas. Before accepting a write, RustFS compares the bucket's current usage plus the requested object size with the configured limit. A write that would exceed the limit is rejected with `InvalidRequest` and a `Bucket quota exceeded` message. + +Quota checks cover these operations: + +| Operation | Quota behavior | +| --- | --- | +| Upload an object | Rejects the upload when current usage plus the object size exceeds the limit. | +| Complete a multipart upload | Checks the completed object size before committing the upload. | +| Copy an object into the bucket | Checks the source object size against the destination bucket quota. | +| Delete an object | Always allowed so that you can recover capacity. | + +The quota applies to object data, not the number of objects or request rate. A bucket without a configured limit is unlimited. + +:::note[Replacement uploads] + +The quota check reserves the full incoming object size against the bucket's current usage. Replacing an existing key can therefore require enough headroom for the complete replacement object. + +::: + +## Configure in the Console + +1. Sign in to the RustFS Console and open **Browser**. +2. Find the bucket and select **Settings**. +3. Under **Capacity & Metadata**, find **Bucket Quota** and select **Edit**. +4. Enable **Bucket Quota**. +5. Enter the quota size and select **MiB**, **GiB**, **TiB**, or **PiB**. +6. Select **Save Quota**. + +![Set Bucket Quota dialog with quota size and unit controls](./images/bucket-quota-console.png) + +To remove the limit, open the dialog again, disable **Bucket Quota**, and save the change. + +## Use rc + +Configure an alias for your RustFS deployment: + +```bash +rc alias set rustfs http://localhost:9000 \ + \ + --region us-east-1 --bucket-lookup path +``` + +Replace `localhost` with the RustFS server address when `rc` runs on another host. + +### Permissions + +Quota operations require these policy actions: + +| Operation | Required action | +| --- | --- | +| Set or clear a quota | `admin:SetBucketQuota` | +| Read quota configuration or statistics | `s3:GetBucketQuota` | +| Check whether a proposed operation fits | `s3:GetBucketQuota` | + +The root credential has these permissions. For delegated administration, attach only the actions needed for the workflow. + +### Set a quota + +Set a 1 GiB hard quota. `rc` accepts byte values or units such as `1G`, `500M`, and `10KB`: + +```bash +rc bucket quota set rustfs/my-bucket 1G +``` + +The response includes the configured limit and current usage: + +```text +Bucket: my-bucket +Quota: 1 GiB +Usage: 0 B +Type: HARD +``` + +### Read the quota + +```bash +rc bucket quota info rustfs/my-bucket +``` + +Use `--json` when another tool needs to process the result: + +```bash +rc bucket quota info rustfs/my-bucket --json +``` + +When no limit is configured, the human-readable output reports `Quota: unlimited`. + +:::note[Configuration propagation] + +A query issued immediately after setting or clearing a quota can briefly return the previous state. Query the quota again and confirm the expected value before starting the verification workflow. + +::: + +### Clear a quota + +Remove the limit without deleting objects: + +```bash +rc bucket quota clear rustfs/my-bucket +rc bucket quota info rustfs/my-bucket +``` + +The bucket becomes unlimited after the configuration change is applied. + +## Advanced quota checks + +`rc 0.1.29` does not expose commands for detailed usage statistics or advisory checks for proposed writes. Use the RustFS Admin API for these operations. Requests must use AWS Signature Version 4 with an active RustFS credential. + +The examples use these shell variables: + +```bash +export RUSTFS_ENDPOINT=http://localhost:9000 +export RUSTFS_ACCESS_KEY= +export RUSTFS_SECRET_KEY= +export BUCKET_NAME=my-bucket +``` + +:::warning[Protect credentials] + +Environment variables are convenient for local testing but may be visible to processes running as the same operating-system user. Use your platform's secret manager or a restricted credentials file in production. + +::: + +### Read detailed usage statistics + +Use the statistics endpoint to retrieve the limit, current usage, remaining bytes, and percentage used: + +```bash +curl --fail-with-body \ + --aws-sigv4 "aws:amz:us-east-1:s3" \ + --user "${RUSTFS_ACCESS_KEY}:${RUSTFS_SECRET_KEY}" \ + "${RUSTFS_ENDPOINT}/rustfs/admin/v3/quota-stats/${BUCKET_NAME}" +``` + +```json +{ + "bucket": "my-bucket", + "quota_limit": 1073741824, + "current_usage": 1048576, + "remaining_quota": 1072693248, + "usage_percentage": 0.09765625 +} +``` + +Usage values come from RustFS data-usage accounting. Allow time for that view to reflect very recent changes before using the statistics response for external billing or orchestration. + +### Check a proposed upload + +Check whether a 64 MiB upload would fit without writing an object: + +```bash +curl --fail-with-body \ + --aws-sigv4 "aws:amz:us-east-1:s3" \ + --user "${RUSTFS_ACCESS_KEY}:${RUSTFS_SECRET_KEY}" \ + --request POST \ + --header "Content-Type: application/json" \ + --data '{"operation_type":"PUT","operation_size":67108864}' \ + "${RUSTFS_ENDPOINT}/rustfs/admin/v3/quota-check/${BUCKET_NAME}" +``` + +The `allowed` field reports the decision. This check is advisory: another write can consume capacity before the planned upload starts, so the actual upload remains authoritative. + +## Verification + +Set a small test quota, upload an object that fits, and then attempt to upload an object that exceeds the remaining capacity: + +```bash +rc bucket quota set rustfs/my-bucket 1M + +dd if=/dev/zero of=/tmp/quota-small.bin bs=1024 count=256 +dd if=/dev/zero of=/tmp/quota-large.bin bs=1048576 count=2 + +rc object copy /tmp/quota-small.bin rustfs/my-bucket/hello.bin +rc object copy /tmp/quota-large.bin rustfs/my-bucket/too-large.bin +``` + +The 256 KiB object succeeds. The 2 MiB object exceeds the 1 MiB bucket quota, so `rc` exits with an error and RustFS does not create `too-large.bin`. + +Delete the first object and clear the test quota: + +```bash +rc object remove rustfs/my-bucket/hello.bin --force +rc bucket quota clear rustfs/my-bucket +rc bucket quota info rustfs/my-bucket +``` + +Confirm that the final query reports `Quota: unlimited`. + +If quota enforcement cannot read or parse its internal configuration, RustFS logs `Bucket quota check degraded to allow` and permits the write. Monitor for this warning because it means quota enforcement is temporarily unavailable. + +## Next steps + +- [Create a bucket](./creation.md) +- [Manage lifecycle rules](../lifecycle-management.md) +- [Configure observability](/operations/observability) \ No newline at end of file diff --git a/content/de/administration/data/bucket/replication.md b/content/de/administration/data/bucket/replication.md new file mode 100644 index 00000000..37642ae9 --- /dev/null +++ b/content/de/administration/data/bucket/replication.md @@ -0,0 +1,298 @@ +--- +title: "Bucket Replication" +description: "Configure, validate, and monitor asynchronous replication between versioned RustFS buckets." +--- + +RustFS bucket replication copies selected object versions from a source bucket to a target bucket. Use it to maintain a remote copy of bucket data, distribute objects between deployments, or prepare a secondary copy for recovery workflows. + +## Overview + +Bucket replication has two configuration layers: + +1. A **remote target** stores the target endpoint, bucket, credentials, and generated target ARN. +2. An S3 **replication configuration** attaches rules to the source bucket and references that target ARN. + +Both the source and target buckets must have versioning enabled. RustFS verifies the target connection and target-bucket versioning when you register the remote target. It rejects a replication configuration whose enabled rules reference an unknown or stale target ARN. + +Replication is asynchronous by default. A successful source upload means RustFS accepted the source object; it does not mean the target copy is already complete. + +Rules can select objects by prefix or object tags and control these behaviors: + +| Rule setting | Behavior | +| --- | --- | +| `Status` | Enables or disables the rule. | +| `Filter` | Restricts replication by key prefix, object tags, or both. | +| `ExistingObjectReplication` | Includes objects that existed before the rule when set to `Enabled`. | +| `DeleteMarkerReplication` | Replicates delete markers when set to `Enabled`. | +| `DeleteReplication` | Replicates deletion of a specific object version when set to `Enabled`. | +| `Destination` | Identifies the registered remote target by ARN. | + +Bucket replication is directional. Configure a separate target and rule in the opposite direction if both buckets must accept writes and replicate them to each other. Do not confuse bucket replication with [site replication](/operations/high-availability/site-replication), which synchronizes broader site configuration and identity data. + +## Configuration + +### Requirements + +- A source RustFS deployment and a reachable target S3-compatible deployment. +- A source bucket and target bucket with versioning enabled. +- A dedicated target credential that can inspect the target bucket's versioning and write replicated object versions and delete markers. +- A source administrator allowed to manage remote targets and replication configuration. +- Network access from every source node that can run replication work to the target endpoint. +- The RustFS [`rc`](/operations/rc) client installed on an administration host. + +Configure one `rc` alias for each deployment. Use dedicated credentials and replace the example endpoints before running the commands: + +```bash +rc alias set source https://source.example.com:9000 \ + \ + --region us-east-1 --bucket-lookup path + +rc alias set target https://target.example.com:9000 \ + \ + --region us-east-1 --bucket-lookup path + +rc alias list +``` + +The source deployment uses the target alias credentials when it registers the remote target. `rc alias list` displays endpoints but does not print secret keys. + +:::warning[Protect target credentials] + +RustFS stores the target credential as part of the source bucket's remote-target configuration. Use a dedicated credential with access limited to the target bucket and do not reuse a root credential. + +::: + +### Permissions + +Source-side administration uses these policy actions: + +| Operation | Required action | +| --- | --- | +| Register, update, or remove a remote target | `admin:SetBucketTarget` | +| List remote targets | `admin:GetBucketTarget` | +| Read replication metrics | `admin:GetReplicationMetrics` | +| Apply or delete replication configuration | `s3:PutReplicationConfiguration` | +| Read replication configuration | `s3:GetReplicationConfiguration` | + +The target credential must pass RustFS target validation, which checks bucket access, versioning, replicated-object writes, replicated delete markers, and object-version deletion. When Object Lock is enabled on the source bucket, the target must have compatible Object Lock support. + +### Create and version the buckets + +Create the source bucket: + +```bash +rc bucket create source/my-bucket +``` + +Create the target bucket: + +```bash +rc bucket create target/my-bucket-replica +``` + +Enable versioning on both buckets: + +```bash +rc bucket version enable source/my-bucket +rc bucket version enable target/my-bucket-replica + +rc bucket version info source/my-bucket +rc bucket version info target/my-bucket-replica +``` + +Do not suspend source-bucket versioning while replication is configured. + +## Usage + +### Configure replication in the Console + +The RustFS Console combines remote-target registration and replication-rule configuration in one form. + +1. Sign in to the source deployment's Console. +2. Open **Buckets**, locate the source bucket, and select **Settings**. +3. Under **Data Protection**, enable **Versioning** if it is disabled. +4. Under **Automation**, select **Open Bucket Replication**. +5. Select **Add Replication Rule**. +6. Configure the destination and rule: + +| Console field | Value | +| --- | --- | +| **Priority** | Rule evaluation priority. The initial value is `1`. | +| **Mode** | Select **Asynchronous** or **Synchronous**. Asynchronous is selected by default. | +| **Endpoint** | Target S3 API address as `host:port`, without a URL scheme. | +| **Bucket** | Versioned destination bucket name. | +| **Access Key** and **Secret Key** | Dedicated credentials authorized to replicate into the destination bucket. | +| **Region** | Target region. The initial value is `us-east-1`. | +| **Storage Class** | Storage class applied at the destination. The initial value is `STANDARD`. | +| **Prefix** | Optional key prefix used to limit matching objects. | +| **Tags** | Optional object-tag name and value filters. Select **Add Tag** for additional filters. | +| **Use TLS** | Enables HTTPS for the destination connection. | +| **Replicate Existing Objects** | Includes objects created before the rule. Enabled by default. | +| **Replicate Delete Markers** | Copies delete markers to the destination. Enabled by default. | +| **Replicate Delete** | Copies deletion of a specific object version. Enabled by default. | +| **Health Check Interval** | Target health-check interval in seconds. The initial value is `60`. | +| **Bandwidth Limit** | Per-target transfer limit, selectable in KiB/s, MiB/s, or GiB/s. | + +7. Select **Save**. RustFS validates the source bucket, target connection, target credentials, and target-bucket versioning before accepting the rule. +8. Return to **Bucket Replication** to review the rule or select **Refresh** to update its displayed state. + +:::warning[Enable versioning on both deployments] + +The source and destination buckets must both have versioning enabled before you save the rule. Sign in to the target deployment's Console and enable versioning under **Buckets** → **Settings** → **Data Protection** when needed. + +::: + +For repeatable automation or configuration management, use the `rc` workflow below. + +### Add a replication rule with rc + +Create an asynchronous rule that copies new and existing objects and propagates delete markers and explicit version deletions: + +```bash +rc bucket replication add source/my-bucket \ + --remote-bucket target/my-bucket-replica \ + --id replicate-all \ + --priority 1 \ + --replicate delete,delete-marker,existing-objects +``` + +`rc` registers the target, obtains its generated ARN, and applies the replication rule in one operation. Omit `--replicate` flags for behaviors you do not want. Add `--sync` only when writes must wait for synchronous replication. + +To limit replication to a prefix, add `--prefix`: + +```bash +rc bucket replication add source/my-bucket \ + --remote-bucket target/my-bucket-replica \ + --id replicate-documents \ + --priority 2 \ + --prefix documents/ \ + --replicate delete-marker,existing-objects +``` + +Objects that do not match an enabled rule remain only in the source bucket. Use `--bandwidth` to set a byte-per-second limit, `--healthcheck-seconds` to change the target health-check interval, and `--storage-class` to override the destination storage class. + +### List and update rules + +List the active rules: + +```bash +rc bucket replication list source/my-bucket +rc bucket replication list source/my-bucket --json +``` + +Update a rule by its ID. Only supplied settings are changed: + +```bash +rc bucket replication update source/my-bucket \ + --id replicate-all \ + --priority 2 \ + --bandwidth 104857600 \ + --healthcheck-seconds 60 +``` + +Use `--status Enabled|Disabled` to enable or disable a rule and `--sync true|false` to change its replication mode. + +### Export and import configuration + +Export the complete replication configuration for review or backup: + +```bash +rc bucket replication export source/my-bucket --json > replication.json +``` + +The export includes remote-target metadata and the target Access Key, although the Secret Key is omitted. Protect the file as sensitive configuration. + +Import a previously exported configuration: + +```bash +rc bucket replication import source/my-bucket replication.json +``` + +### Delete replication configuration + +```bash +rc bucket replication remove source/my-bucket --id replicate-all + +# Remove every replication rule from the bucket. +rc bucket replication remove source/my-bucket --all +``` + +RustFS also removes replication remote targets referenced by the deleted configuration. It does not delete objects or versions already copied to the target bucket. Remove or retain those objects according to the target bucket's lifecycle and retention requirements. + +:::note[rc 0.1.29 removal response] + +After RustFS removes the replication configuration and its targets, `rc 0.1.29` can report `Remote target not found` while attempting a second target cleanup. Run `rc bucket replication list source/my-bucket --json`; an empty `rules` array confirms that the configuration was removed. + +::: + +## Verification + +### Check target readiness + +`rc bucket replication add` checks source access, target connectivity, target credentials, and bucket versioning before it creates the rule. Confirm the resulting destination and rule settings: + +```bash +rc bucket replication list source/my-bucket --json +``` + +### Replicate an object + +Upload a test object to the source: + +```bash +printf 'hello from RustFS replication\n' > /tmp/hello.txt +rc object copy /tmp/hello.txt source/my-bucket/hello.txt +``` + +Inspect the source object: + +```bash +rc object stat source/my-bucket/hello.txt --json +``` + +Because replication is asynchronous by default, the target object might not appear immediately. Repeat this command until it succeeds: + +```bash +rc object stat target/my-bucket-replica/hello.txt --json +rc object show target/my-bucket-replica/hello.txt +``` + +Compare the source and target `etag` and `size_bytes` values, then confirm that `object show` returns the expected content. + +To verify delete-marker replication, delete the source object and list versions on both buckets: + +```bash +rc object remove source/my-bucket/hello.txt --force + +rc bucket version list source/my-bucket/hello.txt --json +rc bucket version list target/my-bucket-replica/hello.txt --json +``` + +When delete-marker replication succeeds, the latest entry on both buckets has `is_delete_marker: true`. + +### Inspect replication metrics + +```bash +rc bucket replication status source/my-bucket +rc bucket replication status source/my-bucket --json +``` + +The command returns the source node's current in-memory replication statistics. Metrics can remain zero even after an object reaches the destination, so use them together with target-object and version checks rather than as the only verification signal. + +### Troubleshoot failures + +| Symptom | Check | +| --- | --- | +| `bucket versioning must be enabled` | Enable versioning on the source bucket. | +| Target registration reports `not versioned` | Enable versioning on the target bucket. | +| `replication target configuration not found` | Recheck the target alias, credentials, and destination bucket, then rerun `replication add`. | +| Rule reports a stale target | Refresh the rule list and retry `replication add`; if it persists, remove the failed configuration and recreate the target. | +| Target object does not appear | Check target reachability, credentials, target quota, and `rc bucket replication status`. | +| Replication fails | Inspect source-node logs for the target ARN and object key. | +| Target removal is disallowed | Delete or replace the replication configuration before removing a referenced target. | + +## Next steps + +- [Manage bucket lifecycle](../lifecycle-management.md) +- [Configure bucket quotas](./quota.md) +- [Configure observability](/operations/observability) \ No newline at end of file diff --git a/content/de/administration/data/lifecycle-management.md b/content/de/administration/data/lifecycle-management.md new file mode 100644 index 00000000..9cc05f87 --- /dev/null +++ b/content/de/administration/data/lifecycle-management.md @@ -0,0 +1,145 @@ +--- +title: "Lifecycle Management" +description: "Configure, inspect, and maintain RustFS object lifecycle rules with the rc command-line client." +--- + +RustFS lifecycle management applies expiration and transition rules to objects in a bucket. This page shows how to manage those rules with `rc`, verify a transition, and restore a temporary local copy of transitioned data. + +Before you begin, [install `rc`](/operations/rc), configure an alias named `local`, and create `my-bucket`. To transition objects, first register the target in [Tiered Storage](/administration/data/tiered-storage) and note its uppercase tier name. + +:::note[Asynchronous evaluation] + +Lifecycle rules do not process every eligible object immediately. The [Object Scanner](/administration/data/object/scanner) evaluates lifecycle work in the background. + +::: + +## Add an expiration rule + +Expire objects under the `logs/` prefix 30 days after creation: + +```bash +rc bucket lifecycle rule add local/my-bucket \ + --prefix logs/ \ + --expiry-days 30 +``` + +The command creates an enabled rule and returns its generated rule ID. Record that ID when you need to edit or remove this specific rule. + +## Add a transition rule + +Move object data to the registered `COLDTIER` tier after 90 days: + +```bash +rc bucket lifecycle rule add local/my-bucket \ + --transition-days 90 \ + --storage-class COLDTIER +``` + +`--storage-class` must match a registered tier name. Adding the rule does not create the tier or validate an AWS storage-class label as a RustFS tier. + +You can combine expiration and transition options in one rule. Use the noncurrent-version options only for a versioned bucket: + +```bash +rc bucket lifecycle rule add local/my-bucket \ + --noncurrent-transition-days 30 \ + --noncurrent-transition-storage-class COLDTIER \ + --noncurrent-expiry-days 365 +``` + +## Inspect and update rules + +List the current rules and note the ID of the rule you want to change: + +```bash +rc bucket lifecycle rule list local/my-bucket +``` + +Change a rule's expiration period or disable it without deleting it: + +```bash +rc bucket lifecycle rule edit local/my-bucket \ + --id \ + --expiry-days 60 +rc bucket lifecycle rule edit local/my-bucket \ + --id \ + --disable true +``` + +Remove one rule by ID. Use `--all` only when you intend to remove the bucket's complete lifecycle configuration: + +```bash +rc bucket lifecycle rule remove local/my-bucket --id +rc bucket lifecycle rule remove local/my-bucket --all +``` + +## Export and import rules + +Export the bucket's lifecycle configuration before a bulk change: + +```bash +rc bucket lifecycle rule export local/my-bucket > lifecycle.json +``` + +Importing a file replaces the lifecycle configuration sent to the bucket. Review the JSON before applying it: + +```bash +rc bucket lifecycle rule import local/my-bucket lifecycle.json +``` + +## Confirm a lifecycle transition + +After an object becomes eligible under its lifecycle rule, inspect it on the source cluster: + +```bash +aws s3api head-object \ + --bucket my-bucket \ + --key hello.txt \ + --endpoint-url http://localhost:9000 +``` + +After the transition completes, the response reports the registered tier name as the storage class: + +```json +{ + "StorageClass": "COLDTIER" +} +``` + +The response can contain additional object metadata. Before the transition completes, `StorageClass` may be absent or may not yet report `COLDTIER`. + +Read the object through the source RustFS endpoint as usual: + +```bash +aws s3 cp \ + s3://my-bucket/hello.txt \ + /path/to/hello.txt \ + --endpoint-url http://localhost:9000 +``` + +RustFS reads transitioned data through the source bucket and object key. Applications do not need to address the target bucket directly. + +## Restore a local copy + +Request a temporary local copy of a transitioned object and retain it for seven days: + +```bash +rc bucket lifecycle restore local/my-bucket/hello.txt --days 7 +``` + +While the copy-back is running, `HEAD` reports `x-amz-restore: ongoing-request="true"`. After completion, it reports `ongoing-request="false"` with an expiry date. A second restore submitted while one is running returns `RestoreAlreadyInProgress`. + +When the restore period expires, RustFS removes the local restored copy and its restore metadata. The transitioned object remains available from the remote tier. + +## Command compatibility + +The latest `rc` also accepts `rc ilm` as a compatibility alias. We recommend the noun-first `rc bucket lifecycle` form for new commands and scripts: + +```bash +rc bucket lifecycle --help +rc bucket lifecycle rule --help +rc bucket lifecycle tier --help +``` + +## Next steps + +Review [Tiered Storage](/administration/data/tiered-storage) to monitor or maintain remote tiers, and use [object creation](/administration/data/object/creation) to create test objects for a lifecycle rule. diff --git a/content/de/administration/data/meta.json b/content/de/administration/data/meta.json new file mode 100644 index 00000000..e72a1121 --- /dev/null +++ b/content/de/administration/data/meta.json @@ -0,0 +1,9 @@ +{ + "title": "Datenverwaltung", + "pages": [ + "object", + "bucket", + "tiered-storage", + "lifecycle-management" + ] +} \ No newline at end of file diff --git a/content/de/administration/data/object/creation.md b/content/de/administration/data/object/creation.md new file mode 100644 index 00000000..b0c5ca07 --- /dev/null +++ b/content/de/administration/data/object/creation.md @@ -0,0 +1,81 @@ +--- +title: "Object Creation" +description: "Create objects using the RustFS UI, rc, or the S3 API." +--- + +Objects are the fundamental storage units in RustFS, containing data, metadata, and a unique key. This guide covers object creation (upload). + +## Requirements + +- A running RustFS instance (see [Installation Guide](../../../installation/index.md)). +- [`rc`](/operations/rc) installed and configured with an alias for the command-line workflow. +- A target bucket. Create one by following [Bucket Creation](../bucket/creation.md). + +## Creating Objects + +### Using the RustFS UI + +1. Log in to the RustFS Console. +2. Select the target bucket. +3. On the bucket page, in the top right corner, select **New Directory**, **New File**, or **Upload File/Folder**. +4. To upload from your local machine, click **Upload File/Folder**, select the files, and click **Start Upload**. + +![object creation from ui](images/upload_file_from_ui.png) + +Click on an object to view its details. + +![object details info](images/object_details_info.png) + +### Using `rc` + +See the [`rc` guide](/operations/rc) for installation and alias configuration. + +Upload a file: + +```bash +rc object copy /path/to/hello.txt rustfs/my-bucket/hello.txt +rc object list rustfs/my-bucket +``` + +Verify the upload in the RustFS Console. + +### Using the API + +Upload a file via API: + +```http +PUT /{bucketName}/{objectName} HTTP/1.1 +``` + +S3 requests must be signed with AWS Signature V4, so use an S3 client rather than hand-crafting headers. With the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured for your access keys: + +```bash +aws s3api put-object \ + --bucket bucket-creation-by-api \ + --key hello.txt \ + --body /path/to/hello.txt \ + --endpoint-url http://localhost:9000 +``` + +Verify the upload in the RustFS Console. + +## Deleting Objects + +See [Object Deletion](./deletion.md). + +Use the following API for file deletion: + +```http +DELETE /{bucketName}/{objectName} HTTP/1.1 +``` + +Request example: + +```bash +aws s3api delete-object \ + --bucket bucket-creation-by-api \ + --key hello.txt \ + --endpoint-url http://localhost:9000 +``` + +You can confirm the file has been deleted on the RustFS UI. diff --git a/content/de/administration/data/object/deletion.md b/content/de/administration/data/object/deletion.md new file mode 100644 index 00000000..5e310623 --- /dev/null +++ b/content/de/administration/data/object/deletion.md @@ -0,0 +1,55 @@ +--- +title: "Object Deletion" +description: "Delete objects using the RustFS UI, rc, or the S3 API." +--- + +This guide covers object deletion. + +## Requirements + +- Install and configure [`rc`](/operations/rc) before using the command-line workflow. +- Confirm the alias, bucket, and object key before deleting an object. + +## Using the RustFS UI + +1. Log in to the RustFS Console. +2. Select the bucket containing the file to be deleted. +3. On the bucket page, select the file to be deleted. +4. Click **Delete Selected Items** in the upper right corner, then click **Confirm** in the popup dialog. + +![object deletion from ui](images/delete_file_from_ui.png) + +## Using `rc` + +Delete a file: + +```bash +rc object remove rustfs/my-bucket/hello.txt +rc object list rustfs/my-bucket +``` + +```text +Removed: rustfs/my-bucket/hello.txt +✓ Removed 1 object(s). +``` + +Verify the deletion in the RustFS Console. + +## Using the API + +Delete a file via API: + +```http +DELETE /{bucketName}/{objectName} HTTP/1.1 +``` + +S3 requests must be signed with AWS Signature V4, so use an S3 client rather than hand-crafting headers. With the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured for your access keys: + +```bash +aws s3api delete-object \ + --bucket bucket-creation-by-api \ + --key hello.txt \ + --endpoint-url http://localhost:9000 +``` + +Verify the deletion in the RustFS Console. diff --git a/content/de/administration/data/object/images/delete_file_from_ui.png b/content/de/administration/data/object/images/delete_file_from_ui.png new file mode 100644 index 00000000..171c3e3b Binary files /dev/null and b/content/de/administration/data/object/images/delete_file_from_ui.png differ diff --git a/content/de/administration/data/object/images/multipart-upload-console.png b/content/de/administration/data/object/images/multipart-upload-console.png new file mode 100644 index 00000000..5276735c Binary files /dev/null and b/content/de/administration/data/object/images/multipart-upload-console.png differ diff --git a/content/de/administration/data/object/images/object-lock-create-bucket.png b/content/de/administration/data/object/images/object-lock-create-bucket.png new file mode 100644 index 00000000..6264eb17 Binary files /dev/null and b/content/de/administration/data/object/images/object-lock-create-bucket.png differ diff --git a/content/de/administration/data/object/images/object_details_info.png b/content/de/administration/data/object/images/object_details_info.png new file mode 100644 index 00000000..086f2eab Binary files /dev/null and b/content/de/administration/data/object/images/object_details_info.png differ diff --git a/content/de/administration/data/object/images/upload_file_from_ui.png b/content/de/administration/data/object/images/upload_file_from_ui.png new file mode 100644 index 00000000..c09abff3 Binary files /dev/null and b/content/de/administration/data/object/images/upload_file_from_ui.png differ diff --git a/content/de/administration/data/object/meta.json b/content/de/administration/data/object/meta.json new file mode 100644 index 00000000..f52acabb --- /dev/null +++ b/content/de/administration/data/object/meta.json @@ -0,0 +1,11 @@ +{ + "title": "Objekte", + "pages": [ + "creation", + "deletion", + "object-lock", + "versioning", + "multipart-upload", + "scanner" + ] +} \ No newline at end of file diff --git a/content/de/administration/data/object/multipart-upload.md b/content/de/administration/data/object/multipart-upload.md new file mode 100644 index 00000000..d4a6af66 --- /dev/null +++ b/content/de/administration/data/object/multipart-upload.md @@ -0,0 +1,101 @@ +--- +title: "Multipart Upload" +description: "Upload large objects in parts, monitor Console upload tasks, and verify completed objects with rc." +--- + +Multipart upload splits one object into independently uploaded parts and assembles them on the server. Use it for large objects, retryable transfers, and browser uploads that need progress and cancellation controls. + +## Overview + +The S3 multipart workflow has three required phases: + +1. **CreateMultipartUpload** returns an upload ID. +2. **UploadPart** uploads numbered parts and returns an ETag for each part. +3. **CompleteMultipartUpload** submits the ordered part numbers and ETags to create the final object. + +Abort an unfinished upload with **AbortMultipartUpload** so temporary parts do not continue consuming storage. + +RustFS accepts part numbers from `1` through `10000`. `ListParts` and `ListMultipartUploads` return at most 1,000 entries per response and use markers for pagination. The completed object appears only after the completion request succeeds. + +## Upload in the Console + +1. Sign in to the RustFS Console and open the destination bucket. +2. Select **Upload File/Folder**. +3. Optionally enter a **Current Prefix**. +4. Select **Select File** or **Select Folder**, then choose the content to upload. +5. Review the selected names and sizes, then select **Start Upload**. + +![Console upload dialog with a selected large file](./images/multipart-upload-console.png) + +The observed Console upload dialog supports up to 10,000 selected files and reports a 512 GB maximum for a single file. These are Console upload limits; S3 clients can have different local limits. + +After upload starts, **Task Management** groups tasks into Pending, Processing, Completed, Failed, and Canceled states. Each processing task shows progress and a **Cancel** action. Canceling an active multipart task aborts the current part request and marks the task as canceled. + +Refresh the bucket after completion and confirm the object size and modification time. + +## Use rc + +Upload and verify an object with `rc`: + +```bash +rc object copy /path/to/hello.txt rustfs/my-bucket/hello.txt +rc object stat rustfs/my-bucket/hello.txt --json +``` + +:::note[rc 0.1.29 multipart boundary] + +`rc 0.1.29` does not expose create, upload-part, complete, list-parts, or abort multipart commands. A validated 20 MiB `rc object copy` used a single `PutObject` request rather than multipart upload. Use the Console or an S3 SDK when explicit multipart behavior is required; use `rc object stat` to verify the completed object. + +::: + +For a versioned bucket, list the version created by a completed multipart upload: + +```bash +rc bucket version list rustfs/my-bucket/hello.txt --json +``` + +RustFS returns a version ID when multipart completion succeeds in a versioned bucket. + +## S3 multipart operations + +Use an S3 SDK that supports these standard operations: + +| Phase | S3 operation | Required values | +| --- | --- | --- | +| Initiate | `CreateMultipartUpload` | Bucket, key, metadata, encryption, and optional Object Lock settings. | +| Upload | `UploadPart` | Bucket, key, upload ID, part number, and body. Save the returned ETag. | +| Inspect | `ListParts` | Bucket, key, and upload ID. Paginate when needed. | +| Complete | `CompleteMultipartUpload` | Ordered part numbers and their exact ETags. | +| Cancel | `AbortMultipartUpload` | Bucket, key, and upload ID. | +| Discover | `ListMultipartUploads` | Bucket and optional prefix. Paginate when needed. | + +Do not reuse an upload ID for a different key. Submit completed parts in ascending part-number order and preserve each ETag exactly as returned. + +## Verification + +After completion: + +```bash +rc object stat rustfs/my-bucket/hello.txt --json +rc object show rustfs/my-bucket/hello.txt > /tmp/hello-downloaded.txt +cmp /path/to/hello.txt /tmp/hello-downloaded.txt +``` + +Confirm that `size_bytes` matches the local file and that `cmp` exits successfully. For a versioned bucket, also confirm that `rc bucket version list` returns a version ID. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Upload remains in Processing | Check browser connectivity and keep the Console tab open; cancel and retry when the connection is interrupted. | +| A part request fails | Retry that part with the same upload ID and part number, then use the latest returned ETag. | +| Completion reports an invalid part | Verify the submitted part numbers, ordering, and ETags against `ListParts`. | +| Object is absent after uploading parts | Send `CompleteMultipartUpload`; uploaded parts alone do not create the object. | +| Temporary storage continues growing | List incomplete uploads and abort those that are no longer needed. | +| Completion fails on a locked key | Check Object Lock retention or Legal Hold on the current destination version. | + +## Next steps + +- [Create and inspect objects](./creation.md) +- [Manage object versions](./versioning.md) +- [Protect objects with Object Lock](./object-lock.md) \ No newline at end of file diff --git a/content/de/administration/data/object/object-lock.md b/content/de/administration/data/object/object-lock.md new file mode 100644 index 00000000..8e35fbf7 --- /dev/null +++ b/content/de/administration/data/object/object-lock.md @@ -0,0 +1,103 @@ +--- +title: "Object Lock" +description: "Protect versioned objects from deletion with retention periods and legal holds." +--- + +RustFS Object Lock applies write-once, read-many protection to individual object versions. Use retention periods for time-bound protection and Legal Hold for protection without a predefined expiration date. + +## Overview + +Object Lock requires bucket versioning and must be enabled when the bucket is created. Each overwrite creates a new version; retention and Legal Hold protect a specific version rather than the object key as a whole. + +| Protection | Behavior | +| --- | --- | +| `GOVERNANCE` retention | Blocks deletion and retention shortening unless the caller has bypass permission and explicitly requests a bypass. Retention can be extended without bypass. | +| `COMPLIANCE` retention | Blocks deletion even with governance bypass. The date can be extended but not shortened. | +| Legal Hold | Blocks deletion until the hold is set to `OFF`. It has no expiration date and governance bypass does not override it. | + +Deleting a key without a version ID creates a delete marker. The protected version remains stored and can still be retrieved by version ID. + +:::danger[Plan retention before enabling it] + +COMPLIANCE retention cannot be bypassed or shortened before its retain-until date. Test policies in a non-production bucket and verify time synchronization, permissions, lifecycle rules, replication, and backup procedures before protecting production data. + +::: + +## Configure in the Console + +1. Sign in to the RustFS Console and open **Buckets**. +2. Select **Create Bucket** and enter the bucket name. +3. Enable **Object Lock**. The Console also enables **Version** because Object Lock requires versioning. +4. To apply retention automatically to new object versions, enable **Retention**. +5. Select **COMPLIANCE** or **GOVERNANCE**, enter the validity, and select **Day** or **Year**. +6. Select **Create**. + +![Create Bucket dialog with versioning, Object Lock, and default retention enabled](./images/object-lock-create-bucket.png) + +The Console initially displays `180` days when default retention is enabled. Replace it with the period required by your retention policy; it is a UI initial value, not a general recommendation. + +To inspect a protected object, open the bucket, select the object name, and use the **Versions** and **Info** tabs. The Info tab exposes **Legal Hold** and **RetentionPolicy** fields. + +## Use rc + +Configure an alias before running these examples: + +```bash +rc alias set rustfs http://localhost:9000 \ + \ + --region us-east-1 --bucket-lookup path +``` + +:::note[rc 0.1.29 bucket creation boundary] + +`rc bucket create --with-lock` and `--with-versioning` appear in the `rc 0.1.29` help output but return `not implemented` when executed. Create the Object Lock bucket in the Console. The upload headers shown below are supported and were validated against RustFS. + +::: + +Upload an object version with GOVERNANCE retention. Supply both retention headers together and use an RFC 3339 UTC timestamp in the future: + +```bash +rc object copy /path/to/hello.txt rustfs/my-bucket/hello.txt \ + -H "x-amz-object-lock-mode:GOVERNANCE" \ + -H "x-amz-object-lock-retain-until-date:2027-01-01T00:00:00Z" +``` + +Upload an object version with Legal Hold enabled: + +```bash +rc object copy /path/to/hello.txt rustfs/my-bucket/legal-hold.txt \ + -H "x-amz-object-lock-legal-hold:ON" +``` + +List the protected versions and inspect a specific version: + +```bash +rc bucket version list rustfs/my-bucket --json +rc object stat rustfs/my-bucket/hello.txt \ + --version-id --json +``` + +`rc 0.1.29` does not provide commands to change retention or switch an existing Legal Hold between `ON` and `OFF`. Use the Console or an S3 SDK for those operations. + +## Verify protection + +1. Upload a test object with GOVERNANCE retention or Legal Hold. +2. Record its version ID with `rc bucket version list`. +3. Attempt to delete that exact version through an S3 client without bypass. RustFS must return `AccessDenied` while protection is active. +4. Confirm the version remains visible in the Console and through `rc object stat --version-id`. + +For GOVERNANCE retention, test bypass only with a dedicated administrative identity that has `s3:BypassGovernanceRetention`. COMPLIANCE retention and Legal Hold remain protected from governance bypass. + +## Operational considerations + +- Default bucket retention is calculated when a new object version or multipart upload is initiated. +- Copying an object creates a new destination version; destination retention policy applies independently. +- Suspending versioning is not appropriate for an Object Lock bucket. +- Lifecycle expiration cannot remove a version while retention or Legal Hold blocks deletion. +- Replication targets for locked objects must support compatible Object Lock behavior. + +## Next steps + +- [Manage object versions](./versioning.md) +- [Configure bucket replication](../bucket/replication.md) +- [Review the security checklist](/installation/requirement/checklists/security-checklists) \ No newline at end of file diff --git a/content/de/administration/data/object/scanner.md b/content/de/administration/data/object/scanner.md new file mode 100644 index 00000000..6330c38f --- /dev/null +++ b/content/de/administration/data/object/scanner.md @@ -0,0 +1,32 @@ +--- +title: "Object Scanning" +description: "Guide to the RustFS object scanner, including design, implementation, and monitoring." +--- + +This guide covers the design and implementation of the RustFS object scanner, including its integration with Erasure Coding, Scrub & Repair mechanisms, scheduling strategies, monitoring metrics, and troubleshooting. + +## Overview + +The RustFS object scanner is built into the storage engine and is responsible for periodically checking object integrity and executing scheduled operations. +Scanning tasks include disk usage statistics, lifecycle management rule evaluation, object replication execution, and triggering corrupted object self-healing. + +## Architecture and Design Principles + +### Scanner Architecture + +The RustFS scanner uses a hash sampling mechanism, selecting one out of every 1024 objects for inspection based on object name hashing to minimize performance impact. +The scanner is deeply integrated with the Erasure Coding module, utilizing redundant shards for online reconstruction when detecting lost or corrupted shards, ensuring high data availability and consistency. + +## Data Verification and Recovery + +The RustFS data verification mechanism checks metadata consistency and performs bit-by-bit reading and verification to discover hidden bad blocks. The object scanner detects issues like bit rot and triggers repair processes when necessary. + +## Scanning Modes and Scheduling + +RustFS supports three scanning modes: online scanning during reads, background periodic scanning, and manual full scanning, balancing performance and reliability. +Similar to the `osd_scrub_begin_hour` configuration in Ceph, administrators can set scanning start times and frequency. For example, light verification is set to once daily by default. + +## Monitoring and Metrics + +Scanner statistics include total task count, failure count, and time distribution, exposing metrics through the Prometheus data model such as `rustfs_scanner_jobs_total`, `rustfs_scanner_failures_total`, and `rustfs_scanner_duration_seconds`. +Combined with monitoring systems, alerts can be set based on scanning failure rates and duration to promptly discover and locate potential issues at the storage or network levels. \ No newline at end of file diff --git a/content/de/administration/data/object/versioning.md b/content/de/administration/data/object/versioning.md new file mode 100644 index 00000000..b891c756 --- /dev/null +++ b/content/de/administration/data/object/versioning.md @@ -0,0 +1,125 @@ +--- +title: "Versioning" +description: "Enable bucket versioning, recover earlier object versions, and manage delete markers with rc and the Console." +--- + +RustFS bucket versioning preserves multiple versions of the same object key. Use it to recover from accidental overwrites and deletes and to satisfy prerequisites for Object Lock and bucket replication. + +## Overview + +When versioning is enabled: + +- Each upload or copy to an existing key creates a new version ID. +- A normal delete creates a delete marker instead of removing older versions. +- Reading the key without a version ID returns the latest visible version, or `NotFound` when the latest entry is a delete marker. +- Reading or deleting a specific version requires its version ID. + +Suspending versioning stops normal creation of new version IDs but preserves existing versions and delete markers. Suspension is not the same as disabling or removing version history. + +## Configure in the Console + +### Enable at bucket creation + +1. Open **Buckets** and select **Create Bucket**. +2. Enter the bucket name. +3. Enable **Version**. +4. Select **Create**. + +### Enable or suspend an existing bucket + +1. Open **Buckets**, locate the bucket, and select **Settings**. +2. Under **Data Protection**, locate **Versioning**. +3. Enable versioning or suspend it as required. + +The settings page reports `Enabled`, `Suspended`, or `Disabled`. It also notes that suspension preserves existing versions. + +### Browse and recover versions + +1. Open the bucket and enable **Show Deleted Objects** when you need to see keys hidden by delete markers. +2. Select an object name to open **Object Details**. +3. Select **Versions** to inspect available versions and delete markers. +4. Download the required version and upload it again to make its content the latest version. + +## Use rc + +Create a bucket and enable versioning: + +```bash +rc bucket create rustfs/my-bucket +rc bucket version enable rustfs/my-bucket +rc bucket version info rustfs/my-bucket +``` + +Upload two versions of the same key: + +```bash +printf 'version one\n' > /tmp/hello.txt +rc object copy /tmp/hello.txt rustfs/my-bucket/hello.txt + +printf 'version two\n' > /tmp/hello.txt +rc object copy /tmp/hello.txt rustfs/my-bucket/hello.txt --overwrite +``` + +List versions and record the version ID you want to recover: + +```bash +rc bucket version list rustfs/my-bucket/hello.txt --json +rc object stat rustfs/my-bucket/hello.txt \ + --version-id --json +``` + +Restore an earlier version by downloading its body and uploading it as a new latest version: + +```bash +rc object show rustfs/my-bucket/hello.txt \ + --version-id > /tmp/hello-restored.txt + +rc object copy /tmp/hello-restored.txt \ + rustfs/my-bucket/hello.txt --overwrite +``` + +Delete the current key to create a delete marker, then inspect it: + +```bash +rc object remove rustfs/my-bucket/hello.txt --force +rc bucket version list rustfs/my-bucket/hello.txt --json +``` + +The latest entry should show `is_delete_marker: true`. The earlier versions remain available by version ID. + +Suspend versioning when you no longer want normal writes to create new numbered versions: + +```bash +rc bucket version suspend rustfs/my-bucket +rc bucket version info rustfs/my-bucket +``` + +:::warning[rc 0.1.29 version deletion boundary] + +`rc object remove --versions` is listed in the `rc 0.1.29` help output but returns `not implemented`. Use the Console or an S3 SDK to permanently delete selected version IDs. Do not remove an entire version history unless you have verified retention, replication, and recovery requirements. + +::: + +## Verification + +After enabling versioning: + +1. Upload the same key twice with different content. +2. Run `rc bucket version list` and confirm that both entries have distinct version IDs. +3. Retrieve each version with `rc object show --version-id` and compare its content. +4. Delete the key without a version ID and confirm that a delete marker appears. +5. Retrieve an older version by version ID to confirm it remains recoverable. + +## Operational considerations + +- Versioning increases storage use because overwrites and deletes retain older data. +- Configure lifecycle rules for noncurrent versions only after defining recovery and retention periods. +- Bucket replication requires versioning on both source and destination buckets. +- Object Lock depends on versioning and protects individual versions. +- Multipart completion creates one new object version in a versioned bucket. + +## Next steps + +- [Protect versions with Object Lock](./object-lock.md) +- [Upload large objects](./multipart-upload.md) +- [Manage bucket lifecycle](../lifecycle-management.md) \ No newline at end of file diff --git a/content/de/administration/data/tiered-storage.md b/content/de/administration/data/tiered-storage.md new file mode 100644 index 00000000..d3625768 --- /dev/null +++ b/content/de/administration/data/tiered-storage.md @@ -0,0 +1,179 @@ +--- +title: "Tiered Storage" +description: "Configure lifecycle-driven object transitions from RustFS to a remote storage tier." +--- + +RustFS tiered storage moves objects from local storage to a configured remote backend. This page explains the supported targets and shows how to add and maintain a RustFS tier in the Console. + +Tiering is asynchronous. RustFS keeps the object metadata locally, transfers the object data to the remote tier, and continues to serve S3 reads through the original bucket and object key. + +```mermaid +flowchart LR + A[Application] -->|S3 requests| H[Hot RustFS cluster] + H -->|Lifecycle transition| T[Remote tier bucket] + H -->|Read transitioned object| T +``` + +:::note[Tier names are not AWS storage classes] + +Lifecycle rules refer to a registered tier by its uppercase name, such as `COLDTIER`. Do not substitute AWS class labels such as `INTELLIGENT_TIERING`, `GLACIER`, or `DEEP_ARCHIVE` unless you have registered a RustFS tier with that exact valid name and verified the target behavior. + +::: + +## Supported backends + +The RustFS source defines warm-backend implementations for these target types: + +| Type | Configuration key | Typical target | +| --- | --- | --- | +| RustFS | `rustfs` | Another RustFS deployment | +| S3 | `s3` | Amazon S3 or an S3 endpoint supported by this backend | +| Wasabi | `wasabi` | Wasabi object storage | +| MinIO | `minio` | A MinIO deployment | +| Aliyun | `aliyun` | Alibaba Cloud Object Storage Service (OSS) | +| Tencent | `tencent` | Tencent Cloud Object Storage (COS) | +| Huaweicloud | `huaweicloud` | Huawei Cloud Object Storage Service (OBS) | +| Azure | `azure` | Azure Blob Storage | +| GCS | `gcs` | Google Cloud Storage | +| R2 | `r2` | Cloudflare R2 | + +Provider payloads and credential requirements differ. The complete workflow below uses the RustFS backend because the RustFS source includes an end-to-end hot-cluster-to-cold-cluster test for this path. + +## Before you begin + +Prepare the following: + +- A source RustFS deployment that stores the hot data. +- A separate target RustFS deployment and an existing target bucket. This example uses `my-bucket` on the target. +- Target credentials with permission to put, get, list, and delete objects in that bucket. +- Access to the source deployment's RustFS Console. + +Use TLS for both deployments in production. Restrict the target credentials to the dedicated tier bucket and prefix. + +## 1. Open Tiered Storage + +Sign in to the source deployment's RustFS Console. In the left navigation, select **Tiered Storage**, then select **Add Tier** in the upper-right corner. + +## 2. Select the target + +Select the target provider. This example uses **RustFS** to connect the source deployment to another RustFS deployment. + +## 3. Enter the target details + +Complete the form: + +| Field | Value | +| --- | --- | +| **Name (A-Z,0-9,_)** | Enter a unique uppercase tier name, such as `COLDTIER`. | +| **Endpoint** | Enter the target RustFS S3 endpoint. | +| **Access Key** | Enter the access key for the target deployment. | +| **Secret Key** | Enter the secret key for the target deployment. | +| **Bucket** | Enter the existing target bucket name, such as `my-bucket`. | +| **Prefix (Optional)** | Optionally enter a prefix dedicated to tiered objects. | +| **Region** | Optionally enter the target region, such as `us-east-1`. | + +Leave **Storage Class** at its default unless the target backend requires a different supported storage class. + +:::warning[Protect tier credentials] + +The form contains a secret key. Use credentials restricted to the target bucket and prefix. Do not expose the key in screenshots, tickets, or logs. + +::: + +## 4. Save the tier + +Select **Save**. RustFS validates the backend and probes it by writing, reading, and removing a small object. Saving fails when the endpoint, credentials, bucket permissions, or backend configuration cannot complete that probe. + +After the tier appears in the **Tiers** list, configure a transition rule in [Lifecycle Management](/administration/data/lifecycle-management). A registered tier does not move objects until a lifecycle rule references its name. + +## Manage tiers with `rc` + +Install and configure [`rc`](/operations/rc), then list the tiers registered on the source deployment: + +```bash +rc bucket lifecycle tier list local +``` + +Add a RustFS tier with the same settings described in the Console workflow: + +```bash +rc bucket lifecycle tier add rustfs COLDTIER local \ + --endpoint \ + --access-key \ + --secret-key \ + --bucket my-bucket \ + --region us-east-1 +``` + +Inspect the tier configuration and available statistics: + +```bash +rc bucket lifecycle tier info COLDTIER local +``` + +The remaining tier commands update credentials or remove a tier: + +```bash +rc bucket lifecycle tier edit COLDTIER local \ + --access-key \ + --secret-key +rc bucket lifecycle tier remove COLDTIER local +``` + +See [Lifecycle Management](/administration/data/lifecycle-management) to create transition rules, confirm transitions, and restore local copies. Run `rc bucket lifecycle tier --help` to inspect provider-specific options before changing a tier. + +## Monitor tier activity + +Use the tier statistics endpoint with a SigV4-signed request and `admin:ListTier` permission: + +```http +GET /rustfs/admin/v3/tier-stats?tier=COLDTIER HTTP/1.1 +Host: +``` + +Monitor transition failures together with source-cluster capacity and target-side availability. A configured tier adds the target service and network path to the read path for objects that do not have a restored local copy. + +## Change or remove a tier + +The Admin API exposes these mutation routes, all requiring `admin:SetTier`: + +| Operation | Route | +| --- | --- | +| Edit a tier | `POST /rustfs/admin/v3/tier/{tiername}` | +| Remove a tier | `DELETE /rustfs/admin/v3/tier/{tiername}` | +| Clear all tiers | `POST /rustfs/admin/v3/tier/clear` | + +Before editing or removing a tier: + +1. Disable lifecycle rules that reference the tier. +2. Confirm that no transition jobs are still using it. +3. Confirm that no source objects depend on data stored in its target bucket or prefix. +4. Back up the tier configuration and record the target location. + +Normal tier mutations check backend usage and protect non-empty targets. Do not use a `force` option to bypass those checks unless you have independently proved that every transitioned object remains recoverable. Removing configuration for an active tier can make transitioned objects unreadable from the source cluster. + +:::warning[Do not modify tier objects directly] + +Do not rename, overwrite, or delete generated objects in the target bucket. Manage source objects through the hot RustFS cluster so RustFS can keep local transition metadata and remote data consistent. + +::: + +## Admin API reference + +The current RustFS source registers these tier routes: + +| Method | Route | Permission | +| --- | --- | --- | +| `PUT` | `/rustfs/admin/v3/tier` | `admin:SetTier` | +| `POST` | `/rustfs/admin/v3/tier/{tiername}` | `admin:SetTier` | +| `DELETE` | `/rustfs/admin/v3/tier/{tiername}` | `admin:SetTier` | +| `POST` | `/rustfs/admin/v3/tier/clear` | `admin:SetTier` | +| `GET` | `/rustfs/admin/v3/tier` (list configurations) | `admin:ListTier` | +| `GET` | `/rustfs/admin/v3/tier/{tier}` (verify connectivity) | `admin:ListTier` | +| `GET` | `/rustfs/admin/v3/tier-stats` (read statistics) | `admin:ListTier` | + +All Admin API requests require SigV4 authentication. These routes are an administrative interface, not ordinary S3 bucket operations. + +## Next steps + +Review [lifecycle management](/administration/data/lifecycle-management) and configure [access policies](/security-compliance/iam/policies) for the administrators and service credentials used by tiered storage. \ No newline at end of file diff --git a/content/de/administration/index.md b/content/de/administration/index.md new file mode 100644 index 00000000..2c059b42 --- /dev/null +++ b/content/de/administration/index.md @@ -0,0 +1,16 @@ +--- +title: "Administration" +description: "Administer RustFS data, access protocols, and browser-based management." +--- + +Use this section to manage RustFS through the Console, administer buckets and objects, and configure access protocols. + +## Administration areas + +- [Console](./console/index.md) covers browser-based administration and sign-in methods. +- [Data Management](./data/object/object-lock.md) covers buckets, objects, lifecycle behavior, and data protection features. +- [Protocol Support](./protocols/s3.md) covers S3, WebDAV, FTPS, and SFTP access. +- [CORS Configuration](./cors/index.md) covers cross-origin access to RustFS services. +- [Virtual-Host Access](/integration/virtual) covers domain-based S3 addressing. + +For identity, encryption, transport security, and auditing, continue with [Security & Compliance](/security-compliance). diff --git a/content/de/administration/meta.json b/content/de/administration/meta.json new file mode 100644 index 00000000..f013a165 --- /dev/null +++ b/content/de/administration/meta.json @@ -0,0 +1,12 @@ +{ + "title": "Administration", + "icon": "Settings", + "defaultOpen": true, + "pages": [ + "[Console](/de/administration/console)", + "data", + "protocols", + "cors", + "[Virtual-Host-Zugriff](/de/integration/virtual)" + ] +} diff --git a/content/de/administration/protocols/ftps.md b/content/de/administration/protocols/ftps.md new file mode 100644 index 00000000..e7031a02 --- /dev/null +++ b/content/de/administration/protocols/ftps.md @@ -0,0 +1,246 @@ +--- +title: "FTP(S)" +description: "Configure the RustFS FTP or FTPS gateway and manage buckets and objects with FTP clients." +--- + +RustFS includes an FTP gateway that exposes buckets and objects to standard FTP clients. You can run it as unencrypted FTP for isolated local testing or as explicit FTP over TLS (FTPS) for encrypted connections. RustFS authenticates each session against Identity and Access Management (IAM) and applies the user's S3 policies to storage operations. + +FTP and FTPS support is compiled into the standard RustFS binary, but both listeners are disabled at runtime by default. Enable only the listener you intend to use. + +## Overview + +The gateway maps FTP paths to RustFS resources: + +| FTP path | RustFS resource | +| --- | --- | +| `/` | All buckets visible to the authenticated user | +| `/my-bucket/` | The `my-bucket` bucket | +| `/my-bucket/hello.txt` | The `hello.txt` object in `my-bucket` | + +The following FTP operations are supported: + +| FTP command | Operation | +| --- | --- | +| `LIST` | List buckets at the root or objects and prefixes in a bucket | +| `MKD` | Create a bucket | +| `CWD` | Enter a bucket | +| `STOR` | Upload an object | +| `RETR` | Download an object | +| `DELE` | Delete an object | +| `RMD` | Recursively delete a bucket and its objects | + +RustFS does not currently support FTP rename operations or uploads that append to an existing object. Because S3 storage has no native working-directory or POSIX directory model, some client-specific filesystem operations may not behave like a traditional FTP server. + +Enter a RustFS access key as the FTP username and its secret key as the password. Invalid usernames and passwords both return `530 Not logged in`. After login, RustFS checks the IAM user's S3 permissions for each operation. + +:::danger[Plain FTP exposes credentials] + +FTP sends credentials and data without encryption. Bind plain FTP to a loopback or isolated test network only. Use FTPS for every remote or production connection. + +::: + +:::warning[Bucket deletion is recursive] + +The current `RMD` implementation deletes the objects in the target bucket before deleting the bucket. Confirm the path before running `rmdir` in an FTP client. + +::: + +## Configuration + +FTP and FTPS use separate listeners and environment variables. + +### FTP variables + +| Variable | Description | Default | +| --- | --- | --- | +| `RUSTFS_FTP_ENABLE` | Enables the unencrypted FTP listener. | `false` | +| `RUSTFS_FTP_ADDRESS` | Bind address for FTP control connections. | `0.0.0.0:8021` | +| `RUSTFS_FTP_PASSIVE_PORTS` | Inclusive passive data port range in `start-end` format. | `40000-50000` | +| `RUSTFS_FTP_EXTERNAL_IP` | Public IP or hostname advertised to passive clients when RustFS is behind NAT. | Not set | + +### FTPS variables + +| Variable | Description | Default | +| --- | --- | --- | +| `RUSTFS_FTPS_ENABLE` | Enables the explicit FTPS listener. | `false` | +| `RUSTFS_FTPS_ADDRESS` | Bind address for FTPS control connections. | `0.0.0.0:8022` | +| `RUSTFS_FTPS_TLS_ENABLED` | Enables TLS on the FTPS listener. Keep this enabled for FTPS. | `true` | +| `RUSTFS_FTPS_CERTS_DIR` | Directory containing the FTPS certificate and private key. Required for FTPS. | Not set | +| `RUSTFS_FTPS_PASSIVE_PORTS` | Inclusive passive data port range in `start-end` format. | `40000-50000` | +| `RUSTFS_FTPS_EXTERNAL_IP` | Public IP or hostname advertised to passive clients when RustFS is behind NAT. | Not set | + +The standard RustFS build enables the `ftps` compile-time feature, which provides both FTP and FTPS. If you build RustFS with `--no-default-features`, include the feature explicitly: + +```bash +cargo build --release --features ftps +``` + +### Local testing with FTP + +Start an unencrypted FTP listener on the loopback interface: + +```bash +export RUSTFS_FTP_ENABLE=true +export RUSTFS_FTP_ADDRESS=127.0.0.1:8021 +export RUSTFS_FTP_PASSIVE_PORTS=40000-40010 +export RUSTFS_ACCESS_KEY= +export RUSTFS_SECRET_KEY= + +rustfs /path/to/data +``` + +Binding to `127.0.0.1` prevents remote hosts from connecting to the unencrypted listener. + +### Prepare a test certificate + +FTPS expects `rustfs_cert.pem` and `rustfs_key.pem` in the certificate directory. For local testing, create a short-lived self-signed certificate: + +```bash +mkdir -p /path/to/ftps-certs + +openssl req -x509 -newkey rsa:2048 -nodes \ + -keyout /path/to/ftps-certs/rustfs_key.pem \ + -out /path/to/ftps-certs/rustfs_cert.pem \ + -days 7 \ + -subj "/CN=localhost" \ + -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" +``` + +Use a certificate issued by a trusted certificate authority in production. Protect `rustfs_key.pem` from unauthorized access and ensure the certificate subject alternative names match the host clients use. + +### Start FTPS + +Configure an explicit FTPS listener: + +```bash +export RUSTFS_FTPS_ENABLE=true +export RUSTFS_FTPS_ADDRESS=0.0.0.0:8022 +export RUSTFS_FTPS_TLS_ENABLED=true +export RUSTFS_FTPS_CERTS_DIR=/path/to/ftps-certs +export RUSTFS_FTPS_PASSIVE_PORTS=40000-50000 + +rustfs /path/to/data +``` + +RustFS requires TLS on both the FTPS control and data channels. Configure clients for **explicit FTP over TLS**, sometimes labeled **FTPES** or **Require explicit FTP over TLS**. Implicit FTPS is not the mode implemented by this listener. + +If the certificate directory is missing, does not exist, or does not contain a usable certificate and key, FTPS initialization fails. See [Configure TLS](/integration/tls-configured) for general certificate guidance. + +### Configure passive connections + +FTP uses the control port for commands and a separate data connection for listings and file transfers. For passive mode: + +1. Allow inbound TCP traffic to the configured control port. +2. Allow inbound TCP traffic to every port in the configured passive range. +3. Set `RUSTFS_FTP_EXTERNAL_IP` or `RUSTFS_FTPS_EXTERNAL_IP` when clients connect through NAT or a load balancer. + +For example, an FTPS server behind NAT can advertise its public hostname: + +```bash +export RUSTFS_FTPS_EXTERNAL_IP=storage.example.com +export RUSTFS_FTPS_PASSIVE_PORTS=40000-40100 +``` + +Use the corresponding FTP-prefixed variables for the plain FTP listener. RustFS supports both active and passive transfer modes, but passive mode is generally easier to operate through client firewalls. + +## Usage + +The examples use [`lftp`](https://lftp.yar.ru/) and the canonical `my-bucket` and `hello.txt` names. The IAM user must have the S3 permissions required for each operation. + +### Connect with FTP + +Use plain FTP only with the loopback test listener: + +```bash +lftp -u ftp://127.0.0.1:8021 +``` + +Enter the secret key when `lftp` prompts for a password. + +### Connect with FTPS + +Connect to the explicit FTPS listener and require encryption for control and data connections: + +```bash +lftp -u -e ' + set ftp:ssl-force true; + set ftp:ssl-protect-data true; + open ftp://storage.example.com:8022 +' +``` + +Enter the secret key when prompted. Keep certificate verification enabled in production. + +For the self-signed local test certificate only, connect to `localhost` and disable certificate verification for that session: + +```bash +lftp -u -e ' + set ftp:ssl-force true; + set ftp:ssl-protect-data true; + set ssl:verify-certificate no; + open ftp://localhost:8022 +' +``` + +:::warning + +Do not disable certificate verification in production. Install the issuing CA in the client trust store instead. + +::: + +### List buckets and create a bucket + +At the `lftp` prompt, list the visible buckets and create `my-bucket`: + +```text +lftp> cls -1 / +lftp> mkdir my-bucket +lftp> cd my-bucket +``` + +Bucket names must follow the same naming rules as buckets created through the S3 API. + +### Upload and download an object + +Upload `/path/to/hello.txt`, list the bucket, and download the object: + +```text +lftp> put /path/to/hello.txt -o hello.txt +lftp> ls +lftp> get hello.txt -o hello.txt +``` + +The upload replaces an object with the same key. Appending to an existing object is not supported. + +### Delete an object and bucket + +Delete the object, return to the root, and delete the bucket: + +```text +lftp> rm hello.txt +lftp> cd / +lftp> rmdir my-bucket +``` + +Remember that `rmdir` recursively removes remaining objects from the bucket before deleting it. + +### Connect other clients + +Use these settings in graphical FTP clients such as FileZilla or Cyberduck: + +| Setting | FTP test listener | FTPS listener | +| --- | --- | --- | +| Protocol | FTP | FTP over TLS (explicit) | +| Host | `127.0.0.1` | Your FTPS hostname | +| Port | `8021` | `8022` | +| Username | RustFS access key | RustFS access key | +| Password | RustFS secret key | RustFS secret key | +| Transfer mode | Passive | Passive | + +If login succeeds but directory listings or transfers time out, check the passive port firewall rules and external IP setting first. + +## Next steps + +- [Manage credentials](/operations/credentials) +- [Configure TLS](/integration/tls-configured) +- [Check service status](/operations/status-check) \ No newline at end of file diff --git a/content/de/administration/protocols/meta.json b/content/de/administration/protocols/meta.json new file mode 100644 index 00000000..1c3e718c --- /dev/null +++ b/content/de/administration/protocols/meta.json @@ -0,0 +1,10 @@ +{ + "title": "Protokollunterstützung", + "pages": [ + "s3", + "webdav", + "ftps", + "sftp", + "[MCP-Server](/de/developer/mcp)" + ] +} \ No newline at end of file diff --git a/content/de/administration/protocols/s3.md b/content/de/administration/protocols/s3.md new file mode 100644 index 00000000..0148cd28 --- /dev/null +++ b/content/de/administration/protocols/s3.md @@ -0,0 +1,200 @@ +--- +title: "S3" +description: "Configure and use the RustFS S3-compatible API with AWS CLI, AWS SDKs, and other S3 clients." +--- + +RustFS exposes an S3-compatible REST API for common object-storage workloads. It supports AWS Signature Version 4 (SigV4) and works with AWS CLI, AWS SDKs, and other S3 clients when you configure the RustFS endpoint, region, credentials, and addressing style. + +## Overview + +S3 organizes data into buckets and objects. With the default path-style addressing, a RustFS URL maps to storage as follows: + +```text +http://localhost:9000/my-bucket/path/to/hello.txt + | bucket | |---- object key ----| +``` + +RustFS covers the common S3 data plane used by applications, backup tools, and SDKs: + +| Area | Supported workflows | +| --- | --- | +| Buckets | Create, delete, list, inspect, and retrieve location | +| Objects | Put, get, head, copy, delete, multi-delete, range reads, conditional requests, and user metadata | +| Listing | `ListObjects`, `ListObjectsV2`, prefixes, delimiters, markers, and pagination | +| Large objects | Create, upload, copy, list, complete, and abort multipart uploads | +| Data management | Versioning, lifecycle rules, bucket and object tags, checksums, and object lock | +| Access control | IAM credentials and policies, bucket policies, public access block, and presigned GET/PUT URLs | +| Integration | CORS, bucket notifications, replication configuration, and server-side encryption workflows | + +Authenticated clients sign requests with an access key and secret key. Anonymous requests are evaluated against the applicable bucket policy. + +:::note[Compatibility scope] + +RustFS provides broad S3 API compatibility for supported features, but it is not identical to every AWS S3 or MinIO API. Review [Known compatibility differences](#known-compatibility-differences) before depending on advanced or vendor-specific behavior, and validate it against your target RustFS release. + +::: + +## Configuration + +The S3 API is enabled on the main RustFS listener. It does not require a separate protocol switch. + +| Variable | Purpose | Default | +| --- | --- | --- | +| `RUSTFS_ADDRESS` | S3 API bind address | `:9000` | +| `RUSTFS_REGION` | Region used for request signing and region-aware clients | `us-east-1` | +| `RUSTFS_ACCESS_KEY` | Root access key | Installation-specific | +| `RUSTFS_SECRET_KEY` | Root secret key | Installation-specific | +| `RUSTFS_SERVER_DOMAINS` | Comma-separated domains for virtual-hosted-style requests | Not set | +| `RUSTFS_TLS_PATH` | Directory containing `rustfs_cert.pem` and `rustfs_key.pem` | Not set | + +Use IAM users or service accounts instead of root credentials for applications, and grant only the required S3 actions. + +### Path-style addressing + +Path-style addressing is the default and requires no DNS configuration. The bucket name is the first component of the request path: + +```text +http://localhost:9000/my-bucket/hello.txt +``` + +Configure clients with: + +- Endpoint: `http://localhost:9000` +- Region: `us-east-1` +- Access key: `` +- Secret key: `` +- Addressing style: path-style + +Some clients default to virtual-hosted-style requests. Enable their path-style option, such as `force_path_style=true` in AWS SDK configuration or `s3_use_path_style = true` in Terraform. + +### Virtual-hosted-style addressing + +Virtual-hosted-style addressing places the bucket name in the hostname. Set `RUSTFS_SERVER_DOMAINS` to each base domain that RustFS should recognize: + +```bash +export RUSTFS_SERVER_DOMAINS="s3.example.com" +``` + +The same object is then addressed as: + +```text +https://my-bucket.s3.example.com/hello.txt +``` + +Configure wildcard DNS, such as `*.s3.example.com`, to resolve to the RustFS endpoint. For HTTPS, the certificate must also cover the bucket hostnames. Bucket names containing dots may require explicit certificate names because a single-label wildcard does not span multiple labels. + +:::warning[Configure the server domain] + +Do not send virtual-hosted-style requests unless `RUSTFS_SERVER_DOMAINS` is configured. RustFS otherwise treats requests as path-style and cannot derive the bucket from the hostname. + +::: + +### TLS and network access + +For production, set `RUSTFS_TLS_PATH` to a directory containing `rustfs_cert.pem` and `rustfs_key.pem`, then use an `https://` endpoint. Ensure clients trust the issuing certificate authority. + +Allow the configured S3 API port through host firewalls and load balancers. The default is TCP port `9000`; this listener also carries RustFS administrative and internode traffic, so do not expose it anonymously without appropriate network and IAM controls. + +## Usage + +The following examples use AWS CLI with the canonical local endpoint. Install and configure [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) before continuing. + +### Configure AWS CLI + +```bash +aws configure +``` + +Enter these values when prompted: + +```text +AWS Access Key ID [None]: +AWS Secret Access Key [None]: +Default region name [None]: us-east-1 +Default output format [None]: json +``` + +Pass the RustFS endpoint to every command. The custom endpoint keeps these examples on RustFS instead of AWS S3. + +```bash +export RUSTFS_ENDPOINT="http://localhost:9000" +``` + +### Create and list buckets + +```bash +aws --endpoint-url "$RUSTFS_ENDPOINT" s3api create-bucket \ + --bucket my-bucket + +aws --endpoint-url "$RUSTFS_ENDPOINT" s3api list-buckets +``` + +### Upload and list objects + +```bash +printf 'Hello from RustFS\n' > hello.txt + +aws --endpoint-url "$RUSTFS_ENDPOINT" s3 cp \ + hello.txt s3://my-bucket/path/to/hello.txt + +aws --endpoint-url "$RUSTFS_ENDPOINT" s3 ls \ + s3://my-bucket/path/to/ +``` + +AWS CLI automatically uses multipart upload when its transfer configuration selects it for a large object. RustFS supports the standard multipart create, upload-part, complete, list, and abort workflows. + +### Download and inspect an object + +```bash +aws --endpoint-url "$RUSTFS_ENDPOINT" s3 cp \ + s3://my-bucket/path/to/hello.txt ./downloaded-hello.txt + +aws --endpoint-url "$RUSTFS_ENDPOINT" s3api head-object \ + --bucket my-bucket \ + --key path/to/hello.txt +``` + +### Create a presigned URL + +Generate a time-limited URL that can download an object without exposing credentials: + +```bash +aws --endpoint-url "$RUSTFS_ENDPOINT" s3 presign \ + s3://my-bucket/path/to/hello.txt \ + --expires-in 3600 +``` + +The hostname, scheme, port, region, and object path used by the recipient must match the values used to sign the URL. Use the externally reachable endpoint when generating URLs for another machine. + +### Delete objects and buckets + +```bash +aws --endpoint-url "$RUSTFS_ENDPOINT" s3 rm \ + s3://my-bucket/path/to/hello.txt + +aws --endpoint-url "$RUSTFS_ENDPOINT" s3api delete-bucket \ + --bucket my-bucket +``` + +An S3 bucket must be empty before it can be deleted. + +## Known compatibility differences + +Current RustFS compatibility tests cover the common workflows described above. The following areas are not equivalent to AWS S3 or MinIO: + +- Bucket access logging and bucket ownership controls are planned rather than complete. +- ACL authorization is intentionally unsupported. Canned ACL headers have partial compatibility, while XML grant policies return `NotImplemented`. Prefer IAM and bucket policies. +- POST Object form uploads are implemented, but checksum handling for that workflow is not complete. +- Some multipart upload listing and part-lookup edge cases are outside the default compatibility gate. +- S3 Select currently accepts only uncompressed input. +- Access Point and Outposts copy-source forms are not implemented. +- MinIO administrative APIs are a separate compatibility surface and should not be inferred from S3 data-plane support. + +Client behavior can also differ when a tool assumes AWS-specific services, storage classes, account ownership controls, or endpoint discovery. Always set the RustFS endpoint explicitly and test the exact operations your application uses. + +## Next steps + +- Follow the complete [AWS CLI example](/developer/examples/aws-cli). +- Choose an [S3 SDK](/developer/sdk) for application integration. +- Configure [credentials and access policies](/operations/credentials). +- Enable [TLS](/integration/tls-configured) before exposing the endpoint outside a trusted network. diff --git a/content/de/administration/protocols/sftp.md b/content/de/administration/protocols/sftp.md new file mode 100644 index 00000000..3579364f --- /dev/null +++ b/content/de/administration/protocols/sftp.md @@ -0,0 +1,181 @@ +--- +title: "SFTP" +description: "Build, configure, and use the RustFS SFTP gateway to access buckets and objects with SFTP clients." +--- + +RustFS includes an SSH File Transfer Protocol (SFTP) gateway that exposes buckets and objects to SFTP clients over an encrypted SSH connection. You can list and create buckets and upload, download, rename, or delete objects while RustFS enforces the permissions of the authenticated Identity and Access Management (IAM) user. + +SFTP is an optional compile-time feature and is not included in the default RustFS build. Build RustFS with the `sftp` or `full` feature before enabling the listener. The listener is disabled at runtime by default. + +## Overview + +The gateway maps SFTP paths to RustFS resources: + +| SFTP path | RustFS resource | +| --- | --- | +| `/` | All buckets visible to the authenticated user | +| `/my-bucket/` | The `my-bucket` bucket | +| `/my-bucket/hello.txt` | The `hello.txt` object in `my-bucket` | +| `/my-bucket/docs/hello.txt` | The `docs/hello.txt` object in `my-bucket` | + +Creating or removing a directory directly below `/` creates or removes a bucket. Directories below a bucket map to object key prefixes and do not exist as independent filesystem directories. + +RustFS supports the following operations through the SFTP gateway: + +| SFTP operation | RustFS operation | +| --- | --- | +| List `/` | List visible buckets | +| List a bucket or prefix | List objects and prefixes | +| `mkdir` at `/` | Create a bucket | +| `put` | Upload an object | +| `get` | Download an object | +| `rename` | Copy an object and then delete the source | +| `rm` | Delete an object | +| `rmdir` at `/` | Delete an empty bucket | + +Enter a RustFS access key as the SFTP username and its secret key as the password. RustFS checks the IAM user's S3 policies for every operation. + +:::warning[Password authentication only] + +The SFTP gateway does not support client public-key authentication or anonymous access. SSH host keys identify the server; they do not authenticate clients. Do not configure an `authorized_keys` file for RustFS SFTP. + +::: + +## Configuration + +### Build with SFTP support + +Build RustFS with the SFTP feature: + +```bash +cargo build --release --features sftp +``` + +To enable all optional RustFS features, including SFTP, use `--features full` instead. Setting `RUSTFS_SFTP_ENABLE=true` has no effect in a binary built without the `sftp` feature. + +### SFTP variables + +| Variable | Description | Default | +| --- | --- | --- | +| `RUSTFS_SFTP_ENABLE` | Enables the SFTP listener. | `false` | +| `RUSTFS_SFTP_ADDRESS` | Bind address for SFTP connections. | `0.0.0.0:2222` | +| `RUSTFS_SFTP_HOST_KEY_DIR` | Directory containing at least one unencrypted SSH private host key. Required when SFTP is enabled. | Not set | +| `RUSTFS_SFTP_READ_ONLY` | Rejects operations that modify buckets or objects. | `false` | +| `RUSTFS_SFTP_IDLE_TIMEOUT` | SSH inactivity timeout in seconds. Must be greater than zero. | `600` | +| `RUSTFS_SFTP_PART_SIZE` | Multipart upload part size in bytes. | `16777216` (16 MiB) | + +The default port `2222` avoids the privileged SSH port `22`. The address can use IPv4 or IPv6 syntax, for example `127.0.0.1:2222` or `[::]:2222`. + +### Prepare a host key + +Create a persistent host-key directory and generate an Ed25519 host key: + +```bash +mkdir -p /path/to/sftp-keys +ssh-keygen -t ed25519 \ + -f /path/to/sftp-keys/ssh_host_ed25519_key \ + -N "" +chmod 600 /path/to/sftp-keys/ssh_host_ed25519_key* +``` + +The host key must not use a passphrase. RustFS also accepts decodable ECDSA and RSA private host keys. + +On Unix, every regular file in the host-key directory must be accessible only by its owner. The `ssh-keygen` command also creates a `.pub` file; either keep it with owner-only permissions as shown or remove it because RustFS does not read it. RustFS refuses to start if the directory is missing, contains no usable private key, or contains a regular file with group or other permission bits set. + +:::warning[Protect the host key] + +Keep the same host key across restarts so clients can verify the server identity. Restrict access to the account that runs RustFS and back up the key securely. A changed host key causes clients to display a possible man-in-the-middle warning. + +::: + +### Start the SFTP listener + +The following example starts SFTP on the loopback interface for local testing: + +```bash +export RUSTFS_SFTP_ENABLE=true +export RUSTFS_SFTP_ADDRESS=127.0.0.1:2222 +export RUSTFS_SFTP_HOST_KEY_DIR=/path/to/sftp-keys +export RUSTFS_ACCESS_KEY= +export RUSTFS_SECRET_KEY= + +rustfs /path/to/data +``` + +For remote connections, bind to an appropriate network interface and allow inbound TCP traffic to the configured SFTP port. This listener is separate from the S3 API and Console listeners. + +## Usage + +The following examples use the OpenSSH `sftp` client, the canonical `my-bucket` and `hello.txt` names, and a local listener on port `2222`. + +### Connect + +```bash +sftp -P 2222 @127.0.0.1 +``` + +Enter the RustFS secret key when prompted for the password. On the first connection, verify the displayed host-key fingerprint before accepting it. + +### List and create buckets + +At the `sftp` prompt, list the buckets visible to the IAM user and create `my-bucket`: + +```text +sftp> ls / +sftp> mkdir /my-bucket +sftp> cd /my-bucket +``` + +Bucket names must follow the same naming rules as buckets created through the S3 API. Files cannot be created directly under `/`. + +### Upload and download an object + +Upload `/path/to/hello.txt`, list the bucket, and download the object: + +```text +sftp> put /path/to/hello.txt /my-bucket/hello.txt +sftp> ls /my-bucket +sftp> get /my-bucket/hello.txt hello.txt +``` + +Uploads must be sequential from the beginning of the file. Resume, append, in-place edits, and segmented multi-connection uploads of one object are not supported. + +### Rename an object + +```text +sftp> rename /my-bucket/hello.txt /my-bucket/greeting.txt +``` + +RustFS implements rename as a server-side copy followed by deletion of the source. The operation is not atomic, and renaming a bucket is not supported. The IAM user needs permission to read and delete the source and to write the destination. + +### Delete an object and bucket + +Delete the object and then remove the empty bucket: + +```text +sftp> rm /my-bucket/greeting.txt +sftp> rmdir /my-bucket +``` + +RustFS does not recursively delete a non-empty bucket through SFTP. + +### Connect a desktop client + +Use these settings in graphical clients such as FileZilla, Cyberduck, or WinSCP: + +| Setting | Value | +| --- | --- | +| Protocol | SFTP (SSH File Transfer Protocol) | +| Host | Your RustFS SFTP hostname | +| Port | `2222`, or the port in `RUSTFS_SFTP_ADDRESS` | +| Username | RustFS access key | +| Password | RustFS secret key | +| Authentication | Password | + +Configure the client for whole-file, single-connection uploads. Symbolic links and POSIX ownership, permission, or timestamp changes are not supported because the gateway maps SFTP operations to object storage. + +## Next steps + +- [Manage credentials](/operations/credentials) +- [Check service status](/operations/status-check) +- [Configure lifecycle management](/administration/data/lifecycle-management) \ No newline at end of file diff --git a/content/de/administration/protocols/webdav.md b/content/de/administration/protocols/webdav.md new file mode 100644 index 00000000..ab6a80ff --- /dev/null +++ b/content/de/administration/protocols/webdav.md @@ -0,0 +1,208 @@ +--- +title: "WebDAV" +description: "Configure the RustFS WebDAV gateway and access buckets and objects with WebDAV clients." +--- + +RustFS includes a Web Distributed Authoring and Versioning (WebDAV) gateway that exposes buckets and objects to WebDAV clients over HTTP or HTTPS. You can browse buckets, create collections, and upload, download, rename, or delete objects while RustFS enforces the permissions of the authenticated Identity and Access Management (IAM) user. + +WebDAV support is compiled into the standard RustFS binary, but the gateway is disabled at runtime by default. You must enable and configure it before connecting a client. + +## Overview + +The gateway maps WebDAV paths to RustFS resources: + +| WebDAV path | RustFS resource | +| --- | --- | +| `/` | All buckets visible to the authenticated user | +| `/my-bucket/` | The `my-bucket` bucket | +| `/my-bucket/hello.txt` | The `hello.txt` object in `my-bucket` | +| `/my-bucket/docs/hello.txt` | The `docs/hello.txt` object in `my-bucket` | + +RustFS supports the following operations through the gateway: + +| Method | Operation | +| --- | --- | +| `PROPFIND` | List buckets or objects and read metadata | +| `MKCOL` | Create a bucket or a directory prefix | +| `PUT` | Upload an object | +| `GET` | Download an object | +| `HEAD` | Read object metadata | +| `MOVE` | Rename or move an object or directory | +| `DELETE` | Delete an object, directory, or bucket | + +Use `PROPFIND`, not `GET`, to list a collection. The current gateway returns `405 Method Not Allowed` for `GET` requests to directories. + +WebDAV uses HTTP Basic authentication. Enter a RustFS access key as the username and its secret key as the password. The gateway authenticates the credentials against RustFS IAM and applies the user's S3 policies to each operation. + +:::warning[Protect credentials] + +Basic authentication does not encrypt credentials. Disable TLS only for isolated local testing. Use HTTPS for every remote or production connection. + +::: + +## Configuration + +Configure the gateway with environment variables before starting RustFS: + +| Variable | Description | Default | +| --- | --- | --- | +| `RUSTFS_WEBDAV_ENABLE` | Enables the WebDAV gateway. | `false` | +| `RUSTFS_WEBDAV_ADDRESS` | Bind address for WebDAV connections. | `0.0.0.0:8080` | +| `RUSTFS_WEBDAV_TLS_ENABLED` | Enables TLS for the WebDAV listener. | `true` | +| `RUSTFS_WEBDAV_CERTS_DIR` | Certificate directory used by the RustFS TLS runtime. Required when TLS is enabled. | Not set | +| `RUSTFS_WEBDAV_MAX_BODY_SIZE` | Maximum request body size in bytes. Must be greater than zero. | `5368709120` (5 GiB) | + +The standard RustFS build enables the WebDAV compile-time feature. If you build RustFS with `--no-default-features`, include the feature explicitly: + +```bash +cargo build --release --features webdav +``` + +### Local testing without TLS + +The following example starts RustFS with an HTTP WebDAV listener on port `8080`: + +```bash +export RUSTFS_WEBDAV_ENABLE=true +export RUSTFS_WEBDAV_ADDRESS=127.0.0.1:8080 +export RUSTFS_WEBDAV_TLS_ENABLED=false +export RUSTFS_ACCESS_KEY= +export RUSTFS_SECRET_KEY= + +rustfs /path/to/data +``` + +Binding to `127.0.0.1` prevents other hosts from connecting to the unencrypted test endpoint. + +### HTTPS + +For a remote or production connection, enable TLS and provide a certificate directory: + +```bash +export RUSTFS_WEBDAV_ENABLE=true +export RUSTFS_WEBDAV_ADDRESS=0.0.0.0:8080 +export RUSTFS_WEBDAV_TLS_ENABLED=true +export RUSTFS_WEBDAV_CERTS_DIR=/path/to/certs + +rustfs /path/to/data +``` + +If TLS is enabled without `RUSTFS_WEBDAV_CERTS_DIR`, or the directory does not exist, WebDAV initialization fails. See [Configure TLS](/integration/tls-configured) for certificate preparation guidance. + +Allow inbound TCP traffic to the configured WebDAV port. This listener is separate from the S3 API and Console listeners. + +## Usage + +The following commands use an HTTP endpoint for local testing. Replace the endpoint and credentials with your HTTPS WebDAV endpoint and an IAM user that has the required bucket and object permissions. + +Set reusable shell variables without placing the secret directly in each command: + +```bash +export WEBDAV_URL=http://127.0.0.1:8080 +export WEBDAV_USER= +read -s WEBDAV_PASSWORD +export WEBDAV_PASSWORD +``` + +### List buckets + +Send `PROPFIND` with `Depth: 1` to list the buckets visible to the user: + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --request PROPFIND \ + --header "Depth: 1" \ + "$WEBDAV_URL/" +``` + +### Create a bucket + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --request MKCOL \ + "$WEBDAV_URL/my-bucket/" +``` + +Bucket names must follow the same naming rules as buckets created through the S3 API. + +### Upload and download an object + +Upload `/path/to/hello.txt`: + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --upload-file /path/to/hello.txt \ + "$WEBDAV_URL/my-bucket/hello.txt" +``` + +Download the object: + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --output hello.txt \ + "$WEBDAV_URL/my-bucket/hello.txt" +``` + +### Create and list a directory + +WebDAV directories below a bucket map to object key prefixes: + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --request MKCOL \ + "$WEBDAV_URL/my-bucket/docs/" + +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --request PROPFIND \ + --header "Depth: 1" \ + "$WEBDAV_URL/my-bucket/docs/" +``` + +### Rename an object + +Use `MOVE` with a destination path on the same WebDAV endpoint: + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --request MOVE \ + --header "Destination: $WEBDAV_URL/my-bucket/greeting.txt" \ + "$WEBDAV_URL/my-bucket/hello.txt" +``` + +The IAM user needs permission to read and delete the source and to write the destination. If authorization fails, RustFS leaves the source unchanged. + +### Delete an object or bucket + +Delete an object: + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --request DELETE \ + "$WEBDAV_URL/my-bucket/greeting.txt" +``` + +Delete the bucket after removing its contents: + +```bash +curl --user "$WEBDAV_USER:$WEBDAV_PASSWORD" \ + --request DELETE \ + "$WEBDAV_URL/my-bucket/" +``` + +### Connect a desktop client + +Use the WebDAV endpoint in a client that supports Basic authentication: + +| Client | Connection address | +| --- | --- | +| GNOME Files | `dav://:8080/` for HTTP or `davs://:8080/` for HTTPS | +| macOS Finder | `http://:8080/` or `https://:8080/` | +| Windows File Explorer | `https://:8080/` | + +Enter the RustFS access key and secret key when the client prompts for credentials. Client behavior and supported WebDAV methods vary; use `curl` to isolate server-side errors when troubleshooting. + +## Next steps + +- [Manage credentials](/operations/credentials) +- [Configure TLS](/integration/tls-configured) +- [Check service status](/operations/status-check) \ No newline at end of file diff --git a/content/de/developer/examples/aws-cli.md b/content/de/developer/examples/aws-cli.md new file mode 100644 index 00000000..2e7dad35 --- /dev/null +++ b/content/de/developer/examples/aws-cli.md @@ -0,0 +1,73 @@ +--- +title: "AWS CLI" +description: "Connect the AWS CLI to RustFS and perform basic object operations." +--- + +The [AWS CLI](https://docs.aws.amazon.com/cli/) is Amazon's official command-line tool for S3, and it works with RustFS via the `--endpoint-url` flag. + +## Install + +Follow the [official install guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), or on macOS: + +```bash +brew install awscli +``` + +## Configure + +Set your [access keys](../../security-compliance/iam/access-token.md) and region: + +```bash +aws configure +``` + +```text +AWS Access Key ID [None]: +AWS Secret Access Key [None]: +Default region name [None]: us-east-1 +Default output format [None]: json +``` + +Pass your RustFS address with `--endpoint-url` on every command. Replace `http://localhost:9000` with your server address. When `--endpoint-url` is set, the AWS CLI uses path-style addressing, which is what RustFS requires. + +:::note + +If you did not set credentials at install, the local-test default is `rustfsadmin` / `rustfsadmin` — never use it beyond a throwaway local trial. + +::: + +## Verify + +Create a bucket: + +```bash +aws s3 mb s3://my-bucket --endpoint-url http://localhost:9000 +``` + +```text +make_bucket: my-bucket +``` + +Upload a file: + +```bash +aws s3 cp /path/to/hello.txt s3://my-bucket/ --endpoint-url http://localhost:9000 +``` + +```text +upload: ../path/to/hello.txt to s3://my-bucket/hello.txt +``` + +List the bucket: + +```bash +aws s3 ls s3://my-bucket --endpoint-url http://localhost:9000 +``` + +```text +2026-07-15 10:30:00 12 hello.txt +``` + +## Next steps + +Build applications against RustFS with an [S3 SDK](../sdk/index.md), or manage objects with [`rc`](/operations/rc). diff --git a/content/de/developer/examples/aws-sdk-go.md b/content/de/developer/examples/aws-sdk-go.md new file mode 100644 index 00000000..70958b90 --- /dev/null +++ b/content/de/developer/examples/aws-sdk-go.md @@ -0,0 +1,55 @@ +--- +title: "AWS SDK for Go" +description: "Connect the AWS SDK for Go v2 to RustFS and perform basic object operations." +--- + +The [AWS SDK for Go v2](https://aws.github.io/aws-sdk-go-v2/docs/) connects to RustFS through a custom base endpoint. This is the minimal connection recipe; see the [Go SDK guide](../sdk/go.md) for a full program. + +## Install + +```bash +go get github.com/aws/aws-sdk-go-v2/aws +go get github.com/aws/aws-sdk-go-v2/credentials +go get github.com/aws/aws-sdk-go-v2/service/s3 +``` + +## Configure + +Replace `http://localhost:9000` with your server address and use your own [access keys](../../security-compliance/iam/access-token.md). RustFS requires path-style addressing (`UsePathStyle: true`): + +```go {9,11} +import ( + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/credentials" + "github.com/aws/aws-sdk-go-v2/service/s3" +) + +cfg := aws.Config{ + Region: "us-east-1", + Credentials: aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider("", "", "")), +} +client := s3.NewFromConfig(cfg, func(o *s3.Options) { + o.BaseEndpoint = aws.String("http://localhost:9000") + o.UsePathStyle = true +}) +``` + +## Verify + +```go +ctx := context.Background() +client.CreateBucket(ctx, &s3.CreateBucketInput{Bucket: aws.String("my-bucket")}) + +out, _ := client.ListBuckets(ctx, &s3.ListBucketsInput{}) +for _, b := range out.Buckets { + fmt.Println(*b.Name) +} +``` + +```text +my-bucket +``` + +## Next steps + +See the full [Go SDK guide](../sdk/go.md), or manage objects with [`rc`](/operations/rc). diff --git a/content/de/developer/examples/aws-sdk-js.md b/content/de/developer/examples/aws-sdk-js.md new file mode 100644 index 00000000..408757a1 --- /dev/null +++ b/content/de/developer/examples/aws-sdk-js.md @@ -0,0 +1,46 @@ +--- +title: "AWS SDK for JavaScript" +description: "Connect the AWS SDK for JavaScript v3 to RustFS and perform basic object operations from Node.js." +--- + +The [AWS SDK for JavaScript v3](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/) connects to RustFS through a custom endpoint. This is the minimal connection recipe; see the [JavaScript SDK guide](../sdk/javascript.md) for a full program. + +## Install + +```bash +npm install @aws-sdk/client-s3 +``` + +## Configure + +Replace `http://localhost:9000` with your server address and use your own [access keys](../../security-compliance/iam/access-token.md). RustFS requires path-style addressing (`forcePathStyle: true`): + +```javascript title="index.mjs" {8} +import { S3Client, CreateBucketCommand, PutObjectCommand, ListObjectsV2Command } from "@aws-sdk/client-s3"; +import { readFileSync } from "node:fs"; + +const s3 = new S3Client({ + endpoint: "http://localhost:9000", + region: "us-east-1", + credentials: { accessKeyId: "", secretAccessKey: "" }, + forcePathStyle: true, +}); +``` + +## Verify + +```javascript +await s3.send(new CreateBucketCommand({ Bucket: "my-bucket" })); +await s3.send(new PutObjectCommand({ Bucket: "my-bucket", Key: "hello.txt", Body: readFileSync("/path/to/hello.txt") })); + +const out = await s3.send(new ListObjectsV2Command({ Bucket: "my-bucket" })); +for (const obj of out.Contents ?? []) console.log(obj.Key, obj.Size); +``` + +```text +hello.txt 12 +``` + +## Next steps + +See the full [JavaScript SDK guide](../sdk/javascript.md), or manage objects with [`rc`](/operations/rc). diff --git a/content/de/developer/examples/boto3.md b/content/de/developer/examples/boto3.md new file mode 100644 index 00000000..1f598b68 --- /dev/null +++ b/content/de/developer/examples/boto3.md @@ -0,0 +1,53 @@ +--- +title: "boto3 (Python)" +description: "Connect boto3 to RustFS and perform basic object operations from Python." +--- + +[boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) is the AWS SDK for Python and connects to RustFS through a custom endpoint. + +## Install + +```bash +pip install boto3 +``` + +## Configure + +Point the client at your RustFS endpoint. Replace `http://localhost:9000` with your server address, and use your own [access keys](../../security-compliance/iam/access-token.md). RustFS requires path-style addressing, set via botocore `Config`: + +```python +import boto3 +from botocore.config import Config + +s3 = boto3.client( + "s3", + endpoint_url="http://localhost:9000", + aws_access_key_id="", + aws_secret_access_key="", + region_name="us-east-1", + config=Config(s3={"addressing_style": "path"}), +) +``` + +## Verify + +Create a bucket, upload a file, and list the bucket: + +```python +s3.create_bucket(Bucket="my-bucket") + +s3.upload_file("/path/to/hello.txt", "my-bucket", "hello.txt") + +for obj in s3.list_objects_v2(Bucket="my-bucket").get("Contents", []): + print(obj["Key"], obj["Size"]) +``` + +Expected output: + +```text +hello.txt 12 +``` + +## Next steps + +See the [S3 SDK overview](../sdk/index.md) for more languages, or manage objects with [`rc`](/operations/rc). diff --git a/content/de/developer/examples/meta.json b/content/de/developer/examples/meta.json new file mode 100644 index 00000000..35a973ec --- /dev/null +++ b/content/de/developer/examples/meta.json @@ -0,0 +1,13 @@ +{ + "title": "Beispiele", + "pages": [ + "[rc](/de/operations/rc)", + "[AWS CLI](/de/developer/examples/aws-cli)", + "[s3cmd](/de/developer/examples/s3cmd)", + "[rclone](/de/developer/examples/rclone)", + "[boto3 (Python)](/de/developer/examples/boto3)", + "[AWS SDK for JavaScript](/de/developer/examples/aws-sdk-js)", + "[AWS SDK for Go](/de/developer/examples/aws-sdk-go)", + "[Terraform](/de/developer/examples/terraform)" + ] +} diff --git a/content/de/developer/examples/rclone.md b/content/de/developer/examples/rclone.md new file mode 100644 index 00000000..b93bedbf --- /dev/null +++ b/content/de/developer/examples/rclone.md @@ -0,0 +1,59 @@ +--- +title: "rclone" +description: "Connect rclone to RustFS and perform basic object operations." +--- + +[rclone](https://rclone.org/) is a command-line program for syncing files with cloud storage and speaks the S3 protocol that RustFS implements. + +## Install + +```bash +curl https://rclone.org/install.sh | sudo bash +``` + +Or see the [official install guide](https://rclone.org/install/). + +## Configure + +Add a remote to `~/.config/rclone/rclone.conf`. Replace `http://localhost:9000` with your server address, and use your own [access keys](../../security-compliance/iam/access-token.md). `force_path_style = true` is required because RustFS uses path-style addressing: + +```ini title="~/.config/rclone/rclone.conf" +[rustfs] +type = s3 +provider = Other +access_key_id = +secret_access_key = +endpoint = http://localhost:9000 +region = us-east-1 +force_path_style = true +``` + +## Verify + +Create a bucket: + +```bash +rclone mkdir rustfs:my-bucket +``` + +Upload a file: + +```bash +rclone copy /path/to/hello.txt rustfs:my-bucket +``` + +List buckets and contents: + +```bash +rclone lsd rustfs: +rclone ls rustfs:my-bucket +``` + +```text + -1 2026-07-15 10:30:00 -1 my-bucket + 12 hello.txt +``` + +## Next steps + +Build applications against RustFS with an [S3 SDK](../sdk/index.md), or manage objects with [`rc`](/operations/rc). diff --git a/content/de/developer/examples/s3cmd.md b/content/de/developer/examples/s3cmd.md new file mode 100644 index 00000000..16fbf31c --- /dev/null +++ b/content/de/developer/examples/s3cmd.md @@ -0,0 +1,56 @@ +--- +title: "s3cmd" +description: "Connect s3cmd to RustFS and perform basic object operations from the command line." +--- + +[s3cmd](https://s3tools.org/s3cmd) is a command-line client for S3-compatible storage. Point it at your RustFS endpoint with a small config file. + +## Install + +```bash +# macOS +brew install s3cmd +# Debian/Ubuntu +sudo apt install s3cmd +# or via pip +pip install s3cmd +``` + +## Configure + +Create `~/.s3cfg`. Replace `localhost:9000` with your server address and use your own [access keys](../../security-compliance/iam/access-token.md). RustFS uses path-style addressing, so set the bucket host to the same endpoint: + +```ini title="~/.s3cfg" +[default] +access_key = +secret_key = +host_base = localhost:9000 +host_bucket = localhost:9000 +use_https = False +signature_v2 = False +``` + +:::note + +Set `use_https = True` and point at port `9000` if you have [configured TLS](../../integration/tls-configured.md). + +::: + +## Verify + +Create a bucket, upload a file, and list it: + +```bash +s3cmd mb s3://my-bucket +s3cmd put /path/to/hello.txt s3://my-bucket/hello.txt +s3cmd ls s3://my-bucket +``` + +```text +upload: '/path/to/hello.txt' -> 's3://my-bucket/hello.txt' [1 of 1] +2026-07-16 10:00 12 s3://my-bucket/hello.txt +``` + +## Next steps + +See the [SDK overview](../sdk/index.md) to connect an application, or manage objects with [`rc`](/operations/rc). diff --git a/content/de/developer/examples/terraform.md b/content/de/developer/examples/terraform.md new file mode 100644 index 00000000..08b7ea2d --- /dev/null +++ b/content/de/developer/examples/terraform.md @@ -0,0 +1,56 @@ +--- +title: "Terraform" +description: "Manage RustFS buckets and objects with Terraform using the AWS provider pointed at a custom endpoint." +--- + +The [Terraform AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest) works against RustFS when you point its S3 endpoint at your server and enable path-style addressing. + +## Configure + +Replace `http://localhost:9000` with your server address and use your own [access keys](../../security-compliance/iam/access-token.md). The skip flags stop the provider from calling AWS-only metadata and STS endpoints: + +```hcl title="main.tf" +provider "aws" { + access_key = "" + secret_key = "" + region = "us-east-1" + s3_use_path_style = true + skip_credentials_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + + endpoints { + s3 = "http://localhost:9000" + } +} + +resource "aws_s3_bucket" "demo" { + bucket = "my-bucket" +} + +resource "aws_s3_object" "hello" { + bucket = aws_s3_bucket.demo.id + key = "hello.txt" + source = "/path/to/hello.txt" +} +``` + +## Apply + +```bash +terraform init +terraform apply +``` + +```text +Plan: 2 to add, 0 to change, 0 to destroy. +... +aws_s3_bucket.demo: Creation complete after 0s [id=my-bucket] +aws_s3_object.hello: Creation complete after 0s [id=hello.txt] + +Apply complete! Resources: 2 added, 0 changed, 0 destroyed. +``` + +## Next steps + +See the [SDK overview](../sdk/index.md) to connect an application, or the [AWS CLI example](aws-cli.md) for ad-hoc commands. diff --git a/content/de/developer/images/add-rustfs-mcp-succ.png b/content/de/developer/images/add-rustfs-mcp-succ.png new file mode 100644 index 00000000..c321d3c7 Binary files /dev/null and b/content/de/developer/images/add-rustfs-mcp-succ.png differ diff --git a/content/de/developer/images/list-rustfs-bucket-with-mcp.png b/content/de/developer/images/list-rustfs-bucket-with-mcp.png new file mode 100644 index 00000000..f59e288c Binary files /dev/null and b/content/de/developer/images/list-rustfs-bucket-with-mcp.png differ diff --git a/content/de/developer/index.md b/content/de/developer/index.md new file mode 100644 index 00000000..5141f84f --- /dev/null +++ b/content/de/developer/index.md @@ -0,0 +1,12 @@ +--- +title: Developer +description: "Integrate applications with RustFS by using STS, MCP, SDKs, and the S3-compatible API." +--- + +Use this section to integrate applications with RustFS through temporary credentials, the Model Context Protocol (MCP), SDKs, and the S3-compatible API. + +- [Security Token Service (STS)](/security-compliance/iam/sts) explains how to request temporary credentials. +- [MCP](/developer/mcp) connects AI tools and agents to RustFS. +- [SDKs](/developer/sdk) provide language-specific integration examples. +- [Integration guides](/developer/integration) cover proxies and data platforms. +- [S3 API](/administration/protocols/s3) documents client configuration and compatibility considerations. diff --git a/content/de/developer/integration/big-data/iceberg.md b/content/de/developer/integration/big-data/iceberg.md new file mode 100644 index 00000000..568a3699 --- /dev/null +++ b/content/de/developer/integration/big-data/iceberg.md @@ -0,0 +1,271 @@ +--- +title: "Apache Iceberg" +description: "Run Apache Iceberg with Spark, a REST catalog, and RustFS object storage using Docker Compose." +--- + +This guide runs **Apache Iceberg** with Spark, an Iceberg REST catalog, and **RustFS** as the S3-compatible warehouse. You will create an Iceberg table, write rows, query them, and verify that the table files are stored in RustFS. + +You need Docker with the Compose plugin and enough local resources to run four containers. This deployment is intended for local integration testing, not production. + +:::note[Upstream status] + +Apache Iceberg [PR #14928](https://github.com/apache/iceberg/pull/14928) demonstrated the same Spark, REST catalog, `S3FileIO`, and RustFS workflow, including table creation and writes. The pull request was closed without merging, and the current [Spark quickstart](https://iceberg.apache.org/spark-quickstart/#docker-compose) still uses another S3-compatible store. The configuration below therefore documents a RustFS integration rather than an Apache Iceberg default. + +::: + +## Architecture + +```mermaid +flowchart LR + Spark["Spark SQL"] -->|REST catalog API| Catalog["Iceberg REST catalog"] + Spark -->|S3FileIO| RustFS["RustFS S3 API"] + Catalog -->|S3FileIO| RustFS + Init["Bucket initializer"] -->|Create my-bucket| RustFS +``` + +Spark uses the REST service for catalog operations. Both Spark and the REST catalog receive the RustFS endpoint, region, credentials, and path-style setting so they can access metadata and data files in `s3://my-bucket/warehouse`. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-iceberg +cd rustfs-iceberg +``` + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use dedicated credentials for the warehouse bucket. Do not commit `.env` to source control. + +Create the Spark catalog configuration: + +```ini title="spark-defaults.conf" +spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions +spark.sql.catalog.demo org.apache.iceberg.spark.SparkCatalog +spark.sql.catalog.demo.type rest +spark.sql.catalog.demo.uri http://rest:8181 +spark.sql.catalog.demo.io-impl org.apache.iceberg.aws.s3.S3FileIO +spark.sql.catalog.demo.warehouse s3://my-bucket/warehouse +spark.sql.catalog.demo.s3.endpoint http://rustfs:9000 +spark.sql.catalog.demo.s3.path-style-access true +spark.sql.defaultCatalog demo +spark.sql.catalogImplementation in-memory +``` + +Path-style access is required for this container-network endpoint. The hostname `rustfs` is resolvable only inside the Compose network; clients running on the host use `http://localhost:9000` instead. + +Create the Compose file: + +```yaml title="compose.yaml" +services: + rustfs: + image: rustfs/rustfs:1.0.0-alpha.83 + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_ADDRESS: 0.0.0.0:9000 + RUSTFS_CONSOLE_ADDRESS: 0.0.0.0:9001 + RUSTFS_CONSOLE_ENABLE: "true" + volumes: + - rustfs-data:/data + ports: + - "9000:9000" + - "9001:9001" + networks: + - iceberg + + create-bucket: + image: rustfs/rc:latest + depends_on: + - rustfs + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + entrypoint: + - /bin/sh + - -c + - | + until /usr/bin/rc alias set rustfs http://rustfs:9000 "$${RUSTFS_ACCESS_KEY}" "$${RUSTFS_SECRET_KEY}"; do + echo "Waiting for RustFS..." + sleep 2 + done + /usr/bin/rc ls rustfs/my-bucket >/dev/null 2>&1 || /usr/bin/rc mb rustfs/my-bucket + networks: + - iceberg + + rest: + image: apache/iceberg-rest-fixture + depends_on: + create-bucket: + condition: service_completed_successfully + environment: + AWS_ACCESS_KEY_ID: ${RUSTFS_ACCESS_KEY} + AWS_SECRET_ACCESS_KEY: ${RUSTFS_SECRET_KEY} + AWS_REGION: us-east-1 + CATALOG_WAREHOUSE: s3://my-bucket/warehouse + CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO + CATALOG_S3_ENDPOINT: http://rustfs:9000 + CATALOG_S3_PATH__STYLE__ACCESS: "true" + ports: + - "8181:8181" + networks: + - iceberg + + spark-iceberg: + image: tabulario/spark-iceberg + depends_on: + create-bucket: + condition: service_completed_successfully + rest: + condition: service_started + environment: + AWS_ACCESS_KEY_ID: ${RUSTFS_ACCESS_KEY} + AWS_SECRET_ACCESS_KEY: ${RUSTFS_SECRET_KEY} + AWS_REGION: us-east-1 + volumes: + - ./spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro + ports: + - "8888:8888" + - "8080:8080" + networks: + - iceberg + +networks: + iceberg: + +volumes: + rustfs-data: +``` + +The [`rc` image](https://github.com/rustfs/cli) provides the official RustFS command-line client. The initializer checks for `my-bucket` before creating it, so repeated starts do not delete existing warehouse data. The RustFS volume preserves warehouse objects across container recreation. + +:::warning[Image versions] + +The Apache Iceberg quickstart images are published without stable version tags in the upstream example. Before using this pattern beyond local testing, pin every image to a tested tag or digest and validate the Spark, Iceberg runtime, and REST catalog versions together. + +::: + +## 2. Validate and start the deployment + +Resolve the Compose file before starting containers: + +```bash +docker compose config +``` + +Start the services and wait for the bucket initializer to finish: + +```bash +docker compose up -d +docker compose ps -a +``` + +The `create-bucket` service should show an exit code of `0`. Check its logs if it does not complete: + +```bash +docker compose logs create-bucket +``` + +Open the RustFS Console at `http://localhost:9001`. The REST catalog is available at `http://localhost:8181`, and the Spark notebook server is available at `http://localhost:8888`. + +## 3. Create and query an Iceberg table + +Start Spark SQL: + +```bash +docker compose exec spark-iceberg spark-sql +``` + +Create a namespace and a partitioned table: + +```sql +CREATE NAMESPACE IF NOT EXISTS demo.nyc; + +CREATE TABLE demo.nyc.taxis +( + vendor_id bigint, + trip_id bigint, + trip_distance float, + fare_amount double, + store_and_fwd_flag string +) +PARTITIONED BY (vendor_id); +``` + +Insert and query sample rows: + +```sql +INSERT INTO demo.nyc.taxis +VALUES + (1, 1000371, 1.8, 15.32, 'N'), + (2, 1000372, 2.5, 22.15, 'N'), + (2, 1000373, 0.9, 9.01, 'N'), + (1, 1000374, 8.4, 42.13, 'Y'); + +SELECT * FROM demo.nyc.taxis ORDER BY trip_id; +``` + +The query should return four rows: + +```text +1 1000371 1.8 15.32 N +2 1000372 2.5 22.15 N +2 1000373 0.9 9.01 N +1 1000374 8.4 42.13 Y +``` + +## 4. Verify objects in RustFS + +List the warehouse from the bucket-initializer image: + +```bash +docker compose run --rm --entrypoint /bin/sh create-bucket -c \ + '/usr/bin/rc alias set rustfs http://rustfs:9000 "$RUSTFS_ACCESS_KEY" "$RUSTFS_SECRET_KEY" >/dev/null && /usr/bin/rc find rustfs/my-bucket/warehouse' +``` + +The output should include Iceberg metadata and data objects below the `warehouse/nyc/taxis` prefix. You can also inspect the `my-bucket` bucket in the RustFS Console. + +## 5. Stop or reset the stack + +Stop the containers while keeping the RustFS data volume: + +```bash +docker compose down +``` + +To delete the local warehouse and start from an empty RustFS volume, explicitly include `--volumes`: + +```bash +docker compose down --volumes +``` + +## Troubleshooting + +### Spark cannot reach RustFS + +Use `http://rustfs:9000` inside Compose. `http://localhost:9000` refers to the Spark container itself when used in `spark-defaults.conf`. + +Confirm that `spark.sql.catalog.demo.s3.path-style-access` is `true`. Virtual-hosted requests require additional RustFS domain and DNS configuration. + +### The catalog returns an S3 error + +Check that the credentials in `.env` match the RustFS credentials and that the `create-bucket` service completed successfully: + +```bash +docker compose logs create-bucket rest +``` + +The REST catalog property uses doubled underscores in `CATALOG_IO__IMPL` and `CATALOG_S3_PATH__STYLE__ACCESS`; the fixture converts them to the dotted and hyphenated Iceberg property names. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Iceberg operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Apache Iceberg Spark documentation](https://iceberg.apache.org/docs/latest/spark-getting-started/) to configure your existing Spark environment. \ No newline at end of file diff --git a/content/de/developer/integration/big-data/index.md b/content/de/developer/integration/big-data/index.md new file mode 100644 index 00000000..89f9fe9f --- /dev/null +++ b/content/de/developer/integration/big-data/index.md @@ -0,0 +1,13 @@ +--- +title: "Big Data" +description: "Connect big data systems to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for big data systems that support an S3-compatible endpoint. + +## Systems + +- [Iceberg](./iceberg.md) +- [Milvus](./milvus.md) + +Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file diff --git a/content/de/developer/integration/big-data/meta.json b/content/de/developer/integration/big-data/meta.json new file mode 100644 index 00000000..c3f673eb --- /dev/null +++ b/content/de/developer/integration/big-data/meta.json @@ -0,0 +1,7 @@ +{ + "title": "Big Data", + "pages": [ + "iceberg", + "milvus" + ] +} \ No newline at end of file diff --git a/content/de/developer/integration/big-data/milvus.md b/content/de/developer/integration/big-data/milvus.md new file mode 100644 index 00000000..869be2d3 --- /dev/null +++ b/content/de/developer/integration/big-data/milvus.md @@ -0,0 +1,352 @@ +--- +title: "Milvus" +description: "Run Milvus with RustFS as its S3-compatible object storage backend using Docker Compose." +--- + +This guide runs **Milvus Standalone** with **RustFS** as its S3-compatible object storage backend. You will start Milvus, etcd, RustFS, and Attu; insert sample vectors; and verify that Milvus persists objects in RustFS. + +You need Docker with the Compose plugin and Python 3.9 or later. This deployment is intended for local integration testing, not production. + +:::note[Milvus configuration names] + +Milvus groups S3-compatible storage settings under the `minio` configuration key. The name does not require a MinIO server. In this guide, `minio.address` points to the RustFS service, and Milvus uses the S3 API exposed by RustFS. + +::: + +## Architecture + +```mermaid +flowchart LR + Client["Milvus client"] -->|gRPC :19530| Milvus["Milvus Standalone"] + Attu["Attu"] -->|gRPC :19530| Milvus + Milvus -->|Metadata| Etcd["etcd"] + Milvus -->|S3 API| RustFS["RustFS"] + Init["Bucket initializer"] -->|Create my-bucket| RustFS +``` + +Milvus stores service metadata in etcd and persists vector data, indexes, and related objects under `s3://my-bucket/milvus` in RustFS. The local Milvus volume remains necessary for runtime data and caches. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-milvus +cd rustfs-milvus +``` + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use dedicated credentials for the Milvus bucket. Do not commit `.env` to source control. + +Create the Milvus storage override: + +```yaml title="user.yaml" +common: + storageType: remote + +minio: + address: rustfs:9000 + port: 9000 + bucketName: my-bucket + rootPath: milvus + useSSL: false + useIAM: false + cloudProvider: aws + region: us-east-1 + useVirtualHost: false +``` + +`useVirtualHost: false` selects path-style S3 requests. The hostname `rustfs` resolves inside the Compose network; clients on the host use `http://localhost:9000`. + +Create the Compose file: + +```yaml title="compose.yaml" +services: + etcd: + image: quay.io/coreos/etcd:v3.5.18 + environment: + ETCD_AUTO_COMPACTION_MODE: revision + ETCD_AUTO_COMPACTION_RETENTION: "1000" + ETCD_QUOTA_BACKEND_BYTES: "4294967296" + ETCD_SNAPSHOT_COUNT: "50000" + command: + - etcd + - --advertise-client-urls=http://etcd:2379 + - --listen-client-urls=http://0.0.0.0:2379 + - --data-dir=/etcd + volumes: + - etcd-data:/etcd + healthcheck: + test: ["CMD", "etcdctl", "endpoint", "health"] + interval: 30s + timeout: 20s + retries: 3 + networks: + - milvus + + rustfs: + image: rustfs/rustfs:1.0.0-alpha.83 + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_ADDRESS: 0.0.0.0:9000 + RUSTFS_CONSOLE_ADDRESS: 0.0.0.0:9001 + RUSTFS_CONSOLE_ENABLE: "true" + volumes: + - rustfs-data:/data + ports: + - "9000:9000" + - "9001:9001" + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:9000/health/ready"] + interval: 10s + timeout: 5s + retries: 12 + start_period: 20s + networks: + - milvus + + create-bucket: + image: rustfs/rc:latest + depends_on: + rustfs: + condition: service_healthy + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + entrypoint: + - /bin/sh + - -c + - | + /usr/bin/rc alias set rustfs http://rustfs:9000 "$${RUSTFS_ACCESS_KEY}" "$${RUSTFS_SECRET_KEY}" \ + --region us-east-1 --bucket-lookup path + /usr/bin/rc bucket create rustfs/my-bucket --ignore-existing + networks: + - milvus + + standalone: + image: milvusdb/milvus:v2.6.0 + command: ["milvus", "run", "standalone"] + security_opt: + - seccomp:unconfined + depends_on: + etcd: + condition: service_healthy + create-bucket: + condition: service_completed_successfully + environment: + ETCD_ENDPOINTS: etcd:2379 + MINIO_ADDRESS: rustfs:9000 + MINIO_ACCESS_KEY_ID: ${RUSTFS_ACCESS_KEY} + MINIO_SECRET_ACCESS_KEY: ${RUSTFS_SECRET_KEY} + MINIO_REGION: us-east-1 + MQ_TYPE: woodpecker + volumes: + - milvus-data:/var/lib/milvus + - ./user.yaml:/milvus/configs/user.yaml:ro + ports: + - "19530:19530" + - "9091:9091" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"] + interval: 30s + timeout: 20s + retries: 3 + start_period: 90s + networks: + - milvus + + attu: + image: zilliz/attu:v2.6.5 + depends_on: + standalone: + condition: service_healthy + environment: + MILVUS_URL: standalone:19530 + ports: + - "8000:3000" + networks: + - milvus + +networks: + milvus: + +volumes: + etcd-data: + rustfs-data: + milvus-data: +``` + +The `create-bucket` service uses the official [`rc`](https://github.com/rustfs/cli) image and exits after ensuring that `my-bucket` exists. Named volumes preserve etcd metadata, RustFS objects, and Milvus local data when containers are recreated. + +:::warning[Protect local service ports] + +The Compose file publishes the RustFS API and Console, Milvus gRPC and health ports, and Attu on the host for local testing. Do not expose these ports to an untrusted network. Production deployments require scoped credentials, TLS, authentication, resource planning, backups, and independently operated dependencies. + +::: + +## 2. Validate and start the deployment + +Resolve the Compose file before starting containers: + +```bash +docker compose config +``` + +Start the services: + +```bash +docker compose up -d +docker compose ps -a +``` + +The `create-bucket` service should exit with code `0`, and `etcd`, `rustfs`, and `standalone` should become healthy. Inspect logs if a service does not reach its expected state: + +```bash +docker compose logs create-bucket rustfs standalone +``` + +Open these local interfaces: + +- RustFS Console: `http://localhost:9001` +- Attu: `http://localhost:8000` +- Milvus health endpoint: `http://localhost:9091/healthz` + +Attu connects to `standalone:19530` through the Compose network. If Attu asks for a connection address, use that service name instead of `localhost:19530`. + +## 3. Insert and query sample vectors + +Create a Python virtual environment and install the Milvus client version that matches the server release: + +```bash +python3 -m venv .venv +source .venv/bin/activate +python -m pip install "pymilvus==2.6.0" +``` + +Create a test script: + +```python title="verify_milvus.py" +from pymilvus import MilvusClient + +client = MilvusClient(uri="http://localhost:19530") +collection_name = "rustfs_demo" + +if client.has_collection(collection_name=collection_name): + client.drop_collection(collection_name=collection_name) + +client.create_collection( + collection_name=collection_name, + dimension=4, +) + +client.insert( + collection_name=collection_name, + data=[ + {"id": 1, "vector": [0.1, 0.2, 0.3, 0.4]}, + {"id": 2, "vector": [0.2, 0.3, 0.4, 0.5]}, + {"id": 3, "vector": [0.9, 0.8, 0.7, 0.6]}, + ], +) + +client.flush(collection_name=collection_name) +results = client.search( + collection_name=collection_name, + data=[[0.1, 0.2, 0.3, 0.4]], + limit=2, + output_fields=["id"], +) + +print(results) +client.close() +``` + +Run the script: + +```bash +python verify_milvus.py +``` + +The result should rank the row with ID `1` first. Open Attu and confirm that the `rustfs_demo` collection contains three entities. + +## 4. Verify Milvus objects in RustFS + +Use the bucket-initializer image to list objects below the configured `milvus` root path: + +```bash +docker compose run --rm --entrypoint /bin/sh create-bucket -c \ + '/usr/bin/rc alias set rustfs http://rustfs:9000 "$RUSTFS_ACCESS_KEY" "$RUSTFS_SECRET_KEY" --region us-east-1 --bucket-lookup path >/dev/null && /usr/bin/rc find rustfs/my-bucket/milvus' +``` + +The output should contain objects created by Milvus below the `milvus/` prefix. You can also open `my-bucket` in the RustFS Console. + +Milvus may buffer or compact data before every expected object appears. The successful insert, flush, query, and RustFS object listing together validate the integration path. + +## 5. Stop or reset the stack + +Stop the containers while retaining all named volumes: + +```bash +docker compose down +``` + +To delete the local test data, including the Milvus bucket contents and etcd metadata, explicitly remove the volumes: + +```bash +docker compose down --volumes +``` + +:::warning[Reset removes the test data] + +The `--volumes` option permanently deletes the named volumes used by this Compose project. Do not run it against data you need to retain. + +::: + +## Troubleshooting + +### Milvus cannot reach RustFS + +Use `rustfs:9000` as the S3 endpoint inside Compose. `localhost:9000` inside the Milvus container refers to that container, not RustFS. + +Confirm that `useVirtualHost` remains `false` and that the credential values passed to Milvus match the RustFS credentials: + +```bash +docker compose logs standalone rustfs +``` + +### The bucket initializer fails + +Check RustFS readiness and the initializer logs: + +```bash +curl -fsS http://localhost:9000/health/ready +docker compose logs create-bucket +``` + +Verify that `.env` contains non-empty credentials and that `docker compose config` resolves both variables. + +### Milvus starts without existing data + +Do not change `minio.bucketName`, `minio.rootPath`, or the etcd root for an existing deployment. Confirm that the `rustfs-data`, `etcd-data`, and `milvus-data` volumes still exist and that the same Compose project name is in use. + +### Attu cannot connect + +The Attu container must use `standalone:19530`. A browser or host-side client uses `localhost:19530`. Check Milvus health and Attu logs: + +```bash +curl -fsS http://localhost:9091/healthz +docker compose logs attu standalone +``` + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before enabling additional Milvus storage features. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Milvus documentation](https://milvus.io/docs) when adapting this local pattern to a managed or distributed deployment. \ No newline at end of file diff --git a/content/de/developer/integration/index.md b/content/de/developer/integration/index.md new file mode 100644 index 00000000..36086197 --- /dev/null +++ b/content/de/developer/integration/index.md @@ -0,0 +1,13 @@ +--- +title: "Integration" +description: "Integrate RustFS with reverse proxies and big data systems." +--- + +Use this section to connect **RustFS** to infrastructure and application platforms through its S3-compatible API. + +## Integration categories + +- [Reverse Proxy](./reverse-proxy/index.md) covers Nginx, Traefik, Caddy, and HAProxy. +- [Big Data](./big-data/index.md) covers Iceberg. + +Each guide identifies the RustFS endpoint and addressing requirements to use when configuring the integrating system. \ No newline at end of file diff --git a/content/de/developer/integration/meta.json b/content/de/developer/integration/meta.json new file mode 100644 index 00000000..96a0425f --- /dev/null +++ b/content/de/developer/integration/meta.json @@ -0,0 +1,7 @@ +{ + "title": "Integration", + "pages": [ + "reverse-proxy", + "big-data" + ] +} \ No newline at end of file diff --git a/content/de/developer/integration/reverse-proxy/caddy.md b/content/de/developer/integration/reverse-proxy/caddy.md new file mode 100644 index 00000000..57c1a55b --- /dev/null +++ b/content/de/developer/integration/reverse-proxy/caddy.md @@ -0,0 +1,207 @@ +--- +title: "Caddy" +description: "Deploy RustFS behind Caddy with automatic HTTPS for separate S3 API and Console hostnames." +--- + +Use **Caddy** to obtain and renew TLS certificates automatically and route separate hostnames to the RustFS S3 API and Console. This deployment runs Caddy and a single-node RustFS instance with Docker Compose. You need Docker Engine, Docker Compose, a public server, and two DNS records that resolve to that server. + +This guide uses these example hostnames: + +- `s3.example.com` for the S3 API +- `console.example.com` for the Console + +Replace them with your public hostnames. Caddy's default ACME challenges require inbound access to ports `80` and `443`. + +:::warning[Serve S3 from the root path] + +Do not publish the S3 API under a path such as `/s3/`. AWS Signature Version 4 includes the request path and host, so rewriting either value can invalidate signed requests. + +::: + +## 1. Create the deployment directory + +Create a directory for the deployment: + +```bash +mkdir rustfs-caddy +cd rustfs-caddy +``` + +## 2. Set deployment variables + +Create an environment file and replace each value: + +```ini title=".env" +S3_HOSTNAME=s3.example.com +CONSOLE_HOSTNAME=console.example.com +ACME_EMAIL=admin@example.com +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use an email address that receives certificate notices. Do not commit `.env` to source control. + +## 3. Configure Caddy + +Create a Caddyfile with one site block for each RustFS endpoint: + +```text title="Caddyfile" +{ + email {$ACME_EMAIL} +} + +{$S3_HOSTNAME} { + reverse_proxy rustfs:9000 { + health_uri /health/ready + health_interval 10s + health_timeout 5s + health_fails 3 + health_passes 2 + lb_try_duration 5s + } +} + +{$CONSOLE_HOSTNAME} { + reverse_proxy rustfs:9001 { + health_uri /rustfs/console/health + health_interval 10s + health_timeout 5s + health_fails 3 + health_passes 2 + lb_try_duration 5s + } +} +``` + +Caddy preserves the incoming `Host` header, HTTP method, and request URI by default. It also forwards client information through `X-Forwarded-*` headers and handles Console WebSocket upgrades without additional header rules. + +## 4. Create the Compose file + +Create the Caddy and RustFS services: + +```yaml title="compose.yaml" +services: + caddy: + image: caddy:2.10-alpine + restart: unless-stopped + depends_on: + rustfs: + condition: service_healthy + environment: + S3_HOSTNAME: ${S3_HOSTNAME} + CONSOLE_HOSTNAME: ${CONSOLE_HOSTNAME} + ACME_EMAIL: ${ACME_EMAIL} + ports: + - "80:80" + - "443:443" + - "443:443/udp" + volumes: + - ./Caddyfile:/etc/caddy/Caddyfile:ro + - caddy-data:/data + - caddy-config:/config + networks: + - rustfs + + rustfs: + image: rustfs/rustfs:latest + restart: unless-stopped + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_ADDRESS: "0.0.0.0:9000" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" + expose: + - "9000" + - "9001" + volumes: + - rustfs-data:/data + healthcheck: + test: ["CMD-SHELL", "curl --fail http://127.0.0.1:9000/health/ready && curl --fail http://127.0.0.1:9001/rustfs/console/health"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + networks: + - rustfs + +volumes: + caddy-data: + caddy-config: + rustfs-data: + +networks: + rustfs: +``` + +The persistent `caddy-data` volume stores certificates, private keys, and ACME account state. Back up this volume and do not share its contents. Only Caddy publishes host ports; RustFS remains reachable inside the Compose network. + +## 5. Validate and start the deployment + +Render the Compose configuration and start RustFS: + +```bash +docker compose config +docker compose up -d rustfs +``` + +Validate the Caddyfile with the same image used by the deployment: + +```bash +docker compose run --rm --no-deps caddy caddy validate --config /etc/caddy/Caddyfile +``` + +Start Caddy and check both services: + +```bash +docker compose up -d caddy +docker compose ps +docker compose logs --follow caddy +``` + +Caddy obtains certificates in the background and redirects HTTP requests to HTTPS. If issuance fails, confirm that both DNS records resolve to this host, ports `80` and `443` are reachable, and the `caddy-data` volume is writable. + +## 6. Verify both endpoints + +Verify the API and Console through their public HTTPS hostnames: + +```bash +curl --fail https://s3.example.com/health/ready +curl --fail https://console.example.com/rustfs/console/health +``` + +Configure S3 clients with `https://s3.example.com` as the endpoint and enable path-style addressing. Open `https://console.example.com` to sign in to the Console. + +## Multi-node upstreams + +For a distributed RustFS deployment, list every node in the corresponding site block: + +```text title="Caddyfile" +{$S3_HOSTNAME} { + reverse_proxy node1.example.net:9000 node2.example.net:9000 node3.example.net:9000 node4.example.net:9000 { + lb_policy least_conn + health_uri /health/ready + health_interval 10s + health_timeout 5s + lb_try_duration 5s + } +} + +{$CONSOLE_HOSTNAME} { + reverse_proxy node1.example.net:9001 node2.example.net:9001 node3.example.net:9001 node4.example.net:9001 { + lb_policy cookie rustfs_console + health_uri /rustfs/console/health + health_interval 10s + health_timeout 5s + lb_try_duration 5s + } +} +``` + +Replace `` with a random secret shared by all Caddy instances. Console affinity keeps an in-progress OpenID Connect login on the RustFS node that created its `state`. Keep port `9000` open directly between RustFS nodes because internal node RPC uses the same listener. + +## Next steps + +- [Configure an S3 client](/developer/examples/aws-cli) +- [Enable virtual-hosted-style bucket URLs](/integration/virtual) +- [Review health and readiness endpoints](/operations/status-check) \ No newline at end of file diff --git a/content/de/developer/integration/reverse-proxy/haproxy.md b/content/de/developer/integration/reverse-proxy/haproxy.md new file mode 100644 index 00000000..faac09ef --- /dev/null +++ b/content/de/developer/integration/reverse-proxy/haproxy.md @@ -0,0 +1,234 @@ +--- +title: "HAProxy" +description: "Deploy RustFS behind HAProxy with separate HTTPS routes for the S3 API and Console." +--- + +Use **HAProxy** to terminate TLS and route separate hostnames to the RustFS S3 API and Console. This deployment runs HAProxy and a single-node RustFS instance with Docker Compose. You need Docker Engine, Docker Compose, two DNS records, and a TLS certificate that covers both hostnames. + +This guide uses these example hostnames: + +- `s3.example.com` for the S3 API +- `console.example.com` for the Console + +Replace them with hostnames that resolve to the Docker host. + +:::warning[Serve S3 from the root path] + +Do not publish the S3 API under a path such as `/s3/`. AWS Signature Version 4 includes the request path and host, so rewriting either value can invalidate signed requests. + +::: + +## 1. Create the deployment directories + +Create directories for the HAProxy configuration and TLS certificate: + +```bash +mkdir -p rustfs-haproxy/config rustfs-haproxy/certs +cd rustfs-haproxy +``` + +HAProxy expects the certificate chain and private key in one PEM file. Combine them in this order: + +```bash +cat fullchain.pem privkey.pem > certs/rustfs.pem +chmod 600 certs/rustfs.pem +``` + +The certificate must cover both public hostnames. + +## 2. Set RustFS credentials + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Do not commit this file or the certificate private key to source control. + +## 3. Configure HAProxy + +Create the HAProxy configuration: + +```text title="config/haproxy.cfg" +global + log stdout format raw local0 + +defaults + log global + mode http + option httplog + timeout connect 10s + timeout client 1h + timeout server 1h + timeout http-request 30s + timeout tunnel 1h + +frontend http + bind :80 + http-request redirect scheme https code 301 + +frontend https + bind :443 ssl crt /usr/local/etc/haproxy/certs/rustfs.pem alpn h2,http/1.1 + + acl host_s3 hdr(host) -i s3.example.com + acl host_console hdr(host) -i console.example.com + + use_backend rustfs_s3 if host_s3 + use_backend rustfs_console if host_console + default_backend reject_unknown_host + +backend reject_unknown_host + http-request deny deny_status 404 + +backend rustfs_s3 + balance leastconn + option httpchk GET /health/ready + http-check expect status 200 + server rustfs rustfs:9000 check inter 10s fall 3 rise 2 + +backend rustfs_console + balance leastconn + cookie RUSTFS_CONSOLE insert indirect nocache secure httponly + option httpchk GET /rustfs/console/health + http-check expect status 200 + server rustfs rustfs:9001 check inter 10s fall 3 rise 2 cookie rustfs +``` + +HAProxy preserves the incoming host and request path unless you explicitly rewrite them. The long client, server, and tunnel timeouts accommodate streaming S3 operations and Console WebSocket connections. + +The Console backend sets an affinity cookie. With one RustFS server it has no routing effect, but keeping it in the base configuration makes the behavior consistent when you add nodes. + +## 4. Create the Compose file + +Create the HAProxy and RustFS services: + +```yaml title="compose.yaml" +services: + haproxy: + image: haproxy:3.2-alpine + restart: unless-stopped + depends_on: + rustfs: + condition: service_healthy + ports: + - "80:80" + - "443:443" + volumes: + - ./config/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro + - ./certs:/usr/local/etc/haproxy/certs:ro + networks: + - rustfs + + rustfs: + image: rustfs/rustfs:latest + restart: unless-stopped + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_ADDRESS: "0.0.0.0:9000" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" + expose: + - "9000" + - "9001" + volumes: + - rustfs-data:/data + healthcheck: + test: ["CMD-SHELL", "curl --fail http://127.0.0.1:9000/health/ready && curl --fail http://127.0.0.1:9001/rustfs/console/health"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + networks: + - rustfs + +volumes: + rustfs-data: + +networks: + rustfs: +``` + +Only HAProxy publishes host ports. RustFS ports `9000` and `9001` remain reachable inside the Compose network. + +## 5. Validate and start the deployment + +Render the Compose configuration and start RustFS: + +```bash +docker compose config +docker compose up -d rustfs +``` + +Validate the HAProxy configuration with the same image used by the deployment: + +```bash +docker compose run --rm --no-deps haproxy haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg +``` + +Start HAProxy and check both services: + +```bash +docker compose up -d haproxy +docker compose ps +``` + +If a service does not become healthy, inspect its logs: + +```bash +docker compose logs haproxy +docker compose logs rustfs +``` + +## 6. Verify both endpoints + +Verify the API and Console through their public HTTPS hostnames: + +```bash +curl --fail https://s3.example.com/health/ready +curl --fail https://console.example.com/rustfs/console/health +``` + +Configure S3 clients with `https://s3.example.com` as the endpoint and enable path-style addressing. Open `https://console.example.com` to sign in to the Console. + +When you replace a renewed `certs/rustfs.pem`, validate the configuration and recreate the HAProxy container to load it: + +```bash +docker compose run --rm --no-deps haproxy haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg +docker compose up -d --force-recreate haproxy +``` + +## Multi-node backends + +For a distributed RustFS deployment, add every RustFS node to both backends: + +```text title="config/haproxy.cfg" +backend rustfs_s3 + balance leastconn + option httpchk GET /health/ready + http-check expect status 200 + server node1 node1.example.net:9000 check inter 10s fall 3 rise 2 + server node2 node2.example.net:9000 check inter 10s fall 3 rise 2 + server node3 node3.example.net:9000 check inter 10s fall 3 rise 2 + server node4 node4.example.net:9000 check inter 10s fall 3 rise 2 + +backend rustfs_console + balance leastconn + cookie RUSTFS_CONSOLE insert indirect nocache secure httponly + option httpchk GET /rustfs/console/health + http-check expect status 200 + server node1 node1.example.net:9001 check inter 10s fall 3 rise 2 cookie node1 + server node2 node2.example.net:9001 check inter 10s fall 3 rise 2 cookie node2 + server node3 node3.example.net:9001 check inter 10s fall 3 rise 2 cookie node3 + server node4 node4.example.net:9001 check inter 10s fall 3 rise 2 cookie node4 +``` + +The Console cookie keeps an in-progress OpenID Connect login on the RustFS node that created its `state`. Keep port `9000` open directly between RustFS nodes because internal node RPC uses the same listener. + +## Next steps + +- [Configure an S3 client](/developer/examples/aws-cli) +- [Enable virtual-hosted-style bucket URLs](/integration/virtual) +- [Review health and readiness endpoints](/operations/status-check) \ No newline at end of file diff --git a/content/de/developer/integration/reverse-proxy/index.md b/content/de/developer/integration/reverse-proxy/index.md new file mode 100644 index 00000000..d7b0c617 --- /dev/null +++ b/content/de/developer/integration/reverse-proxy/index.md @@ -0,0 +1,19 @@ +--- +title: "Reverse Proxy" +description: "Choose and configure a reverse proxy for the RustFS S3 API and Console." +--- + +Use a reverse proxy to expose the **RustFS** S3 API and Console through managed hostnames and TLS endpoints. + +We recommend using separate hostnames for the S3 API on port `9000` and the Console on port `9001`. Serve the S3 API from the root of its hostname because S3 clients sign the request path. + +## Supported guides + +- [Nginx](./nginx.md) +- [Traefik](./traefik.md) +- [Caddy](./caddy.md) +- [HAProxy](./haproxy.md) + +## Related configuration + +See [Virtual-Host Access](/integration/virtual) when clients access buckets through virtual-hosted-style URLs. \ No newline at end of file diff --git a/content/de/developer/integration/reverse-proxy/meta.json b/content/de/developer/integration/reverse-proxy/meta.json new file mode 100644 index 00000000..ea7175e0 --- /dev/null +++ b/content/de/developer/integration/reverse-proxy/meta.json @@ -0,0 +1,9 @@ +{ + "title": "Reverse Proxy", + "pages": [ + "nginx", + "traefik", + "caddy", + "haproxy" + ] +} \ No newline at end of file diff --git a/content/de/developer/integration/reverse-proxy/nginx.md b/content/de/developer/integration/reverse-proxy/nginx.md new file mode 100644 index 00000000..12eb5230 --- /dev/null +++ b/content/de/developer/integration/reverse-proxy/nginx.md @@ -0,0 +1,267 @@ +--- +title: "Nginx" +description: "Deploy RustFS behind Nginx with separate HTTPS endpoints for the S3 API and Console." +--- + +Use **Nginx** to terminate TLS and route separate hostnames to the RustFS S3 API and Console. This deployment runs Nginx and a single-node RustFS instance with Docker Compose. You need Docker Engine, Docker Compose, two DNS records, and a TLS certificate that covers both hostnames. + +This guide uses these example hostnames: + +- `s3.example.com` for the S3 API +- `console.example.com` for the Console + +Replace them with hostnames that resolve to the Docker host. + +:::warning[Serve S3 from the root path] + +Do not publish the S3 API under a path such as `/s3/`. AWS Signature Version 4 includes the request path and host, so rewriting either value can invalidate signed requests. + +::: + +## 1. Create the deployment directories + +Create directories for the Nginx configuration and TLS certificate: + +```bash +mkdir -p rustfs-nginx/sites rustfs-nginx/certs +cd rustfs-nginx +``` + +Copy your certificate chain and private key into `certs/`: + +```text +rustfs-nginx/ +├── certs/ +│ ├── fullchain.pem +│ └── privkey.pem +└── sites/ +``` + +Restrict access to the private key: + +```bash +chmod 600 certs/privkey.pem +``` + +## 2. Set RustFS credentials + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Do not commit this file to source control. + +## 3. Configure Nginx + +Create the Nginx configuration: + +```nginx title="sites/rustfs.conf" +map $http_upgrade $connection_upgrade { + default upgrade; + '' ''; +} + +upstream rustfs_s3 { + server rustfs:9000; + keepalive 32; +} + +upstream rustfs_console { + server rustfs:9001; + keepalive 16; +} + +server { + listen 80; + listen [::]:80; + server_name s3.example.com console.example.com; + + return 301 https://$host$request_uri; +} + +server { + listen 443 ssl; + listen [::]:443 ssl; + http2 on; + server_name s3.example.com; + + ssl_certificate /etc/nginx/certs/fullchain.pem; + ssl_certificate_key /etc/nginx/certs/privkey.pem; + + ignore_invalid_headers off; + client_max_body_size 0; + proxy_buffering off; + proxy_request_buffering off; + + location / { + proxy_http_version 1.1; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Connection ''; + proxy_cache_convert_head off; + proxy_connect_timeout 300s; + chunked_transfer_encoding off; + proxy_pass http://rustfs_s3; + } +} + +server { + listen 443 ssl; + listen [::]:443 ssl; + http2 on; + server_name console.example.com; + + ssl_certificate /etc/nginx/certs/fullchain.pem; + ssl_certificate_key /etc/nginx/certs/privkey.pem; + + client_max_body_size 0; + proxy_buffering off; + proxy_request_buffering off; + + location / { + proxy_http_version 1.1; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_connect_timeout 300s; + proxy_pass http://rustfs_console; + } +} +``` + +The S3 server preserves the original host and request path, disables request buffering for streaming uploads, and does not convert signed `HEAD` requests. The Console server also forwards WebSocket upgrade headers. + +## 4. Create the Compose file + +Create the deployment definition: + +```yaml title="compose.yaml" +services: + nginx: + image: nginx:1.25-alpine + restart: unless-stopped + depends_on: + rustfs: + condition: service_healthy + ports: + - "80:80" + - "443:443" + volumes: + - ./sites/rustfs.conf:/etc/nginx/conf.d/rustfs.conf:ro + - ./certs:/etc/nginx/certs:ro + networks: + - rustfs + + rustfs: + image: rustfs/rustfs:latest + restart: unless-stopped + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_ADDRESS: "0.0.0.0:9000" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" + expose: + - "9000" + - "9001" + volumes: + - rustfs-data:/data + healthcheck: + test: ["CMD-SHELL", "curl --fail http://127.0.0.1:9000/health/ready && curl --fail http://127.0.0.1:9001/rustfs/console/health"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + networks: + - rustfs + +volumes: + rustfs-data: + +networks: + rustfs: +``` + +Only Nginx publishes host ports. RustFS ports `9000` and `9001` remain reachable inside the Compose network. + +## 5. Validate and start the deployment + +Validate both configuration files before starting the services: + +```bash +docker compose config +docker compose up -d rustfs +docker compose run --rm --no-deps nginx nginx -t +``` + +Start Nginx and check both services: + +```bash +docker compose up -d nginx +docker compose ps +``` + +If a service does not become healthy, inspect its logs: + +```bash +docker compose logs nginx +docker compose logs rustfs +``` + +## 6. Verify both endpoints + +Verify the API and Console through their public HTTPS hostnames: + +```bash +curl --fail https://s3.example.com/health/ready +curl --fail https://console.example.com/rustfs/console/health +``` + +Configure S3 clients with `https://s3.example.com` as the endpoint and enable path-style addressing. Open `https://console.example.com` to sign in to the Console. + +When you replace a renewed certificate or key in `certs/`, validate and reload Nginx without interrupting active connections: + +```bash +docker compose exec nginx nginx -t +docker compose exec nginx nginx -s reload +``` + +## Multi-node upstreams + +For a distributed RustFS deployment, replace the single server in each upstream with all RustFS nodes: + +```nginx title="sites/rustfs.conf" +upstream rustfs_s3 { + least_conn; + server node1.example.net:9000; + server node2.example.net:9000; + server node3.example.net:9000; + server node4.example.net:9000; + keepalive 32; +} + +upstream rustfs_console { + ip_hash; + server node1.example.net:9001; + server node2.example.net:9001; + server node3.example.net:9001; + server node4.example.net:9001; + keepalive 16; +} +``` + +The Console upstream uses client affinity because an in-progress OpenID Connect login stores its `state` on one RustFS node. Keep port `9000` open between RustFS nodes because internal node RPC uses the same listener. + +## Next steps + +- [Configure an S3 client](/developer/examples/aws-cli) +- [Enable virtual-hosted-style bucket URLs](/integration/virtual) +- [Review health and readiness endpoints](/operations/status-check) \ No newline at end of file diff --git a/content/de/developer/integration/reverse-proxy/traefik.md b/content/de/developer/integration/reverse-proxy/traefik.md new file mode 100644 index 00000000..7e319593 --- /dev/null +++ b/content/de/developer/integration/reverse-proxy/traefik.md @@ -0,0 +1,241 @@ +--- +title: "Traefik" +description: "Deploy RustFS behind Traefik with automatic TLS for separate S3 API and Console hostnames." +--- + +Use **Traefik** and its Docker provider to discover RustFS, obtain TLS certificates from Let's Encrypt, and route separate hostnames to the S3 API and Console. You need Docker Engine, Docker Compose, a public server, and two DNS records that resolve to that server. + +This guide uses these example hostnames: + +- `s3.example.com` for the S3 API +- `console.example.com` for the Console + +Replace them with your public hostnames. The ACME HTTP-01 challenge requires inbound access to ports `80` and `443`. + +:::warning[Serve S3 from the root path] + +Do not publish the S3 API under a path such as `/s3/`. AWS Signature Version 4 includes the request path and host, so rewriting either value can invalidate signed requests. + +::: + +## 1. Create the deployment directory + +Create a directory and an empty ACME storage file. Traefik requires restrictive permissions on this file: + +```bash +mkdir rustfs-traefik +cd rustfs-traefik +touch acme.json +chmod 600 acme.json +``` + +## 2. Set deployment variables + +Create an environment file and replace each value: + +```ini title=".env" +S3_HOSTNAME=s3.example.com +CONSOLE_HOSTNAME=console.example.com +ACME_EMAIL=admin@example.com +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use an email address that receives certificate expiration notices. Do not commit `.env` or `acme.json` to source control. + +## 3. Create the Compose file + +Create the Traefik and RustFS services: + +```yaml title="compose.yaml" +services: + traefik: + image: traefik:v3.6.5 + restart: unless-stopped + command: + - --log.level=INFO + - --accesslog=true + - --providers.docker=true + - --providers.docker.endpoint=unix:///var/run/docker.sock + - --providers.docker.exposedbydefault=false + - --providers.docker.network=rustfs + - --entrypoints.web.address=:80 + - --entrypoints.websecure.address=:443 + - --entrypoints.web.http.redirections.entrypoint.to=websecure + - --entrypoints.web.http.redirections.entrypoint.scheme=https + - --certificatesresolvers.le.acme.email=${ACME_EMAIL} + - --certificatesresolvers.le.acme.storage=/etc/traefik/acme.json + - --certificatesresolvers.le.acme.httpchallenge=true + - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web + ports: + - "80:80" + - "443:443" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - ./acme.json:/etc/traefik/acme.json + networks: + - rustfs + + rustfs: + image: rustfs/rustfs:latest + restart: unless-stopped + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_ADDRESS: "0.0.0.0:9000" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" + expose: + - "9000" + - "9001" + volumes: + - rustfs-data:/data + healthcheck: + test: ["CMD-SHELL", "curl --fail http://127.0.0.1:9000/health/ready && curl --fail http://127.0.0.1:9001/rustfs/console/health"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + labels: + - traefik.enable=true + - traefik.docker.network=rustfs + - traefik.http.routers.rustfs-s3.rule=Host(`${S3_HOSTNAME}`) + - traefik.http.routers.rustfs-s3.entrypoints=websecure + - traefik.http.routers.rustfs-s3.tls=true + - traefik.http.routers.rustfs-s3.tls.certresolver=le + - traefik.http.routers.rustfs-s3.service=rustfs-s3 + - traefik.http.services.rustfs-s3.loadbalancer.server.port=9000 + - traefik.http.services.rustfs-s3.loadbalancer.passhostheader=true + - traefik.http.routers.rustfs-console.rule=Host(`${CONSOLE_HOSTNAME}`) + - traefik.http.routers.rustfs-console.entrypoints=websecure + - traefik.http.routers.rustfs-console.tls=true + - traefik.http.routers.rustfs-console.tls.certresolver=le + - traefik.http.routers.rustfs-console.service=rustfs-console + - traefik.http.services.rustfs-console.loadbalancer.server.port=9001 + - traefik.http.services.rustfs-console.loadbalancer.passhostheader=true + networks: + - rustfs + +volumes: + rustfs-data: + +networks: + rustfs: + name: rustfs +``` + +The two routers use different host rules and backend ports. RustFS does not publish ports `9000` or `9001` on the Docker host, and the Traefik Dashboard is not exposed. + +:::note[Docker socket access] + +Traefik reads container labels through the read-only Docker socket mount. Anyone who can modify Docker workloads can influence routes discovered by the Docker provider. Restrict Docker access on the proxy host. + +::: + +## 4. Validate and start the deployment + +Render the Compose configuration and check that all variables resolve: + +```bash +docker compose config +``` + +Start both services: + +```bash +docker compose up -d +docker compose ps +``` + +Follow the Traefik logs while it completes the ACME challenge and creates both certificates: + +```bash +docker compose logs --follow traefik +``` + +If certificate issuance fails, confirm that both DNS records resolve to this host and that ports `80` and `443` are reachable from the internet. Let's Encrypt rate limits apply, so correct DNS and firewall problems before repeatedly recreating the deployment. + +## 5. Verify both endpoints + +Verify the S3 API readiness endpoint and Console health endpoint through Traefik: + +```bash +curl --fail https://s3.example.com/health/ready +curl --fail https://console.example.com/rustfs/console/health +``` + +Configure S3 clients with `https://s3.example.com` as the endpoint and enable path-style addressing. Open `https://console.example.com` to sign in to the Console. + +## Multi-node services + +For an external multi-node RustFS cluster, enable Traefik's file provider in the `traefik` service: + +```yaml title="compose.yaml" +services: + traefik: + command: + - --providers.file.filename=/etc/traefik/dynamic.yaml + - --providers.file.watch=true + volumes: + - ./dynamic.yaml:/etc/traefik/dynamic.yaml:ro +``` + +Create the dynamic configuration with every RustFS node: + +```yaml title="dynamic.yaml" +http: + routers: + rustfs-s3: + rule: Host(`s3.example.com`) + entryPoints: + - websecure + service: rustfs-s3 + tls: + certResolver: le + rustfs-console: + rule: Host(`console.example.com`) + entryPoints: + - websecure + service: rustfs-console + tls: + certResolver: le + + services: + rustfs-s3: + loadBalancer: + passHostHeader: true + healthCheck: + path: /health/ready + interval: 10s + timeout: 5s + servers: + - url: http://node1.example.net:9000 + - url: http://node2.example.net:9000 + - url: http://node3.example.net:9000 + - url: http://node4.example.net:9000 + rustfs-console: + loadBalancer: + passHostHeader: true + sticky: + cookie: + name: rustfs_console + secure: true + httpOnly: true + healthCheck: + path: /rustfs/console/health + interval: 10s + timeout: 5s + servers: + - url: http://node1.example.net:9001 + - url: http://node2.example.net:9001 + - url: http://node3.example.net:9001 + - url: http://node4.example.net:9001 +``` + +Configure sticky sessions for the Console service when you use OpenID Connect. An in-progress login stores its `state` on one RustFS node and the callback must return to that node. Keep port `9000` open directly between RustFS nodes because internal node RPC uses the same listener. + +## Next steps + +- [Configure an S3 client](/developer/examples/aws-cli) +- [Enable virtual-hosted-style bucket URLs](/integration/virtual) +- [Review health and readiness endpoints](/operations/status-check) \ No newline at end of file diff --git a/content/de/developer/license.md b/content/de/developer/license.md new file mode 100644 index 00000000..ded25904 --- /dev/null +++ b/content/de/developer/license.md @@ -0,0 +1,203 @@ +--- +title: RustFS Open Source License +description: RustFS is released under the Apache 2.0 license. +--- + +## Open Source License + +- RustFS is released under the Apache 2.0 license. +- [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) + +## License Terms + +_Version 2.0, January 2004_ +http://www.apache.org/licenses/ + +### Terms and Conditions for use, reproduction, and distribution + +#### Definitions + +"License" shall mean the terms and conditions for use, reproduction, and +distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright +owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities +that control, are controlled by, or are under common control with that entity. +For the purposes of this definition, "control" means **(i)** the power, direct or +indirect, to cause the direction or management of such entity, whether by +contract or otherwise, or **(ii)** ownership of fifty percent (50%) or more of the +outstanding shares, or **(iii)** beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising +permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including +but not limited to software source code, documentation source, and configuration +files. + +"Object" form shall mean any form resulting from mechanical transformation or +translation of a Source form, including but not limited to compiled object code, +generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made +available under the License, as indicated by a copyright notice that is included +in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that +is based on (or derived from) the Work and for which the editorial revisions, +annotations, elaborations, or other modifications represent, as a whole, an +original work of authorship. For the purposes of this License, Derivative Works +shall not include works that remain separable from, or merely link (or bind by +name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version +of the Work and any modifications or additions to that Work or Derivative Works +thereof, that is intentionally submitted to Licensor for inclusion in the Work +by the copyright owner or by an individual or Legal Entity authorized to submit +on behalf of the copyright owner. For the purposes of this definition, +"submitted" means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, and +issue tracking systems that are managed by, or on behalf of, the Licensor for +the purpose of discussing and improving the Work, but excluding communication +that is conspicuously marked or otherwise designated in writing by the copyright +owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf +of whom a Contribution has been received by Licensor and subsequently +incorporated within the Work. + +#### Grant of Copyright License + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the Work and such +Derivative Works in Source or Object form. + +#### Grant of Patent License + +Subject to the terms and conditions of this License, each Contributor hereby +grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, +irrevocable (except as stated in this section) patent license to make, have +made, use, offer to sell, sell, import, and otherwise transfer the Work, where +such license applies only to those patent claims licensable by such Contributor +that are necessarily infringed by their Contribution(s) alone or by combination +of their Contribution(s) with the Work to which such Contribution(s) was +submitted. If You institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work or a +Contribution incorporated within the Work constitutes direct or contributory +patent infringement, then any patent licenses granted to You under this License +for that Work shall terminate as of the date such litigation is filed. + +#### Redistribution + +You may reproduce and distribute copies of the Work or Derivative Works thereof +in any medium, with or without modifications, and in Source or Object form, +provided that You meet the following conditions: + +* **(a)** You must give any other recipients of the Work or Derivative Works a copy of +this License; and +* **(b)** You must cause any modified files to carry prominent notices stating that You +changed the files; and +* **(c)** You must retain, in the Source form of any Derivative Works that You distribute, +all copyright, patent, trademark, and attribution notices from the Source form +of the Work, excluding those notices that do not pertain to any part of the +Derivative Works; and +* **(d)** If the Work includes a "NOTICE" text file as part of its distribution, then any +Derivative Works that You distribute must include a readable copy of the +attribution notices contained within such NOTICE file, excluding those notices +that do not pertain to any part of the Derivative Works, in at least one of the +following places: within a NOTICE text file distributed as part of the +Derivative Works; within the Source form or documentation, if provided along +with the Derivative Works; or, within a display generated by the Derivative +Works, if and wherever such third-party notices normally appear. The contents of +the NOTICE file are for informational purposes only and do not modify the +License. You may add Your own attribution notices within Derivative Works that +You distribute, alongside or as an addendum to the NOTICE text from the Work, +provided that such additional attribution notices cannot be construed as +modifying the License. + +You may add Your own copyright statement to Your modifications and may provide +additional or different license terms and conditions for use, reproduction, or +distribution of Your modifications, or for any such Derivative Works as a whole, +provided Your use, reproduction, and distribution of the Work otherwise complies +with the conditions stated in this License. + +#### Submission of Contributions + +Unless You explicitly state otherwise, any Contribution intentionally submitted +for inclusion in the Work by You to the Licensor shall be under the terms and +conditions of this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify the terms of +any separate license agreement you may have executed with Licensor regarding +such Contributions. + +#### Trademarks + +This License does not grant permission to use the trade names, trademarks, +service marks, or product names of the Licensor, except as required for +reasonable and customary use in describing the origin of the Work and +reproducing the content of the NOTICE file. + +#### Disclaimer of Warranty + +Unless required by applicable law or agreed to in writing, Licensor provides the +Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, +including, without limitation, any warranties or conditions of TITLE, +NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are +solely responsible for determining the appropriateness of using or +redistributing the Work and assume any risks associated with Your exercise of +permissions under this License. + +#### Limitation of Liability + +In no event and under no legal theory, whether in tort (including negligence), +contract, or otherwise, unless required by applicable law (such as deliberate +and grossly negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, incidental, +or consequential damages of any character arising as a result of this License or +out of the use or inability to use the Work (including but not limited to +damages for loss of goodwill, work stoppage, computer failure or malfunction, or +any and all other commercial damages or losses), even if such Contributor has +been advised of the possibility of such damages. + +#### Accepting Warranty or Additional Liability + +While redistributing the Work or Derivative Works thereof, You may choose to +offer, and charge a fee for, acceptance of support, warranty, indemnity, or +other liability obligations and/or rights consistent with this License. However, +in accepting such obligations, You may act only on Your own behalf and on Your +sole responsibility, not on behalf of any other Contributor, and only if You +agree to indemnify, defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason of your +accepting any such warranty or additional liability. + +_END OF TERMS AND CONDITIONS_ + +### APPENDIX: How to apply the Apache License to your work + +To apply the Apache License to your work, attach the following boilerplate +notice, with the fields enclosed by brackets `[]` replaced with your own +identifying information. (Don't include the brackets!) The text should be +enclosed in the appropriate comment syntax for the file format. We also +recommend that a file or class name and description of purpose be included on +the same "printed page" as the copyright notice for easier identification within +third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/content/de/developer/mcp.md b/content/de/developer/mcp.md new file mode 100644 index 00000000..fd997b01 --- /dev/null +++ b/content/de/developer/mcp.md @@ -0,0 +1,269 @@ +--- +title: "RustFS MCP" +description: "RustFS MCP Usage Guide" +--- + +**RustFS MCP Server** is a high-performance [Model Context Protocol (MCP)](https://www.anthropic.com/news/model-context-protocol) server that provides AI/LLM tools with seamless access to S3-compatible object storage operations. Built with Rust for performance and security, it enables AI assistants like Claude Desktop to interact with cloud storage through standardized protocols. + +### What is MCP? + +The Model Context Protocol is an open standard that enables AI applications to establish secure, controlled connections with external systems. This server acts as a bridge between AI tools and S3-compatible storage services, providing structured access to file operations while maintaining security and observability. + +## ✨ Features + +### Supported S3 Operations + +- **List Buckets**: Lists all accessible S3 buckets. +- **List Objects**: Browses bucket contents with optional prefix filtering. +- **Upload Files**: Uploads local files with automatic MIME type detection and cache control. +- **Get Objects**: Retrieves objects from S3 storage, supporting read or download modes. + +## 🔧 Installation + +:::warning[Build instructions may be outdated] + +The `rustfs-mcp` crate is no longer part of the current `rustfs/rustfs` main branch, so the build commands below may fail against a fresh clone. Check the [RustFS GitHub organization](https://github.com/rustfs) for the MCP server's current location before building from source. + +::: + +### Prerequisites + +- Rust 1.75+ (for building from source) +- Configured AWS credentials (via environment variables, AWS CLI, or IAM roles) +- Access to S3-compatible storage services + +### Building from Source + +```bash +# Clone the repository +git clone https://github.com/rustfs/rustfs.git +cd rustfs + +# Build the MCP server +cargo build --release -p rustfs-mcp + +# Binary will be available at +./target/release/rustfs-mcp +``` + +## ⚙️ Configuration + +### Environment Variables + +```bash +# AWS credentials (required) +export AWS_ACCESS_KEY_ID=your_access_key +export AWS_SECRET_ACCESS_KEY=your_secret_key +export AWS_REGION=us-east-1 # optional, defaults to us-east-1 + +# Optional: Custom S3 endpoint (for MinIO, etc.) +export AWS_ENDPOINT_URL=http://localhost:9000 + +# Log level (optional) +export RUST_LOG=info +``` + +### Command Line Options + +```bash +rustfs-mcp --help +``` + +The server supports various command line options to customize behavior: + +- `--access-key-id`: AWS access key ID for S3 authentication +- `--secret-access-key`: AWS secret key for S3 authentication +- `--region`: AWS region for S3 operations (default: us-east-1) +- `--endpoint-url`: Custom S3 endpoint URL (for MinIO, LocalStack, etc.) +- `--log-level`: Log level configuration (default: rustfs_mcp_server=info) + +## 🚀 Usage + +### Starting the Server + +```bash +# Start the MCP server +rustfs-mcp + +# Or with custom options +rustfs-mcp --log-level debug --region us-west-2 +``` + +### Integration with Chat Clients + +#### Option 1: Using Command Line Arguments + +```json +{ + "mcpServers": { + "rustfs-mcp": { + "command": "/path/to/rustfs-mcp", + "args": [ + "--access-key-id", "your_access_key", + "--secret-access-key", "your_secret_key", + "--region", "us-west-2", + "--log-level", "info" + ] + } + } +} +``` + +#### Option 2: Using Environment Variables + +```json +{ + "mcpServers": { + "rustfs-mcp": { + "command": "/path/to/rustfs-mcp", + "env": { + "AWS_ACCESS_KEY_ID": "your_access_key", + "AWS_SECRET_ACCESS_KEY": "your_secret_key", + "AWS_REGION": "us-east-1" + } + } + } +} +``` + +### Using with Docker + +[RustFS MCP officially provides a Dockerfile](https://github.com/rustfs/rustfs/tree/main/crates/mcp) that can be used to build container images for using RustFS MCP. + +```bash +# Clone RustFS repository code +git clone https://github.com/rustfs/rustfs.git + +# Build Docker image +docker build -f crates/mcp/Dockerfile -t rustfs/rustfs-mcp . +``` + +After successful build, you can configure it in the MCP configuration of AI IDEs. + +#### Configuring MCP in AI IDEs + +Currently, mainstream AI IDEs such as Cursor, Windsurf, Trae, etc. all support MCP. For example, in Trae, add the following content to the MCP configuration (**MCP --> Add**): + +```json +{ + "mcpServers": { + "rustfs-mcp": { + "command": "docker", + "args": [ + "run", + "--rm", + "-i", + "-e", + "AWS_ACCESS_KEY_ID", + "-e", + "AWS_SECRET_ACCESS_KEY", + "-e", + "AWS_REGION", + "-e", + "AWS_ENDPOINT_URL", + "rustfs/rustfs-mcp" + ], + "env": { + "AWS_ACCESS_KEY_ID": "rustfs_access_key", + "AWS_SECRET_ACCESS_KEY": "rustfs_secret_key", + "AWS_REGION": "us-east-1", + "AWS_ENDPOINT_URL": "rustfs_instance_url" + } + } + } +} +``` + +> `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are RustFS access keys. You can refer to the [Access Key Management chapter](../security-compliance/iam/access-token.md) for creation. + +If added successfully, you can list the [available tools](#️-available-tools) on the MCP configuration page. + +![add rustfs mcp in trae mcp configuration successfully](images/add-rustfs-mcp-succ.png) + +In Trae, you can use the corresponding tools by entering the corresponding prompts. For example, in Trae's chat dialog, enter: + +```text +Please help me list the buckets in the current rustfs instance, thank you! +``` + +Returns the following response: + +![list rustfs bucket with rustfs mcp](images/list-rustfs-bucket-with-mcp.png) + +Trae uses **Builder with MCP** mode, calling the `list_buckets` tool to list all buckets in the configured RustFS instance. The same applies to calls to other tools. + +## 🛠️ Available Tools + +The MCP server exposes the following tools that AI assistants can use: + +### `list_buckets` + +Lists all S3 buckets accessible with the configured credentials. + +**Parameters**: None + +### `list_objects` + +Lists objects in an S3 bucket with optional prefix filtering. + +**Parameters**: + +- `bucket_name` (string): Name of the S3 bucket +- `prefix` (string, optional): Prefix for filtering objects + +### `upload_file` + +Uploads a local file to S3 with automatic MIME type detection. + +**Parameters**: + +- `local_file_path` (string): Local file path +- `bucket_name` (string): Target S3 bucket +- `object_key` (string): S3 object key (target path) +- `content_type` (string, optional): Content type (auto-detected if not provided) +- `storage_class` (string, optional): S3 storage class +- `cache_control` (string, optional): Cache control header + +### `get_object` + +Retrieves objects from S3 with two operation modes: direct content reading or download to file. + +**Parameters**: + +- `bucket_name` (string): Source S3 bucket +- `object_key` (string): S3 object key +- `version_id` (string, optional): Version ID for versioned objects +- `mode` (string, optional): Operation mode - "read" (default) returns content directly, "download" saves to local file +- `local_path` (string, optional): Local file path (required when mode is "download") +- `max_content_size` (number, optional): Maximum content size for read mode in bytes (default: 1MB) + +### `create_bucket` + +Creates a new RustFS bucket. + +**Parameters**: + +- `bucket_name` (string): Name of the bucket to create. + +### `delete_bucket` + +Deletes the specified RustFS bucket. + +**Parameters**: + +- `bucket_name` (string): Name of the bucket to delete. + +## Architecture + +The MCP server is built with a modular architecture: + +```text +rustfs-mcp/ +├── src/ +│ ├── main.rs # Entry point, CLI parsing and server initialization +│ ├── server.rs # MCP server implementation and tool handlers +│ ├── s3_client.rs # S3 client wrapper with async operations +│ ├── config.rs # Configuration management and CLI options +│ └── lib.rs # Library exports and public API +└── Cargo.toml # Dependencies, metadata and binary configuration +``` diff --git a/content/de/developer/meta.json b/content/de/developer/meta.json new file mode 100644 index 00000000..efafd206 --- /dev/null +++ b/content/de/developer/meta.json @@ -0,0 +1,10 @@ +{ + "title": "Entwicklung", + "icon": "CodeXml", + "defaultOpen": false, + "pages": [ + "sdk", + "integration", + "license" + ] +} diff --git a/content/de/developer/sdk/go.md b/content/de/developer/sdk/go.md new file mode 100644 index 00000000..b40cab27 --- /dev/null +++ b/content/de/developer/sdk/go.md @@ -0,0 +1,222 @@ +--- +title: "Golang SDK Guide" +description: "Use the AWS SDK for Go v2 to operate on RustFS instances, including creating and deleting buckets and objects." +--- + +RustFS ships no first-party Go SDK — it is fully S3-compatible, so you use the official AWS SDK for Go v2 configured to point at your RustFS server. Through the SDK, you can operate on RustFS, including creating and deleting buckets/objects, uploading and downloading files, etc. + +## Prerequisites + +- Go 1.21 or later +- A working RustFS instance (refer to [Installation Guide](../../installation/index.md)) — the S3 API listens on port `9000`, the Console on port `9001` +- Access keys, set at install time via the `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` environment variables (refer to [Access Key Management](../../security-compliance/iam/access-token.md)) + +:::tip[Local test] + +If you did not set credentials at install time, the server defaults to `rustfsadmin` / `rustfsadmin` — fine for a throwaway local trial, never for anything reachable by others. + +::: + +Install the SDK modules: + +```bash +go mod init rustfs-go-demo +go get github.com/aws/aws-sdk-go-v2/aws +go get github.com/aws/aws-sdk-go-v2/credentials +go get github.com/aws/aws-sdk-go-v2/service/s3 +``` + +## Initializing the Client + +The following is a complete, runnable program. It reads its configuration from environment variables and initializes the client from an `aws.Config`: + +```go title="main.go" +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/credentials" + "github.com/aws/aws-sdk-go-v2/service/s3" +) + +func main() { + region := os.Getenv("RUSTFS_REGION") + accessKeyID := os.Getenv("RUSTFS_ACCESS_KEY_ID") + secretAccessKey := os.Getenv("RUSTFS_SECRET_ACCESS_KEY") + endpoint := os.Getenv("RUSTFS_ENDPOINT_URL") + + if accessKeyID == "" || secretAccessKey == "" || region == "" || endpoint == "" { + log.Fatal("missing the env: RUSTFS_ACCESS_KEY_ID / RUSTFS_SECRET_ACCESS_KEY / RUSTFS_REGION / RUSTFS_ENDPOINT_URL") + } + + // build aws.Config + cfg := aws.Config{ + Region: region, + Credentials: aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(accessKeyID, secretAccessKey, "")), + } + + // build S3 client + client := s3.NewFromConfig(cfg, func(o *s3.Options) { + o.BaseEndpoint = aws.String(endpoint) + // RustFS uses path-style URLs by default; virtual-host style requires RUSTFS_SERVER_DOMAINS + o.UsePathStyle = true + }) + + ctx := context.Background() + + resp, err := client.ListBuckets(ctx, &s3.ListBucketsInput{}) + if err != nil { + log.Fatalf("list buckets failed: %v", err) + } + + fmt.Println("Buckets:") + for _, b := range resp.Buckets { + fmt.Println(" -", *b.Name) + } +} +``` + +:::note + +These environment variable names (`RUSTFS_ENDPOINT_URL`, `RUSTFS_REGION`, `RUSTFS_ACCESS_KEY_ID`, `RUSTFS_SECRET_ACCESS_KEY`) are just this example's client-side conventions — they are read by your Go program, not by RustFS. They are distinct from the server-side `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` variables used when installing RustFS. + +::: + +Run it (replace `localhost` with your server's IP address if RustFS runs on another machine): + +```bash +export RUSTFS_ENDPOINT_URL="http://localhost:9000" +export RUSTFS_REGION="us-east-1" +export RUSTFS_ACCESS_KEY_ID="" +export RUSTFS_SECRET_ACCESS_KEY="" +go run main.go +``` + +```text +Buckets: + - my-bucket +``` + +You can now perform bucket and object operations. The snippets below run inside the `main` function above, reusing `client` and `ctx`. + +## Create Bucket + +```go +_, err = client.CreateBucket(ctx, &s3.CreateBucketInput{ + Bucket: aws.String("my-bucket"), +}) +if err != nil { + log.Fatalf("create bucket failed: %v", err) +} +fmt.Println("bucket created") +``` + +```text +bucket created +``` + +## List Buckets + +```go +resp, err := client.ListBuckets(ctx, &s3.ListBucketsInput{}) +if err != nil { + log.Fatalf("list buckets failed: %v", err) +} + +fmt.Println("Buckets:") +for _, b := range resp.Buckets { + fmt.Println(" -", *b.Name) +} +``` + +```text +Buckets: + - my-bucket +``` + +## Delete Bucket + +```go +_, err = client.DeleteBucket(ctx, &s3.DeleteBucketInput{ + Bucket: aws.String("my-bucket"), +}) +if err != nil { + log.Fatalf("delete bucket failed: %v", err) +} +fmt.Println("bucket deleted") +``` + +```text +bucket deleted +``` + +## List Objects + +```go +resp, err := client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{ + Bucket: aws.String("my-bucket"), +}) +if err != nil { + log.Fatalf("list object failed: %v", err) +} +for _, obj := range resp.Contents { + fmt.Println(" -", *obj.Key) +} +``` + +```text + - hello.txt +``` + +## Upload Object + +Uploading a string body (add `"strings"` to your imports): + +```go +_, err = client.PutObject(ctx, &s3.PutObjectInput{ + Bucket: aws.String("my-bucket"), + Key: aws.String("hello.txt"), + Body: strings.NewReader("hello rustfs"), +}) +if err != nil { + log.Fatalf("upload object failed: %v", err) +} +fmt.Println("object uploaded") +``` + +```text +object uploaded +``` + +## Download Object + +Reading the object body (add `"io"` to your imports): + +```go +resp, err := client.GetObject(ctx, &s3.GetObjectInput{ + Bucket: aws.String("my-bucket"), + Key: aws.String("hello.txt"), +}) +if err != nil { + log.Fatalf("download object fail: %v", err) +} +defer resp.Body.Close() + +// read object content +data, err := io.ReadAll(resp.Body) +if err != nil { + log.Fatalf("read object content fail: %v", err) +} +fmt.Println("content is :", string(data)) +``` + +```text +content is : hello rustfs +``` + +For other operations (presigned URLs, multipart uploads, and more), see the [AWS SDK for Go v2 documentation](https://aws.github.io/aws-sdk-go-v2/docs/) — every S3-compatible call works against RustFS the same way. diff --git a/content/de/developer/sdk/index.md b/content/de/developer/sdk/index.md new file mode 100644 index 00000000..f57b4164 --- /dev/null +++ b/content/de/developer/sdk/index.md @@ -0,0 +1,43 @@ +--- +title: "RustFS SDK Overview" +description: "Overview of supported S3 SDKs for RustFS." +--- + +RustFS is a distributed object storage system fully compatible with the S3 protocol. Users can: + +- Manage RustFS through the Console management interface. +- Manage RustFS through S3 clients. +- Implement object storage operations and management on the business side through SDKs. + +Currently, the SDKs provided by RustFS include: + +- [Java SDK](./java.md) +- [JavaScript SDK](./javascript.md) +- [Python SDK](./python.md) +- [Rust SDK](./rust.md) +- [TypeScript SDK](./typescript.md) +- [Golang SDK](./go.md) + +## Terminology + +Amazon S3 (Simple Storage Service) was the first widely adopted object storage service. Its API has become the de facto standard for object storage. In this documentation, "S3" refers to the protocol. + +## SDK Recommendations + +We recommend using the official AWS S3 SDKs. These SDKs are mature, well-maintained, and highly optimized. + +If you have a familiar and trusted SDK from a vendor, you can use it. + +Some third-party SDKs may have non-standard implementations. We recommend avoiding SDKs that are not strictly S3-compliant. + +## Compatibility with MinIO SDKs + +Yes, RustFS is fully compatible with MinIO SDKs. + +If you are using MinIO SDKs, you can modify the Endpoint, AK, and SK to be directly compatible with RustFS. + +## Handling Incompatible SDKs + +If you encounter an SDK that does not support standard S3, MinIO, or RustFS: + +We recommend switching to a standard AWS S3 SDK. diff --git a/content/de/developer/sdk/java.md b/content/de/developer/sdk/java.md new file mode 100644 index 00000000..938a7f16 --- /dev/null +++ b/content/de/developer/sdk/java.md @@ -0,0 +1,345 @@ +--- +title: "Java SDK Guide" +description: "Use the official AWS SDK for Java v2 with RustFS." +--- + +RustFS ships no first-party Java SDK — it is S3-compatible, so you use the official AWS SDK for Java v2 configured to point at your RustFS server. + +## 1. Prerequisites + +* Java 8 or later and Maven (or Gradle) +* A running RustFS instance (see the [Installation Guide](../../installation/index.md)) — the S3 API listens on port `9000`, the Console on port `9001` +* Access keys, set at install time via the `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` environment variables (see [Access Key Management](../../security-compliance/iam/access-token.md)) + +:::tip[Local test] + +If you did not set credentials at install time, the server defaults to `rustfsadmin` / `rustfsadmin` — fine for a throwaway local trial, never for anything reachable by others. + +::: + +### 1.1 Maven Project Setup + +Create a new Maven project: + +```text +rustfs-java-s3-demo/ +├── pom.xml +└── src/ + └── main/ + └── java/ + └── com/ + └── example/ + └── RustfsS3Example.java +``` + +### 1.2 Add Dependencies + +Add AWS SDK dependencies in `pom.xml`: + +```xml title="pom.xml" + + + software.amazon.awssdk + s3 + 2.25.27 + + +``` + +> Recommend using AWS SDK v2, which has more complete features and supports async, reactive, and other patterns. + +--- + +## 2. Connecting to RustFS + +### 2.1 Complete Example + +The following class compiles and runs as-is. Replace `localhost` with your server's IP address if RustFS runs on another machine, and fill in your own access keys: + +```java title="RustfsS3Example.java" +package com.example; + +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.*; + +import java.net.URI; +import java.nio.file.Paths; + +public class RustfsS3Example { + + public static void main(String[] args) { + // 1. Initialize S3 client + S3Client s3 = S3Client.builder() + .endpointOverride(URI.create("http://localhost:9000")) // RustFS S3 API address + .region(Region.US_EAST_1) // RustFS default region + .credentialsProvider( + StaticCredentialsProvider.create( + AwsBasicCredentials.create("", "") + ) + ) + // RustFS uses path-style URLs by default; virtual-host style requires RUSTFS_SERVER_DOMAINS + .forcePathStyle(true) + .build(); + + // 2. Create bucket + String bucket = "my-bucket"; + try { + s3.createBucket(CreateBucketRequest.builder().bucket(bucket).build()); + System.out.println("Bucket created: " + bucket); + } catch (BucketAlreadyExistsException | BucketAlreadyOwnedByYouException e) { + System.out.println("Bucket already exists."); + } + + // 3. Upload file + s3.putObject( + PutObjectRequest.builder().bucket(bucket).key("hello.txt").build(), + Paths.get("/path/to/hello.txt") + ); + System.out.println("Uploaded hello.txt"); + + // 4. Download file + s3.getObject( + GetObjectRequest.builder().bucket(bucket).key("hello.txt").build(), + Paths.get("downloaded-hello.txt") + ); + System.out.println("Downloaded hello.txt"); + + // 5. List objects + ListObjectsV2Response listResponse = s3.listObjectsV2(ListObjectsV2Request.builder().bucket(bucket).build()); + listResponse.contents().forEach(obj -> System.out.println("Found object: " + obj.key())); + + // 6. Delete object + s3.deleteObject(DeleteObjectRequest.builder().bucket(bucket).key("hello.txt").build()); + System.out.println("Deleted hello.txt"); + + // 7. Delete bucket (optional) + // s3.deleteBucket(DeleteBucketRequest.builder().bucket(bucket).build()); + } +} +``` + +Expected output: + +```text +Bucket created: my-bucket +Uploaded hello.txt +Downloaded hello.txt +Found object: hello.txt +Deleted hello.txt +``` + +--- + +## 3. Common Issues and Troubleshooting + +| Issue | Cause | Solution | +| -------------------------------------- | ------------------------------------ | ----------------------------------------- | +| `S3Exception: 301 Moved Permanently` | Path-style not enabled or region error | Set `.forcePathStyle(true)` and use region `us-east-1` | +| `ConnectException: Connection refused` | RustFS not started or incorrect port | Check RustFS status and port | +| `403 Forbidden` | AccessKey / SecretKey error | Check authentication configuration | +| Upload fails with no response | SDK defaults to HTTPS, RustFS only supports HTTP (or needs certificates) | Use `http://` address and configure `endpointOverride` | + +--- + +## 4. Appendix + +### 4.1 Maven Package and Run + +Package project: + +```bash +mvn clean package +``` + +Execute: + +```bash +java -cp target/rustfs-java-s3-demo-1.0-SNAPSHOT.jar com.example.RustfsS3Example +``` + +### 4.2 RustFS Configuration Recommendations + +* Ensure SSL validation is disabled when service uses HTTP protocol. +* Enable CORS support (if used for web frontend). +* Recommend setting limits like `max_object_size` and `max_part_size` to prevent large file transfer failures. + +--- + +The following advanced examples cover: + +* Presigned URL generation and usage +* Multipart Upload complete process + +--- + +## 5. Java Advanced Features Examples + +### 5.1 Generate and Use Presigned URLs + +> Presigned URLs allow clients to temporarily access private objects without exposing credentials, widely used for browser direct upload or download file scenarios. + +#### 5.1.1 Generate Download Link (GET) + +```java +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Configuration; +import software.amazon.awssdk.services.s3.model.GetObjectRequest; +import software.amazon.awssdk.services.s3.presigner.S3Presigner; +import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest; +import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest; + +import java.net.URI; +import java.time.Duration; + +S3Presigner presigner = S3Presigner.builder() + .endpointOverride(URI.create("http://localhost:9000")) + .region(Region.US_EAST_1) + .credentialsProvider( + StaticCredentialsProvider.create( + AwsBasicCredentials.create("", "") + ) + ) + // The presigner must also sign path-style URLs + .serviceConfiguration( + S3Configuration.builder().pathStyleAccessEnabled(true).build() + ) + .build(); + +GetObjectRequest getObjectRequest = GetObjectRequest.builder() + .bucket("my-bucket") + .key("hello.txt") + .build(); + +GetObjectPresignRequest presignRequest = GetObjectPresignRequest.builder() + .getObjectRequest(getObjectRequest) + .signatureDuration(Duration.ofMinutes(15)) // 15 minutes validity + .build(); + +PresignedGetObjectRequest presignedRequest = presigner.presignGetObject(presignRequest); + +System.out.println("Presigned URL: " + presignedRequest.url()); +``` + +```text +Presigned URL: http://localhost:9000/my-bucket/hello.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&... +``` + +> 🔗 Open the link in browser to access the object. + +#### 5.1.2 Upload Presigned URL (PUT) + +Similarly, you can also generate upload URLs: + +```java +import software.amazon.awssdk.services.s3.model.PutObjectRequest; +import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest; +import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest; + +PutObjectRequest putRequest = PutObjectRequest.builder() + .bucket("my-bucket") + .key("upload.txt") + .build(); + +PresignedPutObjectRequest presignedPut = presigner.presignPutObject( + PutObjectPresignRequest.builder() + .putObjectRequest(putRequest) + .signatureDuration(Duration.ofMinutes(10)) + .build() +); + +System.out.println("Upload URL: " + presignedPut.url()); +``` + +--- + +### 5.2 Implement Multipart Upload + +> Multipart Upload is the recommended way for large file uploads, enabling resume from breakpoint during network fluctuations. + +The examples below reuse the `s3` client from section 2 and need these additional imports: + +```java +import software.amazon.awssdk.services.s3.model.*; + +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +``` + +#### 5.2.1 Start Multipart Upload + +```java +CreateMultipartUploadRequest createRequest = CreateMultipartUploadRequest.builder() + .bucket("my-bucket") + .key("bigfile.zip") + .build(); + +CreateMultipartUploadResponse createResponse = s3.createMultipartUpload(createRequest); +String uploadId = createResponse.uploadId(); +``` + +#### 5.2.2 Upload Each Part + +```java +List completedParts = new ArrayList<>(); +for (int i = 1; i <= 3; i++) { + String partPath = "part" + i + ".bin"; // Assume each part is a local file + UploadPartRequest uploadPartRequest = UploadPartRequest.builder() + .bucket("my-bucket") + .key("bigfile.zip") + .uploadId(uploadId) + .partNumber(i) + .build(); + + UploadPartResponse uploadPartResponse = s3.uploadPart(uploadPartRequest, Paths.get(partPath)); + completedParts.add( + CompletedPart.builder() + .partNumber(i) + .eTag(uploadPartResponse.eTag()) + .build() + ); +} +``` + +#### 5.2.3 Complete Multipart Upload + +```java +CompletedMultipartUpload completedUpload = CompletedMultipartUpload.builder() + .parts(completedParts) + .build(); + +CompleteMultipartUploadRequest completeRequest = CompleteMultipartUploadRequest.builder() + .bucket("my-bucket") + .key("bigfile.zip") + .uploadId(uploadId) + .multipartUpload(completedUpload) + .build(); + +s3.completeMultipartUpload(completeRequest); +System.out.println("Multipart upload completed."); +``` + +```text +Multipart upload completed. +``` + +#### 5.2.4 Abort Upload on Exception (Optional) + +```java +AbortMultipartUploadRequest abortRequest = AbortMultipartUploadRequest.builder() + .bucket("my-bucket") + .key("bigfile.zip") + .uploadId(uploadId) + .build(); + +s3.abortMultipartUpload(abortRequest); +``` + +--- + +For other operations (object tagging, bucket policies, and more), see the [AWS SDK for Java v2 documentation](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/) — every S3-compatible call works against RustFS the same way. diff --git a/content/de/developer/sdk/javascript.md b/content/de/developer/sdk/javascript.md new file mode 100644 index 00000000..f010b46f --- /dev/null +++ b/content/de/developer/sdk/javascript.md @@ -0,0 +1,315 @@ +--- +title: "JavaScript SDK Guide" +description: "Use the official AWS SDK for JavaScript v3 with RustFS." +--- + +## I. Overview + +RustFS ships no first-party JavaScript SDK — it is S3-compatible, so you use the official AWS SDK for JavaScript (v3) configured to point at your RustFS server. This guide shows how to connect to RustFS and perform common object storage operations. + +## II. Prerequisites + +* Node.js 18 or later +* A running RustFS instance (see the [Installation Guide](../../installation/index.md)) — the S3 API listens on port `9000`, the Console on port `9001` +* Access keys, set at install time via the `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` environment variables (see [Access Key Management](../../security-compliance/iam/access-token.md)) + +:::tip[Local test] + +If you did not set credentials at install time, the server defaults to `rustfsadmin` / `rustfsadmin` — fine for a throwaway local trial, never for anything reachable by others. + +::: + +### 2.1 SDK Installation + +Install the required AWS SDK v3 modules with NPM: + +```bash +npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner +``` + +The examples below use ES modules (`import`). Set `"type": "module"` in your `package.json`, or save the files with the `.mjs` extension. + +--- + +## III. Initializing the Client + +The following is a complete, runnable script. Replace `localhost` with your server's IP address if RustFS runs on another machine, and fill in your own access keys: + +```js title="main.mjs" +import { S3Client, ListBucketsCommand } from "@aws-sdk/client-s3"; + +const s3 = new S3Client({ + endpoint: "http://localhost:9000", // RustFS S3 API address + region: "us-east-1", // RustFS default region + credentials: { + accessKeyId: "", + secretAccessKey: "", + }, + // RustFS uses path-style URLs by default; virtual-host style requires RUSTFS_SERVER_DOMAINS + forcePathStyle: true, +}); + +const { Buckets } = await s3.send(new ListBucketsCommand({})); +console.log(Buckets?.map((b) => b.Name) ?? []); +``` + +Run it: + +```bash +node main.mjs +``` + +```text +[ 'my-bucket' ] +``` + +All snippets below reuse this `s3` client. + +--- + +## IV. Basic Operations + +### 4.1 Create Bucket + +```js +import { CreateBucketCommand } from "@aws-sdk/client-s3"; + +await s3.send(new CreateBucketCommand({ Bucket: "my-bucket" })); +console.log("Bucket created"); +``` + +```text +Bucket created +``` + +--- + +### 4.2 Upload Object + +```js +import { PutObjectCommand } from "@aws-sdk/client-s3"; +import { readFileSync } from "fs"; + +const data = readFileSync("/path/to/hello.txt"); + +await s3.send( + new PutObjectCommand({ + Bucket: "my-bucket", + Key: "hello.txt", + Body: data, + }) +); + +console.log("File uploaded"); +``` + +```text +File uploaded +``` + +--- + +### 4.3 Download Object + +```js +import { GetObjectCommand } from "@aws-sdk/client-s3"; +import { writeFile } from "fs/promises"; + +const response = await s3.send( + new GetObjectCommand({ Bucket: "my-bucket", Key: "hello.txt" }) +); + +const streamToBuffer = async (stream) => { + const chunks = []; + for await (const chunk of stream) chunks.push(chunk); + return Buffer.concat(chunks); +}; + +const buffer = await streamToBuffer(response.Body); +await writeFile("downloaded.txt", buffer); + +console.log("File downloaded"); +``` + +```text +File downloaded +``` + +--- + +### 4.4 List Objects + +```js +import { ListObjectsV2Command } from "@aws-sdk/client-s3"; + +const res = await s3.send(new ListObjectsV2Command({ Bucket: "my-bucket" })); +res.Contents?.forEach((obj) => console.log(`${obj.Key} (${obj.Size} bytes)`)); +``` + +```text +hello.txt (12 bytes) +``` + +--- + +### 4.5 Delete Object + +```js +import { DeleteObjectCommand } from "@aws-sdk/client-s3"; + +await s3.send(new DeleteObjectCommand({ Bucket: "my-bucket", Key: "hello.txt" })); +console.log("File deleted"); +``` + +```text +File deleted +``` + +--- + +## V. Advanced Features + +### 5.1 Generate Presigned URLs + +> Allows frontend or third parties to use temporary links for uploading/downloading files + +#### Download (GET) + +```js +import { GetObjectCommand } from "@aws-sdk/client-s3"; +import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; + +const url = await getSignedUrl( + s3, + new GetObjectCommand({ Bucket: "my-bucket", Key: "hello.txt" }), + { expiresIn: 600 } +); + +console.log("Presigned GET URL:", url); +``` + +```text +Presigned GET URL: http://localhost:9000/my-bucket/hello.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&... +``` + +#### Upload (PUT) + +```js +import { PutObjectCommand } from "@aws-sdk/client-s3"; +import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; + +const url = await getSignedUrl( + s3, + new PutObjectCommand({ Bucket: "my-bucket", Key: "upload.txt" }), + { expiresIn: 600 } +); + +console.log("Presigned PUT URL:", url); +``` + +--- + +### 5.2 Multipart Upload + +```js +import { + CreateMultipartUploadCommand, + UploadPartCommand, + CompleteMultipartUploadCommand, + AbortMultipartUploadCommand, +} from "@aws-sdk/client-s3"; +import { statSync, openSync, readSync, closeSync } from "fs"; + +const bucket = "my-bucket"; +const key = "large-file.zip"; +const filePath = "./large-file.zip"; +const partSize = 5 * 1024 * 1024; // 5 MB + +// 1. Create upload task +const createRes = await s3.send( + new CreateMultipartUploadCommand({ Bucket: bucket, Key: key }) +); +const uploadId = createRes.UploadId; + +// 2. Segmented upload +const fileSize = statSync(filePath).size; +const fd = openSync(filePath, "r"); +const parts = []; + +for (let partNumber = 1, offset = 0; offset < fileSize; partNumber++) { + const buffer = Buffer.alloc(Math.min(partSize, fileSize - offset)); + readSync(fd, buffer, 0, buffer.length, offset); + + const uploadPartRes = await s3.send( + new UploadPartCommand({ + Bucket: bucket, + Key: key, + UploadId: uploadId, + PartNumber: partNumber, + Body: buffer, + }) + ); + + parts.push({ ETag: uploadPartRes.ETag, PartNumber: partNumber }); + offset += partSize; +} + +closeSync(fd); + +// 3. Complete upload +await s3.send( + new CompleteMultipartUploadCommand({ + Bucket: bucket, + Key: key, + UploadId: uploadId, + MultipartUpload: { Parts: parts }, + }) +); + +console.log("Multipart upload completed"); +``` + +```text +Multipart upload completed +``` + +--- + +## VI. Common Issues and Notes + +| Problem | Cause | Solution | +| --------------------------- | -------------------- | --------------------------------------- | +| SignatureDoesNotMatch | Wrong signature version | JS SDK v3 uses v4 by default, ensure RustFS supports v4 | +| EndpointConnectionError | Endpoint address misconfigured or not started | Check if RustFS address is accessible | +| NoSuchKey | File does not exist | Check if `Key` is spelled correctly | +| InvalidAccessKeyId / Secret | Credentials misconfigured | Check `accessKeyId` / `secretAccessKey` configuration | +| Upload failure (path issue) | Path-style not enabled | Set `forcePathStyle: true` | + +--- + +## VII. Appendix: Frontend Upload Adaptation + +Using presigned URLs allows browsers to upload files directly without passing AccessKey. + +Frontend (HTML+JS) upload example: + +```html + + +``` + +For other operations (object tagging, bucket policies, and more), see the [AWS SDK for JavaScript v3 documentation](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/) — every S3-compatible call works against RustFS the same way. diff --git a/content/de/developer/sdk/meta.json b/content/de/developer/sdk/meta.json new file mode 100644 index 00000000..b0d40a7c --- /dev/null +++ b/content/de/developer/sdk/meta.json @@ -0,0 +1,12 @@ +{ + "title": "SDK", + "pages": [ + "[Java](/de/developer/sdk/java)", + "[Python](/de/developer/sdk/python)", + "[Rust](/de/developer/sdk/rust)", + "[JavaScript](/de/developer/sdk/javascript)", + "[TypeScript](/de/developer/sdk/typescript)", + "[Golang](/de/developer/sdk/go)", + "[Weitere SDKs](/de/developer/sdk/other)" + ] +} diff --git a/content/de/developer/sdk/other.md b/content/de/developer/sdk/other.md new file mode 100644 index 00000000..4b438096 --- /dev/null +++ b/content/de/developer/sdk/other.md @@ -0,0 +1,78 @@ +--- +title: "Other SDKs" +description: "This article mainly explains the usage of various other language SDKs in RustFS." +--- + +If AWS S3 doesn't officially support your language, you can adopt the following strategies to integrate with RustFS: + +## 1. Use HTTP Interface Direct Requests (Based on S3 API Protocol) + +The S3 protocol is a standard RESTful API. You can encapsulate access logic yourself using any language that supports HTTP requests (such as C, Rust, Lua, Erlang). + +### Key points include: + +* **Signature Algorithm**: Implement AWS Signature Version 4 signature (more complex) +* **Construct correct Headers and Canonical Request** +* **Use HTTPS/HTTP client to send requests** + +👉 Recommended to reference open-source project signature implementations, for example: + +* [https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html](https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html) + +--- + +## 2. Call CLI Tools or Middleware Services of Existing SDKs + +If you don't want to implement signatures yourself, you can: + +### 2.1. Use AWS CLI tools with existing language support: + +For example, call through Shell: + +```bash +aws s3 cp local.txt s3://mybucket/myfile.txt --endpoint-url http://rustfs.local:9000 +``` + +Or write a simple relay service using Node.js/Python SDK, and your language uploads/downloads by calling this service. + +### 2.2. Set up a Proxy (such as Flask, FastAPI, Express) + +Let clients that don't support S3 call your encapsulated HTTP API: + +```http +POST /upload -> Service internally calls SDK to upload objects to RustFS +GET /presigned-url -> Generate presigned URL for frontend/client use +``` + +--- + +## 3. Find Third-Party Community SDKs + +Although AWS doesn't have official SDKs, some language communities have developed unofficial S3 clients. For example: + +* Haskell: `amazonka-s3` +* Rust: `rusoto` (deprecated) or `aws-sdk-rust` +* OCaml: May implement through `cohttp` yourself +* Delphi: Has commercial libraries supporting S3 protocol + +Community SDKs vary greatly in stability, so you should evaluate activity, documentation, and compatibility before use. + +--- + +## 4. Delegate Core Upload Logic to Platform Hosting + +For example: + +* Delegate frontend (Web/Mobile) upload tasks to browser or App side execution (using presigned URLs) +* Backend uses Node.js/Python/Go and other proxies to implement upload logic + +--- + +## Summary Recommendations + +| Scenario | Recommended Solution | +| ------------- | ---------------------------------- | +| Need complete control/embedded environment | Implement Signature V4 self-signing | +| Weak language support but has Shell | Call upload through AWS CLI | +| Can deploy relay service | Use Python/Node to build S3 API gateway | +| Frontend upload | Use presigned URLs | diff --git a/content/de/developer/sdk/python.md b/content/de/developer/sdk/python.md new file mode 100644 index 00000000..922ec994 --- /dev/null +++ b/content/de/developer/sdk/python.md @@ -0,0 +1,287 @@ +--- +title: "Python SDK Guide" +description: "Use the official AWS SDK for Python (Boto3) with RustFS." +--- + +## 1. Overview + +RustFS ships no first-party SDKs — it is S3-compatible, so you use the official AWS SDK for Python, [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html), configured to point at your RustFS server. + +This guide covers: + +* Bucket creation/deletion +* Object upload/download/deletion +* Listing objects +* Generating presigned URLs +* Multipart upload for large files + +--- + +## 2. Prerequisites + +* Python 3.8 or later +* A running RustFS instance (see the [Installation Guide](../../installation/index.md)) — the S3 API listens on port `9000`, the Console on port `9001` +* Access keys, set at install time via the `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` environment variables (see [Access Key Management](../../security-compliance/iam/access-token.md)) + +:::tip[Local test] + +If you did not set credentials at install time, the server defaults to `rustfsadmin` / `rustfsadmin` — fine for a throwaway local trial, never for anything reachable by others. + +::: + +### 2.1 Install Boto3 + +We recommend using a virtual environment: + +```bash +python3 -m venv venv +source venv/bin/activate +pip install boto3 +``` + +> Boto3 depends on `botocore`, which will be installed automatically. + +--- + +## 3. Connecting to RustFS + +The following is a complete, runnable script. Replace `localhost` with your server's IP address if RustFS runs on another machine, and fill in your own access keys: + +```python title="main.py" +import boto3 +from botocore.client import Config + +s3 = boto3.client( + 's3', + endpoint_url='http://localhost:9000', + aws_access_key_id='', + aws_secret_access_key='', + region_name='us-east-1', + config=Config( + signature_version='s3v4', + s3={'addressing_style': 'path'}, + ), +) + +response = s3.list_buckets() +for bucket in response['Buckets']: + print(bucket['Name']) +``` + +Run it: + +```bash +python main.py +``` + +:::note[Client parameters explained] + +- `endpoint_url` — points to your RustFS S3 API (port `9000`, not the Console port `9001`) +- `signature_version='s3v4'` — RustFS supports v4 signatures +- `region_name='us-east-1'` — RustFS's default region +- `addressing_style='path'` — RustFS uses path-style URLs by default; virtual-host style requires `RUSTFS_SERVER_DOMAINS` + +::: + +--- + +## 4. Basic Operations + +### 4.1 Create Bucket + +```python +bucket_name = 'my-bucket' + +try: + s3.create_bucket(Bucket=bucket_name) + print(f'Bucket {bucket_name} created.') +except s3.exceptions.BucketAlreadyOwnedByYou: + print(f'Bucket {bucket_name} already exists.') +``` + +```text +Bucket my-bucket created. +``` + +--- + +### 4.2 Upload File + +```python +s3.upload_file('/path/to/hello.txt', bucket_name, 'hello.txt') +print('File uploaded.') +``` + +```text +File uploaded. +``` + +--- + +### 4.3 Download File + +```python +s3.download_file(bucket_name, 'hello.txt', 'hello-downloaded.txt') +print('File downloaded.') +``` + +```text +File downloaded. +``` + +--- + +### 4.4 List Objects + +```python +response = s3.list_objects_v2(Bucket=bucket_name) +for obj in response.get('Contents', []): + print(f"- {obj['Key']} ({obj['Size']} bytes)") +``` + +```text +- hello.txt (12 bytes) +``` + +--- + +### 4.5 Delete Object and Bucket + +```python +s3.delete_object(Bucket=bucket_name, Key='hello.txt') +print('Object deleted.') + +s3.delete_bucket(Bucket=bucket_name) +print('Bucket deleted.') +``` + +```text +Object deleted. +Bucket deleted. +``` + +--- + +## 5. Advanced Features + +### 5.1 Generate Presigned URLs + +#### 5.1.1 Download Link (GET) + +```python +url = s3.generate_presigned_url( + ClientMethod='get_object', + Params={'Bucket': bucket_name, 'Key': 'hello.txt'}, + ExpiresIn=600, # 10 minutes validity +) + +print('Presigned GET URL:', url) +``` + +```text +Presigned GET URL: http://localhost:9000/my-bucket/hello.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&... +``` + +#### 5.1.2 Upload Link (PUT) + +```python +url = s3.generate_presigned_url( + ClientMethod='put_object', + Params={'Bucket': bucket_name, 'Key': 'upload-by-url.txt'}, + ExpiresIn=600, +) + +print('Presigned PUT URL:', url) +``` + +You can use the `curl` tool to upload: + +```bash +curl -X PUT --upload-file /path/to/hello.txt "http://localhost:9000/my-bucket/upload-by-url.txt?X-Amz-Algorithm=..." +``` + +--- + +### 5.2 Multipart Upload + +Suitable for files larger than 10 MB, allows manual control of each part. + +```python +file_path = 'largefile.bin' +key = 'largefile.bin' +part_size = 5 * 1024 * 1024 # 5 MB + +# 1. Start upload +response = s3.create_multipart_upload(Bucket=bucket_name, Key=key) +upload_id = response['UploadId'] +parts = [] + +try: + with open(file_path, 'rb') as f: + part_number = 1 + while True: + data = f.read(part_size) + if not data: + break + + part = s3.upload_part( + Bucket=bucket_name, + Key=key, + PartNumber=part_number, + UploadId=upload_id, + Body=data, + ) + + parts.append({'ETag': part['ETag'], 'PartNumber': part_number}) + print(f'Uploaded part {part_number}') + part_number += 1 + + # 2. Complete upload + s3.complete_multipart_upload( + Bucket=bucket_name, + Key=key, + UploadId=upload_id, + MultipartUpload={'Parts': parts}, + ) + print('Multipart upload complete.') + +except Exception as e: + # Abort upload + s3.abort_multipart_upload(Bucket=bucket_name, Key=key, UploadId=upload_id) + print('Multipart upload aborted due to error:', e) +``` + +```text +Uploaded part 1 +Uploaded part 2 +Uploaded part 3 +Multipart upload complete. +``` + +--- + +## 6. Common Issue Troubleshooting + +| Issue | Cause | Solution | +| ------------------------- | ----------------- | -------------------------------------------------------------- | +| `SignatureDoesNotMatch` | Not using v4 signature | Set `signature_version='s3v4'` | +| `EndpointConnectionError` | Wrong RustFS address or service not started | Check endpoint and RustFS service status | +| `AccessDenied` | Wrong credentials or insufficient permissions | Check AccessKey/SecretKey or bucket policies | +| `PermanentRedirect` / wrong bucket URL | Path-style not enabled | Set `s3={'addressing_style': 'path'}` in the `Config` | + +--- + +## 7. Appendix: Quick Upload/Download Script Template + +```python +def upload_file(local_path, bucket, object_key): + s3.upload_file(local_path, bucket, object_key) + print(f"Uploaded {local_path} to s3://{bucket}/{object_key}") + + +def download_file(bucket, object_key, local_path): + s3.download_file(bucket, object_key, local_path) + print(f"Downloaded s3://{bucket}/{object_key} to {local_path}") +``` + +For other operations (object tagging, bucket policies, and more), see the [Boto3 S3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html) — every S3-compatible call works against RustFS the same way. diff --git a/content/de/developer/sdk/rust.md b/content/de/developer/sdk/rust.md new file mode 100644 index 00000000..9d818134 --- /dev/null +++ b/content/de/developer/sdk/rust.md @@ -0,0 +1,287 @@ +--- +title: "Rust SDK Guide" +description: "Operate RustFS instances through the AWS SDK for Rust, including creation and deletion of buckets and objects." +--- + +RustFS ships no first-party Rust client crate — it is fully S3-compatible, so you use the official AWS SDK for Rust (`aws-sdk-s3`) configured to point at your RustFS server. Through the SDK, you can operate RustFS, including creation and deletion of buckets/objects, file upload and download, etc. + +## Prerequisites + +- Rust 1.78 or later (install via [rustup](https://rustup.rs/)) +- An available RustFS instance (refer to [Installation Guide](../../installation/index.md)) — the S3 API listens on port `9000`, the Console on port `9001` +- Access keys, set at install time via the `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` environment variables (refer to [Access Key Management](../../security-compliance/iam/access-token.md)) + +:::tip[Local test] + +If you did not set credentials at install time, the server defaults to `rustfsadmin` / `rustfsadmin` — fine for a throwaway local trial, never for anything reachable by others. + +::: + +Create a project and add the dependencies: + +```bash +cargo new rustfs-rust-demo && cd rustfs-rust-demo +cargo add aws-config aws-sdk-s3 anyhow +cargo add tokio --features full +``` + +Your `Cargo.toml` should contain: + +```toml title="Cargo.toml" +[dependencies] +anyhow = "1" +aws-config = "1" +aws-sdk-s3 = "1" +tokio = { version = "1", features = ["full"] } +``` + +## Initializing the Client + +The following is a complete, runnable program. It loads the connection settings from environment variables, initializes the S3 client, and lists your buckets: + +```rust title="src/main.rs" +use anyhow::Result; +use aws_config::BehaviorVersion; +use aws_sdk_s3::config::{Credentials, Region}; +use aws_sdk_s3::Client; +use std::env; + +pub struct Config { + pub region: String, + pub access_key_id: String, + pub secret_access_key: String, + pub endpoint_url: String, +} + +impl Config { + pub fn from_env() -> Result { + let region = env::var("RUSTFS_REGION")?; + let access_key_id = env::var("RUSTFS_ACCESS_KEY_ID")?; + let secret_access_key = env::var("RUSTFS_SECRET_ACCESS_KEY")?; + let endpoint_url = env::var("RUSTFS_ENDPOINT_URL")?; + + Ok(Config { + region, + access_key_id, + secret_access_key, + endpoint_url, + }) + } +} + +#[tokio::main] +async fn main() -> Result<()> { + let config = Config::from_env()?; + + let credentials = Credentials::new( + config.access_key_id, + config.secret_access_key, + None, + None, + "rustfs", + ); + + let region = Region::new(config.region); + + let shared_config = aws_config::defaults(BehaviorVersion::latest()) + .region(region) + .credentials_provider(credentials) + .endpoint_url(config.endpoint_url) + .load() + .await; + + // RustFS uses path-style URLs by default; virtual-host style requires RUSTFS_SERVER_DOMAINS + let s3_config = aws_sdk_s3::config::Builder::from(&shared_config) + .force_path_style(true) + .build(); + + let rustfs_client = Client::from_conf(s3_config); + + let res = rustfs_client.list_buckets().send().await?; + for bucket in res.buckets() { + println!("Bucket: {:?}", bucket.name()); + } + + Ok(()) +} +``` + +:::note + +These environment variable names (`RUSTFS_ENDPOINT_URL`, `RUSTFS_REGION`, `RUSTFS_ACCESS_KEY_ID`, `RUSTFS_SECRET_ACCESS_KEY`) are just this example's client-side conventions — they are read by your program, not by RustFS. They are distinct from the server-side `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` variables used when installing RustFS. + +::: + +Run it (replace `localhost` with your server's IP address if RustFS runs on another machine): + +```bash +export RUSTFS_ENDPOINT_URL="http://localhost:9000" +export RUSTFS_REGION="us-east-1" +export RUSTFS_ACCESS_KEY_ID="" +export RUSTFS_SECRET_ACCESS_KEY="" +cargo run +``` + +```text +Bucket: Some("my-bucket") +``` + +You can now use the client for the operations below. Each snippet runs inside `main`, reusing `rustfs_client`. + +## Create Bucket + +```rust +match rustfs_client + .create_bucket() + .bucket("my-bucket") + .send() + .await +{ + Ok(_) => { + println!("Bucket created successfully"); + } + Err(e) => { + println!("Error creating bucket: {:?}", e); + return Err(e.into()); + } +} +``` + +```text +Bucket created successfully +``` + +## Delete Bucket + +```rust +match rustfs_client + .delete_bucket() + .bucket("my-bucket") + .send() + .await +{ + Ok(_) => { + println!("Bucket deleted successfully"); + } + Err(e) => { + println!("Error deleting bucket: {:?}", e); + return Err(e.into()); + } +} +``` + +```text +Bucket deleted successfully +``` + +## List Buckets + +```rust +match rustfs_client.list_buckets().send().await { + Ok(res) => { + println!("Total buckets number is {:?}", res.buckets().len()); + for bucket in res.buckets() { + println!("Bucket: {:?}", bucket.name()); + } + } + Err(e) => { + println!("Error listing buckets: {:?}", e); + return Err(e.into()); + } +} +``` + +```text +Total buckets number is 1 +Bucket: Some("my-bucket") +``` + +## List Objects + +```rust +match rustfs_client + .list_objects_v2() + .bucket("my-bucket") + .send() + .await +{ + Ok(res) => { + println!("Total objects number is {:?}", res.contents().len()); + for object in res.contents() { + println!("Object: {:?}", object.key()); + } + } + Err(e) => { + println!("Error listing objects: {:?}", e); + return Err(e.into()); + } +} +``` + +```text +Total objects number is 1 +Object: Some("hello.txt") +``` + +## Upload File + +Add these imports at the top of `src/main.rs`: + +```rust +use aws_sdk_s3::primitives::ByteStream; +use tokio::fs; +``` + +Then upload a local file: + +```rust +let data = fs::read("/path/to/hello.txt").await.expect("can not open the file"); + +match rustfs_client + .put_object() + .bucket("my-bucket") + .key("hello.txt") + .body(ByteStream::from(data)) + .send() + .await +{ + Ok(res) => { + println!("Object uploaded successfully, res: {:?}", res); + } + Err(e) => { + println!("Error uploading object: {:?}", e); + return Err(e.into()); + } +} +``` + +```text +Object uploaded successfully, res: PutObjectOutput { e_tag: Some("\"...\""), ... } +``` + +## Download Object + +```rust +match rustfs_client + .get_object() + .bucket("my-bucket") + .key("hello.txt") + .send() + .await +{ + Ok(res) => { + let data = res.body.collect().await?.into_bytes(); + println!("Object content: {}", String::from_utf8_lossy(&data)); + } + Err(e) => { + println!("Error downloading object: {:?}", e); + return Err(e.into()); + } +} +``` + +```text +Object content: hello rustfs +``` + +For other operations (presigned URLs, multipart uploads, and more), see the [AWS SDK for Rust documentation](https://docs.aws.amazon.com/sdk-for-rust/latest/dg/) — every S3-compatible call works against RustFS the same way. diff --git a/content/de/developer/sdk/typescript.md b/content/de/developer/sdk/typescript.md new file mode 100644 index 00000000..d26b7a93 --- /dev/null +++ b/content/de/developer/sdk/typescript.md @@ -0,0 +1,204 @@ +--- +title: "RustFS TypeScript SDK Usage Guide" +description: "Operate RustFS from TypeScript with the AWS SDK for JavaScript v3, including creation and deletion of buckets and objects." +--- + +RustFS ships no first-party TypeScript SDK — it is fully S3-compatible, so you use the official AWS SDK for JavaScript v3 (which ships its own TypeScript type definitions) configured to point at your RustFS server. Through the SDK, you can operate RustFS, including creation and deletion of buckets/objects, file upload and download, etc. + +## Prerequisites + +- Node.js 18 or later (the examples use ES modules — set `"type": "module"` in your `package.json`) +- An available RustFS instance (refer to [Installation Guide](../../installation/index.md) for installation) — the S3 API listens on port `9000`, the Console on port `9001` +- Access keys, set at install time via the `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` environment variables (refer to [Access Key Management](../../security-compliance/iam/access-token.md) for creation) + +:::tip[Local test] + +If you did not set credentials at install time, the server defaults to `rustfsadmin` / `rustfsadmin` — fine for a throwaway local trial, never for anything reachable by others. + +::: + +Install the dependencies: + +```bash +npm install @aws-sdk/client-s3 +npm install --save-dev typescript tsx @types/node +``` + +## Initializing the Client + +The following is a complete, runnable example. Replace `localhost` with your server's IP address if RustFS runs on another machine, and fill in your own access keys: + +```typescript title="main.ts" +import { + S3Client, + CreateBucketCommand, + DeleteBucketCommand, + ListBucketsCommand, + ListObjectsV2Command, + PutObjectCommand, + GetObjectCommand, +} from "@aws-sdk/client-s3"; +import * as fs from "fs"; + +const rustfs_client = new S3Client({ + region: "us-east-1", // RustFS default region + endpoint: "http://localhost:9000", // RustFS S3 API address + credentials: { + accessKeyId: "", + secretAccessKey: "", + }, + // RustFS uses path-style URLs by default; virtual-host style requires RUSTFS_SERVER_DOMAINS + forcePathStyle: true, +}); + +const response = await rustfs_client.send(new ListBucketsCommand({})); +console.log(response.Buckets?.map((bucket) => bucket.Name) ?? []); +``` + +Run it: + +```bash +npx tsx main.ts +``` + +```text +[ 'my-bucket' ] +``` + +Then use the constructed `rustfs_client` for the operations below. + +## Create Bucket + +```typescript +async function createBucket() { + try { + const response = await rustfs_client.send( + new CreateBucketCommand({ + Bucket: "my-bucket", + }) + ); + console.log("Bucket created:", response.Location); + } catch (error) { + console.log(error); + } +} +``` + +```text +Bucket created: /my-bucket +``` + +## Delete Bucket + +```typescript +async function deleteBucket() { + try { + await rustfs_client.send( + new DeleteBucketCommand({ + Bucket: "my-bucket", + }) + ); + console.log("Bucket deleted"); + } catch (error) { + console.log(error); + } +} +``` + +```text +Bucket deleted +``` + +## List Buckets + +```typescript +async function listBuckets() { + try { + const response = await rustfs_client.send(new ListBucketsCommand({})); + response.Buckets?.forEach((bucket) => console.log(bucket.Name)); + } catch (error) { + console.log(error); + } +} +``` + +```text +my-bucket +``` + +## List Objects + +```typescript +async function listObjects() { + try { + const response = await rustfs_client.send( + new ListObjectsV2Command({ + Bucket: "my-bucket", + }) + ); + response.Contents?.forEach((obj) => console.log(`${obj.Key} (${obj.Size} bytes)`)); + } catch (error) { + console.log(error); + } +} +``` + +```text +test/hello.txt (12 bytes) +``` + +## Upload File + +```typescript +async function uploadFile() { + try { + await rustfs_client.send( + new PutObjectCommand({ + Bucket: "my-bucket", + Key: "test/hello.txt", + Body: fs.createReadStream("/path/to/hello.txt"), + }) + ); + console.log("Object uploaded"); + } catch (error) { + console.log(error); + } +} +``` + +```text +Object uploaded +``` + +## Download Object + +```typescript +async function getObject() { + try { + const response = await rustfs_client.send( + new GetObjectCommand({ + Bucket: "my-bucket", + Key: "test/hello.txt", + }) + ); + + // get object content + if (response.Body) { + const chunks: Buffer[] = []; + for await (const chunk of response.Body as any) { + chunks.push(chunk as Buffer); + } + const data = Buffer.concat(chunks).toString("utf-8"); + console.log("Object content:", data); + } + } catch (error) { + console.log(error); + } +} +``` + +```text +Object content: hello rustfs +``` + +For other operations (presigned URLs, multipart uploads, and more), see the [AWS SDK for JavaScript v3 documentation](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/) — every S3-compatible call works against RustFS the same way. diff --git a/content/de/index.mdx b/content/de/index.mdx new file mode 100644 index 00000000..a8a8e8cc --- /dev/null +++ b/content/de/index.mdx @@ -0,0 +1,83 @@ +--- +title: "RustFS-Dokumentation" +description: "Installieren Sie RustFS unter Linux, in Containern, auf Kubernetes, unter macOS oder Windows." +--- + +
+
+
+ RustFS ist ein in Rust entwickeltes, verteiltes Objektspeichersystem für + S3-kompatible Workloads. Diese Dokumentation behandelt die Installation, + Administration, Sicherheit und den Betrieb von RustFS-Clustern. +
+ + Installationsvoraussetzungen prüfen + +
+ +
+ + + + Unter Linux installieren + Beginnen Sie auf einem Linux-Server und wählen Sie anschließend eine Einzelknoten- oder verteilte Topologie. + + + + + + + Mit Docker installieren + Führen Sie RustFS zur lokalen Evaluierung oder für verwaltete Bereitstellungen als Container aus. + + + + + + + Mit Podman installieren + Stellen Sie mit Podman einen RustFS-Container ohne Root-Rechte oder Daemon bereit. + + + + + + + Mit Helm installieren + Stellen Sie mit dem RustFS-Helm-Chart einen RustFS-Cluster auf Kubernetes bereit. + + + + + + + Mit dem Operator installieren + Verwalten Sie Kubernetes-native RustFS-Tenants, Speicherpools und Lebenszyklusvorgänge. + + + + + + + Unter macOS installieren + Führen Sie RustFS lokal auf Macs mit Apple-Chip oder Intel-Prozessor aus. + + + + + + + Unter Windows installieren + Installieren und betreiben Sie RustFS in einer unterstützten Windows-Umgebung. + + +
+
diff --git a/content/de/installation/cloud-native/helm-chart/cert-manager.md b/content/de/installation/cloud-native/helm-chart/cert-manager.md new file mode 100644 index 00000000..5386eaf3 --- /dev/null +++ b/content/de/installation/cloud-native/helm-chart/cert-manager.md @@ -0,0 +1,69 @@ +--- +title: cert-manager +description: Use cert-manager to issue and renew TLS certificates for a RustFS Helm deployment. +--- + +Use **cert-manager** to issue the certificate referenced by the RustFS Ingress. This provides HTTPS for the public S3 API and Console endpoint and allows cert-manager to renew the certificate. + +## Requirements + +- cert-manager is installed and its controller Pods are ready. +- An `Issuer` or `ClusterIssuer` is ready. +- The RustFS hostname resolves to the Ingress controller. + +Verify the issuer before installing RustFS: + +```bash +kubectl get clusterissuer letsencrypt-prod +kubectl -n cert-manager get pods +``` + +## 1. Configure Ingress TLS + +Add the Ingress and certificate settings to your standalone or distributed values file: + +```yaml title="values.yaml" +ingress: + enabled: true + className: nginx + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + hosts: + - host: s3.example.com + paths: + - path: / + pathType: Prefix + tls: + enabled: true + certManager: + enabled: true + existingSecret: + enabled: false + name: "" +``` + +Replace the Ingress class, issuer, and hostname for your cluster. For a namespace-scoped Issuer, use the `cert-manager.io/issuer` annotation instead. + +## 2. Apply the configuration + +```bash +helm upgrade rustfs ./helm/rustfs \ + --namespace rustfs \ + -f values.yaml +``` + +The Ingress references the `rustfs-tls` Secret for a release named `rustfs`. cert-manager ingress-shim reads the issuer annotation and creates a Certificate that writes to this Secret. + +## 3. Verify the certificate + +```bash +kubectl -n rustfs get ingress,certificate,certificaterequest +kubectl -n rustfs describe certificate rustfs-tls +kubectl -n rustfs get secret rustfs-tls +``` + +Wait for the Certificate to report `Ready=True`, then open `https://s3.example.com`. + +## Use cert-manager with mTLS + +The chart also uses cert-manager to issue server and client certificates when `mtls.enabled=true`. See [mTLS](./mtls.md) to use the chart-managed CA or reference an existing issuer. \ No newline at end of file diff --git a/content/de/installation/cloud-native/helm-chart/index.md b/content/de/installation/cloud-native/helm-chart/index.md new file mode 100644 index 00000000..26652e5b --- /dev/null +++ b/content/de/installation/cloud-native/helm-chart/index.md @@ -0,0 +1,21 @@ +--- +title: Overview +description: Understand how the RustFS Helm chart deploys standalone and distributed clusters on Kubernetes. +--- + +The official **RustFS Helm chart** deploys one RustFS cluster directly into Kubernetes. Helm renders the workload, Services, credentials, configuration, PersistentVolumeClaims (PVCs), Ingress, and optional certificate resources from a single values file. + +The chart supports two deployment modes: + +- **Standalone** creates one Pod with one data PVC. Use it for evaluation and development. +- **Distributed** creates a StatefulSet with multiple Pods and data PVCs. `replicaCount` controls the number of Pods and `drivesPerNode` controls the number of data PVCs mounted by each Pod. + +Distributed mode is enabled by default. The chart also supports multiple append-only server pools, but a single explicit topology is easier to operate for an initial deployment. Credentials must be supplied through chart values or an existing Secret; the chart rejects empty and well-known default credentials unless insecure development defaults are explicitly enabled. + +Use the Helm chart when you want Helm to manage one RustFS cluster. Use the [RustFS Operator](../operator/index.md) when you need Kubernetes custom resources, multiple Tenants, or Operator-driven pool management. + +## Helm chart workflows + +- [Install](./installation.mdx) covers requirements and standalone or distributed deployment. +- [mTLS](./mtls.md) encrypts and authenticates traffic between RustFS Pods. +- [cert-manager](./cert-manager.md) issues and renews certificates for RustFS Ingress and mTLS. \ No newline at end of file diff --git a/content/de/installation/cloud-native/helm-chart/installation.mdx b/content/de/installation/cloud-native/helm-chart/installation.mdx new file mode 100644 index 00000000..5523fffb --- /dev/null +++ b/content/de/installation/cloud-native/helm-chart/installation.mdx @@ -0,0 +1,168 @@ +--- +title: Install +description: Install standalone or distributed RustFS on Kubernetes with the official Helm chart. +--- + +This guide gets the official RustFS chart from source or the Helm repository, installs it, and verifies access to the S3 API and Console. + +## Requirements + +| Component | Requirement | +|-----------|-------------| +| Helm | Version 3 | +| Kubernetes | A cluster reachable with `kubectl` | +| StorageClass | Dynamic PVC provisioning for RustFS data | +| RustFS | `1.0.0-alpha.69` or later | + +An Ingress controller is optional. If you enable Ingress, set `ingress.className` to the controller in your cluster, such as `nginx` or `traefik`. + +Check the active cluster and available StorageClasses: + +```bash +kubectl config current-context +kubectl get storageclass +helm version --short +``` + +## 1. Get the chart + +Choose how Helm should access the chart. Both options set `RUSTFS_CHART`, which the installation commands below use. + + + + +Clone the RustFS source repository to use the chart at `helm/rustfs`: + +```bash +git clone https://github.com/rustfs/rustfs.git +cd rustfs +export RUSTFS_CHART=./helm/rustfs +``` + + + + +Add the RustFS repository listed on [Artifact Hub](https://artifacthub.io/packages/helm/rustfs/rustfs), then update the local repository index: + +```bash +helm repo add rustfs https://charts.rustfs.com +helm repo update +export RUSTFS_CHART=rustfs/rustfs +``` + + + + +## 2. Install standalone mode + +Create a values file for a one-Pod development deployment: + +```yaml title="standalone-values.yaml" +mode: + standalone: + enabled: true + distributed: + enabled: false + +secret: + rustfs: + access_key: "" + secret_key: "" + +storageclass: + name: standard + dataStorageSize: 10Gi + logStorageSize: 1Gi + +ingress: + enabled: false +``` + +Replace `standard` with a StorageClass in your cluster, then install: + +```bash +helm upgrade --install rustfs "$RUSTFS_CHART" \ + --namespace rustfs \ + --create-namespace \ + -f standalone-values.yaml +``` + +## 3. Install distributed mode + +For a distributed cluster, set the Pod and drive counts explicitly. Total data drives equal `replicaCount * drivesPerNode`. + +```yaml title="distributed-values.yaml" +mode: + standalone: + enabled: false + distributed: + enabled: true + +replicaCount: 4 +drivesPerNode: 2 + +secret: + rustfs: + access_key: "" + secret_key: "" + +storageclass: + name: standard + dataStorageSize: 100Gi + logStorageSize: 1Gi + +ingress: + enabled: false +``` + +The example creates four Pods and eight data PVCs. Ensure the cluster can schedule all Pods and provision all PVCs, then install: + +```bash +helm upgrade --install rustfs "$RUSTFS_CHART" \ + --namespace rustfs \ + --create-namespace \ + -f distributed-values.yaml +``` + +:::warning[Choose the topology before installation] + +Kubernetes does not allow updates to StatefulSet `volumeClaimTemplates`. Changing `drivesPerNode` later requires StatefulSet recreation or a new installation. + +::: + +## 4. Verify and access RustFS + +```bash +kubectl -n rustfs get pods,pvc,services +kubectl -n rustfs rollout status statefulset/rustfs +``` + +Standalone mode creates a Deployment instead of a StatefulSet. Check it with: + +```bash +kubectl -n rustfs rollout status deployment/rustfs +``` + +Forward the S3 API and Console to your workstation: + +```bash +kubectl -n rustfs port-forward svc/rustfs 9000:9000 9001:9001 +``` + +Use `http://localhost:9000` as the S3 endpoint and open `http://localhost:9001` for the Console. + +## Key values + +| Value | Purpose | Chart default | +|-------|---------|---------------| +| `mode.standalone.enabled` | Enable one-Pod standalone mode | `false` | +| `mode.distributed.enabled` | Enable distributed mode | `true` | +| `replicaCount` | Distributed Pod count | `4` | +| `drivesPerNode` | Data PVCs per Pod | Inferred from `replicaCount` | +| `storageclass.name` | StorageClass for PVCs | `local-path` | +| `storageclass.dataStorageSize` | Size of each data PVC | `256Mi` | +| `storageclass.logStorageSize` | Size of each log PVC | `256Mi` | +| `service.endpoint.port` | S3 API port | `9000` | +| `service.console.port` | Console port | `9001` | + +We recommend setting storage sizes explicitly; the chart defaults are intended only for basic evaluation. \ No newline at end of file diff --git a/content/de/installation/cloud-native/helm-chart/meta.json b/content/de/installation/cloud-native/helm-chart/meta.json new file mode 100644 index 00000000..bc612082 --- /dev/null +++ b/content/de/installation/cloud-native/helm-chart/meta.json @@ -0,0 +1,8 @@ +{ + "title": "Helm-Chart", + "pages": [ + "installation", + "mtls", + "cert-manager" + ] +} \ No newline at end of file diff --git a/content/de/installation/cloud-native/helm-chart/mtls.md b/content/de/installation/cloud-native/helm-chart/mtls.md new file mode 100644 index 00000000..5144b472 --- /dev/null +++ b/content/de/installation/cloud-native/helm-chart/mtls.md @@ -0,0 +1,66 @@ +--- +title: mTLS +description: Configure mutual TLS between RustFS Pods deployed by the Helm chart. +--- + +The chart can enable mutual Transport Layer Security (mTLS) for RustFS Pod communication. When enabled, RustFS requires client certificates, uses HTTPS for generated peer URLs, and mounts server, client, and CA material into every Pod. + +## Requirements + +mTLS uses cert-manager `Issuer` and `Certificate` resources. Install cert-manager before enabling it and confirm that its CRDs are available: + +```bash +kubectl get crd certificates.cert-manager.io issuers.cert-manager.io +``` + +## 1. Use the chart-managed CA + +Add the following setting to your existing values file: + +```yaml title="values.yaml" +mtls: + enabled: true +``` + +Upgrade the release: + +```bash +helm upgrade rustfs ./helm/rustfs \ + --namespace rustfs \ + -f values.yaml +``` + +The chart creates a self-signed root CA, a namespace Issuer, and server and client Certificates. It mounts the resulting Secrets and configures RustFS with `RUSTFS_SERVER_MTLS_ENABLE=1` and `RUSTFS_TLS_PATH=/opt/tls`. Health probes also use the generated client certificate. + +## 2. Use an existing Issuer + +To use an Issuer or ClusterIssuer already managed by your platform, configure its reference: + +```yaml title="values.yaml" +mtls: + enabled: true + existingIssuerRef: + enabled: true + name: internal-ca + kind: ClusterIssuer + group: cert-manager.io +``` + +The issuer must be ready and able to issue both server and client certificates in the `rustfs` namespace. Use `kind: Issuer` for a namespace-scoped issuer. + +## 3. Verify mTLS + +```bash +kubectl -n rustfs get issuer,certificate,secret +kubectl -n rustfs describe certificate rustfs-server-tls +kubectl -n rustfs describe certificate rustfs-client-tls +kubectl -n rustfs get pods +``` + +For a release named `rustfs`, the generated certificate Secrets are `rustfs-server-tls` and `rustfs-client-tls`. + +:::warning[Plan external access] + +mTLS requires clients to present a trusted certificate. Validate how your Ingress controller or other external client presents that certificate before enabling mTLS on an existing deployment. + +::: \ No newline at end of file diff --git a/content/de/installation/cloud-native/index.md b/content/de/installation/cloud-native/index.md new file mode 100644 index 00000000..ef338b58 --- /dev/null +++ b/content/de/installation/cloud-native/index.md @@ -0,0 +1,201 @@ +--- +title: "Kubernetes Installation (Helm)" +description: "Deploy RustFS on Kubernetes with the official Helm chart: standalone or distributed mode, storage sizing, probes, production hardening, and server-pool expansion." +--- + +RustFS ships an official Helm chart that deploys either a single-node instance (a `Deployment` with one PVC) or a distributed cluster (a `StatefulSet` with multiple pods and PVCs). This guide walks through installing the chart, choosing a deployment mode, sizing storage correctly, and the production options the chart provides. + +**Prerequisites** + +- A Kubernetes cluster and `kubectl` access +- Helm 3 +- RustFS image version `>= 1.0.0-alpha.69` (the chart requirement) +- A StorageClass with a working provisioner — the chart defaults to [`local-path`](https://github.com/rancher/local-path-provisioner); set `storageclass.name` to use your own +- [`rc`](/operations/rc) installed on the administration host before using the server-pool commands in this guide + +The chart lives in the RustFS source repository under `helm/rustfs`: + +```bash +git clone https://github.com/rustfs/rustfs.git +cd rustfs/helm/rustfs +``` + +## 1. Quick install + +Install into a dedicated namespace, setting your own credentials and a realistic data size: + +```bash +helm install rustfs . \ + --namespace rustfs --create-namespace \ + --set secret.rustfs.access_key= \ + --set secret.rustfs.secret_key= \ + --set storageclass.dataStorageSize=100Gi \ + --set storageclass.logStorageSize=1Gi +``` + +:::note[The chart refuses default credentials] + +Rendering fails by default unless one of the following is true: + +1. `secret.existingSecret` names a Kubernetes Secret you control, or +2. `secret.rustfs.access_key` and `secret.rustfs.secret_key` are **both** set to non-empty, non-default values, or +3. `secret.allowInsecureDefaults: true` is set (only for local development). + +This prevents accidental deployment with the well-known default `rustfsadmin`/`rustfsadmin` credentials. Setting only one of the two keys is also rejected, so the chart never silently falls back to a default for the missing key. + +::: + +Watch the pods come up: + +```bash +kubectl -n rustfs get pods -w +``` + +```text +NAME READY STATUS RESTARTS AGE +rustfs-0 1/1 Running 0 2m27s +rustfs-1 1/1 Running 0 2m27s +rustfs-2 1/1 Running 0 2m27s +rustfs-3 1/1 Running 0 2m27s +``` + +## 2. Choose a deployment mode + +The chart supports two modes, selected via the `mode` values: + +| Mode | Values | Workload | Layout | +| --- | --- | --- | --- | +| Distributed (**default**) | `mode.distributed.enabled=true` | StatefulSet | `replicaCount: 4` pods, 4 data PVCs each (16 drives total) — or `replicaCount: 16` for 16 pods with 1 data PVC each | +| Standalone | `mode.standalone.enabled=true`, `mode.distributed.enabled=false` | Deployment | 1 pod, 1 data PVC (single node single disk) | + +- **Standalone** matches single-node single-disk: no erasure-coding redundancy across nodes. Use it for development, testing, or small setups where the underlying storage provides its own durability. It can reuse existing PVCs via `mode.standalone.existingClaim.dataClaim` / `mode.standalone.existingClaim.logsClaim`. +- **Distributed** behaves like [multiple node multiple disk](../linux/multiple-node-multiple-disk.md): objects are erasure-coded across pods and PVCs. `replicaCount` must be `4` (each pod gets 4 PVCs) or `16` (each pod gets 1 PVC); pick based on how many nodes your cluster can spread pods across. + +```bash +# Standalone mode +helm install rustfs . -n rustfs --create-namespace \ + --set mode.standalone.enabled=true \ + --set mode.distributed.enabled=false \ + --set secret.rustfs.access_key= \ + --set secret.rustfs.secret_key= +``` + +## 3. Storage sizing + +PVC sizes come from the `storageclass` block: + +```yaml title="values-prod.yaml" +storageclass: + name: local-path # your StorageClass + dataStorageSize: 256Mi # per data PVC + logStorageSize: 256Mi # per logs PVC +``` + +:::warning[The default PVC size is 256Mi — change it] + +The chart's default size for the data and logs volumes is **256Mi**, which is only enough to verify the chart works. For any real workload set `storageclass.dataStorageSize` (for example `1Ti`) and `storageclass.logStorageSize` (for example `1Gi`) at install time. In distributed mode the data size applies to **each** data PVC (16 PVCs by default). + +::: + +Setting `config.rustfs.obs_log_directory` to `""` disables the log PVCs and mounts entirely. Custom PVC annotations go under `storageclass.pvcAnnotations.data` / `storageclass.pvcAnnotations.logs`. + +## 4. Health probes + +The chart templates HTTP probes on the S3 port (9000) out of the box, matching the server's health endpoints: + +- **Liveness**: `GET /health` (`livenessProbe.httpGet.path`), initial delay 30s, period 5s +- **Readiness**: `GET /health/ready` (`readinessProbe.httpGet.path`), initial delay 10s, period 5s + +A pod is only added to the Service endpoints once `/health/ready` returns `200`, which in distributed mode requires the storage quorum to be met. Thresholds and timings are tunable via the `livenessProbe.*` and `readinessProbe.*` values. + +## 5. Production hardening + +### Pod Disruption Budget + +Disabled by default. Enable it so voluntary evictions (node drains, cluster upgrades) never take more than one pod down at a time: + +```bash +--set pdb.create=true # pdb.maxUnavailable defaults to 1 +``` + +### Anti-affinity and topology spread + +`affinity.podAntiAffinity.enabled` defaults to `true` with `topologyKey: kubernetes.io/hostname`, spreading pods across distinct nodes. For zone-level spreading, enable `topologySpreadConstraints.enabled` and supply raw constraint entries under `topologySpreadConstraints.constraints` (applied to the distributed StatefulSet). + +### Inter-pod mTLS (cert-manager) + +Set `mtls.enabled=true` to encrypt traffic between pods; the chart renders cert-manager `Issuer`/`Certificate` resources for a CA, server, and client certificates. To use an issuer you already operate, set `mtls.existingIssuerRef.enabled=true` with its `name`, `kind` (`Issuer` or `ClusterIssuer`), and `group`. + +### Ingress and Gateway API + +Ingress is enabled by default (`ingress.enabled=true`) with `ingress.className: nginx`; set it to `traefik` if that is your controller — the chart applies the matching session-stickiness annotations for each. Set your domain via `ingress.hosts[0].host` (default `example.rustfs.com`). For HTTPS, enable `ingress.tls.enabled` and either pass the certificate with `--set-file ingress.tls.crt=./tls.crt --set-file ingress.tls.key=./tls.key`, point at an existing secret (`ingress.tls.existingSecret`), or let cert-manager issue one (`ingress.tls.certManager.enabled=true`). + +The chart also has alpha [Gateway API](https://gateway-api.sigs.k8s.io/) support (`gatewayApi.enabled=true` together with `ingress.enabled=false`, Traefik gateway class), rendering `Gateway` and `HTTPRoute` resources. + +## 6. Access RustFS + +Without an ingress, port-forward the Service: + +```bash +kubectl -n rustfs port-forward svc/rustfs 9000:9000 9001:9001 +``` + +- S3 API: `http://localhost:9000` +- Console: `http://localhost:9001` + +Log in to the Console with the access key and secret key you set at install time. With ingress enabled, use your configured host instead (check `kubectl -n rustfs get ing`). The Service defaults to `ClusterIP`; `service.type` can be switched to `NodePort` (S3 on `service.endpoint.nodePort: 32000`, Console on `service.console.nodePort: 32001`) or `LoadBalancer`. + +## 7. Scaling out with server pools + +In distributed mode the chart can run multiple **server pools** — independent StatefulSets whose drives together form one cluster. This is the chart-level equivalent of adding a Server Pool as described in [Pool Expansion](../../operations/scaling/storage-pool-expansion.md). + +To expand an existing deployment, enable pools and describe the current layout as pool 0 plus your new capacity: + +```yaml title="values-prod.yaml (pools)" +pools: + enabled: true + list: + - {} # pool 0: inherits top-level values and keeps the + # existing StatefulSet/pod/PVC names and data + - replicaCount: 4 # pool 1: new capacity (4 or 16) + storageclass: + dataStorageSize: 10Gi +``` + +Then apply with `helm upgrade`. Each entry may set `replicaCount` (4 or 16) and/or a `storageclass` block; omitted fields inherit the top-level values. Additional pools render as `-pool` StatefulSets; all pools share the headless service, the main service, the configuration, and the credentials. + +:::warning[Pools are append-only] + +The list index determines the StatefulSet name — never remove or reorder entries. Retire a pool with `rc admin decommission` before removing it from the list. + +::: + +What to expect during the rollout, per the chart's documentation: + +- **Crash/restart cycles are normal.** Pods restart until every pod of every pool is resolvable — the server refuses to start with unresolvable peers, so expect a few crash loops before the cluster converges. This is harmless. +- **Rebalance afterwards.** After the cluster converges, run `rc admin rebalance start ` to spread existing objects across the new pool. +- The PodDisruptionBudget spans all pools: with the default `pdb.maxUnavailable: 1`, at most one pod of the whole cluster may be evicted at a time. + +:::note + +`rc` is the RustFS command-line client. Use `rc admin pool list`, `expand`, `rebalance`, and `decommission` for the server-pool workflows described by the chart. + +::: + +## 8. Uninstall + +```bash +helm uninstall rustfs -n rustfs +``` + +:::note + +Helm does not delete PVCs created by StatefulSet volume claim templates. If you intend to discard the data, remove the PVCs explicitly (`kubectl -n rustfs delete pvc -l app.kubernetes.io/name=rustfs`) — otherwise a later reinstall with the same release name reattaches them. + +::: + +## Next steps + +- [Pool Expansion](../../operations/scaling/storage-pool-expansion.md) — how Server Pool expansion works at the cluster level +- [Kubernetes Upgrade](../../operations/upgrade/kubernetes/index.md) — upgrade Helm- and Operator-managed deployments +- [TLS configuration](../../integration/tls-configured.md) — end-to-end TLS options diff --git a/content/de/installation/cloud-native/meta.json b/content/de/installation/cloud-native/meta.json new file mode 100644 index 00000000..0cfcaba9 --- /dev/null +++ b/content/de/installation/cloud-native/meta.json @@ -0,0 +1,7 @@ +{ + "title": "Cloud-native", + "pages": [ + "helm-chart", + "operator" + ] +} diff --git a/content/de/installation/cloud-native/operator/index.md b/content/de/installation/cloud-native/operator/index.md new file mode 100644 index 00000000..68777f69 --- /dev/null +++ b/content/de/installation/cloud-native/operator/index.md @@ -0,0 +1,28 @@ +--- +title: Overview +description: Understand how RustFS Operator manages RustFS clusters on Kubernetes. +--- + +**RustFS Operator** applies the Kubernetes Operator pattern to RustFS clusters. Instead of creating StatefulSets, Services, PersistentVolumeClaims (PVCs), and configuration by hand, you declare the required storage cluster as a Kubernetes custom resource. The controller watches that resource and continuously reconciles the running cluster with the declared state. + +The Operator installs two Custom Resource Definitions (CRDs): + +- `Tenant` (`rustfs.com/v1alpha1`) represents one RustFS cluster. It defines storage pools, credentials, scheduling, Transport Layer Security (TLS), and Key Management Service (KMS) settings. +- `PolicyBinding` (`sts.rustfs.com/v1alpha1`) maps a Kubernetes ServiceAccount to RustFS policies when workloads request temporary credentials from the Operator Security Token Service (STS). + +One Operator can manage multiple Tenants across namespaces. Each Tenant has independent storage, credentials, S3 and Console services, and lifecycle. The Operator creates one StatefulSet for each pool, so you can add capacity by appending a pool without rebuilding the cluster. It also reports `Ready`, `Progressing`, or `Degraded` conditions and Kubernetes Events, and exposes health and metrics endpoints for cluster monitoring. + +The same API covers small test clusters and distributed deployments. Sensitive credentials and KMS material stay in Kubernetes Secrets, while version-controlled Tenant manifests hold only Secret references. This makes deployments repeatable, supports GitOps workflows, and keeps routine operations such as multi-tenant management, pool expansion, TLS, and encryption within Kubernetes-native tools. + +## Operator workflows + +- [Install](./installation.md) covers requirements, Helm installation, Console access, and TLS configuration. +- [Multi-Tenant](./tenant.md) creates isolated RustFS clusters for different teams or workloads. +- [Pool Expansion](./pool-expansion.md) adds storage capacity by appending a pool to an existing Tenant. +- [KMS Integration](./kms.md) configures local or HashiCorp Vault key management for encrypted data. + +:::warning[Pre-release software] + +RustFS Operator is currently `v0.1.0` pre-release software under active development. Validate upgrades and Tenant changes in a non-production cluster first. + +::: \ No newline at end of file diff --git a/content/de/installation/cloud-native/operator/installation.md b/content/de/installation/cloud-native/operator/installation.md new file mode 100644 index 00000000..778625e7 --- /dev/null +++ b/content/de/installation/cloud-native/operator/installation.md @@ -0,0 +1,119 @@ +--- +title: Install +description: Prepare Kubernetes, install RustFS Operator, and access its Console securely. +--- + +This guide installs the Operator with Helm, verifies the deployment, and exposes the Operator Console locally or through HTTPS. + +## Requirements + +| Component | Requirement | +|-----------|-------------| +| Kubernetes | `v1.30` or later | +| Helm | `v3.0` or later | +| kubectl | Compatible with the Kubernetes cluster | +| StorageClass | Dynamic PVC provisioning for Tenant storage | + +Your account must be able to create CRDs, cluster RBAC, Deployments, and Services. Confirm the target cluster before installation: + +```bash +kubectl config current-context +kubectl get storageclass +``` + +## 1. Install the Operator + +The Helm chart is stored in the Operator repository: + +```bash +git clone https://github.com/rustfs/operator.git +cd operator + +helm upgrade --install rustfs-operator deploy/rustfs-operator/ \ + --namespace rustfs-system \ + --create-namespace +``` + +Common settings belong in a values file: + +```yaml title="values.yaml" +operator: + replicas: 1 + metrics: + enabled: true + tenantMonitor: + enabled: true + intervalSeconds: 300 +console: + enabled: true + service: + type: ClusterIP +``` + +Apply the file with `-f values.yaml`. The chart generates `OPERATOR_*` variables from these values; do not duplicate them under `operator.env`. + +## 2. Verify the installation + +```bash +kubectl -n rustfs-system get pods,services +kubectl get crd tenants.rustfs.com +kubectl -n rustfs-system rollout status deployment/rustfs-operator +kubectl -n rustfs-system rollout status deployment/rustfs-operator-console +``` + +## 3. Access the Operator Console + +The Console listens on port `9090`. Generate a short-lived login token: + +```bash +kubectl -n rustfs-system create token rustfs-operator-console --duration=24h +``` + +Forward the Console service to your workstation: + +```bash +kubectl -n rustfs-system port-forward \ + svc/rustfs-operator-console 19090:9090 +``` + +Open `http://127.0.0.1:19090` and paste the token into the login form. The Helm installation notes print the exact ServiceAccount and Service names when release names or namespaces differ. + +:::note[Local HTTP access] + +If your browser does not retain the login over HTTP, set `CONSOLE_COOKIE_SECURE=false` under `console.env` for local testing only. Keep secure cookies enabled for HTTPS. + +::: + +## 4. Configure Console TLS + +Use one HTTPS hostname for both the Console UI and `/api/v1`. Create a TLS Secret, or let cert-manager create it, then enable Ingress: + +```yaml title="values.yaml" +console: + ingress: + enabled: true + className: nginx + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + hosts: + - host: console.example.com + paths: + - path: / + pathType: Prefix + tls: + - secretName: console-tls + hosts: + - console.example.com +``` + +Upgrade the release with the values file: + +```bash +helm upgrade rustfs-operator deploy/rustfs-operator/ \ + --namespace rustfs-system \ + -f values.yaml +``` + +Replace the Ingress class, issuer, and hostname for your environment. If cert-manager is not installed, create the `console-tls` Secret with your certificate and private key before the upgrade. + +Next, [create a Tenant](./tenant.md). \ No newline at end of file diff --git a/content/de/installation/cloud-native/operator/kms.md b/content/de/installation/cloud-native/operator/kms.md new file mode 100644 index 00000000..60c89950 --- /dev/null +++ b/content/de/installation/cloud-native/operator/kms.md @@ -0,0 +1,89 @@ +--- +title: KMS Integration +description: Configure local or HashiCorp Vault key management for a RustFS Tenant through the Operator encryption API. +--- + +Configure Key Management Service (KMS) integration through `spec.encryption`. Do not add `RUSTFS_KMS_*` variables to `spec.env`; the Operator generates them from the structured Tenant configuration and Secret references. + +## Choose a backend + +Use `local` only for a single-server Tenant. Use `vault` for distributed deployments where every Tenant Pod can reach HashiCorp Vault. + +## Local KMS + +Create a master key Secret: + +```yaml title="local-kms-secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: rustfs-local-kms + namespace: storage-a +type: Opaque +stringData: + local-master-key: "replace-with-a-random-master-key" +``` + +Add the encryption block to the existing Tenant manifest: + +```yaml title="tenant.yaml" +spec: + encryption: + enabled: true + backend: local + local: + keyDirectory: /data/rustfs0/.kms-keys + masterKeySecretRef: + name: rustfs-local-kms + key: local-master-key + defaultKeyId: tenant-default +``` + +The key directory must be inside a mounted data path so it survives Pod replacement. + +## HashiCorp Vault KMS + +Create a Secret containing a Vault token: + +```yaml title="vault-kms-secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: rustfs-kms + namespace: storage-a +type: Opaque +stringData: + vault-token: "replace-with-vault-token" +``` + +Add the Vault configuration to the existing Tenant manifest: + +```yaml title="tenant.yaml" +spec: + encryption: + enabled: true + backend: vault + vault: + endpoint: https://vault.example.com:8200 + kmsSecret: + name: rustfs-kms + defaultKeyId: tenant-default +``` + +Every Tenant Pod must be able to resolve and connect to the Vault endpoint and trust its certificate. + +## Apply the configuration + +```bash +kubectl apply -f local-kms-secret.yaml +kubectl apply -f tenant.yaml +kubectl -n storage-a describe tenant tenant-a +``` + +For Vault, apply `vault-kms-secret.yaml` before `tenant.yaml`. Changing encryption settings rolls the affected StatefulSets. + +:::warning[Protect encryption keys] + +Back up key material and test recovery before storing production data. Losing the local master key or Vault keys can make encrypted objects unrecoverable. + +::: \ No newline at end of file diff --git a/content/de/installation/cloud-native/operator/meta.json b/content/de/installation/cloud-native/operator/meta.json new file mode 100644 index 00000000..843952a8 --- /dev/null +++ b/content/de/installation/cloud-native/operator/meta.json @@ -0,0 +1,9 @@ +{ + "title": "Operator", + "pages": [ + "installation", + "tenant", + "pool-expansion", + "kms" + ] +} \ No newline at end of file diff --git a/content/de/installation/cloud-native/operator/pool-expansion.md b/content/de/installation/cloud-native/operator/pool-expansion.md new file mode 100644 index 00000000..482c99b5 --- /dev/null +++ b/content/de/installation/cloud-native/operator/pool-expansion.md @@ -0,0 +1,55 @@ +--- +title: Pool Expansion +description: Add storage capacity to a RustFS Tenant by appending a storage pool. +--- + +All pools in a Tenant form one RustFS cluster. Add capacity by appending a new pool to `spec.pools`; do not change the shape of an existing pool. + +:::warning[Existing pools are immutable] + +Do not change `servers` or `persistence.volumesPerServer` on an existing pool. The Operator creates an immutable StatefulSet for each pool. + +::: + +## 1. Check the Tenant + +```bash +kubectl -n storage-a get tenant tenant-a +kubectl -n storage-a get pods,pvc -l rustfs.tenant=tenant-a +``` + +Confirm that the Tenant is `Ready` and the cluster has enough compute and storage capacity. + +## 2. Add a pool + +Append the following entry to the existing `spec.pools` list in `tenant.yaml`. Keep all existing entries unchanged. + +```yaml title="tenant.yaml" +- name: pool-1 + servers: 2 + persistence: + volumesPerServer: 2 + volumeClaimTemplate: + storageClassName: standard + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 100Gi +``` + +This pool creates four PVCs: two servers multiplied by two volumes per server. Apply the complete Tenant manifest: + +```bash +kubectl apply -f tenant.yaml +``` + +## 3. Watch the expansion + +```bash +kubectl -n storage-a get tenant tenant-a -w +kubectl -n storage-a get pods,pvc \ + -l rustfs.pool=pool-1 +``` + +Wait for the Tenant to return to `Ready` before making another topology change. Increasing an existing PVC size is a separate Kubernetes storage operation and depends on the StorageClass. \ No newline at end of file diff --git a/content/de/installation/cloud-native/operator/tenant.md b/content/de/installation/cloud-native/operator/tenant.md new file mode 100644 index 00000000..08bec042 --- /dev/null +++ b/content/de/installation/cloud-native/operator/tenant.md @@ -0,0 +1,74 @@ +--- +title: Multi-Tenant +description: Create isolated RustFS Tenants and access their S3 API and Console services. +--- + +A `Tenant` represents one independent RustFS cluster. Use a separate namespace, credentials Secret, and Tenant resource for each team or workload. + +## 1. Create a namespace and credentials + +Create the Secret directly so credentials are not stored in a manifest: + +```bash +kubectl create namespace storage-a +kubectl -n storage-a create secret generic rustfs-tenant-creds \ + --from-literal=accesskey='' \ + --from-literal=secretkey='' +``` + +## 2. Define the Tenant + +This development example creates one RustFS Pod and one `10Gi` PVC. Replace `standard` with a StorageClass in your cluster. + +```yaml title="tenant.yaml" +apiVersion: rustfs.com/v1alpha1 +kind: Tenant +metadata: + name: tenant-a + namespace: storage-a +spec: + image: rustfs/rustfs:1.0.0-beta.10 + credsSecret: + name: rustfs-tenant-creds + pools: + - name: pool-0 + servers: 1 + persistence: + volumesPerServer: 1 + volumeClaimTemplate: + storageClassName: standard + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi +``` + +## 3. Apply and verify + +```bash +kubectl apply -f tenant.yaml +kubectl -n storage-a get tenant,pods,pvc,svc +kubectl -n storage-a describe tenant tenant-a +``` + +## 4. Access RustFS + +```bash +kubectl -n storage-a port-forward svc/tenant-a-io 9000:9000 +kubectl -n storage-a port-forward svc/tenant-a-console 9001:9001 +``` + +Run the commands in separate terminals. Use `http://localhost:9000` as the S3 endpoint and open `http://localhost:9001` for the Tenant Console. + +To add another Tenant, repeat the process with a different namespace, Secret, and Tenant name. List all managed Tenants with: + +```bash +kubectl get tenants --all-namespaces +``` + +:::warning[Production topology] + +The one-server example is for evaluation. Production Tenants need a distributed pool layout, resource requests, scheduling constraints, and an immutable image reference. + +::: \ No newline at end of file diff --git a/content/de/installation/container/docker.md b/content/de/installation/container/docker.md new file mode 100644 index 00000000..26124785 --- /dev/null +++ b/content/de/installation/container/docker.md @@ -0,0 +1,61 @@ +--- +title: "Docker" +description: "Run a single-node RustFS instance with Docker and persistent container storage." +--- + +Use the official RustFS image to start a single-node instance with persistent storage. You need a working Docker Engine and permission to run containers. + +## 1. Pull the image + +```bash +docker pull rustfs/rustfs:latest +``` + +## 2. Create persistent storage + +Create a named volume so object data remains available when you replace the container: + +```bash +docker volume create rustfs-data +``` + +## 3. Start RustFS + +Replace the credential placeholders before running the container: + +```bash +docker run -d \ + --name rustfs \ + --restart unless-stopped \ + -p 9000:9000 \ + -p 9001:9001 \ + -v rustfs-data:/data \ + -e RUSTFS_ACCESS_KEY="" \ + -e RUSTFS_SECRET_KEY="" \ + -e RUSTFS_CONSOLE_ENABLE=true \ + rustfs/rustfs:latest \ + /data +``` + +:::warning[Set credentials when creating the container] + +Set unique `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` environment variables before exposing RustFS to a network. Do not use the well-known `rustfsadmin` value for either credential. If the container was started without custom credentials, stop and recreate it with both `-e` options shown above; the `rustfs-data` volume remains intact. + +::: + +## 4. Verify the deployment + +Check the container and the S3 API health endpoint: + +```bash +docker ps --filter name=rustfs +curl --fail http://localhost:9000/health +``` + +The S3 API is available at `http://localhost:9000`, and the Console is available at `http://localhost:9001`. + +## Next steps + +- [RustFS Console](/administration/console) +- [Configure an S3 client](../../developer/examples/aws-cli.md) +- [TLS configuration](../../integration/tls-configured.md) \ No newline at end of file diff --git a/content/de/installation/container/index.mdx b/content/de/installation/container/index.mdx new file mode 100644 index 00000000..b7528a27 --- /dev/null +++ b/content/de/installation/container/index.mdx @@ -0,0 +1,302 @@ +--- +title: "Installing RustFS with Docker" +description: "RustFS Docker deployment." +--- + +RustFS is a high-performance, S3-compatible open-source distributed object storage system. In single-node single-disk (SNSD) deployment mode, the backend uses zero erasure coding without additional data redundancy, suitable for local testing and small-scale scenarios. +This article is based on RustFS official Linux binary packages, packaging RustFS and its runtime environment into containers through custom Dockerfile, and configuring data volumes and environment variables for one-click service startup. + +--- + +## 1. Prerequisites + +1. **Host Requirements** + + * Docker installed (≥ 20.10) and able to pull images and run containers normally + * Local path `/mnt/rustfs/data` (or custom path) for mounting object data + * [`rc`](/operations/rc) installed on the administration host for the verification workflow +2. **Network and Firewall** + + * Ensure host ports 9000 (S3 API) and 9001 (Console) are open to external access (or consistent with custom ports) + +3. **Directory Permissions** + + * The RustFS container runs as non-root user `rustfs` with id `10001`. If you run docker with `-v` to mount a host directory into the container, make sure the owner of the host directory is `10001`, otherwise you will encounter permission denied errors. Run `chown -R 10001:10001 /path/to/host_directory` to grant the necessary permissions. + +--- + +## 2. Pull the RustFS Official Image + +Pull the official image (Alpine-based) from Docker Hub: + +```bash +docker pull rustfs/rustfs +``` + +--- + + + +## 3. Run RustFS Container + +RustFS SNSD Docker running method, combining the above image and configuration, execute: + +```bash + docker run -d \ + --name rustfs_local \ + -p 9000:9000 \ + -p 9001:9001 \ + -v /mnt/rustfs/data:/data \ + rustfs/rustfs:latest \ + /data +``` + +Parameter descriptions: + +* `-p 9000:9000`: Map host port 9000 to container +* `-v /mnt/rustfs/data:/data`: Mount data volume +* `--name rustfs_local`: Custom container name +* `-d`: Run in background + +--- + +### Complete Configuration Example + +Configuration can be passed as environment variables (recommended) or as command-line flags — pick one style; when both are present, command-line flags win. The volume path (`/data`) always comes last. + + + + +```bash {7,8} +# Use a unique access key and a strong, random secret (e.g. openssl rand -base64 24) +docker run -d \ + --name rustfs \ + -p 9000:9000 \ + -p 9001:9001 \ + -v /mnt/rustfs/data:/data \ + -e RUSTFS_ACCESS_KEY="" \ + -e RUSTFS_SECRET_KEY="" \ + -e RUSTFS_ADDRESS=:9000 \ + -e RUSTFS_CONSOLE_ENABLE=true \ + rustfs/rustfs:latest \ + /data +``` + + + + +```bash {8,9} +# Use a unique access key and a strong, random secret (e.g. openssl rand -base64 24) +docker run -d \ + --name rustfs \ + -p 9000:9000 \ + -p 9001:9001 \ + -v /mnt/rustfs/data:/data \ + rustfs/rustfs:latest \ + --access-key "" \ + --secret-key "" \ + --address :9000 \ + --console-enable \ + /data +``` + + + + +### Common Configuration Combinations + +1. **Basic Configuration**: + ```bash + docker run -d \ + -p 9000:9000 \ + -p 9001:9001 \ + -v /mnt/data:/data \ + rustfs/rustfs:latest \ + /data + ``` + +2. **Enable Console**: + ```bash + docker run -d \ + -p 9000:9000 \ + -p 9001:9001 \ + -v /mnt/data:/data \ + -e RUSTFS_CONSOLE_ENABLE=true \ + rustfs/rustfs:latest \ + --console-enable \ + /data + ``` + +3. **Custom Authentication Keys**: + ```bash + # Use a unique access key and a strong, random secret (e.g. openssl rand -base64 24) + docker run -d \ + -p 9000:9000 \ + -p 9001:9001 \ + -v /mnt/data:/data \ + -e RUSTFS_ACCESS_KEY="" \ + -e RUSTFS_SECRET_KEY="" \ + rustfs/rustfs:latest \ + --access-key "" \ + --secret-key "" \ + /data + ``` + +### Important Notes + +1. Port mapping must correspond: + - Service port defaults to 9000 (`-p 9000:9000`) + +2. Data volumes must be persistent: + - `-v /host/path:/container/path` + +3. Environment variables and command line parameters can be mixed, but command line parameters have higher priority + +4. If [using TLS](../../integration/tls-configured.md), additional certificate path mounting is needed: + + ```bash + -v /path/to/certs:/certs \ + -e RUSTFS_TLS_PATH=/certs \ + ``` + +### Docker Compose Installation + +RustFS officially provides a Docker Compose installation method. The [`docker-compose.yml`](https://github.com/rustfs/rustfs/blob/main/docker-compose.yml) file includes multiple services, such as `grafana`, `prometheus`, `otel-collector`, and `jaeger`, mainly for observability. If you want to deploy these services together, clone the [RustFS code repository](https://github.com/rustfs/rustfs) locally, + +```bash +git clone https://github.com/rustfs/rustfs.git +``` + +Running the command under root directory, + +```bash +docker compose --profile observability up -d +``` + +Providing the necessary permissions. An initialization container is necessary to grant the correct access rights to rustfs using the `depends_on` keyword. In the example below the `rustfs_perms` service is added to the `docker-compose.yml` to handle this. To ensure logs are persisted and accessible, we map the host log directory to the container's `/logs` path + +```yaml title="docker-compose.yml" + services: + # grant the necessary permissions to RUSTFS volumes path + rustfs_perms: + image: alpine + user: root + volumes: + - /path/to/host_directory/volumes:/fix_path + command: chown -R 10001:10001 /fix_path + + rustfs: + image: rustfs/rustfs:latest + depends_on: + rustfs_perms: + condition: service_completed_successfully + volumes: + - /path/to_host_directory/volumes/data:/data + - /path/to_host_directory/volumes/logs:/logs + environment: + - RUSTFS_OBS_LOG_DIRECTORY=/logs + + # ... other configurations +``` + +Started containers is as below, + +```text +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +c13c23fe3d9d rustfs/rustfs:latest "/entrypoint.sh rust…" 6 seconds ago Up 5 seconds (health: starting) 0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp rustfs-server +e3f4fc4a83a2 grafana/grafana:latest "/run.sh" 7 seconds ago Up 5 seconds 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafana +71ef1b8212cf prom/prometheus:latest "/bin/prometheus --c…" 7 seconds ago Up 5 seconds 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus +e7db806b2d6f jaegertracing/all-in-one:latest "/go/bin/all-in-one-…" 7 seconds ago Up 5 seconds 4317-4318/tcp, 9411/tcp, 0.0.0.0:14250->14250/tcp, :::14250->14250/tcp, 14268/tcp, 0.0.0.0:16686->16686/tcp, :::16686->16686/tcp jaeger +1897830a2f1e otel/opentelemetry-collector-contrib:latest "/otelcol-contrib --…" 7 seconds ago Up 5 seconds 0.0.0.0:4317-4318->4317-4318/tcp, :::4317-4318->4317-4318/tcp, 0.0.0.0:8888-8889->8888-8889/tcp, :::8888-8889->8888-8889/tcp, 55679/tcp otel-collector +``` + +If you only want to install RustFS without Grafana, Prometheus, and the other observability services, start just the `rustfs` service (the compose file marks the collector dependency as optional): + +```bash +docker compose -f docker-compose.yml up -d rustfs +``` + +This way will only install and start `rustfs-server` service, namely rustfs container, + +```text +docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +e07121ecdd39 rustfs/rustfs:latest "/entrypoint.sh rust…" 2 seconds ago Up 1 second (health: starting) 0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp rustfs-server +``` + +Whether you start only the `rustfs-server` or together with observability services, the S3 API is served at `http://localhost:9000`, and the RustFS Console is at `http://localhost:9001` — open it in a browser and log in with the access key and secret key you configured above (the `` / `` placeholders). Generate a strong secret with, for example, `openssl rand -base64 24`, and never ship the placeholder values to production. + +:::warning[Set container credentials immediately] + +Set unique `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` environment variables before exposing RustFS to a network. Do not use the well-known `rustfsadmin` value for either credential. For `docker run`, pass both values with `-e`. For Docker Compose, define both variables in the `rustfs` service environment or in the environment file used for variable substitution, then recreate the service with `docker compose up -d rustfs`. + +::: + +## 4. Verification and Access + +1. **View Container Status and Logs:** + + ```bash + docker logs rustfs_local + ``` + + Logs should show successful service startup and listening on port 9000. + +2. **Test S3 API:** + + Use `rc` to verify the S3 API: + + ```bash + # Use a unique access key and a strong, random secret (e.g. openssl rand -base64 24) + rc alias set rustfs http://localhost:9000 + rc bucket create rustfs/my-bucket + rc bucket list rustfs/ + ``` + + If buckets can be successfully created and listed, deployment is effective. + +## 5. Multiple Nodes + +Dockers default bridge networking does not support multi-node deployments. Use `--network host` so each container can communicate directly with other nodes. + +Run the following on **each node** + +```bash +# Use a unique access key and a strong, random secret (e.g. openssl rand -base64 24) +docker run -d \ + --name rustfs \ + --network host \ + -v /mnt/rustfs/data:/data \ + -e RUSTFS_ACCESS_KEY="" \ + -e RUSTFS_SECRET_KEY="" \ + -e RUSTFS_CONSOLE_ENABLE=true \ + -e RUSTFS_VOLUMES="http://node{1...4}:9000/data/rustfs{0...3}" \ + rustfs/rustfs:latest +``` + +Add the entries to `/etc/hosts` on **every** node: +```ini title="/etc/hosts" +192.168.1.1 node1 +192.168.1.2 node2 +192.168.1.3 node3 +192.168.1.4 node4 +``` + +## 6. Other Recommendations + +1. Production Environment Recommendations: +- Use multi-node deployment architecture +- [Enable TLS encrypted communication](../../integration/tls-configured.md) +- Configure log rotation strategy +- Set up regular backup strategy + +2. Storage Recommendations: +- Use local SSD/NVMe storage +- Avoid using network file systems (NFS) +- Ensure storage directory exclusive access + +--- + +## Summary + +This article explains how to deploy RustFS using Docker with best practices, starting with a single-node single-disk (SNSD) setup and then extending to a multi-node deployment option. diff --git a/content/de/installation/container/meta.json b/content/de/installation/container/meta.json new file mode 100644 index 00000000..be4e8d9a --- /dev/null +++ b/content/de/installation/container/meta.json @@ -0,0 +1,7 @@ +{ + "title": "Container", + "pages": [ + "docker", + "podman" + ] +} \ No newline at end of file diff --git a/content/de/installation/container/podman.md b/content/de/installation/container/podman.md new file mode 100644 index 00000000..fdcb320a --- /dev/null +++ b/content/de/installation/container/podman.md @@ -0,0 +1,60 @@ +--- +title: "Podman" +description: "Run a single-node RustFS instance with Podman and persistent container storage." +--- + +Podman can run the official RustFS Open Container Initiative (OCI) image without a daemon. You need a working Podman installation and permission to create containers. + +## 1. Pull the image + +```bash +podman pull docker.io/rustfs/rustfs:latest +``` + +## 2. Create persistent storage + +Create a named volume so object data remains available when you replace the container: + +```bash +podman volume create rustfs-data +``` + +## 3. Start RustFS + +Replace the credential placeholders before running the container: + +```bash +podman run -d \ + --name rustfs \ + -p 9000:9000 \ + -p 9001:9001 \ + -v rustfs-data:/data \ + -e RUSTFS_ACCESS_KEY="" \ + -e RUSTFS_SECRET_KEY="" \ + -e RUSTFS_CONSOLE_ENABLE=true \ + docker.io/rustfs/rustfs:latest \ + /data +``` + +:::warning[Set credentials when creating the container] + +Set unique `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` environment variables before exposing RustFS to a network. Do not use the well-known `rustfsadmin` value for either credential. If the container was started without custom credentials, stop and recreate it with both `-e` options shown above; the `rustfs-data` volume remains intact. + +::: + +## 4. Verify the deployment + +Check the container and the S3 API health endpoint: + +```bash +podman ps --filter name=rustfs +curl --fail http://localhost:9000/health +``` + +The S3 API is available at `http://localhost:9000`, and the Console is available at `http://localhost:9001`. + +## Next steps + +- [RustFS Console](/administration/console) +- [Configure an S3 client](../../developer/examples/aws-cli.md) +- [TLS configuration](../../integration/tls-configured.md) \ No newline at end of file diff --git a/content/de/installation/index.md b/content/de/installation/index.md new file mode 100644 index 00000000..e7985e53 --- /dev/null +++ b/content/de/installation/index.md @@ -0,0 +1,37 @@ +--- +title: "Installation" +description: "Choose the right RustFS deployment path: local trial, single-node or multi-node production, containers, and Kubernetes." +--- + +RustFS is a distributed object storage system written in Rust and released under the Apache 2.0 license. It runs on Linux, Windows, macOS, FreeBSD, and containers, across x86, ARM, RISC-V, and other CPU architectures. + +:::warning[Change the default credentials immediately] + +After installation, set unique values for `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` before exposing RustFS to a network. Do not use the well-known `rustfsadmin` value for either credential. + +- For a Linux binary or systemd installation, set both values in `/etc/default/rustfs`, then restart the `rustfs` service. +- For Docker, Podman, or Docker Compose, pass both values as container environment variables when creating the container. Recreate an existing container if it was started without them. + +::: + +## Choose Your Path + +| Your goal | Recommended path | Guide | +| - | - | - | +| Try RustFS on a local machine | One-command install script, or a container | [Linux Quick Start](./linux/quick-start.md) · [Container](./container/index.mdx) | +| Single-server production | SNSD (one disk) or SNMD (multiple disks) | [Installing RustFS on Linux](./linux/index.md) | +| Multi-server production cluster | MNMD, after completing the production checklists | [Multiple Node Multiple Disk](./linux/multiple-node-multiple-disk.md) · [Checklists](./requirement/checklists/index.md) | +| Kubernetes / cloud-native | Container orchestration deployment | [Cloud Native](./cloud-native/index.md) | +| Windows or macOS host | Native installation | [Windows](./windows/index.md) · [macOS](./macos/index.md) | + +## Deployment Mode Comparison + +| Mode | Nodes | Disks | Fault Tolerance | Typical Use | +| - | - | - | - | - | +| [SNSD](./linux/single-node-single-disk.md) | 1 | 1 | None — rely on backups | Development, testing, low-density non-critical business | +| [SNMD](./linux/single-node-multiple-disk.md) | 1 | Multiple | Up to M parity disks within the node | Medium, non-critical business on a single server | +| [MNMD](./linux/multiple-node-multiple-disk.md) | 4+ | Multiple per node | Disk- and node-level via erasure coding across servers | Production workloads | + +## Checklist + +Before any production deployment, work through the [Pre-Installation Checklists](./requirement/checklists/index.md) — hardware, network, software, and security — to make sure your environment meets production guidance. diff --git a/content/de/installation/linux/images/console.jpg b/content/de/installation/linux/images/console.jpg new file mode 100644 index 00000000..2a6224d5 Binary files /dev/null and b/content/de/installation/linux/images/console.jpg differ diff --git a/content/de/installation/linux/index.md b/content/de/installation/linux/index.md new file mode 100644 index 00000000..8f9f4f1b --- /dev/null +++ b/content/de/installation/linux/index.md @@ -0,0 +1,34 @@ +--- +title: "Installing RustFS on Linux" +description: "Choose a Linux deployment mode — SNSD, SNMD, or MNMD — and install RustFS on one or more servers." +--- + +This section covers installing RustFS on Linux servers. For a one-command trial installation, use the [Quick Start](./quick-start.md). For a manual installation, pick one of the three deployment modes below — all three share the same [prerequisites and service setup](./prerequisites-and-service.md), and differ only in topology and volume configuration. + +:::warning[Change the default credentials immediately] + +After installation, set unique `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` values in `/etc/default/rustfs`. Do not use the well-known `rustfsadmin` value for either credential. Restart the service with `sudo systemctl restart rustfs` after changing the file. + +::: + +## Single Node Single Disk (SNSD) + +One server, one data disk. The simplest mode, with no redundancy — a disk failure means data loss, so rely on backups. Suitable for development, testing, and low-density non-critical business. + +→ [Single Node Single Disk installation](./single-node-single-disk.md) + +## Single Node Multiple Disk (SNMD) + +One server, multiple data disks. Erasure coding shards data across the disks, so the node tolerates a limited number of disk failures, but a whole-server failure still means data loss. Suitable for medium, non-critical business on a single machine. + +→ [Single Node Multiple Disk installation](./single-node-multiple-disk.md) + +## Multiple Node Multiple Disk (MNMD) + +Four or more servers, each with one or more disks. Erasure coding spans servers, providing disk- and node-level fault tolerance plus horizontal scalability. This is the mode for production workloads. + +→ [Multiple Node Multiple Disk installation](./multiple-node-multiple-disk.md) + +## Before Production + +Work through the [Pre-Installation Checklists](../requirement/checklists/index.md) — hardware, network, software, and security — before deploying to production. If you don't need production standards, you can skip them. diff --git a/content/de/installation/linux/meta.json b/content/de/installation/linux/meta.json new file mode 100644 index 00000000..b2cf42a9 --- /dev/null +++ b/content/de/installation/linux/meta.json @@ -0,0 +1,10 @@ +{ + "title": "Linux", + "pages": [ + "[Schnellstart](/de/installation/linux/quick-start)", + "[Voraussetzungen und Diensteinrichtung](/de/installation/linux/prerequisites-and-service)", + "[Einzelner Knoten mit einem Datenträger (SNSD)](/de/installation/linux/single-node-single-disk)", + "[Einzelner Knoten mit mehreren Datenträgern (SNMD)](/de/installation/linux/single-node-multiple-disk)", + "[Mehrere Knoten mit mehreren Datenträgern (MNMD)](/de/installation/linux/multiple-node-multiple-disk)" + ] +} diff --git a/content/de/installation/linux/multiple-node-multiple-disk.md b/content/de/installation/linux/multiple-node-multiple-disk.md new file mode 100644 index 00000000..eaf7bfb3 --- /dev/null +++ b/content/de/installation/linux/multiple-node-multiple-disk.md @@ -0,0 +1,122 @@ +--- +title: "RustFS Multiple Node Multiple Disk Installation" +description: "Install RustFS on multiple servers with multiple disks, data will be stored on multiple disks across multiple servers." +--- + +Multiple Node Multiple Disk (MNMD) mode is the deployment mode for production workloads, providing enterprise-grade performance, security, and scalability. A minimum of **4 servers** is required, each with at least 1 disk, to safely start a distributed object storage cluster. + +## Topology and Planning + +In the following architecture, requests are distributed to the servers through load balancing. With the default 12 + 4 erasure coding layout, each object is split into 12 data shards and 4 parity shards stored on different disks across different servers: + +- Any single server failure or maintenance does not affect data security. +- Corruption of up to 4 disks does not affect data security. + +```mermaid +flowchart TD + LB([Load Balancing]) --> S1[Server Node1] & S2[Server Node2] & S3[Server Node3] & S4[Server Node4] + + S1 --> N1D1[(Disk1)] & N1D2[(Disk2)] & N1D3[(Disk3)] & N1D4[(Disk4)] + S2 --> N2D1[(Disk1)] & N2D2[(Disk2)] & N2D3[(Disk3)] & N2D4[(Disk4)] + S3 --> N3D1[(Disk1)] & N3D2[(Disk2)] & N3D3[(Disk3)] & N3D4[(Disk4)] + S4 --> N4D1[(Disk1)] & N4D2[(Disk2)] & N4D3[(Disk3)] & N4D4[(Disk4)] + + classDef lb fill:#f3f4f6,stroke:#9ca3af,stroke-width:2px,color:#1e293b; + classDef server fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#1e293b; + classDef disk fill:#dcfce7,stroke:#22c55e,stroke-width:2px,color:#1e293b; + class LB lb + class S1,S2,S3,S4 server + class N1D1,N1D2,N1D3,N1D4,N2D1,N2D2,N2D3,N2D4,N3D1,N3D2,N3D3,N3D4,N4D1,N4D2,N4D3,N4D4 disk +``` + +Before installation, review the [Pre-Installation Checklists](../requirement/checklists/index.md) and ensure all items meet production guidance. + +## Hostnames + +Creating a RustFS cluster requires **identical, sequential** hostnames. There are two ways to achieve sequential hostnames: + +**1. DNS Configuration:** + +Configure your DNS resolution server to ensure name continuity. + +**2. HOSTS Configuration:** + +Modify the local alias settings in `/etc/hosts` as follows: + +```bash title="/etc/hosts" +vim /etc/hosts +127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 +::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 +192.168.1.1 node1 +192.168.1.2 node2 +192.168.1.3 node3 +192.168.1.4 node4 +``` + +## Prerequisites and Service Setup + +On **every node**, complete the [common prerequisites and service setup](./prerequisites-and-service.md) — operating system, firewall, time synchronization, disk formatting, service user, binary download, and systemd unit — then continue below. Remember that all nodes must use the same listening port and must have synchronized clocks. + +## Configure Environment Variables + +1. Create the same configuration file on every node. `RUSTFS_VOLUMES` uses brace expansion to enumerate all nodes and all disk mount points (this example: 4 nodes × 4 disks): + +```ini title="/etc/default/rustfs" +# Use a unique access key and a strong, random secret (e.g. openssl rand -base64 24) +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +RUSTFS_VOLUMES="http://node{1...4}:9000/data/rustfs{0...3}" +RUSTFS_ADDRESS=":9000" +RUSTFS_CONSOLE_ENABLE=true +RUST_LOG=error +RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/" +``` + +:::note + +The access key, secret key, and `RUSTFS_VOLUMES` value must be identical on all nodes. The hostnames (`node1` – `node4`) must match the DNS or `/etc/hosts` configuration above. + +::: + +2. Create the storage and log directories on every node: + +```bash +sudo mkdir -p /data/rustfs{0..3} /var/logs/rustfs /opt/tls +sudo chmod -R 750 /data/rustfs* /var/logs/rustfs +``` + +## Start Service and Verification + +1. Start the service on every node and enable auto-start on boot: + +```bash +sudo systemctl enable --now rustfs +``` + +2. Verify the service status: + +```bash +systemctl status rustfs +``` + +3. Check the service port: + +```bash +netstat -ntpl +``` + +4. View log files: + +```bash +tail -f /var/logs/rustfs/rustfs*.log +``` + +5. Access the console: enter any node's IP address (or the load balancer address) and the console port (default 9001) in a browser. You should see: + +![Console](./images/console.jpg) + +## Next Steps + +- Put a load balancer in front of the cluster — see the [Nginx integration guide](/developer/integration/reverse-proxy/nginx). +- Enable TLS for production traffic — see [TLS configuration](../../integration/tls-configured.md). +- Review [Pool Expansion](../../operations/scaling/storage-pool-expansion.md) before scaling. diff --git a/content/de/installation/linux/prerequisites-and-service.md b/content/de/installation/linux/prerequisites-and-service.md new file mode 100644 index 00000000..0714609f --- /dev/null +++ b/content/de/installation/linux/prerequisites-and-service.md @@ -0,0 +1,231 @@ +--- +title: "Linux Prerequisites and Service Setup" +description: "Common prerequisites, disk preparation, binary installation, and systemd service setup shared by all RustFS Linux deployment modes." +--- + +This page contains the prerequisites and service setup steps shared by all three Linux deployment modes — [SNSD](./single-node-single-disk.md), [SNMD](./single-node-multiple-disk.md), and [MNMD](./multiple-node-multiple-disk.md). Complete these steps first, then return to your mode page to configure the environment file and start the service. + +## Operating System Version + +We recommend Linux kernel version 4.x or later; versions 5.x/6.x achieve better I/O throughput and network performance. Ubuntu 22.04 and RHEL 8.x are both suitable for installing RustFS. + +## Firewall + +Linux systems have firewalls enabled by default. Check the firewall status with: + +```bash +systemctl status firewalld +``` + +If your firewall status is "active", you can disable the firewall: + +```bash +systemctl stop firewalld +systemctl disable firewalld +``` + +Or allow the RustFS S3 port (9000) and console port (9001): + +```bash +firewall-cmd --zone=public --add-port=9000/tcp --permanent +firewall-cmd --zone=public --add-port=9001/tcp --permanent +firewall-cmd --reload +``` + +All RustFS servers in a deployment **must** use the same listening port. If you use port 9000, every other server must also use port 9000. + +## Memory Requirements + +RustFS requires at least 2 GB of memory for a test environment; production environments require a minimum of 128 GB of memory. + +## Time Synchronization + +Multi-node consistency requires a time server to keep clocks consistent, otherwise services may fail to start. Use tools such as `ntp`, `timedatectl`, or `timesyncd`. + +Check the synchronization status with: + +```bash +timedatectl status +``` + +If the status is "synchronized", time synchronization is working properly. + +## Capacity Planning + +When planning object storage capacity, we recommend considering: + +- Initial data volume: How much data do you plan to migrate or store at once? (e.g., 500 TB) +- Data growth volume: Daily/weekly/monthly data growth capacity +- Planning cycle: How long should this hardware planning last? (recommended: 3 years) +- Your company's hardware iteration and update cycles. + +Review [EC Configuration](../requirement/ec-configuration.md) to calculate usable capacity, understand the automatic parity defaults, and validate any explicit parity or erasure-set width before deployment. + +## Disk Planning + +Because NFS generates phantom writes and lock issues under high I/O, **NFS is prohibited** as the underlying storage medium for RustFS. We strongly recommend **JBOD (Just a Bunch of Disks)** mode: expose physical disks directly and independently to the operating system, and let the RustFS software layer handle data redundancy and protection. + +The reasons are as follows: + +- **Better Performance:** RustFS's Erasure Coding engine is highly optimized and reads/writes multiple disks concurrently, achieving higher throughput than hardware RAID controllers. Hardware RAID becomes a performance bottleneck. +- **Lower Cost:** No expensive RAID cards needed, reducing hardware procurement costs. +- **Simpler Management:** RustFS manages disks uniformly, simplifying storage layer operations and maintenance. +- **Faster Fault Recovery:** The RustFS healing process is faster than a traditional RAID rebuild and has less impact on cluster performance. + +We recommend NVMe SSDs as the storage medium for higher performance and throughput. + +## File System Selection + +RustFS strongly recommends formatting all storage disks with the XFS file system. RustFS development and testing are based on XFS, ensuring optimal performance and stability. Avoid other file systems such as ext4, BTRFS, or ZFS, as they may cause performance degradation or unpredictable issues. + +XFS suits RustFS's workload for three reasons: + +- **High-concurrency I/O:** XFS was designed for high performance and scalability. Its internal journaling and data structures (such as B+ trees) efficiently handle large numbers of parallel read/write requests, matching how RustFS shards large objects and reads/writes multiple disks in an erasure set in parallel. +- **Massive files and large file sizes:** XFS is a 64-bit file system supporting extremely large files (up to 8 EB). Its metadata management stays efficient even with millions of files in a single directory — important because RustFS stores each object (or object version) as an independent file. +- **Space reservation:** XFS provides an efficient `fallocate` API. RustFS uses it to reserve contiguous disk space before writing objects, avoiding the overhead of dynamic expansion and metadata updates during writes and minimizing file fragmentation. + +For better disk discovery, we recommend using **Label** tags when formatting XFS file systems. + +First, check the disk layout: + +```bash +sudo lsblk + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +sda 8:0 0 465.7G 0 disk +├─sda1 8:1 0 512M 0 part /boot/efi +└─sda2 8:2 0 465.2G 0 part / +nvme0n1 8:16 0 3.7T 0 disk <-- if this is our format new disk +nvme1n1 8:32 0 3.7T 0 disk <-- if this is our format new disk +nvme2n1 8:48 0 3.7T 0 disk +``` + +Format each data disk: + +```bash +sudo mkfs.xfs -i size=512 -n ftype=1 -L RUSTFS0 /dev/sdb +``` + +Formatting options: + +- `-L