Skip to content

acc: Fix mlops-stacks deploy test for upstream stacks_version 0.5 #5929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shreyas-goenka merged 1 commit into from
Jul 15, 2026
Merged

acc: Fix mlops-stacks deploy test for upstream stacks_version 0.5 #5929

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion acceptance/bundle/deploy/mlops-stacks/config.json.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
{
"input_project_name": "project_name_$UNIQUE_NAME",
"input_root_dir": "test_repo_mlops_stacks",
"input_include_models_in_unity_catalog": "no",
"input_cloud": "$CLOUD_ENV_BASE"
}
1 change: 1 addition & 0 deletions acceptance/bundle/deploy/mlops-stacks/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

62 changes: 38 additions & 24 deletions acceptance/bundle/deploy/mlops-stacks/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,14 @@
{
"input_project_name": "project_name_[UNIQUE_NAME]",
"input_root_dir": "test_repo_mlops_stacks",
"input_include_models_in_unity_catalog": "no",
"input_cloud": "[CLOUD_ENV_BASE]"
}

>>> [CLI] schemas create project_name_[UNIQUE_NAME] main
{
"full_name": "main.project_name_[UNIQUE_NAME]"
}

>>> [CLI] bundle init mlops-stacks --config-file config.json
Welcome to MLOps Stacks. For detailed information on project generation, see the README at https://github.com/databricks/mlops-stacks/blob/main/README.md.

Expand All @@ -20,14 +24,14 @@ Please refer to the README.md for further instructions on getting started.
This directory contains an ML project based on the default
[Databricks MLOps Stacks](https://github.com/databricks/mlops-stacks),

>>> [CLI] bundle summary
>>> [CLI] bundle summary --var catalog_name=main
Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:34:12
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:21:7
in resources/ml-artifacts-resource.yml:28:7

Name: project_name_[UNIQUE_NAME]
Target: dev
Expand All @@ -46,19 +50,19 @@ Resources:
model_training_job:
Name: [dev [USERNAME]] dev-project_name_[UNIQUE_NAME]-model-training-job
URL: (not deployed)
Models:
Registered Models:
model:
Name: [dev [USERNAME]] dev-project_name_[UNIQUE_NAME]-model
Name: dev_[USERNAME]_project_name_[UNIQUE_NAME]-model
URL: (not deployed)

>>> [CLI] bundle validate
>>> [CLI] bundle validate --var catalog_name=main
Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:34:12
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:21:7
in resources/ml-artifacts-resource.yml:28:7

Name: project_name_[UNIQUE_NAME]
Target: dev
Expand All @@ -68,73 +72,83 @@ Workspace:

Found 2 warnings

>>> [CLI] bundle deploy
>>> [CLI] bundle deploy --var catalog_name=main
Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:34:12
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:21:7
in resources/ml-artifacts-resource.yml:28:7

Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/project_name_[UNIQUE_NAME]/dev/files...
Deploying resources...
Updating deployment state...
Deployment complete!

>>> [CLI] bundle summary -o json
>>> [CLI] bundle summary --var catalog_name=main -o json
Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:34:12
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:21:7
in resources/ml-artifacts-resource.yml:28:7

{
"experiment_id": "[NUMID]",
"model_id": "[dev [USERNAME]] dev-project_name_[UNIQUE_NAME]-model",
"model_id": "main.project_name_[UNIQUE_NAME].dev_[USERNAME]_project_name_[UNIQUE_NAME]-model",
"inference_job_id": "[NUMID]",
"training_job_id": "[NUMID]"
}

=== Assert the batch inference job actually exists
Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:34:12
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:21:7
in resources/ml-artifacts-resource.yml:28:7

Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:34:12
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:21:7
in resources/ml-artifacts-resource.yml:28:7

{
"name": "[dev [USERNAME]] dev-project_name_[UNIQUE_NAME]-batch-inference-job"
}

>>> [CLI] bundle destroy --auto-approve
>>> [CLI] bundle destroy --var catalog_name=main --auto-approve
Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:34:12
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:21:7
in resources/ml-artifacts-resource.yml:28:7

The following resources will be deleted:
delete resources.experiments.experiment
delete resources.jobs.batch_inference_job
delete resources.jobs.model_training_job
delete resources.models.model
delete resources.registered_models.model

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/project_name_[UNIQUE_NAME]/dev

Deleting files...
Destroy complete!

>>> [CLI] schemas delete main.project_name_[UNIQUE_NAME]
Warning: expected a string value, found null
at targets.dev.workspace.host
in databricks.yml:37:12

Warning: unknown field: description
at resources.experiments.experiment
in resources/ml-artifacts-resource.yml:28:7

21 changes: 15 additions & 6 deletions acceptance/bundle/deploy/mlops-stacks/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,24 +13,33 @@
envsubst < config.json.tmpl > config.json
trace cat config.json

# The template hard-codes the model's schema name to the project name and always
# registers the model in Unity Catalog, so the schema must already exist in the
# target catalog. Create it in the shared "main" catalog (the convention used by
# other cloud UC acceptance tests) and override catalog_name to match.
SCHEMA_NAME="project_name_${UNIQUE_NAME}"

cleanup() {
trace $CLI bundle destroy --auto-approve
trace $CLI bundle destroy --var catalog_name=main --auto-approve
trace $CLI schemas delete "main.${SCHEMA_NAME}"
cd ../..
rm -rf test_repo_mlops_stacks
}
trap cleanup EXIT

trace $CLI schemas create "${SCHEMA_NAME}" main | jq '{full_name}'

trace $CLI bundle init mlops-stacks --config-file config.json
trace cat test_repo_mlops_stacks/README.md | head -n 4

cd "test_repo_mlops_stacks/project_name_${UNIQUE_NAME}" || exit 1

trace $CLI bundle summary
trace $CLI bundle validate
trace $CLI bundle deploy
trace $CLI bundle summary --var catalog_name=main
trace $CLI bundle validate --var catalog_name=main
trace $CLI bundle deploy --var catalog_name=main
Comment on lines +37 to +39

@janniklasrose janniklasrose Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: consider test.toml with env BUNDLE_VAR_catalog_name="main" to avoid all the --var in various places in this script


trace $CLI bundle summary -o json | jq -r '{experiment_id: .resources.experiments.experiment.id, model_id: .resources.models.model.id, inference_job_id: .resources.jobs.batch_inference_job.id, training_job_id: .resources.jobs.model_training_job.id}'
trace $CLI bundle summary --var catalog_name=main -o json | jq -r '{experiment_id: .resources.experiments.experiment.id, model_id: .resources.registered_models.model.id, inference_job_id: .resources.jobs.batch_inference_job.id, training_job_id: .resources.jobs.model_training_job.id}'

title "Assert the batch inference job actually exists\n"
JOB_ID=$($CLI bundle summary -o json | jq -r '.resources.jobs.batch_inference_job.id')
JOB_ID=$($CLI bundle summary --var catalog_name=main -o json | jq -r '.resources.jobs.batch_inference_job.id')
$CLI jobs get "${JOB_ID}" | jq '{name: .settings.name}'
6 changes: 6 additions & 0 deletions acceptance/bundle/deploy/mlops-stacks/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
Cloud=true
Local=false

# The MLOps Stacks template registers the model in Unity Catalog and runs its
# jobs on serverless compute, so this test only works on a UC-enabled workspace
# (which also has serverless jobs). Gate on UC so it skips on the non-UC cloud
# envs instead of failing with METASTORE_DOES_NOT_EXIST.
RequiresUnityCatalog = true

Badness = "the newly initialized bundle from the 'mlops-stacks' template contains two validation warnings in the configuration"

Ignore = [
Expand Down
Loading