Skip to content

Commit 268b3a5

Browse files
committed
Update README to reflect new changes
1 parent a7ac70e commit 268b3a5

File tree

1 file changed

+9
-4
lines changed

1 file changed

+9
-4
lines changed

README.md

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,9 @@ vec-inf launch Meta-Llama-3.1-8B-Instruct
1515
```
1616
You should see an output like the following:
1717

18-
<img width="450" alt="launch_img" src="https://github.com/user-attachments/assets/557eb421-47db-4810-bccd-c49c526b1b43">
18+
<img width="400" alt="launch_img" src="https://github.com/user-attachments/assets/557eb421-47db-4810-bccd-c49c526b1b43">
1919

20-
The model would be launched using the [default parameters](vec-inf/models/models.csv), you can override these values by providing additional options, use `--help` to see the full list.
21-
If you'd like to see the Slurm logs, they are located in the `.vec-inf-logs` folder in your home directory. The log folder path can be modified by using the `--log-dir` option.
20+
The model would be launched using the [default parameters](vec-inf/models/models.csv), you can override these values by providing additional options, use `--help` to see the full list. You can also launch your own customized model as long as the model architecture is [supported by vLLM](https://docs.vllm.ai/en/stable/models/supported_models.html), you'll need to specify all model launching related options to run a successful run.
2221

2322
You can check the inference server status by providing the Slurm job ID to the `status` command:
2423
```bash
@@ -27,7 +26,7 @@ vec-inf status 13014393
2726

2827
You should see an output like the following:
2928

30-
<img width="450" alt="status_img" src="https://github.com/user-attachments/assets/7385b9ca-9159-4ca9-bae2-7e26d80d9747">
29+
<img width="400" alt="status_img" src="https://github.com/user-attachments/assets/7385b9ca-9159-4ca9-bae2-7e26d80d9747">
3130

3231
There are 5 possible states:
3332

@@ -52,6 +51,12 @@ vec-inf list
5251
```
5352
<img width="1200" alt="list_img" src="https://github.com/user-attachments/assets/a4f0d896-989d-43bf-82a2-6a6e5d0d288f">
5453

54+
You can also view the default setup for a specific supported model by providing the model name, for example `Meta-Llama-3.1-70B-Instruct`:
55+
```bash
56+
vec-inf list Meta-Llama-3.1-70B-Instruct
57+
```
58+
<img width="400" alt="list_model_img" src="https://github.com/user-attachments/assets/5dec7a33-ba6b-490d-af47-4cf7341d0b42">
59+
5560
`launch`, `list`, and `status` command supports `--json-mode`, where the command output would be structured as a JSON string.
5661

5762
## Send inference requests

0 commit comments

Comments
 (0)