Skip to content

Converted hard-coded port to use environment variable SERVER_PORT #274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
auser wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Converted hard-coded port to use environment variable SERVER_PORT #274

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
35373bb
Converted hard-coded port to use environment variable `SERVER_PORT`
auser Mar 23, 2023
485a27e
Updated README with SERVER_PORT option
auser Mar 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
image/
98 changes: 54 additions & 44 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Visual ChatGPT
# Visual ChatGPT

**Visual ChatGPT** connects ChatGPT and a series of Visual Foundation Models to enable **sending** and **receiving** images during chatting.

Expand All @@ -13,34 +13,33 @@ See our paper: [<font size=5>Visual ChatGPT: Talking, Drawing and Editing with V
</a>

## Updates:

- We propose the **template** idea in Visual ChatGPT!
- A template is a **pre-defined execution flow** that assists ChatGPT in assembling complex tasks involving multiple foundation models.
- A template contains the **experiential solution** to complex tasks as determined by humans.
- A template can **invoke multiple foundation models** or even **establish a new ChatGPT session**
- To define a **template**, simply adding a class with attributes `template_model = True`
- A template is a **pre-defined execution flow** that assists ChatGPT in assembling complex tasks involving multiple foundation models.
- A template contains the **experiential solution** to complex tasks as determined by humans.
- A template can **invoke multiple foundation models** or even **establish a new ChatGPT session**
- To define a **template**, simply adding a class with attributes `template_model = True`
- Thanks to **@ShengmingYin** and **@thebestannie** for providing a template example in `InfinityOutPainting` class (see the following gif)
- Firstly, run `python visual_chatgpt.py --load "ImageCaptioning_cuda:0,ImageEditing_cuda:1,VisualQuestionAnswering_cuda:2"`
- Secondly, say `extend the image to 2048x1024` to Visual ChatGPT!
- By simply creating an `InfinityOutPainting` template, Visual ChatGPT can seamlessly extend images to any size through collaboration with existing `ImageCaptioning`, `ImageEditing`, and `VisualQuestionAnswering` foundation models, **without the need for additional training**.
- Firstly, run `python visual_chatgpt.py --load "ImageCaptioning_cuda:0,ImageEditing_cuda:1,VisualQuestionAnswering_cuda:2"`
- Secondly, say `extend the image to 2048x1024` to Visual ChatGPT!
- By simply creating an `InfinityOutPainting` template, Visual ChatGPT can seamlessly extend images to any size through collaboration with existing `ImageCaptioning`, `ImageEditing`, and `VisualQuestionAnswering` foundation models, **without the need for additional training**.
- **Visual ChatGPT needs the effort of the community! We crave your contribution to add new and interesting features!**
<img src="./assets/demo_inf.gif" width="750">

<img src="./assets/demo_inf.gif" width="750">

## Insight & Goal:

On the one hand, **ChatGPT (or LLMs)** serves as a **general interface** that provides a broad and diverse understanding of a
wide range of topics. On the other hand, **Foundation Models** serve as **domain experts** by providing deep knowledge in specific domains.
By leveraging **both general and deep knowledge**, we aim at building an AI that is capable of handling various tasks.

## Demo

## Demo
<img src="./assets/demo_short.gif" width="750">

## System Architecture
## System Architecture


<p align="center"><img src="./assets/figure.jpg" alt="Logo"></p>


## Quick Start

```
Expand All @@ -66,7 +65,7 @@ export OPENAI_API_KEY={Your_Private_Openai_Key}
set OPENAI_API_KEY={Your_Private_Openai_Key}

# Start Visual ChatGPT !
# You can specify the GPU/CPU assignment by "--load", the parameter indicates which
# You can specify the GPU/CPU assignment by "--load", the parameter indicates which
# Visual Foundation Model to use and where it will be loaded to
# The model and device are separated by underline '_', the different models are separated by comma ','
# The available Visual Foundation Models can be found in the following table
Expand All @@ -76,59 +75,70 @@ set OPENAI_API_KEY={Your_Private_Openai_Key}
# Advice for CPU Users
python visual_chatgpt.py --load ImageCaptioning_cpu,Text2Image_cpu

# Advice for 1 Tesla T4 15GB (Google Colab)
# Advice for 1 Tesla T4 15GB (Google Colab)
python visual_chatgpt.py --load "ImageCaptioning_cuda:0,Text2Image_cuda:0"
# Advice for 4 Tesla V100 32GB

# Advice for 4 Tesla V100 32GB
python visual_chatgpt.py --load "ImageCaptioning_cuda:0,ImageEditing_cuda:0,
Text2Image_cuda:1,Image2Canny_cpu,CannyText2Image_cuda:1,
Image2Depth_cpu,DepthText2Image_cuda:1,VisualQuestionAnswering_cuda:2,
InstructPix2Pix_cuda:2,Image2Scribble_cpu,ScribbleText2Image_cuda:2,
Image2Seg_cpu,SegText2Image_cuda:2,Image2Pose_cpu,PoseText2Image_cuda:2,
Image2Hed_cpu,HedText2Image_cuda:3,Image2Normal_cpu,
NormalText2Image_cuda:3,Image2Line_cpu,LineText2Image_cuda:3"

```

## Run options

Set the port to be running using the `SERVER_PORT` environment variable

| Option | Description | Default |
| ----------- | ------------------------- | ------- |
| SERVER_PORT | Port to start gradio demo | 1015 |

## GPU memory usage

Here we list the GPU memory usage of each visual foundation model, you can specify which one you like:

| Foundation Model | GPU Memory (MB) |
|------------------------|-----------------|
| ImageEditing | 3981 |
| InstructPix2Pix | 2827 |
| Text2Image | 3385 |
| ImageCaptioning | 1209 |
| Image2Canny | 0 |
| CannyText2Image | 3531 |
| Image2Line | 0 |
| LineText2Image | 3529 |
| Image2Hed | 0 |
| HedText2Image | 3529 |
| Image2Scribble | 0 |
| ScribbleText2Image | 3531 |
| Image2Pose | 0 |
| PoseText2Image | 3529 |
| Image2Seg | 919 |
| SegText2Image | 3529 |
| Image2Depth | 0 |
| DepthText2Image | 3531 |
| Image2Normal | 0 |
| NormalText2Image | 3529 |
| VisualQuestionAnswering| 1495 |
| ----------------------- | --------------- |
| ImageEditing | 3981 |
| InstructPix2Pix | 2827 |
| Text2Image | 3385 |
| ImageCaptioning | 1209 |
| Image2Canny | 0 |
| CannyText2Image | 3531 |
| Image2Line | 0 |
| LineText2Image | 3529 |
| Image2Hed | 0 |
| HedText2Image | 3529 |
| Image2Scribble | 0 |
| ScribbleText2Image | 3531 |
| Image2Pose | 0 |
| PoseText2Image | 3529 |
| Image2Seg | 919 |
| SegText2Image | 3529 |
| Image2Depth | 0 |
| DepthText2Image | 3531 |
| Image2Normal | 0 |
| NormalText2Image | 3529 |
| VisualQuestionAnswering | 1495 |

## Acknowledgement

We appreciate the open source of the following projects:

[Hugging Face](https://github.com/huggingface) &#8194;
[LangChain](https://github.com/hwchase17/langchain) &#8194;
[Stable Diffusion](https://github.com/CompVis/stable-diffusion) &#8194;
[ControlNet](https://github.com/lllyasviel/ControlNet) &#8194;
[InstructPix2Pix](https://github.com/timothybrooks/instruct-pix2pix) &#8194;
[Stable Diffusion](https://github.com/CompVis/stable-diffusion) &#8194;
[ControlNet](https://github.com/lllyasviel/ControlNet) &#8194;
[InstructPix2Pix](https://github.com/timothybrooks/instruct-pix2pix) &#8194;
[CLIPSeg](https://github.com/timojl/clipseg) &#8194;
[BLIP](https://github.com/salesforce/BLIP) &#8194;

## Contact Information

For help or issues using the Visual ChatGPT, please submit a GitHub issue.

For other communications, please contact Chenfei WU (chewu@microsoft.com) or Nan DUAN (nanduan@microsoft.com).
4 changes: 2 additions & 2 deletions visual_chatgpt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
Thought: Do I need to use a tool? {agent_scratchpad}"""

os.makedirs('image', exist_ok=True)

server_port = int(os.getenv('SERVER_PORT', "1015"))

def seed_everything(seed):
random.seed(seed)
Expand Down Expand Up @@ -1066,4 +1066,4 @@ def run_image(self, image, state, txt):
clear.click(bot.memory.clear)
clear.click(lambda: [], None, chatbot)
clear.click(lambda: [], None, state)
demo.launch(server_name="0.0.0.0", server_port=1015)
demo.launch(server_name="0.0.0.0", server_port=server_port)