Enable AMP (Automatic Mixed Precision ) in Tensorflow Serving.

Describe the problem the feature is intended to solve

AMP accelerates inference significantly.

Describe the solution

A flag for enabling AMP

Describe alternatives you've considered

There is no alternative with Tensorflow Serving

Additional context

N/A

This I think should be very high priority ( at the least FP16) , otherwise the case for TFS becomes weak.

The AMP is mainly target on training instead of serving.(https://www.tensorflow.org/guide/keras/mixed_precision)

Have you observed the significant performance difference for serving as well? If so, could you share the benchmark and related numbers?

How do I turn on AMP in serving ? I have observed 50% improvement in processiong time with fp16 over fp32 without any noticeable change in accuracy. Reduced precision is one of the corner stones of Nvidia TensorRT, etc. See this one also - https://medium.com/@whatdhack/neural-network-inference-optimization-8651b95e44ee .

Is there a way to do the following in TFS ?

config = tf.ConfigProto()
config.graph_options.rewrite_options.auto_mixed_precision = 1
sess = tf.Session(config=config)

I just ran some tests on a MaskRCNN Saved Model in nvcr.io/nvidia/tensorflow:20.03-tf1-py3. TF_ENABLE_AUTO_MIXED_PRECISION seems to work very well for inference - requires less memory and speeds up significantly. The following are the numbers , if you need more convincing.

TF_ENABLE_AUTO_MIXED_PRECISION =1, memory = 4.2GB , inference speed 0.25 sec .

vs

memory = 7.1 GB , inference speed 0.53 sec .

Thanks for the experiments and numbers! Based on the number, we could add the option. I will also follow up with our GPU team.

Any update here? Also, is it possible to enable JIT/XLA as well like #1515 ?

Any update here?

I'd really appreciate this feature being added too

Hi, Any updates here?

Hello @ALL Thank you so much for showing interest! We truly appreciate your enthusiasm and support.
We regret to inform you that we are not accepting new implementation requests or feature additions at this time. Please feel free to fork the repository and submit pull requests if you have any suggestions for enhancements, problem fixes, or ideas. We’d be more than happy to review and collaborate! If you would like to contribute , please follow our CONTRIBUTION GUIDELINES here and submit a pull request.
Thank you again for your interest, and we look forward to seeing your contributions!

BFloat16 Automatic Mixed Precision support has been available on CPU for some time now after #2193.
It can be enabled by adding the --mixed_precision=bfloat16 flag when starting the TensorFlow model server.

Closing this due to inactivity. Please feel free to reopen and post your comments (if you still have queries on this). Thank you!

Navigation Menu

Repository navigation

Enable AMP (Automatic Mixed Precision ) in Tensorflow Serving. #1583

Describe the problem the feature is intended to solve

Describe the solution

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Enable AMP (Automatic Mixed Precision ) in Tensorflow Serving. #1583

Description

Describe the problem the feature is intended to solve

Describe the solution

Describe alternatives you've considered

Additional context

Activity

whatdhack commented on Apr 3, 2020

shadowdragon89 commented on Apr 10, 2020

whatdhack commented on Apr 12, 2020

whatdhack commented on Apr 13, 2020

whatdhack commented on May 14, 2020

shadowdragon89 commented on May 14, 2020

jeisinge commented on Nov 2, 2020

lre commented on Feb 22, 2022

DerryFitz commented on Apr 12, 2022

junA2Z commented on May 17, 2023

BobLiu20 commented on Sep 11, 2023

bharatjetti commented on Mar 16, 2026

alex-chaiko commented on Mar 25, 2026

jeelania10 commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions