WPB-11455 Allow List for 401 rate limiting#24
Conversation
echoes-hq
Bot
added
the
echoes/initiative: federation-and-mls-on-wire-c...
|
|
||
| /* Check the mtime of the allow list, do we need to update? */ | ||
| struct stat fstat; | ||
| if (stat(allowlist, &fstat) == 0) { |
There was a problem hiding this comment.
question: are we opening, reading and closing the allowlist file for each request?
There was a problem hiding this comment.
We are doing stat() and then opening the file if and only if the mtime changes between rate-limited 401 requests
We could try setting a "timeout" for how often the allow-list is updated, but then we'd still take a small hit for comparing the times, in addition to checking the mtime and opening the file. Both way seem valid. But to me this seems like less bloat.
There was a problem hiding this comment.
Was there a requirement that the allow list be editable without restarting? If not, I suppose another option is to just put the entire allow list in the configuration file and load it one at startup.
There was a problem hiding this comment.
@sgodin so @julialongtin told me that it needs to be updated during runtime, otherwise yes we can move this to the server initialization code. I suspect we want it to update during runtime to prevent having to disruptions during adding/removing federated servers and adding larger users to the allow list.
There was a problem hiding this comment.
Yes, this is an operational requirement. A restart of this service is a Big Deal(TM).
There was a problem hiding this comment.
Thank you for elaborating, that sounds very reasonable.
There was a problem hiding this comment.
I think a stat does a system call which reads inode info, I think we should avoid doing this.
Better to update the allowlist on a signal?
There was a problem hiding this comment.
signal would safer and avoid potential blocking from filesystem IO :) There is a signal handler already that re-reads certificate files, perhaps you can attach to that.
There was a problem hiding this comment.
Let me look to see how viable this is. Mind you this wouldn't be every request, as this is only run when we do the rate-limiting - not every request
But still, I will try to figure out ASAP if its possible to attach this to a signal to reload the file
|
It would be good to add all your new command line / configuration options to the sample turnserver.conf file: |
Added a new `--401-allowlist` to allow adding a path of an allow-list of IPv4/IPv6 addresses that can bypass the 401 rate-limit settings. This can also be set in the configuration file via `401-allowlist=.` The allow-list is updated when the allowlist file is updated during runtime. The allow-list format is one (1) IP address per line.
added |
|
@e-lisa If you can please avoid squashing the commits (at least for now), it will be easier to see what has been changed from the commits that result from review comments. :) |
|
@sgodin Added a global lock for the |
|
Moved all the allowlist loading and initialization code to the mainserver startup in Assigned the real-time signal
|
…e() in the future
…e() in the future
| char line[1024]; | ||
| /* Rebuild map */ | ||
| TURN_MUTEX_LOCK(&rate_limit_allowlist_mutex); | ||
| ur_addr_map_clean(rate_limit_allowlist_map); |
There was a problem hiding this comment.
You probably want to hold the lock for the entire read process, otherwise you run the risk of rate limiting an allowed endpoint between the list being cleared and the entries being added back.
|
@z-dule can you please go ahead and merge it? |
7 participants
PR Submission Checklist for internal contributors
The PR Title
SQPIT-764The PR Description
What's new in this PR?
Allowlist for the 401 rate-limit features.
Added a new
--401-allowlistto allow adding a path of an allow-list ofIPv4/IPv6 addresses that can bypass the 401 rate-limit settings. This
can also be set in the configuration file via
401-allowlist=.The allow-list is updated when the allowlist file is updated during
runtime.
The allow-list format is one (1) IP address per line.
Issues
An allow-list to bypass the 401 rate-limit feature is required for federated services and other high volume IP addresses.
Solutions
A new option
--401-allowlist=and configuration file setting401-allowlist=have been added to allow specifying a file to contain the allow list. The allow list is updated every time the allow list file is updated based on its mtime (modification time).The allow list is formatted with one IPv4/IPv6 address per line as seen in this example allow list:
Needs releases with:
Testing
Test Coverage (Optional)
How to Test
Briefly describe how this change was tested and if applicable the exact steps taken to verify that it works as expected.
Notes (Optional)
Specify here any other facts that you think are important for this issue.
Attachments (Optional)
Attachments like images, videos, etc. (drag and drop in the text box)
PR Post Submission Checklist for internal contributors (Optional)
PR Post Merge Checklist for internal contributors
References
feat(conversation-list): Sort conversations by most emojis in the title #SQPIT-764.