slice: purge every block of an object, not just those before a gap

doc/admin-guide/plugins/slice.en.rst

@@ -186,7 +186,21 @@ The slice plugin supports the following options::
         that causes `cache_range_requests` to be bypassed in such requests, and
         allow ATS to handle those range requests internally.
 
-
+    --purge-probe-blocks=<int> (optional)
+        Default is 8
+        How many consecutive uncached slice blocks a PURGE walks, before any block
+        has reported the object's extent, until it concludes that nothing about the
+        object is cached.  May be overridden per request with the header named by
+        ``--purge-probe-header``.  See `Purge Requests`_.
+        -q for short
+
+    --purge-probe-header=<string> (optional)
+        Default is X-Slice-Purge-Probe
+        Name of the request header a PURGE may use to override
+        ``--purge-probe-blocks`` for that request.  A malformed value is ignored
+        in favour of the configured default.  Slice strips this header from the
+        block requests it issues.
+        -H for short
 
 Examples::
 
@@ -338,17 +352,70 @@ requests may end up being served from temporally different assets.
 Purge Requests
 --------------
 
-The slice plugin supports PURGE requests, discarding the requested object from cache.
-If a range is given in the client request, only the slice blocks from the
-requested range will be purged (if in cache). If not, all of the blocks will be discarded
-from the cache.
-
-If a block receives a 404, indicating the requested block to be purged is not in the cache,
-slice will not continue to purge the following blocks.
-
-The functionality works with `--ref-relative` both enabled and disabled. If `--ref-relative` is
-disabled (using slice 0 as the reference block), requesting to PURGE a block that does not have
-slice 0 in its range will still PURGE the slice 0 block, as the reference block is always processed.
+The slice plugin supports PURGE requests, discarding the requested object from
+cache. Without a range every block is discarded; with a range, the blocks that
+range covers are. Two cases below purge more than the range names: a suffix range,
+and block 0 when ``--ref-relative`` is disabled.
+
+Slice issues one PURGE per block and walks every block it was asked for, whether
+or not each one is currently cached. A block that is already absent answers 404
+internally; that is simply noted and the walk continues, so a gap left by
+per-block eviction cannot leave the blocks behind it in cache.
+
+Slice learns where the object ends from the blocks it removes. PURGE is a Traffic
+Server extension, so a successful block purge reports the removed object's extent
+in a ``X-Purged-Content-Range`` header, and the walk continues to the last block that
+extent implies. Blocks of one object can disagree about its length when the origin
+object has been replaced in place; slice takes the largest extent any block
+reports, so the longer generation's tail is not left behind.
+
+Until some block has reported an extent, the walk has no end but the miss bound:
+it stops after ``--purge-probe-blocks`` consecutive uncached blocks and reports
+that nothing was found. That is what bounds a PURGE for a URL which is not cached
+at all.
+
+An operator often knows more about the object than the plugin does, since the
+block count is just the object's size divided by the block size. That count can be
+supplied per request with the header named by ``--purge-probe-header``, default
+``X-Slice-Purge-Probe``::
+
+    PURGE /obj HTTP/1.1
+    X-Slice-Purge-Probe: 64
+
+This only changes how long the walk keeps going without having found anything; it
+never limits how many blocks are purged once an extent is known.
+
+The bound has to be able to span a whole object, because in the worst case only
+the object's last block is still cached, so the value an operator wants is the
+object's size divided by the block size: a 10 GB object in 1 MB blocks needs
+10240. There is no ceiling on it beyond that, since reaching the bound costs one
+internal cache lookup per block and PURGE is already restricted by
+:file:`ip_allow.yaml`. A malformed value is ignored in favour of the configured
+default, and slice strips the header from the block requests it issues.
+
+If the bound is reached, slice logs that it gave up and reports ``404`` even though
+later blocks may still be cached. Raise ``--purge-probe-blocks``, or send the
+override, for objects whose leading blocks are routinely absent.
+
+A client range that is already closed, such as ``bytes=0-6399999999``, bounds the
+walk directly, and is clamped against the object's extent as soon as some block
+reports one. An over-estimate therefore costs no extra block PURGEs beyond the end
+of the object, and if no block is cached at all the miss bound stops the walk.
+
+A suffix range, ``bytes=-<n>``, names its blocks by their distance from an end
+slice does not know yet, and purging is the only way it could find out. Rather
+than guess at the start, such a purge is widened to the whole object: a superset of
+what was asked for, so the named blocks certainly go. Note this is the one place a
+PURGE removes more than its range names; a ``GET`` with the same header is
+unaffected and still returns exactly the last *n* bytes.
+
+The response is sent once the walk is complete: ``200`` if at least one block was
+removed, ``404`` if none was found. This matches what Traffic Server reports for a
+PURGE of an object that is not sliced.
+
+The functionality works with ``--ref-relative`` both enabled and disabled. With it
+disabled, block 0 is always the first block walked, so a PURGE whose range does not
+cover block 0 still purges it.
 
 Conditional Slicing
 -------------------

doc/admin-guide/storage/index.en.rst

@@ -304,14 +304,29 @@ from any other IP, we connect to the daemon via localhost: ::
       > Host: example.com
       > Accept: */*
       >
-      < HTTP/1.1 200 Ok
+      < HTTP/1.1 200 OK
       < Date: Thu, 08 Jan 2010 20:32:07 GMT
       < Connection: keep-alive
 
 The next time Traffic Server receives a request for the removed object,
 it will contact the origin server to retrieve a new copy, which will replace
 the previously cached version in Traffic Server.
 
+If the removed object was stored as a partial response, that is if it carried a
+``Content-Range``, then the ``200 OK`` also reports that range back in a
+``X-Purged-Content-Range`` header::
+
+      < HTTP/1.1 200 OK
+      < X-Purged-Content-Range: bytes 0-1048575/9437184
+
+This lets a caller that holds one piece of a larger resource learn the whole
+resource's extent without a second lookup. It is what allows the
+:ref:`admin-plugins-slice` plugin to purge an object block by block and know which
+block is the last one. The range is reported under its own header name rather than
+as ``Content-Range``, because ``Content-Range`` on a ``200`` response has no
+meaning under :rfc:`9110` and is read by other components as a sign that a stored
+partial response is being served.
+
 This procedure only removes the index to the object from a specific Traffic Server
 cache. While the object remains on disk, Traffic Server will no longer able to find
 the object. The next request for that object will result in a fresh copy of the

plugins/slice/Config.cc

@@ -27,8 +27,9 @@
 
 namespace
 {
-constexpr std::string_view DefaultSliceSkipHeader = {"X-Slicer-Info"};
-constexpr std::string_view DefaultCrrIdentHeader  = {"X-Crr-Ident"};
+constexpr std::string_view DefaultSliceSkipHeader  = {"X-Slicer-Info"};
+constexpr std::string_view DefaultCrrIdentHeader   = {"X-Crr-Ident"};
+constexpr std::string_view DefaultPurgeProbeHeader = {"X-Slice-Purge-Probe"};
 } // namespace
 
 Config::~Config()
@@ -121,13 +122,15 @@ Config::fromArgs(int const argc, char const *const argv[])
     {const_cast<char *>("minimum-size"),         required_argument, nullptr, 'm'},
     {const_cast<char *>("metadata-cache-size"),  required_argument, nullptr, 'z'},
     {const_cast<char *>("stats-prefix"),         required_argument, nullptr, 'x'},
+    {const_cast<char *>("purge-probe-blocks"),   required_argument, nullptr, 'q'},
+    {const_cast<char *>("purge-probe-header"),   required_argument, nullptr, 'H'},
     {nullptr,                                    0,                 nullptr, 0  },
   };
 
   // getopt assumes args start at '1' so this hack is needed
   char *const *argvp = (const_cast<char *const *>(argv) - 1);
   for (;;) {
-    int const opt = getopt_long(argc + 1, argvp, "b:de:g:i:lm:p:r:s:t:x:z:", longopts, nullptr);
+    int const opt = getopt_long(argc + 1, argvp, "b:de:g:H:i:lm:p:q:r:s:t:x:z:", longopts, nullptr);
     if (-1 == opt) {
       break;
     }
@@ -248,6 +251,19 @@ Config::fromArgs(int const argc, char const *const argv[])
       stat_prefix = optarg;
       DEBUG_LOG("Stat prefix: %s", stat_prefix.c_str());
     } break;
+    case 'q': {
+      int const blocksread = atoi(optarg);
+      if (0 < blocksread) {
+        m_purge_probe_blocks = blocksread;
+        DEBUG_LOG("Using purge probe blocks %d", m_purge_probe_blocks);
+      } else {
+        ERROR_LOG("Invalid purge-probe-blocks: %s", optarg);
+      }
+    } break;
+    case 'H': {
+      m_purge_probe_header.assign(optarg);
+      DEBUG_LOG("Using purge probe header %s", optarg);
+    } break;
     default:
       break;
     }
@@ -275,6 +291,10 @@ Config::fromArgs(int const argc, char const *const argv[])
     m_skip_header = DefaultSliceSkipHeader;
     DEBUG_LOG("Using default slice skip header %s", m_skip_header.c_str());
   }
+  if (m_purge_probe_header.empty()) {
+    m_purge_probe_header = DefaultPurgeProbeHeader;
+    DEBUG_LOG("Using default purge probe header %s", m_purge_probe_header.c_str());
+  }
 
   if (m_min_size_to_slice > 0) {
     if (m_oscache.has_value()) {

plugins/slice/Config.h

@@ -36,6 +36,8 @@ struct Config {
   static constexpr int64_t const blockbytesmax     = 1024 * 1024 * 128; // 128MB
   static constexpr int64_t const blockbytesdefault = 1024 * 1024;       // 1MB
 
+  static constexpr int const purgeprobeblocksdefault = 8;
+
   int64_t     m_blockbytes{blockbytesdefault};
   std::string m_remaphost; // remap host to use for loopback slice GET
   std::string m_regexstr;  // regex string for things to slice (default all)
@@ -49,8 +51,12 @@ struct Config {
   bool     m_head_strip_range{false}; // strip range header for head requests
   uint64_t m_min_size_to_slice{0};    // Only strip objects larger than this
 
+  // consecutive uncached blocks a purge tolerates before giving up on the object
+  int m_purge_probe_blocks{purgeprobeblocksdefault};
+
   std::string m_skip_header;
   std::string m_crr_ident_header;
+  std::string m_purge_probe_header; // request header overriding m_purge_probe_blocks
 
   // Convert optarg to bytes
   static int64_t bytesFrom(char const *const valstr);

plugins/slice/Data.h

@@ -77,6 +77,10 @@ struct Data {
   int64_t m_blockskip{0};     // number of bytes to skip in this block
   int64_t m_blockconsumed{0}; // body bytes consumed
 
+  int64_t m_purge_hits{0};       // blocks a purge actually removed
+  int     m_purge_misses{0};     // consecutive uncached blocks the walk has seen
+  int     m_purge_miss_bound{0}; // from the config or the request header
+
   BlockState m_blockstate{Pending}; // is there an active slice block
 
   int64_t m_bytestosend{0}; // header + content bytes to send
@@ -109,11 +113,25 @@ struct Data {
     memset(&m_client_ip, 0, sizeof(m_client_ip));
   }
 
-  // Check if response only expects header
+  // HEAD only; a purge sends just a header too but never reaches the transfer path
   bool
   onlyHeader() const
   {
-    return (m_method_type == TS_HTTP_METHOD_HEAD || m_method_type == TS_HTTP_METHOD_PURGE);
+    return m_method_type == TS_HTTP_METHOD_HEAD;
+  }
+
+  bool
+  is_purge() const
+  {
+    return m_method_type == TS_HTTP_METHOD_PURGE;
+  }
+
+  // The purge range, closed against the object length once known. m_req_range
+  // stays as sent so a longer extent can widen the walk; a clamp could only shrink.
+  Range
+  purge_range() const
+  {
+    return (m_contentlen < 0) ? m_req_range : m_req_range.intersectedWith(Range(0, m_contentlen));
   }
 
   ~Data()

plugins/slice/HttpHeader.cc

@@ -326,6 +326,32 @@ HttpHeader::toString() const
 
 /////// HdrMgr
 
+bool
+HdrMgr::create_response(TSHttpStatus const status)
+{
+  resetHeader();
+
+  if (nullptr == m_buffer) {
+    m_buffer = TSMBufferCreate();
+  }
+
+  m_lochdr = TSHttpHdrCreate(m_buffer);
+  if (nullptr == m_lochdr) {
+    return false;
+  }
+
+  TSHttpHdrTypeSet(m_buffer, m_lochdr, TS_HTTP_TYPE_RESPONSE);
+  TSHttpHdrVersionSet(m_buffer, m_lochdr, TS_HTTP_VERSION(1, 1));
+  TSHttpHdrStatusSet(m_buffer, m_lochdr, status);
+
+  char const *const reason = TSHttpHdrReasonLookup(status);
+  if (nullptr != reason) {
+    TSHttpHdrReasonSet(m_buffer, m_lochdr, reason, strlen(reason));
+  }
+
+  return true;
+}
+
 TSParseResult
 HdrMgr::populateFrom(TSHttpParser const http_parser, TSIOBufferReader const reader, HeaderParseFunc const parsefunc,
                      int64_t *const bytes)

plugins/slice/HttpHeader.h

@@ -39,6 +39,10 @@
 constexpr std::string_view SLICE_CRR_HEADER = {"Slice-Crr-Status"};
 constexpr std::string_view SLICE_CRR_VAL    = "1";
 
+// extent of the object a PURGE removed, reported by ATS on a successful purge.
+// Emitted by HttpTransact::delete_all_document_alternates_and_return.
+constexpr std::string_view PURGED_CONTENT_RANGE = {"X-Purged-Content-Range"};
+
 /**
   Designed to be a cheap throwaway struct which allows a
   consumer to make various calls to manipulate headers.
@@ -207,6 +211,13 @@ struct HdrMgr {
     }
   }
 
+  /** Create an owned HTTP/1.1 response header with the given status.
+   *
+   * For a response slice forms itself, with no server response to relay. An
+   * intercept is an HTTP/1.x channel, so the version is not negotiable.
+   */
+  bool create_response(TSHttpStatus const status);
+
   void
   resetHeader()
   {