The Hypertext Transfer Protocol (HTTP) Entity Tag ("ETag") Response Header in Write Operations

Note that there is a related problem: modifying content-negotiated resources. Here the consensus seems to be simply not to do it. Instead, the origin server should reveal specific URIs of content that is not content-negotiated in the Content-Location response header ([RFC2616], Section 14.14), and user agents should use this more specific URI for authoring. Thus, the remainder of this document will focus on resources for which no content negotiation takes place.¶

Another related question is the usability of the weak entity tags for authoring (see [RFC2616], Section 13.3.3). Although this document focuses on the usage of strong entity tags, it is believed that the changes suggested in this document could be applied to weak entity tags as well.¶

For a long time, nobody realized that there was a problem at all, or those who realized preferred to ignore it.¶

Server implementers added code that would return the new value of the "ETag" header in a response to a successful PUT request. After all, the client could be interested in it.¶

User agent developers in turn were happy to get a new "ETag" value, saving a subsequent HEAD request to retrieve the new entity tag.¶

However, at some point of time, potentially during a Web Distributed Authoring and Versioning (WebDAV, [RFC2518], obsoleted by [RFC4918]) interoperability event, client programmers asked server programmers to always return "ETag" headers upon PUT, never ever to change the entity tag without "good reason", and - by the way - always to guarantee that the server stores the new content octet-by-octet.¶

From the perspective of client software that wants to treat an HTTP server as a file system replacement, this makes a lot of sense. After all, when one writes to a file one usually expects the file system to store what was written, and not to unexpectedly change the contents.¶

However, in general, an HTTP server is not a file system replacement. There may be some that have been designed that way, and some that expose some parts of their namespace that have this quality. But in general, HTTP server implementers have a lot of freedom in how resources are implemented. Indeed, this flexibility is one of the reasons for HTTP's success, allowing it to be used for a wide range of tasks, of which replacing file systems is just one (and not necessarily the most interesting one).¶

In particular: ¶

• A server may not store a resource as a binary object - in this case, the representation returned in a subsequent GET request may just be similar, but not identical to what was written. Good examples are servers that use HTTP to access XML data ([RFC4825]), Calendaring data ([RFC4791]) or newsfeed data ([RFC5023]).
• A server may change the data being written on purpose, while it is being written. Examples that immediately come to mind are keyword substitution in a source control system, or filters that remove potentially insecure parts out of HTML pages.

Furthermore: ¶

• An "unrelated" method such as WebDAV's PROPPATCH (see [RFC4918], Section 9.2) may affect the entity body and therefore the entity tag in an unexpected way, because the server stores some or all of the WebDAV properties inside the entity body (for instance, GPS information inside a JPG image file).

As long as servers store the content octet-by-octet, and return exactly what the client wrote, there is no problem at all.¶

Things get more interesting when a server does change the content, such as in the "simple authoring" example given in Appendix A.1. Here, the server does change the content upon writing to the resource, yet no harm is done, because the final state of the resource on the server does not depend on the client being aware of that.¶

All of the content rewriting examples mentioned above have this quality: the client can safely continue to edit the entity it sent, because the result of the transformation done by the server will be the same in the end. Formally, if we call the server-side transformation "t", the initial content "c", and the client-side editing steps "e1" and "e2", then¶

t(e2(e1(c))) = t(e2(t(e1(c))))

e.g., it is irrelevant whether the client obtained the current entity body before doing its second edit.

[example.for.non.safe.rewrite: Question: does anybody know a real-world example for server-side content rewriting where the above is not true? --julian.reschke@greenbytes.de] ¶

Problems will only occur if the client uses the entity body it sent, and the entity tag it obtained in return, in subsequent requests that only transfer part of the entity body, such as GET or PUT requests using the "Range" request header (see [RFC2616], Section 14.35).¶

Furthermore, some clients need to expose the actual contents to the end user. These clients will have to ensure that they really have the current representation.¶

Entity bodies (and thus entity tags) changing due to side effects of seemingly unrelated requests are indeed a problem, as demonstrated in Appendix A.3, and this specification proposes a way to resolve this in Section 3.¶

There are several places in the HTTP/1.1 specification ([RFC2616]) mentioning the "ETag" response header.¶

The ETag response-header field provides the current value of the entity tag for the requested variant. Sections 14.24, 14.26 and 14.44 describe the headers used with entity tags. The entity tag MAY be used for comparison with other entities from the same resource (see Section 13.3.3).¶

The response-header fields allow the server to pass additional information about the response which cannot be placed in the Status-Line. These header fields give information about the server and about further access to the resource identified by the Request-URI.¶

The "ETag" response header itself is mentioned mainly in the context of cache validation, such as in Section 13.3.2. What is missing is a coherent description on how the origin server can notify the user-agent when the entity tag changes as result of a write operation, such as PUT.¶

A 201 response MAY contain an ETag response header field indicating the current value of the entity tag for the requested variant just created, see Section 14.19.¶

The "ETag" response header is mentioned again in the definition of 206 Partial Content (Section 10.2.7) and 304 Not Modified (Section 10.3.5), but notably missing are statements about other 2xx series status codes that can occur upon a successful PUT operation, such as 200 OK (Section 10.2.1) and 204 No Content (Section 10.2.5).¶

Summarizing, the specification is a bit vague about what an ETag response header upon a write operation means, but this problem is somewhat mitigated by the precise definition of a response header. A proposal for enhancing [RFC2616] in this regard is made in Section 3 below.¶

While working on the revision of [RFC2518], the IETF WebDAV working group realized that this is a generic problem that needs attention independently of WebDAV. An initial attempt was made with [draft-whitehead-http-etag] in February 2006, but no progress was made since.¶

Since then, two specifications based on HTTP were just published by the RFC Editor, taking two opposite approaches: ¶

• Section 8.5 of [RFC4825] makes it a MUST-level requirement to return an entity tag upon PUT, even though the very nature of an XCAP server will cause it to rewrite contents (due to its XML-based storage).
• Section 5.3.4 of [RFC4791] explicitly forbids ("MUST NOT") returning an entity tag upon PUT if the content was rewritten.

In essence, this makes it impossible to implement an HTTP resource that conforms to both specifications. Due to the differing use cases of XCAP and CalDAV, this may not be a problem in practice, but the disagreement in itself is scary. Publication of these specifications on the standards track will make it much harder for future protocols to deal with this topic in a meaningful way (comments were sent during IETF Last Call for CalDAV, see <http://www1.ietf.org/mail-archive/web/ietf/current/msg43021.html>).¶

Content was stored octet-by-octet:

HTTP/1.1 200 OK
ETag: "1"
Entity-Transform: identity "1"

Content was transformed:

HTTP/1.1 200 OK
ETag: "2"
Entity-Transform: unspecified "2"

Response containing a stale "Entity-Transform" header:

HTTP/1.1 200 OK
ETag: "3"
Entity-Transform: unspecified "2"

Note that in this case the newly assigned entity tag and the entity tag returned in "Entity-Transform" do not match, thus the client is aware that the header value is stale and can't be used.

8.1. Normative References

[RFC2119]

Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels”, BCP 14, RFC 2119, March 1997.

[RFC2616]

Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, “Hypertext Transfer Protocol -- HTTP/1.1”, RFC 2616, June 1999.

8.2. Informative References

[draft-whitehead-http-etag]

Whitehead, J., “Design Considerations for State Identifiers in HTTP and WebDAV”, Internet-Draft draft-whitehead-http-etag-00 (work in progress), February 2006.
(Not updated since.)

[HTML]

Raggett, D., Hors, A., and I. Jacobs, “HTML 4.01 Specification”, W3C REC-html401-19991224, December 1999, <http://www.w3.org/TR/html401/>.

[RFC2518]

Goland, Y., Whitehead, E., Faizi, A., Carter, S., and D. Jensen, “HTTP Extensions for Distributed Authoring -- WEBDAV”, RFC 2518, February 1999.

[RFC4791]

Daboo, C., Desruisseaux, B., and L. Dusseault, “Calendaring Extensions to WebDAV (CalDAV)”, RFC 4791, March 2007.

[RFC4825]

Rosenberg, J., “The Extensible Markup Language (XML) Configuration Access Protocol (XCAP)”, RFC 4825, May 2007.

[RFC4918]

Dusseault, L., Ed., “HTTP Extensions for Web Distributed Authoring and Versioning (WebDAV)”, RFC 4918, June 2007.

[RFC5023]

Gregorio, J., Ed. and B. de hOra, Ed., “The Atom Publishing Protocol”, RFC 5023, October 2007.

Let us consider a server not having the quality of preserving octet-by-octet identity, for instance because of SVN-style keyword expansion in text content (<http://svnbook.red-bean.com/en/1.2/svn-book.html#svn.advanced.props.special.keywords>).¶

>> Request (1)

GET /test HTTP/1.1
Host: example.com

>> Response (1)

HTTP/1.1 200 OK
Content-Type: text/plain
ETag: "1"

# $Revision: 1 $
Sample text.

>> Request (2)

PUT /test HTTP/1.1
Host: example.com
If-Match: "1"

# $Revision: 1 $
New sample text.

>> Response (2a)

HTTP/1.1 412 Precondition Failed
Content-Type: text/plain

Precondition Failed: entity tag supplied in If-Match request header
did not match current.

In this case, the client usually has take care of merging the changes made locally with those made on the server ("Merge Conflict").¶

>> Response (2b)

HTTP/1.1 200 OK
Content-Type: text/plain
ETag: "2"

Note however, that at this point the client knows the new entity tag, but doesn not know the current representation, which will have been updated by the server to:

# $Revision: 2 $
New sample text.

>> Request (3)

PUT /test HTTP/1.1
Host: example.com
If-Match: "2"

# $Revision: 1 $
A third attempt.

>> Response (3)

HTTP/1.1 200 OK
Content-Type: text/plain
ETag: "3"

Note that the only problem here is that the client doesn't have an exact copy of the entity it is editing. However, from the server's point of view this is entirely irrelevant, because the "Revision" keyword will be automatically updated upon every write anyway.¶

In the example above (Appendix A.1, ERROR: xref to unknown element: preamble (at line 936)), the server returns an ETag response header upon PUT, although the content actually was rewritten due to keyword substitution.¶

>> Response (2c)

HTTP/1.1 200 OK
Content-Type: text/plain
ETag: "2"
Entity-Transform: unspecified "2"

This indicates that the content that was sent is not octet-by-octet identical to what a subsequent GET request would retrieve. It is then up to the client to decide whether it wants to re-fetch the content before continuing edits.¶

In this example, the server exposes data extracted from the HTML <title> element ([HTML], Section 7.4.2) as a custom WebDAV property ([RFC4918], Section 4), allowing both read and write access.¶

>> Request (1)

GET /test.html HTTP/1.1
Host: example.com

>> Response (1)

HTTP/1.1 200 OK
Content-Type: text/html
ETag: "A"

<html>
  <head>
  </head>
  <body>
  </body>
</html>

>> Request (2)

PUT /test.html HTTP/1.1
Host: example.com
If-Match: "A"

<html>
  <head>
  </head>
  <body>
    <p>First paragraph.</p>
  </body>
</html>

>> Response (2)

HTTP/1.1 200 OK
ETag: "B"

>> Request (3)

PROPPATCH /test.html HTTP/1.1
Host: example.com
Content-Type: application/xml

<proppatch xmlns="DAV:">
  <set>
    <prop>
      <title xmlns="http://ns.example.org/"
      >A sample document</title>
    </prop>
  </set>
</proppatch>

>> Response (3)

HTTP/1.1 207 Multi-Status
Content-Type: application/xml

<multistatus xmlns="DAV:">
  <response>
    <href>/test.html</href>
    <propstat>
      <prop>
        <title xmlns="http://ns.example.org/"/>
      </prop>
      <status>HTTP/1.1 2OO OK</status>
    </propstat>
  </response>
</multistatus>

The server knows how to propagate property changes into the HTML content, so it updates the entity by adding an HTML title document accordingly. This causes the entity tag changing to "C".¶

The new entity body is shown below, but the client does not realize that it did change at all.

<html>
  <head>
    <title>A sample document</title>
  </head>
  <body>
    <p>First paragraph.</p>
  </body>
</html>

A subsequent attempt by the client to update the entity body will fail, unless it realizes that changing WebDAV properties may affect the entity as well. In this case, it would have had to get the current entity tag before proceeding. Of course, this introduces an additional round-trip, and a timing window during which overlapping updates by other clients would go unnoticed.¶

• Note that a client would be affected by this behaviour even in the presence of an exclusive WebDAV write lock (see [RFC4918], Section 7).

Below we repeat the example from above (Appendix A.3), but here the origin server returns entity tags for all write operations, and the user agent knows how to deal with them. That is, both take advantage of what [RFC2616] already allows.¶

>> Request (3)

PROPPATCH /test.html HTTP/1.1
Host: example.com
Content-Type: application/xml

<proppatch xmlns="DAV:">
  <set>
    <prop>
      <title xmlns="http://ns.example.org/">
        A sample document
      </title>
    </prop>
  </set>
</proppatch>

>> Response (3)

HTTP/1.1 207 Multi-Status
Content-Type: application/xml
ETag: "C"

<multistatus xmlns="DAV:">
  <response>
    <href>/test.html</href>
    <propstat>
      <prop>
        <title xmlns="http://ns.example.org/"/>
      </prop>
      <status>HTTP/1.1 2OO OK</status>
    </propstat>
  </response>
</multistatus>

As before, this causes the entity to change, and a new entity tag to be assigned. But in this case, the origin server actually notifies the client of the changed state by including the "ETag" response header.¶

The client now will be aware that the requested entity change, and can use the new entity tag in subsequent requests (potentially after refreshing the local copy).¶

Add and resolves issues "entity-header" and "extensibility", by removing the extension hooks, and by redefining the header to it can be used as an Entity header.¶

Update APP and CALDAV references. Remove RFC3986 reference (not needed anymore after the simplication in draft 01). Fix typo in header description ("submitted entity", not "stored entity"). Remove comparison about how XCAP and CALDAV profile RFC2616: after all, both mandate a behaviour that was legal but optional before. Add "Updates: RFC2616".¶

In the references, note that there was no activitiy on draft-whitehead-http-etag-00 anymore. Change intended status to "Experimental". Update APP reference. Update statements about current status of XCAP and CALDAV. Add and resolve "clarify-extension".¶

Update XCAP reference. Update header definition to use prose rules rather than comments. Clarify "identity" keyword when appearing with a weak entity-tag. In PROPPATCH/HTML example, fix whitespace so that the HTML title element has exactly the same value as the one that was PROPPATCHed. Mention that ETags changing due to a PROPPATCH could even occur in presence of a WebDAV write lock. Expand keyword substitution example with variant where Entity-Transform is returned.¶

Update APP reference. Update CalDAV reference (published as RFC4791). Update "prior work" (RFC4791 being published, XCAP having been approved).¶

Update XCAP reference (published as RFC4825). Update "prior work" (RFC4825 being published).¶

Replace RFC2518 references with RFC4918 where appropriate. Update APP reference.¶

Update APP reference. Reference the "requested variant" issue on the RFC2616bis issues list (<http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i69>).¶

Update pointer to RFC2616bis "requested variant" issue.¶