Skip to content

Commit 6db3f78

Browse files
committed
Preserve probe run when a framework fails a baseline test
1 parent a1973d7 commit 6db3f78

34 files changed

Lines changed: 435 additions & 279 deletions

File tree

.github/workflows/probe.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,7 @@ jobs:
293293
f.write('\n'.join(lines))
294294
295295
if baseline_failures:
296-
sys.exit(1)
296+
print('::warning::Baseline failures detected — results will still be published')
297297
PYEOF
298298
299299
# ── Upload / publish ───────────────────────────────────────────

AGENTS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -273,12 +273,12 @@ Your server MUST listen on **port 8080** and implement these endpoints:
273273
| Endpoint | Method | Behavior |
274274
|----------|--------|----------|
275275
| `/` | `GET` | Return `200 OK` |
276-
| `/` | `HEAD` | Return `200 OK` with no body |
277276
| `/` | `POST` | Read the full request body and return it in the response body |
278-
| `/` | `OPTIONS` | Return `200 OK` |
279277
| `/echo` | `GET`, `POST` | Return all received request headers in the response body, one per line as `Name: Value` |
280278
| `/cookie` | `GET`, `POST` | Parse the `Cookie` header and return each cookie as `name=value` on its own line |
281279

280+
HEAD and OPTIONS are handled automatically by virtually all frameworks — do not implement them explicitly.
281+
282282
The `/echo` endpoint is critical for normalization tests. It must echo back all headers the server received, preserving the names as the server internally represents them.
283283

284284
Example `/echo` response body:

docs/content/compliance/_index.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,12 +62,10 @@ Each test sends a request that violates a specific **MUST** or **MUST NOT** requ
6262
'COMP-DUPLICATE-CT','COMP-POST-UNSUPPORTED-CT',
6363
'COMP-ACCEPT-NONSENSE'
6464
]},
65-
{ key: 'methods-upgrade', label: 'Methods & Upgrade', testIds: [
65+
{ key: 'methods-headers', label: 'Methods & Headers', testIds: [
6666
'COMP-METHOD-CONNECT',
6767
'COMP-UNKNOWN-TE-501','COMP-EXPECT-UNKNOWN','COMP-METHOD-TRACE',
6868
'COMP-TRACE-WITH-BODY','COMP-TRACE-SENSITIVE',
69-
'COMP-UPGRADE-POST','COMP-UPGRADE-MISSING-CONN',
70-
'COMP-UPGRADE-UNKNOWN','COMP-UPGRADE-INVALID-VER','COMP-UPGRADE-HTTP10',
7169
'COMP-CONNECTION-CLOSE','COMP-HTTP10-DEFAULT-CLOSE','COMP-HTTP10-NO-HOST'
7270
]}
7371
];

docs/content/docs/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ Reference documentation for every test in Http11Probe, organized by topic. Each
2121
{{< card link="body" title="Body Handling" subtitle="Content-Length body consumption, chunked transfer encoding, incomplete bodies, chunk extensions." icon="document-download" >}}
2222
{{< card link="smuggling" title="Request Smuggling" subtitle="CL+TE conflicts, TE obfuscation, pipeline injection, and why ambiguous framing is dangerous." icon="shield-exclamation" >}}
2323
{{< card link="malformed-input" title="Malformed Input" subtitle="Binary garbage, oversized fields, control characters, incomplete requests." icon="lightning-bolt" >}}
24-
{{< card link="upgrade" title="Upgrade / WebSocket" subtitle="Protocol upgrade validation, WebSocket handshake method and version checks." icon="arrow-up" >}}
24+
{{< card link="websockets" title="WebSockets" subtitle="Protocol upgrade validation, WebSocket handshake method and version checks." icon="arrow-up" >}}
2525
{{< card link="normalization" title="Header Normalization" subtitle="Echo-based tests checking if servers normalize malformed header names (underscore, tab, casing)." icon="adjustments" >}}
2626
{{< card link="caching" title="Caching" subtitle="Optional feature probes — conditional requests, ETag handling, caching behavior." icon="beaker" >}}
2727
{{< /cards >}}

docs/content/docs/rfc-requirement-dashboard.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ The RFC requires rejection, but the mechanism (400 status or connection close) h
110110
| 57 | `COMP-POST-CL-ZERO` | Compliance | [RFC 9112 §6.2](https://www.rfc-editor.org/rfc/rfc9112#section-6.2) | Content-Length: 0 is a valid 1*DIGIT value. Server **MUST** accept zero-length body. |
111111
| 58 | `COMP-POST-NO-CL-NO-TE` | Compliance | [RFC 9112 §6.3](https://www.rfc-editor.org/rfc/rfc9112#section-6.3) | "If this is a request message and none of the above are true, then the message body length is zero (no message body is present)." Server **MUST** treat as zero-length. |
112112
| 59 | `COMP-RANGE-POST` | Compliance | [RFC 9110 §14.2](https://www.rfc-editor.org/rfc/rfc9110#section-14.2) | "A server **MUST** ignore a Range header field received with a request method other than GET." |
113-
| 60 | `COMP-UPGRADE-HTTP10` | Compliance | [RFC 9110 §7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) | "A server **MUST** ignore an Upgrade header field that is received in an HTTP/1.0 request." |
113+
| 60 | `WS-UPGRADE-HTTP10` | WebSockets | [RFC 9110 §7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) | "A server **MUST** ignore an Upgrade header field that is received in an HTTP/1.0 request." |
114114
| 59 | `COMP-CHUNKED-BODY` | Compliance | [RFC 9112 §7.1](https://www.rfc-editor.org/rfc/rfc9112#section-7.1) | "A recipient **MUST** be able to parse and decode the chunked transfer coding." |
115115
| 60 | `COMP-CHUNKED-MULTI` | Compliance | [RFC 9112 §7.1](https://www.rfc-editor.org/rfc/rfc9112#section-7.1) | Same — multi-chunk is the standard chunked format. **MUST** accept. |
116116
| 61 | `COMP-CHUNKED-EMPTY` | Compliance | [RFC 9112 §7.1](https://www.rfc-editor.org/rfc/rfc9112#section-7.1) | A zero-length chunked body (just `0\r\n\r\n`) is valid. **MUST** accept. |
@@ -119,9 +119,9 @@ The RFC requires rejection, but the mechanism (400 status or connection close) h
119119
| 64 | `COMP-CHUNKED-EXTENSION` | Compliance | [RFC 9112 §7.1.1](https://www.rfc-editor.org/rfc/rfc9112#section-7.1.1) | "A recipient **MUST** ignore unrecognized chunk extensions." Server **MUST** accept and ignore. |
120120
| 65 | `COMP-POST-CL-UNDERSEND` | Compliance | [RFC 9112 §6.2](https://www.rfc-editor.org/rfc/rfc9112#section-6.2) | "If the sender closes the connection or the recipient times out before the indicated number of octets are received, the recipient **MUST** consider the message to be incomplete and close the connection." |
121121
| 66 | `COMP-CHUNKED-NO-FINAL` | Compliance | [RFC 9112 §7.1](https://www.rfc-editor.org/rfc/rfc9112#section-7.1) | "The message body length is determined by reading and decoding the chunked data until the transfer coding indicates the data is complete." Without a zero terminator, the transfer is incomplete. Server **MUST** not process as complete. |
122-
| 67 | `COMP-UPGRADE-POST` | Compliance | [RFC 6455 §4.1](https://www.rfc-editor.org/rfc/rfc6455#section-4.1) | "The method of the request **MUST** be GET, and the HTTP version **MUST** be at least 1.1." WebSocket upgrade via POST **MUST** not succeed. |
123-
| 68 | `COMP-UPGRADE-MISSING-CONN` | Compliance | [RFC 9110 §7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) | "A sender of Upgrade **MUST** also send an 'Upgrade' connection option in the Connection header field." Without Connection: Upgrade, the server **MUST NOT** switch protocols. |
124-
| 69 | `COMP-UPGRADE-UNKNOWN` | Compliance | [RFC 9110 §7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) | "A server **MUST NOT** switch to a protocol that was not indicated by the client in the corresponding request's Upgrade header field." Unknown protocol — **MUST NOT** return 101. |
122+
| 67 | `WS-UPGRADE-POST` | WebSockets | [RFC 6455 §4.1](https://www.rfc-editor.org/rfc/rfc6455#section-4.1) | "The method of the request **MUST** be GET, and the HTTP version **MUST** be at least 1.1." WebSocket upgrade via POST **MUST** not succeed. |
123+
| 68 | `WS-UPGRADE-MISSING-CONN` | WebSockets | [RFC 9110 §7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) | "A sender of Upgrade **MUST** also send an 'Upgrade' connection option in the Connection header field." Without Connection: Upgrade, the server **MUST NOT** switch protocols. |
124+
| 69 | `WS-UPGRADE-UNKNOWN` | WebSockets | [RFC 9110 §7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) | "A server **MUST NOT** switch to a protocol that was not indicated by the client in the corresponding request's Upgrade header field." Unknown protocol — **MUST NOT** return 101. |
125125
| 70 | `MAL-NUL-IN-URL` | Malformed | [RFC 9112 §2.2](https://www.rfc-editor.org/rfc/rfc9112#section-2.2) | NUL byte is not valid in any protocol element. Grammar violation — **MUST** reject. |
126126
| 71 | `MAL-CONTROL-CHARS-HEADER` | Malformed | [RFC 9110 §5.5](https://www.rfc-editor.org/rfc/rfc9110#section-5.5) | "Field values containing CR, LF, or NUL characters are invalid and dangerous... a recipient **MUST** either reject the message or replace each of those characters with SP." (CTL characters outside safe context.) |
127127
| 72 | `MAL-NON-ASCII-HEADER-NAME` | Malformed | [RFC 9112 §5](https://www.rfc-editor.org/rfc/rfc9112#section-5) | `field-name = token = 1*tchar`. Non-ASCII bytes are not tchar. Grammar violation — **MUST** reject. |
@@ -133,7 +133,7 @@ The RFC requires rejection, but the mechanism (400 status or connection close) h
133133
| 78 | `MAL-URL-OVERLONG-UTF8` | Malformed | [RFC 3629 §3](https://www.rfc-editor.org/rfc/rfc3629#section-3) | "Implementations of the decoding algorithm above **MUST** protect against decoding invalid sequences." Overlong UTF-8 (0xC0 0xAF) is explicitly invalid per RFC 3629. |
134134
| 79 | `NORM-SP-BEFORE-COLON-CL` | Normalization | [RFC 9112 §5.1](https://www.rfc-editor.org/rfc/rfc9112#section-5) | "A server **MUST** reject, with a response status code of 400 (Bad Request), any received request message that contains whitespace between a header field name and colon." |
135135
| 80 | `NORM-TAB-IN-NAME` | Normalization | [RFC 9112 §5](https://www.rfc-editor.org/rfc/rfc9112#section-5) | `field-name = token = 1*tchar`. Tab (0x09) is not a tchar. **MUST** reject — invalid token character. |
136-
| 81 | `COMP-UPGRADE-INVALID-VER` | Compliance | [RFC 6455 §4.4](https://www.rfc-editor.org/rfc/rfc6455#section-4.4) | "If the server doesn't support the requested version, it **MUST** abort the WebSocket handshake." (426 Upgrade Required preferred.) |
136+
| 81 | `WS-UPGRADE-INVALID-VER` | WebSockets | [RFC 6455 §4.4](https://www.rfc-editor.org/rfc/rfc6455#section-4.4) | "If the server doesn't support the requested version, it **MUST** abort the WebSocket handshake." (426 Upgrade Required preferred.) |
137137
| 82 | `COMP-UNKNOWN-TE-501` | Compliance | [RFC 9112 §6.1](https://www.rfc-editor.org/rfc/rfc9112#section-6.1) | "A server that receives a request message with a transfer coding it does not understand **SHOULD** respond with 501." Combined with unknown-TE-without-CL making body length indeterminate: **MUST** reject. |
138138
| 83 | `SMUG-TE-TRAILING-SPACE` | Smuggling | [RFC 9112 §6.1](https://www.rfc-editor.org/rfc/rfc9112#section-6.1) | "chunked " (with trailing space) is not an exact match for the registered coding "chunked". Combined with CL present: "the server **MUST** close the connection after responding." |
139139
| 84 | `MAL-POST-CL-HUGE-NO-BODY` | Malformed | [RFC 9112 §6.2](https://www.rfc-editor.org/rfc/rfc9112#section-6.2) | "If the sender closes the connection or the recipient times out before the indicated number of octets are received, the recipient **MUST** consider the message to be incomplete and close the connection." |
Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
2-
title: Upgrade / WebSocket
3-
description: "Upgrade / WebSocket — Http11Probe documentation"
2+
title: WebSockets
3+
description: "WebSockets — Http11Probe documentation"
44
weight: 12
55
sidebar:
66
open: false
@@ -26,6 +26,7 @@ The HTTP `Upgrade` mechanism allows a client to request switching to a different
2626
{{< card link="upgrade-post" title="UPGRADE-POST" subtitle="WebSocket upgrade via POST must not return 101." >}}
2727
{{< card link="upgrade-missing-conn" title="UPGRADE-MISSING-CONN" subtitle="Upgrade without Connection: Upgrade must not switch." >}}
2828
{{< card link="upgrade-unknown" title="UPGRADE-UNKNOWN" subtitle="Upgrade to unknown protocol must not return 101." >}}
29+
{{< card link="upgrade-http10" title="UPGRADE-HTTP10" subtitle="Upgrade header in HTTP/1.0 request must be ignored." >}}
2930
{{< /cards >}}
3031

3132
### Unscored

docs/content/docs/upgrade/upgrade-http10.md renamed to docs/content/docs/websockets/upgrade-http10.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ weight: 5
66

77
| | |
88
|---|---|
9-
| **Test ID** | `COMP-UPGRADE-HTTP10` |
10-
| **Category** | Compliance |
9+
| **Test ID** | `WS-UPGRADE-HTTP10` |
10+
| **Category** | WebSockets |
1111
| **Scored** | Yes |
1212
| **RFC** | [RFC 9110 §7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) |
1313
| **RFC Level** | MUST |

docs/content/docs/upgrade/upgrade-invalid-ver.md renamed to docs/content/docs/websockets/upgrade-invalid-ver.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ weight: 4
66

77
| | |
88
|---|---|
9-
| **Test ID** | `COMP-UPGRADE-INVALID-VER` |
10-
| **Category** | Compliance |
9+
| **Test ID** | `WS-UPGRADE-INVALID-VER` |
10+
| **Category** | WebSockets |
1111
| **RFC** | [RFC 6455 Section 4.4](https://www.rfc-editor.org/rfc/rfc6455#section-4.4) |
1212
| **Requirement** | MUST abort handshake (426 preferred) |
1313
| **Expected** | `426` or non-`101` |

docs/content/docs/upgrade/upgrade-missing-conn.md renamed to docs/content/docs/websockets/upgrade-missing-conn.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ weight: 2
66

77
| | |
88
|---|---|
9-
| **Test ID** | `COMP-UPGRADE-MISSING-CONN` |
10-
| **Category** | Compliance |
9+
| **Test ID** | `WS-UPGRADE-MISSING-CONN` |
10+
| **Category** | WebSockets |
1111
| **RFC** | [RFC 9110 Section 7.8](https://www.rfc-editor.org/rfc/rfc9110#section-7.8) |
1212
| **Requirement** | MUST NOT switch |
1313
| **Expected** | any non-`101` response |

docs/content/docs/upgrade/upgrade-post.md renamed to docs/content/docs/websockets/upgrade-post.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ weight: 1
66

77
| | |
88
|---|---|
9-
| **Test ID** | `COMP-UPGRADE-POST` |
10-
| **Category** | Compliance |
9+
| **Test ID** | `WS-UPGRADE-POST` |
10+
| **Category** | WebSockets |
1111
| **RFC** | [RFC 6455 Section 4.1](https://www.rfc-editor.org/rfc/rfc6455#section-4.1) |
1212
| **Requirement** | MUST use GET |
1313
| **Expected** | any non-`101` response |

0 commit comments

Comments
 (0)