Testing and debugging guide

This pages provides links and resources for the most common ways to debug and test your TCP, and our most important notes. We’ll keep this document updated with any known issues or tips.

For the most up-to-date notes, please see EdStem, specifically our TCP FAQ/reading list post.

How to test…

Sending and receiving files

For a demo of sending and receiving files, please see the video for Gearup III.

Zero-window probing (ZWP)

For a demo of sending and receiving files, please see the video for Gearup III.

Important notes

  1. There is a known bug in the reference’s implementation of zero-window probing that resends the probe byte after space is available in the window. You are not required to implement this behavior. For more discussion on this, see the gearup III notes, page 26.
  2. Wireshark may flag packets incorrectly when ZWP ends (e.g. “ACKed unseen segment”, “previous segment not captured”). This behavior is expected, and seems to be a limitation in Wireshark’s packet analysis.

For examples of how ZWP should look in Wireshark independent of items (1) and (2), see EdStem #536 and Edstem #523.

Retransmissions

To test retransmissions independently of sending files, we recommend starting up a connection and then configuring your router to drop packets, either by taking down an interface, or by using the drop command in the reference router.

An example test may look like this (try with wireshark open):

  1. Start up the linear-r1h2 network, make a TCP connection between h1 and h2
  2. Send a couple of data segments (which should compute an initial RTO estimate)
  3. Configure the router to drop all packets (down if0 or drop 1)
  4. Try to send a segment. In Wireshark, you should see the segment be sent (and not ACKed, because it was dropped by the router), then retransmitted after the RTO timer expires. To make sure the timeout value matches what you expect, check Wireshark’s timeout values.

For a more comprehensive test, we recommend trying to send a file between hosts with a loss rate of at most 2% (which is the highest value we will use when testing). For a large enough file (100KB-1MB) you should see plenty of examples of retransmissions that you can check by comparing what you see in Wireshark with any log output, timestamps, etc.

Early arrivals / out-of-order packets

Testing early arrivals can work similarly to testing retransmissions, as in the previous section. Here’s an example for how a test could look:

  1. Setup: for testing purposes, hard-code your RTO to be a super-high value (e.g., 10 seconds or greater), and disable any code that updates the RTO when you receive an ACK
  2. Configure your router to drop all packets (e.g., down if0)
  3. Send a segment (we’ll call it S1)
  4. Re-enable the router’s interface
  5. Send another segment (S2)

Since the RTO is very high, S2 will arrive before S1, and therefore S2 (and any segments you send after it) will be early arrivals, allowing you to see how this processing would happen with a small number of packets.

How to debug…

General tips

As you debug, don’t be afraid to disable or modify certain parts of your TCP to help you test others. Some examples:

  1. If you use a lot of print statements set your minimum RTO to value >=250ms. With a lower value (eg. 1ms), print statements may slow down your threads enough to cause spurious retransmissions.

  2. Wireshark output too confusing to read? Consider disabling retransmissions or setting a very high RTO (many seconds). If you are testing without packet loss, lots of retransmissions may be a sign of a bug that occurred in an earlier step, so turning off retransmissions entirely may help rule out certain issues. Similarly, consider increasing your zero-window probe interval, to reduce the number of probe packets.

  3. If you are debugging issues receiving files, consider disabling early-arrival processing (don’t add to the queue), to help rule this out.

  4. To debug possible race conditions, consider slowing down your sender by, eg, adding a sleep in your logic for sf. You may also consider slowing down the sending process inside your TCP stack (though this often also requires changing your RTO).

Invalid or missing data in received file

If you notice your receiver writing extra bytes to a file, or otherwise receiving invalid data, a good place to start is to figure out if the data is being sent correctly. Wireshark has some features that can help with this! To try it:

  1. Start a new Wireshark capture and send a file with known content. For best results, it may be easiest to send a small file, so you can check all the content.

  2. In Wireshark, Right-click on any packet in the TCP stream and go to Follow > TCP Stream

  3. Wireshark should open a new window showing (roughly) the content of your file, like the figure below. These are the bytes Wireshark reconstructed from the TCP stream! To look more closely, click on any byte in the TCP stream view–your main wireshark window should jump to the segment that contained this byte, helping you pinpoint issues!

The TCP stream is a good way to help diagnose problems:

  • If your TCP stream looks correct but your file/VRead is wrong, this means your receiver logic may be processing segments is incorrectly
  • If Wireshark’s TCP stream looks incorrect, this means something may be wrong about how the segments are sent

A deadlock (in Go)

Go provides a super helpful feature that can help debug a hang or deadlock: in any Go program, pressing Ctrl+\ will exit the program and dump a stack trace of all goroutines. When using vnet_run with tmux, the output of the stack trace can be hard to read–here’s a tutorial on how to make this easier:

  1. In tmux, put your cursor in the terminal for the node that is hanging
  2. Enter the tmux command Ctrl+b z. (That is, press Ctrl+b, let go, then press z). This “zooms in” on the current tmux terminal, making it fill the screen. (To go back, press Ctrl+b z again.)
  3. With your terminal pane zoomed-in, press Ctrl+\. This will abort your node and print a very long stack trace.

Scrolling up: The stack trace is likely too big to read in your terminal. To “scroll up” inside tmux, do the following:

  1. Press Ctrl+b [
  2. Use the arrow keys or page up/page down to move in the window. (When you’re done, press Ctrl+c to go back to normal mode.)

Want to see it live? Here’s a video:

Reading the stack trace: The output contains one stack trace for each goroutine in your program. This will look pretty gnarly at first, but, fortunately, you only need to pay attention to a small part of it. Here’s a strategy:

  1. Ignore goroutines where you don’t recognize any functions. Many of these are part of Go itself (eg. garbage collection), and have are not related to your code.
  2. Focus on what you do recognize. As you scroll through the log, you’ll eventually see references to your own code. Examine these to try and figure out which thread it is, and what code it is executing. (Then, consider, is this what you expect that thread to be doing?)
  3. Go adds a hint at the top of each stack trace: for example, it identifies which threads are blocked on channels, which are waiting for I/O (files, network), and the threads that are part of go. Use these to help sort through the list!

For an example of (2) and (3), here’s a screenshot of a stack trace in the reference, which is waiting on a channel:

Using the debugger

Sick of print statements? With ~5min of setup, you can run one of your hosts in VSCode’s debugger–see here for instructions!

When using the debugger, note that you may need to disable retransmissions, or set your RTO to a very high value, in order to avoid retransmissions occurring while you are trying to debug.

Creating a capture file

For your final submission, you will need to submit a capture file of a file transmission with your TCP. A capture file is a standardized format for exporting packets from Wireshark and other packet-capture tools. To create a capture file from an existing Wireshark view:

  1. Stop your current capture
  2. Go to File > Save as…
  3. Save the file somewhere in your repo directory

You may also find capture files for debugging purposes (eg. to save an error case to examine later). The course staff may also request you send a capture file when debugging certain issues on Edstem.