4. netstat

The netstat utility summarizes a variety of characteristics of the networking stack. With netstat you can learn a number of important things. If no other type of data is requested it will report on the state of all active sockets. You can however request the routing table, masquerading table, network interface statistics, and network stack statistics [60].

4.1. Displaying socket status with netstat

One of the most common uses of the netstat utility is to determine the state of sockets on a machine. There are many questions that netstat can answer with the right set of options. Here's a list of some of the things different things we can learn.

  • which services are listening on which sockets

  • what process (and controlling PID) is listening on a given socket

  • whether data is waiting to be read on a socket

  • what connections are currently established to which sockets

By invoking netstat without any options, you are asking for a list of all currently open connections to and from the networking stack on the local machine. This means IP network connections, unix domain sockets, IPX sockets and Appletalk sockets among others. Naturally, we'll skip over the non-IP sockets since this is about IP networking with linux.

Assume the --inet switch in all cases below unless we are examining a particular higher layer protocol (e.g., TCP with the --tcp switch or UDP with --udp switch.

A convenient feature of netstat is its ability to differentiate between two different sorts of name lookup. Normally the -n specifies no name lookup, but this is ambiguous when there are hostnames, port names, and user names. Fortunately, netstat offers the following options to differentiate the different forms of lookup and suppress only the [un-]desired lookup.

  • --numeric-hosts

  • --numeric-ports

  • --numeric-users

The option -n (my favorite), suppress all hostname, port name and username lookup, and is a synonym for --numeric. I'll reiterate that hostnames and DNS in particular can be confusing, or worse, misleading when trying to diagnose or debug a networking related issue, so it is wise to suppress hostname lookups in these sorts of situations.

In Example G.11, “Displaying IP socket status with netstat we will look at netstat's numeric output and then we'll invoke the same command but suppress the host lookups. Though the output is almost the same, a particular situation might call for one or the other invocation.

Example G.11. Displaying IP socket status with netstat

[root@morgan]# netstat --inet -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0    192 192.168.98.82:22        192.168.99.35:40991     ESTABLISHED
tcp        0      0 192.168.98.82:42929     192.168.100.17:993      ESTABLISHED
tcp       96      0 127.0.0.1:40863         127.0.0.1:6010          ESTABLISHED
tcp        0      0 127.0.0.1:6010          127.0.0.1:40863         ESTABLISHED
tcp        0      0 127.0.0.1:38502         127.0.0.1:6010          ESTABLISHED
tcp        0      0 127.0.0.1:6010          127.0.0.1:38502         ESTABLISHED
tcp        0      0 192.168.98.82:53733     209.10.26.51:80         SYN_SENT
tcp        0      0 192.168.98.82:44468     192.168.100.17:993      ESTABLISHED
tcp        0      0 192.168.98.82:44320     192.168.100.17:139      TIME_WAIT
[root@morgan]# netstat --inet --numeric-hosts
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 192.168.98.82:ssh       192.168.99.35:40991     ESTABLISHED
tcp        0      0 192.168.98.82:42929     192.168.100.17:imaps    ESTABLISHED
tcp        0      0 127.0.0.1:40863         127.0.0.:x11-ssh-offset ESTABLISHED
tcp        0      0 127.0.0.:x11-ssh-offset 127.0.0.1:40863         ESTABLISHED
tcp        0      0 127.0.0.1:38502         127.0.0.:x11-ssh-offset ESTABLISHED
tcp        0      0 127.0.0.:x11-ssh-offset 127.0.0.1:38502         ESTABLISHED
tcp        0      0 192.168.98.82:53733     209.10.26.51:http       SYN_SENT
tcp        0      0 192.168.98.82:44468     192.168.100.17:imaps    ESTABLISHED
tcp        0      0 192.168.98.82:44320     192.168.100:netbios-ssn TIME_WAIT
          

Each line represents a either the sending or receiving half of a connection. In the above output on morgan it appears that there are no connections other than TCP connections. If you are very familiar with TCP ports and the service associated with that port, then the first format will suffice in most cases. A possibly misleading aspect of the latter output is visible in the connections to and from localhost and the final line. netstat abbreviates the IP endpoint in order to reproduce the entire string retrieved from the port lookup (in /etc/services). Also interestingly, this line conveys to us (in the first output) that the kernel is waiting for the remote endpoint to acknowledge the 192 bytes which are still in the Send-Q buffer.

The first line describes a TCP connection to the IP locally hosted on morgan's Ethernet interface. The connection was initiated from an ephemeral port (40991) on tristan to a service running on port 22. The service normally running on this well-known port is sshd, so we can conclude that somebody on tristan has connected to the morgan's ssh server. The second line describes a TCP session open to port 993 on isolde, which probably means that the user on morgan has an open connection to an IMAP over SSL server.

The third through the sixth lines can be understood in pairs. By examining the source and destination IP and port pairs, we can see that two different TCP sessions have been established with the source and destination address of 127.0.0.1. For an administrator to publish services on localhost is not at all uncommon. This makes the service harder to abuse from the network. In this case, when we allow the service lookup, the port in question (6010) appears to be used to tunnel forwarded X applications over ssh.

The next line is the first TCP session in our output which is not in a state of ESTABLISHED. Refer to Table G.1, “Possible Session States in netstat output” for a full list of the possible values of the State field in the netstat output. The state SYN_SENT means that an application has made arequest for a TCP session, but has not yet received the return SYN+ACK packet.

The final line of our netstatoutput shows a connection in the TIME_WAIT state, which means that the TCP sessions have been terminated, but the kernel is waiting for any packets which may still be left on the network for this session. It is not at all abnormal for sockets to be in a TIME_WAIT state for a short period of time after a TCP session has ended.

If we needed to know exactly which application owned a particular network connection, we would use the -p | --program switch which gives us the PID and process name of the owner process. If we want to see the unix user and the PID and process we'll add the -e | --extend switch.

Example G.12. Displaying IP socket status details with netstat

[root@masq-gw]# netstat -p -e --inet --numeric-hosts
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name   
tcp        0      0 192.168.100.254:ssh     192.168.100.17:49796    ESTABLISHED root       25453      6326/sshd
tcp        0    240 192.168.99.254:ssh      192.168.99.35:42948     ESTABLISHED root       171748     31535/sshd
          

There doesn't appear to be a large number of connections to and from the masq-gw host. The two sessions are initiated to the sshd running on port 22, and the process which owns each socket is a root process.

Table G.1. Possible Session States in netstat output

StateDescription
LISTENaccepting connections
ESTABLISHEDconnection up and passing data
SYN_SENTTCP; session has been requested by us; waiting for reply from remote endpoint
SYN_RECVTCP; session has been requested by a remote endpoint for a socket on which we were listening
LAST_ACKTCP; our socket is closed; remote endpoint has also shut down; we are waiting for a final acknowledgement
CLOSE_WAITTCP; remote endpoint has shut down; the kernel is waiting for the application to close the socket
TIME_WAITTCP; socket is waiting after closing for any packets left on the network
CLOSEDsocket is not being used (FIXME. What does mean?)
CLOSINGTCP; our socket is shut down; remote endpoint is shut down; not all data has been sent
FIN_WAIT1TCP; our socket has closed; we are in the process of tearing down the connection
FIN_WAIT2TCP; the connection has been closed; our socket is waiting for the remote endpoint to shut down

4.2. Displaying the main routing table with netstat

One of the most common uses of netstat, especially in cross-platform environments is the reporting of the main routing table. On many platforms, netstat -rn is the preferred method of displaying routing information, although linux provides at least two alternatives to this: route and ip route show.

Example G.13. Displaying the main routing table with netstat

[root@morgan]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.98.0    0.0.0.0         255.255.255.0   U        40 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U        40 0          0 lo
0.0.0.0         192.168.98.254  0.0.0.0         UG       40 0          0 eth0
          

This output should look familiar. The routing cache itself may not be as familiar to most, but can also be displayed with netstat. The ouput below is exactly the same as the ouput from route -enC. Refer also to Example D.3, “Viewing the routing cache with route.

Example G.14. Displaying the routing cache with netstat

[root@tristan]# netstat -rnC
Kernel IP routing cache
Source          Destination     Gateway         Flags   MSS Window  irtt Iface
194.52.197.133  192.168.99.35   192.168.99.35     l      40 0          0 lo
192.168.99.35   194.52.197.133  192.168.99.254         1500 0         29 eth0
192.168.99.35   192.168.99.254  192.168.99.254         1500 0          0 eth0
192.168.99.254  192.168.99.35   192.168.99.35     il     40 0          0 lo
192.168.99.35   192.168.99.35   192.168.99.35     l   16436 0          0 lo
192.168.99.35   194.52.197.133  192.168.99.254         1500 0          0 eth0
192.168.99.35   192.168.99.254  192.168.99.254         1500 0          0 eth0
          

Consult Section 1.1, “Displaying the routing table with route for more detail on reading and interpreting the data in this output. Because this is simply another way of reporting the routing table information, we'll skip over any detailed description.

4.3. Displaying network interface statistics with netstat command

netstat -i summarizes interface statistics in a terse format. This format

OK! This is strange. netstat -ie looks exactly like ifconfig output. That's weird!

4.4. Displaying network stack statistics with netstat

4.5. Displaying the masquerading table with netstat

For machines which perform masquerading, typically dual-homed packet-filtering firewalls like masq-gw a tool to list the current state of the masquerading table is convenient.

Each masqueraded connection can be described by a tuple of six pieces of data: the source IP and source port, the destination IP and destination port, and the (usually implicit) locally hosted IP and a local port.

Example G.15. Displaying the masquerading table with netstat

[root@masq-gw]# netstat -Mn


          

FIXME; this command seems to fail on all of the iptables boxen, even if I'm using the -j MASQUERADE target. I can use it successfully on ipchains boxen. Anybody have any ideas or explanation here?



[60] Additionally, netstat can display multicast information with the --group switch. I have zero experience here. Anybody with experience want to write about this?