Server Bug Fix: server is not responding on SYN packets

Original Source Link

On the attached tcp dump, the first two SYN packets (#21800 and 21801) came to the server, however SYN ACK was sent for the second SYN. Is that correct behaviour? My understanding is that the client is trying to establish two TCP connections from different src ports, so both connections should have been established. After 4 retries the client changed its src port from 13158 to 2352, and the TCP connection succeeded.
Is this an issue on the client or server side?

Similar article below doesn’t seem to be related.
Why would a server not send a SYN/ACK packet in response to a SYN packet

TCP Dump (client IP masked):

https://www.dropbox.com/s/3qkh1jw8emimh21/tcpdump1.png?dl=0

Seems to be an issue on the server side. The client retransmitted the initial packet around 16:30:44 (the black sequence of lines on your screenshot) and the server finally replied. So it appears the very first packet was either lost on the server side, or server could not handle it properly for some reason (listen queue overflow, not enough workers, CPU saturation or something else).

Tagged : / / / /

Server Bug Fix: tcpdump, how to capture actual data only?

Original Source Link

For example I have a server listening on port 8001, a client programe open a tcp socket connect to that port, send some binary data, I want to capture the actual data only without any TCP/IP header such as tcp handshake stuff..

Is that possible with tcpdump?

Nope, the tcpdump haven’t features to exctract payload (cut off the headers). Also the aren’t any built-in features about deep analysis of application layer data. But you can write the traffic into the file, and extract the actual data with wireshark features.

You can obtain something similar using tcpdump -i any <your_filter> -A. From man tcpdump:

-A Print each packet (minus its link level header) in ASCII. Handy for capturing web pages.

To obtain a more focused dump (ie: remove some handshake packet) you can play with the filter rules, for example removing tcp-syn packet from filtering.

Tagged :

Server Bug Fix: Re-routing DNS queries with iptables properly without resolv.conf

Original Source Link

I run a custom DNS service on 127.0.0.1:53 and I dont just want all queries to go through it, I also want only that service to respond back. Which is where the problem arises. My NAT iptables setup:

iptables -t nat -A OUTPUT -p tcp --dport 53 -j DNAT --to-destination 127.0.0.1:53
iptables -t nat -A OUTPUT -p udp --dport 53 -j DNAT --to-destination 127.0.0.1:53

Here tcpdump of me, pinging ya.ru:

[[email protected] ~]# tcpdump -nnSXvv -i any udp
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
18:34:15.974227 IP (tos 0x0, ttl 64, id 20601, offset 0, flags [DF], proto UDP (17), length 51)
    192.168.0.6.54387 > 127.0.0.1.53: [bad udp cksum 0x3fe0 -> 0x2c11!] 58964+ A? ya.ru. (23)
    <data here>
18:34:15.974457 IP (tos 0x0, ttl 64, id 26083, offset 0, flags [DF], proto UDP (17), length 604)
    192.168.0.6.56172 > 77.66.84.233.443: [bad udp cksum 0x6533 -> 0x48f0!] UDP, length 576
    <data here>
18:34:16.028783 IP (tos 0x0, ttl 57, id 7923, offset 0, flags [none], proto UDP (17), length 204)
    77.66.84.233.443 > 192.168.0.6.56172: [udp sum ok] UDP, length 176
    <data here>
18:34:16.029127 IP (tos 0x0, ttl 64, id 12032, offset 0, flags [DF], proto UDP (17), length 78)
    1.1.1.1.53 > 192.168.0.6.54387: [bad udp cksum 0xc2fb -> 0xc42e!] 58964 q: A? ya.ru. 1/0/1 ya.ru. A 87.250.250.242 ar: . OPT UDPsize=1252 (50)
    <data here>
18:34:16.075993 IP (tos 0x0, ttl 64, id 20662, offset 0, flags [DF], proto UDP (17), length 73)
    192.168.0.6.38972 > 127.0.0.1.53: [bad udp cksum 0x3ff6 -> 0x7be2!] 32011+ PTR? 242.250.250.87.in-addr.arpa. (45)
    <data here>
18:34:16.076448 IP (tos 0x0, ttl 64, id 26142, offset 0, flags [DF], proto UDP (17), length 604)
    192.168.0.6.53415 > 77.66.84.233.443: [bad udp cksum 0x6533 -> 0xf1ca!] UDP, length 576
    <data here>
18:34:16.186718 IP (tos 0x0, ttl 57, id 7930, offset 0, flags [none], proto UDP (17), length 268)
    77.66.84.233.443 > 192.168.0.6.53415: [udp sum ok] UDP, length 240
    <data here>
18:34:16.187096 IP (tos 0x0, ttl 64, id 12081, offset 0, flags [DF], proto UDP (17), length 103)
    1.1.1.1.53 > 192.168.0.6.38972: [bad udp cksum 0xc314 -> 0xfa7e!] 32011 q: PTR? 242.250.250.87.in-addr.arpa. 1/0/1 242.250.250.87.in-addr.arpa. PTR ya.ru. ar: . OPT UDPsize=1252 (75)
    <data here>

1) process with local port 54387 sends request and it is being redirected to 127.0.0.1:53 as expected
2) local custom DNS service sends request to remote DNS server
3) remote server responds back
4) ??? the original process with local port 54387 receives response from 1.1.1.1:53 ???
5+) repeat for PTR

What and why happened here and how can this be avoided without editing resolv? It wouldnt be a big issue overall but the DNS queries from custom service are encrypted while response from 1.1.1.1:53 isnt.

For reference, current resolv.conf updated via DHCP lease:

1.1.1.1
8.8.8.8

I don’t have the ability to edit it because original question was intended for Android 9+, which simply does not have a place where you can just set DNS and be happee. I managed to repro the same situation on my CentOS8 setup so the question should applicable.

The request in point 1) above is seen AFTER DNAT in one of the prerouting chains took place. This is why you see destination address of 127.0.0.1 instead of the ORIGNAL destination address of 1.1.1.1. This packets and up at your resolver and then you seem to proceed with DNS over HTTPS.

When the response back from your local resolver is sent back to the requestor it ALSO goes through NAT. This NAT will ensure the SOURCE address of the response is the ORIGINAL DESTINATION address of the request. This is why you see 1.1.1.1 as the source address in point 4) above.

Basically NAT is taking place at two places:

  1. It modifies the OUTGOING packet according to the rule you supplied.
  2. It also modifies the INCOMING packet to revert the modification above and to ensure return packet is properly matched to correct socket at the “originating” node.
Tagged : / / /

Server Bug Fix: identifying vlan packets using tcpdump

Original Source Link

I’m trying to figure out the vlan tagged packets that my host receives or sends to other hosts.
I tried

tcpdump -i eth1 vlan 0x0070

But it didnt work. Has anyone tried to view the vlan packets through tcpdump before?
Couldn’t find much help searching the web!

I think you’re using the wrong tool to be honest – tcpdump is more tied to IP (L3) whereas VLANs are a feature of L2 – try using wireshark instead.

If your host is connected to an access port, the switch will likely strip the VLAN tag off before it reaches your host. As a result, running TCPDump on the host in question will never see the VLAN tags.

You would need to setup a SPAN port and/or introduce a network tap into your network somewhere to grab traffic before the tags are dropped off the packets in order to see them in a network dump/trace.

you can actually use linux to “decode” 802.1q (vlan tagging). you can effectively turn linux into a “router on a stick” and route between vlans, with a single ethernet port, on a fancy cisco layer 2 switch (that has lots of vlans).

the main ethernet has “subinterfaces” which correscpond with the vlan id. you can then route, and iptables (firewall) the subinterfaces individually.

this is an easy way to have a perimeter linux firewall connected to isp, and 10 vlans behind it, but only using a single ethernet interface.

dot q is the standard, even tho cisco thinks it made it up, so it runs great on linux.

EDIT: to enable this

modprobe 8021q

you can then run tcpdump to listen on the subinterfaces

Tagged : / /

Linux HowTo: have tcpdump show the ‘conversation’ as i would get from wireshark?

Original Source Link

Is it possible to get the same output i would get from:

tcpdump port 80 -w log.pcap
wireshark log.pcap

in wireshark Analyze > Follow TCP Stream and then in the bottom drop down Entire Conversation

but without using wireshark? ideally using tcpdump or other widely available tool (netcat?) in the console.

Newer versions of TShark should support “-z follow” for this:

   −z follow,prot,mode,filter[,range]
       Displays the contents of a TCP or UDP stream between two nodes.
       The data sent by the second node is prefixed with a tab to
       differentiate it from the data sent by the first node.

       prot specifies the transport protocol.  It can be one of:
           tcp   TCP
           udp   UDP
           ssl   SSL

       mode specifies the output mode.  It can be one of:
           ascii ASCII output with dots for non‐printable characters
           hex   Hexadecimal and ASCII data with offsets
           raw   Hexadecimal data

       Since the output in ascii mode may contain newlines, the length
       of each section of output plus a newline precedes each section
       of output.

       filter specifies the stream to be displayed.  UDP streams are
       selected with IP address plus port pairs.  TCP streams are
       selected with either the stream index or IP address plus port
       pairs.  For example:
           ip−addr0:port0,ip−addr1:port1
           tcp‐stream‐index

       range optionally specifies which "chunks" of the stream should
       be displayed.

       Example: −z "follow,tcp,hex,1" will display the contents of the
       first TCP stream in "hex" format.

     ===================================================================
     Follow: tcp,hex
     Filter: tcp.stream eq 1
     Node 0: 200.57.7.197:32891
     Node 1: 200.57.7.198:2906
     00000000  00 00 00 22 00 00 00 07  00 0a 85 02 07 e9 00 02  ...".... ........
     00000010  07 e9 06 0f 00 0d 00 04  00 00 00 01 00 03 00 06  ........ ........
     00000020  1f 00 06 04 00 00                     ......
         00000000  00 01 00 00                   ....
         00000026  00 02 00 00

       Example: −z
       "follow,tcp,ascii,200.57.7.197:32891,200.57.7.198:2906" will
       display the contents of a TCP stream between 200.57.7.197 port
       32891 and 200.57.7.98 port 2906.

     ===================================================================
     Follow: tcp,ascii
     Filter: (ommitted for readability)
     Node 0: 200.57.7.197:32891
     Node 1: 200.57.7.198:2906
     38
     ...".....
     ................
         4
         ....

So, although tcpdump can’t do this for you, newer versions of TShark can do so, and TShark is a tty-mode (what the youngsters call “console-mode” :-)) program.

Tagged : / / /

Making Game: have tcpdump show the ‘conversation’ as i would get from wireshark?

Original Source Link

Is it possible to get the same output i would get from:

tcpdump port 80 -w log.pcap
wireshark log.pcap

in wireshark Analyze > Follow TCP Stream and then in the bottom drop down Entire Conversation

but without using wireshark? ideally using tcpdump or other widely available tool (netcat?) in the console.

Newer versions of TShark should support “-z follow” for this:

   −z follow,prot,mode,filter[,range]
       Displays the contents of a TCP or UDP stream between two nodes.
       The data sent by the second node is prefixed with a tab to
       differentiate it from the data sent by the first node.

       prot specifies the transport protocol.  It can be one of:
           tcp   TCP
           udp   UDP
           ssl   SSL

       mode specifies the output mode.  It can be one of:
           ascii ASCII output with dots for non‐printable characters
           hex   Hexadecimal and ASCII data with offsets
           raw   Hexadecimal data

       Since the output in ascii mode may contain newlines, the length
       of each section of output plus a newline precedes each section
       of output.

       filter specifies the stream to be displayed.  UDP streams are
       selected with IP address plus port pairs.  TCP streams are
       selected with either the stream index or IP address plus port
       pairs.  For example:
           ip−addr0:port0,ip−addr1:port1
           tcp‐stream‐index

       range optionally specifies which "chunks" of the stream should
       be displayed.

       Example: −z "follow,tcp,hex,1" will display the contents of the
       first TCP stream in "hex" format.

     ===================================================================
     Follow: tcp,hex
     Filter: tcp.stream eq 1
     Node 0: 200.57.7.197:32891
     Node 1: 200.57.7.198:2906
     00000000  00 00 00 22 00 00 00 07  00 0a 85 02 07 e9 00 02  ...".... ........
     00000010  07 e9 06 0f 00 0d 00 04  00 00 00 01 00 03 00 06  ........ ........
     00000020  1f 00 06 04 00 00                     ......
         00000000  00 01 00 00                   ....
         00000026  00 02 00 00

       Example: −z
       "follow,tcp,ascii,200.57.7.197:32891,200.57.7.198:2906" will
       display the contents of a TCP stream between 200.57.7.197 port
       32891 and 200.57.7.98 port 2906.

     ===================================================================
     Follow: tcp,ascii
     Filter: (ommitted for readability)
     Node 0: 200.57.7.197:32891
     Node 1: 200.57.7.198:2906
     38
     ...".....
     ................
         4
         ....

So, although tcpdump can’t do this for you, newer versions of TShark can do so, and TShark is a tty-mode (what the youngsters call “console-mode” :-)) program.

Tagged : / / /

Server Bug Fix: Trying to capture 3-way handshake in linux

Original Source Link

I am trying to capture 3-way handshake using the following in linux, it does not seems to work…

sudo tcpdump "tcp[tcpflags] & (tcp-syn) !=0" -w filename.pcap -i eth0

Could you someone suggest me an edit to it??

Might be tough capture only the three way handshake. So a traditional three way handshake would be a SYN packet, a SYN/ACK packet to acknowledge the original SYN, and then an ACK to acknowledge that SYN/ACK. From a filtering standpoint, you probably could capture the second part of the three way handshake with a (tcp-syn&tcp-ack), but that third ACK would be hard to filter out from all the other normal ACK packets sent during a TCP conversation.

In this case, I would probably just do a far simplier filter to capture the whole conversation (maybe avoid the whole packet to keep size in line), and then just use wireshark (or tcpdump itself) to see the three way handshake. So something like

sudo tcpdump -s 32 -w filename.pcap -i eth0

Tagged : /

Server Bug Fix: TCPDUMP capture new connections only

Original Source Link

I am using TCPDUMP to capture traffic from specific IP address.
Is there the possibility to capture new connections only, meaning TCP streams that start with SYN packet?

Thank you

To capture only TCP SYN packets:

# tcpdump -i <interface> "tcp[tcpflags] & (tcp-syn) != 0"

The following will capture both TCP-SYN and SYN-ACK packets.

tcpdump -i <interface> "tcp[tcpflags] & (tcp-syn) !=0"

The following will only capture TCP-SYN packets.

tcpdump -i <interface> "tcp[tcpflags] & (tcp-syn) !=0 and tcp[tcpflags] & (tcp-ack) =0"

The reason is, SYN-ACK packets include both the SYN and ACK flags. The first filter only looked for the presence of a SYN flag.

If you want to filter on inbound only, add the -Q in option.

tcpdump -i <interface> -Q in "tcp[tcpflags] & (tcp-syn) !=0 and tcp[tcpflags] & (tcp-ack) =0"

Tagged :

Server Bug Fix: How we have differentiate the packet capture based on the RX & TX

Original Source Link

I am working in the security industry. We have taken packet capture on the firewall multiple times.

  1. Can we take Pcap based on RX and & tx and force the tcpdump command to write the output in two direct files based on TX and RX?
  2. At the same time, if the traffic has not reached on the TX and getting the drop on TX level, can we write this output in a different file (drop)?

I have gone through couple of documents and articles but was not able to integrate this in one command.

In addition I have tried this command

tcpdump -nnei any -Q in -w /var/log/rx.pcap -C 50 -W2 & -Q out -w /var/log/tx.pcap

but it saves only RX not TX

Tagged : / / / /

Server Bug Fix: Proxy service only responding on some IP addresses (others suck on SYN_RECV)

Original Source Link

I have a server using a proxy service (WAF etc) which forwards packets to my server.

I can see established SSL conenctions from all proxy netstat -an and the rest stuck in SYN_RECV:

tcp        0      0 192.168.102.11:443      185.93.230.20:64966     SYN_RECV
tcp        0      0 192.168.102.11:443      192.88.135.20:8306      SYN_RECV
tcp        0      0 192.168.102.11:443      66.248.202.20:10750     SYN_RECV
tcp        0      0 192.168.102.11:443      185.93.230.20:2213      SYN_RECV
tcp        0      0 192.168.102.11:443      66.248.202.20:7494      SYN_RECV
tcp        0      0 192.168.102.11:443      185.93.231.20:32752     ESTABLISHED
tcp        0      0 192.168.102.11:443      185.93.231.20:31910     ESTABLISHED

I can see traffic hit tcpdump port 443 and '(tcp-syn|tcp-ack)!=0' -nn:

For 185.93.231.20.2139

20:36:35.263777 IP 192.168.102.11.443 > 185.93.231.20.2139: Flags [FP.], seq 203642186:203642217, ack 1968471817, win 258, options [nop,nop,TS val 32827456 ecr 876705214], length 31
20:36:36.901357 IP 192.168.102.11.443 > 185.93.231.20.2137: Flags [P.], seq 418165034:418165065, ack 2875697257, win 258, options [nop,nop,TS val 32829093 ecr 876704135], length 31

For 185.93.230.20

20:36:49.098560 IP 185.93.230.20.20721 > 192.168.102.11.443: Flags [S], seq 2855805773, win 29200, options [mss 1460,sackOK,TS val 882921029 ecr 0,nop,wscale 9], length 0
20:36:49.098638 IP 192.168.102.11.443 > 185.93.230.20.20721: Flags [S.], seq 268496949, ack 2855805774, win 28960, options [mss 1460,sackOK,TS val 32841290 ecr 882921029,nop,wscale 7], length 0

For 66.248.202.20:

20:37:02.042048 IP 66.248.202.20.49557 > 192.168.102.11.443: Flags [S], seq 3837436386, win 29200, options [mss 1460,sackOK,TS val 791596242 ecr 0,nop,wscale 9], length 0
20:37:02.042116 IP 192.168.102.11.443 > 66.248.202.20.49557: Flags [S.], seq 2339555392, ack 3837436387, win 28960, options [mss 1460,sackOK,TS val 32854234 ecr 791596242,nop,wscale 7], length 0

For 192.88.135.20:

20:36:39.595087 IP 185.93.228.20.23354 > 192.168.102.11.443: Flags [S], seq 1334433323, win 29200, options [mss 1460,sackOK,TS val 274977072 ecr 0,nop,wscale 9], length 0
20:36:39.595120 IP 192.168.102.11.443 > 185.93.228.20.23354: Flags [S.], seq 1203016390, ack 1334433324, win 28960, options [mss 1460,sackOK,TS val 32831787 ecr 274970056,nop,wscale 7], length 

But only traffic from 185.93.231.20 is getting logged in domlogs:

185.93.231.20 - - [22/May/2020:19:55:37 +0400] "GET /blog/video-gallery/ HTTP/1.1" 200 12716 "https://www.example.com/blog/publications/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0" 747893
185.93.231.20 - - [22/May/2020:19:55:39 +0400] "GET /wp-content/uploads/2020/02/Thumbnail72.jpg HTTP/1.1" 200 181941 "https://www.example.com/blog/video-gallery/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0" 1283052
185.93.231.20 - - [22/May/2020:19:55:39 +0400] "GET /wp-content/uploads/2020/02/Thumbnail68.jpg HTTP/1.1" 200 180934 "https://www.example.com/blog/video-gallery/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0" 952373

Any ideas as to what to check next? I have disabled all firewall rules and ensured NAT is correctly working between WAN and host (inbound and out) – no config changes happened this just stopped working.

This turned out to be an asymmetrical routing issue, in that packets could reach the server but the network was not able to return them (due to a routing failure).

netstat with partial connections stuck in SYN_RECV and tcpdump with its flags being returned:

  • [S] inbound from remote server to us to SYN establish a connection
  • [S.] reply from us to remote server to SYN+ACK establish connection
    request

This was identified on the server with the below SYN request from the remote server:

20:36:49.098560 IP 185.93.230.20.20721 > 192.168.102.11.443: Flags [S], seq 2855805773, win 29200, options [mss 1460,sackOK,TS val 882921029 ecr 0,nop,wscale 9], length 0

Which put the socket into a half-open state:

tcp        0      0 192.168.102.11:443      185.93.230.20:64966     SYN_RECV

And then we respond (note the flags S. meaning SYN+ACK):

20:36:49.098638 IP 192.168.102.11.443 > 185.93.230.20.20721: Flags [S.], seq 268496949, ack 2855805774, win 28960, options [mss 1460,sackOK,TS val 32841290 ecr 882921029,nop,wscale 7], length 0

But this never reaches the remote server, so it never in turn responds with further packets that would complete the handshake and set the socket to ESTABLISHED.

This was resolved by the ISP and associated counter parties to resolve routing issues.

This could have also indicated firewalls dropping packets or misconfigured NAT rules, these were ruled out prior to reaching out to ISP.

Tagged : / / /