Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate and fix network protocol data read from connection #4496

Closed
tarunKoyalwar opened this issue Dec 17, 2023 · 1 comment
Closed
Labels
Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@tarunKoyalwar
Copy link
Member

Nuclei version:

Problem Statement

In golang when we do

b := make([]byte,1024)
n , err := conn.Read(b)

length or b is not necessarily 1024 as we expect this is due to how net.Conn and network I/O works .

Context

In golang network I/O is Non-Blocking stream based hence it returns as soon as there is some data available . Let's say if server has 500ms difference between two consecutive messages it sends then even if both of them can fit into buffer/ byte array b only 1st one is read . and there are also other factors here i.e

  • underlying buffers (os)
  • network congestion / latency
  • Non Blocking I/O design of golang

i.e why all stdlib implementations of a protocol use for loop to make sure we are reading xyz bytes of data so instead of above example below is preferred or used everywhere

func ReadFull(conn net.Conn, buf []byte) (int, error) {
    bytesRead := 0
    for bytesRead < len(buf) {
        n, err := conn.Read(buf[bytesRead:])
        if err != nil {
            return bytesRead, err
        }
        bytesRead += n
    }
    return bytesRead, nil
}

or something similar the idea is to read continuously in a loop until N bytes are read successfully

In Nuclei

  • In Nuclei we used to have just net.Conn.Read() (example 1) in network protocols so even when template says 1024 bytes it doesn't necessarily read 1024 bytes . this was a identified in javascript net module and fixed in nuclei v3.0.3 see Incomplete Data Received with Nuclei's JavaScript NetConn ('nuclei/net')  #4285
  • but it was identified recently with Read and Name attributes of network.Input() not working #4454 that there was a timeout for network protocols while reading and the reason was that in template we were trying to read 1024 bytes and only 256 bytes i returned by server and hence the connection timeout out after waiting for timeout [Note: protocols like ftp etc are dynamic length based protocols so we don't know upfront what the size of next response is but such protocols have a delimeter for example ftp has \r\n similar to other protocols] ( in earlier implementation it would just return whatever was available
  • currently we are resolving/resolved this by reverting to old behaviour and furthur investigation is needed to identify what would be the best behaviour and changes that need to be made
@tarunKoyalwar tarunKoyalwar added the Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Dec 17, 2023
@tarunKoyalwar
Copy link
Member Author

This is not a bug but optional pattern to implement but is limited to cases where we are 💯 sure that response length will be xyz bytes and we are willing to wait

@tarunKoyalwar tarunKoyalwar closed this as not planned Won't fix, can't repro, duplicate, stale Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Development

No branches or pull requests

1 participant