Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarking, packaging, and fix issues, tests and perfomance #797

Merged
merged 11 commits into from
Aug 11, 2020

Conversation

urbanishimwe
Copy link
Collaborator

@urbanishimwe urbanishimwe commented Jul 25, 2020

performance

  • handling of the very big packet(any size that can be buffered)
  • speeding up TCP sessions by using message hints: Added proto.HasFullPayload that helps to validate the entire HTTP request, it supports Chunked encoding too! Added proto.HasRequestTitle and proto.HasResponseTitle for validating the beginning of HTTP request. Those methods are used input_raw.go with TCP.
  • supports Keep-Alive: the above functions helps to support keep-alive

Packaging

  • capture: engines(capture/doc.go)
  • tcp: tcp message parser (tcp/doc.go)

benchmarking

  • capture.BenchmarkPcapDump: the benchmarks regarding dumping packets in a pcap file
  • capture.BenchmarkPcapFile: the benchmarks of reading packets from a pcap file
  • capture.BenchmarkPcap: the benchmarks of parsing packets from the loopback interface with pcap handles
  • proto.BenchmarkHasFullPayload:: benchmarking this function which validates the HTTP payload
  • tcp.BenchmarkPacketParseAndSort: benchmarks of parsing and sorting packets
  • tcp.BenchmarkMessageParserWithoutHint: benchmarks of message reasembling by using SYN and FIN flag
  • tcp.BenchmarkMessageParserWithHint: benchmarks of message reasembling by using proto.HasRequestTitle and proto.HasFullPayload flag

issues

see linked issues

tests

  • fixed input raw and engine tests

Most of the changed of the files, was about using functionalities of tcp and capture in existing functionalities

http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
input_raw.go Outdated Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
input_raw.go Outdated Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
http_modifier_settings.go Show resolved Hide resolved
@urbanishimwe urbanishimwe force-pushed the benchmarking-engine branch 2 times, most recently from 186ab87 to 16fbbea Compare August 2, 2020 22:01
@urbanishimwe urbanishimwe marked this pull request as ready for review August 2, 2020 22:05
@urbanishimwe urbanishimwe force-pushed the benchmarking-engine branch 2 times, most recently from 8bb1b2a to fc533bc Compare August 2, 2020 23:03
func TestSetInterfaces(t *testing.T) {
l := &Listener{}
l.host = "127.0.0.1"
l.setInterfaces()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for errors. This is at multiple places.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we handled the check at line 30

capture/capture_test.go Outdated Show resolved Hide resolved
emitter.go Show resolved Hide resolved
}
origin := &http.Server{
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}),
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ab"))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error check. Multiple places.

Copy link
Collaborator Author

@urbanishimwe urbanishimwe Aug 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I handled them with other functionalities to avoid unnecessarily allocation


func testPcapDumpEngine(f string, t *testing.T) {
defer os.Remove(f)
l, err := NewListener(f, 8000, "", EnginePcapFile, true)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check if err != nil

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's checked at the line 134

capture/capture.go Outdated Show resolved Hide resolved
input_raw.go Show resolved Hide resolved
tcp/tcp_test.go Outdated Show resolved Hide resolved
proto/proto_test.go Outdated Show resolved Hide resolved
@buger buger merged commit fdc8b09 into master Aug 11, 2020
@urbanishimwe urbanishimwe deleted the benchmarking-engine branch August 11, 2020 09:49
xingren23 referenced this pull request in xingren23/goreplay Sep 4, 2021
### performance
- handling of the very big packet(any size that can be buffered)
- speeding up TCP sessions by using message hints: Added **proto.HasFullPayload** that helps to validate the entire HTTP request, it supports `Chunked` encoding too! Added **proto.HasRequestTitle** and **proto.HasResponseTitle** for validating the beginning of HTTP request. Those methods are used `input_raw.go` with `TCP`.
- supports Keep-Alive: the above functions helps to support keep-alive

### Packaging
- **capture:** engines(capture/doc.go)
- **tcp:** tcp message parser (tcp/doc.go)

### benchmarking
- **capture.BenchmarkPcapDump:** the benchmarks regarding dumping packets in a pcap file
- **capture.BenchmarkPcapFile:** the benchmarks of reading packets from a pcap file
- **capture.BenchmarkPcap:** the benchmarks of parsing packets from the loopback interface with pcap handles
- **proto.BenchmarkHasFullPayload:**: benchmarking this function which validates the HTTP payload
- **tcp.BenchmarkPacketParseAndSort:** benchmarks of parsing and sorting packets
- **tcp.BenchmarkMessageParserWithoutHint:** benchmarks of message reasembling by using `SYN` and `FIN` flag
- **tcp.BenchmarkMessageParserWithHint:** benchmarks of message reasembling by using `proto.HasRequestTitle` and `proto.HasFullPayload` flag

### issues
see linked issues

###  tests
- fixed input raw and engine tests

**Most of the changed of the files, was about using functionalities of** `tcp` **and** `capture` **in existing functionalities**
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants