move eventlogs to an http endpoint

[whyrusleeping]

This PR makes eventlogs not get written to disk, instead, they are thrown away unless a client is connected to the API endpoint at /logs

License: MIT
Signed-off-by: Jeromy jeromyj@gmail.com

[jbenet]

clever

[cryptix]

Do these block after the client closes the request? if you get a write error to a closed HTTP connection you also don't see the error. Also, the MirrorWriter drops them as well.

[whyrusleeping]

@cryptix that was intended. I assume that the only error we will encounter is the http connection being closed. The first error encountered will drop the connection from the mirrorwriter, and then this will exit. I could add a default case to the send on the write notifier to ensure that misuse of the code doesnt cause a hang

[cryptix]

i see. Wasn't sure if there are are other errors. Sorry but I still don't see where the Writer is removed from the mirror.

[whyrusleeping]

the MirrorWriter loops over all writes on each write call, any writer that throws an error gets dropped from the list of writers.

[whyrusleeping]

or rather, any writer that doesnt throw an error is kept around for another write

[cryptix]

Aaah.. there it is! Glancing over it, I assumed to pushes to messages to the filter but it recreates the writer slice every time.. that seems wasteful to be honest.

[whyrusleeping]

yeah... the alternative was to keep an array of the items to be removed, wait til the write portion of the call was done, then rebuild the slice sans the dead writers. which was really ugly. I chose <32 bytes worth of wasted memory (on average) per write over ugly code.

[jbenet]

if worried about perf (which we may be, given that this is per event (and there are lots...), another option is:

// in Write()

// write to all writers, and nil out the broken ones.
for i, w := range mw.writers {
  _, err := w.Write(b)
  if err != nil {
    mw.writers[i] = nil
  }
}

// consolidate the slice
for i := 0; i < len(mw.writers); i++ {
  if mw.writers[i] != nil {
    continue
  }

  j := len(mw.writers)
  for  ; j > i; j-- {
    if mw.writers[j] != nil {
      mw.writers[i], mw.writers[j] = mw.writers[j], nil // swap
      break
    }
  }
  mw.writers = mw.writers[:j]
}

seem to work: http://play.golang.org/p/AV9UeZphgq

[whyrusleeping]

Yeah, that works. still a bit more complicated, but it may help.

[jbenet]

@whyrusleeping this LGTM. it should have a sharness test case though to make sure it gets mounted correctly and we don't break it.

[whyrusleeping]

@jbenet a test just to make sure theres an endpoint there? It would be rather hard to 'connect for two seconds' in a test, or something like that

[ghost]

The Log config code should be removed then, now or soon.

[whyrusleeping]

good point. I'll pull that out

[rht]

Nice, don't forget to close #866, #904, #868, #1224.

[cryptix]

invert if, log and continue?

[whyrusleeping]

I don't really want to log these errors, worst case, a client connection drops.

[whyrusleeping]

@jbenet whats with the swarm and notifications failures happening more often? I don't remember them popping up this much in the past.

[jbenet]

@whyrusleeping

@jbenet a test just to make sure theres an endpoint there? It would be rather hard to 'connect for two seconds' in a test, or something like that

Yeah. if we dont test these things we're bound to have regressions. we have to test everything.

Maybe can use the pollEndpoint binary that's already there. or maybe even curl | head -n5 | grep <expected>

@jbenet whats with the swarm and notifications failures happening more often? I don't remember them popping up this much in the past.

not sure, will take a look. IIRC, its timing related. so overloaded machines will fail more often.

[whyrusleeping]

@jbenet this is an odd failure: https://travis-ci.org/ipfs/go-ipfs/jobs/67385491

[jbenet]

@whyrusleeping yeah wat. it's totally nil. having hard time repro-ing it... looks like somewhere in the dialing both the conn and err are nil, so the flow doesn't go down. we can guard against it with: multiformats/go-multiaddr-net@b9c1a08

but we should probably get to the bottom of it someday.

[jbenet]

@whyrusleeping thanks LGTM