[subexp-daq] lwrocmerge stream server problem coupled with Go4

Tue Jan 21 08:20:15 CET 2025

Dear Martin,

Long story short:

drasi sent 0 as the fourth member ('streams') of the 16-byte info header 
of transport and stream server data.  That 0 is interpreted by go4 
(eventapi) do treat the data differently somehow.  Continous events 
without blocks?

https://github.com/gsi-ee/go4/blob/master/MbsAPI/f_evt.c#L716
https://github.com/gsi-ee/go4/blob/master/MbsAPI/f_evt.c#L750

With a 1, it works for me.

---

Longer story; I keep it just since the netcat (nc) reading of transport 
server data may be handy.

I was at first able to 'halfway' reproduce an issue.  With go4 6.2.0 that 
I had at hand and the Go4ExampleSimple.

Reading data from a drasi transport server, go4 stops and refuses after 
what seems to be a first buffer (status bar tells 6210 events).  If I feed 
the same data as a file however, it looks like it is eating it all...

Reproduction:

Data generation (ucesb):

file_input/empty_file --lmd --caen-v775=7 | \
   empty/empty --file=- --server=trans:7777,hold

lwrocmerge:

bin/lwrocmerge --buf=size=100M \
   --log-no-start-wait
   --merge-mode=event --merge-no-validate \
   --trans=localhost:7777 --server=trans \
   --max-ev-size=0x40000

Connecting go4 to this gives an go4 analysis terminal error:

GO4-!> Analysis UserClient-subfy09.fy.chalmers.se-19025 ERROR:  from event source TGo4MbsTransport:localhost
GO4-*> AnalysisClient UserClient-subfy09.fy.chalmers.se-19025 has STOPPED analysis processing.

One can get the data sent through the transport this way:

nc localhost 6000 | dd bs=1M count=100 iflag=fullblock > trans1.lmd

Get rid of the first 16 bytes (transport info header):

dd if=trans1.lmd of=trans1a.lmd bs=16 skip=1

This go4 is willing to read.

...

After trying the same with ucesb, and some 'hexdump -C', the info header 
difference became suspicious...

---

Thanks for reporting this --- ehm --- anomaly!

Cheers,
Håkan

On Mon, 20 Jan 2025, Bajzek, Martin wrote:

> 
> Dear Joern,
> 
> 
> that could explain it. I'll have a brief look at the documentation later
> this week.
> 
> Thank you for investigating!
> 
> 
> Cheers,
> 
> -- Martin
> 
> ____________________________________________________________________________
> From: Adamczewski-Musch, Joern Dr.
> Sent: Monday, January 20, 2025 9:43:29 AM
> To: Bajzek, Martin
> Subject: Re: lwrocmerge stream server problem coupled with Go4  
> Dear Martin,
> 
> thank you for this request.
> 
>  From the error message in the Go4 debug output, it seems that the event
> that Go4 receives from your stream or transport server implementation
> does not like the event type in the header
> 
> (please see
> https://github.com/gsi-ee/go4/blob/c7bbf7a991e6794637f116ef2b3c26b6446ae1da
> /Go4EventServer/TGo4MbsSource.cxx#L231)
> 
> The TGo4MbsSource currently does accept MBS events of types 10 or 4 only.
> 
> This event type is set in the event header field s_ve10_1.i_type (in Go4
> the event header is passed to fxEvent by f_evt_get_event() of the
> underlying MBS event api).
> 
> In your case, Go4 finds 0 as i_type value and thus rejects it.
> 
> So either your drasi server does not fill the i_type correctly when
> sending the event buffer, or the first data in the received buffer do
> not contain an event structure, but something else that Go4 does not
> understand, as it always expects it as s_ve10_1.
> 
> Maybe this helps to debug the issue.
> 
> 
> Cheers,
> 
> Joern
> 
> 
> 
> 
> 
> On 1/17/25 16:53, Bajzek, Martin wrote:
> > Dear Jörn,
> >
> >
> > there's a small issue with Go4 (version 603-00), where if the Go4
> > analysis connects to a client (transport or stream),
> >
> > spawned from a drasi process, and not MBS, or UCESB/MREV, then the
> > following error happens:
> >
> >
> >  > GO4-*> Analysis loop is starting...
> >  > GO4-*> MBS source:: First event = 0 step = 0
> >  >
> >  > GO4-d>  !!! Mbs Source --  ERROR: Unknown event type 0 !!!
> >
> > In gdb I see only:
> >
> > Catchpoint 1 (exception thrown), 0x00007ffff72a90a1 in __cxa_throw ()
> > from /lib/x86_64-linux-gnu/libstdc++.so.6
> > (gdb) bt
> > #0  0x00007ffff72a90a1 in __cxa_throw () from
> > /lib/x86_64-linux-gnu/libstdc++.so.6
> > #1  0x00007ffff7ee4865 in TGo4MbsSource::BuildMbsEvent(TGo4MbsEvent*) ()
> > from /cvmfs/eel.gsi.de/debian12-x86_64/go4/603-00/lib/libGo4Analysis.so
> > #2  0x00007ffff7ee42c7 in TGo4MbsSource::BuildEvent(TGo4EventElement*)
> > () from /cvmfs/eel.gsi.de/debian12-x86_64/go4/603-00/lib/libGo4Analysis.so
> >
> > I'm forwarding Haakan's ideas, maybe you would know why is this a case?
> >
> > -- Martin
> >
> > ------------------------------------------------------------------------
> > *From:* Håkan T Johansson <f96hajo at chalmers.se>
> > *Sent:* Friday, January 17, 2025 3:50:48 PM
> > *To:* Bajzek, Martin
> > *Cc:* Klenze, Philipp; Hubbard, Nicolas James Dr.;
> > subexp-daq at lists.chalmers.se
> > *Subject:* Re: lwrocmerge stream server problem coupled with Go4
> >
> > Dear Martin,
> >
> > Does it work to connect the go4 to a transport server?
> >
> > (The protocols are very similar, the only on-wire difference is actually
> > that a stream server sends back a request to get a new block of data every
> > time.  but drasi does not care about that, it just sends data anyhow, and
> > eats whatever garbage (request) the consumer is sending to it).
> >
> > Both transport and stream protocols start by sendsing a 16-byte 'info'
> > header to the client.  This I have abused a bit to send some additional
> > information about port mapping.  This may actually be only in the stream
> > case and not in the transport case, since there the data transmission
> > starts without any further request, such that I cannot play that trick.)
> >
> > It may be that go4 does not like that abuse.  However, those port map
> > tricks were first done in ucesb, so should be the same.  So that is a bit
> > of a wonder then...
> >
> > Other test: can you use a plain go4 ('empty' equivalent) and see if that
> > is willing to eat the data from drasi?  If not, we have source so can
> > debug...
> >
> > Hmm, the drasi '--server' option has the 'forcemap' option, but that would
> > surely not help.  One could implement a 'nomap' ('noportmap') option to
> > turn these tricks off, if they turn out to be the issue.  That then leads
> > to the fun with 'port-in-use', but that's another story...
> >
> > What may then be nicer to use is (if transport protocol works) to just use
> > a transport server, but on another port.  And tell that --server to be
> > 'nohold'.  That is then the equivalent of a transport server in that it
> > does not hinder the DAQ if it does not keep up.
> >
> > I'm Cc'ing the subexp-daq at lists.chalmers.se also, as this info may be
> > useful for someone else.  (And also that this might be a point that could
> > use better documentation...  A small protocol description...)
> >
> > Cheers,
> > Håkan
> >
> >
> >
> > On Fri, 17 Jan 2025, Bajzek, Martin wrote:
> >
> >>
> >> Dear Haakan,
> >>
> >>
> >> there's an issue if a Go4 client connects to a stream server from drasi
> >> timeorderer (lwrocmerge executable).
> >>
> >> The client connects successfully but throws some error (can't debug with
> >> -ggdb3, as the build is not ours) on the very first event.
> >>
> >> Says that something is wrong with the event source.
> >>
> >>
> >> However, if I spawn] a UCESB intermediary:
> >>
> >> ./empty/empty stream://daq --server=stream
> >>
> >>
> >> Then everything works fine..
> >>
> >>
> >> Could this be that the LMD packets in lwrocmerge stream server are
> different
> >> than in the UCESB empty server?
> >>
> >> If not, what would be the suspect?
> >>
> >>
> >> Cheers,
> >>
> >> -- Martin
> >>
> >>
> >>
> 
> --
> /////////////////////////////////////////////////////////////////////
> // Dr. J"orn Adamczewski-Musch              (J.Adamczewski at gsi.de)
> // GO4 project team / data processing group Tel: +49-6159-71-1337
> // Experiment Electronics department (EEL)  FAX: +49-6159-71-2986
> ////////////////////////////////////////////////////////////////////
> // GSI Helmholtzzentrum f"ur Schwerionenforschung GmbH
> // Planckstraße 1
> // D-64291 Darmstadt, Germany
> // www.gsi.de
> //
> // Commercial Register / Handelsregister:
> //      Amtsgericht Darmstadt, HRB 1528
> // Managing Directors / Gesch"aftsf"uhrung:
> //      Prof. Dr. Thomas Nilsson
> //      Dr. Katharina Stummeyer
> //      J"org Blaurock
> // Chairman of the Supervisory Board / Vorsitzender des Aufsichtsrates:
> //      Ministerialdirigent Dr. Volkmar Dietz
> 
> 
>