[subexp-daq] How to start a DAQ system after NURDLIB, TRLOII, DRASI, etc. were updated

Thu Jan 18 11:29:39 CET 2024

Dear friends,

I now started the DAQ and it looks like there is a problem with the VETAR2 module. Attached please find the output once the DAQ is started on the RIO4.

Here ist the error message:

11: a:1: .Slow module[1]=GSI_VETAR. (crate/crate.c:866)
11: a:1: .Gsi Etherbone init_slow { (module/gsi_etherbone/gsi_etherbone.c:110)
5: a:1: ..Gsi Etherbone Could not open /sys/class/vetar/vetar0/dactl: No such file or directory.
 (module/gsi_etherbone/gsi_etherbone.c:120)
5: a:1: ..Calling exit(EXIT_FAILURE)... (module/gsi_etherbone/gsi_etherbone.c:120)

How to proceed?

Many thanks and best greetings
Günter

________________________________
Von: subexp-daq <subexp-daq-bounces at lists.chalmers.se> im Auftrag von Håkan T Johansson <f96hajo at chalmers.se>
Gesendet: Dienstag, 16. Januar 2024 19:20:54
An: Discuss use of Nurdlib, TRLO II, drasi and UCESB.
Betreff: Re: [subexp-daq] How to start a DAQ system after NURDLIB, TRLOII, DRASI, etc. were updated

On Tue, 16 Jan 2024, Weber, Guenter Dr. wrote:

>
> Dear friends,
>
>
> I am just trying to figure out how all the various scripts/commands work
> together for setting up our DAQ system. Maybe, you can help to clearify some
> things.
>
>
>
> - master.bash

This would be intended to run on the readout board (RIO).  The other
scripts below is for the PC.

>
> source ../env/env.sh
> ./trloii_setup.sh
> # gdb --args \
> ../r3bfuser/build_cc_ppc-linux_4.2.2_debug_drasi/m_read_meb.drasi \
>         --label=MCAL1 \
>         --triva=master,fctime=10,ctime=50 \
>         --log-no-rate-limit \
>         --server=drasi,dest=lyserv \
>         --buf=size=400Mi \
>         --max-ev-size=0x1000000 \
>         --subev=crate=1,type=20,subtype=2,control=9,procid=1 \
>         --eb=lyserv \
>         "$@"
>
> In the first line some environment variables are set. The second line tells
> the VULOM4 how to operate (i. e. selection from modes of operation defined
> in vulom.trlo). Then DRASI is started with a bunch of parameters. Is there
> somewhere an explanation of the purpose of all these settings?

The drasi command line options should be described here:

http://fy.chalmers.se/~f96hajo/drasi/doc/drasi_cmdline.html

(Note: 'lyserv' is the name of your PC, at least as seen from the RIO.)

The TRLO II setup file (vulom.trlo) is worse in terms of documentation.
The format as such is hopefully rather straightforward.  A description is
here: http://fy.chalmers.se/~f96hajo/trloii/man5/trloconf.5.html .
The actual meaning is more complex, the TRLO II description can be found
here:

http://fy.chalmers.se/~f96hajo/trloii/vulom4_trlo/descr_defs_frame.html

(left pane), which just is a colourised version of this:
http://fy.chalmers.se/~f96hajo/trloii/description.txt

It does not describe the .trlo file as such, but what the gateware tries
to achieve.

> And what is in the last line? I never came accross "$@" before.

That would be all the (non-shifted, i.e. not removed) options given to the
script itself, i.e. master.bash.

> - eb.bash
>
> ../drasi/bin/lwrocmerge \
>
>     --label=MCAL_EB \
>     --merge-mode=event \
>     --server=trans,flush=1 \
>     --server=stream,flush=1 \
>     --buf=size=1600Mi \
>     --max-ev-size=20Mi \
>     --eb-master=rio4-mcal-1 \
>     --file-writer \
>     --drasi=rio4-mcal-1
>
> What is the purpose of this command?

This runs an 'event builder' on the PC.  Not strictly needed, but this
way, the RIO only ever needs to send data once, to this event-builder
(EB).  File writing, and streaming of on-line data is then handled by this
process.  It is cheaper to have better network cards and so on in a plain
PC.

> Why are values for buffer size and max
> event size different then in the command before?

The nomenclaturure is unfortunately a bit convoluted below with ucesb
regards 'buffer size'.  The --buf in the two commands above give the size
of the memory used to buffer data inside the process, and depends on how
much memory each machine has.  The --max-ev-size tells how large each .lmd
event can be.  The --max-ev-size configured in the event builder needs to
be as large as the sum of all event sources it has.  I.e. should be able
to be the same as in your single source above.

In the readout itself, any event that is read out cannot be larger than
this.  If it is, then the DAQ will report a fatal failure.

> - fanout.bash
>
> #!/bin/bash
> while :
> do
> ~/mbsrun/rio4/mcalstruck/ucesb/empty/empty \
>     stream://localhost \
>     --server=stream:8001,bufsize=10Mi,flush=1,dataport:7001
> sleep 5
> done
>
> This looks like setting up a connection point to the outside world, right?

Yes.  It accepts connections and will serve them data for online analysis.

stream://localhost  tells it where to read data (here the PC itself where
it is running).  Using the 'stream' protocol, which esentially is just raw
LMD events.  (As is the 'transport protocol'.)  That will be served on the
default port used for the stream server set up by the EB above.

--server is then the server that accepts connections, and serves data on
another port (8001).

> And we find the third (random?) value for a buffer size.

That bufsize is actually the LMD buffer size used by the stream server.
The corresponding value in the readout and EB is set implicitly from the
maximum event size...

> - rate.bash
>
> ../drasi/bin/lwrocmon --rate rio4-mcal-1
>
> Ok, this I understand. We can look the DAQ over the shoulder and see what
> the thing is doing in terms of number of trigger events, transported data,
> etc.

> - log.bash
>
> ../drasi/bin/lwrocmon --log rio4-mcal-1 localhost
>
> This is also easy to understand.

:-)

Then also try to do (on the PC)

drasi/bin/lwrocmon --tree rio4-mcal-1 localhost

which would show a 'tree-view' monitoring of the running system.

Cheers,
Håkan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.chalmers.se/pipermail/subexp-daq/attachments/20240118/b94a3f21/attachment-0001.html>
-------------- next part --------------
RIO4-MCAL-1 mbsdaq > ./master.bash
hwmap_mapvme.c:240: LOG: Virtual address for TRLO II @ VME 0x03000000 is 0x3005e000.
LOG: TRLO: MD5SUM: 0x426cb99c (CT: 5bba6507 = 2018-10-07 19:56:55 UTC)
Clear module setup.  
Loading config file 'vulom.trlo'.
Executing 'standalone'.
Executing 'module_trigger'.
10: lwroc_hostname_util.c:105: Host 'lyserv' known as 192.168.1.1 (port: 56583).
Thread has no error buffer yet...
CPUS: 1
delay: 1
10: lwroc_hostname_util.c:105: Host 'lyserv' known as 192.168.1.1 (port: 56583).
Thread has no error buffer yet...
HOST: RIO4-MCAL-1
Token: d6d68d7b (d6d68d7b:d6d68d7b) [/mbsusr/mbsdaq/.drasi_tokens/mcal]
10: lwroc_data_pipe.c:122: Data buffer READOUT_PIPE, size 209715200 = 0x0c800000, 1 consumers.
11: lwroc_triva_readout.c:53: TRIVA: master,fctime=10,ctime=50
11: lwroc_triva_readout.c:210: TRIVA mode: master=1
11: lwroc_triva_readout.c:222: TRIVA VME map (addr=0x02000000)
11: hwmap_mapvme.c:240: Virtual address for TRIVA @ VME 0x02000000 is 0x30025000.
11: lwroc_triva_readout.c:264: TRIVA off 0x00 = 00001f20
10: lwroc_triva_readout.c:66: Silence TRIVA  (HALT)
10: lwroc_net_io.c:164: Started server on port 56583 (data port 46671).
client union size: 244 192 184 508 640 3048 204 => 3048
10: lwroc_readout.c:100: call readout_init...
10: lwroc_thread_util.c:105: This is the triva control thread!
10: lwroc_thread_util.c:105: This is the net io thread!
10: lwroc_thread_util.c:105: This is the data server thread!
10: lwroc_net_trans.c:1073: [drasi] Transport client connected (data) [192.168.1.1].
network client -> data network thread
using blocking writes
10: lwroc_message_internal.c:418: Message client connected!
11: lwroc_triva_readout.c:606: Readout thread waiting for inject request...
11: lwroc_triva_control.c:350: TRIVA control: startup (wait for init request).
move to data server thread...
11: lwroc_net_message.c:192: [msgclnt: 192.168.1.1] Connected.
10: lwroc_triva_control.c:370: Setup TRIVA  (DISBUS, HALT, MASTER, RESET)
11: lwroc_triva_control.c:397: Set FCAtime to 10 and Ctime to 50...
10: lwroc_triva_control.c:418: Minimum event time ctime(5000)+1*rd(690)+3*wr(633)+fctime(1000)=8589 ns (116.428 kHz)
11: lwroc_triva_control.c:420: Clearing TRIVA... (HALT, DIS_IRQ, CLEAR=RESET)
10: lwroc_triva_state.c:1412: (Re)send ident messages...
11: lwroc_triva_readout.c:628: Readout thread injecting ID message!
11: lwroc_triva_control.c:478: TRIVA control: initialised (wait for test request).
10: lwroc_triva_control.c:495: START TEST ACQ: HALT, CLEAR=RESET, MT=1
9: lwroc_triva_control.c:507: TEST: GO
11: lwroc_triva_control.c:522: status = 0x00008021
11: lwroc_triva_control.c:670: TRIVA control: initialised (going for run).
11: lwroc_triva_control.c:679: status = 0x00000021
10: lwroc_triva_control.c:700: RUN: RESET
10: lwroc_triva_control.c:704: RUN: MT=14
9: lwroc_triva_control.c:712:   GO (1 good test triggers done) (max 116.4 kHz)
11: lwroc_triva_readout.c:683: Readout thread loop active!
10: lwroc_triva_readout.c:369: Trigger 14 seen.
11: lwroc_triva_control.c:745: TRIVA control: running...
10: a:1: Global log level=verbose. (config/config.c:1292)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/crate.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/crate.cfg'. (config/parser.c:298)
8: lwroc_triva_state.c:2322: Master: deadtime: 1.  Status: 0x10 (IN_READOUT).  EC: 1
10: lwroc_triva_state.c:2351: [EB lyserv] EB: Status: 0x0.
8: lwroc_triva_state.c:2411: Node(s) busy in readout, waiting...
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/gsi_vulom.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/gsi_vulom.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/gsi_vetar.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/gsi_vetar.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/sis_3316.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/sis_3316.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/sis_3316.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/sis_3316.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/sis_3316.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/sis_3316.cfg'. (config/parser.c:298)
11: a:1: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:286)
11: a:1: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2021_mcalstruck/nurdlib/cfg/default/module_log_level.cfg'. (config/parser.c:298)
11: a:1: Closed './main.cfg'. (config/parser.c:298)
10: a:1: crate_create { (crate/crate.c:301)
11: a:1: .Crate="MCAL". (crate/crate.c:316)
11: a:1: .Adaptive CVT=no. (crate/crate.c:321)
11: a:1: .Shadow readout disabled. (crate/crate.c:332)
11: a:1: .Early deadtime release=no. (crate/crate.c:337)
11: a:1: .event_max override=0. (crate/crate.c:342)
11: a:1: .Re-init sleep=1s. (crate/crate.c:347)
11: a:1: .Gsi Siderem crate_create { (module/gsi_siderem/gsi_siderem.c:51)
11: a:1: .Gsi Siderem crate_create } (module/gsi_siderem/gsi_siderem.c:55)
11: a:1: .Gsi Tacquila crate_create { (module/gsi_tacquila/gsi_tacquila.c:103)
11: a:1: .Gsi Tacquila crate_create } (module/gsi_tacquila/gsi_tacquila.c:106)
11: a:1: .Pnpi Cros3 crate_create { (module/pnpi_cros3/pnpi_cros3.c:216)
11: a:1: .Pnpi Cros3 crate_create } (module/pnpi_cros3/pnpi_cros3.c:219)
11: a:1: .Gsi Vulom create { (module/gsi_vulom/gsi_vulom.c:5)
11: a:1: ..TRLO II create { (module/trloii/trloii.c:300)
11: a:1: ...Address=03000000. (module/trloii/trloii.c:303)
11: a:1: ...Has timestamps. (module/trloii/trloii.c:307)
11: a:1: ..TRLO II create } (module/trloii/trloii.c:329)
11: a:1: .Gsi Vulom create } (module/gsi_vulom/gsi_vulom.c:5)
11: a:1: .module[0]=GSI_VULOM. (crate/crate.c:468)
11: a:1: .Gsi Etherbone create { (module/gsi_etherbone/gsi_etherbone.c:72)
11: a:1: ..Mode=etherbone. (module/gsi_etherbone/gsi_etherbone.c:82)
11: a:1: ..Address=0x50000000. (module/gsi_etherbone/gsi_etherbone.c:87)
11: a:1: .Gsi Etherbone create } (module/gsi_etherbone/gsi_etherbone.c:90)
11: a:1: .module[1]=GSI_VETAR. (crate/crate.c:468)
11: a:1: .Gsi Vulom are_distinguishable. (module/gsi_vulom/gsi_vulom.c:5)
11: a:1: .sis3316 create { (module/sis_3316/sis_3316.c:225)
11: a:1: ..Address = 30000000 (module/sis_3316/sis_3316.c:236)
11: a:1: ..sis3316 get_config { (module/sis_3316/sis_3316.c:3106)
11: a:1: ...channels_to_read = mask 0x0000ffff (module/sis_3316/sis_3316.c:3116)
11: a:1: ...use_tau_correction = mask 0x00000000 (module/sis_3316/sis_3316.c:3128)
11: a:1: ...use_external_trigger = mask 0x0000ffff (module/sis_3316/sis_3316.c:3134)
11: a:1: ...use_internal_trigger = mask 0x00000000 (module/sis_3316/sis_3316.c:3140)
11: a:1: ...trigger_output = mask 0x0000ffff (module/sis_3316/sis_3316.c:3146)

... a few pages of SIS3316 stuff that seems to work fine ...

11: a:1: ...sample_length_maw_e[0] = 2048. (module/sis_3316/sis_3316.c:3594)
11: a:1: ...sample_length_maw_e[1] = 2048. (module/sis_3316/sis_3316.c:3594)
11: a:1: ...sample_length_maw_e[2] = 2048. (module/sis_3316/sis_3316.c:3594)
11: a:1: ...sample_length_maw_e[3] = 2048. (module/sis_3316/sis_3316.c:3594)
11: a:1: ...pretrigger_delay_maw_e[0] = 100. (module/sis_3316/sis_3316.c:3603)
11: a:1: ...pretrigger_delay_maw_e[1] = 100. (module/sis_3316/sis_3316.c:3603)
11: a:1: ...pretrigger_delay_maw_e[2] = 100. (module/sis_3316/sis_3316.c:3603)
11: a:1: ...pretrigger_delay_maw_e[3] = 100. (module/sis_3316/sis_3316.c:3603)
11: a:1: ...gate[0].delay = 0 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[0].width = 30 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ...gate[1].delay = 40 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[1].width = 30 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ...gate[2].delay = 40 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[2].width = 60 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ...gate[3].delay = 0 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[3].width = 100 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ...gate[4].delay = 0 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[4].width = 200 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ...gate[5].delay = 0 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[5].width = 500 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ...gate[6].delay = 0 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[6].width = 500 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ...gate[7].delay = 0 ns. (module/sis_3316/sis_3316.c:3621)
11: a:1: ...gate[7].width = 500 ns. (module/sis_3316/sis_3316.c:3629)
11: a:1: ..sis3316 get_config } (module/sis_3316/sis_3316.c:3636)
11: a:1: .sis3316 create } (module/sis_3316/sis_3316.c:247)
11: a:1: .module[4]=SIS_3316. (crate/crate.c:468)
11: a:1: .Counter=auto:Default. (crate/crate.c:572)
10: a:1: crate_create(MCAL) } (crate/crate.c:622)
10: a:1: crate_init(MCAL) { (crate/crate.c:841)
11: a:1: .map_setup { (module/map/map.c:183)
11: a:1: ..sicy_setup { (module/map/map_xpc_3310.c:82)
11: a:1: ...bridge_setup { (module/map/map_xpc_3310.c:52)
11: a:1: ...bridge_setup } (module/map/map_xpc_3310.c:60)
11: a:1: ..sicy_setup } (module/map/map_xpc_3310.c:84)
11: a:1: ..blt_setup { (module/map/map_xpc_3310.c:126)
11: a:1: ...bridge_setup { (module/map/map_xpc_3310.c:52)
11: a:1: ...bridge_setup } (module/map/map_xpc_3310.c:60)
11: a:1: ..blt_setup } (module/map/map_xpc_3310.c:136)
11: a:1: ..Broken BLT return val = yes. (module/map/map.c:186)
11: a:1: .map_setup } (module/map/map.c:193)
11: a:1: .Slow module[0]=GSI_VULOM. (crate/crate.c:866)
11: a:1: .Gsi Vulom init_slow { (module/gsi_vulom/gsi_vulom.c:5)
11: a:1: ..TRLO II init_slow { (module/trloii/trloii.c:404)
11: a:1: ...map_map(addr=0x03000000,size=0x00016000,mode=noblt,poke=0x0000000c/32) { (module/map/map.c:127)
11: a:1: ....map_poke { (module/map/map.c:171)
11: a:1: .....sicy_map(addr=0x0300000c,size=0x4) { (module/map/map.c:255)
11: a:1: .....sicy_map(mapped=0x3001f00c) } (module/map/map.c:267)
11: a:1: .....Poke reading 0x3001f00c... (module/map/map_sicy_common.c:38)
11: a:1: .....sicy_unmap { (module/map/map.c:275)
11: a:1: .....sicy_unmap } (module/map/map.c:277)
11: a:1: ....map_poke } (module/map/map.c:177)
11: a:1: ....sicy_map(addr=0x03000000,size=0x16000) { (module/map/map.c:255)
11: a:1: ....sicy_map(mapped=0x30036000) } (module/map/map.c:267)
11: a:1: ...map_map(mapped=0x30036000) } (module/map/map.c:156)
11: a:1: ..TRLO II init_slow } (module/trloii/trloii.c:442)
11: a:1: .Gsi Vulom init_slow } (module/gsi_vulom/gsi_vulom.c:5)
11: a:1: .Slow module[1]=GSI_VETAR. (crate/crate.c:866)
11: a:1: .Gsi Etherbone init_slow { (module/gsi_etherbone/gsi_etherbone.c:110)
5: a:1: ..Gsi Etherbone Could not open /sys/class/vetar/vetar0/dactl: No such file or directory.
 (module/gsi_etherbone/gsi_etherbone.c:120)
5: a:1: ..Calling exit(EXIT_FAILURE)... (module/gsi_etherbone/gsi_etherbone.c:120)
Performing hardware cleanup (TRIVA HALT, RESET)...