[subexp-daq] How to start a DAQ system after NURDLIB, TRLOII, DRASI, etc. were updated
Weber, Guenter Dr.
g.weber at hi-jena.gsi.de
Fri Jan 19 15:22:41 CET 2024
Dear Hans,
with the help of Håkan, I managed to get the VULOM running. So, we are back to starting the DRASI. There we encountered first the fact that in main.cfg a BARRIER is now required between the initialization of all modules (before there was only a single BARRIER command in the whole main.cfg.
Now, we are again encounter the VETAR error message. See below:
10: lwroc_hostname_util.c:108: Host 'lyserv' known as 192.168.1.1 (port: 56583).
Thread has no error buffer yet...
CPUS: 1
delay: 1
10: lwroc_hostname_util.c:108: Host 'lyserv' known as 192.168.1.1 (port: 56583).
Thread has no error buffer yet...
HOST: RIO4-MCAL-1
Token: d6d68d7b (d6d68d7b:d6d68d7b) [/mbsusr/mbsdaq/.drasi_tokens/mcal]
10: lwroc_hostname_util.c:457: Own address: 192.168.1.71/255.255.255.0 (eth1).
10: lwroc_data_pipe.c:145: Data buffer READOUT_PIPE, size 419430400 = 0x19000000, 1 consumers.
10: lwroc_triva_readout.c:66: Silence TRIVA (HALT)
10: lwroc_net_io.c:167: Started server on port 56583 (data port 50126).
client union size: 244 208 188 508 640 204 204 => 640
10: lwroc_udp_awaken_hints.c:159: UDP awaken hints file: /tmp/drasi.u1001/drasi.hints.u1001.RIO4-MCAL-1:56583
10: lwroc_main.c:706: Log message rate limit not in effect.
10: lwroc_readout.c:112: call readout_init...
10: lwroc_thread_util.c:117: This is the triva control thread!
10: lwroc_thread_util.c:117: This is the net io thread!
10: lwroc_thread_util.c:117: This is the slow_async thread!
10: lwroc_thread_util.c:117: This is the data server thread!
10: lwroc_message_internal.c:472: Message client connected!
10: lwroc_net_trans.c:1156: [drasi] Transport client connected (data) [192.168.1.1].
10: lwroc_triva_control.c:370: Setup TRIVA (DISBUS, HALT, MASTER, RESET)
10: lwroc_triva_control.c:418: Minimum event time ctime(5000)+1*rd(688)+3*wr(634)+fctime(1000)=8590 ns (116.414 kHz)
10: lwroc_triva_state.c:1486: (Re)send ident messages...
10: lwroc_triva_control.c:495: START TEST ACQ: HALT, CLEAR=RESET, MT=1
9: lwroc_triva_control.c:507: TEST: GO
10: lwroc_triva_control.c:725: RUN: RESET
10: lwroc_triva_control.c:729: RUN: MT=14
9: lwroc_triva_control.c:737: GO (1 good test triggers done) (max 116.4 kHz)
10: lwroc_triva_readout.c:376: Trigger 14 seen.
10: config/config.c:181: Will try default cfg path='/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default', can be set with NURDLIB_DEF_PATH.
10: config/parser.c:287: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/global.cfg' {
10: config/parser.c:299: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/global.cfg' }
8: lwroc_triva_state.c:2399: Master: deadtime: 1. Status: 0x10 (IN_READOUT). EC: 1
10: lwroc_triva_state.c:2428: [EB lyserv] EB: Status: 0x0.
8: lwroc_triva_state.c:2488: Node(s) busy in readout, waiting...
10: config/parser.c:287: Opened './main.cfg' {
10: config/config.c:1299: .Global log level=verbose.
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/crate.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/crate.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/gsi_vulom.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/gsi_vulom.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/gsi_vetar.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/gsi_vetar.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/sis_3316.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/sis_3316.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/sis_3316.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/sis_3316.cfg' }
10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' {
10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' }
10: config/parser.c:299: Closed './main.cfg' }
10: crate/crate.c:347: crate_create {
10: crate/crate.c:673: crate_create(MCAL) }
10: crate/crate.c:899: crate_init(MCAL) {
10: crate/crate.c:923: .Slow-init module[0]=GSI_VULOM.
LOG: TRLO: MD5SUM: 0x1409285e (CT: 63bb1d44 = 2023-01-08 19:45:08 UTC)
10: crate/crate.c:923: .Slow-init module[1]=GSI_VETAR.
5: module/gsi_etherbone/gsi_etherbone.c:431: ..Gsi Etherbone /sys/class/vetar/vetar0/dactl: Failed dactl fopen: No such file or directory.
5: module/gsi_etherbone/gsi_etherbone.c:431: ..Calling abort()...
We are sure that the right NURDLIB path is used and also DRASI is started in the correct folder.
What we found in the enironment variables are two lines, where 'white rabbit' is mentioned. As I understand, this is connected to the VETAR2 module.
LD_LIBRARY_PATH=/mbs/driv/white_rabbit/cherry/RIO4_Linux_OSV_2_6_33_SuHa/lib:/mbsusr/mbsdaq/mbsrun/whiterabbit/enigma/RIO4_Linux_OSV_2_6_33_SuHa/lib
PATH=/mbs/driv/white_rabbit/cherry/RIO4_Linux_OSV_2_6_33_SuHa/bin:/bin:/bin/ces:/usr/bin:/mbsusr/mbsdaq/bin:/usr/local/bin:/usr/bin/X11:/etc:/usr/etc:.:/mbsusr/mbsdaq/tools
Maybe, it is necessary to update something there?
Anyway, if you have now a switch in NURDLIB to deal with old software for the VETAR, then we should proceed with removing NURDLIB and getting the newer version, right?
Many thanks and have a nice weekend!
Best greetings
Günter
________________________________
Von: subexp-daq <subexp-daq-bounces at lists.chalmers.se> im Auftrag von Hans Toshihide Törnqvist <hans.tornqvist at chalmers.se>
Gesendet: Donnerstag, 18. Januar 2024 13:28:36
An: Discuss use of Nurdlib, TRLO II, drasi and UCESB.; Weber, Guenter Dr.
Betreff: Re: [subexp-daq] How to start a DAQ system after NURDLIB, TRLOII, DRASI, etc. were updated
Dear Günter,
It looks like either the Vetar driver failed, or the fw/driver is old
and nurdlib expects a file that didn't exist a while back. If it's the
latter I can add a switch to ignore this file, it's not critical for
readout and is used to enter a faster readout mode.
Which version of nurdlib is this? The files and line numbers don't match
with what I see in the source code, and the last time gsi_etherbone.c
changed was early last year.
The message "Could not open so-and-so" should be for the device, e.g.
/dev/pcie_wb0. This is used for the readout.
The dactl file should fail with "Failed dactl fopen".
To summarise, I would suggest to make sure that the correct nurdlib and
r3bfuser are executed by the DAQ scripts. Meanwhile I will prepare a
switch to ignore the dactl file which could anyhow be useful for old
configurations, and to clean up the Etherbone error messages.
Best regards,
Hans
On 2024-01-18 11:29, Weber, Guenter Dr. wrote:
> Dear friends,
>
>
> I now started the DAQ and it looks like there is a problem with the
> VETAR2 module. Attached please find the output once the DAQ is started
> on the RIO4.
>
>
> Here ist the error message:
>
>
> 11: a:1: .Slow module[1]=GSI_VETAR. (crate/crate.c:866)
> 11: a:1: .Gsi Etherbone init_slow {
> (module/gsi_etherbone/gsi_etherbone.c:110)
> 5: a:1: ..Gsi Etherbone Could not open /sys/class/vetar/vetar0/dactl: No
> such file or directory.
> (module/gsi_etherbone/gsi_etherbone.c:120)
> 5: a:1: ..Calling exit(EXIT_FAILURE)...
> (module/gsi_etherbone/gsi_etherbone.c:120)
>
> How to proceed?
>
>
>
> Many thanks and best greetings
> Günter
>
>
>
>
> ------------------------------------------------------------------------
> *Von:* subexp-daq <subexp-daq-bounces at lists.chalmers.se> im Auftrag von
> Håkan T Johansson <f96hajo at chalmers.se>
> *Gesendet:* Dienstag, 16. Januar 2024 19:20:54
> *An:* Discuss use of Nurdlib, TRLO II, drasi and UCESB.
> *Betreff:* Re: [subexp-daq] How to start a DAQ system after NURDLIB,
> TRLOII, DRASI, etc. were updated
>
> On Tue, 16 Jan 2024, Weber, Guenter Dr. wrote:
>
>>
>> Dear friends,
>>
>>
>> I am just trying to figure out how all the various scripts/commands work
>> together for setting up our DAQ system. Maybe, you can help to clearify some
>> things.
>>
>>
>>
>> - master.bash
>
> This would be intended to run on the readout board (RIO). The other
> scripts below is for the PC.
>
>>
>> source ../env/env.sh
>> ./trloii_setup.sh
>> # gdb --args \
>> ../r3bfuser/build_cc_ppc-linux_4.2.2_debug_drasi/m_read_meb.drasi \
>> --label=MCAL1 \
>> --triva=master,fctime=10,ctime=50 \
>> --log-no-rate-limit \
>> --server=drasi,dest=lyserv \
>> --buf=size=400Mi \
>> --max-ev-size=0x1000000 \
>> --subev=crate=1,type=20,subtype=2,control=9,procid=1 \
>> --eb=lyserv \
>> "$@"
>>
>> In the first line some environment variables are set. The second line tells
>> the VULOM4 how to operate (i. e. selection from modes of operation defined
>> in vulom.trlo). Then DRASI is started with a bunch of parameters. Is there
>> somewhere an explanation of the purpose of all these settings?
>
> The drasi command line options should be described here:
>
> http://fy.chalmers.se/~f96hajo/drasi/doc/drasi_cmdline.html
> <http://fy.chalmers.se/~f96hajo/drasi/doc/drasi_cmdline.html>
>
> (Note: 'lyserv' is the name of your PC, at least as seen from the RIO.)
>
> The TRLO II setup file (vulom.trlo) is worse in terms of documentation.
> The format as such is hopefully rather straightforward. A description is
> here: http://fy.chalmers.se/~f96hajo/trloii/man5/trloconf.5.html
> <http://fy.chalmers.se/~f96hajo/trloii/man5/trloconf.5.html> .
> The actual meaning is more complex, the TRLO II description can be found
> here:
>
> http://fy.chalmers.se/~f96hajo/trloii/vulom4_trlo/descr_defs_frame.html
> <http://fy.chalmers.se/~f96hajo/trloii/vulom4_trlo/descr_defs_frame.html>
>
> (left pane), which just is a colourised version of this:
> http://fy.chalmers.se/~f96hajo/trloii/description.txt
> <http://fy.chalmers.se/~f96hajo/trloii/description.txt>
>
> It does not describe the .trlo file as such, but what the gateware tries
> to achieve.
>
>> And what is in the last line? I never came accross "$@" before.
>
> That would be all the (non-shifted, i.e. not removed) options given to the
> script itself, i.e. master.bash.
>
>> - eb.bash
>>
>> ../drasi/bin/lwrocmerge \
>>
>> --label=MCAL_EB \
>> --merge-mode=event \
>> --server=trans,flush=1 \
>> --server=stream,flush=1 \
>> --buf=size=1600Mi \
>> --max-ev-size=20Mi \
>> --eb-master=rio4-mcal-1 \
>> --file-writer \
>> --drasi=rio4-mcal-1
>>
>> What is the purpose of this command?
>
> This runs an 'event builder' on the PC. Not strictly needed, but this
> way, the RIO only ever needs to send data once, to this event-builder
> (EB). File writing, and streaming of on-line data is then handled by this
> process. It is cheaper to have better network cards and so on in a plain
> PC.
>
>> Why are values for buffer size and max
>> event size different then in the command before?
>
> The nomenclaturure is unfortunately a bit convoluted below with ucesb
> regards 'buffer size'. The --buf in the two commands above give the size
> of the memory used to buffer data inside the process, and depends on how
> much memory each machine has. The --max-ev-size tells how large each .lmd
> event can be. The --max-ev-size configured in the event builder needs to
> be as large as the sum of all event sources it has. I.e. should be able
> to be the same as in your single source above.
>
> In the readout itself, any event that is read out cannot be larger than
> this. If it is, then the DAQ will report a fatal failure.
>
>> - fanout.bash
>>
>> #!/bin/bash
>> while :
>> do
>> ~/mbsrun/rio4/mcalstruck/ucesb/empty/empty \
>> stream://localhost \
>> --server=stream:8001,bufsize=10Mi,flush=1,dataport:7001
>> sleep 5
>> done
>>
>> This looks like setting up a connection point to the outside world, right?
>
> Yes. It accepts connections and will serve them data for online analysis.
>
> stream://localhost tells it where to read data (here the PC itself where
> it is running). Using the 'stream' protocol, which esentially is just raw
> LMD events. (As is the 'transport protocol'.) That will be served on the
> default port used for the stream server set up by the EB above.
>
> --server is then the server that accepts connections, and serves data on
> another port (8001).
>
>> And we find the third (random?) value for a buffer size.
>
> That bufsize is actually the LMD buffer size used by the stream server.
> The corresponding value in the readout and EB is set implicitly from the
> maximum event size...
>
>> - rate.bash
>>
>> ../drasi/bin/lwrocmon --rate rio4-mcal-1
>>
>> Ok, this I understand. We can look the DAQ over the shoulder and see what
>> the thing is doing in terms of number of trigger events, transported data,
>> etc.
>
>> - log.bash
>>
>> ../drasi/bin/lwrocmon --log rio4-mcal-1 localhost
>>
>> This is also easy to understand.
>
> :-)
>
> Then also try to do (on the PC)
>
> drasi/bin/lwrocmon --tree rio4-mcal-1 localhost
>
> which would show a 'tree-view' monitoring of the running system.
>
>
> Cheers,
> Håkan
>
--
subexp-daq mailing list
subexp-daq at lists.chalmers.se
https://lists.chalmers.se/mailman/listinfo/subexp-daq
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.chalmers.se/pipermail/subexp-daq/attachments/20240119/c594e2c2/attachment-0001.html>
More information about the subexp-daq
mailing list