<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:14pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>Dear Hans,</p>
<p><br>
</p>
<p>many thanks! And in particular for all the detailed explanations.</p>
<p><br>
</p>
<p>For the VULOM "sleep 1" did not do the trick, but "sleep 10" worked. Is there any chance to ask the VULOM if it feels ready to do the job, instead of using a random waiting time?</p>
<p><br>
</p>
<p>Also I noticed that when aksing the VULOM which firmware it is using, we get a slightly different reply than the actual firmware number:</p>
<p><span></p>
<div><span style="font-size:8pt">RIO4-MCAL-1 mbsdaq > vulomflash --addr=3 --read</span><br>
<span style="font-size:8pt">VULOM base address: 0x03000000</span><br>
<span style="font-size:8pt">hwmap_mapvme.c:398: LOG: Virtual address for VULOM/TRIDI @ VME 0x03000000 is 0x3005e000.</span><br>
<span style="font-size:8pt">Performing command 'read'...</span><br>
<span style="font-size:8pt; color:rgb(255,0,0)"><b>VOLUM+0 => 0x14091f20</b></span><br>
<span style="font-size:8pt">VOLUM+RANGE_REG(0x800000) => 0x0000006a</span><br>
<span style="font-size:8pt">Released vme ptr.</span></div>
</span>But the actual firmware number is <b><span style="color:rgb(255,0,0)">1409285e</span></b>.<br>
<p></p>
<p>For comparison should one look only at the first four hex numbers? Or is there more to take into account?<br>
</p>
<p><br>
</p>
<p>For the V560 module, misusing the bitmask for the counter resolved the issue. At the end of this mail, I attach the new log. Maybe you find something notable, but to me it looks fine now.</p>
<p><br>
</p>
<p>Our next steps would be as follows:</p>
<p><br>
</p>
<p>1) Wait for you to implement the bugfixes of the last days into NURDLIB.</p>
<p>2) Setting up the test system with the most recent version of NURDLIB and checking, if our minimal system with VULOM and V560 is now running smoothly.</p>
<p>3) Hammering the V767 TDC into NURDLIB.</p>
<p>4) Once we have achieved this, we would go back to testing the SIS3316 modules.<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>Best greetings</p>
<p>Günter<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p></p>
<div><span style="font-size:8pt">10: lwroc_hostname_util.c:108: Host 'lyserv' known as 192.168.1.1 (port: 56583).</span><br>
<span style="font-size:8pt">Thread has no error buffer yet...</span><br>
<span style="font-size:8pt">CPUS: 1</span><br>
<span style="font-size:8pt">delay: 1</span><br>
<span style="font-size:8pt">10: lwroc_hostname_util.c:108: Host 'lyserv' known as 192.168.1.1 (port: 56583).</span><br>
<span style="font-size:8pt">Thread has no error buffer yet...</span><br>
<span style="font-size:8pt">HOST: RIO4-MCAL-1</span><br>
<span style="font-size:8pt">Token: d6d68d7b (d6d68d7b:d6d68d7b) [/mbsusr/mbsdaq/.drasi_tokens/mcal]</span><br>
<span style="font-size:8pt">10: lwroc_hostname_util.c:457: Own address: 192.168.1.71/255.255.255.0 (eth1).</span><br>
<span style="font-size:8pt">10: lwroc_data_pipe.c:145: Data buffer READOUT_PIPE, size 419430400 = 0x19000000, 1 consumers.</span><br>
<span style="font-size:8pt">10: lwroc_triva_readout.c:66: Silence TRIVA (HALT)</span><br>
<span style="font-size:8pt">10: lwroc_net_io.c:167: Started server on port 56583 (data port 39534).</span><br>
<span style="font-size:8pt">client union size: 244 208 188 508 640 204 204 => 640</span><br>
<span style="font-size:8pt">10: lwroc_udp_awaken_hints.c:159: UDP awaken hints file: /tmp/drasi.u1001/drasi.hints.u1001.RIO4-MCAL-1:56583</span><br>
<span style="font-size:8pt">10: lwroc_main.c:706: Log message rate limit not in effect.</span><br>
<span style="font-size:8pt">10: lwroc_readout.c:112: call readout_init...</span><br>
<span style="font-size:8pt">10: lwroc_thread_util.c:117: This is the triva control thread!</span><br>
<span style="font-size:8pt">10: lwroc_thread_util.c:117: This is the net io thread!</span><br>
<span style="font-size:8pt">10: lwroc_thread_util.c:117: This is the slow_async thread!</span><br>
<span style="font-size:8pt">10: lwroc_thread_util.c:117: This is the data server thread!</span><br>
<span style="font-size:8pt">8: lwroc_message_wait.c:86: Waited 1 seconds for msg client.</span><br>
<span style="font-size:8pt">8: lwroc_triva_state.c:414: Waited 1 seconds for initial slave and EB connection(s):</span><br>
<span style="font-size:8pt">8: lwroc_triva_state.c:422: [EB lyserv] (state 0)</span><br>
<span style="font-size:8pt">10: lwroc_message_internal.c:472: Message client connected!</span><br>
<span style="font-size:8pt">10: lwroc_net_trans.c:1156: [drasi] Transport client connected (data) [192.168.1.1].</span><br>
<span style="font-size:8pt">10: lwroc_triva_control.c:370: Setup TRIVA (DISBUS, HALT, MASTER, RESET)</span><br>
<span style="font-size:8pt">10: lwroc_triva_control.c:418: Minimum event time ctime(5000)+1*rd(686)+3*wr(634)+fctime(1000)=8588 ns (116.442 kHz)</span><br>
<span style="font-size:8pt">10: lwroc_triva_state.c:1486: (Re)send ident messages...</span><br>
<span style="font-size:8pt">10: lwroc_triva_control.c:495: START TEST ACQ: HALT, CLEAR=RESET, MT=1</span><br>
<span style="font-size:8pt">9: lwroc_triva_control.c:507: TEST: GO</span><br>
<span style="font-size:8pt">10: lwroc_triva_control.c:725: RUN: RESET</span><br>
<span style="font-size:8pt">10: lwroc_triva_control.c:729: RUN: MT=14</span><br>
<span style="font-size:8pt">9: lwroc_triva_control.c:737: GO (1 good test triggers done) (max 116.4 kHz)</span><br>
<span style="font-size:8pt">10: lwroc_triva_readout.c:376: Trigger 14 seen.</span><br>
<span style="font-size:8pt">10: config/config.c:181: Will try default cfg path='/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default', can be set with NURDLIB_DEF_PATH.</span><br>
<span style="font-size:8pt">8: lwroc_triva_state.c:2399: Master: deadtime: 1. Status: 0x10 (IN_READOUT). EC: 1</span><br>
<span style="font-size:8pt">10: lwroc_triva_state.c:2428: [EB lyserv] EB: Status: 0x0.</span><br>
<span style="font-size:8pt">8: lwroc_triva_state.c:2488: Node(s) busy in readout, waiting...</span><br>
<span style="font-size:8pt">10: config/parser.c:287: Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/global.cfg' {</span><br>
<span style="font-size:8pt">10: config/parser.c:299: Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/global.cfg' }</span><br>
<span style="font-size:8pt">10: config/parser.c:287: Opened './main.cfg' {</span><br>
<span style="font-size:8pt">10: config/config.c:1299: .Global log level=debug.</span><br>
<span style="font-size:8pt">10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/crate.cfg' {</span><br>
<span style="font-size:8pt">10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/crate.cfg' }</span><br>
<span style="font-size:8pt">10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/gsi_vulom.cfg' {</span><br>
<span style="font-size:8pt">10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/gsi_vulom.cfg' }</span><br>
<span style="font-size:8pt">10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' {</span><br>
<span style="font-size:8pt">10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' }</span><br>
<span style="font-size:8pt">10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/caen_v560.cfg' {</span><br>
<span style="font-size:8pt">10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/caen_v560.cfg' }</span><br>
<span style="font-size:8pt">10: config/parser.c:287: .Opened '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' {</span><br>
<span style="font-size:8pt">10: config/parser.c:299: .Closed '/LynxOS/mbsusr/mbsdaq/mbsrun/rio4/2024_mcalstruck/nurdlib/cfg/default/module_log_level.cfg' }</span><br>
<span style="font-size:8pt">10: config/parser.c:299: Closed './main.cfg' }</span><br>
<span style="font-size:8pt">10: crate/crate.c:348: crate_create {</span><br>
<span style="font-size:8pt">10: crate/crate.c:674: crate_create(MCAL) }</span><br>
<span style="font-size:8pt">10: crate/crate.c:900: crate_init(MCAL) {</span><br>
<span style="font-size:8pt">10: crate/crate.c:924: .Slow-init module[0]=GSI_VULOM.</span><br>
<span style="font-size:8pt">LOG: TRLO: MD5SUM: 0x1409285e (CT: 63bb1d44 = 2023-01-08 19:45:08 UTC)</span><br>
<span style="font-size:8pt">10: crate/crate.c:924: .Slow-init module[1]=CAEN_V560.</span><br>
<span style="font-size:8pt">10: module/map/map.c:224: ...rd(0x33333300+0xfa/16)=963ns wr(0x33333300+0x50/16)=713ns.</span><br>
<span style="font-size:8pt">10: crate/crate.c:977: .Fast-init module[0]=GSI_VULOM.</span><br>
<span style="font-size:8pt">10: crate/crate.c:977: .Fast-init module[1]=CAEN_V560.</span><br>
<span style="font-size:8pt">10: crate/crate.c:1074: crate_init(MCAL) }</span><br>
<span style="font-size:8pt">10: ctrl/ctrl.c:788: Control server online.</span><br>
<span style="font-size:8pt">Thread has no error buffer yet...</span><br>
<span style="font-size:8pt">10: f_user.c:559: WR ID=0x200.</span><br>
<span style="font-size:8pt">10: f_user.c:565: TS offset unset. Will not modify stamp.</span><br>
<span style="font-size:8pt">10: f_user.c:572: TPAT: No.</span><br>
<span style="font-size:8pt">10: f_user.c:573: Sync-check: No.</span><br>
<span style="font-size:8pt">10: f_user.c:575: Spill triggers: No.</span><br>
<span style="font-size:8pt">10: f_user.c:576: LMU: No.</span><br>
<span style="font-size:8pt">10: f_user.c:577: Timer latches: No.</span><br>
<span style="font-size:8pt">10: f_user.c:578: Spill shape: No.</span><br>
<span style="font-size:8pt">10: f_user.c:579: Micro-structure: No.</span><br>
<span style="font-size:8pt">10: f_user.c:581: Multi-event flag: No.</span><br>
<span style="font-size:8pt">10: f_user.c:586: UDP destination: None.</span><br>
<span style="font-size:8pt">GSI_VULOM: diff_module: 0, module_crate_counter: 0, module_event_counter: 0, module_this_minus_crate: 0</span><br>
<span style="font-size:8pt">CAEN_V560: diff_module: 0, module_crate_counter: 0, module_event_counter: 0, module_this_minus_crate: 0</span></div>
...<br>
<p></p>
<p><br>
</p>
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>Von:</b> Hans Toshihide Törnqvist <hans.tornqvist@chalmers.se><br>
<b>Gesendet:</b> Donnerstag, 22. Februar 2024 15:26:15<br>
<b>An:</b> Weber, Guenter Dr.; Hċkan T Johansson<br>
<b>Betreff:</b> Re: AW: [subexp-daq] Report of a possible bug of the CAEN_V560 module</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Dear Günter,<br>
<br>
I have some ideas after your two e-mails, logs and config files are <br>
really useful :) I'll shorten the log for my comments. for the tl;dr <br>
version, just skip to the bottom code change suggestion.<br>
<br>
On 2024-02-22 14:21, Weber, Guenter Dr. wrote:<br>
> Dear Hans,<br>
> <br>
> here is the output of the DRASI log from the test system. I am using the <br>
> most recent (or almost most recent) versions of the various software <br>
> packages from GITLAB. The system currently only has a VULOM and a V560 <br>
> module.<br>
> <br>
> I added some comments to the output. To me it looks, the VULOM has some <br>
> problems at the beginning. And, separate from the VULOM issue, the math <br>
> of the difference in counters does not work out for the V560 module.<br>
> <br>
> The VULOM issue I did not notice before. So, maybe by adding a lot of <br>
> output lines into crate.c and then removing them I have broken <br>
> something. But it also possible that before I simply overlooked these <br>
> error message as in the end it looks like the DAQ is working fine.<br>
> <br>
> Best greetings<br>
> <br>
> Günter<br>
> <br>
> 10: f_user.c:559: WR ID=0x200.<br>
> 10: f_user.c:565: TS offset unset. Will not modify stamp.<br>
> 10: f_user.c:572: TPAT: No.<br>
> 10: f_user.c:573: Sync-check: No.<br>
> 10: f_user.c:575: Spill triggers: No.<br>
> 10: f_user.c:576: LMU: No.<br>
> 10: f_user.c:577: Timer latches: No.<br>
> 10: f_user.c:578: Spill shape: No.<br>
> 10: f_user.c:579: Micro-structure: No.<br>
> 10: f_user.c:581: Multi-event flag: No.<br>
> 10: f_user.c:586: UDP destination: None.<br>
> ***** looks like the VULOM has a problem. I did not notice this before *****<br>
> 5: module/gsi_vulom/gsi_vulom.c:12: .Serial timestamp receiver has had <br>
> sync failure, status=0x000a8000.<br>
> 5: module/gsi_vulom/gsi_vulom.c:12: .Serial timestamp receiver not <br>
> synced, status=0x000a8000.<br>
> 5: crate/crate.c:1250: .MCAL[0]=GSI_VULOM: readout_dt failed = 0x00000004.<br>
> 5: crate/crate.c:1322: .MCAL[0]=GSI_VULOM: Event counter: <br>
> crate=0x00000000/32, this-crate=0x00000000, module=0x00000000/31 <br>
> diff=0xdeadbeef, shadow=0x00000000/31 diff=0xdeadbeef.<br>
<br>
The sync could be bad if the DAQ starts too fast after setting up the <br>
timestamp source. Try "sleep 1" after setting up the vulom so the <br>
timestamp receiver in the vulom can latch onto its input, even if it's <br>
wired internally.<br>
<br>
> ***** here we see the counter mismatch - I did add a 250 ms delay to <br>
> crate.c, before it does readout_dt again to avoid having thousands of <br>
> output lines here******<br>
> *<br>
> CAEN_V560: diff_module: 4294967295, module_crate_counter: 0, <br>
> module_event_counter: 1, module_this_minus_crate: 0<br>
> CAEN_V560: diff_module: 4294967294, module_crate_counter: 0, <br>
> module_event_counter: 2, module_this_minus_crate: 0<br>
> CAEN_V560: diff_module: 4294967293, module_crate_counter: 0, <br>
> module_event_counter: 3, module_this_minus_crate: 0<br>
> CAEN_V560: diff_module: 4294967292, module_crate_counter: 0, <br>
> module_event_counter: 4, module_this_minus_crate: 0<br>
> ***** after four trials of readout_dt of V560, we reach the timeout of 1 <br>
> second*****<br>
> 5: crate/crate.c:1287: .MCAL[1]=CAEN_V560: readout_dt timeout.<br>
> 5: crate/crate.c:1322: .MCAL[1]=CAEN_V560: Event counter: <br>
> crate=0x00000000/32, this-crate=0x00000000, module=0x00000004/32 <br>
> diff=0xfffffffc, shadow=0x00000000/32 diff=0x00000000.<br>
<br>
This is an artifact of the soft counter in the v560, obviously we don't <br>
expect the module to have more accepted events while polling it, but the <br>
real problem comes a bit later.<br>
<br>
> 5: crate/crate.c:1394: .MCAL: readout_dt failed!<br>
> 5: crate/crate.c:1501: .MCAL: had problems, re-initializing.<br>
> 10: crate/crate.c:684: .crate_deinit(MCAL) {<br>
> 10: crate/crate.c:708: .crate_deinit(MCAL) }<br>
> 8: lwroc_triva_state.c:2028: Master TRIVA/MI no progress last second, <br>
> and in deadtime.<br>
> 8: lwroc_triva_state.c:2399: Master: deadtime: 1. Status: 0x10 <br>
> (IN_READOUT). EC: 2<br>
> 10: lwroc_triva_state.c:2428: [EB lyserv] EB: Status: 0x0.<br>
> 8: lwroc_triva_state.c:2488: Node(s) busy in readout, waiting...<br>
> 10: crate/crate.c:900: .crate_init(MCAL) {<br>
> 10: crate/crate.c:924: ..Slow-init module[0]=GSI_VULOM.<br>
> LOG: TRLO: MD5SUM: 0x1409285e (CT: 63bb1d44 = 2023-01-08 19:45:08 UTC)<br>
> 10: crate/crate.c:924: ..Slow-init module[1]=CAEN_V560.<br>
> 10: module/map/map.c:224: ....rd(0x33333300+0xfa/16)=963ns <br>
> wr(0x33333300+0x50/16)=713ns.<br>
> 10: crate/crate.c:977: ..Fast-init module[0]=GSI_VULOM.<br>
> 10: crate/crate.c:977: ..Fast-init module[1]=CAEN_V560.<br>
> 10: crate/crate.c:1074: .crate_init(MCAL) }<br>
> 5: f_user.c:1257: .had readout error, ret=0x14, trigger=14, prev=0<br>
<br>
This is the last log from the readout part, it says trigger 14 was <br>
handled. This trigger is always fired in MBS-like DAQ's before starting <br>
the event loop, but no master start is delivered to the modules. No <br>
physical event is associated with it, so we expect no counter increases <br>
and no event data.<br>
<br>
In general we read out "everything" for every trigger and rely on <br>
modules reporting their status/content properly. Modules like the v560 <br>
ruin this since we always check it for all events and the counter always <br>
increments, and it clearly shouldn't for trigger 14. So, in this case it <br>
really did test the incorrect software logic...<br>
<br>
I had a look in the v560 manual once more and only now did I realize <br>
that it is not trigger based, the scalers are only available on-the-fly. <br>
The event counter makes no sense then, so I will concede and suggest you <br>
set the module counter mask to 0.<br>
<br>
Have a look in module/gsi_vftx2/gsi_vftx2.c line 79, another module <br>
without an event-counter which skips the whole counting stuff:<br>
<br>
vftx2->module.event_counter.mask = 0;<br>
<br>
Put something similar in module/caen_v560/caen_v560.c line 41, and feel <br>
free to remove the increment in readout_dt.<br>
<br>
Hope the extra info isn't too verbose...<br>
<br>
Best regards,<br>
Hans<br>
</div>
</span></font>
</body>
</html>