Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example 3 - Cross Correlation Sample and Phase Coherent Check - Exception Failed to put IQ Samples into the Queue #7

Closed
anthfuller opened this issue Sep 13, 2022 · 50 comments

Comments

@anthfuller
Copy link

Receiving a numerous amount of these when running the example below:
Example 3 - Cross Correlation Sample and Phase Coherent Check

Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
Sending control message
Control interface communication finished
Reconfiguration succesfully finished
/usr/local/lib/python3.10/dist-packages/gnuradio/krakensdr/krakensdr_correlator.py:46: RuntimeWarning: divide by zero encountered in log10
x_corr_plot = 10*np.log10(np.abs(x_corr))
Failed to put IQ Samples into the Queue
Exception:

@anthfuller
Copy link
Author

I meant to mention, this isn't running on a Pi4, but a x86 machine with Ubuntu 22.04 running. I also used the automatic script, had trouble with the manual. This is also a fresh OS install.

@anthfuller
Copy link
Author

This is probably root cause of the issues:

"The problem is that taking the log10 of anything <= 0 is undefined mathematically, and Python is (quite rightly)
raising an exception.

I guess the KrakenSDR correlator block calls log10 underneath, and it doesn't take care to avoid the zone
of undefinedness.

When I've encountered this problem, I just add a tiny amount to my samples to make sure they're never
<= 0. This usually only happens at startup, when buffers may have been zero-padded in the beginning."

Is there a fix or workaround for this or should I look into the sample size?

@anthfuller
Copy link
Author

anthfuller commented Sep 13, 2022

I just built a simple KrakenSDR two-channel QT Gui Sink with the appropriate GRC blocks and I'm still getting the error:
/usr/local/lib/python3.10/dist-packages/gnuradio/krakensdr/krakensdr_correlator.py:46: RuntimeWarning: divide by zero encountered in log10
x_corr_plot = 10*np.log10(np.abs(x_corr))
Failed to put IQ Samples into the Queue
Exception:

I purchased this specifically because it was stated when Kraken solicited funds for this, that the built in RTL-SDRs were coherent! The wait was close to a year for this thing! I don't have the patience to build coherency into a product in my opinion should have had this.
krakenrf/krakensdr_docs#9

@krakenrf
Copy link
Owner

The RuntimeWarning isn't important, it's not the cause of the exception, its only a warning. Once valid data starts flowing in there will be no more zeros.

The problem seems to be "Failed to put IQ Samples into the Queue". Seems like the GNU Radio program isn't receiving samples.

I assume that you are running heimdall on the same machine? Is the Kraken powered with a good 2.4A+ supply?

@anthfuller
Copy link
Author

Hello, if IQ Samples are not going into the Queue, then the issue would most likely be the Heimdall software. I've created several OOT GRC blocks, and it seems like the Heimdall isn't running correctly. I'm running the Kraken off of 5.0V at 2.0 A

@anthfuller
Copy link
Author

Yes, Heimdall is running on the same machine. Tweak the shmem to eth. Even configured the noise source. I've blown the x86 device away twice, ran the KrakenSDR install script as well.

@krakenrf
Copy link
Owner

I'm running the Kraken off of 5.0V at 2.0 A

This could be insufficient. The Kraken draws about 2.1A under normal load, peaking up to about 2.2A when the noise source is active. So it's possible that the tuners start, but fail when they actually start pumping IQ as they can't get enough current.

If you don't have a better power supply around, you could try testing by reducing the receiver count num_ch in the heimdall Firmware/daq_chain_config.ini config file and in the GNU Radio source block.

@anthfuller
Copy link
Author

anthfuller commented Sep 13, 2022

I've noticed something, if I stop and start Heimdall (./heimdall_only_stop.sh) and start it back up, let the software Heimdall sit for 10 minutes, then the KrakenSDR will run simple GNU Radio block programs. Like this one, but if I start Heimdall and run GNURadio after even 15 seconds, the error messages come up

  • [https://wiki.gnuradio.org/index.php?title=Add_Const ]

@anthfuller
Copy link
Author

It's pretty late for me. I have to get up for work in 4 hours. I'll pickup a new power supply.

@krakenrf
Copy link
Owner

krakenrf commented Sep 13, 2022

I've noticed something, if I stop and start Heimdall (./heimdall_only_stop.sh) and start it back up, let the software Heimdall sit for 10 minutes, then the KrakenSDR will run simple GNU Radio block programs. Like this one, but if I start Heimdall and run GNURadio after even 15 seconds, the error messages come up

After 15s heimdall might still be calibrating. If you give it 1-2 minutes does it work?

If it's taking longer it could also just be a power issue causing sample drops resulting in calibration failing.

@anthfuller
Copy link
Author

anthfuller commented Sep 13, 2022

After all this time, yes just noticed if it sits for 10 minutes. Runs well
Screenshots attached. This is simple GNU radio blocks that were failing but seem now to be working.

@anthfuller
Copy link
Author

20220913_021643
One error about an array

@anthfuller
Copy link
Author

20220913_021613

@anthfuller
Copy link
Author

20220913_021607

@anthfuller
Copy link
Author

The sample rates, I've tried 32K, 1000k, 3.8m just to see what Heimdall would do

@anthfuller
Copy link
Author

20220913_022244
Last one. I'll try replacing the power supply tomorrow.

@anthfuller
Copy link
Author

Hello, power supply replaced and I'm still having the same issues. I even blew away the OS and reloaded everything from the install script. This is fourth time. The KrakenSDR runs the FFT example with exception on IQ Samples and does the same thing on the Correlation check. The Music DOA works fine. This is getting frustrating, is the KrakenSDR not stable? I need help with this, otherwise I need to return this product as not advertise for five channel coherency! The power charger is rated at 2.4 A and has other slots with ratings up to 3A. I'm using USB c cables, what is the issue now??

@anthfuller
Copy link
Author

No configurations in GNU Radio, your examples or my own GNU Radio blocks are without these "Exception: Failed to put IQ Samples into the Queue"

20220915_230625

@anthfuller
Copy link
Author

Is this KrakenSDR defective?

@krakenrf
Copy link
Owner

krakenrf commented Sep 16, 2022

What are you PC specs? This error means that buffer is full, meaning that the downstream DSP blocks weren't able to process data from the buffer fast enough. The DOA example is quite low in CPU complexity so that could be why you have no errors there.

If your application is tolerant to dropped samples it's safe to ignore.

If you're plotting to a sink the bandwidth / samp_rate should be fixed at 2.4 MHz as that is what heimdall is outputting by default. GNU Radio might be doing something strange and not sampling data fast enough if it's set wrongly, so that could be another reason for the buffer filling up.

@anthfuller
Copy link
Author

Where would I set the buffer, in the rtl-daq-c config file?

@anthfuller
Copy link
Author

Or would I do that in the GNU Radio blocks?

@anthfuller
Copy link
Author

Screenshot from 2022-09-16 01-17-25

Screenshot from 2022-09-16 01-21-35

@anthfuller
Copy link
Author

Screenshot from 2022-09-16 01-20-34

@anthfuller
Copy link
Author

My GNU radio blocks run, but some times the Exception errors come back. Where should I adjust the buffers, please?

@krakenrf
Copy link
Owner

krakenrf commented Sep 16, 2022

Change samp_rate from 2.56 to 2.4 MHz in your GNU Radio sink blocks.

The buffer this is error is referring to is the buffer in the KrakenSDR Sink block code, in the gr-krakensdr code. But I don't think you should adjust the buffers, if you increase them you're just adding a delay until the buffer fills up, which will eventually happen I assume if the sink samp_rate is greater than what heimdall is actually outputting.

@anthfuller
Copy link
Author

That does make sense! I'll test out later in the day.
Question, is the 2.4A the max for the KrakenSDR?
Currently I'm now running, 2.4Amps.

@krakenrf
Copy link
Owner

krakenrf commented Sep 16, 2022

Either that or the PC just isn't handling the output. On my 6th gen intel running a VM, it does show that warning and drop samples with 5 channels active. But with only two it's fine. For RA probably dropping a few samples is fine.

If you want to experiment with increasing the buffer size (maybe there's only bursts of slowdowns in your application), https://github.com/krakenrf/gr-krakensdr/blob/1d95d2f9b1923e63c9dec241b0dd7fd187617223/python/krakensdr/krakensdr_source.py#L62, change 10 to 100 or more maybe, then redo make, sudo make install from the gr-krakensdr readme.

Also from your screenshot I suspect your Steam to Vector and Mult Conj blocks are using a vector length of just 1? You probably want to use a larger vector for some processing gain and efficiency, try cpi_size or some power of 2 for num items in Stream to Vector, and Vector Length in Mult Conj. Then use a vector to stream block to connect to your GUI Sink.

Around 2.1A is the max.

@anthfuller
Copy link
Author

Either that or the PC just isn't handling the output. On my 6th gen intel running a VM, it does show that warning and drop samples with 5 channels active. But with only two it's fine. For RA probably dropping a few samples is fine.

If you want to experiment with increasing the buffer size (maybe there's only bursts of slowdowns in your application), https://github.com/krakenrf/gr-krakensdr/blob/1d95d2f9b1923e63c9dec241b0dd7fd187617223/python/krakensdr/krakensdr_source.py#L62, change 10 to 100 or more maybe, then redo make, sudo make install from the gr-krakensdr readme.

Also from your screenshot I suspect your Steam to Vector and Mult Conj blocks are using a vector length of just 1? You probably want to use a larger vector for some processing gain and efficiency, try cpi_size or some power of 2 for num items in Stream to Vector, and Vector Length in Mult Conj. Then use a vector to stream block to connect to your GUI Sink.

Around 2.1A is the max.

Sure, thank you! I'll give it a try tonight over tomorrow. Looking forward to testing this out and I really need this to work. I'm using the KrakenSDR for a Radio Astronomy Interferometer and it's not good if it doesn't work. USRB-210 are expensive but I'm contemplating on purchasing one, case the KrakenSDR proves unstable. Shame, also a year I had to wait.

@krakenrf
Copy link
Owner

I should clarify just in case: 2.1A is the max, if you're not using the bias tees. If you use the bias tee it's 2.1A + whatever your bias tee accessories need.

@anthfuller
Copy link
Author

I should clarify just in case: 2.1A is the max, if you're not using the bias tees. If you use the bias tee it's 2.1A + whatever your bias tee accessories need.

So will 2.1A work, 2.4A or 3A suffice?

@anthfuller
Copy link
Author

I did purchase a USRP-B210, three hours ago, in case this KrakenSDR is unstable. Please keep in mind, my test and the data I'm providing is also being pushed out to the Society of Amateur Radio Astronomers or SARA. A lot of members are being entertained. They are curious, if the Kraken's claims of your five channel, RTL-SDR is stabe, and in addition to your internal noise source. provides a Disciplined GPS system; truly coherent for a RAS interferometer.

@anthfuller
Copy link
Author

So, the issue could be the kraken itself! Maybe the USB cables, power supply, computer, stoage drive, PC speed, RAM, buffer or just poor architecture. I don't know, but well find out soon..... Good night

@krakenrf
Copy link
Owner

2.1A is the minimum, so any supply that has high capabilities will work.

I'm sorry to hear that you continue to have problems. What are the problems that you continue to have?

Can you please upload your GNU Radio interferometer flowgraph here so I can try it on my systems?

@anthfuller
Copy link
Author

Back to testing tonight.

@krakenrf
Copy link
Owner

One note that might not be in the docs:

If you're changing num_ch in the heimdall config file, make sure to change the length of en_bias_tee as well.

e.g.

num_ch = 2
en_bias_tee = 0,0

@krakenrf
Copy link
Owner

Also in one of your screenshots the center frequency was set to 1420508 MHz, which is well out of bounds. Remember the frequency is entered in MHz, so it should read as 1420.40575 MHz.

@krakenrf
Copy link
Owner

Quick test of a 2ch correlator, with noise source forced on, vec_len 2**16.

You should see a peak in the center indicating coherence.

image

@anthfuller
Copy link
Author

Even following your configurations, I'm unable to add the variable "vec_len" to the "Multiply Conjugate" block. SO without the "vec_len" variable, I'm still getting the exceptions with an array issues. See screenshot. Perhaps, I should send the KrakenSDR back, maybe this device is defective?
Screenshot from 2022-09-21 12-52-06

@anthfuller
Copy link
Author

Here is my block, similar to yours, without the vec_len added to the Multiply Conjugate block
Screenshot from 2022-09-21 12-57-18

@anthfuller
Copy link
Author

Now look what happens when I modify the "Multiply Conjugate" with the variable "vec_len", the error on the GNU Radio "Stream to Vector" and Multiply Conjugate!

Screenshot from 2022-09-21 12-58-25

@anthfuller
Copy link
Author

I am getting the peak, with out modifying the Multiply Conjugate, but with those array 4096 errors or warnings and exception for IQ samples not being sent to GNU Radio, sigh.

Screenshot from 2022-09-21 13-01-27

@anthfuller
Copy link
Author

Takes about 15 seconds, then the errors show up, and this is just with the Multiply Conjugate vector set to "1". If I add in your variable of vec_len 2**14 it GNU radio won't run!
Screenshot from 2022-09-21 13-02-15

@anthfuller
Copy link
Author

Any ideas?

@anthfuller
Copy link
Author

Used a stop watch, errors or exceptions start at 13.63 seconds, after running the test.

@anthfuller
Copy link
Author

What is the variable CPI_Size used for?

@anthfuller
Copy link
Author

anthfuller commented Sep 22, 2022

Ok, so far the KrakenSDR is stable when I perform the 2 channel correlator test without the variable => vec_len. Starts up runs fine. Ran the KrakenSDR for several hours. Then stopped it, closed everything down and restarted all software and it ran fine. I may get the exceptions issue once, but if I stop the Heimdall TCP socket connection and restart it things are fine. In fact after maybe one incidence it never comes back.

I'm wondering if it's the mini PC I'm running Ubuntu 22.04 on? Below is the link.
https://www.amazon.com/gp/product/B099MT9ZRR/ref=ppx_yo_dt_b_asin_title_o05_s00?ie=UTF8&psc=1

@anthfuller
Copy link
Author

anthfuller commented Sep 22, 2022

I've decided to aquire another PC, 16 GB of RAM, 128G SSD with 1TB HDD with 2x USB 2.0, 4x USB 3.0, Core I3 4005U (CPU) processor, etc. That should do it, plus I plan on running a few other programs, when I'm not running an interferometer setup.

@krakenrf
Copy link
Owner

Sorry was away a couple of days. Try this .grc file. The problem you're having with vec_len is that you need to propagate the vector length through your Stream to Vector and Vector to Stream blocks as well.

kraken_multconj.zip

cpi_size is the natural vector length (a block of samples) coming out of heimdall. GNU Radio slices this block of samples up into a stream based on the way the scheduler works. But then you can recombine into any vector length desired downstream with Stream to Vector blocks.

I'm not too familiar with Interferometry in practice, but I think you will want to process data based on some larger vector length in order to get some processing gain?

The mini PC is a little underpowered, but I think it's OK for two channels. In my experience though GNU Radio can be quite bad at performance when plotting to a graph sink. The actual DSP is fast, but plotting is bad.

@anthfuller
Copy link
Author

The two channel correlator is working fine! The issues is not the KrakenSDR but the under powered mini PC which runs fine testing two channels on the KrakenSDR.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants