How to make RF calibration behave like in core 2.4.2?

[TD-er]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: other
• Core Version: [latest git hash or date]
• Development Env: [Platformio]
• Operating System: [Windows]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [dio]
• Flash Size: [4MB]
• lwip Variant: [v1.4|v2 Lower Memory|Higher Bandwidth]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

I do have a module here which does work well with a build of ESPEasy of 2019/02/26, but not on later builds. (core 2.4.2)
Symptoms are:

• Able to connect to AP
• Most of the time it is able to receive an IP
• Hardly any IP traffic is possible (pings do not even get answered)
• Access point kicks off node after set timeout in AP (On a Mikrotik you can extend this, so I am sure it is the AP that kicks off the node)

After lots and lots of debugging I found out that the specific board seems to have an antenna which is not matched very well to the WiFi frequency.
If I hold the board between thumb and index finger on either the begin or end of the PCB antenna trace, the node suddenly starts answering to pings.
This means the resonance frequency of the antenna is too high.
N.B. the antenna also feels warm.

A user sent me this board, which is part of a larger batch which all show the same symptoms.

Normally I would toss this away as it is clearly something wrong in the antenna tuning.
However, on core 2.4.2 the board runs fine.
So I wonder if it is possible to fix this in software?

Apparently the RF calibration used in core 2.4.2 was able to adjust parameters a bit further to compensate for this compared to what later core revisions would allow.

While debugging, I noticed quite a few other issues which also may sound very familiar to anyone who has had issues with WiFi since core 2.5.0.
Using very sophisticated tools like my fingers, ping and live RSSI charts in the MikroTik web interface I was able to 'fine tune' the antenna of the board and noticed the following:

• RF calibration can correct the tuning if the antenna is somewhat in the acceptable range when starting. -> RSSI values are significantly higher on the AP, compared to when reconnecting with fingers on the antenna after boot.
• By slightly moving my fingers or changing pressure I could change responsiveness of the web UI and ping times.
• Reducing TX power does lower temperature of antenna, but also "tuning" using my fingers. (no crashes or other strange issues by the way caused by RF reflected signals back into the ESP)

These findings were very well reproducible, so I do feel confident it was not just a fluke.

This looks like a lot of ESP devices out there may have antennas which are not optimal and maybe just on the edge of what the RF calibration in core 2.5.0 and later can correct.
N.B. enclosures or presence of other materials close to the antenna can cause an antenna to shift its resonance frequency.

Is there a way to change the RF calibration to allow it to accept a larger range to correct an antenna?
It was working in core 2.4.2, so the ESP is capable of correcting for it.

One part of the code which does seem related is this:

Arduino/cores/esp8266/core_esp8266_phy.cpp

Lines 268 to 277 in 60fe7b4

	// freq_correct_en
	// bit[0]:0->do not correct frequency offset, 1->correct frequency offset.
	// bit[1]:0->bbpll is 168M, it can correct + and - frequency offset, 1->bbpll is 160M, it only can correct + frequency offset
	// bit[2]:0->auto measure frequency offset and correct it, 1->use 113 byte force_freq_offset to correct frequency offset.
	// 0: do not correct frequency offset.
	// 1: auto measure frequency offset and correct it, bbpll is 168M, it can correct + and - frequency offset.
	// 3: auto measure frequency offset and correct it, bbpll is 160M, it only can correct + frequency offset.
	// 5: use 113 byte force_freq_offset to correct frequency offset, bbpll is 168M, it can correct + and - frequency offset.
	// 7: use 113 byte force_freq_offset to correct frequency offset, bbpll is 160M , it only can correct + frequency offset.
	/*[112] =*/ 0,

Is this assumption correct that this may have to do with these symptoms?

[JAndrassy]

2.5.0 was the version with SDK pre-3 and many users reported problems with WiFi. 2.5.1 reverted to SDK 2.2.2.
core_esp8266_phy.cpp was changed from C to CPP in 2.5.0

[TD-er]

@JAndrassy Yep, the file was changed, but not its value.
What I meant with the suggested setting is, would it be possible to tell the RF calibration to correct for these symptoms by changing this byte to allow for using a frequency offset?

[earlephilhower]

Moved to a discussion since it's not really a bug and more of a discussion on how to hack the rf calibration bits...

[dalbert2]

Would you mind adding a link to where this discussion can be found? I think many folks have found the ESP8266 WiFi connectivity to be intermittent and this sounds like a discussion I would like to follow. Thank you!

[dalbert2]

Oh, I'm sorry, I see...this is the discussion; I misunderstood @earlephilhower 's message.

I've spent several days on ESP8266 connectivity problems recently:

OBSERVATIONS

1. The ESP8266 intermittently has great difficulty connecting to an AP; this is reported regularly in almost every ESP8266 forum and the problem appears to be persistent across resets, but does eventually clear (hence my interest in the RF calibration). I test with an application (Core 3.1.1, SDK 3.0.5, LWIP 2.1.3) that every 15 minutes: enters STA mode, connects to an AP, dumps some data via HTTPS, and then WIFI_OFF. The AP is dedicated (i.e. nothing else is using it). The ESP8266 is about 2' away and has a very strong RSSI (including when it can't connect to the AP). Everything works fine for many hours, but invariably, at some point, the ESP8266 will stop connecting to the AP, often for hours. In this state, the stack cyclically tries to connect and fails (STA disconnect: 2 or STA disconnect: 4). A wifi scan during this state will see the AP perfectly. Free heap in this state is >27K with the largest free block > 20K. When the ESP8266 cannot connect, a phone or laptop can connect to the AP with no problem. This condition is NOT cleared by a reset.

2. Adding a delay(250 - 1000) after each WiFi operation dramatically improves the overall WiFi performance (memory usage, incidences of poor connectivity) even though the main loop is littered with yield()s.

3. Changing the PHY mode (B->G or G->B) appears to correct or at least improve the connectivity issue temporarily (does this trigger an RF re-calibration?).

4. LWIP and/or Bear SSL hold onto heap memory too long, particularly when connections fail with any error. Failed connection attempts are retried every 30s in my case. Particularly with SSL connect timeouts/failures, all of the heap can be consumed before the memory is released.

QUESTIONS/SUGGESTIONS:

1. Is there a way to get LWIP and/or Bear SSL to release resources programmatically? Obviously one strategy is just to reset periodically or when the largest free heap block is too small, but it would be far better if there were a way to reset the state of the LWIP and Bear SSL stacks and/or force them to free allocated memory. I can't think of any reason they should be holding onto buffers once the WiFi mode is OFF.
2. Is there a way to force a the WiFi PHY to do an RF calibration?

Thanks for any insights or ideas for how to approach this!

[TD-er]

Yep, calling delay() with at least 50 - 100 msec right after each WiFi related call dramatically improves WiFi performance and stability.
All these symptoms you observed are very relatable, however I don't think those are related to RF calibration.

What I did observe was that some nodes were running perfectly well with older SDK (< 2.5, not 100% sure whether all 2.4.x versions were acting the same), but no longer able to connect using any newer SDK unless you firmly press your thumb on the antenna.
Also the antenna feels warm to the touch on those units.
This can be explained by the fact that pressing some body part to the antenna will de-tune the antenna and thus it will have a different resonance frequency.
Thus older SDK could compensate for this, but newer SDK can't. -> RF tuning

What I think is also happening on the newer SDKs is that the WiFi stack/code isn't completely cleared or initialized when you turn off the WiFi radio or get disconnected.
To support this hypothesis; I do see WiFi events which are clearly part of the previous connection or connection attempt when trying to make a new connection.
So what I do in ESPEasy is to turn off the WiFi radio and start over, when I do see events which I don't expect.
However this does make the WiFi code extremely complex :(

About the memory consumption by LWIP and WiFiClient based objects...
I think there is some queue of packets which have to be processed, which eventually get destructed when not handled within the timeout.
But I have no clue how to handle or flush these buffers when they are not part of the current connection.
I did add some code to destruct the WiFi object and recreate it, which did seem to help with this, but it doesn't seem to be a magic fix for everything.

Another issue with setting up a WiFi connection is that it is quite sensitive to being forced to handle traffic before the connection is fully established.
For example, when I immediately try to perform a NTP call (UDP traffic seems to differ in this case from TCP traffic) immediately after I get the event I got an IP-address, the unit stalls for tens of seconds.
If I wait a few-100 msec, or even better, delay(100) when making the first NTP call, it is handled almost immediately.
However, it might be that some left-over buffers in LWIP are blocking other replies to be handled.
So I am not 100% sure this is solely an Espressif SDK issue as it might be a combination with other parts of the code interacting with the network stack.

I also got the impression these bugs are a very timing critical issue.
For example, a new build with hardly any WiFi related changes (e.g. only some flash string changes) may yield a build which acts completely different when looking at the WiFi performance.
So another hypothesis of mine is that either the order of how object files are being linked matters, or maybe something has changed regarding alignment, causing some timing differences when using it.
Or there is some very silly bug somewhere, where we have a char-array which isn't properly 0-terminated and thus sensitive to previous values stored there. This can be an explanation to why the WiFi connection fails with errors related to not being authorized (forgot the exact error)

[dalbert2]

@TD-er FYI, if needed, I have lab grade RF gear (VNAs, spectrum analyzers, etc.) so any RF concerns can be tested. However, at this point, I don't suspect RF issues (at least not antenna tuning) for several reasons.

Sorry if this is obvious: antenna tuning is primarily a function of the shape of the antenna and the ground plane (including the RF shield) below it, and any matching components (Ls, Cs) in the feed path. It's also affected by nearby materials and it's certainly possible that the inverted meandering printed F antenna used in the ESP12E modules is not tuned properly and that bringing other materials close to or in contact with them will change this as you've noticed - I can check this with a VNA. However, if this were the case, it should be reflected in poor RSSI levels and I'm not seeing that.

A tuned antenna is one whose resonant frequency is in the target band and so transmits and receives efficiently at those frequencies 2.401 - 2.495GHz (for 802.11g). If properly tuned, it should have low return loss/VSWR. A de-tuned antenna will work poorly (inefficiently) in the target frequency band and we should see this in low RSSI levels (and it should not be intermittent). For this reason, I'm skeptical that this is the problem (at least my problem). My modules will often connect and communicate perfectly for hours with strong RSSI, but over time something happens and they fail to connect repeatedly for extended periods). Moreover, the problem occurs with the module so close to the AP that it would likely work with no antenna at all (and the RSSI reflects that). When the module fails to connect, a wifi scan will still see the AP with strong RSSI. Would you mind checking the RSSI levels on your module with and without de-tuning and at either end of the band (ch 1 and ch 13 or 14)?

With respect to heating: the maximum transmit power from most ESP8266 modules is around +20dBm (100mW) that is unlikely to get your antenna warm even if terribly matched; I suspect something else is getting warm and the heat is just being conducted through the PCB. Over-heating could be causing other problems: heat affects the crystal oscillator that generates the transmission frequencies. It would be good to know exactly what the module does when it performs calibration (and against what standard). FYI, the shield on my RF modules only gets very slightly warm.

I will try downgrading the Core to 2.4.2 and see if it helps with what I am seeing and report back.

If you test as above and that testing suggests that this is may be an RF issue, I can do two tests:

1. Measure the transmit frequencies on a spectrum analyzer as the module warms up to see if they drift significantly.
2. Measure the antenna return loss and see if it is decent in the B/G/N bands.

However neither of the above should be affected by the Core firmware version so if 2.4.2 works and 3.0.x does not, I suspect we should be focused on stack issues.

[TD-er]

About the de-tune of the antenna's.
Not only would you see it in the reported RSSI, but a badly tuned antenna also has another significant side-effect.
It doesn't match the standing wave, and thus the transmitted signal will be reflected back into the radio.
That's why -on some modules- you see much better WiFi stability when lowering the TX power.
For example the firmware running on the ESP-01 modules in Prusa Mini 3D printers showed exactly this problem and I had a discussion with someone about this who finally did report this to the devs and I think they did apply exactly this fix to the firmware on those units.
Also the first "Wemos D1 mini pro" units that didn't have a metal cap over the ESP appeared to show nearly impossible to reproduce errors which "magically" were solved when reducing TX power.

With badly tuned antenna's you also will have a different current and voltage distribution. So -to me- it makes sense the antenna will heat up as there is power injected into it which isn't radiated away as RF.

But as I said, with the 'standard' castelated ESP-12E/F/S modules it isn't likely the antenna will have a mismatch.

Anyway, you could try to test a bit with changing the TX power.
If that makes a significant difference, then you know there is either some RF mismatch, or some issue with power stability. (can also be too thin traces to the ESP)

[dalbert2]

Yes, RF energy not radiated (due to reflection or resistive loss) will be dissipated as heat; I was just trying to point out that 100mW max TX power is not enough to cause heating that could cause malfunction of the part. According to EspressIF, the ESP8266 is qualified for up to 125C and even if the ESP8266 package had extremely high thermal resistance (θJA), you wouldn't expect more than a few tens of degrees of thermal rise over ambient from 100mW...even if you ran the transmitter continuously into a short or open; it's just not enough to cause trouble. If you are noticing significant heating on your modules, that is certainly something to look into, but I doubt it's being caused by reflected transmit energy due to poor VSWR. Do you have a VNA? If not, the nano-VNA is inexpensive and remarkably useful; it would let you conclusively determine whether you are experiencing issues with antenna match. I have one and have compared it with my lab gear and it is surprisingly accurate; more than enough for this sort of measurement.

However, it's interesting that you and others are seeing improved performance by turning down transmit power. This is a half-duplex transceiver; the internal RF switch should severely attenuate tx energy that could damage the receiver. Enough reflected energy could damage the transmitter output stage, but that too seems unlikely (but not impossible) at +20dBm max power and would still likely appear as low RSSI at the AP and would not result in intermittent problems. It would be interesting to know specifically what changed in the output signal on the units whose problems were solved by reducing Tx Power. Did anybody look at the output with a spectrum analyzer?

Back to point: I think we agree that an RF mismatch should not be causing comms problems that don't also manifest as low RSSI. A poor antenna match will typically reduce PA power consumption, so it seems unlikely that this would be causing power-related problems (did you measure current consumption as you varied the match)? As you note, high impedance supply lines/insufficient de-coupling (or high ESR de-coupling) could cause issues in some designs. However I'm reasonable confident they are not the problem in my design where I have low-ESR decoupling caps (22uF + 100nF MLCC) located very close to the input castellation, 32-mil traces for Vcc and a well-stitched solid ground plane; I've done many RF designs at significantly higher power so I don't think layout is the issue here. The internal layout for the ESP12E module also looks reasonable.

Most importantly, none of these things would be affected by the firmware version (assuming they all set the same transmit power) or result in comms that are intermittent, particularly when the module is 2-feet from the AP with strong RSSI. This seems very much like a software problem.

I'm not questioning that you and others are seeing improved performance with reduced transmit power or experiencing module heating; I'm just interested in getting to the bottom of why these modules seem to have intermittent connectivity problems.

[TD-er]

I do have 2 different VNA's, but I don't see how I could possibly measure the antenna resonance frequency and/or impedance using those with PCB trace antennas.
Simply trying to connect them somehow will affect the measurement.

Right now I can't perform measurements on those specific 'problematic' units as I have no idea in which box they are.

[dalbert2]

I should do more digging and less speculating.

I've attached some relevant links at the end, but the EspressIF datasheets seem very light on RF details; I've found nothing to indicate minimum VSWR requirements, max RF input levels, or much else in the way of receiver specs (IP3, etc.); my google-fu seems to be failing me; are you aware of any detailed RF specs for the SOC?

Apparently, some past module printed antennae have been sub-optimal. The ESP12E was revised to 12F and apparently the only change was the PIFA itself (which is described as "better optimized")...which lends credence to your belief that there are antenna match problems with some modules.

The FCC test report for the AIThinker ESP12F module has no conducted emission measured above +16.4dBm (i.e. they're not transmitting at max power). Also of note is that the module is mounted with the antenna off the edge of the carrier PCB (no FR4 beneath it at all).

If it's only one or two modules, it might not be worthwhile, but if you are seeing this with many, it might be worth removing the shield and then the ESP8266 SOC from a board, attaching a short piece of micro-coax to the input to the antenna matching network (e.g. for the ESP12E or F you would attach it to the pads for L2 in the matching Pi network and then set your VNA measurement plane at the there), then replace the shield with the micro-coax running out the end opposite the antenna. Obviously it would depend on the module you're using, it would be some time/work, and the result wouldn't be perfect, but it would certainly be interesting to sweep the antenna for S11 around 2.4GHz and also see how well matched it is! I'm going to do some more googling to see if someone has already done this; otherwise, next time I'm working on a dead unit I'll try that and post the results.

I still don't suspect RF as the core connectivity issue in my case due to strong RSSI and the intermittent nature of the problem: when it works, it works perfectly, when it doesn't work, it won't connect to the AP at all despite very strong RSSI seen in WiFi scan, and the only thing changing between these states is run time. However, as my time permits, I'll dig into that and try to make some measurements suggested above.

Some possibly useful links:

• https://www.espressif.com/sites/default/files/documentation/esp8266-hardware_matching_guide_en.pdf
• https://www.espressif.com/sites/default/files/documentation/0a-esp8266ex_datasheet_en.pdf
• https://components101.com/sites/default/files/2021-09/ESP12E-Datasheet.pdf
• https://fcc.report/FCC-ID/2AHMR-ESP12F/3564264

[dalbert2]

Thank you, I will try reducing power shortly (and I read through the trail of comments on #1200 too). I have been focused on judicious introduction of delay() statements after each significant interaction with the WiFi and SSL stacks; this has so far made the biggest difference and dramatically improved performance so that I have nearly perfect connectivity every time. I'm pretty sure my problems are firmware-related and wondered if they might have to do with the periodic re-calibration done by the SDK (I'd really like to know what is being calibrated and against what standard).

I'm using different modules than what are being used in the Prusa printers so I'm not sure how closely what I'm experiencing tracks what you are. As @TD-er pointed out, the placement of the ESP-01 antenna directly above the aluminum electrolytic cap in those printers is a poor design choice. I haven't used ESP-01 modules, but judging from the photo, another unfortunate aspect of their use is that they are almost certainly not modular certified (no shield) and so their use in the printers is almost certainly an FCC violation.

Thanks again for the pointer and suggestion!

[TD-er]

Another interesting test would be to transmit at full power in both new and old SDK, problematic and working modules, to see if there are any differences in the radiated signal. Hypothesis being older SDK were somehow throttling down TX power at runtime.

That's still happening.
In ESPEasy, I have added an "ECO mode" option, which essentially does call delay() as often as possible whenever there's nothing to do.
This does reduce the power consumption significantly.
It seems to increase the DTIM interval and reduce TX power.

However, if you start sending pings to the node, the node will no longer enter this low power mode as long as you keep sending pings to the ESP.
So keep this in mind when testing.
To keep the TX power and DTIM at the set/known values, keep sending pings to the node.

But.... this will also affect switches and access points in your network as there is a known issue where the ESP may miss replying to ARP requests.
This way switches may at some point forget at which port the ESP is reachable.
To mitigate this, I do send gratuituous ARP packets periodically.

[TD-er]

Hmm just an idea...
What if the DTIM is set to an incorrect value when the ESP needs to reconnect?
What will happen then?
I think the ESP will not receive the reply from the AP.

[dalbert2]

@TD-er looking more closely at timing is a great idea; suggestions for what/how to test are welcome!

I tried reducing maximum transmit power didn't help, but thanks for the suggestion though @CarlosGS and @TD-er

A few other things I've noticed with respect to connectivity that may give clues (or at least help others):

1. the default wifi country is CN. Even when I explicitly set the country to my country (US), and the initial channel to 11, attempts to connect to the AP often use channels greater than 11.

        wifi_country_t usa = {
            .cc = "US",
            .schan = 1,
            .nchan = 11,
            .policy = WIFI_COUNTRY_POLICY_AUTO
        };
        wifi_set_country(&usa);

I'm probably doing something wrong here, but if anyone knows more about this, I'd appreciate it.

2. At times, LWIP provides an APIPA address; when this happens, WiFi will show as connected,
   (which it is), but that connection is useless - there is no IP connectivity, so any code that is testing to
   see if there is really a connection must also check for ! WiFi.localIP().isLocal().

3. I'd like to understand the need for (and how to use) the magic delay()s that seem to make the code work. It's not clear to me that the duration of the delay matters (i.e. I'm testing to see whether delay(1) has the same impact as delay(1000). However it seems clear that delay() statements are necessary for proper network operation. Decades ago, I knew LWIP much better and IIRC, it used protothreads rather than relying on OS tasking; does anyone know when LWIP actually gets to run in the Arduino/SDK environment?

[TD-er]

All I know is that a call to delay() does -on Espressif microprocessors- do some background tasks which also service WiFi and LWIP packet handling.
Typically I use 1 - 30 msec for operations on WiFi which do not actually need to wait for some interaction with the AP.
For example changing TX power.
For those actions that need some interaction with an AP or changing WiFi mode, I typically use 100 msec.

As far as I know, the country settings should hardly make any difference when connecting to an AP.
Some countries (e.g. USA) allow for more TX power, but the Espressif radio can't use > 14 dBm on 802.11n mode, 17.5 dBm on 802.11g and 20.5 dBm on 802.11b
The other limits implied by the country settings is related to the nr. of channels.
But in STA mode, you just follow the channel used by the AP you connect to.

[mhightower83]

Could you try #8858 (comment) and see if it has any effect?

[dalbert2]

Thanks @mhightower83 , I tried it with both SDK 2.2.2 and 3.0.5 but unfortunately no change.

[mhightower83]

@dalbert2 Thanks for checking.

[dalbert2]

After some changes, connections to the AP have been perfect for more than 24 hours: it connects the first time, every time. I'm going to back out the changes one at a time to find out what did it, but I strongly suspect it was changes made on the router:

1. Channel from 11 to 6
2. Security mode to WPA2 only
3. Cipher type to TKIP and AES

It's possible it was a code change, but they were small:

• reduced several delay()s from 1s to 100ms
• returned from SDK 3.0.5 to 2.2.2 (I went to 3.0.5 to try to fix this problem)
• Stopped forcing the PHY MODE to B or G (also done to try to fix this problem)

[TD-er]

• returned from SDK 3.0.5 to 2.2.2 (I went to 3.0.5 to try to fix this problem)

Which SDK22x version?

• PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK22x_190703 does appear to be "the most stable" or "least unstable", depending on your reference :)
• PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK22x_190313 is the last one for SDK22x which appears to be reporting the correct extensions to the AP when connecting in 802.11n mode.

Also some APs supporting "WiFi Mesh" (doesn't have to be enabled) try to disconnect Espressif nodes when they try to connect using 802.11n.
It is like they force a node to reconnect to the "strongest" AP by sending something to the node which causes the ESP to actually disconnect. Not sure if they just send a disconnect message, or some other message type misinterpreted by the ESP, causing it to disconnect.
If the node connects using 802.11g, this won't happen.
I don't know if using PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK22x_190313 also shows this behavior, as I don't own such APs which actively disconnect ESPs.
I only have Fritzbox and MikroTik APs and the Fritzbox APs actually are using WiFi mesh, but not causing issues here.

[dalbert2]

I'm building using the default current Arduino core from PlatformIO: Core:3.1.1 SDK:2.2.2-dev(38a443e) LWIP:2.1.3

I initially was having trouble with the units connecting to Ubiquiti non-mesh APs. Before joining this thread, I tried to isolate the problem so I set up an older D-Link b/g/n AP (DIR-645) on my desk dedicated to serving ESP8266 testing; the problem persisted and the D-Link unit is what I've been testing with since then.

Since the last round of changes, I've just let it run to see if the problem re-asserts, but so far it's gone >42 hours working like clockwork with the ESP8266 connecting to the AP and dumping data every 15 minutes. There are no retries on connection whereas before it often took many retries to connect and even when connected, the ESP8266 would sometimes end up with an APIPA address (DHCP failed), or HTTPS connection failure. It looks very much like there were severe problems with the communications link and one of the changes resolved it.

I am going to stop the tests at 48hrs and start backing out changes one at a time to see which made the difference.