USB Issue Troubleshooting

[aolofsson]

Hi all,

There are quite a few posts related to USB port failures. The goal of this post is to summarize all of them and to set the record straight.

The issue:
The USB phy does not always get initialized properly coming out of reset on some boards. After successful initialization, the USB port works fine.

Statements:
*Since day one, we have tested the USB port on every board shipped
*We have shipped 10,000 Parallella boards to 7,000 customers to date and have only received issue reports from a small number of customers
*We have spent significant time debugging this problem (internal to Adapteva and external) and unfortunately we have not found an acceptable bullet proof workaround yet
*We will keep working on resolving this issue, but given our limited resources and other high priority items that need to be dealt with, we can't give a time line for when this will be resolved.

Some data:
* The probability of successful USB initialization varies between boards.
* Keeping USB device plugged in during powerup increases probability of success
* A power cycle reset is more likely to work than a push button reset
* Probability of success increased by using longer USB cables
* Probability of success increased by using certain USB peripherals/hub setups
* Most boards work with all combination of hubs/peripherals (no restrictions)

How to report a USB problem:
* Failure mode (sometimes or always?)
* Headless/hdmi configuration
* FPGA/Linux/Ubuntu version being used
* Fan used?
* SKU#
* For SKU A101040 boards owners, an indication whether the U13 device is populated
* Power supply used
* Recommended: A picture of your USB setup (as shown here)

Sincerely,
Andreas

[ajtravis]

[...]
Statements:
*Since day one, we have tested the USB port on every board shipped
*We have shipped 10,000 Parallella boards to 7,000 customers to date and have only received issue reports from a small number of customers
*We have spent significant time debugging this problem (internal to Adapteva and external) and unfortunately we have not found an acceptable bullet proof workaround yet
*We will keep working on resolving this issue, but given our limited resources and other high priority items that need to be dealt with, we can't give a time line for when this will be resolved.
[...]

Hi, Andreas.

I don't understand why you have invested so much time and effort debugging a problem that only affects a small number of customers when you could have resolved the matter by issuing RMA's for the correspondingly small number of faulty boards.

I requested an RMA for my 4@SKU101020 cards, but then decided to continue trying to identify where the USB problem was because other people seem to be having similar problems. Is it possible that these problems are confined to a particular SKU?

Bye,

Tony.

[Morgaine]

The issue:
The USB phy does not always get initialized properly coming out of reset on some boards. After successful initialization, the USB port works fine.

Statements:
* Since day one, we have tested the USB port on every board shipped
* We have shipped 10,000 Parallella boards to 7,000 customers to date and have only received issue reports from a small number of customers
* We have spent significant time debugging this problem (internal to Adapteva and external) and unfortunately we have not found an acceptable bullet proof workaround yet
* We will keep working on resolving this issue, but given our limited resources and other high priority items that need to be dealt with, we can't give a time line for when this will be resolved.

It's good to see direct recognition that the problem exists on some boards, since that's the first step towards fixing it.

Some aspects of this situation are extremely puzzling though. For example, how can all 4 of ajtravis's boards suffer from the same problem? This can't be attributed to natural variability in individual units unless a whole production batch is faulty and ajtravis was unlucky enough to have received all boards from that batch.

Also, while the official Statements recognize that a USB initialization problem does manifest itself on some boards, they don't actually say whether the cause of this problem has been identified yet. While it's good to know that a workaround is being sought, the more important step is first identifying the cause, otherwise workarounds are just guesswork. Has the cause of unreliable USB initialization been determined yet?

Morgaine.

[theover]

Hi,

In my case of the 7010 board without mounted U13 chip, usb regularly fails at startup, maybe 50 % chance, i don't know.

If I reset with the button, USB always works, but of course this can upset the OS.

I've made a de-bounced startup circuit with the supplied wall wart, which seems to increase the chance of a startup with USB.

I don't think HDMI or not made a difference, I used the latest Linux and FPGA of a couple of months ago.

I've used various USB units, a long extension cable improved startup chance (but not near 100%), and conrad switchable hub.

T.

[ajtravis]

[...]
How to report a USB problem:
* Failure mode (sometimes or always?)

Sometimes: 10 runs booting node1-4 of my 4@A101020 Parallella cluster (0 = USB phy failed to initialise, 1 = initialised)

Code: Select all

run   node1   node2   node3   node4   
1   0   0   0   0   
2   1   0   0   1   
3   1   0   1   0   
4   0   0   0   1   
5   0   0   0   0   
6   0   0   1   1   
7   0   0   1   1   
8   0   0   1   1   
9   0   0   1   1   
10   0   0   0   1   
               
total   2   0   5   7   35%


* Headless/hdmi configuration

Default HDMI config from http://www.parallella.org/create-sdcard/

* FPGA/Linux/Ubuntu version being used

kernel-hdmi-140603.tgz
parallella_e16_hdmi_gpiose_7010.bit.bin

* Fan used?

80mm 12V fan run at 5V

* SKU#

4@A101020 (node1-4)

* For SKU A101040 boards owners, an indication whether the U13 device is populated

N/A

* Power supply used

Adapteva supplied 5V/2.5A Jameco PSU with US->UK 240V adapter

* Recommended: A picture of your USB setup (as shown here)

IMG_2278.JPG (211.72 KiB) Viewed 25682 times

IMG_2276.JPG (240.87 KiB) Viewed 25682 times

Hi, Andreas.

In all cases, I could ssh into the nodes and run "lsusb" as well as test the wireless keyboard and mouse.
This is the "dmesg" output when the USB phy fails to initialise

USB-phy-missing.log

(12.26 KiB) Downloaded 612 times

. When the USB phy was detected I got this from "lsusb":

Code: Select all

Bus 001 Device 002: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub


Please let me know if you need any other information?

Bye,

Tony.

[aolofsson]

Tony,

Thanks for the detailed feedback. I noticed in the picture that only one board had USB/HDMI connected. Under what conditions were the 10 tries done for board 1-4? Were all of them connected to USB dongles at power on? Your results fit reasonably well with our data if board 4 is the top one and you only had that USB cable connected. If this was not the case, then we are back to head scratching...

Andreas

[ajtravis]

Tony,

Thanks for the detailed feedback. I noticed in the picture that only one board had USB/HDMI connected. Under what conditions were the 10 tries done for board 1-4? Were all of them connected to USB dongles at power on? Your results fit reasonably well with our data if board 4 is the top one and you only had that USB cable connected. If this was not the case, then we are back to head scratching...

Hi, Andreas.

I tested one node at a time: I moved the uSD card, HDMI and USB cable to the next node after completing 10 runs on each node. The USB OTG cable and wireless dongle were attached before powering up the node. All the inactive nodes were powered off during my tests. The top board is node1 and the bottom board is node4.

Although these are the same nodes that I reported on in my previous tests http://forums.parallella.org/viewtopic.php?f=50&t=1483#p10180 I made a new default HDMI config uSD card and used it to re-test all the nodes. In my previous tests, I used a Maplin 5V/8.4A USB charger and the rear USB connector to power the boards, but this time I used the Adapteva-supplied PSU plugged into the barrel power connector and changed the PSU option link which had previously been set for USB power.

The boards work well, except for the problem initialising the USB. I've not had any NIC problems and I was able to demo "dsh" and MPI etc. without any problem. I've run the Epiphany demos on node1 OK, but I've not tested Epiphany on node2-4 yet.

Bye,

Tony.

[RomanLudwig]

* Failure mode (sometimes or always?)

Sometimes

* Headless/hdmi configuration

hdmi configuration

* FPGA/Linux/Ubuntu version being used

ubuntu-14.04-140611.img

* Fan used?

5V 0.09A 60 mm Fan

* SKU#

SKU P1602-DK02 SN 0011927

* Power supply used

5V 2.0A FAIRWAY PSU

Hi Andreas,

i have more than 50% failures for the USB port on startup. As my wireless keyboard/mouse set has no numlock LED, it turned out to be helpful to plug in a Belkin Numeric Keypad at boot time. If the USB port starts up successfully the numlock LED of the keypad turns off after less than a second (and turns on again a couple of seconds later). On USB port failure the numlock LED shines constantly. So i can decide after less than a second whether to recycle power to try again. Once the numlock LED turned off, i can successfully change to my wireless dongle at any phase of the boot proccess.


Greetings

Roman

[Sean S]

Hi Andreas

Can you please clarify "only received issue reports from a small number of customers"? Through what means are you referring to for receipt of issue reports and do you think this is a realistic reflection of the number of customers that are experiencing USB problems?

If you consider the number of view on the subject of "USB not working" which currently stands at "Views: 4882" then one of two things are possible

- either the same people have nothing better to do than view this thread over and over desperately hoping for a solution...
- or many people are having issues with USB.

You claim to have shipped 10 000 boards to 7 000 customers... Which would mean that roughly half of the people that have boards have viewed this thread - implying that they have issues with USB too. That in my head does not account for a "small number of customers".

It has been suggested that I RMA my board but being in the UK, I can't justify the cost of shipping at my expense. In addition as the board was a gift, I am being told that I cannot RMA the board but need to hassle someone else to go through the RMA process which I don't think it fair. So please don't make that suggestion to me.

I am happy to see that after all of noise, you have finally made a formal statement. However, this needs to be followed up with a full solution - as you said. And I think you need to offer people their money back as well as an option. Many people require USB to be stable.

Honestly, the Parallella concept is great. Just a pity that such a key feature has no resolution or understanding after such a long time.

Sean

[AndyC]

* Failure mode (sometimes or always?)

75% failure


* Headless/hdmi configuration

hdmi


* FPGA/Linux/Ubuntu version being used

Standard ones from http://www.parallella.org/create-sdcard/


* Fan used?

5V60mm - avg temp 55


* SKU#

P1602-DK02


* Power supply used

4A 5V switched