USB Issue Troubleshooting

Hardware related problems and workarounds

Re: USB Issue Troubleshooting

Postby xilman » Thu Aug 21, 2014 8:48 am

aolofsson wrote:Hi all,

There are quite a few posts related to USB port failures. The goal of this post is to summarize all of them and to set the record straight.

The issue:
The USB phy does not always get initialized properly coming out of reset on some boards. After successful initialization, the USB port works fine.

...

Some data:
* The probability of successful USB initialization varies between boards.
* Keeping USB device plugged in during powerup increases probability of success
* A power cycle reset is more likely to work than a push button reset

Please could you explain how that is best done for a 4-board cluster in a common case fed by a common PSU? If the probability of failure is about 50% (two out of four presently do not have USB and that's typical in my experience) the likelihood of the whole cluster working is an unacceptably low 6.25%
aolofsson wrote:* Probability of success increased by using longer USB cables

How long is "longer"? If need be I'm prepared to buy and then leave four otherwise useless cables plugged in 24/7 if it means that the USB can be expected to work for the odd occasions I might need it.
aolofsson wrote:* Probability of success increased by using certain USB peripherals/hub setups
* Most boards work with all combination of hubs/peripherals (no restrictions)

A powered USB hub arrived this morning, purchased on the basis of information already posted on the forum. It was plugged into one of the boards which had not initialised its USB (two out of four were in that state) and rebooted. Note, not power-cycled. After reboot the USB still failed to work.

If "long" cables are left plugged in 24/7, must a mostly-useless active device also be connected to the far end? I'm prepared to dig up some otherwise useless devices from my parts bin (a wireless dongle with no Linux driver comes to mind, as do a couple of very small thumb drives) to leave dangling from the cluster until the rare occasion when I may want to plug something useful in there. The great majority of the time everything, especially large scale and properly backed-up storage and user I/O, will be driven over the ethernet.

I'm not entirely happy with the current system but I'm prepared to indulge in reasonable and inexpensive work-arounds. I've been at the sharp end of analogous situations and know what it's like. That's why I'm prepared to cut you some slack.
xilman
 
Posts: 80
Joined: Sat May 10, 2014 8:10 pm
Location: UK

Re: USB Issue Troubleshooting

Postby cmcconnell » Thu Aug 21, 2014 11:24 am

xilman wrote:How long is "longer"? If need be I'm prepared to buy and then leave four otherwise useless cables plugged in 24/7 if it means that the USB can be expected to work for the odd occasions I might need it.

For what it's worth, the problem was solved for me with the addition of a USB extension cable about 75cm long (which I just happened to have lying around), attached to the micro-to-full-USB conversion cable supplied by Adapteva, which is itself around 15cm long.

That being said, I've only ever tried it with one particular USB device (a wireless dongle for a combined keyboard/mouse), so I can't be sure the extension cable would work in other configurations (e.g., if I wanted to hang a hub off it).
Colin.
cmcconnell
 
Posts: 99
Joined: Thu May 22, 2014 6:58 pm

Re: USB Issue Troubleshooting

Postby Morgaine » Fri Aug 22, 2014 6:32 am

Problems that have this high a rate of occurrence don't happen because of unknown voodoo or a passing alpha particle, but because of quantifiable causes such as incorrect power sequencing, or out of spec rise times, or external devices not being held in reset while the FPGA is configuring its GPIOs, or race conditions, or noise on lines, or reflections owing to impedance mismatch, or inadequate on-board voltage regulation / current starvation on power-up surges, or etc etc etc .... you get the idea. There are a lot of possible causes, many more than I've mentioned.

Note that even if the board design is essentially correct, it's clear from the error rate that the chosen design margins are insufficient for reliable operation.

The feedback to Andreas about how often users find USB to be non-operational after boot is interesting as an indicator of individual experiences, but it probably won't do much to narrow down the cause. That's going to require some good old-fashioned engineering legwork with sampling scopes, logic analysers and other instrumentation, and meticulous re-examination of datasheets. There's also a prerequisite --- the Adapteva engineers first need a System Under Test that fails regularly, otherwise fault inspection is really hard and slow.

Am I right in thinking that such a reliably failing system is not yet to hand? If it were, I suspect Adapteva wouldn't be asking these questions at all, they'd just be getting on with diagnosing the cause, and then correcting the board design so that the situation does not occur.

Morgaine.
Morgaine
 
Posts: 42
Joined: Tue Jul 02, 2013 8:29 pm

Re: USB Issue Troubleshooting

Postby ajtravis » Sat Aug 23, 2014 12:26 am

Morgaine wrote:[...]
Am I right in thinking that such a reliably failing system is not yet to hand? If it were, I suspect Adapteva wouldn't be asking these questions at all, they'd just be getting on with diagnosing the cause, and then correcting the board design so that the situation does not occur.

Hi, Morgaine.

Well, I've got four ;-)

Andreas said he thought that my results were similar to what Adapteva found in their USB tests...

Quite often, USB works on my boards from a cold start (i.e. power has been off for several hours). In that respect, I believe that Adapteva have tested all the boards they shipped and found them to be working. However, in the good old days, we 'soak-tested' machines - It was not realistic for Adapteva to do that and sell the boards for $99: Remember that they did not make money on the 'backer' and 'pre-order' Parallella boards like mine. Let's not forget that this is a community-driven venture too. Adapteva would not have produced the Parallella boards without their successful Kickstarter campaign backed by us.

I agree that Andreas' did make it seem that only a few of us are having problems with USB, in which case I would have expected Adapteva to simply issue RMA's for a few faulty boards. However, I think the problem is more serious than that and only manifests itself when the systems are actually being used as opposed to being tested from a cold-start before shipping.

So far, only a few people have provided the information about USB problems that Andreas requested on this thread. If the problems with USB are as serious as I think they are I would have expected more people to post to this thread. Adapteva need to know if we are *really* having problems with USB so anyone reading this thread who is having problems with USB please post the information that Andreas requested and let Adapteva know the real level of USB problems we are experiencing with Parallella.

Thanks,

Tony.
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
User avatar
ajtravis
 
Posts: 152
Joined: Fri Jul 18, 2014 12:54 pm
Location: Scotland (UK)

Re: USB Issue Troubleshooting

Postby Morgaine » Sat Aug 23, 2014 8:30 am

According to this post of Fred's, the root cause is actually known now --- it's a timing fault:

FHuettig wrote:There is no logic that we can look at, it's all in a hard-IP block that's part of the Zynq peripherals. The signals are assigned to certain FPGA pins and then routed to the PHY.


In other words, it seems the Zynq's USB output timing is not compatible with the PHY's input timing requirement, which is why they're trying to slow the signal by adding capacitance. That itself can cause problems if the PHY input has a maximum rise-time that isn't met by the slowed edge (depends on the type of input logic), so we may not be out of the woods yet. Fingers crossed.

The extreme length of delay needed is puzzling though --- 30-42 milliseconds! That's an eternity in logic terms. Clearly the Zynq isn't expecting the PHY to be taking a holiday, but it's highly likely to be in the startup spec for the device unless it's actually caused by the board (for example through current starvation).

Morgaine.
Morgaine
 
Posts: 42
Joined: Tue Jul 02, 2013 8:29 pm

Re: USB Issue Troubleshooting

Postby Frida » Sat Aug 23, 2014 12:30 pm

The extreme length of delay needed is puzzling though --- 30-42 milliseconds!


Yes and I can extend it to ca. 10sec. without trouble.

I don't know anything about FPGA, but somebody must could explain the timing on these signals.
In particular STP, is it held high until the communication is ready from the FPGA?

A selection from the datasheet.

Datasheet
6.3.3.1
Start up Protection
Upon start-up, when the transceiver de-asserts DIR, the Link must be ready to receive commands and
drive Idle on the data bus. If the Link is not ready to receive commands or drive Idle, it must assert
STP before DIR is de-asserted. The Link can then de-assert STP when it has completed its start-up.
If the Link doesn’t assert STP before it can receive commands, the transceiver may interpret the data
bus state as a TX CMD and transmit invalid data onto the USB bus, or make invalid register writes.
User avatar
Frida
 
Posts: 21
Joined: Wed Sep 04, 2013 2:37 pm
Location: Middelfart, Denmark

Re: USB Issue Troubleshooting

Postby greytery » Sat Aug 23, 2014 3:37 pm

ajtravis wrote:So far, only a few people have provided the information about USB problems that Andreas requested on this thread..

A possible reason for 'under reporting' the USB problem is that boards that run headless are unlikely/less likely to hit it.
If so, then many cluster owners, for example, may simply have by-passed this issue on 'most' of their boards.

Cheers,
tery
User avatar
greytery
 
Posts: 205
Joined: Sat Dec 07, 2013 12:19 pm
Location: ^Wycombe, UK

Re: USB Issue Troubleshooting

Postby Frida » Sat Aug 23, 2014 5:55 pm

aolofsson wrote:* Failure mode (sometimes or always?)

Always, until I put on my LaunchPad
aolofsson wrote:* Headless/hdmi configuration

hdmi configuration
aolofsson wrote:* FPGA/Linux/Ubuntu version being used

ubuntu-14.04-140611.img
aolofsson wrote:* Fan used?

12V 0.60A 70 mm Fan Running at 5V
aolofsson wrote:* SKU#

SKU A101020
SN 0009801
U13 missing
aolofsson wrote:* Power supply used

5V 2.5A OEM PSU
Attachments
CAM00056a.jpg
CAM00056a.jpg (372.12 KiB) Viewed 11030 times
User avatar
Frida
 
Posts: 21
Joined: Wed Sep 04, 2013 2:37 pm
Location: Middelfart, Denmark

Re: USB Issue Troubleshooting

Postby ajtravis » Sat Aug 30, 2014 4:39 pm

aolofsson wrote:Hi all,

There are quite a few posts related to USB port failures. The goal of this post is to summarize all of them and to set the record straight.
[...]

Hi, Andreas.

I'm surprised how few people posted to this thread but if you are right and only a few of us are affected by the problem then I think it is time for Adepteva to take action and issue RMA's for the faulty boards. SInce we bought them in good faith expecting USB to work I think it would be unfair for the few people affected to bear the cost of shipping them back to Adepteva from the UK. Could we, for example, send them to RS Components in the UK for a replacement instead?

I think the positive publicity for Adapteva in resolving what is, for some of us, a very real problem would be worth the cost.

Tony.
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
User avatar
ajtravis
 
Posts: 152
Joined: Fri Jul 18, 2014 12:54 pm
Location: Scotland (UK)

Re: USB Issue Troubleshooting

Postby leon_heller » Sat Aug 30, 2014 5:49 pm

RS replaces faulty items without any problems.
Leon Heller
G1HSM
leon_heller
 
Posts: 67
Joined: Mon Dec 17, 2012 4:01 am
Location: St. Leonards-on-Sea, E. Sussex, UK.

PreviousNext

Return to Troubleshooting

Who is online

Users browsing this forum: No registered users and 4 guests