Embedded Systems-Can software kill?

[Lothar]

I'm married to an electrical engineer who worked for Hewlett Packard for 30 years

My dad was a Senior Software Engineer for Qwest (as Mountain Bell and US West) for 30 years. My wife is an embedded software engineer working on the very systems you're talking about. And her dad was an EE who worked for HP for 17 years designing integrated circuits.

Different companies have different cultures and expectations. Those companies also change over time; "normal" practices in 1974 are different from current practices, as software development has matured a great deal in that time period. And of course the type of system they're developing matters; the level of checking that goes into a chip in a $10 toaster is different from the checking that goes into a $150 million fighter jet.

Sometimes the entire code for a product was tossed out and redone!

Good programmers do this piece by piece anyway; it's called "refactoring". Write it, get it to work, and then throw it out and rewrite it now that you know how to do it. Eliminates tons of bugs.

Of course, it's a lot more expensive if you build up a bunch of bad software before you fix it.

you can't test in quality control, you really have to design it in if you want reliability

Of course. That's the whole point of "Software Engineering", as compared to merely programming.

it appears that software engineering is a totally different animal.... I couldn't find this type of engineering course at OSU's engineering school.

http://web.engr.oregonstate.edu/~cscaff ... _F09.shtml
http://beaversource.oregonstate.edu/soc ... fall-2009/

SE has been around for a very long time, and it has become more common to see full-fledged degrees with that title instead of just one or two courses in a more generic CS program.

[Tunnelcat]

Ferno, HP used to care about the quality of their products. Sadly, not anymore. I wouldn't trust them building a PC anymore!

Well, I guess things have changed over the years. Thanks for the links Lothar, I'll show them to my husband. He'll be interested in what's being taught now in the field of computer and software engineering. Looks like a radical change has occurred from what was around 30 years ago. It's good to see SE as a full-fledged discipline now, we'll need it in the future if we are to survive our embedded code.

I'd send him back to college so that he could learn some new things, he'd be fascinated, but he's too old and unhealthy to go through it anymore. Now if I could just get him to quit his griping and complaining about software whenever our PC crashes or locks up............

[Foil]

tc, I had to laugh a bit at some of those bits about your husband's experiences, because at my first programming job, I worked with some coders who did the same things. They were mostly business-folk who had been taught some minimal amount of coding. Sometimes it drove me nuts to troubleshoot their code, because it was just cobbled together and often made little or no sense. Even worse, some of those folks put little to no commentary in their code. Like your husband said, sometimes it just had to be thrown out and rewritten.

The thing is, those weak programmers generally don't survive in more technically demanding fields. The people I mentioned a moment ago would never make it in my current workplace (not that they'd be hired on in the first place). In fact, my current company recently let go of a couple developers who I thought did pretty decent work; for the rest of us, it was a pretty clear message about the quality we're expected to maintain.

Back to the original topic, perhaps the thing to look at is the software quality standards of a given industry and/or company. From what I understand, Toyota has been at or near the top of the quality measures in the auto industry for years (anecdotally, I can confirm this, as my wife and I own two of their vehicles and I have driven them for years); I'd tend to expect the same from their software development group.

[Spidey]

Well…I’m convinced.

[Tunnelcat]

tc, I had to laugh a bit at some of those bits about your husband's experiences, because at my first programming job, I worked with some coders who did the same things. They were mostly business-folk who had been taught some minimal amount of coding. Sometimes it drove me nuts to troubleshoot their code, because it was just cobbled together and often made little or no sense. Even worse, some of those folks put little to no commentary in their code. Like your husband said, sometimes it just had to be thrown out and rewritten.

The thing is, those weak programmers generally don't survive in more technically demanding fields. The people I mentioned a moment ago would never make it in my current workplace (not that they'd be hired on in the first place). In fact, my current company recently let go of a couple developers who I thought did pretty decent work; for the rest of us, it was a pretty clear message about the quality we're expected to maintain.

Glad to hear other stories about sloppy business developers. At least there seems to be some effort to organize and improve business software development today. My husband compared some of the developers he used to work with to novelists trying to write a novel without using a storyline, or sometimes without even paper, just the memory in their heads! Now if Microsoft would just get a clue...

Back to the original topic, perhaps the thing to look at is the software quality standards of a given industry and/or company. From what I understand, Toyota has been at or near the top of the quality measures in the auto industry for years (anecdotally, I can confirm this, as my wife and I own two of their vehicles and I have driven them for years); I'd tend to expect the same from their software development group.

It seems that Toyota's quality has been slipping in the last few years due to their rapid expansion of production and now, slipping sales. A few of my friends that own Toyotas have had more than a few gripes about the general quality and body fit/rattles. It seems that the expansion of production into the U.S. and South America has had it's pitfalls. The recall rate is creeping up too. They might just be losing their crown.

http://247wallst.com/2009/01/28/toyota-tm-recal/

This from 2007:

http://www.businessweek.com/autos/autob ... ality.html

[SirSamII]

91 Eagle Talon, pedal stuck when I floored it to pass someone on the interstate, hit 85 before I was able to reach down & pull it back to release it with my hand.

05 Jeep Liberty
92 Lexus Sc400
Not sure how I seem to possibly get this to occur on both vehicles. First time was purely by accident, If you push brake & gas over & over a few times the throttle sticks, not the pedal. Flooring the brake does nothing but squeals your tires until you throw it in neutral & releases the throttle just before redline. So is this an electrical or mechanical issue? Maybe this is an easter egg on the electronic side for those you are looking to unlock that hidden code left in the programming lol.

[Lothar]

I've definitely had throttle cables stick before in my 1982 Chevy van (NOT an integrated circuit problem.) The pedal was loose, but somewhere in the engine, the cable was snagging so it wouldn't release the whole way. Kicking the gas pedal down over and over again would eventually jerk the cable free, but I had a couple of adventures before I got to be able to do that reliably.

[Floyd]

I've definitely had throttle cables stick before in my 1982 Chevy van (NOT an integrated circuit problem.) The pedal was loose, but somewhere in the engine, the cable was snagging so it wouldn't release the whole way. Kicking the gas pedal down over and over again would eventually jerk the cable free, but I had a couple of adventures before I got to be able to do that reliably.

in other words, you didn't fix it the first time it happened?

[Lothar]

in other words, you didn't fix it the first time it happened?

I was sixteen. I didn't exactly know a lot about engines.

I did get the pedal to un-snag the first time, and drove on my way just fine. It wasn't until it became a recurring problem that I had somebody with more skillz look at it and repair it.

[Tunnelcat]

VW had a problem with their new diesel engine design in the early 1980's. They were prone to head gasket failures and cracked blocks. The engine was essentially a reworked gasoline engine they were using at the time.

Well, when either of these failures occurred, the crankcase would overpressure and force a lot of crankcase oil into the air plenum. Sometimes so much oil got in there that the engine would start to aspirate it and then it would just take off AT FULL TILT, TOTALLY UNTHROTTLED (diesels have no throttle plates anyway)! One time one of the other mechanics was working on one of these engineering junk heaps and it happened right in the shop after he started it! He had to race and pull off the air plenum cover and actually STUFF RAGS into the intake to shut it down! There was no other way! Fortunately, it didn't damage the engine, but it was scary none the less.

I loved the Dilbert cartoon from Dec. 16. It expressed some of my software sentiments.

http://dilbert.com/2009-12-15/

[Tunnelcat]

Had to bump this one up with all the finger pointing going on and Toyota sliding into meltdown mode. Since they've stopped selling 8 of their most of their popular models of Toyotas for the time being, the blame game is now starting. Lentz, the American President of Toyota, was on the Today show this morning and he seemed a little evasive and not very reassuring about whether THIS was the definitive fix for the unintended acceleration problem. He seemed to hedge on whether the problem was even noted BEFORE last October (cases go back to 1999). Also, the 8 models involved don't include others that have shown the problem too, Lexus for one.

USA Today

Now Toyota is trying to blame the pedal maker, CTS, for the faulty pedals. Even though CTS is sending out a redesigned pedal assembly for installation, they claim they weren't consulted about any recall. CTS also claims that unintended sudden acceleration was identified by Toyota in 1999, well before the 2005 date that CTS pedals were first used in Toyotas.

http://www.nationalpost.com/news/story.html?id=2508074

It turns out that Toyota gas pedals are all electronic, no mechanical linkage to the throttle at all, which is no surprise since autos have entered the computer age. I've noticed that no one in the press has noted that fact. Ignorance or maybe to not scare the average newer car owner that electronics are the sole link to controlling engine speed in this day and age. There are also 2 versions of the part by 2 suppliers, CTS and Denso. If you own a Toyota and want to look for what version you have in your car, check here:

http://www.tuneyfish.com/blog/cts-pedal ... ta-recall/

IF that turns out to be the eventual culprit, that is........

[woodchip]

TC, I'm impressed with your knowledgeable replies in this thread. I had a friend who had a early chevy pickup with a converted gas to diesel engine. The problem he had was the head gaskets couldn't handle the pressure and would constantly blow out. Can't remember if there was a machining issue on the block where the gaskets seated or if it was something else. The guy ultimately got rid of the vehicle.

What is interesting now is the perceived quality of companies like Ford ascending and Toyota descending.

[SilverFJ]

Get a horse. It'll carry your wasted ass back from the bar, and eats $12 a day.

[Tunnelcat]

TC, I'm impressed with your knowledgeable replies in this thread. I had a friend who had a early chevy pickup with a converted gas to diesel engine. The problem he had was the head gaskets couldn't handle the pressure and would constantly blow out. Can't remember if there was a machining issue on the block where the gaskets seated or if it was something else. The guy ultimately got rid of the vehicle.

What is interesting now is the perceived quality of companies like Ford ascending and Toyota descending.

Yeah, I remember these little gems from GM, LOL! How about the Cadillac V8-6-4? How do you think the Japanese swooped in during the 1990's and cleaned the American Auto Maker's clocks? Toyota's just fallen victim to the same thing, over expansion and lax quality management. By the way, the first GM Diesel was NOT a converted gasoline engine if it was original to the truck.

http://forums.caranddriver.com/showthread.php?t=686953

http://www.kitcarusa.com/kb.php?aid=62

But even the venerable 350 gasoline V8 was plagued with soft camshaft lobes that would go flat after average mileage use. It wasn't a well known problem, but I heard about it years ago. GM blamed the problem on lack of proper oil additives, but it turned out that the supplier hadn't properly hardened the cams after all.

I remember about 15 years ago, that Ford had the quality and image problems.

They were usually called by:

Found On th Road Dead

Fix Or Repair Daily

F**ked Over Reworked Dodge

Plus a few more:

http://www.abbreviations.com/FORD

http://www.all-acronyms.com/cat/9/FORD/ ... mp/1148722

However, the VW diesel was an absolute piece of crap when it first came out in the 1980's. They'd taken an aluminum gasoline engine block, beefed up the mains, bearings, pistons and crankshaft and put on an aluminum head hoping for a success. But as with the GM diesels, if the driver abused the engine or wasn't familiar with it's operation (the auto maker's usual response to bad design), head gasket failure was very common. We had diesel after diesel come into the shop either leaking coolant down the back of the engine or worse, coolant in the oil, unfortunately fatal to the bearings.

SilverFJ, horses don't go very fast for very far, they poop all the time indiscriminately and they require more upkeep and attention. And if they take exception to you, they'll kick you, buck you off or just stand stock still when ordered to move.

[SilverFJ]

Haha, I'd like to see your car make another car out of it's rear.

[S13driftAZ]

However, the VW diesel was an absolute piece of crap when it first came out in the 1980's. They'd taken an aluminum gasoline engine block

Um, from what I recall, VW has always used magnesium blocks

[Spidey]

I don’t think you could make an engine block out of pure magnesium, probably a aluminium-magnesium alloy.

[Will Robinson]

Haha, I'd like to see your car make another car out of it's rear.

Lol! You win!

[S13driftAZ]

I don’t think you could make an engine block out of pure magnesium, probably a aluminium-magnesium alloy.

That would not be very good. Aluminum expands as it is heated, probably MUCH more than magnesium, and in theory your block would fail.
But yeah, they're just straight up magnesium.

[Krom]

I don’t think you could make an engine block out of pure magnesium, probably a aluminium-magnesium alloy.

That would not be very good. Aluminum expands as it is heated, probably MUCH more than magnesium, and in theory your block would fail.
But yeah, they're just straight up magnesium.

Pure magnesium is extremely flammable and burns blindingly bright which is why it is used in various flares and fireworks. Additionally it reacts to water and carbon dioxide, making extinguishing it a challenge without the correct chemical fire extinguisher. An engine block made from pure magnesium would be a stretch...

[S13driftAZ]

Pure magnesium is extremely flammable and burns blindingly bright which is why it is used in various flares and fireworks. An engine block made from pure magnesium would be a stretch...

Ah, just did some research and yes you guys were right, it is a kind of alloy. AMC-SC1.

But still, I don't think it would be much of a stretch to make one out of pure magnesium. To ignite magnesium strips you have to get it really hot, as in glowing bright orange, before it ignites. I could be wrong tho. I don't know if temperatures inside the cylinders reach that hot.

[Grendel]

Pure magnesium is extremely flammable and burns blindingly bright which is why it is used in various flares and fireworks. Additionally it reacts to water and carbon dioxide, making extinguishing it a challenge without the correct chemical fire extinguisher.

Partly true -- ref: "Magnesium is a highly flammable metal, but while it is easy to ignite when powdered or shaved into thin strips, it is difficult to ignite in mass or bulk." Check the "Applications" section, it's being used quite a bit in cars -- incl. engine blocks. Sometimes it does become a factor in accidents, the most notable was the '55 Le Mans disaster.

[Tunnelcat]

Haha, I'd like to see your car make another car out of it's rear.

ROFL, The one advantage, doh! Then I wouldn't have to buy a new car when the old one wore out! You just have make sure they mated before they were past their prime mileage!

Um, from what I recall, VW has always used magnesium blocks

Nope. The early 1980's VW 1.6 diesel block was made from grey cast iron. The cylinder head was made from an aluminum alloy. Probably one of the reasons for many of the head gasket failures. The 2 metals have different coefficients of expansion. Not ideal for a head gasket in a high compression environment.

Heavy SOB of an engine for a little car like the Rabbit it was put into back then. Drove like a slug crawling on dry pavement too. The reason VW went with a diesel back then for their cars was because they wanted a simpler engine that didn't need all those emission controls.

[Tunnelcat]

Things are going even worse for Toyota. Now it's a brake problem with the Prius that actually MAY be a software glitch. It WAS caught and fixed at the factory for newer cars that were still at the factory, but nothing was done for cars that were already shipped to dealers and sold. They didn't even warn the public or government about it, until now. A recall is pending, maybe. Not looking good for them in the public relations trust department.

http://news.yahoo.com/s/ap/20100204/ap_ ... ota_recall

A funny side story concerns Steve Wozniak. He claims that his Prius goes WOT while in cruise control and Toyota refuses to deal with him about the problem.

http://www.gearlog.com/2010/02/runaway_ ... blem_w.php

But the ROFL part is that it may be caused by user error.

http://priuschat.com/news/steve-wozniak ... ntrol-kind

We'll see as things develop. But I see that The Woz has gotten a little portly in his old age. Maybe that's the problem. His Prius is trying to get away from him, screaming \"YOU'RE TO BIG, DON'T SIT IN ME\" or \"I'M TRYING TO GO AS FAST AS I CAN WITH MY TINY ENGINE!

BIG GEEK

An Asian acquaintance of ours came up with a funny conspiracy theory, industrial sabotage. There's a fierce and nasty centuries old rivalry between Japan and Korea, no love lost between those 2 nations. In fact, Koreans HATE the Japanese. Fast forward to today and the widespread practice of the outsourcing of components for a company's products. Japanese companies ARE using outsourced electronic parts from Korea in their products, cars too. It's a global economy.

Every new car now has a C.A.N. or 'car area network' to allow all of the car's system processors to talk to one another and coordinate operations. Our 2002 PT Cruiser has about 14 or so separate little processors that talk to each other on a main bus, some newer luxury cars can have around 70 or 80. What if a Korean electronics supplier, that was making a part for Toyota, decided to sabotage Toyota by burying malicious code in a small system processor. Say they decided to create a little virus that was a random number generator, for example, and then hid it deep in the embedded OS of that particular part's chip. Then, depending on a one in a billion chance, a random number would come up and cause little glitches or failures in another interconnected system or systems. It would cause no end of impossible to diagnose or find problems. You'll notice that Hyundai of Korea car sales have gone up 24% since Toyota started having 'troubles'.

[Grendel]

Every new car now has a C.A.N. or 'car area network' to allow all of the car's system processors to talk to one another and coordinate operations.

*cringe* -- Controller Area Network.

What if a Korean electronics supplier, that was making a part for Toyota, decided to sabotage Toyota by burying malicious code in a small system processor. Say they decided to create a little virus that was a random number generator, for example, and then hid it deep in the embedded OS of that particular part's chip. Then, depending on a one in a billion chance, a random number would come up and cause little glitches or failures in another interconnected system or systems. It would cause no end of impossible to diagnose or find problems. You'll notice that Hyundai of Korea car sales have gone up 24% since Toyota started having 'troubles'.

Amusing. But rather unlikely since your 1/1e9 random message needs a receiver that would interpret it as a command or sensor value. The probability for a match would be 1/2^11 or smaller, not even considering the probability of the data portion matching the expected format

[Tunnelcat]

You're right Grendel. My factory service manual actually calls it the Car Area Network, though I can see why 'controller' is a more accurate descriptor. That's what I get for reading a service manual written by a tech writer dweeb.

My conspiracy theory was just meant as a joke, I'm not a software expert!

However, it's come to light that the Prius has had over 1300 previous and present unresolved complaints with the headlights turning OFF on their OWN at inopportune times, LIKE AT NIGHT. More Toyota sloppiness with customer gripes.

http://seattletimes.nwsource.com/html/n ... ota05.html

[Tunnelcat]

Here's a new twist, a guy convicted of murder using his Toyota car may have really been a victom of Toyota's new found sloppy quality:

http://chicagopressrelease.com/press-re ... -in-prison

http://www.opposingviews.com/i/man-serv ... ing-toyota

To make matters worse, he was probably originally convicted due to local cultural bias since he's a Laotian Hmong and his accident happened just months after 3 Wisconsin hunters were murdered by another Hmong immigrant. Shows you the jury didn't or couldn't take into consideration that a problem with the car could have caused the unintended acceleration. I mean the guy had his FAMILY in the car with him! Why would he intentionally wreck his own car and injure his family?

There was also interesting testimony from an American engineer during the Toyota Hearings. He claimed that he could get the engine control unit from a new Toyota to enter a state of uncontrolled throttle response after about 3 hours of experimenting. I'll search for his testimony.

[Will Robinson]

Here's a new twist, a guy convicted of murder using his Toyota car may have really been a victom of Toyota's new found sloppy quality:

http://chicagopressrelease.com/press-re ... -in-prison

http://www.opposingviews.com/i/man-serv ... ing-toyota

To make matters worse, he was probably originally convicted due to local cultural bias since he's a Laotian Hmong and his accident happened just months after 3 Wisconsin hunters were murdered by another Hmong immigrant. Shows you the jury didn't or couldn't take into consideration that a problem with the car could have caused the unintended acceleration. I mean the guy had his FAMILY in the car with him! Why would he intentionally wreck his own car and injure his family?

There was also interesting testimony from an American engineer during the Toyota Hearings. He claimed that he could get the engine control unit from a new Toyota to enter a state of uncontrolled throttle response after about 3 hours of experimenting. I'll search for his testimony.

Are you saying if that zipperhead had been driving a Gran Torino he would be free today?

[roid]

zipperhead

you sure he wasn't a ★■◆●?

[woodchip]

It would seem the part the Hmong played in aiding us during the Vietnam war is now largely forgotten.:

\"The loss of 60,000 American lives for a no-win war in Vietnam was a tragedy to the huge nation of America, but it was a relatively small percentage of the nation compared to the loss the Hmong people suffered. In 1969, at the time when Congress first learned of our secret war in Laos, about 18,000 Hmong soldiers had already been killed in battle died, and many women and children had died as well. The Hmong were taking a great risk in boldly fighting for the United States, trusting that we would stand by them. But in 1973, the U.S. began to pull out of Laos, leaving the Hmong on their own to fight thousands of North Vietnamese troops in Laos. By 1975, Laos had fallen completely into Communist hands, and the lives of all Hmong people who helped fight the Communists were in jeopardy. More than 100,000 Hmong fled to Thai refugee camps. Many would be killed along the way, especially when crossing the Mekong River to get to Thailand. An estimated 30,000 Hmong would be killed by Communist forces while trying to reach Thailand. Over 100,000 Hmong people died as a result of the war, and today nearly every Hmong family in the US has terrible tales of loss and tragedy relating to the war.\"

[Tunnelcat]

Are you saying if that zipperhead had been driving a Gran Torino he would be free today?

You don't sound like someone who has ever been a victim of discrimination or bias. Step off your pedestal. WHY would a mid-west, probably all-white jury convict an immigrant man of vehicular homicide, that same man who had HIS OWN FAMILY riding in the car at the time of the accident? The jury had to be either culturally biased or assume that Toyota cars ALWAYS worked right mechanically, OR ASSUMED BOTH INSTANCES, to come up with that crazy, senseless verdict. I guess since the jury couldn't be shown a fault with the car as a cause of the collision, the only other explanation was that the driver was just another evil immigrant murderer out to kill white folks for pleasure or revenge.

Yes woodchip, the U.S. seems to have forgotten who our allies were during the Vietnam War. We've pushed the Hmong under the rug like our nation always seems to do with those who were friendly allies during some war or another we've had in the past. Use and throw away soldiers of convenience. Worse, a lot of times, even our own.

[Duper]

Just a small correction woody (and I'm sure you know this, but for the younger crowd) 'Nam was technically a \"policing action\" and was not fought like a war. War was never \"declared\" as the Gov feared an all out with China or the Soviets or both. THAT is why we got our butts handed to us there. Incidentally, what's the source of that quote. it's a nice piece.
Don't get me wrong. I'm not belittling our boy's duties or sacrifices there (some of them were my Dad's best friends) But it's important to note on of the major flaws of that conflict. (which why it's called a \"conflict\" aw you well know)

You are now returning you to your regularly scheduled debate.

[Grendel]

Gran Torino, very good movie BTW.