Bugs and System Validation

[BS8899]

At its core, Lucid’s core-competencies are electro-mechanical and batteries. Based on these competencies, they built an industry-leading skateboard and a low-drag body. Together, these building blocks gave the Air superiority on mpge, range, and acceleration. These are the main reasons why owners like us buy their Lucid Airs. While the long-term reliability of this industry-leading skateboard still needs to be proven, it is off to a good start on the electro-mechanical front.

But it takes more than the skateboard and a sleek body to build a luxury sedan. It needs reliable, (nearly) bug-free SYSTEM integration, a state-of-the-art computer system on wheels that harmonizes the raw power, safety, Dream-Drive, navigation, infotainment, etc..



Inasmuch as Lucid did a great job on the electrotechnical platform, it appears they under-estimated the challenges of the System Integration and Validation.



Lucid has been shipping cars for over one year. The “bugs” we are complaining about are unlikely bugs related to a basic function simply not working. More likely, these “bugs” originate from unintended interactions at the system level that result in random events. These events, though real, are often non-systematic and hard to reproduce (hence, hard to fix).

These types of “System Bugs” are not uncommon in a complex computing system where the SW and HW for each sub-function was developed and validated by individual engineers or teams. Yet, this ensemble of validated SW modules WILL interact with each other in some unanticipated way under unforeseen circumstances. These circumstances are pseudo-random and not easily reproducible.

Computer manufactures have long recognized the need to do SYSTEM VALIDATION and REGRESSION in addition to just validating individual SW and HW blocks. System Validation and Regression Analyses are the manufacturers’ proxy to simulating the use environment. Done rigorously, it will help to uncover many of the “bugs” that result from the unintended interactions between the SW modules as well as the hardware.

Doing proper System Validation requires more than just technical expertise (that’s a given). By definition, system-level “bugs” are likely cross-functional, i.e., unintended interactions between its building blocks. Often, these issues cross organizational boundaries. As such, System Validation also requires an empowered organizational structure that can arbitrate and implement the resultant changes/tradeoffs.

So, the question is, is Lucid Motors doing adequate System Validation and Regression on their Air products?

I don’t pretend to know.

However, if you look hard, you can find some hints. Lucid appointed its first VP of SW Validation 18 mo ago. That’s good. But, what about system validation? It appears that Lucid is now trying to ramp up its validation activities. If you look at job openings at Lucid (e.g., Indeed.com, Newark CA job openings), you will see a flood of postings for system validation engineers (40+ openings). It will take time to hire, train, and integrate these new hires into a cohesive team.

 

[joec]

At its core, Lucid’s core-competencies are electro-mechanical and batteries. Based on these competencies, they built an industry-leading skateboard and a low-drag body. Together, these building blocks gave the Air superiority on mpge, range, and acceleration. These are the main reasons why owners like us buy their Lucid Airs. While the long-term reliability of this industry-leading skateboard still needs to be proven, it is off to a good start on the electro-mechanical front.

But it takes more than the skateboard and a sleek body to build a luxury sedan. It needs reliable, (nearly) bug-free SYSTEM integration, a state-of-the-art computer system on wheels that harmonizes the raw power, safety, Dream-Drive, navigation, infotainment, etc..



Inasmuch as Lucid did a great job on the electrotechnical platform, it appears they under-estimated the challenges of the System Integration and Validation.



Lucid has been shipping cars for over one year. The “bugs” we are complaining about are unlikely bugs related to a basic function simply not working. More likely, these “bugs” originate from unintended interactions at the system level that result in random events. These events, though real, are often non-systematic and hard to reproduce (hence, hard to fix).

These types of “System Bugs” are not uncommon in a complex computing system where the SW and HW for each sub-function was developed and validated by individual engineers or teams. Yet, this ensemble of validated SW modules WILL interact with each other in some unanticipated way under unforeseen circumstances. These circumstances are pseudo-random and not easily reproducible.

Computer manufactures have long recognized the need to do SYSTEM VALIDATION and REGRESSION in addition to just validating individual SW and HW blocks. System Validation and Regression Analyses are the manufacturers’ proxy to simulating the use environment. Done rigorously, it will help to uncover many of the “bugs” that result from the unintended interactions between the SW modules as well as the hardware.

Doing proper System Validation requires more than just technical expertise (that’s a given). By definition, system-level “bugs” are likely cross-functional, i.e., unintended interactions between its building blocks. Often, these issues cross organizational boundaries. As such, System Validation also requires an empowered organizational structure that can arbitrate and implement the resultant changes/tradeoffs.

So, the question is, is Lucid Motors doing adequate System Validation and Regression on their Air products?

I don’t pretend to know.

However, if you look hard, you can find some hints. Lucid appointed its first VP of SW Validation 18 mo ago. That’s good. But, what about system validation? It appears that Lucid is now trying to ramp up its validation activities. If you look at job openings at Lucid (e.g., Indeed.com, Newark CA job openings), you will see a flood of postings for system validation engineers (40+ openings). It will take time to hire, train, and integrate these new hires into a cohesive team.

Exactly. Lucid seems to be paying for some poor choices on the software front early on, and it will take a while to make things right.

It also just takes time to build this sort of infrastructure. And the results are not immediately evident to the customer (at least not as evident as shipping shiny new features), so most companies deprioritize things like validation, to their detriment.

What gives me confidence Lucid will set it right is that list of open job positions. They know where their weaknesses are, and they are working to rectify the situation.

Evidence that Lucid is willing to make serious change in order to set things right is plentiful. It’s been a tumultuous year, but they are way stronger now than they were going into 2022. If they can continue to improve on the software front in 2023, they should be unstoppable.

 

[Keithmaj]

At its core, Lucid’s core-competencies are electro-mechanical and batteries. Based on these competencies, they built an industry-leading skateboard and a low-drag body. Together, these building blocks gave the Air superiority on mpge, range, and acceleration. These are the main reasons why owners like us buy their Lucid Airs. While the long-term reliability of this industry-leading skateboard still needs to be proven, it is off to a good start on the electro-mechanical front.

But it takes more than the skateboard and a sleek body to build a luxury sedan. It needs reliable, (nearly) bug-free SYSTEM integration, a state-of-the-art computer system on wheels that harmonizes the raw power, safety, Dream-Drive, navigation, infotainment, etc..



Inasmuch as Lucid did a great job on the electrotechnical platform, it appears they under-estimated the challenges of the System Integration and Validation.



Lucid has been shipping cars for over one year. The “bugs” we are complaining about are unlikely bugs related to a basic function simply not working. More likely, these “bugs” originate from unintended interactions at the system level that result in random events. These events, though real, are often non-systematic and hard to reproduce (hence, hard to fix).

These types of “System Bugs” are not uncommon in a complex computing system where the SW and HW for each sub-function was developed and validated by individual engineers or teams. Yet, this ensemble of validated SW modules WILL interact with each other in some unanticipated way under unforeseen circumstances. These circumstances are pseudo-random and not easily reproducible.

Computer manufactures have long recognized the need to do SYSTEM VALIDATION and REGRESSION in addition to just validating individual SW and HW blocks. System Validation and Regression Analyses are the manufacturers’ proxy to simulating the use environment. Done rigorously, it will help to uncover many of the “bugs” that result from the unintended interactions between the SW modules as well as the hardware.

Doing proper System Validation requires more than just technical expertise (that’s a given). By definition, system-level “bugs” are likely cross-functional, i.e., unintended interactions between its building blocks. Often, these issues cross organizational boundaries. As such, System Validation also requires an empowered organizational structure that can arbitrate and implement the resultant changes/tradeoffs.

So, the question is, is Lucid Motors doing adequate System Validation and Regression on their Air products?

I don’t pretend to know.

However, if you look hard, you can find some hints. Lucid appointed its first VP of SW Validation 18 mo ago. That’s good. But, what about system validation? It appears that Lucid is now trying to ramp up its validation activities. If you look at job openings at Lucid (e.g., Indeed.com, Newark CA job openings), you will see a flood of postings for system validation engineers (40+ openings). It will take time to hire, train, and integrate these new hires into a cohesive team.

You have hit the nail on the head! System integration design and test is critical for complex systems. I hope Lucid has taken a page out of the aerospace industry and has been conducting Failures Modes, Effects and Criticality Analysis (FMECA), Worst Case Circuit Analysis (WCCA), and Sneak Circuit Analysis (SCA). These analyses consider the cause and effects of the overall hierarchical system design. I can't imagine how large these analyses must be for the Air, but once implemented they are usually a key part of system level integration and test. Regression testing of design changes can be more easily assessed as can intermittent, timing, and environmentally driven conditions. Lucid has come a long way since their 2.x rewrite, here's hoping they are able to continue that growth!

 

[TCGT]

If they are indeed looking for help, you guys sound ready!

 

[Mossnado]

At its core, Lucid’s core-competencies are electro-mechanical and batteries. Based on these competencies, they built an industry-leading skateboard and a low-drag body. Together, these building blocks gave the Air superiority on mpge, range, and acceleration. These are the main reasons why owners like us buy their Lucid Airs. While the long-term reliability of this industry-leading skateboard still needs to be proven, it is off to a good start on the electro-mechanical front.

But it takes more than the skateboard and a sleek body to build a luxury sedan. It needs reliable, (nearly) bug-free SYSTEM integration, a state-of-the-art computer system on wheels that harmonizes the raw power, safety, Dream-Drive, navigation, infotainment, etc..



Inasmuch as Lucid did a great job on the electrotechnical platform, it appears they under-estimated the challenges of the System Integration and Validation.



Lucid has been shipping cars for over one year. The “bugs” we are complaining about are unlikely bugs related to a basic function simply not working. More likely, these “bugs” originate from unintended interactions at the system level that result in random events. These events, though real, are often non-systematic and hard to reproduce (hence, hard to fix).

These types of “System Bugs” are not uncommon in a complex computing system where the SW and HW for each sub-function was developed and validated by individual engineers or teams. Yet, this ensemble of validated SW modules WILL interact with each other in some unanticipated way under unforeseen circumstances. These circumstances are pseudo-random and not easily reproducible.

Computer manufactures have long recognized the need to do SYSTEM VALIDATION and REGRESSION in addition to just validating individual SW and HW blocks. System Validation and Regression Analyses are the manufacturers’ proxy to simulating the use environment. Done rigorously, it will help to uncover many of the “bugs” that result from the unintended interactions between the SW modules as well as the hardware.

Doing proper System Validation requires more than just technical expertise (that’s a given). By definition, system-level “bugs” are likely cross-functional, i.e., unintended interactions between its building blocks. Often, these issues cross organizational boundaries. As such, System Validation also requires an empowered organizational structure that can arbitrate and implement the resultant changes/tradeoffs.

So, the question is, is Lucid Motors doing adequate System Validation and Regression on their Air products?

I don’t pretend to know.

However, if you look hard, you can find some hints. Lucid appointed its first VP of SW Validation 18 mo ago. That’s good. But, what about system validation? It appears that Lucid is now trying to ramp up its validation activities. If you look at job openings at Lucid (e.g., Indeed.com, Newark CA job openings), you will see a flood of postings for system validation engineers (40+ openings). It will take time to hire, train, and integrate these new hires into a cohesive team.

Great post and thread. You have nailed the essence of my concerns, after seemingly random but unsettling behaviors like the car refusing to go into Park and backing up while in Drive. This thread helps restore my confidence in Lucid, which has been shaken.

 

[pjdcorp]

At its core, Lucid’s core-competencies are electro-mechanical and batteries. Based on these competencies, they built an industry-leading skateboard and a low-drag body. Together, these building blocks gave the Air superiority on mpge, range, and acceleration. These are the main reasons why owners like us buy their Lucid Airs. While the long-term reliability of this industry-leading skateboard still needs to be proven, it is off to a good start on the electro-mechanical front.

But it takes more than the skateboard and a sleek body to build a luxury sedan. It needs reliable, (nearly) bug-free SYSTEM integration, a state-of-the-art computer system on wheels that harmonizes the raw power, safety, Dream-Drive, navigation, infotainment, etc..



Inasmuch as Lucid did a great job on the electrotechnical platform, it appears they under-estimated the challenges of the System Integration and Validation.



Lucid has been shipping cars for over one year. The “bugs” we are complaining about are unlikely bugs related to a basic function simply not working. More likely, these “bugs” originate from unintended interactions at the system level that result in random events. These events, though real, are often non-systematic and hard to reproduce (hence, hard to fix).

These types of “System Bugs” are not uncommon in a complex computing system where the SW and HW for each sub-function was developed and validated by individual engineers or teams. Yet, this ensemble of validated SW modules WILL interact with each other in some unanticipated way under unforeseen circumstances. These circumstances are pseudo-random and not easily reproducible.

Computer manufactures have long recognized the need to do SYSTEM VALIDATION and REGRESSION in addition to just validating individual SW and HW blocks. System Validation and Regression Analyses are the manufacturers’ proxy to simulating the use environment. Done rigorously, it will help to uncover many of the “bugs” that result from the unintended interactions between the SW modules as well as the hardware.

Doing proper System Validation requires more than just technical expertise (that’s a given). By definition, system-level “bugs” are likely cross-functional, i.e., unintended interactions between its building blocks. Often, these issues cross organizational boundaries. As such, System Validation also requires an empowered organizational structure that can arbitrate and implement the resultant changes/tradeoffs.

So, the question is, is Lucid Motors doing adequate System Validation and Regression on their Air products?

I don’t pretend to know.

However, if you look hard, you can find some hints. Lucid appointed its first VP of SW Validation 18 mo ago. That’s good. But, what about system validation? It appears that Lucid is now trying to ramp up its validation activities. If you look at job openings at Lucid (e.g., Indeed.com, Newark CA job openings), you will see a flood of postings for system validation engineers (40+ openings). It will take time to hire, train, and integrate these new hires into a cohesive team.

Great observation. Thank you

 

[BS8899]

While Lucid's job postings for validation engineers is encouraging and a step in the right direction, it will be an arduous road getting to a better place.
As I inferred in the original post, SYSTEM VALIDATION is much ore than just SW Validation. An automobile SYSTEM has many interacting factors including HW, SW, weather, elctrical noise (internal and external) etc.. System Validation and remedies involve emcompassing tasks that trascend functional and organizational boundaries. To be successful, it needs senior management oversight and empowerment.

 

[Cosmo Cruz]

The architecture of these computers and controls boggles me...
they are all on a party line...
a "one-wire-to-rule-them-all" design.
so each module has to sort through the stream of electrons and decide which one to ignore and which one to obey?

 

[njg-agt]

The architecture of these computers and controls boggles me...
they are all on a party line...
a "one-wire-to-rule-them-all" design.
so each module has to sort through the stream of electrons and decide which one to ignore and which one to obey?

That is just an old hat distributed computing system, software & hardware techniques that have existed in one form or another for many decades.
Standard networking has the ability for a data generator to send data to a specific recipient, or to all recipients, or to a set of recipients... both "local" (on the local wire) or (via a "router/gateway", remote). Not really significantly different in concept (or implementation) from what our phones do today.

 

[BS8899]

According to Lucid, the electronic modules communicate via an "Ethernet Ring". The Ethernet archtecture has been around since the 1980s and is used extenstively in computer networking. There is no dedicate wires that connects one electronic module to another or one computer to another. It is a network. Data are tarnsmitted via the Ethernet ring. This cuts down the amount of wiring required. Howver, (akin to a highway system where cars can get on and off the highway to different destinations, there could be collisions and congestions) Ethernet traffic are susceptible to delays and corruption (electrical noise, packet corrumptions, inappropriate protocols, etc.). Thus, how well the Ethernet ring performs is a function of the system archtecture and system validation.
I don't know enough about the Lucid Ethernet Ring to offer any inight. Again, system validation is key to flush out these issues.

 

[MoniputerLM]

Great hardware and communications architecture will not fix bad systems design and software.