CONTACT US

X-ray News

Fireside Chat with the Xperts: Data Fusion

Fireside Chat 02102021

If data analysis is going to drive the next industrial revolution, then data fusion will be it’s self-driving vehicle.  Host David Kruidhof, Dr. Bill Cardoso, and their guest, Dr. Glen Thomas, explain data fusion as it relates to manufacturing, it’s benefits and long term implications.

It’s a fascinating conversation that effortlessly weaves science fiction and  history in making the fundamentals for data fusion and its implications on contemporary manufacturing accessible.  Using silos of data in decision making has been useful, but combining isolated data sets through data fusion will be revolutionary.  Listen carefully, as this might be the only time you’ll ever hear Dr. Cardoso suggest that you shouldn’t be looking at your X-ray images.

Enjoy the video, and then reach out to us directly with any question.  Register for upcoming Fireside Chats with the Xperts and view our archives here.

 

Transcript:

David Kruidhof:
Welcome back for another Fireside Chat with the Xperts. Today I have Dr. Bill Cardoso, me again, and Dr. Glen Thomas back with us, today we’re going to be talking about data fusion. So welcome gentlemen, good to have you guys.

Dr. Glen Thomas:
Good to be here again.

Dr. Bill Cardoso:
Good to be here.

David Kruidhof:
So data fusion is a neat topic. I think some people use it differently than what we’ll be talking about so I just want to make sure that we all started off with the same understanding. So Bill, why don’t you give us just a one or two minute, high-level view of what data fusion is?

Dr. Bill Cardoso:
It has to be one or two minutes? Can’t be longer?

David Kruidhof:
Yeah. No. Timer, now.

Dr. Bill Cardoso:
Okay. Glen’s at the clock.

Dr. Glen Thomas:
Yep.

Dr. Bill Cardoso:
So data fusion is a fairly mature concept that relates to combining the different data streams to achieve a specific goal. Right? So instead of looking one data stream or one type of inspection, one type of data separately from another one and another one, what you do is you fuse them together, and there are different algorithms that you can use to combine that information. And out of that information, the combined data you define, it can be design criteria, it can be pass/fail criteria, on the combined data to drive information for your project. So it’s been around for a while. I mean, it goes back to… For example, you’re looking at objects on a field, and let’s say you are doing surveillance from an airplane, if you’re looking for tanks, right? On the enemy territory, and you want to make sure it’s a tank, not a cow, which, believe it or not, is actually not trivial, to make the distinction. You might want a fuse infrared data stream with a visual data stream to better give you that information. Right? Each of these infrared or optical, visible spectrum, each one of those data streams alone might give you a certain level of accuracy, and combined, they usually give a much better accuracy. So that’s data fusion in a nutshell.

David Kruidhof:
Okay. So instead of relying on one source at a time to give you information to make a decision, you’re weighing these different values that you’re getting, weighing the different data inputs, and combined, making a decision.

Dr. Bill Cardoso:
Yes, one plus one equals [crosstalk 00:03:15].

David Kruidhof:
When infrared says, “Hey, that looks like a cow, but I’m 80% sure that’s a cow.” And then optically, it’s, “Hey, I don’t know, 50% of it’s cow.” Then you can make a decision based off that.

Dr. Bill Cardoso:
Exactly. Yeah.

David Kruidhof:
Combined instead of just independently, makes sense. So how is that being used in manufacturing, that you’ve seen so far?

Dr. Bill Cardoso:
So we’ve had deployments for example, where we fused optical data with X-ray data streams to improve detection of counterfeit components. Right? Which is an example you clearly have. So what you do is, every time you do inspection, and I’m sure people on this call are familiar with it, but every time you do inspection, look for a pass/fail criteria, ideally you have a very clear definition where to put a threshold. Right? So everything lower than this threshold is fail, everything above this threshold is pass. Right? Unfortunately, nature is often against us as engineers, so what nature gives us is normal distributions, bell-shaped curves, so instead of a clear place to put a threshold, you usually end up with two overlapping bell-shaped curves, one is the pass and one is the fail population. Right?

Dr. Bill Cardoso:
Ideally, what you want to do is to have these two bell curves as far apart from each other so you can put a threshold somewhere in between and you have a very clear distinction of pass and fail. Again, oftentimes what you end up experiencing is that these two bell curves are close to each other so details overlap. What does it mean? That overlap means that if you put a threshold here, sometimes you’re passing something that should be failed, sometime you’re failing something should be passed. Right? Because you have that tail of the two overlapping bell curves. So data fusion becomes very powerful, because you have the two bell curves for optical inspection and now you have two bell curves for X-ray inspection. Right? So now we have two sets of bell curves, two sets off a pass/fail criteria, and what data fusion allow us to do is to say, “Okay, in the optical, this is marginal, how’s it doing with X-ray?” Right? And you look to see if that is also marginal or if it’s more towards a pass or more towards a fail. Right? So based on that, now your pass/fail criteria is not two-dimensional, we have three-dimensional space to map where you’re going to pass or fail things, which improve your ability to reduce false positives and false negatives. Does it make sense?

Dr. Glen Thomas:

David Kruidhof:
Yeah. I think, historically speaking, electronics, for example, you have a circuit board that’s starting in a certain spot, bare board, it’s going to go through a process. Right?

Dr. Bill Cardoso:
Yeah.

David Kruidhof:
So you’d stick some solder paste on there and it goes to SPI, and SPI is going to say, “That’s good. Go to the next line.”

Dr. Bill Cardoso:
Correct.

David Kruidhof:
That’s pretty much all you get out of it, right? Or it says, “Hey, that’s fail, reprint it.”

Dr. Bill Cardoso:
Correct.

David Kruidhof:
And it’s going to go through the whole thing, and then it hits AOI and the AOI machine says, “Hey, that solder joint doesn’t look good,” Or, “This component’s flipped around,” or something. And that alone is its own kind of island where you make decisions off of. Right? The same for AXI. So the idea with data fusion is that you start saying, “AOI thought this looked bad, maybe it’s a cold solder joint, what does it look like under X-ray?” And if you can fuse that data, you can make a much more educated decision rather than just alone.

Dr. Glen Thomas:
You could use this concept to strengthen your inspection when your inspection is barely meeting the minimum. Right?

Dr. Bill Cardoso:
Yeah.

Dr. Glen Thomas:
Due to circumstances of the inspection itself, not necessarily the modality, but the component may be more difficult to image, the sample is difficult to image, so you have a suspect, no matter how you try to image the product, you have some suspect results. Right? So the more data streams you can add into the equation, the more positive you’re going to get on all of those dubious results. So you can essentially… They’ll all add up to one, whereas singularly, they may be .25, right?

Dr. Bill Cardoso:
Yeah, exactly. I mean, it’s very easy to get 80, 90% accuracy in the single modalities nowadays. Right? SPI machines are highly evolved, so they basically are very good at getting you the most of that Gaussian. Right? The bell curve that is not overlapping with the other one, for the pass side, for the fail side. So the 90 plus percent for all those machines can be achieved today with the high-end systems available in the market. The really tricky thing is, it’s the minutiae returns, to get to the end, to get to the point where we start getting to the overlapping bell curves, that’s where data fusion plays a role. Right? The problem we have is that not a lot of people are doing… I don’t know anyone doing it right now, cross-modalities. Right? It’s not trivial, and that’s something that we’ve been doing with optical and X-ray for a while now, and I think there’s a lot of potential to market for nondestructive testing as we can correlate or fuse laser scanning data with CT and other types of X-ray inspection. I think there’s quite a bit of potential there, and that’s what we’re exploring right now. To people interested in figuring that out, yeah, give us a call, it’s what we do.

Dr. Glen Thomas:
Great.

David Kruidhof:
Yeah. So combining optical and X-ray, we just mentioned electronics, but outside of that, we’ve done, I don’t know how much we can talk about it, but barcode, not just barcode scanning, but integrity of it.

Dr. Bill Cardoso:
Yeah. Yeah.

David Kruidhof:
Integrity of things on the widget that needs to be inspected or looking at, is it pass this test? Is it passing that test? And maybe that’s not quite using it, at least the example I’m thinking of, but it’s a matter-

David Kruidhof:
… of taking all these things, right? Right? Lots of data streams together and saying, “Hey, your product gets a score of seven out of 10 or 9.5, or overall.”

Dr. Bill Cardoso:
And infrared or ultrasound, eddy current, I mean, all these different modalities can be fused with X-ray and optical inspection to augment the probability that you’ve got to make the right decision. Right?

David Kruidhof:
Yeah.

Dr. Bill Cardoso:
And it’s old school to think of each one of data streams as an independent decision-making process. Right? That’s old school. Now, with the computing capabilities we have, it became silly not to take advantage of the different data streams you… Because at the end of the day, what you want to do is transform data into information. Right?

David Kruidhof:
Yeah.

Dr. Bill Cardoso:
Data that you don’t use has a very defined aim, it’s noise, right? It’s useless. If you don’t use it, it’s useless and it’s noise. So what we were talking about here is to design an algorithm that can turn data into information, and information is data you actually use and you actually employ to improve your processes and improve the overall quality of your manufacturing. Right?

David Kruidhof:
Right. Yeah. So have it actionable, or actions come out of it. Right?

Dr. Bill Cardoso:
Exactly.

David Kruidhof:
To have a stream of numbers coming at you, you can’t do much with that, but once you can turn that into something that I can read, I can understand, I know what to do with this product now, maybe I know how to take it back and go improve that. One of the things we had talked about was, you’re going through, again, back to electronics, you put down your solder paste in the board, SPI images the whole thing. Right? You get all your data out of that, but where exactly is the threshold, what is good? What is bad? Well, if you can now take your outputs at the end of the line and AOI gave it a passing score, AXI gave it a passing score but it’s marginal over on the SPI side, you can sort of train your earlier inspection points based on the output and results, if you can track that product through the whole process.

Dr. Bill Cardoso:
Exactly.

Dr. Glen Thomas:
Right. And so we can automate some of those processes. Right? You could automate improvements in the process based on the thresholding.

David Kruidhof:
Yeah. Yeah, exactly. That’s what a lot of people are looking for. Especially now where they’re trying to keep people apart, it’s even more of a push to automate things, make things.

Dr. Glen Thomas:
With that in mind, what’s the ability to augment data fusion with AI?

Dr. Bill Cardoso:
So, yeah. That’s a good question. When we talk about that algorithm that’s going to fuse the data, yeah, AI plays a huge role in that algorithm, the black box that we design to fuse this data. And what we’ve done, it’s very powerful, is to adopt a training process to design the data fusion algorithm. What does it mean? It means that you can actually look at all the data streams and train which data streams, which extreme cases that were previously ambiguous, were hard to determine, you can train AI to identify those extreme cases and give you a pass, an accurate pass, or fail result. Right? So let’s put that into an example, right? To make it less cryptic. You have a board that passes SPI, it’s not 100% but it passes, then it goes from AOI after pick-and-place and passes that as well, and then it passes AXI. Right? And at each one of those steps, the pass was not with flying colors, it was somewhat marginal, but it passed each one of those stages. And you know that that specific board with that set of results, even if it passes ICT, In-Circuit Test, it’s going to fail in the field after six months. Right?

Dr. Bill Cardoso:
So you can imbed that level of knowledge, that level of information into your AI data fusion algorithm to say, “You know what? Even though it passed everything, let’s fail this board.” Right? “Let’s take it aside and rework it, let’s do something with it.” Because they know that there’s a X percent chance that that board is going to become an RMA. Right? So that’s not possible without data fusion, that’s not possible if you’re not looking at your manufacturing process or your SMT line as a fluid process that gives you information. I mean, we know that a lot of times our customers collect data and put it in a box and they don’t do anything with it. Right? It’s not easy, right? And a lot of these machines, SPI, AOI, and ICT machines, they were designed to crank up data and put in a box, they’re not designed to crank up data and share with other people. So there’s work to be done there. Right? As a community of companies that sell equipment, we’re challenged, because at same time you want to collect data and provide information to the users of the data so they can turn to information and improve their process, at same time, we want to keep that information to ourselves. We don’t want to share it with our competitors, so we have the challenge going on now.

Dr. Bill Cardoso:
But I’m a big believer that considering the SMT line as a fluid process that has transparency, and with transparency we have the ability to look at how each machine is performing, what kind of results we’re getting, so that at the end of the day, our customer has the ability to stop a board that will fail, even though it’s not working now. I think it’s very powerful, that’s the future, right? I know, as a community, we’re not there yet, but we have to, very soon.

Dr. Glen Thomas:
Great, so with that thought in mind, what is the feasibility of adding a data stream from the field, from RMAs or from service technicians? That would essentially be an analog data stream, right? So I could see products that have long life cycles and long iterations, let’s say we’re in iteration 11, this has been a 10 year period that we have been building this product, we still don’t have a grasp on some aspects. Would feeding that RMA data and feeding that service data and failure data in the field, or from the field, back into the algorithm and back into the fusion, what would that benefit be? Would there be a benefit to that?

Dr. Bill Cardoso:
The benefit is that it gives you almost like a Minority Report situation. Right? You’d be able to foresee the future, you’d be able to know what’s going to fail before it fails. Right? And again, a lot of these SMT lines today are running at 90 plus percent accuracy, but they don’t really know if that’s true, you’ve done 90% SPI accuracy or 99% SPI accuracy means anything. Right? It’s means something for the SPI machine, but what does it mean for the product that the customer is experiencing? Right?

Dr. Glen Thomas:
Right.

Dr. Bill Cardoso:
So right now we have silos of data being generated in our factories, and I think the great disruption that’s happening, slowly, but it’s happening, is that transparency. So if data can permeate from system to system and this overall data collection can happen… I mean, another analog, or semi, quasi-analog data stream is inventory control. Right? Is the stuff you put in your boards, are they counterfeit or not? Right?

Dr. Glen Thomas:
Right.

Dr. Bill Cardoso:
What’s the chance that the stuff you put on your is actually legit, right? The manufacturing line has very little access to the inventory and vice versa, right? Because it needs access to your ERP system, right? I mean, so that ability to start putting together data streams is critical. Right? I mean, COVID helped a little bit, nudged companies who realized that, “Wait a minute. Now I need to have fewer people on the floor, perhaps I should start looking at automation.” But a lot of the automation conversation we have is a much lower level, is that business intelligence that we have, it’s figuring out how good your machine is from an efficiency perspective. Right? Glenn, you know this better than anybody else, is you basically extract how good operator A is compared to operator B at finding errors. Right? And having those dashboards, managing dashboard, those stuff what we have with TruView BI, but I think the opportunity is much bigger. Right? I don’t know, what do you think? Do you think we’re getting there, or will get there at some point?

Dr. Glen Thomas:
Absolutely, yes. Yeah, as the AI gets more accepted. And I think a lot of it is not necessarily the technology, we have the technology and we have the key algorithms, we have the capability, I think it’s acceptance on the floor, and it’s even manufacturing managers. And some of the mindsets need to change in the manufacturing arena to accept what that black box is giving them. Old school guys, they like joysticks, tried to convince them to use a keyboard, it’s almost impossible. So a lot of it is open-mind, companies with open mind and willing to accept change, and keep fighting that old-school resistance, “Well, we’ve always done it this way and it’s worked just fine.” Right?

Dr. Bill Cardoso:
Exactly.

Dr. Glen Thomas:
So as with any emerging technology, there’s always going to be some resistance and selective… And so the bottom line, the paycheck gets bigger, right? We ship more product, we have less returns, our company is more successful than it was two years ago based on that black box. Right?

Dr. Bill Cardoso:
So I mean, we struggle with the joystick question, right? Customer would like a joystick, can you imagine when we have to convince them that they only need to look at X-ray image at all?

Dr. Glen Thomas:
Exactly.

Dr. Bill Cardoso:
Right? And I mean, if you think about it, why do you have to look at an X-ray image? You have algorithms and a bunch of other technology that can make those decisions for you. Right? The X-ray image, that gray scale image that they see today is, at some point, going to be a thing of the past. Right? I mean, we’ll provide it as a comfort blanket, but at the end of the day, it’s fully automation. Right? If you have an autonomous system, it will become irrelevant or not important for decision-making process. And I don’t know if that might even be a generational thing, and especially when we talk about self-driving cars. Right? My kids have a much, or I would expect, they have a least… Their process to adopt, to be comfortable, in a driverless vehicle is going to be much less friction than what we have, right? Our kids are just going to say, “Yeah, this is awesome. My dad should drive? The schmuck?” No, I don’t have to drive, I just push a button and go where I have to go, right?

David Kruidhof:
Yeah.

Dr. Bill Cardoso:
It makes me feel sad they will never have the pleasure of changing oil and smell gas, and drive a V8 down the street, but I guess that’s the price of evolution, right?

Dr. Glen Thomas:
Right. Right.

David Kruidhof:
Yeah. And I think as we’re along the road to get to this point of AI and data fusion and that adoption, it’s critical that we continue to gather the data. Right? I mean, your example of sending the product out in the field and it fails in six months instead of the expected five years, if you look back, what you had to do at that point and say, “Okay, serial number 123 failed at six months, what do I know about that thing before it left the shop?” And then you go to your AI and you say, “Hey, here’s all the AOI images, here’s all the AXI images, the SPI data, the operator decisions that was made, the In-Circuit Test data, all of this resulted in a premature failure.” And you just throw it out the AI. Right? And then you just have to do that enough times where finally the AI is going to… It’s not going to tell you, “Hey, it’s going to fail because of that.”

David Kruidhof:
But when you’re giving it a ton of information, something that’s going to trigger and say, “Hey, this thing’s probably going to fail. I’m 75% sure this one’s going to fail at eight months.” And that’s the part that makes it hard to adopt AI, it’s not going to tell you why necessarily since it is that black box side of things. If you start feeding it that information now, you should have a pretty good data set a couple of years from now. Going to take time.

Dr. Bill Cardoso:
Yeah.

Dr. Glen Thomas:
Yes. So essentially we could apply the concept of planned obsolescence down to an absolute day, right? The day the warranty runs out or the day your contract expires, it’s going to break.

Dr. Bill Cardoso:
Yeah.

David Kruidhof:
The exact amount of solder I need on that joint to get it to break at 366 days.

Dr. Bill Cardoso:
Exactly.

Dr. Glen Thomas:
Exactly.

Dr. Bill Cardoso:
I mean, the inventor of capitalism, by Adam Smith, to the invention of the production line by Henry Ford, the companies are really good at collecting data. Right? And I think that was the previous industrial revolution. I don’t know, depending how you count it, it can be the third or the second or the fourth. And the next industrial revolution, which, in my book, is the fourth, will not necessarily prove data collection but it will prove data analysis. Right? Is using all the data that companies collect every day, and as I said before, that data dies in black boxes in the line, manufacturing lines around the world. Right? Is you gets that data out of those silos and make that data available so that AI algorithms can figure out how to improve manufacturability and quality with the data collection. Right?

Dr. Glen Thomas:
So the nemesis here is compartmentalization of data, right?

Dr. Bill Cardoso:
Yeah.

David Kruidhof:
Yeah.

Dr. Glen Thomas:
Quality has their compartment, incoming inspection has their compartment, manufacturing has theirs, engineering has theirs. So essentially we’re looking at multiple inroads into that box, but how do we get the data reliably to the person that needs it when they need it? I mean, I think that’s key. If I’ve sent out 10,000 failed products, they’re going to fail as soon as they hit the floor. Right? I’d really needed that data yesterday instead of tomorrow. Right? So how do we make that data accessible to more individuals in the company?

Dr. Bill Cardoso:
Yeah.

Dr. Bill Cardoso:
And also, how you need the data, right? Because just like that person that tells you, “Oh, isn’t water refreshing in a hot day?” When the person’s actually asking for a glass of water. Right? So a lot of times that’s how data gets delivered to decision makers. Right? “Oh, isn’t water refreshing in a hot, sunny day?” When the data’s actually telling you, “I’m thirsty, put a stop button because we’re going to start shipping bad product.” Right? And that’s something that can be topic of next… Your future of Fireside Chats, right, David?

Dr. Glen Thomas:
So with that thought, would that be diluting the problem or diluting the answer?

Dr. Bill Cardoso:
Or both.

Dr. Glen Thomas:
Or both.

David Kruidhof:
Both, talking about water, it’s going to dilute everything. All right, gentlemen. Well, we are at 10:30, we’ll have to stop here.

Dr. Bill Cardoso:
That was great.

David Kruidhof:
I appreciate your time, again. As always, pleasure talking with you, Glenn.

Dr. Glen Thomas:
You bet.

David Kruidhof:
Talk to you all again soon.

Dr. Bill Cardoso:
See, he doesn’t say pleasure talking to me, Glenn, just you. [crosstalk 00:29:14] that.

David Kruidhof:
No.

Dr. Glen Thomas:
Yeah, what can I say.

David Kruidhof:
Wasn’t this time.

Dr. Bill Cardoso:
All right, guys. Talk to you later.

Dr. Glen Thomas:
Thanks.

David Kruidhof:
All right.

Dr. Bill Cardoso:
Bye.

David Kruidhof:
Take care.

Related Posts

Fireside Chat with the Xperts: Or is it “S”perts?

In this week's Fireside Chat, we take the "X" out of Xpert.  With their guest, Ross Williams of HES Solar,  David Kruidhof and Dr. Bill...

Read More
Fireside Chat with the Xperts: Or is it “S”perts?

Fireside Chat with the Xperts: Surgical Tray Inspection

The Xperts are back at it, with their first fireside chat of 2021.  Who though surgical tray inspection could be so interesting, and the challenges...

Read More
Fireside Chat with the Xperts: Surgical Tray Inspection

Fireside Chat with the Xperts: NDT with Jeff Darby

On this week's Fireside Chat with the Xperts, we at Creative Electron are very excited to announce the addition of Jeff Darby to our team,...

Read More
Fireside Chat with the Xperts: NDT with Jeff Darby