Hiring or looking for a job in Digital Health? Check out our Digital Health Jobs board

Articles Blog

November 11, 2022

We Got Audited By a Competent Authority. Here's What You Need To Know.

Dr. Oliver Eidel

We got audited by a competent authority - the LaGeSo in Berlin to be exact, as that’s where our company is located. Let me tell you how it went and what you need to know if you ever run the risk of getting audited by one (which is.. always, if you’re a medical device manufacturer in the EU). And, by the way, this probably is the first public report of a competent authority audit, ever. Cool!

Episode 1: The Letter In The Mailbox

It all started with a letter in the mail in which the LaGeSo announced their audit and set a date which was around 2.5 months in the future (quick aside for readers in developed countries: While Germany counts as a developed country, it still runs on snail mail. Ready for a quick joke? How do you identify a startup founder in Germany? By the fact that they own a printer and constantly have to drop by the office to empty the mailbox).

Back to the letter. It also stated that we’d have to confirm that audit appointment within two weeks. They also set another deadline for a prior hand-in of our documentation which was around 1.5 months in the future. All of this sounded pretty reasonable. Cool!

Side note #1: In theory, these time frames would allow you to create an entire Quality Management System (QMS) and Technical Documentation (Techdoc) from scratch with backdated dates and submit that in time. But that would be highly illegal and while I’m sometimes in favor of taking reasonable regulatory risks as a startup, this is far outside my boundaries for risk tolerance. Still, the possibility is there and a company could theoretically game the system that way. All I’m saying is that it’s not a “true” unannounced audit due to this prior notice weeks ahead of the actual audit date.

Side note #2: You might be wondering why the hell are we getting audited anyway - aren’t we only a consulting company? Good question, Jedi! We actually also act as a legal manufacturer of medical devices. In other words, you can (sort of) throw your medical device software over the fence to us and we bring it to market, handling everything from documentation to customer support. So, in the context of this audit, we’re just another medical device manufacturer. But that also makes it very interesting, because we’ve set up our QMS and technical documentation using our own free templates and our eQMS software, Formwork. So we can use this audit to get some real-world auditor data on how well those templates and Formwork do. Cool!

Now, I’m sure you’re expecting me to rant a lot, just like everyone expected The Rings of Power to be a great Series on Amazon Prime Video. And just like with the Rings of Power, let me dampen your hopes a bit, albeit by a little less: I will be ranting less than usual.

That’s because I know how state institutions, just like hospitals and large enterprise companies, sometimes are willing to improve and act faster, but they can’t because any change within their institution has to be synced across a complex web of stakeholders and interdependent processes. So I’ll try to be more tolerant here. And no, they didn’t force me to be this way. That being said, I’ll still be ranting.. a bit.. we’ll see.

Episode 2: The Coworking Space Situation

So here’s the first rant (that didn’t last long, did it?).

Wait, before I start ranting I’d like to mention that the auditors accommodated our request to move the audit by a few days because we already had made holiday and vacation plans. I thought that was really nice because that’s not necessarily something a company can expect.

But the next problem was that our company is remote-first. Its legal address is at a coworking space which recently scaled down its room capacity significantly “due to Covid” and probably some suboptimal financial planning in which they wrongly projected their growth to transform them into the next WeWork.. then again, WeWork didn’t go so well, so maybe they got it right. (For the record: We’re still grateful to be their customer, the coworking community there is cool, even though many rooms are now located in the cellar. Yes, I’m serious)

Back to the audit. We needed to find some sort of solution with the LaGeSo auditors because meeting at our actual company address was out of the question - or, well, it would have been really awkward because we’d have to do the audit in the common area. Besides being in the cellar, the main problem here would have been that it is, well, a common area. Imagine that as an observer. Coming back from lunch, you’re comfortably shuffling your Birkenstock-wielding feet back to your desk, while you come across some people in a cold sweat because they’re in a middle of an audit. Let’s not let that happen, shall we.

So I reached out to them and explained the situation. I offered two alternatives: We could do the audit remotely. Or, if they really wanted to meet during Covid (this was mid-2022), we would rent an external meeting room on our costs and send them the address.

Now, here’s the first indication that state institutions, just like Universities and large enterprise companies, are by no means rational actors. A rational actor would have said “yeah, you guys are a software company, so there’s no need to see your office. Let’s do a remote audit”. Alternatively, I could have expected a state institution to say something like “Yeah, you guys are a software company, so there’s no need to see your office. But we need to see it anyway for a weird compliance reason”. And I would also have understood that, sort of. But something else happened which I didn’t expect. They replied: “Yeah, you guys are a software company, so there’s no need to see your office. So let’s meet at the external meeting room which you offered!”.

Rationally, that made no sense. Why meet at another random meeting room if meeting at our office was already not necessary?

It later turned out that this was all due to the certified video conferencing solution. You see, they are a state institution, therefore they can’t just use random video conferencing solutions like Google Meet. That’s because they have their own compliance and data protection requirements. Therefore they can only use their own certified video conferencing solution which was provided to them by a certified video conferencing provider.

The main problem of the certified video conferencing solution apparently was that it doesn’t support sharing your screen. Yes, I’m serious: You could not share your screen with it. Therefore, understandably, conducting remote audits with this certified video conferencing solution was very cumbersome, because you literally had to say “okay, now everyone open document X and scroll to page Y, paragraph Z”. Crazy!

Side note #3: If OpenRegulatory ever fails as a business, I’ll attempt to found a certified video conferencing provider which sells certified video conferencing solutions to state institutions. I feel like that would get very close to producing unlimited passive income for me while I wouldn’t have to provide anything of value. But OpenRegulatory is profitable right now, so let’s cautiously place that idea on the backlog (at the top!).

Am I ranting here? I’m actually not sure. I feel like probably 100% of the employees at the LaGeSo dislike the certified video conferencing solution, but some weird internal requirement gave them no other choice. So who’s at fault here? The German bureaucratic system? Weird data protection requirements? Did the certified video conferencing provider kidnap someone’s dog? Hard to tell as an outsider. Maybe we can find a certified whistleblower to enlighten us some time.

Episode 3: The OpenReg GmbH Folder

As the first deadline approached, my colleague Sven submitted all relevant documentation to them. It was good that Sven did this, because by now he has way more regulatory hard skills than I do. The second date approached, so it was time for the audit!

We met the auditors (there were two) at the external meeting room. We greeted each other and got started.

First off: They were really nice and professional. I think that’s often under-appreciated: I really would be scared of having unprofessional or even corrupt auditors. I could imagine that happening in other countries. Luckily that wasn’t the case here. So that’s cool.

They had sent us a draft agenda which started with a short company presentation by us, so I did that because that seemed to be the thing a CEO should do. Instead of the allocated 15 minutes, I was done after two or so. Next, we moved on to the actual audit.

I was already wondering what was contained in the large folder which one of the auditors was lugging around. As she finally laid it on the conference room table, I noticed that it had our company name on it. Enter: The OpenReg GmbH folder. (By the way, when we registered our company, we couldn’t name it “OpenRegulatory GmbH” due to German bureaucracy reasons. Don’t get me started.. another time). It turns out that this folder contained printouts of all documents we had sent to them.

Okay, let’s backtrack a bit. Sven had sent them documents before the first deadline, right? How did that go actually? Well, Sven prepared everything in an orderly fashion: There were PDFs exported from our eQMS, Formwork, and neatly sorted into folders on our file system. Those folders were important, because otherwise everything is a jumbled mess. You e.g. want your CAPAs to be in the CAPA folder and not somewhere else. Time to send them via email. Now, what does a rational human being in the age of the internet do? Right, create a zip archive so that the folder structure is preserved, and send the zip archive via email.

That’s what I did. Some time later, one of the auditors sent me an email “that she had been notified that their email system had rejected an email by me” (mind you, I hadn’t received any notification). She told me that their IT security policy didn’t allow receiving emails with zip archive attachments. So either I’d have to attach each PDF separately to the next email, giving up the folder structure and creating a jumbled mess, or she could create a shareable folder within their certified file sharing system (I know what you’re thinking right now). I opted for the jumbled-mess email solution, threw all PDFs into one email and sent it off into the matrix. It seems to have worked, because it explains the existence of the OpenReg GmbH folder which we’ll come back to now.

So the folder contained a printout of all PDFs we had sent them.

Why?

One of the auditors explained it to me: While competent authorities in some German states have started trialing an electronic document management system (let me guess: a certified electronic document management system), in Berlin they haven’t had the chance to use it yet. Therefore everything is still paper-based. In other words, all company-specific documents must be printed out and filed in the respective company’s folder.

Ready for the next one? This also applies to emails. So they literally have to print out each email we send them and file it in our company folder. I’ll just leave this here, you can do the ranting.

Side note #4: The LaGeSo in Berlin, specifically the department for medical devices, is run by a gentleman who has an unusually high amount of industry and software engineering experience. We need more of these people in this industry. I can imagine he’d be pushing them heavily towards the electronic documentation system and other innovations but is hampered by bureaucratic processes outside of his control. So I really hope that he sticks around and doesn’t give up.

Anyway, that explains the OpenReg GmbH folder. One of the auditors had gone through every (!) document and highlighted sections with a marker. I mean, I had expected them to skim them, but it seems like they read every single word (or got close). Okay. Awkward silence. That brings us back to the audit.

Episode 4: The Very Minor Findings

The two auditors opened the session by stating something along the lines of “so we had a look at the documents you dudes handed in beforehand. All in all, they looked pretty good, better than at some other companies, so we can already tell you now that everything is mostly okay and we only have minor things to discuss”. They had made a list of their findings which they wanted to discuss with us next (why discuss those in person and not asynchronously via email? Why not during a video call? I know, I know).

So we got right into it. One of them actually had installed the software (it’s an app) on their smartphone beforehand to try that out, so that was a welcome surprise - typically auditors don’t install and try out the medical device software (yep).

Before I get into the content of the findings, let me tell you that they had a ginormous amount of tiny findings which are near-ridiculous to even talk about. My favorite one was this: In our SOP for Post-Market Surveillance, we had the following sentence:

Ideally, the responsible person continuously collects information on all the categories described below […]

The auditor mentioned it, looked at us and said “you know, I don’t like the word ‘ideally’ here”.

Awkward silence because no one immediately understood what to do or say next.

I break the silence with: “Okay, so what do we do? Just drop the word ‘ideally’? Just say so. It’s not a big deal”.

Auditor: “Yeah”.

So we changed it to this, as you might have predicted:

The responsible person continuously collects information on all the categories described below […]

I mean, sure, that phrasing is better, and requesting that change is fine, I guess? But is this really what we’re here for? Are we here to discuss specific words in specific sentences? This somehow feels a bit like missing the forest for the trees.

You can find a full list of audit-triggered changes to our templates here. Be warned, it’s quite lengthy. I’ll just throw a list of bullet points at you without going into further detail:

Instead, let’s get into some findings which triggered more elaborate discussions.

Episode 5: Collecting User Emails

The medical device we’re talking about here was a smartphone app used by patients. It had, in my opinion, a pretty cool and rare property of not having a backend. In other words, all data which users enter remains on their smartphone and doesn’t get sent anywhere. While this of course limits us in features we can build, it’s very beneficial from a data protection standpoint because we simply weren’t handling any patient data on our end.

Enter the auditors. In their opinion, all medical device apps must collect contact data of each user, typically an email address, for post-market purposes, e.g. so that we can contact all relevant users when a bug is discovered which could lead to patient harm.

The discussion went along these lines:

Auditors: “So how do you inform users if you discover a critical bug in your software?”

“We could publish a software update and also update the app store description that a critical bug was fixed.”

“But what if users are currently backpacking in Thailand and don’t have access to the internet?”

With this reasoning, they wanted us to collect email addresses of all our users.

Side note #5: If users don’t have internet access in Thailand (which seems unlikely), why would sending them an email solve this problem?

Side note #6: To add a truly unreasonable level of comedy to this situation, I’m literally writing this up while I am Thailand myself. Sometimes it really seems like, when it comes to audits, the universe is playing jokes on us.

Regardless, this whole “collection of all user emails” thing was new to me and this also marked the first time I became slightly enraged. I was like “there’s no guidance which explicitly says that, also, MDD class I devices tend to be low-risk. And finally, what might happen next is that we get audited by the data protection people who will go on to complain that we collect email addresses of all users”.

While the auditors took my reply seriously and replied in a polite way, they remained unmoved and pretty much stated something along the lines of “well, that’s our own internal rule in our department so we require all medical apps to do that”. So that was that and we had to build a backend for our app.

Side note #7: Wouldn’t that be a significant change to a legacy MDD device? Well, technically yes, but as it’s caused by the same auditors who would also assess such changes, we’re in a weird catch-22 situation where everyone just nods and implements the change.

Later that night, I thought about this again. I mean, damn, there are so many more risky hardware medical devices out there which don’t collect any sort of user data. Like contact lenses and contact lens solutions. I mean, that stuff can be seriously risky to people’s health if its quality is compromised, or? What would you say is more risky: A contact lens solution causing someone to go blind, or a buggy class I medical app displaying some wrong data? Why does the medical app have to collect email addresses, at least in Berlin, while the contact lens solution doesn’t?

I feel this is 1) unfair, 2) an unnecessary overreach of the auditors and 3) not backed up by any MDR requirement or guidance.

There are less invasive methods of reaching out to your users in case of a malfunction, like publishing it on your website, the app store entry and/or the BfArM database. That’s how non-software manufacturers do it, right? Sure, you don’t reach all users that way, but that’s the way things currently are, or?

Episode 6: Don’t Call It FMEA

Another point which triggered me in a brutal way was their weird opinion on using an FMEA for risk analysis. Quick reminder: FMEA stands for Failure Modes and Effects Analysis. We have a template for it, and we used it. This is a pretty standard way of doing a risk analysis for medical devices.

Or so we thought.

The auditors were like: “We don’t really like the word ‘FMEA’”.

I was like: “?!?”

“So what do you mean?”

They: “Well, for medical devices, you don’t really do an FMEA, right? So don’t call it that way. In our department, we just don’t like to call it that way.”

I: “Dudes, you can’t be serious. Some time ago I sat in a class IIb notified body audit and the auditor explicitly agreed with our approach to do an FMEA - he literally said, yes, do an FMEA, that’s great. And I’m sure the 14971 mentions the word FMEA somewhere. But now you’re telling me we shouldn’t do an FMEA because your department sees things differently?”

They: “Yes”

I: “So what do we do instead? And how do we call it? Tell me, because you lost me completely.”

They: “Well, just call it a ‘risk analysis’. Do that instead”.

I: “So how does that differ from a FMEA?”

They: “Not much”

I: “So we just rename our FMEA to ‘risk analysis’?”

They: “Kind of”

At this point I didn’t want to press the point any further because this felt like discussing Jesus with Jehova’s Witnesses. Their department had some sort of weird understanding that a medical device risk analysis must not be called FMEA, and they weren’t willing to move away from this. So I just left it there.

Later on, I skimmed the ISO 24971:2020 (which used to be the annex of the old ISO 14971:2009). There, the method of using a FMEA actually has its own section on page 40, section B.5. Crazy, right? So, at the very least, this is proof that the auditors were not aware of the content of the 24971 and have come up with their own naming conventions (“don’t use the word FMEA”). At worst, they have a wrong understanding of risk analysis for medical devices.

My colleague Sven later added an observation which might clarify things: One of their reasons was that the FMEA only covers product-specific risks while a “risk analysis” would also cover process-specific risks. We got away by including process-specific risks in our FMEA. So.. all was good in Middle Earth (exploding head emoji)? I’m still unclear on this and the simple statement of “don’t call it an FMEA” causes a lot of unnecessary confusion.

Episode 7: What’s SOUP?

We were already nearing the end of our audit. We had briefly touched upon the SOUP list of our device (SOUP pretty much means third-party libraries) when one of the auditors asked me: “By the way, what’s SOUP?”

The gravity of this cannot be understated. This is like when a test pilot goes flying in a new airplane and, after landing, asks you “by the way, what’s a propeller?”.

Still, this is no time for ranting and, on some level, I actually think it’s a beneficial and constructive thing if people are open about gaps in their knowledge. So I gave them a quick tutorial on what SOUP is (similar to my video here), both from a regulatory and software engineering perspective.

But it still makes you wonder whether this audit provides any tangible, technical usefulness for the actual software - see the conclusion further below.

Anyway, after that, we all nodded and concluded the learning experience.

What Did We Cover?

It truly was a long audit - we spent the entire day from around 9am to 3pm in that meeting room. I’ve tried my best to condense it down into the most relevant parts. Still, coming at this from another angle, here’s a quick summary on what we covered. Topic-wise, the relevant parts were:

And even though it may seem that I did most of the talking, that wasn’t true; Sven did. He walked them through our documents and answered their questions. I only jumped in when things triggered me, like the FMEA (non-)naming convention.

Time-wise, here’s an estimation how we spent our time:

Conclusion

First off: Thanks for reading this far - damn! That was a long post.

What do I think of this audit? While my points above may sound overly negative, I think a more differentiated conclusion is appropriate here.

It is my personal opinion that our current way of auditing medical device software is inherently broken - instead of documentation-based audits by generic auditors, we’d need deep technical audits by software experts, down to the source code level. This not only applies to competent authorities, but also to notified bodies. Right now we don’t have that, so we’re left with the suboptimal solution of documentation-based audits.

Keeping those limitations in mind, this audit was reasonable. Sure, there certainly was a tendency of diving (too) deep into specific phrasings, but, all in all, this audit achieved what it set out to do, namely checking for whether our documentation was compliant with the applicable regulation.

And although I did end up ranting a bit, it’s really worth highlighting that our auditors were friendly people and the discussions were polite and constructive (even when I got slightly enraged). They were open to discuss some of the findings I didn’t mention above and they actually ended up agreeing with our opinion. I thought that was really constructive. I believe they really did have the right intentions of trying to make medical devices safer, within the limitations of what documentation-based audits can accomplish.

And not only within those limitations, but also within the technical limitations of their institution: As mentioned in the first few episodes, their IT infrastructure (remember the certified video conference solution?) severely lags behind what is state of the art, and this is sad to see. I’d really love to see state institutions use the most polished technology to enable them to do their job in the best possible way, and we’re very far away from that.

That being said, we spent a whole lot of person-hours pouring over documents and correcting all sorts of phrases while not looking at the actual software or even its code. But.. that’s not their fault, I guess. It’s the fault of the current system of regulating medical devices.

Oh, and how did Formwork and our templates do? Formwork passed with flying colors but, that being said, they didn’t look into it deeply. It could export documents and records as PDFs with e-signatures and that was probably sufficient for them. Regarding our templates, we needed to make a huge amount of mostly minor changes which you can view here on GitHub.

Anyway! I hope this was useful to you. I think this is literally the first public report of a competent authority audit, so I hope it’s another step towards making our industry more transparent and enabling startups to enter the market :)

On a slighty different note: You want to get your medical software certified under MDR but don't know where to start? No worries! That's why we built the Wizard. It's a self-guided video course which helps you create your documentation yourself. No prior knowledge required. You should check it out.

Or, if you're looking for the most awesome (in our opinion) eQMS software to manage your documentation, look no further. We've built Formwork, and it even has a free version!

If you're looking for human help, did you know that we also provide some limited consulting? It's limited because we are not many people. We guide startups from start to finish in their medical device compliance.

Congratulations! You read this far.

Get notified when we post something new.

Sign up for our free newsletter.

Dr. Oliver Eidel

I'm a medical doctor, software engineer and regulatory dude. I'm also the founder of OpenRegulatory.

Through OpenRegulatory, I've helped 100+ companies with their medical device compliance. While it's also my job that we stay profitable, I try to dedicate a lot of my time towards writing free content like our articles and templates. Maybe that will make consulting unnecessary some day? :)

If you're still lost and have further questions, reach out any time!

Need help? →

Comments

If you have any questions or would like to share your opinion publicly, feel free to comment below. If you'd like to reach out privately, send us a message.

Why Startups Fail Our Services Free Consulting Community
QMS Software List Notified Body Reviews UDI Video Course Conference 2021
About OpenRegulatory Join Us Digital Health Jobs
No Cookie For You Privacy Policy Imprint
No QMS on this planet will save you from creating crappy software.