A security certification is a snapshot in time, not a guarantee of ongoing protection, according to Mariia Bakhtina, a research specialist at Masaryk University.
What does a certification really promise?
A Common Criteria or FIPS 140 certificate confirms that a product was evaluated against specific security claims at a particular point in time. For Common Criteria, this means the product met the criteria the vendor claimed within a defined scope. FIPS 140, on the other hand, validates a cryptographic module at one of four security levels but says nothing about the surrounding application or system.
Despite this, buyers often misinterpret certifications as a blanket assurance of security. Common misconceptions include the belief that certified products have no vulnerabilities, that the entire product is covered, or that a higher certification level guarantees greater security. Bakhtina clarifies that a higher level doesn’t mean a product is more secure in general, instead it tells you how thoroughly the product was examined.
Tools for transparency
To address these challenges, Bakhtina’s team developed two open-source tools: sec-certs and SCRUTINY. Sec-certs transforms public certification documents into a searchable, linked dataset, making it easier for buyers, vendors, and security labs to access and compare information. SCRUTINY tests the actual product, verifying its cryptographic functions and performance against its certification claims.
Read Also: AI agents now evade detection by mimicking humans perfectly
For companies purchasing certified products, Bakhtina advises reading the scope of the certification and asking vendors to confirm that the specific security function they’re buying is covered in the certification documents. If it’s not, independent testing may be necessary.
Understanding Certification Documents
Picture a procurement manager handed a 300-page certification document for a product they are about to buy. Luckily, certification documents (e.g., Security Target in CC or Security Policy in FIPS-140) are usually up to around 100 pages, not 300. Covering all of those could take more than 1 hour if the procurement manager is not familiar with the certification scheme details (e.g., how the documents are organized).
A procurement manager faced with a certification document might initially search for key details like the product name, version, assurance level, evaluated configuration, and out-of-scope declarations. These elements define the boundaries of the security claims. However, the document’s usefulness diminishes as it only provides a historical snapshot. It doesn’t account for vulnerabilities discovered post-certification or changes made to the product. In her team’s analysis, about 6 in 10 vulnerabilities matched to certified products, smartcards aside, were disclosed after the certification date.
Searchable Certification Documents
When certification documents become searchable, buyers can quickly answer questions that previously required consultants or extensive reading. They can verify a certificate’s validity, check for newer certifications, and explore referenced documents like certified hardware or Protection Profiles.
Read Also: Shoppers Reluctant to Trust AI for Unsupervised Purchases
This searchability also benefits vendors and security labs. Vendors can monitor their entries, track products dependent on theirs, and analyze competitors’ certifications. Labs can identify expiring certificates and potential customers for re-certification testing.
SCRUTINY: Testing the Real Product
It generates a profile for comparison and a readable report.
When presenting findings, SCRUTINY uses clear language and evidence levels to avoid panic or dismissal. It categorizes results as “nothing found,” “needs follow-up,” or “reproducible with strong evidence.” We also map them to vulnerabilities that potentially affect the certified products, and those matches are correct only with roughly 89% precision.
Leave a Reply