Penetration Testing
Vendor Evaluation Checklist
How to compare pentest firms on what actually changes the outcome: who tests, how deep they go, the report, the fixes, and the retest. 30 checks, 12 KPIs, and a scoring rubric.
By Louis Sanchez (OSCP, OSWA, CISSP, CCSK), founder of Voke Cyber, a penetration testing firm in the Charlotte, NC area
What to Look for in a Penetration Testing Firm: The Short Answer
When you evaluate penetration testing firms, look for six things. First, a named, certified tester who also writes the report. Second, a documented manual methodology such as NIST SP 800-115, the OWASP Web Security Testing Guide, or PTES, not a scanner run. Third, written authorization and rules of engagement before testing starts. Fourth, detailed reports that work for both engineers and non-technical stakeholders. Fifth, remediation guidance specific to your technology stack, with a retest included to verify the fixes. Sixth, evidence that your compliance framework (PCI DSS, SOC 2, HIPAA, ISO 27001) will accept. Score each provider on the 30 checks below: a penetration testing vendor that scores 50 of 60 or more, with no hard stops, is a strong choice. The best way to compare pentest companies is to use the same selection criteria and due diligence questions for every one of them, which is what this checklist is for.
Most pentest quotes look alike: a price, a date range, and a list of test types. The differences show up after you sign. Did a senior tester do the work, or a scanner? Can leadership act on the report? Does anyone help you fix what was found, and prove the fix held? This checklist turns those differences into questions you can ask before you sign. It draws on NIST SP 800-115, PCI DSS v4.0.1 Requirement 11.4, and the PCI Security Standards Council’s Penetration Testing Guidance, plus what we have seen across more than 1,000 engagements. We are a penetration testing firm ourselves, so hold us to it too.
6 Evaluation Criteria for Choosing a Penetration Testing Provider
These six criteria separate penetration testing providers more than price does. Each has five checks; score each check 0, 1, or 2, then compare vendors side by side.
Who Actually Does the Testing
The single biggest driver of pentest quality is the person at the keyboard. Get a name, verify the credentials, and confirm that person writes the report.
Methodology and Manual Depth
A credible penetration testing firm names its methodology, tests business logic and authorization by hand, and exploits past detection. Scanners find the easy half.
Scope, Rules of Engagement, and Legal
Before a pentest starts, the vendor should provide written authorization, rules of engagement, proof of insurance, and a data-handling policy. That paperwork protects you too.
Report Quality for Two Audiences
Leadership needs business risk in plain language. Engineers need evidence and reproduction steps. A good report serves both without a translator.
Remediation Guidance and Retesting
Pentest findings only matter once they are fixed. Look for fixes specific to your stack, a review call with the tester, and a retest that proves each fix held.
Compliance and Audit Evidence
If an auditor, customer, or insurer will read the result, confirm the report meets their specific requirements before testing starts, not after.
Only have 15 minutes per vendor? Start here.
- Ask for a sample report. It shows more about a vendor than any sales call. Check it against Area 4.
- Ask who the tester is, then verify them. Look up their certifications with the issuer and ask to speak with them.
- Ask what the retest costs and what it produces. If the answer is “extra” or “a new report later,” factor that into the price.
Red flags in a pentest proposal
- The proposal names a firm but no tester.
- The sample report is a scanner export with a cover page.
- Every finding is rated High or Critical, with no CVSS vectors to check.
- Retesting is a paid add-on or is not offered.
- They cannot explain how they test authorization or business logic.
- “Continuous pentesting” pricing that, on inspection, is automated scanning.
The Full Checklist: 30 Questions to Ask a Pentest Vendor
Why each area matters, and the five checks to score in each.
1. Who does the testing: the question most buyers never ask
Two firms can quote the same scope and deliver very different tests, because the work is only as good as the tester. A proposal that names nobody usually means the work goes to whoever is free that week, sometimes a subcontractor. Ask for the tester by name, check their hands-on certifications through the issuer’s public credential lookup, and ask to speak with them before you sign. The PCI SSC’s Penetration Testing Guidance treats tester qualifications (certifications and practical experience) as something the client should evaluate, not assume.
- 1.1The proposal names the tester(s), and the same person who tests writes the report.
- 1.2Their hands-on certifications (for example OSCP, OSWE, OSWA, GPEN, CREST CRT) are verifiable through the issuer’s public lookup.
- 1.3They can describe years of offensive security work specifically, not general IT or compliance experience.
- 1.4The firm states whether any work is subcontracted or offshored, and where your data and evidence will be handled.
- 1.5You can talk to the tester directly before signing, and during testing.
2. Methodology and manual depth: where scanner-led tests fall short
A pentest led by automated scanners reliably finds missing patches and known misconfigurations. Scanners do not find broken authorization, tenant-isolation failures, or business-logic flaws, which are the findings that lead to real breaches in modern applications. A credible vendor names a public methodology (NIST SP 800-115’s planning, discovery, attack, and reporting phases; the OWASP Web Security Testing Guide; the OWASP Mobile Application Security Testing Guide; or PTES) and can explain how they test what a scanner cannot. PCI DSS v4.0.1 Requirement 11.4.1 goes further and requires an industry-accepted methodology that covers both application-layer and network-layer testing.
- 2.1The vendor names the public methodology it follows (for example NIST SP 800-115, OWASP WSTG or MASTG, or PTES) and, if you are in PCI scope, shows how it meets PCI DSS 11.4.1.
- 2.2They can explain how business-logic and authorization flaws are tested by hand, not just which scanners they run.
- 2.3Authenticated testing uses multiple roles, and multiple tenants where your product is multi-tenant.
- 2.4They attempt exploitation and chain findings to show real impact, within agreed rules, rather than stopping at detection.
- 2.5Every scanner result is manually validated before it reaches the report.
3. Scope and rules of engagement: the paperwork that protects you
A penetration test is authorized attack activity against your systems, so the paperwork matters. Before testing starts you should have a signed authorization covering every in-scope asset (including cloud and hosting providers whose policies apply), rules of engagement that define testing windows and prohibited actions, an emergency stop contact, and the source IP addresses the tester will use so your team can tell test traffic from a real attack. Ask how evidence such as screenshots, credentials, and extracted data is stored, encrypted, and destroyed after the engagement.
- 3.1A written authorization is signed by someone with authority over every in-scope asset, including third-party hosting.
- 3.2Rules of engagement define testing windows, prohibited actions, and an emergency stop contact.
- 3.3The vendor provides certificates of professional liability and cyber insurance on request.
- 3.4A data-handling policy covers how evidence is stored and encrypted and when it is destroyed.
- 3.5Testing source IP addresses are disclosed in advance so your team can separate test traffic from real attacks.
4. Report quality: what makes a good pentest report for leadership and engineers
The report is the only lasting artifact of the engagement, and it has two audiences. Non-technical stakeholders need a short executive summary that states overall risk, what an attacker could actually do, and what to fix first, in plain language. Engineers need every finding with the affected asset, evidence (requests and responses, screenshots), reproduction steps, and a severity score with its CVSS vector so they can check the math. The report should also say what was tested, when, how, and what was out of scope or not reached. Always ask for a real sample report before signing, and ask us for ours too.
- 4.1The executive summary is readable by a non-technical leader and states business impact and priorities.
- 4.2Every finding includes the affected asset, evidence, reproduction steps, and a severity score with its CVSS vector.
- 4.3The report states the scope, test dates, methodology, and limitations, including anything that was not tested.
- 4.4It records attack paths that were attempted and failed, so you can see coverage and not just the failures.
- 4.5The vendor shares a real sample report (redacted or public) before you sign.
5. Remediation and retesting: where the value is realized
A report that lists problems but leaves your team to work out the fixes gives you half the value. Remediation guidance should be specific to your technology stack, prioritized, and include a way to verify the fix. Critical findings should be escalated the day they are confirmed, not held for the final report. And the engagement should include a retest that produces an updated status for each finding. PCI DSS v4.0.1 Requirement 11.4.4 requires exactly this: exploitable vulnerabilities are corrected and the penetration testing is repeated to verify the corrections. Pair this area with our Pentest Remediation Playbook.
- 5.1Remediation guidance is specific to your stack and includes a verification step, not just a generic reference link.
- 5.2Findings come with a prioritized fix order that separates quick wins from structural fixes.
- 5.3A findings review call with the tester is included.
- 5.4A retest is included in the price, within a defined window, and produces an updated status for every finding.
- 5.5Critical findings are escalated as soon as they are confirmed, not held until the final report.
6. Compliance evidence: make sure the auditor accepts it the first time
Many pentests are bought to satisfy a requirement: PCI DSS, SOC 2, HIPAA, ISO/IEC 27001, a customer security questionnaire, or a cyber insurance renewal. Each has its own expectations. PCI DSS 11.4 requires internal and external testing, segmentation testing, and a tester with organizational independence (who does not need to be a QSA or ASV). Auditors want test dates inside the audit period and evidence that findings were tracked to closure. Confirm the vendor knows your framework before testing starts. Our Compliance-to-Pentest Matrix maps ten frameworks to their actual requirements.
- 6.1The vendor can explain your framework’s specific pentest requirements (for example PCI DSS 11.4, SOC 2, HIPAA, ISO/IEC 27001 A.8.8).
- 6.2The tester is organizationally independent of the systems tested, as PCI DSS 11.4.1 requires.
- 6.3An attestation letter or summary is available to share with customers and insurers without exposing the full report.
- 6.4Test dates, scope, and methodology are documented in a way that fits your audit period.
- 6.5The vendor can support annual testing and testing after significant changes.
Detailed Reporting and Remediation Guidance: What Each Stakeholder Should Get
Penetration testing firms and application security services firms are easiest to compare on their reports, because the report is what every stakeholder actually uses. Detailed reporting means one report that works for four audiences: non-technical leaders, engineers, auditors, and the customers or insurers who ask for proof. Remediation guidance means each finding tells the engineer what to change, in their stack, and how to confirm the fix. Use this table to check a sample report from any vendor.
| Stakeholder | What a detailed pentest report gives them | Question to ask the vendor |
|---|---|---|
| Executives and non-technical stakeholders | A one- or two-page executive summary in plain language: overall risk, what an attacker could actually do, and the top three priorities. No CVE lists or tool output. | “Show me the executive summary from a recent report. Could our CEO act on it in five minutes?” |
| Engineers and developers | Every finding with the affected asset, evidence (requests, responses, screenshots), reproduction steps, a CVSS vector, and stack-specific remediation guidance with a verification step. | “Can my team reproduce a finding from the report alone, without a call?” |
| Auditors and compliance | Scope, test dates, methodology, tester independence, limitations, and retest results showing each finding’s status at closure. | “Will this report satisfy PCI DSS 11.4, SOC 2, HIPAA, or ISO 27001 evidence requests as is?” |
| Customers, partners, and insurers | An attestation letter or summary that confirms testing was done and findings were addressed, without exposing exploit details. | “Do you provide a shareable attestation, and what does it include?” |
The Pentest Findings Remediation Process, Step by Step
A good penetration testing vendor supports the whole remediation process, not just the report. These are the six steps for taking pentest findings from report to verified closure, and what the vendor should contribute at each step.
- Triage the findings with the tester. On the findings review call, confirm each finding, check its severity against the CVSS vector, and add business context the tester could not see.
- Assign an owner and a due date to every finding. How long remediation should take after a pentest depends on severity, so set deadlines by severity in your vulnerability management policy before the test starts. PCI DSS v4.0.1 Requirement 6.3.3, for example, requires critical and high-severity security patches within one month of release.
- Fix the root cause, not just the instance. If one endpoint is missing an authorization check, review the shared code path so the same flaw does not survive on the endpoints the tester did not reach.
- Verify the fix internally. Use the report’s reproduction steps to confirm the finding no longer reproduces before you request the retest.
- Retest with the vendor. The tester repeats the attack and records an updated status for every finding. PCI DSS v4.0.1 Requirement 11.4.4 requires this repeat testing for exploitable vulnerabilities.
- Document closure. Attach the retest results to your risk register and audit evidence, and formally accept and document any risk you decide not to fix.
Step-by-step fixes for the 30 most common internal findings are in the Pentest Remediation Playbook.
Manual Penetration Testing vs. Continuous Vulnerability Management for Compliance
Continuous compliance programs run vulnerability management all year: automated scanning, patching, and tracking. A manual penetration test is a separate requirement. Scanning finds known weaknesses; a penetration tester proves which ones can be exploited and finds the authorization and business-logic flaws scanners miss. Where a compliance framework names penetration testing (PCI DSS, NY DFS Part 500, FedRAMP), scanning does not substitute for it: both are required, on different cadences. When you evaluate penetration testing services for compliance, confirm the vendor knows which of these frameworks applies to you.
| Framework | Manual penetration testing | Vulnerability scanning | Citation |
|---|---|---|---|
| PCI DSS v4.0.1 | Required: internal and external, at least annually and after significant changes; segmentation testing every 12 months (every 6 months for service providers) | Quarterly internal scans and quarterly external ASV scans | Req. 11.3, 11.4 |
| SOC 2 | Expected by auditors: penetration testing is named in the CC4.1 points of focus; annual is the de facto standard | Commonly used to satisfy CC7.1 monitoring | CC4.1, CC7.1 |
| HIPAA | Not named in the current Security Rule, but risk analysis and evaluation drive it; annual is the industry standard. A proposed Security Rule update would require it every 12 months | Proposed update would require scanning every 6 months | 45 CFR 164.308 |
| ISO/IEC 27001:2022 | Not named, but auditors expect it as evidence for technical vulnerability management; annual | Part of A.8.8 technical vulnerability management | A.8.8, A.5.36 |
| NY DFS 23 NYCRR 500 | Required: annually, from inside and outside the system boundaries, by a qualified internal or external party | Risk-based automated scans plus manual review (since May 1, 2025) | Section 500.5 |
| FedRAMP | Required: annually, by an independent third-party assessor (3PAO), following FedRAMP penetration test guidance | Monthly, with 30-day remediation for high-risk findings | FedRAMP Pentest Guidance |
Ten frameworks, with scope and sources, are in the Compliance-to-Pentest Matrix.
12 KPIs for Evaluating a Penetration Testing Vendor
Write these into the statement of work so “good” is defined before testing starts.
| KPI | What good looks like | Red flag |
|---|---|---|
| Critical-finding notification | Same business day the finding is confirmed | Criticals first appear in the final report |
| Named tester | Tester named in the SOW, and the same person writes the report | “Our team” with no names |
| Manual validation rate | 100% of reported findings confirmed by hand | Raw scanner output in the findings list |
| Reproducibility | 100% of findings include evidence and reproduction steps | Findings your team cannot reproduce |
| False positives | Zero in the final report; any your team disproves are removed and noted | Long lists padded with informational items |
| Severity scoring | CVSS vector shown for every finding, and the score matches the vector | Scores with no vector, or everything rated High |
| Remediation specificity | Every finding has a stack-specific fix and a verification step | A generic OWASP link as the only guidance |
| Executive summary | A non-technical reader can act on it in five minutes | Jargon, CVE lists, or no summary at all |
| Coverage statement | Scope, dates, methodology, and untested areas are all stated | No way to tell what was actually tested |
| Retest | Included, within a defined window (ours is 30 days), with an updated status per finding | Retest billed separately, or no retest offered |
| Tester access | Direct channel to the tester during testing and after the report | All questions routed through an account manager |
| Time to start | A committed start date once scope and access are agreed | Open-ended scheduling or a rush premium |
Pentest Vendor Comparison Scorecard
How to score and compare penetration testing providers: 30 checks, 2 points each, 60 points per vendor.
Scoring each check
- 2 points: Yes, and the vendor showed evidence (a document, a sample, a name you verified).
- 1 point: Partially, only on request, or a verbal yes with nothing to back it.
- 0 points: No, or they could not answer.
Reading a vendor’s total
A penetration testing vendor’s total out of 60 maps to one of three decisions.
Hard stops, whatever the score
- No named tester, or certifications you cannot verify.
- No sample report, or a sample that is a scanner export.
- No written authorization or rules of engagement before testing.
- No retest, so you cannot prove the fixes held.
| Evaluation criterion | Max | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| 1. Who Actually Does the Testing | 10 | |||
| 2. Methodology and Manual Depth | 10 | |||
| 3. Scope, Rules of Engagement, and Legal | 10 | |||
| 4. Report Quality for Two Audiences | 10 | |||
| 5. Remediation Guidance and Retesting | 10 | |||
| 6. Compliance and Audit Evidence | 10 | |||
| Total | 60 | |||
| Any hard stop? | — |
What’s in the PDF
A printable version to take into vendor calls and attach to your procurement file.
30-Point Scorecard
Every check with columns to score up to three vendors side by side.
12 Vendor KPIs
What good looks like and the red flag for each, ready to paste into a statement of work.
Hard Stops and Red Flags
The answers that should end a vendor conversation, whatever the price.
Standards References
Each pentest checklist item that comes from NIST SP 800-115, PCI DSS v4.0.1 Requirement 11.4, or PCI SSC guidance cites its source.
Frequently Asked Questions
How do I choose a penetration testing company?
Choose a penetration testing company by evaluating six criteria: who does the testing, methodology and manual depth, scope and rules of engagement, report quality, remediation guidance and retesting, and compliance evidence. Ask for the tester by name and verify their certifications, request a real sample report, and confirm a retest is included. Score each of the 30 checks on this page 0 to 2 and shortlist vendors that score 50 or more out of 60 with no hard stops.
What should penetration testing firms provide to non-technical stakeholders?
Penetration testing firms should give non-technical stakeholders a one- or two-page executive summary in plain language: overall risk, what an attacker could actually do, and the top priorities to fix. Detailed technical findings, evidence, and remediation guidance belong in the body of the same report for engineers, and a shareable attestation letter serves customers and insurers.
What should a penetration test report include?
A good penetration test report includes an executive summary written for non-technical readers, then every finding with the affected asset, evidence such as requests, responses, and screenshots, reproduction steps, a severity score with its CVSS vector, and remediation guidance specific to your technology. It should also state the scope, test dates, methodology, and limitations, including anything that was not tested.
What KPIs should I use to measure a pentest vendor?
Useful KPIs include critical-finding notification time (same business day is the standard to ask for), manual validation rate (100% of reported findings), reproducibility (100% of findings with evidence and steps), false positives in the final report (zero), severity scores that match their CVSS vectors, remediation guidance specific to your stack, and an included retest with an updated status for every finding.
Should a penetration test include a retest?
Yes. A retest is the only way to prove that fixes actually closed the vulnerabilities. PCI DSS v4.0.1 Requirement 11.4.4 requires that exploitable vulnerabilities are corrected and the testing repeated to verify the corrections. Ask whether the retest is included in the price, how long the window is, and whether it produces an updated status for every finding.
What certifications should a penetration tester have?
Look for hands-on, practical-exam certifications such as OffSec’s OSCP, OSWE, and OSWA, GIAC’s GPEN and GWAPT, or CREST’s CRT and CCT. Verify them through the issuer’s public credential lookup. Certifications are a floor, not a ceiling: also ask how many years the tester has spent doing offensive security work.
Does continuous vulnerability management replace a penetration test for compliance?
No. Continuous vulnerability management (automated scanning and patching) finds known weaknesses; a manual penetration test proves which ones can be exploited and finds authorization and business-logic flaws that scanners miss. PCI DSS, NY DFS Part 500, and FedRAMP each require both, on different cadences: for example, PCI DSS requires quarterly scans and at least annual penetration testing.
What are the steps to remediate penetration test findings?
Remediate penetration test findings in six steps: triage the findings with the tester, assign an owner and due date by severity, fix the root cause rather than the single instance, verify the fix internally with the report's reproduction steps, retest with the vendor to get an updated status for every finding, and document closure in your risk register and audit evidence.
Which compliance frameworks require manual penetration testing?
PCI DSS v4.0.1 (Requirement 11.4), NY DFS 23 NYCRR 500 (Section 500.5), and FedRAMP require penetration testing outright, at least annually. SOC 2 and ISO/IEC 27001:2022 do not name it as mandatory, but auditors expect it as evidence. The current HIPAA Security Rule does not name it, though a proposed update would require it every 12 months.
Sources and Standards Referenced in This Pentest Checklist
This penetration testing vendor checklist cites only what these primary sources say. The KPI targets and scoring rubric are Voke Cyber’s own recommendations, drawn from more than 1,000 engagements, not requirements of any standard. PTES (the Penetration Testing Execution Standard) is referenced as a methodology name.
- NIST SP 800-115, Technical Guide to Information Security Testing and Assessment: four-phase penetration testing methodology (planning, discovery, attack, reporting) and rules of engagement.
- PCI DSS v4.0.1 and the PCI SSC Penetration Testing Guidance information supplement: Requirements 6.3.3, 11.3, and 11.4.1 through 11.4.6; tester qualifications and report content.
- OWASP Web Security Testing Guide (WSTG): web application testing methodology.
- OWASP Mobile Application Security Testing Guide (MASTG): mobile application testing methodology.
- FIRST Common Vulnerability Scoring System (CVSS): severity scores and vector strings.
- FedRAMP: annual third-party penetration testing and monthly scanning for cloud service providers.
Related Reading and Resources
Use these penetration testing resources alongside the checklist when you compare vendor quotes.
“Ask every vendor for a sample report and the name of the person who will test. Those two answers tell you most of what you need to know.”
Voke Cyber publishes its starting prices, puts you in direct contact with the tester doing the work, and includes a retest with every pentest. Score us with this checklist alongside anyone else you are considering.
Comparing Pentest Vendors Right Now?
Send us your scope and we will have a scoped price back to you within 24 hours. Bring this checklist to the call.