AI governance for private companies

Artificial intelligence has moved rapidly from experimentation to deployment. For fast-growing private companies, the opportunity is substantial: AI can improve productivity, enhance customer experience, accelerate decision-making, and help teams scale without adding commensurate headcount. AI tools are also now increasingly embedded in core business processes and capable of taking actions with limited human involvement.

Notably, nearly three in four companies plan to deploy agentic AI within two years despite only one in five having a mature governance model for autonomous agents.¹

This gap creates risk. Companies that move quickly without appropriate oversight may expose themselves to operational, legal, cybersecurity, customer, and reputational challenges. We believe it is critical for companies to adopt best practices that create the confidence and discipline required to safely deploy AI across their businesses.

Here, we outline the risks AI presents to private companies, explore its evolving regulatory landscape, offer best practices for adoption, and share resources for companies.

Figure 1

AI risks for private companies

Effective AI governance starts with identifying where AI can create risk across data, operations, customer outcomes, third-party relationships, and a range of other domains.

Data, privacy, and intellectual property risks

AI tools increasingly create, process, and rely on large volumes of data across internal systems, customer interactions, and external sources. As adoption grows, companies can lose visibility into how sensitive information moves through their environment, who can access it, and how it is being used. These challenges can create privacy, intellectual property, compliance, and reputational risks. Key examples include the unauthorized use or disclosure of sensitive information, uncertainty regarding ownership and rights associated with AI-generated content, and difficulties demonstrating appropriate governance to customers, regulators, or other stakeholders.

Data, privacy, and intellectual property concerns can be magnified when organizations deploy AI tools before establishing clear controls over data access, retention, and use. Risks may also emerge through employee adoption of unsanctioned AI tools that operate outside established security, privacy, and governance controls (“shadow AI”).

Agentic and autonomous action risk

The risk with agentic AI moves beyond producing a “wrong answer” to taking a “wrong action.” As AI systems gain the ability to access information, interact with applications, and execute multi-step workflows, failures may become more difficult to predict, detect, or reverse. These risks can increase when AI tools are connected to sensitive data, customer-facing channels, payment systems, or other core business processes.

For example, while not a fully autonomous agent, a US auto dealership’s chatbot was manipulated into agreeing to sell a new vehicle for US$1 after a user tested the system’s limits². The incident illustrates a broader governance concern: AI tools can be pushed outside their intended scope when guardrails and escalation controls are weak.

Reliability and performance risk

An AI tool’s performance can also change over time. Model updates, new data sources, evolving business processes, and changing operating environments can all affect reliability, accuracy, and outcomes. Companies that rely on AI for important decisions or customer interactions may experience operational disruption, poor decisions, or degraded performance if systems are not regularly monitored, tested, and reassessed.

Unlike traditional software, AI systems often continue to evolve after deployment, making ongoing oversight critical.

Customer impact and business risk

As AI becomes more deeply embedded in customer-facing processes and business operations, the consequences of failure increase. AI systems increasingly influence recommendations, decisions, and interactions that affect customers directly. This makes it more likely that errors, bias, or inappropriate outputs translate into tangible harm.

Risks may also arise when organizations pursue AI-driven workforce reductions or customer-service automation without fully understanding where human judgment, escalation, empathy, or institutional knowledge remain essential.

For instance, health insurers have faced lawsuits challenging the use of AI and algorithmic tools in claims denials, with plaintiffs alleging that automated systems contributed to improper coverage decisions and adverse customer outcomes³. These legal challenges highlight the broader business risks associated with inadequately governed AI systems, including regulatory scrutiny, litigation, operational disruption, and reputational damage.

Vendor dependence and AI supply chain vulnerabilities
Most companies will build their businesses with AI tools developed by external vendors. As a result, they may become dependent on third-party models, infrastructure providers, and software vendors they do not control. Changes in pricing, model availability, performance, functionality, or terms of service can create operational, financial, and strategic risk. This is particularly impactful when AI is embedded in critical business processes.

Limited visibility into how third-party models are trained, maintained, updated, and governed may also make it difficult to identify risks related to data provenance, intellectual property, security, or compliance.

Regulatory landscape

Most private companies are not yet subject to a comprehensive AI compliance regime, making effective AI governance largely a matter of self-governance today.

However, expectations around AI governance are increasing rapidly from regulators, customers, employees, investors, and business partners. Leaders should expect greater scrutiny of how AI systems are deployed, monitored, and controlled, particularly where they affect customers, employees, or critical business processes.

• Europe: Plan for comprehensive AI regulation
  In the European Union, the EU AI Act of 2024 is being implemented in phases through 2027. By the end of the rollout, organizations developing or deploying higher-risk AI systems will be subject to one of the most comprehensive AI governance frameworks globally. This includes requirements related to transparency, human oversight, risk management, documentation, post-market monitoring, and AI literacy.
  
  Even companies without European operations may encounter EU AI Act expectations through enterprise customers, business partners, or procurement requirements. In practice, many global technology companies are already designing AI systems and governance processes to align with the Act's requirements ahead of full implementation, making its influence broader than the formal compliance timeline may suggest.
• United States: Plan for evolving governance expectations
  The United States has not adopted a comprehensive federal AI law comparable to the EU AI Act. Expectations regarding responsible AI governance continue to evolve through industry standards, sector-specific guidance, customer requirements, and state-level regulation.
  In the absence of federal guidance, standards-setting activity has accelerated across industry groups, standards bodies, and government partnerships. Frameworks such as the NIST AI Risk Management Framework and emerging ISO standards are increasingly influencing customer expectations, procurement processes, industry practices, and sector-specific guidance. Whether through future regulation or industry self-governance, these standards are likely to shape how organizations are expected to develop, deploy, and oversee AI systems.
  
  At the same time, state-level regulation continues to expand, creating a patchwork of evolving requirements. Regulatory activity has been most concentrated in areas where AI may affect employment decisions, access to financial products, healthcare, insurance coverage, housing, education, and other high-impact outcomes. Companies operating in these sectors (or using AI to support these types of decisions) should expect heightened scrutiny and continually evolving regulations.
  
  Ongoing debate regarding the appropriate balance between federal and state oversight may create additional uncertainty, including in areas where states have already enacted AI-related requirements.

Given the pace of change, we believe most companies will be better served by building adaptable governance capabilities than by attempting to comply with individual regulations one at a time. While specific regulations and standards continue to differ across jurisdictions and industries, common themes are emerging around accountability, transparency, human oversight, risk management, documentation, and AI literacy.

Organizations that establish these capabilities early will be well positioned to adapt as legal requirements, industry standards, and stakeholder expectations continue to evolve.

Six AI governance best practices

Below, we share best practices for private companies building AI governance capabilities.

1. Establish clear ownership and accountability
   • Designate a senior executive or accountable function for AI governance.
   • Create a lightweight cross-functional review process involving product, engineering, legal, compliance, security, procurement, and relevant business leaders.
   • Ensure higher-risk AI use cases are escalated to leadership or the board where appropriate.
   • Avoid making governance purely a technical or legal function; it should connect to business strategy, risk, and customer outcomes
2. Start with a business objective instead of a specific AI tool
   • Require teams to define the problem AI is intended to solve before selecting tools or models.
   • Identify expected business and/or customer outcomes upfront.
   • Track measurable results such as cost, speed, quality, revenue, customer satisfaction, error reduction, or risk reduction.
   • Define what would trigger redesign, escalation, or decommissioning.
3. Know where AI is being used and govern based on risk
   • Maintain a practical inventory of AI tools, vendors, models, and use cases.
   • Include internal tools, customer-facing AI, third-party AI-enabled software, and employee-built or unsanctioned uses where possible.
   • Prioritize governance attention based on customer impact, data sensitivity, regulatory exposure, operational importance, and autonomy.
   • Use risk tiering to avoid over-governing low-risk uses while applying stronger controls where potential consequences are higher.
4. Safeguard data, secure systems, and diligence vendors
   • Control what data AI systems can access and how confidential, proprietary, or customer data may be used.
   • Set minimum requirements for data provenance, privacy, cybersecurity, and intellectual-property review before AI tools or vendors are approved.
   • Conduct AI-specific vendor diligence, including data use, model updates, liability, transparency, portability, and lock-in.
   • Provide employees with approved tools, clear usage rules, and practical training so they do not route around governance.
5. Protect customers and other affected stakeholders
   • Assess who could be harmed if AI fails or produces systematic errors.
   • Pay special attention to consequential decisions involving access, eligibility, pricing, claims, recommendations, healthcare, employment, education, financial products, or other sensitive contexts.
   • Build in disclosure, human review, appeal or escalation paths, and monitoring for disparate or unfair outcomes.
   • Avoid deploying AI where the company cannot explain, monitor, or defend the result.
6. Monitor continuously and prepare for failures
   • Test AI before deployment and monitor it after launch.
   • Track accuracy, reliability, bias, drift, explainability, misuse, and repeated failures.
   • For agentic or autonomous AI, add permissions, action logs, human approval for high-stakes actions, and override mechanisms.
   • Establish incident response protocols for pausing, escalating, remediating, and communicating AI-related failures.

Bottom line

AI governance is about ensuring that AI’s rapid innovation creates value without creating unnecessary risk. We believe the companies most likely to benefit from AI will not necessarily be those that deploy it fastest, but rather those that combine experimentation with accountability, customer awareness, and disciplined execution. Companies that establish strong governance foundations will therefore, in our view, be better able to scale AI responsibly, adapt to change, and build trust with customers, employees, investors, and regulators. Critically, governance will need to scale with it, becoming more rigorous as AI becomes increasingly embedded in products, operations, and decision-making.

Appendix A: Prepare for investor questions

Investors are increasingly seeking greater transparency into how companies govern AI, a trend that has expanded beyond the technology sector and is beginning to influence expectations across industries⁴.

AI strategy and value creation

• How is the company using AI today, and which use cases are most important to growth, customer value, operational efficiency, or competitive advantage?
• How does the company prioritize AI investments and measure business outcomes, including ROI, productivity gains, revenue impact, or risk reduction?

Governance and accountability

• What AI governance framework is in place, and who is accountable for overseeing AI-related opportunities, risks, and decision-making?
• How are higher-risk AI use cases identified, reviewed, approved, and monitored throughout their lifecycle?

Risk management and responsible use

• How does the company assess potential impacts of AI on customers, employees, and other stakeholders before deployment?
• What controls are in place to ensure appropriate human oversight, intervention, and accountability for AI-enabled decisions and actions?
• How does the company manage data privacy, cybersecurity, intellectual property, and third-party AI vendor risks?
• How does the company monitor AI systems for reliability, accuracy, unintended outcomes, and changing performance over time?

Organizational readiness and future adaptation

• How does the company train employees to use AI responsibly and effectively?
• How does the company adapt its governance, controls, and practices as AI technology, regulation, customer expectations, and business use cases evolve?

Appendix B: Seven additional best practices for AI developers

Organizations developing AI systems may face additional responsibilities related to product design, testing, transparency, and oversight. The following practices may help AI developers manage these risks throughout the product lifecycle.

1. Set the tone
   • Establish governance, accountability, and controls across the AI product lifecycle (design, development, monitoring, and retirement).
2. Anticipate misuse
   • Understand customer systems, data flows, access rights, and interoperability risks.
   • Build safeguards, monitoring, and escalation paths.
   • Understand customer systems, data flows, access rights, and interoperability risks.
3. Allocate responsibility
   • Define responsibilities for testing, disclosures, security, incident response, oversight, and liability for errors, harms, and noncompliance.
4. Test for fairness and performance
   • Test for bias, disparate impact, accuracy gaps, and degradation throughout the lifecycle.
   • Use lawful, representative data and document sources, limits, and mitigations.
5. Be transparent with users
   • Disclose uses, limitations, oversight needs, known risks, and when users are interacting with AI.
6. Build efficiently
   • Treat model efficiency, energy, and water impacts as product requirements.
   • Measure training and inference impacts, optimize resource use, and evaluate cloud and data-center providers.
7. Review data labor practices
   • Evaluate labor practices, worker protections, and quality controls for data labeling and annotation.

Appendix C: AI governance resources

A growing body of guidance is available to help organizations govern AI effectively. The following resources are among the most widely used and practical starting points.

• NIST AI Risk Management Framework
  • A practical framework for identifying, assessing, and managing AI-related risks across an organization. Widely referenced by regulators, customers, and enterprises, it provides a useful starting point for companies building AI governance programs.
• NIST Generative AI Profile (A1 600-1)
  • Companion guidance to the NIST AI Risk Management Framework focused on generative AI. Addresses risks such as hallucinations, prompt injection, privacy, intellectual property, cybersecurity, and third-party model dependence.
• OWASP Top 10 for LLM Applications
  • A widely used cybersecurity resource that helps organizations identify and mitigate common vulnerabilities in AI applications, including prompt injection, data leakage, excessive permissions, insecure integrations, and agentic AI risks.
• ISO Artificial Intelligence Standards
  • International standards that provide guidance on AI governance, risk management, and implementation. Companies may find ISO/IEC 42001 (AI Management Systems) particularly relevant as customers increasingly evaluate suppliers’ AI governance practices.
• EU AI Act including governance expectations in Europe
  • The European Union’s risk-based AI regulation. Relevant not only for companies operating in Europe, but also for organizations seeking to understand emerging regulatory expectations around transparency, governance, risk management, and human oversight.

• OECD AI Principles
  • Foundational international principles for trustworthy AI, emphasizing transparency, accountability, robustness, human-centered design, and responsible governance. Useful for organizations establishing high-level AI governance objectives and policies.

¹Deloitte. “State of AI in the Enterprise” (2026) | ²Venture Beat. “A Chevy for $1? Car dealer chatbots show perils of AI for customer service.” (2023) | ³Healthcare Finance. “Class action lawsuit against UnitedHealth’s AI claim denials advances.” (2025) | ⁴Harvard Law School Forum on Corporate Governance. “AI in Focus in 2025: Boards and shareholders set their sights on AI” (2025)