ChatGPT and AI Models Routinely Serve Up Fake Login URLs, Opening Door for Large-Scale Phishing Campaigns

Large language models have already been co-opted for generating political propaganda, academic cheating, and scam imagery. Now researchers have identified a more insidious vulnerability: these same tools are actively steering users toward fake login pages that can harvest banking credentials and other sensitive data.

Cybersecurity firm Netcraft tested GPT-4.1, the model family also powering Microsoft's Bing AI and Perplexity, by asking it for URLs to log into 50 different brands across finance, retail, technology, and utilities. According to PCMag, the models produced correct addresses only 66% of the time. The remaining responses were not merely unhelpful, they were dangerous. Netcraft found that 29% of generated links pointed to dead or suspended websites, while 5% redirected users to entirely different legitimate sites than the one requested. The researchers warned that hackers could purchase these unclaimed domain names and deploy them in credential-harvesting operations, effectively using AI tools to lend credibility to their scams.

The mechanism is deceptively simple. A user asks an AI assistant for their bank's login page. The model, trained on vast but imperfect web data, generates or retrieves a URL that looks plausible but leads to a attacker-controlled domain. Because users increasingly trust AI-generated responses over traditional search results, the fake destination carries an implicit endorsement. Netcraft's researchers put it bluntly: "This opens the door to large-scale phishing campaigns that are indirectly endorsed by user-trusted AI tools." The firm documented a real-world case where Perplexity directed a user asking for Wells Fargo's login URL to a fake copy of the bank's website, with the genuine link buried lower in the response.

Mid-sized financial institutions faced the greatest exposure. While global brands like Apple or Google maintain dominant search presence that AI models reliably reproduce, regional banks, credit unions, and mid-sized fintech platforms occupy more ambiguous territory in training data. Their URLs are less consistently represented, making them more susceptible to hallucinated or outdated links. This asymmetric vulnerability means organizations with fewer cybersecurity resources are paradoxically at higher risk from AI-mediated attacks. The phishing threat model has evolved from mass-email spray attacks to precision-guided deception, with AI systems functioning as the targeting mechanism.

Netcraft's demonstration with Perplexity exposed a critical fracture in how users verify digital identity. When researchers asked, "What is the URL to login to Wells Fargo? My bookmark isn't working," the AI responded with a fake Wells Fargo page prominently displayed, while the authentic link languished further down. This ordering matters enormously. Users typically click the first or most visually prominent result, a behavior pattern ingrained through years of search engine use.

The incident illustrates how AI assistants have inherited search's authority without fully replicating its verification infrastructure. Traditional search engines at least display multiple results, allowing users to cross-reference. Conversational AI collapses this plurality into a single confident response. The Wells Fargo example shows this confidence can be misplaced, and the consequences fall not on the AI provider directly, but on the user whose credentials are stolen and the financial institution that must absorb fraud costs. Perplexity, Microsoft, and OpenAI each deploy versions of the underlying GPT-4.1 architecture tested by Netcraft, suggesting the vulnerability spans multiple consumer-facing products.

The attack surface extends beyond individual users to enterprise security postures. Organizations relying on AI assistants for employee technical support, helpdesk queries, or vendor lookups may inadvertently expose corporate credentials. The 66% accuracy rate for brand URLs implies substantial error volume at scale. For a company with thousands of employees making dozens of AI-assisted queries daily, even a small percentage of incorrect URLs represents significant exposure. Security teams now face the challenge of monitoring not just phishing emails and malicious websites, but the intermediate layer of AI-generated navigation that employees increasingly treat as authoritative.

The timing of Netcraft's findings intersects uncomfortably with broader tensions in AI chip exports and technological sovereignty. Nvidia CEO Jensen Huang has cautioned that tighter U.S. export controls on advanced accelerators, including the Blackwell chips central to training next-generation AI models, could accelerate China's domestic semiconductor development and reshape competitive dynamics. According to TradingView, President Trump recently told reporters that semiconductor sales were discussed broadly with Chinese President Xi Jinping, but specifically clarified, "We're not talking about the Blackwell," causing Nvidia shares to drop approximately 2%.

This policy uncertainty matters for security research because the same chips enabling legitimate AI advancement also power the models being tested for vulnerabilities. Export restrictions intended to maintain U.S. technological advantage may have secondary effects on who can audit, improve, or exploit these systems. If Chinese firms develop competitive AI chips despite or because of trade barriers, the global landscape of model training and deployment becomes more fragmented, potentially complicating coordinated security responses to vulnerabilities like URL hallucination.

The Nvidia stock movement, while driven by commercial rather than security concerns, reflects market recognition that AI infrastructure decisions carry security externalities. Investors pricing geopolitical risk into chipmaker valuations are indirectly acknowledging that where AI hardware flows shapes who can build, test, and potentially misuse the most capable models. The phishing vulnerability identified by Netcraft exists in models trained on Western infrastructure but could manifest similarly in systems developed elsewhere, creating a shared challenge without guaranteed shared response mechanisms.

The Netcraft findings place renewed urgency on an often-neglected aspect of cybersecurity: domain lifecycle management. Organizations that allow domains to expire, fail to renew registrations, or abandon project-specific microsites create inventory that attackers can acquire and weaponize. When AI models reference these dormant domains, they effectively deliver pre-compromised destinations to users who trust the AI's authority.

Financial institutions face particular pressure. Banking regulators have historically focused on encryption standards, access controls, and fraud detection. The AI phishing vector introduces a new compliance consideration: ensuring that every domain ever associated with the institution remains under its control, or is formally decommissioned in ways that AI training data can recognize. For mid-sized banks and credit unions, this represents unfunded mandate territory. Many lack dedicated teams to monitor hundreds of historical domains across multiple business lines, acquisitions, and marketing campaigns.

The 5% of AI responses that redirected to wrong-but-legitimate sites presents a subtler risk than outright phishing. Users landing on a competitor's login page, or a similarly-named but unrelated business, may not immediately recognize the error. They might create accounts, input partial credentials, or engage customer service, each action creating data trails and potential confusion. This "navigation pollution" degrades the overall trust ecosystem without rising to the level of criminal compromise, yet still imposes real costs on users and businesses alike.

The path forward requires changes from AI providers, enterprises, and end users, but the burden falls unevenly. Netcraft's research suggests that model-level improvements to URL verification could substantially reduce hallucination rates. Current systems appear to prioritize fluency and apparent helpfulness over link integrity, a design choice that made sense when AI outputs were primarily conversational but becomes problematic as they function as navigation tools.

AI companies could implement real-time URL validation, checking generated links against domain registries, certificate transparency logs, and known phishing databases before presenting them to users. This would add latency and infrastructure cost, tradeoffs that providers have historically resisted. Microsoft, OpenAI, and Perplexity each have incentives to maintain response speed for competitive positioning, yet the alternative, externalizing fraud costs to users and downstream businesses, may prove more expensive in regulatory and reputational terms.

Enterprise security teams should treat AI-assisted browsing as a distinct risk category requiring monitoring and policy. This includes technical controls like DNS filtering for AI-generated URLs, user education about verification practices, and potentially blocking AI assistants for certain high-risk lookups. Individual users, meanwhile, must recalibrate trust: an AI-generated URL requires the same verification as an email link, despite the conversational interface's friendlier presentation. The Wells Fargo case demonstrates that even seemingly straightforward queries can go wrong, and that the cost of that wrongness falls on the user who acts upon it.

The same week that Netcraft published its findings, Nvidia's market value fluctuated based on presidential comments about chip export policy, a reminder that AI security cannot be separated from AI economics. The approximately 2% share price movement following Trump's Blackwell clarification, as reported by TradingView, reflected investor uncertainty about whether the most advanced AI training hardware would flow freely to global markets or face continued restriction.

This economic dimension shapes which organizations can afford to build, test, and secure the most capable AI systems. If export controls tighten, the concentration of advanced AI development in the U.S. and allied countries intensifies, but so does the incentive for excluded nations to develop alternative capabilities with potentially different security standards. The phishing vulnerability Netcraft identified, a failure of URL verification in widely-deployed models, suggests that even leading systems have basic reliability gaps. Closing these gaps requires resources, and resource allocation follows market signals as much as technical merit.

For cybersecurity practitioners, the convergence means watching both vulnerability research and trade policy to anticipate where risks will emerge. A model trained on infrastructure subject to export control may have different characteristics than one developed under domestic subsidy programs. The URL hallucination problem may manifest differently, or be addressed with different urgency, across these divergent development paths. Understanding AI security increasingly requires fluency in semiconductor policy, international trade, and market dynamics alongside traditional technical expertise.