[{"title":"Residential Proxies and the AI Scraper Problem","permalink":"https://gnosyslambda.github.io/en/posts/2026-07-12-residential-proxy-ai-scraper/","summary":"Why residential proxies and AI scrapers are putting pressure on the open web, based on LWN's reporting and the NetNut seizure case.","content":" In brief: Residential proxies and AI scrapers are not merely a traffic-growth problem. They shift the costs of competition for training data onto other people’s devices and independent websites.\nWhat happened On July 10, 2026, LWN reported that large-scale scraper traffic routed through residential proxies had intensified again. The concern is not clearly identified automated access, such as search-engine crawlers. It is traffic that appears briefly from vast numbers of home and mobile-network IP addresses, makes only a few requests from each address, and disappears.\nThat makes the problem much harder for site operators to handle. Over a few hours, requests can arrive from millions of unique IP addresses, with each IP fetching a page only two or three times before vanishing. User-Agent strings are not trustworthy. A bot may be suspected because it does not request images or CSS, but by then that IP often will not be used again.\nLWN identified residential proxies as a major route for this traffic. Ordinary users’ TVs, streaming boxes, phones, and apps can receive commands from a central control server, fetch web pages, and return the results. In some cases, malware infection is involved; in others, the arrangement is disclosed through consent language buried in free VPNs or app SDKs.\nThe confirmed facts can be separated as follows.\nItem Confirmed facts What remains uncertain LWN traffic Large-scale scraper attacks were continuing as of July 2026 The identities of the ultimate buyers have not been made public NetNut and Popa The FBI and industry partners seized domains related to NetNut, and security firms describe at least 2 million devices connected to the Popa botnet The purpose of every customer’s use and whether a specific AI company used the network directly Defenses Proof-of-work systems such as Anubis, login barriers, CAPTCHAs, and data-poisoning tools are spreading Which defenses will be effective over the long term remains unclear Krebs on Security’s coverage of the NetNut seizure made the discussion more concrete. The FBI seized hundreds of domains associated with NetNut, with Google, Lumen, and Shadowserver reportedly cooperating. Citing security-firm analysis, Krebs explained that NetNut infrastructure was connected to the Popa botnet and that household devices such as smart TVs and streaming hardware had been used as proxy nodes.\nThere is an important line to keep clear. LWN wrote that there is no evidence that large AI-model companies directly use these residential proxy networks. By contrast, crawlers from publicly identified large-model companies generally disclose their User-Agent strings and follow rules such as robots.txt to some degree.\nEven so, that gap is precisely what made the community uneasy. Someone is paying to buy this traffic. Someone is taking material from the public web for training data or automation. Yet the costs are dispersed among site operators, real readers, and the owners of compromised devices.\nWhy people reacted The debate over residential proxies extends beyond whether web scraping is good or bad. It combines questions of who can read the public web, under what conditions, and who bears the cost.\nTrust is the first thing to break down. Web-server operators have traditionally used signals such as IP addresses, User-Agent strings, robots.txt, and request frequency to distinguish people, search engines, and harmful bots. Residential proxies deliberately disrupt that signal system. Because the requests appear to come from household internet connections, blocking them can also block real readers.\nAuthorization is another major concern. Installing a free VPN or an app does not necessarily mean a user fully understands that their device may become a transit point for a third party’s large-scale scraping. Even if some form of consent exists in the terms of service, ordinary users are unlikely to anticipate that their networks could be used for ad fraud, account-takeover attempts, or mass content collection.\nThe costs also shift to operators. Independent media outlets and small open-source project sites rarely have the extensive defensive infrastructure available to major platforms. As scraper traffic grows, so do server costs and the time required for caching, rate limiting, bot detection, and log analysis. Put up defenses, and real readers may instead have to solve CAPTCHAs or sign in.\nflowchart LR A[Data consumer] --\u003e|Buys proxy access| B[Residential proxy provider] B --\u003e|SDK, VPN, or compromised app| C[Home devices and mobile networks] C --\u003e|Requests that look human| D[Independent websites] D --\u003e|Returns pages and content| C C --\u003e|Sends collected data| B B --\u003e A D --\u003e|Rising defense costs| E[Operators] D --\u003e|CAPTCHA and login barriers| F[Real readers] There is regulatory risk as well. Krebs’s reporting on NetNut shows that this issue can go beyond a debate over etiquette and lead to law-enforcement seizures. Responsibility becomes blurred when compromised devices, ambiguous consent, and resale or white-label arrangements are mixed together. A proxy vendor may point to legitimate market research or price comparison, while its actual customers can rent the network for far broader purposes.\nMisunderstandings are common, too. Many discussions treat all AI scrapers as a single category. LWN’s distinction is more useful in practice: crawlers that identify themselves and follow rules to some extent, commercial proxies intended to bypass defenses, and malware-based botnets are all forms of automated access, but they create different operational risks.\nIn day-to-day operations, trying to block all automated access in response to abusive traffic often backfires. It can also disrupt search visibility, archive preservation, accessibility tools, and security monitoring. The problem is not automation itself, but automation that avoids identification and accountability.\nThe core issue as I see it The heart of this issue is not scraping but concealed origin. The public web exists to be read, but that does not mean every method of access is justified. Millions of distributed requests that masquerade as human readers are not simply operating within the rules of public access; they are closer to breaking those rules.\nResidential proxies target exactly the part of the system that is hardest to defend. Datacenter IP addresses are comparatively easy to block using reputation signals. Residential and mobile IP addresses, by contrast, are intermingled with real users. Raise the blocking threshold and readers are inconvenienced; lower it and the server may struggle to cope.\nProof of work is not a complete answer either. Systems such as Anubis, which impose a computational cost on visitors, may buy small sites some time. But as LWN notes, if an attacker can use millions of other people’s devices, the computational cost falls not on the attacker but on compromised users.\nThe structure resembles cloud-cost optimization. A system is balanced only when the party that directly pays the costs also has decision-making power. With residential proxies, the data consumer gains the benefit, the proxy provider earns revenue, and the affected sites and device owners share the costs. The market’s cost signals are broken.\nThat is why it is not enough to view this solely as a contest over the performance of bot-blocking tools. Defensive tools are necessary, but tools alone cannot fill the gaps in consent, accountability, and transparency. Once app stores, SDK distribution channels, VPN providers, proxy resellers, and data buyers can all say they are merely intermediaries, operators have to distrust every request.\nGoogle’s response to NetNut also looks incomplete on this point. Detecting NetNut-infected apps in the Play Store is helpful. But questions remain about why apps with residential-proxy functionality could circulate so easily through app stores, how SDKs should obtain user consent, and how far resellers should verify where their customers will use the network.\nThis is not only a problem for small sites. If public documentation, package repositories, community forums, bug trackers, and personal blogs all face the same pressure, the web’s basic usability changes. Those who keep material open absorb the cost of defense; those who close it lose discoverability and public value.\nWhat to look for next The next time news appears about AI scrapers, residential proxies, or botnet seizures, start by separating the sources of the traffic. Even when the activity is all called crawling, the assessment changes depending on whether it comes from an identifiable crawler, a commercial proxy, or a botnet built from compromised devices.\nFor operators, the following guidelines are practical.\nDo not rely on User-Agent strings alone. Consider whether images and CSS are requested, along with session persistence. Before blocking entire IP ranges, consider protecting high-cost endpoints, strengthening caching, and limiting anonymous requests. Classify automated access that should be preserved separately, including search engines, the Internet Archive, and security scanners. Evaluate CAPTCHAs and proof of work with the costs imposed on real readers included. When introducing a login barrier, separate documents that should remain part of the public web from functions that can reasonably be closed off. When adopting an app or SDK, check whether its terms allow device-network traffic to be used for third parties. Policy needs clear standards, too. If a proxy provider claims ethical sourcing, users should at least be able to learn what traffic they are relaying. If the arrangement includes resellers, there should be verification of the end customer’s intended use. App stores should make network-proxy capabilities more explicit within their permission models.\nAI companies cannot avoid the same questions. If they operate public crawlers, they should clearly state what data they collect, how they handle robots.txt and deletion requests, and whether they use third-party data brokers or proxies. No specific company should be accused without evidence, but opaque supply chains invite suspicion.\nSite operators have no completely clean option. Leave the site open and it can be abused; close it and readers and search engines are inconvenienced. Future decisions are therefore likely to focus less on perfect blocking than on limiting the scope of harm.\nReturning to the initial tension, this is less about who has the right to read the web than about who bears the cost of that reading. If the public web is to remain open, automated access must not hide what it is, and costs and accountability must travel with it. If that boundary collapses, what remains is more logins, more CAPTCHAs, and a narrower web.\nReferences [Selected article] An update on residential proxies and the scraper situation - LWN / Hacker News Best [Related] FBI Seizes NetNut Proxy Platform, Popa Botnet - Krebs on Security "},{"title":"Grok Build CLI Security Concerns: Full Repository Uploads","permalink":"https://gnosyslambda.github.io/en/posts/2026-07-12-grok-cli-repo-upload-security/","summary":"An examination of claims that Grok Build CLI sends entire repositories, Git history, and .env secrets to xAI's cloud—and the security questions those claims raise.","content":" In short: The crucial question in debates about AI coding CLIs is not only how well a model understands code. More immediately, it is when, where, and under what consent an entire repository and its secrets move. The Grok Build CLI report is unsettling not simply because of a possible product mistake, but because the trust boundary assumed by AI development tools is still far from settled.\nWhat happened A report posted to Reddit’s LocalLLaMA community describes observing Grok Build CLI v0.2.93 with mitmproxy. According to the report, the entire repository was uploaded as a Git bundle to Google Cloud infrastructure associated with xAI, even after the user explicitly instructed the tool not to open any files.\nThe reporter included do not read or open any files in the prompt. They also placed a canary file in the repository and claim that the file was restored when they git cloned the captured upload. The report further says that files read by the CLI were sent separately to cli-chat-proxy.grok.com, including values such as API_KEY and DB_PASSWORD from .env files.\nIt is important to separate what has been observed from what is being inferred here.\nCategory Details Scope supported by the available evidence The Reddit post and its linked reproduction material concern Grok Build CLI v0.2.93, a particular network capture, and a particular repository setup. The reporter’s claims The full Git history and .env secrets were uploaded, and the model-improvement opt-out did not prevent the upload itself. Points that still need independent confirmation xAI’s official explanation, server-side retention periods, deletion policies, and whether upload scope is consistent across versions. The practical question already worth asking If an AI CLI can send a repository remotely, its toggle labels must be verified against its actual data flows. Treating this solely as a story about xAI would miss the broader issue. Developers reacted because AI coding tools have entered the most sensitive trust boundary in the local development environment.\nCode-generation tools need context to understand a repository. But the risk differs greatly depending on whether that context is a few open files, a partial index, the entire Git history, or the full working directory including .env files.\nflowchart LR A[Developer's local repository] --\u003e B[AI CLI] B --\u003e C{Context collection scope} C --\u003e|Open files| D[Relatively limited transfer] C --\u003e|Entire repository| E[Large-scale source-code transfer] C --\u003e|Git history| F[Previously deleted secrets re-exposed] C --\u003e|.env / configuration files| G[API-key and password exposure risk] D --\u003e H[Model response] E --\u003e I[Remote storage/processing policy needs review] F --\u003e I G --\u003e I Why did people react? The first reason the developer community reacted strongly is that the term opt-out appears to have worked differently from what users expected.\nUsers commonly assume that disabling a toggle such as Improve the model means less of their code leaves their machine. But if the report is accurate, that toggle governs whether data is used for training rather than whether it is uploaded to perform the task at all. From the product’s perspective, those may be separate policies. From the user’s perspective, both involve their repository leaving their environment.\nThe second reason is Git history.\nSecrets absent from the current working tree may still exist in older commits. Even after an API key has been rotated, internal URLs, customer names, infrastructure design, and deployment practices can remain in history. If an AI CLI creates a complete Git bundle, it can move far more information than the files the user can see in front of them.\nThe third reason is .env.\nIn practice, .env is one of the files most likely to be handled carelessly. It can exist locally even when it was never committed to Git. Once an AI tool reads local files and sends them remotely, established controls such as GitHub secret scanning and pull-request review do not help. The incident route now exists outside the repository.\nThe MIT LLM Serve Dashboard example offers a useful contrast. The related Reddit post introduces a dashboard for GPU utilization, per-model throughput, KV/context fill, and system status on a local LLM serving box. It also emphasizes that the frontend is a single index.html, the backend is a Python file built on the standard library, and there are no external requests.\nBy functionality alone, it is a small tool. Yet the community response was less about performance metrics than about its trust model. Claims that there are no external requests, no build step, and that the observed data stays local show what developers increasingly regard as reassuring.\nThe CISA case belongs on the same axis. According to TechCrunch, in May 2026 a CISA contractor employee uploaded sensitive keys and credentials that could be used to access government systems to a public GitHub repository. CISA responded after security researchers and a journalist flagged the issue. In its post-incident account, CISA said it had no prepared incident playbook at the time and had to develop response procedures as the incident unfolded.\nThe two cases look different, but they lead to the same question.\nOnce sensitive development material crosses a boundary, how can an organization know it happened, who must be notified, what must be discarded, and which keys must be rotated? The AI CLI debate is a usability debate, but it is also a debate about incident readiness.\nThe core issue as I see it This is not an argument that remote processing by AI coding tools is inherently wrong.\nLarge models may require remote inference. Repository-level context can also produce better answers. Insisting on local models alone brings different trade-offs in quality, speed, cost, and maintenance.\nBut uploading a repository should not be treated as a minor side effect of an autocomplete feature. It packages source code, configuration, history, developer habits, and sometimes information close to customer data, then sends it to an external system in one operation.\nIn day-to-day engineering work, this discussion usually unfolds along these lines:\nDevelopers want better context. Security teams want to know the scope of data transfer. Legal and privacy teams ask about retention and the purpose of further processing. Platform teams look for a way to enforce a consistent policy across every developer environment. Product teams want to use the tools without sacrificing productivity. To balance those interests, data flows must be disclosed before UI toggles.\nFor example, if an AI CLI performs the following actions, documentation and product UI should describe them separately.\nQuestion What needs to be answered What will it send? Open files, selected folders, the full repository, Git history, and whether ignored files are included. When will it send it? Immediately on launch, after prompt entry, or when a particular command runs. Where will it send it? API domains, object storage, region, and third-party processors. How long will it retain it? Retention periods for logs, caches, bundles, and session data. What will it use it for? Response generation, quality improvement, model training, or abuse detection. How can it be prevented? Allowlists, denylists, .gitignore support, secret redaction, and organization-level policy. What is particularly troubling in this report is that the meaning of opt-out may conflict with users’ intuition. Saying that data will not be used to improve the model is not the same as saying that it will not be uploaded. Yet many users treat the two as the same safeguard.\nIf a product does not explain this distinction plainly, users may learn only afterward that their code has already moved. Once that trust is lost, feature quality alone is unlikely to restore it.\nWhat to evaluate next When assessing AI coding-tool news, look at transfer boundaries before model performance.\nFirst, determine whether full-repository upload is the default—and treat the inclusion of Git history as a separate question. History may contain secrets and internal information that developers believe were deleted.\nSecond, examine how the tool handles .gitignore, .env, and secret patterns. The fact that a file is not committed to Git does not reduce the risk of an AI CLI transferring it. A tool that can read local files creates a new path regardless of Git tracking status.\nThird, do not take opt-out language at face value; read its scope. Training opt-out, telemetry opt-out, prompt-logging opt-out, and file-upload opt-out are different controls. If a product blends them together on one screen, operational risk remains.\nFourth, check whether organization-level controls exist. Relying on every individual developer to be careful does not scale. For company repositories, CLI policy files, centrally managed configuration, network egress controls, secret scanning, and key-rotation procedures should work together.\nFifth, define an incident playbook in advance. As the CISA case illustrates, credential-exposure incidents become harder after discovery. If no one has determined which repositories were transferred, which keys may have been included, which accounts to revoke, and whom to notify, the response becomes an improvised meeting.\nThis is not an argument to stop using AI coding tools. Standards are needed precisely because they are likely to remain in use.\nA good AI CLI is not merely a tool that can read a great deal of code. It is one whose users can predict what it reads. A better AI CLI also lets organizations restrict and audit that scope.\nThe question left by the Grok Build CLI report is simple: do we want a tool that understands our repositories better, or one that lets us know how far our repositories have traveled? We should now demand both.\nFurther reading Selected source: Grok Build CLI uploads your whole repo, full git history + .env secrets, to xAI’s cloud, and the opt-out doesn’t stop it (Reddit LocalLLaMA) Related: Grok Build CLI wire-capture evidence gist (GitHub Gist) Related: MIT LLM Serve Dashboard I am making open source (Reddit LocalLLaMA) Related: US cybersecurity agency CISA had to build its incident playbook during the incident, agency reveals (TechCrunch) "}]