This case places Ravi between two things that appear, at first glance, to point in the same direction but are actually in tension — visible, measurable improvement in public order, and a credible allegation that this improvement has been achieved through a system that discriminates against a specific community. The dilemma is genuine precisely because both the improved law and order and the community’s right to non-discriminatory policing are legitimate claims the state owes its citizens.

Facts

• AI predictive policing system operational for six months, using biometric matching against a data library
• System flagged an immigrant, low-income neighbourhood as a centre for gang violence and drug trafficking
• Visible improvement in public order followed increased patrolling, preventive detentions, and checkposts in that area
• Community leaders, civil rights lawyers, and human rights activists allege the system is built on biased historical data reflecting past discriminatory policing
• Residents report a climate of tension, worsened by having no visibility into the data recorded against their names

Stakeholders

• Residents of the flagged neighbourhood, bearing the direct impact of increased surveillance and detention
• Civil rights groups and activists, representing concerns about systemic bias and due process
• The wider district population, benefiting from improved public order
• Ravi and the police force, responsible for both effective policing and constitutional fidelity

(a) Ethical Issues, Including Bias, in AI-Driven Policing

• Algorithmic Bias and Disparate Impact — an AI system trained on historical arrest data does not learn “where crime occurs,” it learns “where police have historically made arrests,” which itself reflects past discriminatory policing patterns rather than an objective crime distribution
• Feedback Loop / Self-Fulfilling Prediction — flagging a neighbourhood leads to more patrolling there, which leads to more arrests there, which then reinforces the system’s own prediction in the next cycle, regardless of the area’s actual underlying crime rate relative to others
• Proxy Variables for Protected Characteristics — factors like immigration status or income level can function as indirect proxies for ethnicity or origin, allowing discrimination to persist even without any explicit discriminatory input
• The Black-Box / Explainability Problem — residents flagged by the system have no way to understand or contest the basis of the data recorded against them, undermining natural justice and the basic right to know the case against oneself
• Privacy and Purpose Limitation — biometric data captured from a crowd, including from individuals with no connection to any crime, raises serious concerns about the scope and duration for which such data is retained and used
• Automation Bias — officers may over-rely on algorithmic flags as objective fact, reducing independent human judgement precisely in situations where contextual nuance matters most

(b) Alternatives Available to Ravi, and the Justified Course of Action

Option 1 — Continue the system unchanged, citing the visible public order improvement

• Merits: Preserves the operational gains already achieved
• Demerits: Ignores a credible bias allegation, risking entrenched discrimination and eventual legal or constitutional challenge

Option 2 — Discontinue the AI system entirely in response to the memorandum

• Merits: Immediately addresses the discrimination concern
• Demerits: Discards a genuinely useful tool without first establishing whether the bias claim is accurate or fixable, and may reverse the public order gains without addressing their root cause

Option 3 — Suspend unsupervised algorithmic flagging pending an independent bias audit, while retaining human-in-the-loop oversight for any ongoing operations

• Merits: Directly investigates the actual claim rather than assuming it true or false; keeps a functioning, more accountable system in place during the review
• Demerits: Requires time and technical capacity to conduct a proper audit, during which some operational efficiency may be temporarily reduced

Option 3 is the most ethically sound course. It treats the bias allegation with the seriousness it deserves — since historical policing data across many contexts has been shown to reflect precisely this kind of skew — without abandoning a system that may still hold value once corrected.

Justification

• Sir Robert Peel’s foundational principle of policing holds that police effectiveness ultimately depends on public approval and voluntary cooperation, not on enforcement statistics alone — a system generating deep community mistrust undermines this even as it appears to “work” on paper
• The Puttaswamy judgment’s proportionality framework requires that any privacy-infringing state action be necessary, proportionate, and subject to procedural safeguards — continuing unaudited biometric surveillance without a grievance mechanism fails this standard
• A transparent grievance redressal mechanism, allowing residents to know and contest data recorded against them, should be instituted regardless of the audit’s outcome, since the current opacity is itself a due-process failure independent of whether the underlying bias claim is proven

Way Forward

Technology that improves policing outcomes on average can still fail the basic test of justice if those gains are achieved by systematically burdening one community more than others. Ravi’s responsibility is not to choose between effective policing and fair policing, but to insist that the two are only genuinely compatible when the system generating the results can withstand independent scrutiny. An unaudited tool that cannot explain itself to the people it affects has not yet earned the trust its results seem to suggest it deserves.