[{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/tags/architecture/","section":"Tags","summary":"","title":"Architecture","type":"tags"},{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/posts/","section":"Blog","summary":"","title":"Blog","type":"posts"},{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/tags/cloud/","section":"Tags","summary":"","title":"Cloud","type":"tags"},{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/tags/data/","section":"Tags","summary":"","title":"Data","type":"tags"},{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/tags/mid-market/","section":"Tags","summary":"","title":"Mid-Market","type":"tags"},{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/","section":"Pragmatic Architect","summary":"","title":"Pragmatic Architect","type":"page"},{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/tags/pragmatism/","section":"Tags","summary":"","title":"Pragmatism","type":"tags"},{"content":"","date":"14 December 2025","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"I recently worked with one of my long-time customers - a catering company I\u0026rsquo;ve been working with for years. €500M+ revenue, 5.000 employees across multiple sites, sophisticated operations managing workplace canteens, cafeterias, and event catering.\nTheir CTO had built a working demand forecasting prototype on a Raspberry Pi 5. It worked. They wanted to productionize it - run it at scale in the cloud instead of under someone\u0026rsquo;s desk. The hyperscaler solution architect and hyperscaler partner sales showed up with a proposal: data lake architecture, Lake Formation for governance, Glue for ETL, EMR for processing, comprehensive data catalog, multi-phase implementation. They had 30GB of total data in Redshift.\nThe right solution? Deploy SageMaker Unified Studio, point it to Redshift, and keep it simple. Five days from kickoff to forecasting models running in production.\nThis isn\u0026rsquo;t a story about incompetent salespeople or a bad hyperscaler. This is a story about a systemic industry blind spot.\nThe Pattern Nobody Talks About # Mid-market enterprises - roughly €500M to €5B revenue, 1.000 to 10.000 employees - have a specific IT profile that the entire tech industry misunderstands.\nOn paper, this looks like an enterprise IT organization. In practice, it isn\u0026rsquo;t.\nThey might have 100 people in IT. Vendors see this number and think \u0026ldquo;enterprise IT organization.\u0026rdquo; But look closer:\n40 Windows admins and desktop support 20 helpdesk managing tickets 15 network operations keeping infrastructure running 10 people managing SAP/ERP systems 5 security and compliance staff 20-30 people doing cloud infrastructure, data platforms, ML projects, and application development These aren\u0026rsquo;t software companies. They\u0026rsquo;re manufacturing firms, catering operations, logistics providers, service companies. They might have a couple hundred embedded software engineers writing code for their machines or products - and that team is world-class at what they do. But the team handling cloud infrastructure, data platforms, ML projects, and application development (including web interfaces for their products)? Right-sized at 20-30 people.\nAnd here\u0026rsquo;s the key: their products are B2B or internal. They might have 10.000 total users, not millions. The requirements are completely different from running a B2C webshop or consumer application. \u0026ldquo;Infinite scale\u0026rdquo; is never in scope. They don\u0026rsquo;t need architecture designed for viral growth or massive traffic spikes.\nVendors see €500M revenue and 100+ IT headcount and pull out the enterprise playbook — one designed for organizations with hundreds of engineers, silos, and coordination failure.\nThe fatal mistake: trying to \u0026ldquo;scale down\u0026rdquo; enterprise solutions for mid-market execution.\nYou Can\u0026rsquo;t Scale Down Complexity to Get Simplicity # Here\u0026rsquo;s what vendors fundamentally misunderstand: you can\u0026rsquo;t take a solution designed for Netflix\u0026rsquo;s 1.000 microservices and 50 siloed teams and just \u0026ldquo;simplify it\u0026rdquo; for a mid-market company.\nEnterprise architecture isn\u0026rsquo;t about traffic scale. It\u0026rsquo;s about organizational distrust at scale. Mid-market companies usually don\u0026rsquo;t have that problem.\nA company with 30GB or even 100TB of data doesn\u0026rsquo;t need Lake Formation, Glue, EMR, and governance layers. They need a data warehouse (Redshift, Snowflake, BigQuery, ClickHouse). The \u0026ldquo;enterprise data platform\u0026rdquo; adds operational complexity, requires more people to maintain, and solves problems they don\u0026rsquo;t have.\nA team of 25 developers doesn\u0026rsquo;t need \u0026ldquo;simplified service mesh.\u0026rdquo; They need services that can talk to each other directly because everyone sits in the same office and actually communicates.\nThe industry keeps trying to sell:\nOrchestration layers for teams that don\u0026rsquo;t have coordination problems Service isolation for people who share the same codebase Governance frameworks for organizations where everyone knows what\u0026rsquo;s happening Multi-phase implementations for companies that need solutions in weeks, not quarters The Root Cause: Engagement-Driven Content # Why does this happen? Look at what dominates industry content:\n\u0026ldquo;How Netflix built their data platform across 50 teams\u0026rdquo; \u0026ldquo;Scaling microservices to 1.000 services at Uber\u0026rdquo;\n\u0026ldquo;Data mesh architecture for siloed organizations\u0026rdquo;\nThis content generates engagement. It gets clicks, conference slots, and social media shares. It\u0026rsquo;s genuinely valuable - for companies facing similar scale and organizational complexity as Netflix or Uber.\nThe problem: hyperscalers build tools based on these patterns. Solution architects learn from these examples. Blogs and documentation showcase these massive-scale solutions. The entire industry optimizes for problems that 99% of companies will never have.\nAnd for mid-market enterprises specifically: these are mature, stable companies with predictable growth. They\u0026rsquo;re not in hyper-growth mode. They won\u0026rsquo;t suddenly need to coordinate 50 siloed teams or scale to millions of users. The coordination and scaling problems that these enterprise solutions solve? They\u0026rsquo;re never coming.\nWhat doesn\u0026rsquo;t generate engagement?\n\u0026ldquo;We used Redshift and dbt for our 50GB of data and it works fine\u0026rdquo; \u0026ldquo;Our 20 developers share a monorepo and deploy together\u0026rdquo; \u0026ldquo;We kept it simple with YAGNI and KISS principles\u0026rdquo; Boring. Unsexy. No conference talk in that. But it\u0026rsquo;s what actually helps most companies.\nThe Pattern in Practice # The same misunderstanding appears everywhere:\nAI Orchestration platforms designed for teams that don\u0026rsquo;t trust each other\u0026rsquo;s APIs. Your webhook comes in with data - but the \u0026ldquo;enterprise solution\u0026rdquo; makes you write it to a presigned URL, wait for detection, trigger orchestration, run your job, write results to another presigned URL, detect completion, then call back. When your 25-person team could just trigger a Step Function directly with the webhook data.\nBedrock AgentCore Gateway for managing MCP servers across agents. Makes sense when you have multiple teams building different agents that need to discover shared tools. For a team of 20 developers sharing the same codebase? Your tools are just Python functions in a shared library. You don\u0026rsquo;t need service boundaries and discovery layers for people who sit in the same room.\nThese aren\u0026rsquo;t bad solutions. They\u0026rsquo;re solving real problems - for organizations with siloed teams, political boundaries, and coordination complexity. Mid-market companies with 20-30 engineers working together don\u0026rsquo;t have these problems.\nWrap # Mid-market enterprises (€500M-5B revenue) have 20-30 people doing cloud, data, and ML work. They\u0026rsquo;re mature, stable, B2B or internal. Teams actually talk to each other.\nThey don\u0026rsquo;t need:\nData lakes for 30GB (modern DWH handles 100TB+) AI orchestration theater for teams that share code Service mesh for people in the same office Scaled-down Netflix patterns They need: purpose-built simple solutions, not enterprise complexity.\nThe industry optimizes for Netflix-scale content because it generates engagement. For everyone else, it\u0026rsquo;s a trap.\nSomeone needs to write boring content that actually helps. This is me trying.\nUntil vendors design for this reality, architects in the mid-market will keep winning by ignoring the playbooks.\nBuilding for mid-market enterprises? The boring, right-sized solutions nobody writes about are what most companies actually need.\n— The Pragmatical Architect\n","date":"14 December 2025","externalUrl":null,"permalink":"/posts/2025-12-14-why-tech-industry-misunderstands-mid-market/","section":"Blog","summary":"","title":"Why the Entire Tech Industry Misunderstands Mid-Market Enterprises","type":"posts"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/adot/","section":"Tags","summary":"","title":"ADOT","type":"tags"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/agentcore/","section":"Tags","summary":"","title":"AgentCore","type":"tags"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/ai-agents/","section":"Tags","summary":"","title":"AI Agents","type":"tags"},{"content":"If you\u0026rsquo;re trying to integrate Amazon Bedrock AgentCore Observability with AWS Lambda, the docs will confuse the hell out of you.\nNot because they\u0026rsquo;re wrong - they\u0026rsquo;re just written for containerized environments (ECS, EKS) and assume you know the difference between collector-based and embedded agent patterns.\nYou probably don\u0026rsquo;t know which is which. Neither did I - spent a day debugging why none of the documented env vars worked.\nHere\u0026rsquo;s what actually works.\nContext: What We\u0026rsquo;re Building # We\u0026rsquo;re running AI agents in AWS Lambda as async backend tasks - event-driven, not human-interactive. Think SQS triggers, EventBridge events, background processing with AI reasoning.\nThis is different from AgentCore Runtime (the managed, conversational runtime). When you run agents outside AgentCore Runtime, you lose session metrics but keep traces, metrics, and logs.\nI\u0026rsquo;ll write a separate post on which runtime to use when. Today: just getting observability working in Lambda.\nThe Problem with the Docs # AgentCore observability docs assume you\u0026rsquo;re running agents in containers with this architecture:\nBut modern AWS Lambda with ADOT layer uses a completely different pattern:\nNo collector. No sidecar. In-process embedded agent.\nThe docs don\u0026rsquo;t clarify this, so you end up cargo-culting env vars from three different sources that contradict each other.\nPrerequisites # Step 1: Enable CloudWatch Transaction Search # This is a one-time setup per region. You need this before anything else works.\nOption 1: Using AWS CLI\nCreate the resource policy:\naws logs put-resource-policy \\ --policy-name MyResourcePolicy \\ --policy-document \u0026#39;{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [{ \u0026#34;Sid\u0026#34;: \u0026#34;TransactionSearchXRayAccess\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: { \u0026#34;Service\u0026#34;: \u0026#34;xray.amazonaws.com\u0026#34; }, \u0026#34;Action\u0026#34;: \u0026#34;logs:PutLogEvents\u0026#34;, \u0026#34;Resource\u0026#34;: [ \u0026#34;arn:aws:logs:region:account-id:log-group:aws/spans:*\u0026#34;, \u0026#34;arn:aws:logs:region:account-id:log-group:/aws/application-signals/data:*\u0026#34; ], \u0026#34;Condition\u0026#34;: { \u0026#34;ArnLike\u0026#34;: { \u0026#34;aws:SourceArn\u0026#34;: \u0026#34;arn:aws:xray:region:account-id:*\u0026#34; }, \u0026#34;StringEquals\u0026#34;: { \u0026#34;aws:SourceAccount\u0026#34;: \u0026#34;account-id\u0026#34; } } }] }\u0026#39; Configure trace destination:\naws xray update-trace-segment-destination --destination CloudWatchLogs Optional - configure sampling:\naws xray update-indexing-rule \\ --name \u0026#34;Default\u0026#34; \\ --rule \u0026#39;{\u0026#34;Probabilistic\u0026#34;: {\u0026#34;DesiredSamplingPercentage\u0026#34;: 10}}\u0026#39; Option 2: Using Console\nOpen CloudWatch console → Settings → X-Ray traces tab Transaction Search section → View settings → Edit Enable Transaction Search Select \u0026ldquo;Enable Transaction Search\u0026rdquo; and set sampling percentage (1% free tier) Wait until \u0026ldquo;Ingest OpenTelemetry spans\u0026rdquo; shows Enabled AWS Docs: Enabling Transaction Search\nLambda Setup # Runtime Version: Python 3.12 # Python 3.13 is supported but has a bug as of writing. Use Python 3.12 - it\u0026rsquo;s the latest stable version.\nCritical: Your Strands layer must be built with Python 3.12 to match the Lambda runtime (Pydantic is version-sensitive).\nExample Agent Code # Here\u0026rsquo;s a simple Strands agent in Lambda:\nimport json import logging from strands import Agent, tool from strands.models import BedrockModel logger = logging.getLogger() logger.setLevel(logging.INFO) @tool def weather(): \u0026#34;\u0026#34;\u0026#34;Get weather\u0026#34;\u0026#34;\u0026#34; return \u0026#34;sunny\u0026#34; model = BedrockModel(model_id=\u0026#34;eu.anthropic.claude-haiku-4-5-20251001-v1:0\u0026#34;) agent = Agent( model=model, tools=[weather], system_prompt=\u0026#34;You\u0026#39;re a helpful assistant. You can tell the weather.\u0026#34; ) def lambda_handler(event, context): user_input = \u0026#34;How is the weather\u0026#34; response = agent(user_input) result = { \u0026#34;statusCode\u0026#34;: 200, \u0026#34;body\u0026#34;: json.dumps({\u0026#34;message\u0026#34;: response}) } logger.info(\u0026#34;Operation completed successfully\u0026#34;) return result Framework Note: I\u0026rsquo;m using Strands because it\u0026rsquo;s my default agent framework in AWS, but this setup works the same with LangChain, LangGraph, CrewAI, or any other agent framework that supports OpenTelemetry instrumentation. The three environment variables and ADOT layer configuration remain identical.\nCreate Strands Lambda Layer # Build the layer with platform-specific dependencies:\n# Match your Lambda runtime version python3.12 -m pip install strands-agents \\ --platform manylinux2014_x86_64 \\ --only-binary=:all: \\ -t python/ # Or with OpenTelemetry extras python3.12 -m pip install \u0026#39;strands-agents[otel]\u0026#39; \\ --platform manylinux2014_x86_64 \\ --only-binary=:all: \\ -t python/ # Package it zip -r strands-layer.zip python/ Upload to Lambda as a layer.\nAWS Docs: Lambda Layers\nLambda Configuration # 1. Add Layers\nADOT Lambda Layer (find ARN for your region: ADOT Lambda Layers) Your Strands layer 2. Enable Monitoring Tools\nLambda Console → Your Function → Configuration → Monitoring and operations tools → Edit\nEnable both:\n✅ Application Signals ✅ Lambda service traces 3. Add IAM Policy\nAttach CloudWatchLambdaApplicationSignalsExecutionRolePolicy to your Lambda execution role.\n4. Environment Variables\nHere\u0026rsquo;s the magic. Just three environment variables:\nAGENT_OBSERVABILITY_ENABLED=true AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=none That\u0026rsquo;s it. Nothing else.\nWhy This Works # The ADOT Lambda layer uses an embedded agent pattern, not a collector.\nWhen you set AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument, the layer wraps your function with auto-instrumentation that exports directly to CloudWatch and X-Ray.\nWhy OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=none?\nAWS ADOT Lambda layer tries to be lightweight by default - it disables heavier instrumentations (logging integrations, deep library hooks) to keep cold starts low. Setting this to none tells ADOT: \u0026ldquo;no, instrument everything.\u0026rdquo; Without this, you\u0026rsquo;ll miss critical agent framework telemetry.\nAll those other OTEL env vars you see in the docs (OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_LOGS_HEADERS, OTEL_PYTHON_DISTRO, etc.) are for:\nContainerized environments with collector sidecars (ECS/EKS) Legacy Lambda collector pattern (outdated) AgentCore Runtime-specific configuration In modern Lambda with ADOT layer, they\u0026rsquo;re either redundant or conflict with the auto-configuration.\nAGENT_OBSERVABILITY_ENABLED=true tells your agent framework to enable observability. The ADOT layer handles everything else.\nWhat You Get # After deployment and invocation, your traces show up in CloudWatch Application Signals, X-Ray Service Map, and CloudWatch Logs Insights:\nYou\u0026rsquo;ll see:\nAgent invocations Tool calls Model interactions Latency metrics Error traces What you DON\u0026rsquo;T get:\nSession metrics (only available in AgentCore Runtime) Session metrics make sense for conversational, human-interactive agents. If you\u0026rsquo;re running event-driven backend agents in Lambda, you don\u0026rsquo;t need them.\nThe Docs Confusion Explained # If you follow the AgentCore observability setup docs, you\u0026rsquo;ll see instructions for containerized environments:\nWhat the docs say:\n# Configure AWS environment variables export AWS_ACCOUNT_ID=\u0026lt;account id\u0026gt; export AWS_DEFAULT_REGION=\u0026lt;default region\u0026gt; export AWS_REGION=\u0026lt;region\u0026gt; export AWS_ACCESS_KEY_ID=\u0026lt;access key id\u0026gt; export AWS_SECRET_ACCESS_KEY=\u0026lt;secret key\u0026gt; # Configure OpenTelemetry environment variables export AGENT_OBSERVABILITY_ENABLED=true export OTEL_PYTHON_DISTRO=aws_distro export OTEL_PYTHON_CONFIGURATOR=aws_configurator export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=\u0026lt;YOUR-LOG-GROUP\u0026gt;,x-aws-log-stream=\u0026lt;YOUR-LOG-STREAM\u0026gt;,x-aws-metric-namespace=\u0026lt;YOUR-NAMESPACE\u0026gt; export OTEL_RESOURCE_ATTRIBUTES=service.name=\u0026lt;YOUR-AGENT-NAME\u0026gt; The observability configuration docs show even more:\nAGENT_OBSERVABILITY_ENABLED=true OTEL_PYTHON_DISTRO=aws_distro OTEL_PYTHON_CONFIGURATOR=aws_configurator OTEL_RESOURCE_ATTRIBUTES=service.name=\u0026lt;agent-name\u0026gt;,aws.log.group.names=/aws/bedrock-agentcore/runtimes/\u0026lt;agent-id\u0026gt;,cloud.resource_id=\u0026lt;AgentEndpointArn:AgentEndpointName\u0026gt; OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=/aws/bedrock-agentcore/runtimes/\u0026lt;agent-id\u0026gt;,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_TRACES_EXPORTER=otlp Why this doesn\u0026rsquo;t work in Lambda:\nThese docs assume collector pattern architecture - your code exports OTLP to a sidecar collector (see the container diagram above).\nBut Lambda with ADOT layer uses embedded agent pattern (see the FaaS diagram above) - no collector, no sidecar, in-process instrumentation.\nWhat breaks if you use the docs\u0026rsquo; env vars:\nAWS credential exports aren\u0026rsquo;t needed (Lambda IAM role handles this) OTEL_PYTHON_DISTRO=aws_distro and OTEL_PYTHON_CONFIGURATOR=aws_configurator configure collector-less direct export - contradicts the collector architecture they\u0026rsquo;re documenting OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf configures OTLP export to a collector that doesn\u0026rsquo;t exist in Lambda OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=... tries to configure collector endpoint headers aws.log.group.names=/aws/bedrock-agentcore/runtimes/\u0026lt;agent-id\u0026gt; is AgentCore Runtime-specific (managed runtime, not your Lambda) These env vars either conflict with ADOT layer auto-configuration or are completely unnecessary.\nThe docs aren\u0026rsquo;t wrong - they\u0026rsquo;re container-first. Lambda FaaS with embedded ADOT is a different execution model. The docs don\u0026rsquo;t bridge this gap.\nDebugging Tips # If traces don\u0026rsquo;t appear:\nWait 5-10 minutes after enabling Transaction Search Check IAM permissions include CloudWatchLambdaApplicationSignalsExecutionRolePolicy Verify both ADOT and Strands layers are attached Confirm Python runtime is 3.12 Check CloudWatch Logs for OTEL export errors If you see \u0026ldquo;Failed to export\u0026rdquo; errors, you probably have conflicting OTEL env vars from the docs. Remove everything except the three listed above.\nWrap # TL;DR for Lambda + AgentCore Observability:\nEnable CloudWatch Transaction Search (per-region, one-time) Python 3.12 runtime Add ADOT + Strands layers Enable Application Signals + Lambda service traces Add IAM policy Set exactly 3 env vars That\u0026rsquo;s the working config. No collector. No 10+ environment variables. No cargo-culting from container docs.\nIf you\u0026rsquo;re running agents in Lambda for async backend processing, this gets you full observability without the session metrics you don\u0026rsquo;t need anyway.\nRunning AI agents in production? The boring backend stuff nobody talks about is where the real work happens.\n— The Pragmatical Architect\n","date":"10 December 2025","externalUrl":null,"permalink":"/posts/2025-10-10-agentcore-observability-with-aws-lambda/","section":"Blog","summary":"","title":"Amazon Bedrock AgentCore Observability with AWS Lambda: The Working Config","type":"posts"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"AWS","type":"tags"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/bedrock/","section":"Tags","summary":"","title":"Bedrock","type":"tags"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/lambda/","section":"Tags","summary":"","title":"Lambda","type":"tags"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/observability/","section":"Tags","summary":"","title":"Observability","type":"tags"},{"content":"","date":"10 December 2025","externalUrl":null,"permalink":"/tags/opentelemetry/","section":"Tags","summary":"","title":"OpenTelemetry","type":"tags"},{"content":" How I Fell Into the Compliance Rabbit Hole # AWS re:Invent announced Amazon Bedrock AgentCore Evaluations. Managed LLM-as-judge for your AI agents — sounds great, right?\nSo I did what any architect does: I read the documentation to understand how to integrate it.\nWhat started as \u0026ldquo;let me evaluate this new AWS service\u0026rdquo; turned into a deep dive that revealed a serious GDPR compliance risk hiding in plain sight. Not with Bedrock AgentCore Evaluations itself — but with the entire observability pattern we\u0026rsquo;ve all been using for AI agents.\nAnd if you\u0026rsquo;re building AI agents in production, you need to know about this.\nTo be clear: this isn\u0026rsquo;t an \u0026ldquo;AWS is bad\u0026rdquo; problem. It exists across AWS, Azure, GCP, Datadog, and pretty much every observability stack. These systems were designed for debuggability and performance — not per-user erasure under GDPR.\nThe Two Integration Paths # Amazon Bedrock AgentCore Evaluations gives you two options:\nOnline evaluation: Reads conversations directly from CloudWatch Logs + OpenTelemetry\nOn-demand evaluation: Requires extra steps on the agent side (at which point, why not just run your own LLM-as-judge?)\nI started with online evaluation because it seemed simpler.\nThe CloudWatch Trap # Online evaluation means shipping your entire conversation to CloudWatch Logs.\nI live in the EU. I architect data systems. I breathe GDPR compliance. And here\u0026rsquo;s the truth:\nYour conversation history WILL contain PII. Always.\nUsers naturally share:\nNames, locations, email addresses (direct identifiers) Job details, family information, health data (indirect identifiers) Unique writing styles and behavioral patterns Contextual details that combine to identify them You cannot scrub this data without destroying the functionality of your agent. And even if you think you can anonymize it — GDPR says pseudonymized data is still personal data. You must delete the actual data, not just the user-to-ID mapping.\nThe killer: GDPR\u0026rsquo;s right to erasure. Users can request deletion within 1 month. CloudWatch? Partition-based deletion only. You can\u0026rsquo;t delete a single user\u0026rsquo;s history.\n\u0026ldquo;But wait,\u0026rdquo; you might say, \u0026ldquo;what about Amazon Bedrock AgentCore Observability?\u0026rdquo;\nYeah. That\u0026rsquo;s just a wrapper around CloudWatch + X-Ray. Same immutable storage. Same partition-based deletion. Same GDPR problem.\nThen Came the Traces # Reading further, AWS documentation says:\nAgentCore Evaluations integrates with popular agent frameworks including Strands and LangGraph with OpenTelemetry and OpenInference instrumentation libraries…\nWait. Traces?\nClassic distributed traces don\u0026rsquo;t contain message content, right? So I dug into Strands documentation (my framework of choice).\nThe oh-no moment # Check the example trace output. Messages are part of the trace data.\nThen I went down the OpenTelemetry GenAI spec rabbit hole. According to the official examples, traces explicitly include:\nFull user messages AI responses Retrieved context Function arguments This creates a serious GDPR problem.\nWhere Your PII Actually Lives # Here\u0026rsquo;s what nobody tells you when you set up AI agent observability:\nYour PII is now in at minimum three places:\nYour database (hopefully deletable) CloudWatch Logs (immutable, partition-based deletion only) OpenTelemetry traces → Datadog / New Relic / Honeycomb (also immutable) Typical pattern:\nUser → AI Agent → Database (deletable) ↓ CloudWatch / X-Ray (immutable) ↓ OpenTelemetry → Datadog (immutable) Why this breaks GDPR expectations:\nNo granular deletion Full message content in immutable storage Retention often 30–90+ days Multiple processors involved You can delete records from PostgreSQL instantly.\nBut the same conversation still lives in CloudWatch and Datadog — and you can\u0026rsquo;t remove it.\nThe Third-Party Processor Nightmare # You typically have two tracing paths:\nAWS X-Ray\nStays within AWS Covered by your existing AWS DPA Still immutable, partition-based deletion Third-party (Datadog, New Relic, Honeycomb)\nAdditional data processors Cross-company data flow Often US-based No granular deletion What I see in practice:\nTeams start with X-Ray, then export to Datadog anyway. Or they send logs to both CloudWatch and another platform.\nNow full conversations exist in multiple immutable systems across multiple processors.\nThis is often happening by default, just by following \u0026ldquo;observability quickstart\u0026rdquo; guides.\nThe Access Control Blindspot # Even before deletion, there\u0026rsquo;s another issue:\nEveryone with access to observability tools can read user conversations.\nDevelopers debugging bugs? See PII. SREs investigating latency? See PII. Support teams? See PII. Contractors? Also see PII. This violates core GDPR principles:\nData minimization Purpose limitation Access control / least privilege And this is the default behaviour of many AI agent frameworks.\nArchitectural separation is not optional. It\u0026rsquo;s the only sane approach.\nWhat You Should Actually Do # Here\u0026rsquo;s the pattern that actually works.\n1. Architectural Separation (Non-negotiable) # Deletable storage (PostgreSQL, DynamoDB, etc.)\nFull chat messages User profiles All PII Granular, immediate deletion Strict access control Immutable systems (CloudWatch, X-Ray, Datadog)\nMetadata ONLY Token counts Latency Error codes Hashed user IDs (one-way) No message content 30-day retention max When there\u0026rsquo;s no PII, developers can safely access these systems.\n2. Fix Your OpenTelemetry Instrumentation # Default instrumentation is the real danger.\nConfigure it to send:\n✅ Latency\n✅ Token usage\n✅ Error type\n✅ Hashed user ID\n❌ Full prompts\n❌ Model responses\n❌ Retrieved context\n❌ Function arguments with PII\nI\u0026rsquo;ve already opened a feature request with Strands to make this the default. AI frameworks should be compliant by design, not compliant by extra effort.\n3. The 7–14 Day Rule (If you can\u0026rsquo;t fix it yet) # If you absolutely must send message content (legacy reasons, framework limitations, transition period), your retention matters.\n30 days – common and defensible, but extremely tight 90+ days – almost impossible to justify for chat data 7–14 days – much safer, still operationally useful This is mitigation, not a solution.\nThe real fix is: no content in immutable systems.\nQuick Audit Checklist # If you\u0026rsquo;re running AI agents in prod, ask yourself:\nDo any logs or traces contain full prompts or responses? Can I delete a single user\u0026rsquo;s data from all systems? Who inside the company can query these logs? Is retention longer than 30 days? Do third-party tools receive this data? If any of these make you uncomfortable — you\u0026rsquo;ve found your starting point.\nThe Bottom Line # The default observability pattern for AI agents creates major GDPR risk.\nIt\u0026rsquo;s not about CloudWatch vs Datadog. It\u0026rsquo;s about what you send to them.\nAs long as AI frameworks export full conversations to immutable systems, you have:\nNo granular deletion Excessive retention Over-broad internal access Multiple data processors The fix is architectural and simple:\nChat content → Deletable storage only Observability → Metadata only No messages in traces/logs Strict access control What\u0026rsquo;s Next # Honestly? I\u0026rsquo;m still processing this.\nWhat started as \u0026ldquo;let me evaluate a new AWS service\u0026rdquo; turned into realizing that the entire AI observability ecosystem has a compliance problem baked into its defaults.\nIf you\u0026rsquo;re building AI systems in production:\nAudit your pipeline. Fix your retention. Separate your storage.\nWe\u0026rsquo;ll figure out the rest as we go.\nBuilding AI in the EU? Send me your war stories. Misery loves company.\n— The Pragmatical Architect\n","date":"6 December 2025","externalUrl":null,"permalink":"/posts/2025-12-06-aws-bedrock-agentcore-gdpr/","section":"Blog","summary":"","title":"AWS Bedrock AgentCore Evaluations: The GDPR Risk Nobody's Talking About","type":"posts"},{"content":"","date":"6 December 2025","externalUrl":null,"permalink":"/tags/compliance/","section":"Tags","summary":"","title":"Compliance","type":"tags"},{"content":"","date":"6 December 2025","externalUrl":null,"permalink":"/tags/gdpr/","section":"Tags","summary":"","title":"GDPR","type":"tags"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]